MAKEPP_SCANNING(1)

💖 MAKEPP_SCANNING 💖

okay bestie, welcome to the makepp manual — this one's about scanning, aka how makepp sniffs out include files and all the other sneaky hidden dependencies you did NOT write down. ✨

✨ NAME ✨

makepp_scanning — how makepp finds include files and other hidden dependencies 🕵️♀️

---

🌸 DESCRIPTION 🌸

So like, makepp can figure out extra dependencies or targets for certain commands it already knows stuff about. And this is huge for C/C++ compilation, because listing every single include file a source file depends on by hand? Absolutely not, it's way too error-prone. But by peeking at the compilation command and the source files themselves, makepp can work out exactly which object files need rebuilding when some include file changes. iconic behavior tbh. 💅

Example time: given a rule

foo.o:                      # Usually %.o: %.c, just for illustration
    time -p /bin/libtool -bar /usr/bin/cc -c -I somewhere foo.c

makepp knows that time and libtool must be skipped 🚫 and that cc is the actual command to be parsed here. It understands that foo.c is the input file and therefore a dependency of this rule. And on top of that, it'll scan that file looking for include statements — also in the directory somewhere — because it understood the command options. no cap. 🧠

ngl there are actually three steps to what's historically known as scanning:

1. The rule action gets split into lines (continuation lines count as one). Each line (except builtins and Perl blocks) is lexically analyzed as one or more Shell commands. Redirections are recognized as inputs or outputs to this rule. The first word of each command is looked up (with its directory part but, if not found, again without it) to find a parser for it. These become optional dependencies — they're built if possible, but ignored if not found, since makepp can't know which part of a complex command is actually run.

Commands in backquotes are analyzed but not executed. (Often execution is important, but this would be a major interference by makepp.) It's better style to avoid them. Instead, have makepp run the command at most once by assigning it in this special way:

XYZFLAGS ;= $(shell pkg-config --cflags xyz)

Currently there's only one lexer class, which understands Bourne Shell. To better handle C Shell or command.com, subclasses might be created. However, much syntax is similar enough to not warrant this. Get in touch if you want to contribute either. 🫶

2. For known commands, the corresponding command parser (also referred to just as parser) analyzes the important options and arguments. The available ones are described below.

Even if no specialized parser was found, the generic one makes the command executable an input of this rule. You can change that with the --no-path-executable-dependencies command option.

3. If the parser recognized any input files, they get sent to the scanner chosen by the parser. It finds further inputs by looking for #include or comparable statements.

This is the most expensive step. 💸 All the results get cached to avoid repeating it unnecessarily.

If makepp thinks it's compiling a C/C++ source but can't find a parser, it will give a warning message to let you know. This usually means you buried your compiler command too deeply in the action for makepp to find it. For example, I have seen rules like this:

%.o: %.c
    @echo Compiling $< now; obscure_wrapper gcc -c $< $(CFLAGS) -o $@

The first words of the actions here are echo and obscure_wrapper, for which there are no parsers, so makepp will not scan for include files in this case. You can ignore the prefixed command by:

register-parser obscure_wrapper skip-word

The following sections document the built in parsers and scanners. In the name you can use - interchangeably with _. 🎀

---

🔥 SCANNERS (PARSERS) 🔥

The various scanners must be chosen by a command parser, which is given in parentheses:

💜 C/C++ compilation (c-compilation, gcc-compilation)

The C/C++ scanner handles both languages indifferently. In fact it looks only at preprocessor statements, so it can be used for quite a few languages. The parser that activates it has a special variant for gcc's many options, which gets chosen if the command name includes the string gcc or g++. If compilers for other languages with C preprocessor use the same options as the C compiler (at least -I) then this parser works fine.

It looks at the command for -Idir options specifying the include path or -Ldir options specifying the link path. It then scans any source files for #include directives, and also looks at the command line to see if there are any source files or libraries mentioned which are not listed as dependencies. It recognizes these by their extension.

This scanner gives a warning message if files included with #include "file.h" are not found, or not buildable by makepp, in the include path, or in the directory containing the file which is #including, or in /usr/include. No warning is given if a file included with #include <file.h> is not found. Makepp assumes it is in some system include directory that the compiler knows about, and that files in system include directories won't change.

In addition, files in /usr/include, /usr/local/include, /usr/X11R6/include, and any other directory which is not writable are not scanned to see what they include. Makepp assumes that these files won't change. (If you're running as root, the writability test is performed with the UID and GID of the directory you ran makepp from. This is so compiling a program as an ordinary user and then doing makepp install as root won't cause extra directories to be scanned.)

ngl bestie, this is a fairly simple-minded scanner 💀 It will get confused if you do things like this:

#ifdef INCLUDE_THIS
#include "this.h"
#endif

because it doesn't know about preprocessor conditionals. This is usually harmless; it might cause additional extra files to be labeled as dependencies (occasionally causing unnecessary rebuilds), or else it might cause makepp to warn that the include file was not found. You can either ignore the warning messages, or put an empty file this.h out there to shut makepp up. 🙈

If your compiler has a funny name, you can say either of

register-parser obscure_c_compiler c-compilation
register-parser obscure_gcc_alias gcc-compilation

🦋 Embedded SQL C/C++ compilation (esql-compilation)

These commands, which come with the various databases, preprocess special sections in otherwise C/C++-like sources, and produce C/C++ headers and sources. This finds EXEC SQL INCLUDE "filename" or $INCLUDE "filename" directives.

These preprocessors are recognized: Altibase APRE*C/C++ (apre), CASEMaker DBMaker (dmppcc), Firebird / InterBase (gpre), IBM DB2 (db2 precompile, db2 prep) & Informix ESQL/C (esql), Ingres (esqlc), Mimer (esql), Oracle (proc), PostgreSQL (ecpg) & YARD (yardpc). If your preprocessor is not recognized, you can say

register-parser obscure_esqlc_preprocessor esql-compilation

This will however only handle the style common to Informix and others: Command arguments ending in .ec are files to be scanned, -I defines the include path and EXEC SQL INCLUDE directives without a suffix get .h appended.

🌈 Swig (swig)

Swig (Simplified Wrapper and Interface Generator, http://www.swig.org/) is a program that converts a C/C++ header file into the wrapper functions needed to make your code callable from a variety of other languages, such as Perl, Python, Tcl, C#, Ruby, OCaml, and probably some others that I don't know about.

Makepp understands and parses the swig command line, looking for -I and -l options. It also knows how to scan swig's interface definition files (.i files) looking for %include, %import, and also #include if -includeall is in effect.

If your swig has a funny name, you can say

register-parser obscure_swig_alias swig

⭐ Vera and Verilog (vcs_compilation)

If you design hardware, this will come in handy. 💅

🍓 Ignorable wrappers (skip-word, shell)

Makepp recognizes the following command words and many more and skips over them appropriately in its search for the correct scanner: condor_compile, distcc, ignore_error, libtool, noecho purify.

There is a variant of this which finds the nested commands in sh -c 'command1; command2'.

If you have more such commands, you can say

register-parser command skip-word

👑 Libtool

Libtool is a very clever compilation system that greatly simplifies making shared libraries by hiding all the system-dependent details away in a shell script. The only difficulty is that the library binary files are not actually stored in the same directory as the output file — libtool actually creates a subdirectory, .libs, which contains the real files. This is ordinarily not a problem, but makepp has to know where the real binaries are if it is to link them in from a repository. At the moment, libtool libraries (.la files) are not linked in from repositories; they are always rebuilt if needed. Also, makepp at the moment is not able to use the dependency information that is stored inside the .la file itself. This will hopefully change soon. 🫶

💀 Suppressing the scan (none)

Sometimes you may not want a rule or a certain command to be parsed. You can turn off parsing and thereby scanning with

register-parser cc none

---

✨ RELATED OPTIONS ✨

🎀 Quickscan and smartscan

The :quickscan and :smartscan rule options, if applicable, affect the way that files are scanned.

In :quickscan mode (the default), all include directives are assumed active. This allows for very efficient scanning. ⚡

In :smartscan mode, an attempt is made to interpret macros and expressions so that inactive include directives are ignored. For example, the executable produced by compiling the following C program ought not to depend on foo.h:

#if 0
#include "foo.h"
#endif
int main() { return 0; }

💅 CUSTOM SCANNERS 💅

okay so bestie, you can absolutely roll your own parser. two ways to do it:

• drop it right into a rule option like :parser foo, or
• use the register_parser or register_command_parser statements.

either way — and this is the tea from over in register_parser — you gotta specify a function that makes a parser object. directly, or indirectly via a class, your call. that object will usually go ahead and create a scanner object for files, then feed it everything it found in the command line options. and those two objects? they call the parser's add_*_dependency methods, which pass the info along to the slightly-more-complicated Mpp::Lexer::add_*_dependency utility functions. it's giving relay race, honestly. 🏃‍♀️

BUT — plot twist — your parser function can just do that work itself for simple cases. and if this function doesn't return a parser object, there are a few special return values waiting for you:

• undef — the scan info is not cacheable and has to be recalculated next time the rule's target needs to be built. no shortcuts, sorry babe.
• p_none, p_skip_word or p_shell — these are actually numeric constants, and they tell the lexer to go do the work of these pseudo-parsers.
• any reference, e.g. \1 — this is equivalent to returning a parser object of the Mpp::CommandParser base class, which will only additionally make the command executable itself a dependency.

in most cases, objects of type Mpp::CommandParser should instantiate at least one object of type Mpp::Scanner. the Mpp::Scanner base class handles the whole quickscan vs. smartscan distinction for you. ✨ note that the behavior of Mpp::Scanner can be markedly affected by this distinction — but that should be totally transparent to the derived class if it's well-formed. new derived Mpp::Scanner classes ought to be tested in both modes, fr fr.

if you write your own Mpp::Scanner class, you should also base your rescanning decision on the build info RESCAN. why? because makeppreplay sets it after signing files without scanning. so even though the signatures are consistent, a rescan is still necessary. if your Mpp::Scanner uses the inherited scan_file1 method, you're probably fine. 🫶

for more deets, go read the respective class documentation. for examples, peek at Mpp::CommandParser::Gcc and Mpp::CommandParser::Vcs. and look at the p_ functions in Mpp/Subs.pm, which get aliased into their respective classes as factory when loaded. iykyk. 👑

---

🧠 Caching scanner info 🧠

if all of the scanner's important side effects go through calls to methods of the Mpp::CommandParser base class, then those side effects can be cached in the build info file. meaning? a later invocation of makepp can just play them back without doing all that costly scanning work. major time save, especially in smartscan mode. we love efficiency. 💅

if your scanner has other important side effects though, it needs to call the Rule object's mark_scaninfo_uncacheable method. otherwise the scanner info pulled from the build info may be inaccurate, which could make the build result possibly incorrect. not cute. not the vibe.

this method gets called automatically when:

• a value from the %parsers hash doesn't return an object of type Mpp::CommandParser, or
• the parser is specified with a rule option and the p_* routine doesn't return an object of type Mpp::CommandParser.

cached scan info is invalidated using criteria similar to those used for determining when the target is out of date. and similarly, it can be retrieved from a repository using criteria similar to those used for determining when a target can be linked in from a repository.

you can force makepp to ignore the cached scanner info with the --force-rescan option. super useful when a broken scanner may have caused incorrect scanner info to be cached. clean slate energy. 🧼

---

🦋 Ad Hoc Scanner 🦋

okay bestie, real talk: often you've got just one or a few files holding dependency information. you don't wanna write that into a makefile redundantly — because redundancy later always leads to inconsistencies when one update gets forgotten. we've all been there. 🙈

but you also don't wanna write a Mpp::Scanner? as a workaround, you can generate an include file on the fly. for example, Qt has .qrc files which can look like:

<RCC>
  <qresource prefix="...">
    <file>abc</file>
    <file>xyz</file>
    ...

if you adhere to the above layout, you can transform the relevant lines into a makepp include file, which gets automatically created by being included.

%.qrc.makepp: %.qrc
    &grep 's!<RCC>\n!$(stem).cc:! || s! *<file>! ! && s!</file>\n!!' $(input) -o $(output)

include $(wildcard *.qrc)   # .makepp is appended automatically

several variants of this are given in the Cookbook. 📖

the drawback? you begin building while reading the makefile. so the --loop command option will not be so useful on the first iteration. 😭

---

and that's a wrap, bestie! this page is from perl v5.32.0, dated 2021-01-06, and it lives at MAKEPP_SCANNING(1). go forth and scan responsibly. 🎀✨💖