DPB(1)

👑 DPB(1) 👑

General Commands Manual · distributed ports builder

✨ NAME ✨

dpb — distributed ports builder 💖

🦋 SYNOPSIS 🦋

dpb [-acemqRrsUuvx] [-A arch] [-B chroot] [-b logfile] [-C pathlist]
    [-D PARAM=value] [-F m] [-f m] [-h hosts] [-I pathlist] [-J p]
    [-j n] [-L logdir] [-l lockdir] [-M threshold] [-P pathlist]
    [-p parallel] [-S logfile] [-X pathlist] [pathlist ...]

🎀 DESCRIPTION 🎀

Okay bestie, so dpb is used to build ports on a cluster of machines, or on a single machine with several cores. It literally walks the ports tree to figure out dependencies and starts building ports as soon as it can. Main character energy fr fr. ✨

dpb will run with sensible defaults if used without options. But lowkey note this: it will produce logs, lock files, packages, and package installations. No cap.

If run as non-root, dpb will warn. The preferred way is to run it as root (and preferably under a chroot, see bulk(8) and proot(1) for example setups). dpb will then change its identity to different users as needed. See “THE SECURITY MODEL OF DPB” for details. 👑

dpb can be restricted to a subset of the tree by giving it pathlist ... to build as parameters.

A pathlist is either a pkgpath(7) to build, or a filename that contains pkgpaths (one per line). pathlist parameters can also take the form filename*scale in order to multiply the weights of all pkgpath(7) in a file by a given scale, or pkgpath=value, in order to set the weight of a given pkgpath(7) to a specific value. Slay. 💅

dpb supports ‘hot-fixes’: if a particular port errors out, it is possible to fix the problem, remove the corresponding lockfile, and dpb will pick it up without needing to be stopped and restarted. Obsessed with not restarting tbh.

In order to build on a cluster, the ports tree itself should be identical on each machine (shared through NFS or copied at start).

Some directories must be shared: PACKAGE_REPOSITORY, DISTDIR, and PLIST_REPOSITORY. The WRKOBJDIR and LOCKDIR should be local to each machine, and on a high-speed partition.

Also note that dpb's logs and locks are managed by the main dpb process, which runs locally, and hence those directories do not need to be shared on the cluster. That's giving convenience, honestly.

Some log files ("rolling logs") are kept from one run to the next and stored under ${DISTDIR}/build-stats. On each run, the most recent entries for each pkgpath (see STATS_USED) are used to figure out in which order things should be built. Meanwhile, some more entries are kept around (see STATS_BACKLOG) for potential data analysis over longer periods. 🍓

Option -h file is used to specify hosts to use, where file may contain lots of information, but can be as simple as a list of hosts to use, one host per line (however, it is recommended to also include a STARTUP script).

Most filenames will go through some control sequence expansions. For instance, the default logdir location can be specified as %p/logs/%a. The following sequences are recognized:

• %a — architecture being used.
• %d — date at start of dpb, GMtime, formatted as yyyy-mm-dd@hh:mm:ss.
• %f — fetch distfiles location (DISTDIR).
• %h — short hostname running dpb.
• %L — logdir location.
• %p — portsdir location.
• %t — timestamp (number of seconds since January 1 1970) at start of dpb.
• %$ — Pid of the main dpb process .

💅 Options are as follows 💅

• -A arch — Build packages for given architecture, selecting relevant hosts from the cluster. By default, the current host's architecture will be used.
• -a — Walk the whole tree and builds all packages (default if no pathlist is given).
• -B chroot — chroot to chroot before building. See proot(1) for preparing such an environment.
• -b logfile — Explicitly prime the heuristics module with a previous build log, so that packages that take a long time to build will happen earlier. The rolling log file under %f/build-stats/%a is automatically used.
• -C pathlist — Don't clean port working directories after build. Only use simple pkgpath(7) in the list, as this does not take subpackages and flavors into account.
• -c — Clean port working directory and log before each build.
• -D PARAM=value — Set defined parameter to value. Known parameters are as follows:
  • ALWAYS_CLEAN — Set to 1 if dpb should clean work directories even if the port errored out.
  • BUILD_USER — Default value for build_user if you want to specify it on the command line, and want to ensure even the small "discover PORTSDIR" activity at the beginning of dpb is not run as root.
  • COLOR — Set to 1 to have the normal display in color.
  • CONNECTION_TIMEOUT — Connection timeout for ssh. Defaults to 10 seconds (but ssh will retry 3 times).
  • CONTROL — Let dpb create a unix socket of the given name for external control. Defaults to %L/control-%h-%$. If no socket is wanted, explicitly set CONTROL to empty.
  • DISPLAY_TIMEOUT — Display timeout (in seconds) while waiting for jobs to finish, so that the display is updated even if jobs didn't finish. Defaults to 10 seconds.
  • DONT_BUILD_ONCE — By default, dpb will use the BUILD_ONCE optimization (see bsd.port.mk(5)) if run with -a: pseudo-flavors that disable subpackages and are not necessary for bootstrap will be disabled, so that the same port is built once, as far as possible. This flag disables that optimization, which might be desirable if you want to build a small subset of packages which would pull in the kitchen sink otherwise.
  • DONT_CLEAN_LOCKS — By default, dpb will clean old locks from dpb running on the same host that no longer exist, provided they didn't end in error. This is usually the right thing to do after a crash, or after killing dpb abruptly. Sometimes, one may want manual control over which locks to remove.
  • FETCH_JOBS — Alternate way to specify the number of fetch jobs.
  • FETCH_TIMEOUT — Timeout (in seconds) after which fetches that don't show any progress will be killed. This can be instead set in DEFAULT or localhost as the fetch_timeout property.
  • FETCH_CMD — Override for the default FETCH_CMD coming from ports. This might be useful because fetching isn't chroot'd and is run as ${FETCH_USER}.
  • FETCH_USER — User for all fetch activities if possible (defaults to _pfetch).
  • FTP_ONLY — Don't fetch distfiles/don't build packages that are not allowed for ftp.
  • HISTORY_ONLY — Don't fetch or build anything. Only run dpb to figure out old distfiles and update %f/history.
  • LISTING_EXTRA — Alternate way to specify -e.
  • LOCKDIR — Alternate way to specify the locking directory.
  • LOGDIR — Alternate way to specify the logging directory.
  • LOG_USER — User for all log files if possible (defaults to build_user).
  • MIRROR — Applicable to fetch modes. If 0, will only fetch normal DISTFILES (default for dpb -f). If 1, will also fetch extra SUPDISTFILES (default for dpb -F).
  • NEVER_CLEAN — If 1, dpb will never clean any work directory after build.
  • NO_BUILD_STATS — Disable reading/saving of default build stats under ${DISTDIR}/build-stats/${ARCH}.
  • NO_CHECKSUM — Do not run checksum again for files already fetched.
  • NO_CURSOR — Make the terminal cursor invisible if possible. Avoids flickering on slow graphics cards.
  • NO_HISTORY — Do not update the distfiles history. For instance, if dpb is run a second time after a problem during the first run.
  • NO_QUICK_SCAN — Disable the quick scan default heuristic, where full bulks will start by scanning the most prominent ports in former builds.
  • PORT_USER — User that can write to the ports tree. Not really used for anything yet.
  • RECORD — Define a file which will save all terminal output. Mostly useful for presentations, as a way to save dpb output and replay it later at a faster rate. Defaults to %L/term-report.log, can be set to nothing to disable.
  • STARTUP — Define a start-up script on the command-line, override any host file contents.
  • STATS_BACKLOG — Max number of stats (per individual pkgpath) to save in the rolling log file (defaults to 25).
  • STATS_USED — Clamp number of stats (per individual pkgpath) used for computing build order (defaults to 10).
  • STUCK_TIMEOUT — Timeout (in seconds * speed factor) after which tasks that don't show any progress will be killed. This can be instead set on a per-core basis as the stuck property. Note that this will always be divided by the core's speed factor.
  • SYSLOG — Make dpb call syslog(3) on every task start/end while creating packages. This does produce lots of messages, it is intended to route the logging on another machine, while tracking down panics and other hangs.
• WANTSIZE — ngl bestie this is just another way to spell -s, same energy, different outfit 💅
• -e — makes the listing job extra: once it's done it does not go back into the pool. It's a main-character job, it doesn't do returns 🎀
• -F m — fetch-only mode, for mirroring hosts ✨ Do not build any package, just fetch everything, straight up disregarding BROKEN and ONLY_FOR_ARCHS info like they're last season. Creates m localhost jobs for fetching files 📥
• -f m — create m jobs for fetching files. These are separate from the build jobs (they don't eat cpu, they run on the localhost) 🧚 Defaults to 2. You can set it to 0 to bypass fetching jobs entirely and reduce dpb memory footprint by a lot — like, genuinely a lot, no cap 💀
• -h hosts — file with the hosts to use for building. One host per line, plus properties, like so:

espie@aeryn jobs=4 arch=i386

Lines starting with a known variable name, such as

STARTUP=path

or

FETCH_JOBS=5

can also be set inside a configuration file, so you don't have to pass a million options on the command line 🫶

The special hostname DEFAULT can be used to preset defaults — put it at the start of the file, bestie, it's the opening act 👑

Use localhost to specify the local machine. dpb special-cases it and will not use ssh(1) to connect 🙈

Properties are as follows:

• always_clean=n — set to 0 or 1, on a per-host basis. See the ALWAYS_CLEAN parameter 🌸
• arch=value — architecture of the concerned host. (There should be a startup task to check consistency, but currently you have to set this manually on heterogeneous networks. Not me explaining the lore 😭)
• build_user=user — use user for non-root jobs if possible (defaults to the whoami(1) value) 🎀
• chroot=dir — chroot to dir before building 🧚
• fetch_timeout=s — timeout (in seconds) after which fetches that don't show any progress get killed. Only makes sense for DEFAULT or localhost ⏰
• jobs=n — number of jobs to run on that host, defaults to hw.ncpuonline 🔥
• junk=n — junk unused packages each n steps. See the -J option 🗑️
• memory=thr — build everything below that wrkdir threshold with USE_MFS=‘Yes’, assuming the ports tree has been configured so that WRKOBJDIR_MFS points to a memory filesystem. thr is the sum, in KBytes, of ports that will be allowed to build in memory. dpb understands suffixes, such as -M 2G or -M 500M.

⚠️ Note that you should always allow for some margin, as dpb makes its decision based on the size info collected during previous builds — so in cases of significant updates, the work directory size will usually grow 📈

• nochecksum=0/1 — defaults to 1. During the junk stage, run pkg_delete(1) with the -q (no checksum) option ✅
• parallel=p — run big ports on several cores. See the -p option 💪
• parallel2=p — run the largest ports on many cores. Defaults to the same value as the parallel option, but can be increased for, say, chromium 👀
• repair=0/1 — defaults to 1. Run pkg_add(1) with the repair option. Lowkey useful on some bulk machines that tend to crash a lot and leave /var/db/pkg in a weird state 🙈
• sf=n — speed factor. An estimate of that machine's speed with that number of jobs compared to other machines in the same network. Works better with small values, in the range 1..50. The machine (or machines) with the highest speed factor gets access to all jobs, whereas other machines get clamped to stuff that doesn't take too long. Requires previous build information to be effective. Defaults to 1 🏎️
• small=s — small threshold (in seconds × sf): ports known to build under that duration are deemed small, so dpb won't bother calling fine-grained steps for patch/configure/fake — it goes straight to build and package instead. Defaults to 120 seconds ⚡
• squiggles=n — number of squiggles on this host (see “THE SQUIGGLE HEURISTICS” below). Defaults to 1 squiggle for hosts with 4 jobs or more, 0.7 for hosts with more than 1 job, 0 for single-job hosts 🌀
• stuck=s — stuck timeout (in seconds × sf) after which tasks that show no progress get killed 💀
• timeout=s — defines a specific connection timeout for ssh to that host ⏱️

There are no fine-grained options to control ssh(1) options, because those can be specified through virtual host declarations in ssh_config(5) ✨

• -I pathlist — list of pkgpath(7) to install, on the local box. This will also add them to the list of things to build 📝
• -J p — override the value for the “junk” property. Delete unneeded installed packages during the build. Each prepare stage is followed by a show-prepare-results stage. After every p new dependencies, it gets followed by a junk stage, which uses pkg_delete(1) with the -aXI options to delete automatically installed packages that are currently not needed 🗑️

dpb keeps track of the list of dependencies on a given host by storing each dependency list in the lockfile corresponding to the package being built 🔐

To avoid a race condition between the depends and junk stages, dpb allows only one job on a given host to be in the depends … junk stages at one time, by using a per-host lock 🫶

Defaults to 150. Can be disabled by setting to 0.

⚠️ Some ports — most notably cmake-based ones — have an annoying dependency handling bug: they compute their makefile dependencies based on all include files present, not just the ones that are actually enabled. Those ports' builds may be broken by a junk phase that removes some unused includes that were added as makefile prerequisites. Those ports should be annotated with DPB_PROPERTIES = nojunk until that bug is fixed: while a port with the ‘nojunk’ property is building, junk will be postponed 🙏

Those ports get marked with a ! in the display, so it's way more obvious why junk seems to be ineffective 👀

Note that the ‘nojunk’ property is still active for ports in error — in the belief that trivial fixes can be made that will allow the port build to finish 🤞

• -j n — number of jobs to run on a single host (defaults to hw.ncpuonline) 🔥
• -L logdir — choose a log directory. (Defaults to %p/logs/%a.) 📁
• -l lockdir — choose a lock directory. (Defaults to %L/locks.) Override it to keep it local, because locks don't really like NFS 😬
• -M threshold — build ports below the memory threshold under a memory filesystem, as configured through WRKOBJDIR_MFS (see bsd.port.mk(5)). threshold is the sum, in KBytes, of ports allowed to build there 🧠
• -m — force tty-style reporting 🖥️
• -P pathlist — read the list of pkgpath(7) from file 📄
• -p parallel — override the value for the “parallel” property.

Run big jobs on several cores on the same host, by using MAKE_JOBS=k 💪

Once such a job has started, dpb will not start new jobs on the same host until the big job has stolen enough cores from other finishing jobs 😤

Only big ports that are safe for parallel building (annotated with DPB_PROPERTIES = parallel in their Makefile) are affected 🎀

It's advisable to set k to an integral fraction of the number of cores available on a given host. parameter can be an integer, or of the form /n, in which case dpb sets k to a fraction of the total number of jobs on the machine, but never below 2 🧮

Defaults to /2 ✨

• -q — don't quit while errors/locks are around. It's giving "we stay" energy 💅
• -R — rebuild existing packages based on discrepancies between the package signature and what the port says it should be. Concretely, use this to run a partial bulk build after some library change 🔄

⚠️ Note that -R won't always work, since rebuilding a package when another version is already installed is not supported. Building in a chroot is strongly recommended 🧚

• -r — random build order. Disregard any kind of smart heuristics. Useful to try to find missing build dependencies 🎲
• -S logfile — read logfile as an initial workdir size log 📚
• -s — compute workdir sizes before cleaning up, and stash them in the log file %L/size.log. This also maintains a rolling log of build sizes under %f/build-stats/%a-size. To save time, dpb actually won't always compute new sizes for known directories — mostly just for new ones, or when the package name changes ⏳
• -U — insist on updating existing packages during dependency solving, even if the new package apparently didn't change. Like, insist insist 🙅‍♀️
• -u — update the packages you already have while it's solving deps. You can use this to do a bulk-build on a machine that already has stuff installed, but ngl it might break a little, since some packages only build on a totally clean machine right now 🙈
• -X pathlist — read a list of pkgpath(7) from a file and pass them along in the junk phase: these are the packages that should stay on the machine if some dependency pulled them in. Basically the anti-churn flag — use it so you're not endlessly removing and reinstalling the usual suspects, like devel/gmake.
• -x — no tty report, only the actually important stuff gets reported: hosts going down and coming back up, build errors, or builds that just aren't progressing. Minimalist queen 💅

---

dpb figures out the build order on the fly and constantly shows you info about whatever's building right now. There's a list of what's currently running, one line per job, and those jobs are in strict chronological order — which means long-running builds tend to percolate up to the top of the list. Normal jobs look like this:

      www/mozilla-firefox(build) [9452] 41% unchanged for 92 seconds

So what's in there? Let's break it down:

• an optional ‘~’ squiggle marker (more on that below 👀),
• the pkgpath being built,
• the step currently being run,
• an optional ‘!’ for ports with the ‘nojunk’ property,
• an optional ‘+’ for ports built in memory,
• the pid running that task (heads up: this is always a pid on the host running dpb — for distributed builds, it'll be an ssh(1) to another machine),
• the current size of the log file (shown as a percentage if there are previous build stats available),
• and a possible notice that things might be stuck when the log file doesn't change for long periods.

And fetch jobs look like this:

      <dist-3.0.tgz(#1) [4321] 25%

Which contains:

• the file being fetched,
• the number of the MASTER_SITE being tried,
• the pid of the ftp(1) process (note that fetch jobs are always local),
• a progress percentage.

After that comes a host line with the name of each host dpb is using. They're followed by a ‘-'’` for unresponsive hosts, and the pid of the ssh master for distant hosts.

And it all wraps up with a summary display:

• I= — number of built packages that can be installed.
• B= — number of built packages not yet known to be installable, because of run depends that still need to be built.
• Q= — number of packages in the queue, e.g. stuff that can be built now, assuming we have a free slot.
• T= — number of packages to build, where dependencies are not yet resolved.
• F= — number of distfiles to fetch, when -f is used.
• != — number of ignored packages. Details in engine.log.
• L= — list of packages that cannot currently be built because of locks.
• E= — list of packages in error, that cannot currently be built.
• H= — list of packages that haven't shown up yet, usually due to nfs, but watch out for revision bumps 👀

If those three lists are empty, they won't even show up. Packages in errors may be followed by a ‘!’ if they prevent junk from happening.

Heads up: those numbers refer to pkgpaths known to dpb. In general those numbers will be slightly higher than the actual number of packages being built, since several paths may lead to the same package. No cap.

dpb uses some heuristics to try to maximise the queue as soon as possible. There are also provisions for a feedback-directed build, where info from previous builds gets used to try to build the long-running jobs first. Obsessed honestly 💖

Similarly, fetches will use the continue option of ftp(1), since distfiles are checksummed after the fetch anyways.

---

✨ THE SQUIGGLE HEURISTICS ✨

So here's the tea: on machines with lots of cores, the basic scheduling heuristics yields a tail of very small jobs, where dpb mostly just waits on pkg_add(1) to solve dependencies. 😭 Starting with OpenBSD 5.5, a new mechanism (squiggles) was introduced to counter-balance this effect: big machines devote some of their cores to ‘squiggles’, jobs that walk the queue in reverse, thus building the smallest ports first. As a result, small ports get built as a trickle alongside the largest ports, offsetting the negative effect of the exponential queue for a large part. Slay 🦋

Note that ‘squiggles’ can be a non-integral value, usually lower than 1, in which case they represent the fraction of cores that should be affected to squiggles, as decided randomly at the start of each build. 0.7 or 0.8 might be a good choice for dual core machines.

---

🌸 DPB PROPERTIES 🌸

The bsd.port.mk(5) variable DPB_PROPERTIES may hold several annotations that only dpb will look at. These properties are as follows:

• lonesome — a large port that stresses the memory limits of the machine, so it should be built alone. Prevents dpb from scheduling anything else on the same host after it starts building. It's giving "do not perceive me" energy 💅
• noconfigurejunk — a port that looks for unneeded dependencies during its configure phase (typically, optional tools like doxygen to rebuild documentation). Similar to nojunk but less expensive, since the configure phase is most often limited in scope.
• nojunk — a port that hardcodes includes in its Makefile mechanisms. Prevents junk from running while the port is building.
• parallel — a port that can be built in parallel, uses MAKE_JOBS and several build slots.
• parallel2 — a very large port that should be built in parallel, uses MAKE_JOBS and lots of build slots. Main character energy 🔥

---

👑 THE SECURITY MODEL OF DPB 👑

When dpb is run as root, it uses a privilege drop model instead of the dangerous privilege elevation model of doas(1). When run as root, by default, _pbuild is used as the build and log user, and _pfetch is used as the fetch user.

• Start dpb as root.
• dpb will drop privileges for every operation except pkg_add(1), pkg_delete(1) and the STARTUP script.
• For cluster builds, provide an ssh(1) connection to distant hosts from root as root.
• build_user is used to build stuff locally or distantly (can be per-host), using: chroot -u build_user /build_root (with /build_root = / if there's no actual chroot needed). It must have read access to ${DISTDIR} and ${PORTSDIR}, and write access to ${WRKOBJDIR}, ${PACKAGE_REPOSITORY}, and ${PLIST_REPOSITORY}. It does not require network access.
• LOG_USER is used to open all log files. LOG_USER only needs to exist locally. It needs write access to the log directories, including ${DISTDIR}/build-stats. It does not need network access.
• FETCH_USER is used to fetch distfiles and handle corresponding log info. It needs write access to ${DISTDIR}, and network access. Thus, ftp(1) does not happen as root. Petition signed successfully 🫶
• _dpb is used as a fail-safe for any other activities that do not require any rights.
• dpb creates local directories as root, then gives them to the appropriate user.

---

🎀 LOCKS AND ERRORS 🎀

dpb still uses the normal ports tree mechanism while building, which includes LOCKDIR. When starting up, dpb will normally detect stale locks from old dpb runs and remove them. If this doesn't happen, builds will stay stuck in their initial stage, that is: show-prepare-results, patch, build depending on the port. A telltale message ‘Awaiting lock ...’ can be found in the corresponding logfile paths/pkgpath.log.

In addition, when building a package, dpb produces a lockfile in the locks directory, whose name is deduced from the basic pkgpath with slashes replaced by dots. This lockfile is filled with such info as the build start time or the host, or the needed dependencies for this pkgpath.

The lockfile will also contain the name of a parent pkgpath, for paths that were discovered as dependencies. This is particularly useful for bogus paths, where it would be hard to know where the path came from otherwise.

At the end of a successful build, these lockfiles are removed. The lock will stay around in case of errors (raw value from wait(2)), and the name of the next task in the build pipeline (with todo=<nothing> in case of failure during clean-up). Normal list of tasks is: depends prepare fetch patch configure build fake package clean.

At the end of each job, dpb rechecks the locks directory for existing lockfiles. If some locks have vanished, it will put the corresponding paths back in the queue and attempt another build.

This eases manual repairs: if a package does not build, the user can look at the log, go to the port directory, fix the problem, and then remove the lock. dpb will pick up the ball and keep building without interruption. Iconic behaviour, honestly 💜

It is perfectly safe to run several dpb in parallel on the same machine. This is not optimal, since each dpb ignores the others, and only uses the lock info to avoid the other's current work — but it can be handy: in an emergency, one can start a second dpb to obtain a specific package right now, in parallel with the original dpb.

Note that dpb is very careful not to run two builds from the same pkgpath at the same time, even on different machines: in some cases, MULTI_PACKAGES and FLAVOR combinations may lead to the same package being built simultaneously, and since the package repository is shared, this can easily lead to trouble. 💀

Handling of shared log files and history is also done very carefully, by systematically appending to files or using atomic mv operations.

For obvious reasons, this won't work as well with masters running on distinct machines sharing their logs through NFS.

🔍 BUILD CYCLES 🔍

There are some various interdependencies in package builds that can be hard to trace in case something goes wrong. Refer to summary.log to fix those specific issues.

---

🌈 AFFINITY 🌈

dpb now maintains a list of pkgpath-per-host that are currently building in the affinity directory of its log directory, along with building-in-memory status.

That information is only wiped out when a given build finishes successfully.

Otherwise dpb will try to restart that build on the same host, which can be handy if you interrupt dpb while it's building a large port, or if you remove a lock after fixing a problem. It remembers. It cares. 😭✨

🏷️ TAGS FOR EXCLUSIVE BUILDS 🏷️

Okay so storytime bestie 💅 — back in the day we had kde3 and kde4, and the way that these two literally could not be built at the same time 😭 like, a single host had to be exclusively building kde3 ports or kde4 ports at any given moment. No mixing, no vibes.

So the conflicting ports got annotated, right? With DPB_PROPERTIES=tag:kde3 and DPB_PROPERTIES=tag:kde4 respectively. Cute little labels ✨

More generally: with DPB_PROPERTIES=tag:A and DPB_PROPERTIES=tag:B, dpb will keep track of tags. For instance, if host X is building ports tagged with A, then any port with tag B will be prevented from building on X until the next junk phase. It's giving "you can't sit with us" 🚫

This heavily relies on the junk stage to clean up hosts periodically — and it can even forcibly provoke a junk stage even if junk=0. Iconic but aggressive.

That ‘force-junk’ stage is actually implemented as a pseudo path called junk-proxy, which does only junk. Nothing else. Just junk. 🗑️

In order for builds to proceed gracefully, machines should start in a clean slate, without any of the problematic ports installed. Fresh face, no baggage 🧖

As a special-case, failing ports with a tag will not interfere with clean-up, so that hosts do not get locked down to a specific tag. This also means that their dependencies may vanish before human intervention addresses the problem. Yeah. That's the trade-off, ngl 😬

This is supposed to be an exceptional hack, helpful while porters figure out how to remove the deadlock. It's a band-aid, not a lifestyle 🩹

---

🎮 EXTERNAL CONTROL 🎮

By default (see CONTROL), dpb will create a Unix socket at %L/control-%h-%$, only accessible by LOG_USER, that can accept a few commands — e.g., usable as nc -U path. Yes bestie, you can literally just talk to it 🫶

Current commands are as follows:

• addhost hostline — add a new host.
• addpath fullpkgpath ... — add fullpkgpath to scan.
• bye — close the socket connection. Bye 👋
• dontclean pkgpath ... — add new pkgpath to the list of paths that should not be cleaned after build.
• help — self explanatory. (Truly self explanatory. No notes.)
• info cores — debug info for cores (to be extended to other data).
• rescan — force dpb to rescan all ignored paths (for various errors, including bogus dependencies).
• stats — show the current stats line.
• status fullpkgpath ... — show the current status of fullpkgpath: whether it's built, installable, ready to build, to build later, along with current dependencies if applicable.
• stub fullpkgpath ... — stub out fullpkgpath and unlock it if needed.
• wipe fullpkgpath ... — wipe out an existing lock: clean up the corresponding fullpkgpath on the appropriate host, then remove all lock and affinity info pertaining to the port. Full reset energy ✨
• wipe-all — wipe out all existing packages with an error state. This operates like the wipe control.
• wipehost hostname ... — remove all information relevant to a given host from dpb, including running jobs, locks, and affinity information.

---

🛑 SHUTTING DOWN GRACEFULLY 🛑

dpb periodically checks for a file named stop in its log directory. If this file exists, then it won't start new jobs, and it'll shutdown when the current jobs are finished — unless -q. Chill exit 😌

dpb also checks for files named stop-<hostname> in its log directory. If such a file exists, then it won't start new jobs on the corresponding machine. Per-host timeout corner, we love to see it 🎀

---

📁 FILES 📁

Apart from producing packages, dpb may create temporary files as ${FULLDISTDIR}/${DISTFILE}.part.

In fetch mode (-f and -F), dpb populates ${DISTDIR}/by_cipher/sha256 with links. It also uses ${DISTDIR}/distinfo and ${DISTDIR}/history as a ‘permanent log’:

• distinfo — cache of distfiles checksum. Contains all sha256(1) checksums of known files under ${DISTDIR}. Fetching uses this to avoid re-checksumming known files. Efficiency queen 👑
• history — log of old files under distinfo. After successfully scanning a full ports tree (dpb -a), the fetch engine knows precisely which files are needed by the build (and their checksums). Anything that is:
  • recorded in distinfo but unneeded
  • recorded in distinfo but with the wrong checksum
  • not recorded in distinfo, but not needed

…will be entered at the end of history as a line:

ts SHA256 (file) = value

with ts a timestamp from Unix epoch.

When cleaning up old files, with a tool such as clean-old-distfiles(1), it is vital to check both the checksum and the file name: since mirroring stores permanent links under by_cipher, files which are still needed will appear in history under their old checksums, as an indication the link should be removed, but possibly not the file itself. Like, do not just delete stuff, fr fr 😭 check both or perish.

If ${DISTDIR} ever becomes corrupted, removing ${DISTDIR}/distinfo will force dpb into checking all files again. Hard reset, but for checksums 🔄

All those files belong to the FETCH_USER if it is defined. They should be readable for the build_user.

dpb also records rolling build statistics under ${DISTDIR}/build-stats/${ARCH}, and uses them automatically (see STATS_BACKLOG and STATS_USED) in the absence of -b logfile. That file belongs to the LOG_USER if it is defined.

If -s is used, size information for successful builds will be recorded under ${DISTDIR}/build-stats/${ARCH}-size (by default, location adjustable with -S sizelog). This is then reused for the mfs threshold option. That file also belongs to the LOG_USER if it is defined. 📏

dpb also maintains a list of pkgpath frequencies ${DISTDIR}/build-stats/${ARCH}-dependencies, filled at end of LISTING if -a. This list will be automatically reused when restarting a build: a quick LISTING of the most important dependencies will happen before the general LISTING, in order to prime further LISTING steps with most common ports first. Main character ports get the head start 💫

dpb will also create a large number of log files under ${PORTSDIR}/logs/${ARCH}, which will belong to LOG_USER if it is defined:

• affinity/ — affinity information. One file per full pkgpath, with slash replaced by dots like so: affinity/lang.ghc,-main.
• affinity.log — on startup dpb reads existing affinity information, and records it in that log, together with its pid. This log just exists to verify, along with engine.log, whether correct affinity was heeded.
• awaiting-locks.log — this is purely for gathering performance statistics, about how much lock contention happened around pkg_add(1) and pkg_delete(1) usage. Plotting cumulated time may help in fine-tuning squiggles parameters.
• build.log — actual build log. Each line summarizes build of a single pkgpath, as: ‘pkgpath host time logsize (detailed timing)[!]’ where time is the actual build time in seconds, host is the machine name where this occurred, logsize is the corresponding log file size, and a ! is appended in case the build didn't succeed.

The detailed timing info gives a run-down of the build, with clean, fetch, prepare, patch (actually extract+patch), configure, build, fake, package, clean detailed timing info. Note that the actual build time starts at ‘extract’ and finishes at ‘package’.

• built-packages.log — the actual list of fullpkgname.tgz as they get built. Trophies 🏆
• cpu-concurrency.log — shows the actual concurrency achieved as a result of job starvation / parallel handling. Only gets a new line when the value changes: pid timestamp jobs
• debug.log — contains various information related to the main engine spinning (RTFS, haven't figured that one yet) along with the more useful warning and die traces that happen when something wrong occurs. Especially useful for the warning messages that tend to be overwritten by subsequent displays. Will also contain error messages pertaining to failure at parsing existing lock files.
• dist/<distfile>.log — log of the ftp(1) process(es) that attempted to fetch the distfile.
• control-%h-%$ — default name for the external control socket.
• dump.log — a long log file generated at the end of build that yields any information pertinent to ports still in the ‘to build’ and the ‘built’ queues. See also summary.log for an expurged version of same.
• engine.log — build engine log. Each line corresponds to a state change for a pkgpath and starts with the pid of dpb, plus a timestamp of the log entry. The codes, bestie, the codes 👇
  • ^ — pkgpath temporarily put aside, because a job is running in the same directory.
  • ! — pkgpath ignored, either directly, or indirectly because a dependency was ignored. End of the line states reason why ignored.
  • A — affinity mismatch: path considered for build, but not the right host, followed by the affinity information.
  • B — pkgpath built / distfile found.
  • C — forcible clean-up before building a port with a kde tag.
  • E — error in build or fetch.
  • F — distfile queued for download.
  • H — package still not found due to nfs on this run.
  • I — pkgpath can be installable.
  • J — job to build pkgpath started. Also records the host used for the build.
  • K — kde mismatch, no build until host has been cleaned up.
  • L — job did not start, existing lock detected.
  • N — job did not finish. The host may have gone down. 😬
  • P — built package is no longer required for anything.
  • Q — pkgpath queued as buildable whenever a slot is free.
  • T — pkgpath to build / distfile to download.
  • V — pkgpath put back in the buildable queue, after job that was running in the same directory returned.
  • W — only happens when the external control wipe command is used: pkgpath will be cleaned up, next log entry will be ‘N’ since the job did not finish and is ready to restart.
  • X — only happens when rescanning after an error. The engine temporarily locks paths that are incomplete (detained). These will be kept in a separate list for later examination until the end of the new scan.
  • x — only happens when rescanning after an error. Releases a path for building after the new scan is finished. (Yes, X and x are different. Case matters, always 💅)
• Y — affinity mismatch 😭 but the job is gonna start on the wrong host anyways, because the queue literally contains no other buildable path.

The engine is no longer run after each package build event because of performance considerations, so the Q and I changes may be delayed by a few B. fr fr, don't panic if you don't see them instantly.

---

• equiv.log — the lists of equivalent pkgpaths for the build, for when default flavors and default subpackages have been resolved ✨
• fetch/bad.log — list of URLs that did not lead to a correct distfile, either because they weren't responding or because of incorrect checksums 💀
• fetch/good.log — list of URLs that fetched correctly, along with timing statistics 👑
• fetch/manually.log — list of pkgpaths that require manual intervention, in human-readable form 🫶
• <hostname>.sig.log — the complete library signature of the host 🔍
• init.<hostname>.log — captured output of the initialization job for each host 🎀
• junk.log — option -J counts the number of dependencies directly added to decide when to run pkg_delete -a. This file sums up how many ports were built, and how many ports had dependencies each time dpb decides to junk 🗑️
• locks/ — the directory where locks are created. There are three types of locks:
  • pkgpath locks for building, where the slash in a pkgpath is replaced with a dot like so: locks/devel.make — flattens the structure, cute right?
  • distfile locks for fetching, using the distfile name without the path like so: locks/distfile.dist
  • host locks for dependency handling and junking, like so: locks/host:hostname
• packages/pkgname.log — one file or symlink per pkgname 🌸
• paths/some/path.log — one file or symlink per pkgpath 🌸
• performance.log — some parts of dpb are computationally intensive, like the engine runs to determine new stuff that can be built, and the actual display reports 👀

Both those activities are rate-limited, so that dpb doesn't run its engine at each new package build, and doesn't update its display every time there is a phase change.

Lines tagged with ENG correspond to the engine; lines tagged with REP correspond to the display reports.

Lines ending with a dash - correspond to new activity that didn't trigger a computation.

Other lines will feature a plus + for normal runs, or an exclamation point ! for forced runs, followed by two numbers: the next timestamp at which we'll be allowed to run, and a measure of how much time it took to run this pass 🔥

That information is mostly relevant while dpb is building lots of small packages very quickly.

• signature.log — discrepancies between hosts that prevent them from starting up 😬
• size.log — size of work directory at the end of each build, built only with -s 📏
• stats.log — simple log of the B=... line summaries. Mostly useful for making plots and tweaking performance 📈
• stop — not a logfile at all, bestie, it's a file created by the user to stop dpb creating new jobs 🛑
• stop-<hostname> — again, not a logfile at all, but created by the user to stop hostname creating new jobs 🛑
• summary.log — a summary file generated at end of build that lists packages not built or not installable, along with a reason for it. This summarizes packages not built because of existing locks, because of errors, but also because they depend on something that was not built.

In that last case, summary.log contains a chain of dependencies leading to the problematic package, or in case of build cycles, stopping at the first loop 🌀

• term-report.log — saves all terminal output, so that it can be replayed at hi speed with dpb-replay(1) ⏪
• vars.log — logs the directories that were walked in the ports tree for dependency information, including the path to a dependency that triggered this particular step 🌳

---

🔍 DIAGNOSTICS 🔍

okay these are all the messages dpb can throw at you, and what they actually mean. no cap, some of them are actual bugs. 💅

• Waiting for hosts to finish STARTUP... — displayed on the console while dpb is setting up hosts, getting essential data from the ports tree, running a STARTUP script, collecting base library signatures. just vibing, give it a sec ✨
• stuck on <lockfilename> — displayed on the console when dpb detects a "frozen" port has happened outside of dpb's purview, namely because the ports tree itself has that specific port locked without dpb's knowledge. see bsd.port.mk(5), portlock(1) 😭
• (Junk lock obtained for <host> at <time>)
• (Junk lock released for <host> at <time>) — printed in a paths/pkgpath.log file when attempting to get a junk lock. On a given host, all dependency operations are serialized. The dependency computation itself is handled by the main dpb process, which needs to know exactly which dependencies are used at a given point, so that junk can clean up the host correctly. In particular, junk will not clean up dependencies already scheduled for installation. Ports that do not obtain the lock on first try are put to sleep 😴
• Received IO — printed in a paths/pkgpath.log file when woken up before trying attempting to obtain a junk lock again...
• Woken up <fullpkgpath> — printed in a paths/pkgpath.log when waking another task by sending it SIGIO, so that it may attempt to obtain the junk lock again 📨
• (Junk lock failure for <host> at <time>) — all ports sleeping for a junk lock are woken at the same time, so only one of them will obtain the lock, and the others will fail and be put to sleep again 💀
• Short-cut: depends already handled by <fullpkgpath> — printed in a paths/pkgpath.log when a port wakes up after others that ran pkg_add(1). As dpb maintains dependencies for a given host globally, it coalesces depends lists together 👑
• Don't run junk because nojunk in <fullpkgpath> — printed in a paths/pkgpath.log while evaluating whether to run junk. Normally, junk happens at regular intervals, but ports marked nojunk will delay that. dpb still keeps track of attempted junks 🫠
• Still tainted: <bool> — a host may have a tag that prevents building differently tagged ports. This will be cleansed by junk eventually. This prints in path/pkgpath.log to indicate whether this particular junk will keep the host tainted with a tag or not 🎀
• Forced junk, retainting: <tag> — printed at end of prepare-results, when an eventual junk was run even though some ports still hold a tag 🏷️
• Can't run junk because of lock on <fullpkgpath> — junk can't happen because fullpkgpath is locked and is marked nojunk 🔒
• Avoided depends for <dependencies> — as dependencies are handled globally per-host, some ports can avoid pkg_add(1) altogether because another port already installed the correct dependencies 🦋
• SPINNING ON MAIN — printed in debug.log, this is an actual bug: the engine said it can build, there are cores available, but dpb can't start a new build job 💀
• SPINNING ON FETCH — printed in debug.log, this is also an actual bug: the engine said it can fetch, there are fetching cores available, but dpb can't start a new fetch job 💀
• KILLED: <job> stuck at <somewhere> — printed in path/pkgpath.log when a port exceeds its timeout 🔪
• !: <path> tried and didn't get it — printed in engine.log. Scanning the port didn't give us useful information. See vars.log for gory details 🙈

💀 BUGS AND LIMITATIONS 💀

Okay bestie, gather round, because dpb is iconic but she is not perfect, and we need to talk about it. 🫶

• dpb performs best with lots of paths to build. When it's just used to build a few ports, there's a high risk of starvation, because there are bottlenecks in parts of the tree. So like… give her a full plate or she'll get weird about it. 😭
• Fetch jobs don't deal with checksum changes yet. If a fetch fails because of a wrong checksum, and you update the distinfo file and remove the lock, dpb won't pick it up. Not me being ignored after doing everything right 💀
• Heads up: dpb does not manage installed packages in any intelligent way. It will just call pkg_add(1) during its depend stage to install its dependencies. With -u, it will call pkg_add -r. With -U, it will call pkg_add -r -D installed, but there is nothing else going on. This is especially true when using -R — ensure the machine is clean of possibly older packages first, or run dpb with -U. Fr fr, she is not doing the thinking for you here. 🧠✨

In particular -R and -J together may lead to strange issues. Like, "why is everything like this" levels of strange. You've been warned 💅

• On heterogeneous networks, calibration of build info and choice of speed factors is not perfect, and somewhat a dark art. Using distinct speed factors on a build log that comes from a single machine works fine, but using the build info coming from several machines does not work all that well. It's giving ✨ chaos magic ✨ honestly 🔮
• dpb should check /usr/include and /usr/X11R6/include for consistency, but it doesn't. She simply did not. 🙈
• When a host fails consistency check, there is not yet a way to re-add it after fixing the problem. You have to stop dpb, cleanup and restart. Annoying? Yes. Fixable? Not yet, bestie. 😔
• The default limits in login.conf are too small for bulk builds on any kind of parallel machines. Bump number of processes, file descriptors, and memory. Go off, give her room to breathe 🌬️💖
• Even though dpb tries really hard to check heterogeneous networks for sanity (checking shared libraries and .la files), it is still dependent on the user to make sure all the hosts build ports the same way. She's doing her best but she can't do everything 😤
• Make sure your NFS setup is consistent. The ports dir itself should be exported or synchronized. Distfiles, the package repository, and the plist repository should be exported, but WRKOBJDIR should not be on NFS unless you have absolutely no choice, or if you exhibit deep masochistic tendencies 🫣 Pay particular attention to discrepancies in /etc/mk.conf.

Also, dpb connects to external hosts through ssh(1), relying on ssh_config(5) for any special cases. That's the whole vibe. 🐚

• And the classic: when fetching distfiles, dpb may freeze and spin in a tight loop while the last distfiles are being fetched. This is definitely a bug, which has been around for quite some time, which is a bit difficult to reproduce, and hasn't been fixed yet. So if dpb stops updating its display right around the end of fetch, you've hit the bug 💀 Just kill dpb and restart it. Iconic behaviour honestly, she's just taking a little nap 😴

---

🌈 SEE ALSO 🌈

• clean-old-distfiles(1)
• dpb-replay(1)
• proot(1)
• pkgpath(7)
• bulk(8)

Go read them, iykyk, they're the supporting cast and they ate 🎀

---

👑 HISTORY 👑

The original dpb command was written by Nikolay Sturm. This version is a complete rewrite from scratch using all the stuff we learnt over the years to make it better. Glow-up of the century, no cap ✨

---

💖 AUTHORS 💖

Marc Espie <espie@openbsd.org>

Periodt. That's the whole section. 🫡

---

And that's a wrap on the whole page, bestie!! 🎉 This one comes to you from Debian, dated April 9, 2026, and the page itself is DPB(1) 🌸 Now go build something gorgeous, drink some water, and remember: kill dpb and restart it if she starts spinning 💜🦋🎀