MASS-PREBUILD(7)

📦
version 1.8.0 · 2 versions
, in fedora 44

💖 NAME 💖

okay bestie, gather round — this is mass-prebuild, aka the mass-prebuild set of tools 🎀 (the manual name is folded in right here, you're welcome).

✨ DESCRIPTION ✨

So, the mass pre-builder — that's mpb(1) to you and me 🫶 — is a whole little toolkit made to help you build mass rebuilds around a limited set of packages, so you can vibe check how stable a given update actually is. No cap.

The whole idea is lowkey simple. You hand it a package or a bunch of packages — these are the "main packages" — and mass pre-builder goes and calculates the list of their direct reverse dependencies: like, the packages that explicitly mention one of the main packages in their BuildRequires field. 👀

Then the tooling first builds the main packages using the distribution's own facilities (which should include a set of test cases that validate general functionalities). Assuming those main packages build successfully, they get used as base packages to build the reverse dependencies, and those get their own test cases executed too. ✨

That gives you a first set of results — there may be successful builds (hopefully most of them, fr), but also failures that may or may not be because of the changes introduced by modifications of the main packages. 😭 To cut down the uncertainty and hand you a limited list of packages to analyze, the mass pre-builder does something iconic: the moment a failure is detected, it spins up another mass build, running in parallel to the original one, but without the changes that were introduced into the main packages. That's a pristine build 🌈. So this pristine build only includes a sublist of the reverse dependencies — specifically, the ones that failed on the original run.

Once all the package builds are done, you end up with 3 major categories:

  1. The successful ones 🎉
  2. The ones that failed only with the modified packages 💀
  3. The ones that failed with both the modifications and the pristine version 😵‍💫

Out of these, the first category can likely be ignored, since the packages don't seem to have been affected by the changes. slay.

The second category needs much more attention. These are the ones that cry: "Hey there seems to be a big issue with your changes !" 🚨 The failure needs to be analyzed, to figure out if the problem being raised is due to changes that have been introduced, or maybe a mistake from the final user (e.g. Use of a deprecated feature that got removed).

The last category is a bit trickier. Since the build failed with the pristine packages, there may be hidden failures among them that originated from the new changes. 🙈

🦋 Examples of workflows 🦋

🎀 The package pre-release use case (medium sized) 🎀

This is the primary motivation for mpb and may therefore give some insights on decisions taken for default behavior. iykyk.

This use case is basically the one for autoconf: an official release will come out soon, and considering the impact of this component, it makes sense to pre-test it and provide feedback to the community. The idea is, autoconf 2.70 led to failures that were not spotted during tests executed by the community. This resulted in an autoconf 2.71 release soon after 2.70, to fix the major ones. The Mass pre-builder was created to check for autoconf 2.72 candidates regularly, and avoid the same kind of problems. 💅

In this example, an autoconf pre-release tarball is created, which has a specific name generated through git describe. This name is going to be re-used as the project name for MPB. If failures are detected there may be more tarballs to be created while using git bisect.

Each SRPM created that way gets its own folder, and the data collected by MPB is to be centralized. 🗂️

The dependencies need to be automatically calculated, and they should be built to their last known version. Whenever possible the build should be ordered so that if C depends on B, while both depend on Autoconf, then C should wait for B to be built before starting. This should help detect C build failures due to a B misconfiguration (which itself could be due to the new Autoconf version). ✨

By experience, I know that there are transient failures during build in COPR, so builds should be retried once or twice.

In principle, we should run the build for all architectures, but x86_64 is enough to have a good idea on the status. The latest version of each package needs to be built, so we go for rawhide. 👑

Thus, the configuration looks as follows:

name: autoconf-2.72a.17.0f330
package:
  autoconf:
    src_type: file
    src: /home/anexample/work/fedora/autoconf/autoconf-2.72a.17.0f330-1.fc38.src.rpm
retry: 2
data: /home/anexample/work/mpb/autoconf-2.72

The data collected by MPB will be stored under /home/anexample/work/mpb/autoconf-2.72/autoconf-2.72a.17.0f330/. If failures are detected, assuming the problem comes from autoconf, a new build will be generated, with a dedicated name. This new build may either be a full build if there is confidence on the quality of the fix, or a limited one, using a configuration file generated by mpb-failedconf.

Note 📝 Here it shouldn't be forgotten that the whole idea is to validate the main package. Any new version should therefore go through a full rebuild at one point. periodt.

If a failure is due to a bug in a reverse dependency, it falls into 2 categories:

  • There are a handful of failures, they can be restarted manually while modifying the commit ID to be used for the build.
  • The amount of failure is too big to restart manually, mpb-failedconf is used to generate a base config which only contains the failures from the previous build.

In the second case, the generated configuration may need to be modified, e.g. by replacing all the committish: * lines with committish: "@last_build", to ensure that newer version of the packages will be built. 🫶

🌸 The package release use case (medium size, complex dependencies) 🌸

This use case is similar to the previous one, which means that in principle, there may not be any differences. Yet, let's modify slightly the scenario. 👀

As packager, a fork of the original distgit is used to prepare the new release. If downstream patch need to be made, they will be added into distgit and applied on the fly.

Since the dependency graph is complex, MPB may not be able to calculate it, and brute force is to be used for the builds. The project naming is irrelevant.

package:
  ruby:
    src_type: git
    src: https://src.fedoraproject.org/fork/anexample/rpms/ruby
retry: dynamic
data: /home/anexample/work/mpb/ruby-3.5

Since no name for the project is provided, MPB will automatically attribute one. By default, the naming will be mpb.N where N is the build ID.

The data collected will therefore be stored under /home/anexample/work/mpb/ruby-3.5/mpb.5/ (assuming build ID 5).

Note the package fields. The default value for src_type is distgit, which refers to the official package repository for a given distribution. When using non-official distribution git repositories (like in this case, a fork), it is recommended to define this value to git. The default value for committish is the branch corresponding to the fedora release being build for, in our case rawhide. For each new MPB build, the latest version of our fork of the ruby package will therefore be used. 💅

The retry field is set to dynamic: brute force is applied on the reverse dependency builds. A simplified explanation of this is that as long as there are successful builds, the failed packages will be rebuilt (that isn't exactly true, but the idea is there). 😭

The rest of the workflow is similar to the previous one (playing with mpb-failedconf whenever necessary).

🧚 The multi-package use case 🧚

It may happen that a package owner is responsible for multiple packages that depend on each other and therefore need to be built in a specific order. ⛓️

This can be achieved through the following configuration snippet:

packages:
  componentA:
    build_order: 0
  componentB:
    build_order: 1
  componentC:
    build_order: 2

That way, MPB will make sure that componentA is built first, followed by componentB and then componentC. ✨

💜 The package pre-release (big size, overly complex dependencies) 💜

This use case is typically the one that could be expected from a compiler, where thousands of packages will need to be rebuilt. 💀

arch: all
packages:
  gcc:
    deps_only: True
automatic_build_ordering: False
data: /home/anexample/work/mpb/gcc-14
copr:
  additional_repos:
  - https://anexample.fedorapeople.org/fedora-gcc14-${arch}/

🍓 Post-processing reverse dependencies 🍓

For complex scenarios, MPB allows programmatic modification of the reverse dependency list before the build starts. This is achieved through the revdeps:postprocess configuration key, which specifies a path to an executable script.

This feature is powerful but comes with significant security implications. 🚨

Warning: Security Warning 🔥 The revdeps:postprocess feature executes arbitrary code from the user-provided script. The script runs with the same permissions as the mpb command itself. - Trust: Only use scripts that you have written or that come from a highly trusted source. - Code Review: Always review and fully understand any script before configuring it. - Principle of Least Privilege: Run mpb with the lowest possible privileges (never as root). - Environment Access: The script has access to the same environment as mpb, which may contain sensitive information.

The script receives a single command-line argument: the path to a temporary YAML file containing the reverse dependency dictionary. It must print the modified dictionary in YAML format to standard output.

Here is an example postprocess.py script:

#!/usr/bin/env python3
import sys
import yaml

if len(sys.argv) < 2:
    sys.exit("Input file path not provided.")

input_file = sys.argv[1]

with open(input_file, 'r', encoding='utf-8') as f:
    rev_deps = yaml.safe_load(f)

# --- User modifications here ---

# Example 1: Force a package to be built first
if 'important-package' in rev_deps:
    for pkg in rev_deps.values():
        pkg.setdefault('priority', 0)
        pkg['priority'] += 1
    rev_deps['important-package']['priority'] = 0

# Example 2: Remove a package from the list
rev_deps.pop('unstable-package', None)

# --- End of modifications ---

print(yaml.dump(rev_deps))

This script can be enabled in the configuration file like this:

revdeps:
  postprocess: /path/to/your/postprocess.py

Let's go through all these new options. 👑

The arch field is set to all, which is an MPB special option that selects every single architecture COPR supports for rawhide. Like, all of them. The whole buffet. 💅

The deps_only field is basically MPB's way of saying "bestie, we're not rebuilding gcc, just trust me on this one" — it tells MPB that it isn't necessary to rebuild gcc, and to assume the build is a success. Why? Because it takes like 30h in COPR (a whole day and change, no cap 😭), and other means were used for this package instead, like Koji or a dedicated build machine. The resulting packages were put in a custom repository, which gets handed to COPR through the additional_repos field. Iconic. ✨

And since there are about 9k components that depend on gcc, it makes no sense to try to calculate priorities for the builds — they will all be executed in the same COPR batch (with lower priority). For that same reason, it may make sense to have a smoke-test for this project, where a limited set of packages gets built:

arch: all
name: gcc-14_smoketest
packages:
  gcc:
    deps_only: True
revdeps:
  automake:
    committish: "@last_build"
  libtool:
    committish: "@last_build"
data: /home/anexample/work/mpb/gcc-14
copr:
  additional_repos:
  - https://anexample.fedorapeople.org/fedora-gcc14-${arch}/

🌈 RESOURCES 🌈

Wanna know more about the mass-prebuilder? More info lives right here, bestie:

🦋 SEE ALSO 🦋

Go read these too, they're the whole friend group 🫶:

And that's a wrap on the MASS-PREBUILD(7) era 💖 this page was dated 2026-08-27 and it's the MASS-PREBUILD(7) page itself, so yeah — go forth and prebuild, queen 👑🌸

📦 where this page comes from