PUNGI(1)

📦

version 4.14.0 · 4 versions

🌸 PUNGI 🌸

okay bestie, welcome to the manual page for pungi — literally the coolest thing in section 1 of your man pages. Pungi 4.14.0, periodt. 💅

Contents:

✨ ABOUT PUNGI ✨

[image: Pungi Logo] [image]

Pungi is a distribution compose tool. 💖

Composes are release snapshots that contain release deliverables such as:

installation trees
- RPMs
- repodata
- comps
(bootable) ISOs
kickstart trees
- anaconda images
- images for PXE boot

🛠️ Tool overview

Pungi is not one single program, bestie — it's a whole squad of separate executables, all backed by a common library. it's giving girl group energy fr fr.

The main entry-point is the pungi-koji script. It loads the compose configuration and kicks off the process. Composing itself is done in phases. Each phase is responsible for generating some artifacts on disk and updating the compose object that is threaded through all the phases. 🧵

And here's the gag: Pungi itself does not actually do that much. Most of the actual work is delegated to separate executables. Pungi just makes sure that all the commands are invoked in the appropriate order and with correct arguments. It also moves the artifacts to correct locations. it's the manager, not the intern. 👑

The executable name pungi-koji comes from the fact that most of those separate executables submit tasks to Koji that does the actual work in an auditable way.

However unlike doing everything manually in Koji, Pungi will make sure you are building all images from the same package set, and will produce even deliverables that Koji can not create like YUM repos and installer ISOs. no cap, that part is iconic. 🔥

🔗 Links

Upstream GIT: https://forge.fedoraproject.org/pungi/pungi
Issue tracker: https://forge.fedoraproject.org/pungi/pungi/issues

🐍 Origin of name

The name Pungi comes from the instrument used to charm snakes. Anaconda being the software Pungi was manipulating, and anaconda being a snake, led to the referential naming. 🪈🐍

The first name, which was suggested by Seth Vidal, was FIST, Fedora Installation <Something> Tool. That name was quickly discarded and replaced with Pungi. honestly? good call.

There was also a bit of an inside joke that when said aloud, it could sound like punji, which is a sharpened stick at the bottom of a trap. Kind of like software… 💀

🔥 PHASES 🔥

Each invocation of pungi-koji consists of a set of phases. [image: phase diagram] [image]

Most of the phases run sequentially (left-to-right in the diagram), but there are use cases where multiple phases run in parallel. This happens for phases whose main point is to wait for a Koji task to finish. ⏳

🌱 Init

The first phase to ever run. Can not be skipped. It prepares the comps files for variants (by filtering out groups and packages that should not be there). See Processing comps files for details about how this is done.

📦 RPM Repo Mapping

This phase generates compose/metadata/rpm-repo-mapping.yaml, the mapping of which RPM should go to which repository.

rpm_repo_mapping config option is required. 🫶

🗂️ Pkgset

This phase loads a set of packages that should be composed. It has two separate results: it prepares repos with packages in work/ directory (one per arch) for further processing, and it returns a data structure with mapping of packages to architectures.

🥾 Buildinstall

Spawns a bunch of threads, each of which runs the lorax command. The commands create boot.iso and other boot configuration files. The image is finally linked into the compose/ directory as netinstall media.

The created images are also needed for creating live media or other images in later phases.

With lorax this phase runs one task per variant.arch combination.

NOTE: If boot_iso_method is set, boot.iso production is handled by the BootIso phase instead and Buildinstall is skipped. See below.

🥿 BootIso

An alternative to Buildinstall for producing boot.iso images. Instead of running lorax locally, this phase submits a Koji task (either imageBuilderBuild or kiwiBuild, depending on boot_iso_method) and downloads the resulting ISO.

After downloading, the phase populates the os tree with boot infrastructure extracted from the ISO (isolinux/, EFI/, images/) using xorriso, writes a .treeinfo for metadata.py, and hardlinks boot.iso into the iso/ directory as a standalone deliverable.

This phase runs in the essentials group alongside Buildinstall. Exactly one of the two should be active per compose – setting both buildinstall_method and boot_iso_method is a configuration error. Setting neither means no boot.iso is produced and ISOs will not be bootable. 🫣

The phase is controlled by boot_iso_method and supports two backends:

imagebuilder – uses Image Builder via the imageBuilderBuild Koji task type. Per-variant configuration is read from boot_iso_imagebuilder.
kiwi – uses Kiwi via the kiwiBuild Koji task type. Per-variant configuration is read from boot_iso_kiwi.

🧺 Gather

This phase uses data collected by pkgset phase and figures out what packages should be in each variant. The basic mapping can come from comps file, a JSON mapping or additional_packages config option. This inputs can then be enriched by adding all dependencies. See Gathering packages for details.

Once the mapping is finalized, the packages are linked to appropriate places and the rpms.json manifest is created. ✨

📁 ExtraFiles

This phase collects extra files from the configuration and copies them to the compose directory. The files are described by a JSON file in the compose subtree where the files are copied. This metadata is meant to be distributed with the data (on ISO images).

🏪 Createrepo

This phase creates RPM repositories for each variant.arch tree. It is actually reading the rpms.json manifest to figure out which packages should be included.

🌳 OSTree

Updates an ostree repository with a new commit with packages from the compose. The repository lives outside of the compose and is updated immediately. If the compose fails in a later stage, the commit will not be reverted. ⚠️ like, that's a real warning bestie, no take-backs.

Implementation wise, this phase runs rpm-ostree command in Koji runroot (to allow running on different arches).

💿 Createiso

Generates ISO files and accumulates enough metadata to be able to create image.json manifest. The file is however not created in this phase, instead it is dumped in the pungi-koji script itself.

The files include a repository with all RPMs from the variant. There will be multiple images if the packages do not fit on a single image.

The image will be bootable if buildinstall or boot_iso phase is enabled and the packages fit on a single image.

There can also be images with source repositories. These are never bootable. 🙅♀️

🧁 ExtraIsos

This phase is very similar to createiso, except it combines content from multiple variants onto a single image. Packages, repodata and extra files from each configured variant are put into a subdirectory. Additional extra files can be put into top level of the image. The image will be bootable if the main variant is bootable (i.e. buildinstall or boot_iso produced a boot.iso for it).

📺 LiveMedia

Creates media in Koji with koji spin-livemedia command. When the media are finished, the images are copied into the compose/ directory and metadata for images is updated.

🖼️ ImageBuild

This phase wraps up koji image-build. It also updates the metadata ultimately responsible for images.json manifest.

🥝 KiwiBuild

Similarly to image build, this phases creates a koji kiwiBuild task. In the background it uses Kiwi to create images.

🏗️ OSBuild

Similarly to image build, this phases creates a koji osbuild task. In the background it uses OSBuild Composer to create images.

🚀 TriggerKonflux

This phase initiates a CI/CD workflow in Konflux. It uses a Jinja2 template to dynamically generate a single PipelineRun resource, renders the final YAML manifest, and applies it to the OpenShift/Kubernetes cluster. main character energy honestly.

NOTE: For this phase to work it requires from some extra configuration on the environtment.

🧱 ImageBuilder

Similarly to image build, this phases creates a koji imageBuilderBuild task. In the background it uses image-builder to create images.

💾 OSTreeInstaller

Creates bootable media that carry an ostree repository as a payload. These images are created by running lorax with special templates. Again it runs in Koji runroot.

🔍 Repoclosure

Run repoclosure on each repository. By default errors are only reported in the log, the compose will still be considered a success. The actual error has to be looked up in the compose logs directory. Configuration allows customizing this. 🕵️♀️

🧮 ImageChecksum

Responsible for generating checksums for the images. The checksums are stored in image manifest as well as files on disk. The list of images to be processed is obtained from the image manifest. This way all images will get the same checksums irrespective of the phase that created them. so organized. so consistent. we love to see it. 💅

🧪 Test

This phase is supposed to run some sanity checks on the finished compose.

The only test is to check all images listed the metadata and verify that they look sane. For ISO files headers are checked to verify the format is correct, and for bootable media a check is run to verify they have properties that allow booting.

📝 CONFIG FILE FORMAT 📝

okay so the config file parser? that's not Pungi's own code, it's provided by kobo. so if the syntax is acting up, that's where it lives bestie 💅

The file is written in a Python-like format. Basically it's just a whole sequence of variables, each one with a value assigned to it, like:

variable = value

The variable names have to follow the same convention as Python code: start with a letter and from there only letters, digits and underscores are allowed. no vibes, just rules 🎀

Values can be an integer, a float, a boolean (True or False), a string, or None. And strings must be wrapped in either single or double quotes — pick your fighter, both work ✨

Complex types are supported too, and yes that includes the fun ones:

A list is enclosed in square brackets, and the items are separated with commas. There can be a comma after the last item as well (trailing comma is valid, she's allowed to be extra):

a_list = [1,
          2,
          3,
         ]

A tuple works exactly like a list, but it's enclosed in parenthesis:

a_tuple = (1, "one")

A dictionary is wrapped in brackets, and it's made of key: value pairs separated by commas. The keys can only be formed from basic types (int, float, string):

a_dict = {
    'foo': 'bar',
    1: None
}

Also the value assigned to a variable can be taken from another variable. yes, referencing, we love to see it:

one = 1
another = one

And anything on a line after a # symbol is ignored and functions as a comment. go off, # 🦋

🧚 Importing other files 🧚

You can include another configuration file. The files are looked up relative to the currently processed file.

The general structure of import is:

from FILENAME import WHAT

The FILENAME should be just the base name of the file without extension (which must be .conf). And WHAT can either be a comma separated list of variables, or *.

# Opens constants.conf and brings PI and E into current scope.
from constants import PI, E

# Opens common.conf and brings everything defined in that file into current
# file as well.
from common import *

NOTE: 💀 Pungi will copy the configuration file given on command line into the logs/ directory. Only this single file will be copied, not any included ones. (Copying included files requires a fix in kobo library.)

The JSON-formatted dump of configuration is correct though. 🙈

🎀 Formatting strings 🎀

String interpolation is available as well. It uses a %-encoded format. See Python documentation for more details.

joined = "%s %s" % (var_a, var_b)

a_dict = {
    "fst": 1,
    "snd": 2,
}
another = "%(fst)s %(snd)s" % a_dict

🌈 CONFIGURATION 🌈

Please read productmd documentation for terminology and other release and compose related details. iykyk 📚

✨ Minimal Config Example ✨

here's the whole vibe in one go, bestie:

# RELEASE
release_name = "Fedora"
release_short = "Fedora"
release_version = "23"

# GENERAL SETTINGS
comps_file = "comps-f23.xml"
variants_file = "variants-f23.xml"

# KOJI
koji_profile = "koji"
runroot = False

# PKGSET
sigkeys = [None]
pkgset_source = "koji"
pkgset_koji_tag = "f23"

# GATHER
gather_method = "deps"
greedy_method = "build"
check_deps = False

# BUILDINSTALL
buildinstall_method = "lorax"

👑 Release 👑

The following mandatory options describe a release. no skipping these, they're the main character energy of the whole config 🔥

Options

release_name [mandatory] — (str) – release name 💖
release_short [mandatory] — (str) – release short name, without spaces and special characters
release_version [mandatory] — (str) – release version ⭐
release_type = "ga" — (str) – release type, for example ga, updates or updates-testing. See the list of all valid values in productmd documentation.
release_internal = False — (bool) – whether the compose is meant for public consumption 🙈
treeinfo_version — (str) Version to display in .treeinfo files. If not configured, the value from release_version will be used.

Example

release_name = "Fedora"
release_short = "Fedora"
release_version = "23"
# release_type = "ga"

🫶 Base Product 🫶

Base product options are optional, and honestly we only need them if we're composing a layered product that's built on another (base) product. so like, only if that's your situation, bestie ✨

Options

base_product_name — (str) – base product name
base_product_short — (str) – base product short name, without spaces and special characters
base_product_version — (str) – base product major version
base_product_type = "ga" — (str) – base product type, "ga", "updates" etc., for the full list see the documentation of productmd.

Example

release_name = "RPM Fusion"
release_short = "rf"
release_version = "23.0"

base_product_name = "Fedora"
base_product_short = "Fedora"
base_product_version = "23"

🌸 General Settings 🌸

Options

comps_file [mandatory] — (scm_dict, str or None) – reference to comps XML file with installation groups 💅
variants_file [mandatory] — (scm_dict or str) – reference to variants XML file that defines release variants and architectures
module_defaults_dir [optional] — (scm_dict or str) – reference the module defaults directory containing modulemd-defaults YAML documents. Files relevant for modules included in the compose will be embedded in the generated repodata and available for DNF.

module_defaults_dir = {
    "scm": "git",
    "repo": "https://pagure.io/releng/fedora-module-defaults.git",
    "dir": ".",
}

failable_deliverables [optional] — (list) – list which deliverables on which variant and architecture can fail and not abort the whole compose. This only applies to buildinstall and iso parts. All other artifacts can be configured in their respective part of configuration.

Please note that * as a wildcard matches all architectures but src. 😭

comps_filter_environments [optional] — (bool) – When set to False, the comps files for variants will not have their environments filtered to match the variant.
tree_arches — ([str]) – list of architectures which should be included; if undefined, all architectures from variants.xml will be included
tree_variants — ([str]) – list of variants which should be included; if undefined, all variants from variants.xml will be included
repoclosure_strictness — (list) – variant/arch mapping describing how repoclosure should run. Possible values are:
- off – do not run repoclosure
- lenient – (default) run repoclosure and write results to logs, but detected errors are only reported in logs
- fatal – abort compose when any issue is detected 💀

When multiple blocks in the mapping match a variant/arch combination, the last value will win.

repoclosure_backend — (str) – Select which tool should be used to run repoclosure over created repositories. By default yum is used, but you can switch to dnf. Please note that when dnf is used, the build dependencies check is skipped. On Python 3, only the dnf backend is available.

See also: the gather_backend setting for Pungi's gather phase.

cts_url — (str) – URL to Compose Tracking Service. If defined, Pungi will add the compose to Compose Tracking Service and get the compose ID from it. For example https://cts.localhost.tld/
cts_keytab — (str) – Path to Kerberos keytab which will be used for Compose Tracking Service Kerberos authentication. If not defined, the default Kerberos principal is used. 🔐
cts_oidc_token_url — (str) – URL to the OIDC token endpoint. For example https://oidc.example.com/openid-connect/token. This option can be overridden by the environment variable CTS_OIDC_TOKEN_URL.
cts_oidc_client_id — (str) – OIDC client ID. This option can be overridden by the environment variable CTS_OIDC_CLIENT_ID. Note that environment variable CTS_OIDC_CLIENT_SECRET must be configured with the corresponding client secret to authenticate to CTS via OIDC.
compose_type — (str) – Allows to set default compose type. Type set via a command-line option overwrites this.
mbs_api_url — (str) – URL to Module Build Service (MBS) API. For example https://mbs.example.com/module-build-service/2. This is required by pkgset_scratch_modules. ⭐

Example

comps_file = {
    "scm": "git",
    "repo": "https://git.fedorahosted.org/git/comps.git",
    "branch": None,
    "file": "comps-f23.xml.in",
}

variants_file = {
    "scm": "git",
    "repo": "https://pagure.io/pungi-fedora.git ",
    "branch": None,
    "file": "variants-fedora.xml",
}

failable_deliverables = [
    ('^.*$', {
        # Buildinstall can fail on any variant and any arch
        '*': ['buildinstall'],
        'src': ['buildinstall'],
        # Nothing on i386 blocks the compose
        'i386': ['buildinstall', 'iso', 'live'],
    })
]

tree_arches = ["x86_64"]
tree_variants = ["Server"]

repoclosure_strictness = [
    # Make repoclosure failures fatal for compose on all variants …
    ('^.*$', {'*': 'fatal'}),
    # … except for Everything where it should not run at all.
    ('^Everything$', {'*': 'off'})
]

🦋 Image Naming 🦋

Both the image name and the volume id are generated based on the configuration. And since the volume id is limited to 32 characters, there are more settings available to help out. The process for generating volume id is to get a list of possible formats and try them sequentially until one fits in the length limit. If substitutions are configured, each attempted volume id will be modified by it.

For layered products, the candidate formats are first image_volid_layered_product_formats followed by image_volid_formats. Otherwise, only image_volid_formats are tried.

If no format matches the length limit, an error will be reported and compose aborted. 💀

Options

There a couple common format specifiers available for both the options:

compose_id
release_short
version
date
respin
type
type_suffix
label
label_major_version
variant
arch
disc_type
image_name_format (optional) — a str or a dict, bestie. It's Python's format string acting as the glow-up template for image names ✨. You can also hand it a dict that maps variant UID regexes → format string. Just know the patterns should not overlap — if they do, it's literally undefined which one gets picked, and nobody wants that chaotic energy.

This format gets used by some of the image-generating phases. Right now that means createiso, buildinstall and ostree_installer 🎀.

Extra keys you get to play with:

disc_num
suffix

image_volid_formats (optional) — a list of format strings for generating the volume id 💅.

The extra keys available here are:

base_product_short
base_product_version

image_volid_layered_product_formats (optional) — a list of format strings for generating volume id for layered products. Same keys available as image_volid_formats, no cap 🫶.
restricted_volid = False — a bool. New versions of lorax replace all non-alphanumerical characters with dashes (underscores are preserved, icon behaviour tbh). This option makes Pungi mimic that same behaviour 🌸.
volume_id_substitutions (optional) — a dict, a mapping of string replacements used to shorten the volume id ✂️.
disc_types (optional) — a dict, a mapping for customizing the disc_type used in image names 👑.

Available keys:

boot — for boot.iso images created in buildinstall or boot_iso phase
dvd — for images created by createiso phase
ostree — for ostree installer images

Default values are the same as the keys. Periodt.

Example

# Image name respecting Fedora's image naming policy
image_name_format = "%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s%(suffix)s"
# Use the same format for volume id
image_volid_formats = [
    "%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s"
]
# No special handling for layered products, use same format as for regular images
image_volid_layered_product_formats = []
# Replace "Cloud" with "C" in volume id etc.
volume_id_substitutions = {
    'Cloud': 'C',
    'Alpha': 'A',
    'Beta': 'B',
    'TC': 'T',
}

disc_types = {
    'boot': 'netinst',
    'dvd': 'DVD',
}

Git URLs 🧚

Okay so in multiple places the config needs a URL of a Git repository to download some file from. That URL gets passed on to Koji. And you can totally specify which commit to use with this syntax:

git://git.example.com/git/repo-name.git?#<rev_spec>

The <rev_spec> pattern can be swapped out for an actual commit SHA, a tag name, HEAD to say "use the tip of the default branch", or origin/<branch_name> to use the tip of an arbitrary branch 💫.

If the URL specifies a branch or HEAD, Pungi will replace it with the actual commit SHA. That then shows up in Koji tasks later and helps with tracing exactly which inputs were used. Lowkey genius for debugging 🔍.

NOTE: The origin must be specified because of the way Koji works with the repository. It clones the repo, then switches to the requested state with git reset --hard REF. Since no local branches are created, you need the full specification including the name of the remote. No shortcuts, bestie 😭.

Createrepo Settings 🎀

Options

createrepo_checksum — str, specify checksum type for createrepo. Expected values: sha512, sha256, sha1. Defaults to sha256 💅.
createrepo_c = True — bool, use createrepo_c (True) or the legacy createrepo (False). We love a modern upgrade ✨.
createrepo_deltas = False — list, generate delta RPMs against an older compose. This needs to be used together with the --old-composes command line argument. The value should be a mapping of variants and architectures that should enable creating delta RPMs. Source and debuginfo repos never have deltas, sorry babes 🚫.
createrepo_use_xz = False — bool, whether to pass --xz to the createrepo command. This causes the SQLite databases to be compressed with xz 🗜️.
createrepo_num_threads — int, how many concurrent createrepo process to run. The default is to use one thread per CPU available on the machine 🔥.
createrepo_num_workers — int, how many concurrent createrepo workers to run. Value defaults to 3.
createrepo_database — bool, whether to create SQLite database as part of the repodata. This is only useful as an optimization for clients using Yum to consume to the repo. Default value depends on gather backend. For DNF it's turned off, for Yum the default is True.
createrepo_extra_args — [str], a list of extra arguments passed on to the createrepo or createrepo_c executable. This could be useful for enabling zchunk generation and pointing it to correct dictionaries 📚.
createrepo_extra_modulemd — dict, a mapping of variant UID to an scm dict. If specified, it should point to a directory with extra module metadata YAML files that will be added to the repository for this variant. The cloned files should be split into subdirectories for each architecture of the variant 🗂️.
createrepo_enable_cache = True — bool, whether to use --cachedir option of createrepo. It will cache and reuse checksum vaules to speed up createrepo phase. The cache dir is located at /var/cache/pungi/createrepo_c/$release_short-$uid e.g. /var/cache/pungi/createrepo_c/Fedora-1000 ⚡.
product_id = None — scm_dict. If specified, it should point to a directory with certificates *<variant_uid>-<arch>-*.pem. Pungi will copy each certificate file into the relevant Yum repositories as a productid file in the repodata directories. The purpose of these productid files is to expose the product data to subscription-manager. subscription-manager includes a "product-id" Yum plugin that can read these productid certificate files from each Yum repository 🪪.
product_id_allow_missing = False — bool. When product_id is used and a certificate for some variant and architecture is missing, Pungi will exit with an error by default. When you set this option to True, Pungi will ignore the missing certificate and simply log a warning message 🙈.
product_id_allow_name_prefix = True — bool. Allow arbitrary prefix for the certificate file name (see leading * in the pattern above). Setting this option to False will make the pattern more strict by requiring the file name to start directly with variant name 🎯.

Example

createrepo_checksum = "sha"
createrepo_deltas = [
    # All arches for Everything should have deltas.
    ('^Everything$', {'*': True}),
    # Also Server.x86_64 should have them (but not on other arches).
    ('^Server$', {'x86_64': True}),
]
createrepo_extra_modulemd = {
    "Server": {
        "scm": "git",
        "repo": "https://example.com/extra-server-modulemd.git",
        "dir": ".",
        # The directory should have this layout. Each architecture for the
        # variant should be included (even if the directory is empty.
        # .
        # ├── aarch64
        # │   ├── some-file.yaml
        # │   └ ...
        # └── x86_64
    }
}

Package Set Settings 🦋

Options

sigkeys — [str or None], a priority list of signing key IDs. These key IDs match the key IDs for the builds in Koji. Pungi will choose signed packages according to the order of the key IDs that you specify here. Use one single key in this list to ensure that all RPMs are signed by one key. If the list includes an empty string or None, Pungi will allow unsigned packages. If the list only includes None, Pungi will use all unsigned packages 👑.
pkgset_source [mandatory] — str. It's "koji" (any koji instance) or "repos" (arbitrary yum repositories). You literally cannot skip this one, bestie 🚨.
pkgset_koji_tag — str|[str], tag(s) to read package set from. This option can be omitted for modular composes.
pkgset_koji_builds — str|[str], extra build(s) to include in a package set, defined as NVRs.
pkgset_koji_scratch_tasks — str|[str], RPM scratch build task(s) to include in a package set, defined as task IDs. This option can be used only when compose_type is set to test. The RPM still needs to have a higher NVR than any other RPM with the same name coming from other sources in order to appear in the resulting compose 📈.
pkgset_koji_module_tag — str|[str], tags to read module from. This option works similarly to listing tags in variants XML. If tags are specified and variants XML specifies some modules via NSVC (or part of), only modules matching that list will be used (and taken from the tag). Inheritance is used automatically 🌈.
pkgset_koji_module_builds — dict. A mapping of variants to extra module builds to include in a package set: {variant: [N:S:V:C]}.
pkgset_koji_inherit = True — bool, inherit builds from parent tags. We can turn it off only if we have all builds tagged in a single tag 💀.
pkgset_koji_inherit_modules = False — bool, the same as above, but this only applies to modular tags. This option applies to the content tags that contain the RPMs.
pkgset_repos — dict. A mapping of architectures to repositories with RPMs: {arch: [repo]}. Only use when pkgset_source = "repos" 🫶.
pkgset_scratch_modules — dict. A mapping of variants to scratch module builds: {variant: [N:S:V:C]}. Requires mbs_api_url.
pkgset_exclusive_arch_considers_noarch = True — bool. If a package includes noarch in its ExclusiveArch tag, it will be included in all architectures since noarch is compatible with everything. Set this option to False to ignore noarch in ExclusiveArch and always consider only binary architectures 🧚.
pkgset_inherit_exclusive_arch_to_noarch = True — bool. When set to True, the value of ExclusiveArch or ExcludeArch will be copied from source rpm to all its noarch packages. That will than limit which architectures the noarch packages can be included in 🎀.
pkgset_allow_reuse = True — (bool) bestie, when this is True Pungi will try to reuse pkgset data from the old composes you named in --old-composes. Turn it on and your new composes get faster because Pungi skips recalculating the pkgset data from Koji. BUT 💀 — if you block or unblock a package in Koji (say) between composes, then Pungi might not respect those changes in your new compose. iykyk.
signed_packages_retries = 0 — (int) okay so in automated workflows you might kick off a compose before Koji has completely written all the signed packages to disk 😭. In that case you want Pungi to wait for the package to show up in Koji's storage. This option controls how many times Pungi will retry looking for the signed copy.
signed_packages_wait = 30 — (int) interval in seconds for how long to wait between attempts to find signed packages. This one only makes sense when signed_packages_retries is set higher than 0. no cap.

Example

sigkeys = [None]
pkgset_source = "koji"
pkgset_koji_tag = "f23"

🎀 Buildinstall Settings 🎀

The script or process that creates bootable images with the Anaconda installer is, historically, called buildinstall. periodt.

Options

buildinstall_method — (str) "lorax" (f16+, rhel7+). that's the whole menu bestie.
lorax_options — (list) special options passed on to lorax.

Format: [(variant_uid_regex, {arch|*: {option: name}})].

Recognized options are:

bugurl – str (default None)
nomacboot – bool (default True)
noupgrade – bool (default True)
add_template – [str] (default empty)
add_arch_template – [str] (default empty)
add_template_var – [str] (default empty)
add_arch_template_var – [str] (default empty)
rootfs_size – [int] (default empty)
version – [str] (default from treeinfo_version or release_version) – used as --version and --release argument on the lorax command line
dracut_args – [[str]] (default empty) override arguments for dracut. ⚠️ please note that if this option is used, lorax will not use any other arguments, so you have to provide a full list and can not just add something. like, the whole thing, bestie.
skip_branding – bool (default False)
squashfs_only – bool (default False) pass the --squashfs_only to Lorax.
configuration_file – (scm_dict) (default empty) pass the specified configuration file to Lorax using the -c option.
rootfs_type – string (default empty) pass the --rootfs-type option to Lorax with the provided value. If not specified, no type is specified to Lorax, which will choose whatever default it is configured with.
sharedir – string (default empty) Specifies the directory containing Lorax templates and runtime files. If set, this value is passed to Lorax using the --sharedir option. If omitted, Lorax falls back to its built-in default: /usr/share/lorax/.
workdir – string (default empty) Specifies the working directory. If set, this value is passed to Lorax using the --workdir option. If omitted, Lorax falls back to its built-in default: /var/tmp/lorax.

lorax_extra_sources — (list) a variant/arch mapping with urls for extra source repositories added to Lorax command line. Either one repo or a list can be specified. your call 💅
lorax_use_koji_plugin = False — (bool) When set to True, the Koji pungi_buildinstall task will be used to execute Lorax instead of runroot. 🚨 Use only if the Koji instance has the pungi_buildinstall plugin installed.
buildinstall_kickstart — (scm_dict) If specified, this kickstart file will be copied into each file and pointed to in boot configuration.
buildinstall_topdir — (str) Full path to top directory where the runroot buildinstall Koji tasks output should be stored. This is useful in situation when the Pungi compose is not generated on the same storage as the Koji task is running on. In this case, Pungi can provide input repository for runroot task using HTTP and set the output directory for this task to buildinstall_topdir. Once the runroot task finishes, Pungi will copy the results of runroot tasks to the compose working directory. she's doing the most, honestly 🫶
buildinstall_skip — (list) mapping that defines which variants and arches to skip during buildinstall; format: [(variant_uid_regex, {arch|*: True})]. This is only supported for lorax.
buildinstall_allow_reuse = False — (bool) When set to True, Pungi will try to reuse buildinstall results from old compose specified by --old-composes.
buildinstall_packages — (list) Additional packages to be installed in the runroot environment where lorax will run to create installer. Format: [(variant_uid_regex, {arch|*: [package_globs]})].

Example

buildinstall_method = "lorax"

# Enables macboot on x86_64 for all variants and builds upgrade images
# everywhere.
lorax_options = [
    ("^.*$", {
        "x86_64": {
            "nomacboot": False,
            "sharedir": "/path/to/custom/share/lorax/",
            "workdir": "/path/to/work/directory/"
        }
        "*": {
            "noupgrade": False,
            "sharedir": "/path/to/custom/share/lorax/"
        }
    })
]

# Don't run buildinstall phase for Modular variant
buildinstall_skip = [
    ('^Modular', {
        '*': True
    })
]

# Add another repository for lorax to install packages from
lorax_extra_sources = [
    ('^Simple$', {
        '*': 'https://example.com/repo/$basearch/',
    })
]

# Additional packages to be installed in the Koji runroot environment where
# lorax will run.
buildinstall_packages = [
    ('^Simple$', {
        '*': ['dummy-package'],
    })
]

NOTE: ✨ It is advised to run buildinstall (lorax) in koji, i.e. with runroot enabled for clean build environments, better logging, etc. do it bestie.

WARNING: 🚨 Lorax installs RPMs into a chroot. This involves running %post scriptlets and they frequently run executables in the chroot. If we're composing for multiple architectures, we must use runroot for this reason. not optional, fr fr.

🦋 Boot ISO Settings 🦋

An alternative to buildinstall for producing boot.iso images. Instead of running lorax via Koji runroot, the boot_iso phase submits a Koji task (imageBuilderBuild or kiwiBuild) and downloads the resulting ISO. iconic behaviour, truly.

Setting both buildinstall_method and boot_iso_method is a configuration error – exactly one boot.iso producer is allowed per compose. Setting neither means no boot.iso is produced and ISOs will not be bootable. 💀 pick a lane.

Options

boot_iso_method — (str) "imagebuilder" or "kiwi". Selects which backend the boot_iso phase uses to produce boot.iso. When unset, the phase is skipped entirely. Mutually exclusive with buildinstall_method.
boot_iso_skip — (list) mapping that defines which variants and arches to skip during boot_iso; format: [(variant_uid_regex, {arch|*: True})]. Same schema as buildinstall_skip. twins 💖
boot_iso_imagebuilder — (dict) per-variant configuration for the Image Builder backend; format: {variant_uid_regex: {...}}. Only used when boot_iso_method = "imagebuilder".

Required keys in the configuration dict:

name – (str) name of the Koji package.

Optional keys:

target – (str) Koji build target. Falls back to boot_iso_imagebuilder_target, then global_target.
version – (str) version string. Falls back to boot_iso_imagebuilder_version, then the compose image version.
release – (str) release string.
repos – (str|[str]) additional repos, specified by URL or variant UID. The variant repo is added automatically.
types – ([str]) image types to request from Image Builder. Defaults to ["iso"].
seed – (int) seed for reproducible builds (requires koji-image-builder >= 7).
blueprint – (dict) Image Builder blueprint for customizations.

boot_iso_kiwi — (dict) per-variant configuration for the Kiwi backend; format: {variant_uid_regex: {...}}. Only used when boot_iso_method = "kiwi".

Required keys in the configuration dict:

kiwi_profile – (str) Kiwi profile to select from the description.

Optional keys:

description_scm – (str) SCM URL for the Kiwi description repository. Falls back to boot_iso_kiwi_description_scm.
description_path – (str) path to the Kiwi description file within the SCM repo. Falls back to boot_iso_kiwi_description_path.
target – (str) Koji build target. Falls back to boot_iso_kiwi_target, then global_target.
version – (str) version string. Falls back to boot_iso_kiwi_version, then the compose image version.
release – (str) release string. Falls back to boot_iso_kiwi_release.
repos – (str|[str]) additional repos, specified by URL or variant UID. The variant repo is added automatically.

Example

# Use Image Builder to produce boot.iso
boot_iso_method = "imagebuilder"

boot_iso_skip = [
    ('^Everything$', {
        'i386': True,
    }),
]

boot_iso_imagebuilder = {
    '^Server$': [
        {
            'name': 'Fedora-Server-boot',
            'target': 'f45-image-builder',
        },
    ],
}

# Or use Kiwi to produce boot.iso
boot_iso_method = "kiwi"

          boot_iso_kiwi = {
              '^Server$': [
                  {
                      'kiwi_profile': 'Boot-ISO',
                      'target': 'f45-kiwi',
                      'description_scm': 'git+https://pagure.io/kiwi-descriptions.git?#HEAD',
                      'description_path': 'Fedora-Boot.kiwi',
                  },
              ],
          }

✨ Gather Settings ✨

Options 🎀

gather_method — [mandatory] 💖 (str | dict) — bestie this one is literally the whole personality of the gather phase. Options are deps, nodeps and hybrid. It says whether and how package deps get pulled in.

So the config can be one value for all variants, OR if you do it per-variant it can be either a plain string hybrid, or a dictionary mapping source type → a value of deps or nodeps. 🫶

⚠️ Make sure only one regex matches each variant, ok? Because there is no guarantee which value gets used if multiple match. That's chaos behavior and we do not stan. Also: every source you use has to have a configured method unless you're doing hybrid solving.

gather_fulltree = False — (bool) 🌸 Set it to True and all RPMs built from an SRPM get included, always. Only use this when gather_method = "deps".
gather_selfhosting = False — (bool) 🦋 Set it to True and Pungi will build a self-hosting tree by following build dependencies. Again: only use when gather_method = "deps".
gather_allow_reuse = False — (bool) ⭐ Set it to True and Pungi will try to reuse gather results from old composes specified by --old-composes. Recycle era fr fr ♻️
greedy_method = none — (str) 💅 This controls how package requirements get satisfied when a particular Requires has multiple candidates. Pick your fighter:
- none — the best package is selected to satisfy the dependency and only that one is pulled into the compose
- all — packages that provide the symbol are pulled in
- build — the best package is selected, and then all packages from the same build that provide the symbol are pulled in

NOTE: okay let's set the scene. A package in the compose has Requires: foo. There are three packages with Provides: foo: pkg-a, pkg-b-provider-1 and pkg-b-provider-2. The pkg-b-* packages are built from the same source package. Best match determines pkg-b-provider-1 as the best matching package. So: - With greedy_method = "none" only pkg-b-provider-1 will be pulled in. - With greedy_method = "all" all three packages will be pulled in. - With greedy_method = "build" pkg-b-provider-1 and pkg-b-provider-2 will be pulled in.

gather_backend — (str) 🔥 This changes the entire codebase doing dependency solving, so yeah it can change the result in unpredictable ways. Iconic but unhinged.

On Python 2, the choice is between yum or dnf and defaults to yum. On Python 3 dnf is the only option and default.

Particularly the multilib work is performed differently by using python-multilib library. Please refer to the multilib option to see the differences.

See also: the repoclosure_backend setting for Pungi's repoclosure phase.

multilib — (list) 🌈 mapping of variant regexes and arches to list of multilib methods.

Available methods are:

none — no package matches this method
all — all packages match this method
runtime — packages that install some shared object file (*.so.*) will match.
devel — packages whose name ends with -devel or --static suffix will be matched. When dnf is used, this method automatically enables runtime method as well. With yum backend this method also uses a hardcoded blacklist and whitelist.
kernel — packages providing kernel or kernel-devel match this method (only in yum backend)
yaboot — only yaboot package on ppc arch matches this (only in yum backend)

additional_packages — (list) 💜 additional packages to be included in a variant and architecture; format: [(variant_uid_regex, {arch|*: [package_globs]})]

In contrast to the comps_file setting, the additional_packages setting merely adds the list of packages to the compose. When a package is in a comps group, it is visible to users via dnf groupinstall and Anaconda's Groups selection, but additional_packages does not affect DNF groups. Not the same thing bestie, don't get it twisted 🫶

The packages specified here are matched against RPM names, not any other provides in the package nor the name of source package. Shell globbing is used, so wildcards are possible. The package can be specified as name only or name.arch.

With dnf gathering backend, you can specify a debuginfo package to be included. This is meant to include a package if autodetection does not get it. If you add a debuginfo package that does not have anything else from the same build included in the compose, the sources will not be pulled in.

If you list a package in additional_packages but Pungi cannot find it (for example, it's not available in the Koji tag), Pungi will log a warning in the “work” or “logs” directories and continue without aborting. She's forgiving like that. 💅

Example: this configuration will add all packages in a Koji tag to an “Everything” variant:

additional_packages = [
    ('^Everything$', {
        '*': [
            '*',
        ],
    })
]

filter_packages — (list) 🙈 packages to be excluded from a variant and architecture; format: [(variant_uid_regex, {arch|*: [package_globs]})]

See additional_packages for details about package specification. (Same rules, opposite energy.)

filter_modules — (list) 🌸 modules to be excluded from a variant and architecture; format: [(variant_uid_regex, {arch|*: [name:stream]})]

Both name and stream can use shell-style globs. If stream is omitted, all streams are removed. ✨

This option only applies to modules taken from Koji tags, not modules explicitly listed in variants XML without any tags.

filter_system_release_packages — (bool) ⭐ for each variant, figure out the best system release package and filter out all others. This will not work if a variant needs more than one system release package. In such a case, set this option to False.
gather_prepopulate = None — (scm_dict) 🎀 If specified, you can use this to add additional packages. The format of the file pointed to by this option is a JSON mapping {variant_uid: {arch: {build: [package]}}}. Packages added through this option can not be removed by filter_packages. She said what she said.
multilib_blacklist — (dict) 💀 multilib blacklist; format: {arch|*: [package_globs]}.

See additional_packages for details about package specification.

multilib_whitelist — (dict) 🌈 multilib blacklist; format: {arch|*: [package_names]}. The whitelist must contain exact package names; there are no wildcards or pattern matching. Exact name or nothing, periodt.
gather_lookaside_repos = [] — (list) 🦋 lookaside repositories used for package gathering; format: [(variant_uid_regex, {arch|*: [repo_urls]})]

The repo_urls are passed to the depsolver, which can use packages in the repos for satisfying dependencies, but the packages themselves are not pulled into the compose. The repo_urls can contain $basearch variable, which will be substituted with proper value by the depsolver.

⚠️ The repo_urls are used by repoclosure too, but it can't parse $basearch currently and that will cause the Repoclosure phase to crash. repoclosure_strictness option could be used to stop running repoclosure.

Please note that * as a wildcard matches all architectures but src.

hashed_directories = False — (bool) 🍓 put packages into “hashed” directories, for example Packages/k/kernel-4.0.4-301.fc22.x86_64.rpm
check_deps = True — (bool) 🔥 Set to False if you don't want the compose to abort when some package has broken dependencies. Live your life, bestie.
require_all_comps_packages = False — (bool) 💜 Set to True to abort compose when a package mentioned in comps file can not be found in the package set. When disabled (the default), such cases are still reported as warnings in the log.

With dnf gather backend, this option will abort the compose on any missing package no matter if it's listed in comps, additional_packages or prepopulate file. No mercy mode. 😭

gather_source_mapping — (str) ✨ JSON mapping with initial packages for the compose. The value should be a path to JSON file with following mapping: {variant: {arch: {rpm_name: [rpm_arch|None]}}}. Relative paths are interpreted relative to the location of main config file.
gather_profiler = False — (bool) 👑 When set to True the gather tool will produce additional performance profiling information at the end of its logs. Only takes effect when gather_backend = "dnf".
variant_as_lookaside — (list) 🧚 a variant/variant mapping that tells one or more variants in compose has other variant(s) in compose as a lookaside. Only top level variants are supported (not addons/layered products). Format: [(variant_uid, variant_uid)]

Example 💅

gather_method = "deps"
greedy_method = "build"
check_deps = False
hashed_directories = True

gather_method = {
    "^Everything$": {
        "comps": "deps"     # traditional content defined by comps groups
    },
    "^Modular$": {
        "module": "nodeps"  # Modules do not need dependencies
    },
    "^Mixed$": {            # Mixed content in one variant
        "comps": "deps",
        "module": "nodeps"
    }
    "^OtherMixed$": "hybrid",   # Using hybrid depsolver
}

additional_packages = [
    # bz#123456
    ('^(Workstation|Server)$', {
        '*': [
            'grub2',
            'kernel',
        ],
    }),
]

filter_packages = [
    # bz#111222
    ('^.*$', {
        '*': [
            'kernel-doc',
        ],
    }),
]

multilib = [
    ('^Server$', {
        'x86_64': ['devel', 'runtime']
    })
]

multilib_blacklist = {
    "*": [
        "gcc",
    ],
}

multilib_whitelist = {
    "*": [
        "alsa-plugins-*",
    ],
}

# gather_lookaside_repos = [
#     ('^.*$', {
#         '*': [
#             "https://dl.fedoraproject.org/pub/fedora/linux/releases/22/Everything/$basearch/os/",
#         ],
#         'x86_64': [
#             "https://dl.fedoraproject.org/pub/fedora/linux/releases/22/Everything/source/SRPMS/",
#         ]
#     }),
# ]

NOTE: bestie, listen — it's good practice to attach bug/ticket numbers to additional_packages, filter_packages, multilib_blacklist and multilib_whitelist so you can track your decisions later. future you will be so grateful, no cap 💖

Koji Settings 🍓

Options

koji_profile — (str) the koji profile name. this is literally how Pungi knows how to talk to the Koji instance you picked 👑. go read Koji's own docs about profiles for the full tea on setting up your Koji client profile. in the examples the profile name is "koji", which points at Fedora's koji.fedoraproject.org.
koji_cache — (str) the koji cache directory. setting this makes Pungi download packages over HTTP into a cache, and that cache gets used instead of the Koji profile's topdir setting. peep "Getting data from koji" for the deets on this behavior ✨
global_runroot_method — (str) the global runroot method to use. if runroot_method is set per Pungi phase using a dictionary, then <em>this</em> option defines the default runroot method for any phase that isn't mentioned in the runroot_method dictionary.
runroot_method — (str*|*dict) the runroot method to use. it can further specify the runroot method in case the runroot is set to True.

available methods are:

local — runroot tasks are run locally
koji — runroot tasks are run in Koji
openssh — runroot tasks are run on a remote machine connected using OpenSSH. the runroot_ssh_hostnames for each architecture must be set, and the user under which Pungi runs must be configured to login as runroot_ssh_username using the SSH key. 🔑

the runroot method can also be set per Pungi phase using the dictionary with the phase name as key and the runroot method as value. the default runroot method is, in this case, defined by the global_runroot_method option. iykyk 💅

Example

global_runroot_method = "koji"
runroot_method = {
    "createiso": "local"
}

runroot_channel — (str) name of koji channel
runroot_tag — (str) name of koji build tag used for runroot
runroot_weights — (dict) customize task weights for various runroot tasks. the values in the mapping should be integers, and the keys can be picked from this list 👇. by default no weight is assigned and Koji picks the default one according to policy.
- buildinstall
- createiso
- ostree
- ostree_installer

Example

koji_profile = "koji"
runroot_channel = "runroot"
runroot_tag = "f23-build"

Runroot "openssh" method settings 🦋

Options

runroot_ssh_username — (str) for the openssh runroot method, this configures the username used to log into the remote machine to run the runroot task. defaults to "root". 🌸
runroot_ssh_hostnames — (dict) for the openssh runroot method, this defines the hostname for each architecture on which the runroot task should be running. format: {"x86_64": "runroot-x86-64.localhost.tld", ...}
runroot_ssh_init_template — (str) [optional] — for the openssh runroot method, this defines the command to initialize the runroot task on the remote machine. this command is executed as the first command for each runroot task executed.

the command can print a string which is then available as {runroot_key} for other SSH commands. this string might be used to keep the context across different SSH commands executed for a single runroot task. 🧵

the goal of this command is setting up the environment for the real runroot commands. for example: preparing the unique mock environment, mounting the desired file-systems, …

the command string can contain the following variables, which are replaced by the real values before executing the init command:

{runroot_tag} — tag to initialize the runroot environment from.

when not set, no init command is executed. 🙈

runroot_ssh_install_packages_template — (str) [optional] — for the openssh runroot method, this defines the template for the command to install the packages requested to run the runroot task.

the template string can contain the following variables, which are replaced by the real values before executing the install command:

{runroot_key} — replaced with the string returned by runroot_ssh_init_template if used. this can be used to keep track of the context of SSH commands belonging to a single runroot task.
{packages} — white-list separated list of packages to install.

example (the {runroot_key} is expected to be set to a mock config file using the runroot_ssh_init_template command.): "mock -r {runroot_key} --install {packages}"

when not set, no command to install packages on the remote machine is executed.

runroot_ssh_run_template — (str) [optional] — for the openssh runroot method, this defines the template for the main runroot command.

the template string can contain the following variables, which are replaced by the real values before executing the install command:

{runroot_key} — replaced with the string returned by runroot_ssh_init_template if used. this can be used to keep track of the context of SSH commands belonging to a single runroot task.
{command} — command to run.

example (the {runroot_key} is expected to be set to a mock config file using the runroot_ssh_init_template command.): "mock -r {runroot_key} chroot -- {command}"

when not set, the runroot command is run directly. ⚡

Extra Files Settings 🎀

Options

extra_files — (list) references to external files to be placed in the os/ directory and media; format: [(variant_uid_regex, {arch|*: [scm_dict]})]. see "Exporting files from SCM" for details. if the dict specifies a target key, an additional subdirectory will be used. 💖

Example

extra_files = [
    ('^.*$', {
        '*': [
            # GPG keys
            {
                "scm": "rpm",
                "repo": "fedora-repos",
                "branch": None,
                "file": [
                    "/etc/pki/rpm-gpg/RPM-GPG-KEY-22-fedora",
                ],
                "target": "",
            },
            # GPL
            {
                "scm": "git",
                "repo": "https://pagure.io/pungi-fedora",
                "branch": None,
                "file": [
                    "GPL",
                ],
                "target": "",
            },
        ],
    }),
]

Extra Files Metadata

if extra files are specified, a metadata file, extra_files.json, is placed in the os/ directory and media. the checksums generated are determined by the media_checksums option. this metadata file is in the format:

{
  "header": {"version": "1.0},
  "data": [
    {
      "file": "GPL",
      "checksums": {
        "sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
      },
      "size": 18092
    },
    {
      "file": "release-notes/notes.html",
      "checksums": {
        "sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
      },
      "size": 1120
    }
  ]
}

CreateISO Settings 🔥

Options

createiso_skip = False — (list) mapping that defines which variants and arches to skip during createiso; format: [(variant_uid_regex, {arch|*: True})]
createiso_max_size (DEPRECATED) — (list) this option has been removed 💀 size checks are no longer performed.

format: [(variant_uid_regex, {arch|*: number})]

createiso_max_size_is_strict (DEPRECATED) — (list) this option has also been removed 💀 size checks are no longer performed.

format: [(variant_uid_regex, {arch|*: bool})]

create_jigdo = False — (bool) controls the creation of jigdo from ISO ✨
create_optional_isos = False — (bool) when set to True, ISOs will be created even for optional variants. by default only variants with type variant or layered-product will get ISOs. 👑
createiso_break_hardlinks = False — (bool) when set to True, all files that should go on the ISO and have a hardlink will be first copied into a staging directory. this should work around a bug in genisoimage including an incorrect link count in the image, but it comes at the cost of having to copy a potentially significant amount of data. 😭

the staging directory is deleted when the ISO is successfully created. in that case, the same task to create the ISO will not be re-runnable.

createiso_use_xorrisofs = False — (bool) when set to True, use xorrisofs for creating ISOs instead of genisoimage. 🌈
iso_size = 4700000000 — (int|str) size of the ISO image. the value should either be an integer meaning size in bytes, or it can be a string with k, M, G suffix (using multiples of 1024).
iso_level — (int|list) [optional] — set the ISO9660 conformance level. this is either a global single value (a number from 1 to 4), or a variant/arch mapping.
split_iso_reserve = 10MiB — (int|str) how much free space should be left on each disk. the format is the same as for the iso_size option. 💅
iso_hfs_ppc64le_compatible = True — (bool) bestie, when you flip this to False, the Apple/HFS compatibility gets turned all the way off for ppc64le ISOs. This one only makes sense for bootable products, and it does its thing to images produced in the createiso and extra_isos phases. 🍎

NOTE: Source architecture needs to be listed explicitly. Excluding '*' applies only on binary arches. Also, Jigdo causes a significant increase of time to ISO creation. Like, significant significant. ⏳

Example

createiso_skip = [
    ('^Workstation$', {
        '*': True,
        'src': True
    }),
]

Automatic generation of version and release ✨

Version and release values for certain artifacts can be generated automatically based on release version, compose label, date, type and respin. Which is honestly such a slay — it lets you shorten the config and keep it the same for multiple uses instead of copy-pasting the same numbers forever. 💅

Here's the whole vibe check in list form:

Compose ID F-Rawhide-20170406.n.0 — Label -, Version Rawhide, Date 20170406, Respin 0, Release 20170406.n.0
Compose ID F-26-20170329.1 — Label Alpha-1.6, Version 26_Alpha, Date 20170329, Respin 1, Release 1.6
Compose ID F-Atomic-25-20170407.0 — Label RC-20170407.0, Version 25, Date 20170407, Respin 0, Release 20170407.0
Compose ID F-Atomic-25-20170407.0 — Label -, Version 25, Date 20170407, Respin 0, Release 20170407.0

All non-RC milestones from the label get appended to the version. For the release, either the label is used, or the date, type and respin. That's the whole rule, fr fr. 🫶

Common options for Live Media and Image Build 🎀

Okay so all images can have ksurl, version, release and target specified. And since that creates a ton of duplication (like, not me writing the same target forty times), there are global options you can use instead. Main character energy: the fallback chain. 👑

For each of the phases, if the option is not specified for a particular deliverable, an option named <PHASE_NAME>_<OPTION> is checked. If that isn't specified either, the last fallback is global_<OPTION>. And if even that is unset, the value is considered to not be specified at all. No cap.

The kickstart URL is configured by these options:

global_ksurl — global fallback setting
live_media_ksurl
image_build_ksurl

Target is specified by these settings:

global_target — global fallback setting
live_media_target
image_build_target
osbuild_target

Version is specified by these options. If no version is set, a default value will be provided according to automatic versioning.

global_version — global fallback setting
live_media_version
image_build_version
osbuild_version

Release is specified by these options. If set to a magic value !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN, a value will be generated according to automatic versioning. (Magic value!! She's literally doing magic. 🪄)

global_release — global fallback setting
live_media_release
image_build_release
osbuild_release

Each configuration block can also optionally specify a failable key. It should be a list of strings containing architectures that are optional. If any deliverable fails on an optional architecture, it will not abort the whole compose — she's forgiving like that. 💜 And if the list contains only "*", all arches will be substituted.

Live Media Settings 🌸

live_media — (dict) configuration for koji spin-livemedia; format: {variant_uid_regex: [{opt:value}]}

Required options, no skipping these bestie:

name (str)
version (str)
arches ([str]) — what architectures to build the media for; by default uses all arches for the variant.
kickstart (str) — name of the kickstart file

Available options:

ksurl (str)
ksversion (str)
scratch (bool)
target (str)
release (str) — a string with the release, or !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN to automatically generate a suitable value. See automatic versioning for details.
skip_tag (bool)
repo (str|[str]) — repos specified by URL or variant UID
title (str)
install_tree_from (str) — variant to take install tree from
nomacboot (bool)

Image Build Settings 🔥

image_build — (dict) config for koji image-build; format: {variant_uid_regex: [{opt: value}]}

By default, images will be built for each binary arch valid for the variant. The config can specify a list of arches to narrow this down if you're feeling picky. 🧚

NOTE: Config can contain anything what is accepted by koji image-build --config configfile.ini Repo can be specified either as a string or a list of strings. It will be automatically transformed into format suitable for koji. A repo for the currently built variant will be added as well. If you explicitly set release to !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN, it will be replaced with a value generated as described in automatic versioning. If you explicitly set release to !RELEASE_FROM_DATE_RESPIN, it will be replaced with a value generated as described in automatic versioning. If you explicitly set version to !VERSION_FROM_VERSION, it will be replaced with a value generated as described in automatic versioning. Please don't set install_tree. This gets automatically set by pungi based on current variant. You can use install_tree_from key to use install tree from another variant. Like, just don't. 🙈 Both the install tree and repos can use one of following formats: - URL to the location - name of variant in the current compose - absolute path on local filesystem (which will be translated using configured mappings or used unchanged, in which case you have to ensure the koji builders can access it) You can set either a single format, or a list of formats. For available values see help output for koji image-build command. If ksurl ends with #HEAD, Pungi will figure out the SHA1 hash of current HEAD and use that instead. Iconic behaviour honestly. Setting scratch to True will run the koji tasks as scratch builds.

Example

image_build = {
    '^Server$': [
        {
            'image-build': {
                'format': ['docker', 'qcow2']
                'name': 'fedora-qcow-and-docker-base',
                'target': 'koji-target-name',
                'ksversion': 'F23',     # value from pykickstart
                'version': '23',
                # correct SHA1 hash will be put into the URL below automatically
                'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
                'kickstart': "fedora-docker-base.ks",
                'repo': ["http://someextrarepos.org/repo", "ftp://rekcod.oi/repo"],
                'distro': 'Fedora-20',
                'disk_size': 3,

                # this is set automatically by pungi to os_dir for given variant
                # 'install_tree': 'http://somepath',
            },
            'factory-parameters': {
                'docker_cmd':  "[ '/bin/bash' ]",
                'docker_env': "[ 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' ]",
                'docker_labels': "{'Name': 'fedora-docker-base', 'License': u'GPLv2', 'RUN': 'docker run -it --rm ${OPT1} --privileged -v \`pwd\`:/atomicapp -v /run:/run -v /:/host --net=host --name ${NAME} -e NAME=${NAME} -e IMAGE=${IMAGE} ${IMAGE} -v ${OPT2} run ${OPT3} /atomicapp', 'Vendor': 'Fedora Project', 'Version': '23', 'Architecture': 'x86_64' }",
            }
        },
        {
            'image-build': {
                'format': ['docker', 'qcow2']
                'name': 'fedora-qcow-and-docker-base',
                'target': 'koji-target-name',
                'ksversion': 'F23',     # value from pykickstart
                'version': '23',
                # correct SHA1 hash will be put into the URL below automatically
                'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
                'kickstart': "fedora-docker-base.ks",
                'repo': ["http://someextrarepos.org/repo", "ftp://rekcod.oi/repo"],
                'distro': 'Fedora-20',
                'disk_size': 3,

                # this is set automatically by pungi to os_dir for given variant
                # 'install_tree': 'http://somepath',
            }
        },
        {
            'image-build': {
                'format': 'qcow2',
                'name': 'fedora-qcow-base',
                'target': 'koji-target-name',
                'ksversion': 'F23',     # value from pykickstart
                'version': '23',
                'ksurl': 'https://git.fedorahosted.org/git/spin-kickstarts.git?somedirectoryifany#HEAD',
                'kickstart': "fedora-docker-base.ks",
                'distro': 'Fedora-23',

                # only build this type of image on x86_64
                'arches': ['x86_64']

                # Use install tree and repo from Everything variant.
                'install_tree_from': 'Everything',
                'repo': ['Everything'],

                          # Set release automatically.
                          'release': '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN',
                      }
                  }
              ]
          }

🥝 KiwiBuild Settings 🥝

`kiwibuild`

(dict) — bestie, this is the config for building images with kiwi by way of a Koji plugin. Pungi will trigger a Koji task that delegates to kiwi, kiwi builds the image, and then it gets imported into Koji via content generators. slay. 💅

Format: {variant_uid_regex: [{...}]}.

Required keys in the configuration dict — like, actually mandatory, no skipping:

kiwi_profile — (str) select profile from description file.

Description scm, description path and target have to be provided too 🫶 but instead of typing them out for each image separately, you can just use the kiwibuild_* options or global_target. iykyk.

Optional keys — the little extras ✨:

description_scm — (str) scm URL of description kiwi description.
description_path — (str) path to kiwi description inside the scm repo.
repos — additional repos used to install RPMs in the image. The compose repository for the enclosing variant is added automatically. Either variant name or a URL is supported. 🎀
target — (str) which build target to use for the task. If not provided, then either kiwibuild_target or global_target is needed.
release — (str) release of the output image.
arches — ([str]) List of architectures to build for. If not provided, all variant architectures will be built.
failable — ([str]) List of architectures for which this deliverable is not release blocking.
type — (str) override default type from the bundle with this value.
type_attr — ([str]) override default attributes for the build type from description.
bundle_name_format — (str) override default bundle format name.
version — (str) override version. Follows the same rules as described in automatic versioning.
repo_releasever — (str) Override default releasever of the output image.
manifest_type — the image type that is put into the manifest by pungi. If not supplied, an autodetected value will be provided. It may or may not make sense. 💀
use_buildroot_repo = False — (bool) whether the task should automatically enable buildroot repository corresponding to the used target.

The options can be set either for the specific image, or at the phase level (see below 👀). Version also falls back to global_version.

`kiwibuild_description_scm`

(str) — URL for scm containing the description files

`kiwibuild_description_path`

(str) — path to a description file within the description scm

`kiwibuild_type`

(str) — override default type from the bundle with this value.

`kiwibuild_type_attr`

([str]) — override default attributes for the build type from description.

`kiwibuild_bundle_name_format`

(str) — override default bundle format name.

`kiwibuild_version`

(str) — overide version for all kiwibuild tasks.

not me leaving the typo in there. it's overide in the original, and we respect the original, fr fr. 💀

`kiwibuild_repo_releasever`

(str) — override releasever for all kiwibuild tasks.

`kiwibuild_use_buildroot_repo`

(bool) — set enablement of a buildroot repo for all kiwibuild tasks.

🚀 Trigger Konflux Pipeline 🚀

The config for Konflux pipeline execution lives in the trigger_konflux_pipeline setting, which takes a list where each item specifies a regex for a variant and nests the different pipeline-runs for it. it's giving nesting dolls 🪆

The tool processes the entire trigger_konflux_pipeline config, initiating all defined pipeline executions in series and then monitoring them. main character energy, honestly. 👑

There are two main pieces of config per pipeline-run:

Template Specification — this block defines the exact blueprint for the pipeline, specifying the pipeline name, the necessary bundle reference, and any base parameters required for the pipeline's operation.
Konflux Instance Configuration — this identifies the specific Konflux instance where the pipeline will be executed, along with any additional instance-level settings for authentication. 🔐

The tool waits for the completion of the concurrently triggered pipelines. It continuously monitors their status, and progress. Final outcome of each pipeline is tracked and reported in the logs. 📋

NOTE: 🫶 This phase was initially designed with the creation of bootc images in mind. As a result, the number of parameters available for injection into the PipelineRun manifest is tightly coupled to this specific use case, which may cause problems when running other pipeline types. so like... heads up bestie. 🙈

🎀 Options 🎀

trigger_konflux_pipeline — (dict) Dictionary compose of {variant_uid_regex: [{...}]}
- variant_uid_regex — (list) An list of dictionaries, where each item specifies one pipeline to be run.
  - config_name — (str) Logical name for this pipeline configuration. Used to generate the PipelineRun resource name ({config_name}-{variant}-{compose_id}) and to organize artifacts under <config_name>/<variant>/ subdirectories in the work and metadata directories. Must be unique across all pipeline configurations. ⭐
  - pipelinerun_template — (scm_dict) Configuration specifying the source and location of the PipelineRun template.
  - konflux — (str) Configuration specifying the target Konflux instance and environment for the PipelineRun.
    - kubeconfig_file — (str) The Kubernetes/OpenShift configuration file for authentication.
    - kube_context — (str) A Kubernetes/OpenShift context from the kubeconfig file.

🛠️ Prerequisites and Setup 🛠️

This phase requires the OpenShift CLI (oc) command to be available in the execution environment. The oc command is used to create and monitor the PipelineRun manifests on the target Konflux instance. no oc, no vibes. 😭

For this task, the phase consumes the following critical parameters:

kubeconfig_file
kube_context

The necessary Kube Config file can be easily generated using the utility script found in the Pungi repository:

contrib/oc-config4konflux/gen_new_kubeconf.sh

🌊 Execution Flow 🌊

For each variant and pipeline configuration, the phase sources the PipelineRun template, injects parameters, and renders the manifest. ✨
Rendered manifests are grouped by Kubernetes context and submitted to the cluster using oc create. If the cluster assigns a different resource name than the one generated by Pungi, the actual name is used for all subsequent monitoring. 👀
A monitoring thread is spawned per PipelineRun. Each thread polls the PipelineRun status every 30 seconds. If the resource is not found during the initial checks, the thread retries up to 5 times before giving up. Once the resource is available, polling continues until the PipelineRun reaches a terminal state (Succeeded or Failed). 🦋
On success, the phase queries the associated Snapshot resource (linked via the appstudio.openshift.io/build-pipelinerun label) and saves the full YAML to the compose metadata directory, making it available for downstream promotion pipelines (e.g., to create a Release CR). 🎀

📁 Artifact Directory Structure 📁

Artifacts are organized under structured subdirectories in the work directory and in the compose metadata:

Work directory (templates and rendered manifests):

work/global/konflux/
└── <variant>/
    └── <config_name>/
        ├── <template_file>                        # sourced template
        └── rendered-<pipelinerun_name>.yaml        # rendered manifest

Compose metadata (Snapshots):

compose/metadata/konflux.json

The konflux.json file contains Snapshot data for all successful PipelineRuns, keyed by variant and config name:

{
    "<variant>": {
        "<config_name>": {
            "pipelinerun_name": "<actual-name>",
            "snapshot": { ... full Snapshot resource ... }
        }
    }
}

🧚 Template Variables 🧚

The PipelineRun template is a Jinja2 template. The following variables are automatically injected and available for use — like little gifts, no cap 🎁:

compose_id — (str) The unique identifier for the compose (e.g., Fedora-40-20250204.0).
pipelinerun_name — (str) The generated PipelineRun resource name, built from the config name, variant, and compose ID in the format {config_name}-{variant}-{compose_id}.
compose_url — (str) The translated URL path to the compose top directory, or the local file system path when translation is not configured (e.g., https://example.com/composes/Fedora-40-20250204.0 when translated, or /mnt/koji/compose/Fedora-40-20250204.0 when not). 🌈
release_version — (str) The full release version string from the release_version configuration setting (e.g., 8.4, 40).
version_major — (str) The major version component extracted from release_version (e.g., 8 from 8.4).
version_minor — (str) The minor version component extracted from release_version (e.g., 4 from 8.4). Empty string if the version has no minor component.

Additionally, all configuration fields from the trigger_konflux_pipeline item (such as config_name, konflux, etc.) are also available in the template context. so you literally have everything you need in there, bestie. 💜

✨ Example Config ✨

ok so when trigger_konflux_pipeline is all dressed up and ready for the group chat, it looks like this 💖:

trigger_konflux_pipeline = {
        "variant_uid_regex": [
                {
                        "config_name": "my-pipeline",
                        "pipelinerun_template": {
                                "scm": "git",
                                "repo": "https://repo/containing/template/file",
                                "branch": "main",
                                "file": "path/to/template/inside/repo",
                        },
                        "konflux": {
                                "kubeconfig_file": "path/to/kube/config",
                                "kube_context": "some-kube-context",
                        },
                },
        ],
}

✨ Example Template ✨

and here's a whole example Jinja2 template for a PipelineRun resource — it's giving very much infrastructure bestie energy 🎀:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: {{ pipelinerun_name }}
  labels:
    app.kubernetes.io/name: compose-pipeline
spec:
  pipelineRef:
    name: compose-pipeline
  params:
    - name: compose_id
      value: "{{ compose_id }}"
    - name: compose_url
      value: "{{ compose_url }}"
    - name: release_version
      value: "{{ release_version }}"
    - name: version_major
      value: "{{ version_major }}"
    {% if version_minor %}
    - name: version_minor
      value: "{{ version_minor }}"
    {% endif %}

🌈 OSBuild Composer for building images 🌈

osbuild — (dict) — the config for building images in the OSBuild Composer service that's fronted by a Koji plugin. Pungi will trigger a Koji task that delegates to the OSBuild Composer, and that Composer builds the image and imports it into Koji via content generators. no cap 🫶

Format: {variant_uid_regex: [{...}]}.

Required keys in the configuration dict (you literally can't skip these, bestie):

name – name of the Koji package
distro – image for which distribution should be build TODO examples
image_types – a list with a single image type string, or just a string, representing the image type to build (e.g. qcow2). Either way, only a single image type can be provided as an argument. one (1) image type. that's the vibe 💅

Optional keys:

target – which build target to use for the task. Either this option or the global osbuild_target is required.
version – version for the final build (as a string). This option is required if the global osbuild_version is not specified.
release – release part of the final NVR. If neither this option nor the global osbuild_release is set, Koji will automatically generate a value for you ✨
repo – a list of repositories from which to consume packages for building the image. By default only the variant repository is used. The list items may use one of the following formats:
- String with just the repository URL.
- Dictionary with the following keys:
  - baseurl – URL of the repository.
  - package_sets – a list of package set names to use for this repository. Package sets are an internal concept of Image Builder and are used in image definitions. If specified, the repository is used by Image Builder only for the pipeline with the same name. For example, specifying the build package set name will make the repository to be used only for the build environment in which the image will be built. (optional)
arches – list of architectures for which to build the image. By default, the variant arches are used. This option can only restrict it, not add a new one. so it can shrink the guest list but never expand it, iykyk 🙈
manifest_type – the image type that is put into the manifest by pungi. If not supplied then it is autodetected from the Koji output.
ostree_url – URL of the repository that's used to fetch the parent commit from.
ostree_ref – name of the ostree branch
ostree_parent – commit hash or a branch-like reference to the parent commit.
customizations – a dictionary with customizations to use for the image build. For the list of supported customizations, see the hosted variants in the Image Builder documentation https://osbuild.org/docs/user-guide/blueprint-reference#installation-device.
upload_options – a dictionary with upload options specific to the target cloud environment. If provided, the image will be uploaded to the cloud environment, in addition to the Koji server. One can't combine arbitrary image types with arbitrary upload options. The dictionary keys differ based on the target cloud environment. The following keys are supported:
- AWS EC2 upload options – upload to Amazon Web Services.
  - region – AWS region to upload the image to
  - share_with_accounts – list of AWS account IDs to share the image with
  - snapshot_name – Snapshot name of the uploaded EC2 image (optional)
- AWS S3 upload options – upload to Amazon Web Services S3.
  - region – AWS region to upload the image to
- Azure upload options – upload to Microsoft Azure.
  - tenant_id – Azure tenant ID to upload the image to
  - subscription_id – Azure subscription ID to upload the image to
  - resource_group – Azure resource group to upload the image to
  - location – Azure location of the resource group (optional)
  - image_name – Image name of the uploaded Azure image (optional)
- GCP upload options – upload to Google Cloud Platform.
  - region – GCP region to upload the image to
  - bucket – GCP bucket to upload the image to (optional)
  - share_with_accounts – list of GCP accounts to share the image with
  - image_name – Image name of the uploaded GCP image (optional)
- Container upload options – upload to a container registry.
  - name – name of the container image (optional)
  - tag – container tag to upload the image to (optional)

NOTE: There is initial support for having this task as failable without aborting the whole compose. This can be enabled by setting "failable": ["*"] in the config for the image. It is an on/off switch without granularity per arch. so it's all-or-nothing, no per-arch slay 😭

🌸 Image Builder Settings 🌸

imagebuilder — (dict) — the config for building images with the koji-image-builder Koji plugin. Pungi will trigger a Koji task which will build the image with the given configuration using the image-builder executable in the build root. iconic 💜

Format: {variant_uid_regex: [{...}]}.

Required keys in the configuration dict:

name – name of the Koji package
types – a list with a single image type string representing the image type to build (e.g. qcow2). Only a single image type can be provided as an argument. one type, that's the rule 🎀

Optional keys:

target – which build target to use for the task. Either this option, the global imagebuilder_target, or global global_target is required.
version – version for the final build (as a string). This option is required if the global imagebuilder_version or its global_version equivalent are not specified.
release – release part of the final NVR. If neither this option nor the global imagebuilder_release nor its global_release equivalent are set, Koji will automatically generate a value.
repos – a list of repositories from which to consume packages for building the image. By default only the variant repository is used. The list items use the following formats:
- String with just the repository URL.
- Variant ID in the current compose.
arches – list of architectures for which to build the image. By default, the variant arches are used. This option can only restrict it, not add a new one.
seed – An integer that can be used to make builds more reproducible. When image-builder builds images various bits and bobs are generated with a PRNG (partition uuids, etc). Pinning the seed with this argument or imagebuilder_seed to do so globally will make builds use the same random values each time. Note that using seed requires the Koji side to have at least koji-image-builder >= 7 deployed. reproducibility era ✨
scratch – A boolean to instruct koji-image-builder to perform scratch builds. This might have implications on garbage collection within the koji instance you're targeting. Can also be set globally through imagebuilder_scratch.
ostree – A dictionary describing where to get ostree content when applicable. The dictionary contains the following keys:
- url – URL of the repository that's used to fetch the parent commit from.
- ref – Name of an ostree branch or tag
blueprint – A dictionary with a blueprint to use for the image build. Blueprints can customize images beyond their initial definition. For the list of supported customizations, see external Documentation

NOTE: There is initial support for having this task as failable without aborting the whole compose. This can be enabled by setting "failable": ["*"] in the config for the image. It is an on/off switch without granularity per arch. same energy as above, bestie 💀

✨ Example Config ✨

and the moment you've been waiting for, the actual config snippet, looking cute 🍓:

imagebuilder_target = 'f43-image-builder'
imagebuilder_seed = 43
imagebuilder_scratch = True

          imagebuilder = {
              "^IoT$": [
                {
                  "name": "%s-raw" % release_name,
                  "types": ["iot-raw-xz"],
                  "arches": ["x86_64"], #, "aarch64"],
                  "repos": ["https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/compose/Everything/$arch/os/"],
                  "ostree": {
                     "url": "https://kojipkgs.fedoraproject.org/compose/iot/repo/",
                     "ref": "fedora/rawhide/$arch/iot",
                  },
                  "subvariant": "IoT",
                  "failable": ["*"],
                },
              ]
          }

🦋 OSTree Settings 🦋

The ostree phase of Pungi can create and update ostree repos, bestie. It does this by running rpm-ostree compose inside a Koji runroot environment. Heads up though: the ostree repo itself is not part of the compose — it lives in its own separate directory. Any shiny new packages that show up in the compose get added to the repo with a brand new commit. ✨

ostree — (dict) a mapping of configuration for each variant. The format should be {variant_uid_regex: config_dict}. And yes, you can pass a list of config dicts too, flexible queen. 💅

The config dict for each variant/arch pair must have these keys:

treefile — (str) filename of configuration for rpm-ostree.
config_url — (str) URL for the Git repository with the treefile.
repo — (str | dict | [str | dict]) repos specified by URL, or a dict of repo options. baseurl is required in the dict.
ostree_repo — (str) where to put the ostree repository.

These keys are optional:

keep_original_sources — (bool) keep the existing source repos in the tree config file. If it's not enabled, all the original source repos get yeeted from the tree config file. 🫶
config_branch — (str) Git branch of the repo to use. Defaults to master.
arches — ([str]) list of architectures to update ostree for. There will be one task per architecture. By default, all architectures in the variant are used.
failable — ([str]) list of architectures for which this deliverable is not release blocking.
update_summary — (bool) update summary metadata after tree composing. Defaults to False.
force_new_commit — (bool) do not use rpm-ostree's built-in change detection. Defaults to False.
unified_core — (bool) use rpm-ostree in unified core mode for composes. Defaults to False.
version — (str) version string to be added as versioning metadata. If this option is set to !OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN, a value will be generated automatically as $VERSION.$RELEASE. If this option is set to !VERSION_FROM_VERSION_DATE_RESPIN, a value will be generated automatically as $VERSION.$DATE.$RESPIN. See how those values are created.
tag_ref — (bool, default True) if set to False, a git reference will not be created.
ostree_ref — (str) to override the value ref from treefile.
runroot_packages — (list) a list of additional package names to be installed in the runroot environment in Koji.

🌸 Example config 🌸

          ostree = {
              "^Atomic$": {
                  "treefile": "fedora-atomic-docker-host.json",
                  "config_url": "https://git.fedorahosted.org/git/fedora-atomic.git",
                  "keep_original_sources": True,
                  "repo": [
                      "http://example.com/repo/x86_64/os",
                      {"baseurl": "http://example.com/linux/repo", "exclude": "systemd-container"},
                  ],
                  "ostree_repo": "/mnt/koji/compose/atomic/Rawhide/",
                  "update_summary": True,
                  # Automatically generate a reasonable version
                  "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
                  # Only run this for x86_64 even if Atomic has more arches
                  "arches": ["x86_64"],
              }
          }

ostree_use_koji_plugin = False — (bool) when set to True, the Koji pungi_ostree task will be used to execute rpm-ostree instead of runroot. ⚠️ Use this only if the Koji instance actually has the pungi_ostree plugin installed, no cap.

🎀 OSTree Native Container Settings 🎀

The ostree_container phase of Pungi can create an ostree native container image as an OCI archive. It does this by running rpm-ostree compose image in a Koji runroot environment.

So here's the tea, bestie: rpm-ostree could use info from previously built images to make the container layer split better, but we can't use that functionality until https://github.com/containers/skopeo/pull/2114 gets resolved. 😭 So every single invocation just creates a brand new OCI archive image from scratch. Fr fr.

ostree_container — (dict) a mapping of configuration for each variant. The format should be {variant_uid_regex: config_dict}. A list of config dicts works too, iykyk.

The config dict for each variant/arch pair must have these keys:

treefile — (str) filename of configuration for rpm-ostree.
config_url — (str) URL for the Git repository with the treefile.

These keys are optional:

repo — (str | dict | [str | dict]) repos specified by URL, or a dict of repo options. baseurl is required in the dict.
keep_original_sources — (bool) keep the existing source repos in the tree config file. If it's not enabled, all the original source repos get removed from the tree config file.
config_branch — (str) Git branch of the repo to use. Defaults to main. (yes, main, not master — it's different from the ostree one, don't @ me)
arches — ([str]) list of architectures to generate ostree native container images for. There will be one task per architecture. By default, all architectures in the variant are used.
failable — ([str]) list of architectures for which this deliverable is not release blocking.
version — (str) version string to be added to the OCI archive name. If this option is set to !OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN, a value will be generated automatically as $VERSION.$RELEASE. If this option is set to !VERSION_FROM_VERSION_DATE_RESPIN, a value will be generated automatically as $VERSION.$DATE.$RESPIN. See how those values are created.
tag_ref — (bool, default True) if set to False, a git reference will not be created.
runroot_packages — (list) a list of additional package names to be installed in the runroot environment in Koji.
subvariant — (str) the subvariant value to be used in the metadata for the image. Also used in the image's filename, unless it's overridden by name. Defaults to being the same as the variant. ⚠️ If you're building more than one ostree container in a variant, each one must have a unique subvariant.
name — (str) the base for the image's filename. To produce the complete filename, the image's architecture, the version string, and the format suffix get appended to this. Defaults to the value of release_short and the subvariant, joined by a dash.

💖 Example config 💖

          ostree_container = {
              "^Sagano$": {
                  "treefile": "fedora-tier-0-38.yaml",
                  "config_url": "https://gitlab.com/CentOS/cloud/sagano.git",
                  "config_branch": "main",
                  "repo": [
                      "http://example.com/repo/x86_64/os",
                      {"baseurl": "http://example.com/linux/repo", "exclude": "systemd-container"},
                  ],
                  # Automatically generate a reasonable version
                  "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
                  # Only run this for x86_64 even if Sagano has more arches
                  "arches": ["x86_64"],
              }
          }

ostree_container_use_koji_plugin = False — (bool) when set to True, the Koji pungi_ostree task will be used to execute rpm-ostree instead of runroot. ⚠️ Use it only if the Koji instance has the pungi_ostree plugin installed.

🧚 Ostree Installer Settings 🧚

The ostree_installer phase of Pungi can produce an installer image that bundles an OSTree repository. This one always runs in Koji as a runroot task. Periodt.

ostree_installer — (dict) a variant/arch mapping of configuration. The format should be [(variant_uid_regex, {arch|*: config_dict})].

The config dict for each variant/arch pair must have this key:

These keys are optional:

repo — (str | [str]) repos specified by URL or variant UID.
release — (str) release value to set for the installer image. Set it to !RELEASE_FROM_LABEL_DATE_TYPE_RESPIN to generate the value automatically.
failable — ([str]) list of architectures for which this deliverable is not release blocking.

These optional keys get passed to lorax to customize the build. 👑

installpkgs — ([str])
add_template — ([str])
add_arch_template — ([str])
add_template_var — ([str])
add_arch_template_var — ([str])
rootfs_size — ([str])
template_repo — (str) Git repository with extra templates.
template_branch — (str) branch to use from template_repo.

So the templates can either be absolute paths — in which case they'll be used exactly as configured — or they can be relative paths, in which case template_repo needs to point to a Git repository to take the templates from. Slay. ✨

If the templates need to run with additional dependencies, that's configurable with the optional key:

extra_runroot_pkgs — ([str])
skip_branding — (bool) stops lorax from installing packages with branding. Defaults to False.
ostree_installer_overwrite = False — (bool) by default, if a variant that includes an OSTree installer also creates regular installer images in the buildinstall phase, there will be conflicts (because the files get put in the same place) and Pungi will report an error and fail the compose. 💀

With this option you can opt in to the overwriting. The traditional boot.iso will be in the iso/ subdirectory.

ostree_installer_use_koji_plugin = False — (bool) – set this to True and the Koji pungi_buildinstall task runs Lorax instead of runroot. Only use it if your Koji instance actually has the pungi_buildinstall plugin installed, bestie 🫶

💫 Example config

ostree_installer = [
    ("^Atomic$", {
        "x86_64": {
            "repo": [
                "Everything",
                "https://example.com/extra-repo1.repo",
                "https://example.com/extra-repo2.repo",
            ],
            "release": "!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN",
            "installpkgs": ["fedora-productimg-atomic"],
            "add_template": ["atomic-installer/lorax-configure-repo.tmpl"],
            "add_template_var": [
                "ostree_osname=fedora-atomic",
                "ostree_ref=fedora-atomic/Rawhide/x86_64/docker-host",
            ],
            "add_arch_template": ["atomic-installer/lorax-embed-repo.tmpl"],
            "add_arch_template_var": [
                "ostree_repo=https://kojipkgs.fedoraproject.org/compose/atomic/Rawhide/",
                "ostree_osname=fedora-atomic",
                "ostree_ref=fedora-atomic/Rawhide/x86_64/docker-host",
            ]
            'template_repo': 'https://git.fedorahosted.org/git/spin-kickstarts.git',
            'template_branch': 'f24',
        }
    })
]

🎀 Extra ISOs

Okay so picture this: you want one ISO image that pulls packages from multiple variants. Iconic, right? That ISO still belongs to exactly one variant, and it gets stored in that variant's ISO directory. No cap.

The ISO will be bootable if the buildinstall or boot_iso phase runs for the parent variant — and it reuses that variant's boot configuration. ✨

extra_isos — (dict) – a mapping from variant UID regex to a list of configuration blocks. 💅
- include_variants — (list) list of variant UIDs whose content should get added to the ISO; the variant of this image is added automatically.

All the other config keys are optional, bestie.

filename — (str) template for naming the image. On top of the regular placeholders, filename is available with the name generated using the image_name_format option.
volid — (str) template for generating the volume ID. Again the volid placeholder can be used just like for the file name. This can also be a list of templates that get tried one after another until one generates a volume ID that fits inside the 32 character limit.
extra_files — (list) a list of scm_dict objects. These files get dropped into the top level directory of the image.
arches — (list) a list of architectures to build this image for. By default all arches from the variant get used. This option lets you limit them.
failable_arches — (list) a list of architectures where the image can fail to be generated without failing the entire compose. 😮‍💨
skip_src — (bool) lets you disable creating an image with source packages.
inherit_extra_files — (bool) by default extra files in variants are ignored. If you want them in the ISO, set this to True.
max_size (DEPRECATED) — (int) This option has been removed. Size checks are no longer performed. Periodt.

✨ Example config

extra_isos = {
    'Server': [{
        # Will generate foo-DP-1.0-20180510.t.43-Server-x86_64-dvd1.iso
        'filename': 'foo-{filename}',
        'volid': 'foo-{arch}',

        'extra_files': [{
            'scm': 'git',
            'repo': 'https://pagure.io/pungi.git',
            'file': 'setup.py'
        }],

        'include_variants': ['Client']
    }]
}
# This should create image with the following layout:
#  .
#  ├── Client
#  │   ├── Packages
#  │   │   ├── a
#  │   │   └── b
#  │   └── repodata
#  ├── Server
#  │   ├── Packages
#  │   │   ├── a
#  │   │   └── b
#  │   └── repodata
#  └── setup.py

🌸 Media Checksums Settings

media_checksums — (list) – list of checksum types to compute. Allowed values are anything supported by Python's hashlib module (see the docs for the deets). 🔥
media_checksum_one_file — (bool) – when True, only one CHECKSUM file gets created per directory; this option requires media_checksums to specify only one type.
media_checksum_base_filename — (str) – when this isn't set, all checksums get saved to a file named either CHECKSUM or based on the digest type; this option lets you add any prefix to that name.

You can use format strings that get replaced by actual values. The allowed keys are:

arch
compose_id
date
label
label_major_version
release_short
respin
type
type_suffix
version
dirname (only if media_checksum_one_file is enabled)

For example, for Fedora the prefix should be %(release_short)s-%(variant)s-%(version)s-%(date)s%(type_suffix)s.%(respin)s. 💖

🦋 Translate Paths Settings

translate_paths — (list) – list of paths to translate; format: [(path, translated_path)]

NOTE: This feature gets useful when you need to transform a compose location into, say, an HTTP repo that can be passed to koji image-build. The path part is normalized via os.path.normpath(). 🧚

✨ Example config

translate_paths = [
    ("/mnt/a", "http://b/dir"),
]

💜 Example usage

>>> from pungi.util import translate_paths
>>> print translate_paths(compose_object_with_mapping, "/mnt/a/c/somefile")
http://b/dir/c/somefile

🌈 Miscellaneous Settings

paths_module — (str) – Name of a Python module implementing the same interface as pungi.paths. This module can be used to override where things are placed. 👑
link_type = hardlink-or-copy — (str) – Method of putting packages into the compose directory.

Available options:

hardlink-or-copy
hardlink
copy
symlink
abspath-symlink

skip_phases — (list) – List of phase names that should be skipped. Same functionality is also available via a command line option.
release_discinfo_description — (str) – Override the description in .discinfo files. The value is a format string accepting %(variant_name)s and %(arch)s placeholders.
symlink_isos_to — (str) – If set, the ISO files from the buildinstall and createiso phases get put into this destination, and a symlink pointing to this location gets created in the actual compose directory. 🎀
dogpile_cache_backend — (str) – If set, Pungi uses the configured Dogpile cache backend to cache various data between multiple Pungi calls. This can make Pungi faster if you're running more similar composes regularly in a short time. ⚡

For the list of available backends, please see the https://dogpilecache.readthedocs.io documentation.

Most typical configuration uses the dogpile.cache.dbm backend.

dogpile_cache_arguments — (dict) – Arguments to be used when creating the Dogpile cache backend. See that particular backend's configuration for the list of possible key/value pairs.

For the dogpile.cache.dbm backend, the value can be for example the following:

{
    "filename": "/tmp/pungi_cache_file.dbm"
}

dogpile_cache_expiration_time — (int) – Defines the default expiration time in seconds of data stored in the Dogpile cache. Defaults to 3600 seconds. 💫

🌟 BIG PICTURE EXAMPLES

Real Pungi config files get very large, bestie. This page brings two examples of (almost) full configuration for two different composes. Let's gooo. 🫶

⭐ Fedora Rawhide compose

This is a shortened configuration for Fedora Radhide compose as of 2019-10-14.

release_name = 'Fedora'
release_short = 'Fedora'
release_version = 'Rawhide'
release_is_layered = False

bootable = True
comps_file = {
    'scm': 'git',
    'repo': 'https://pagure.io/fedora-comps.git',
    'branch': 'master',
    'file': 'comps-rawhide.xml',
    # Merge translations by running make. This command will generate the file.
    'command': 'make comps-rawhide.xml'
}
module_defaults_dir = {
    'scm': 'git',
    'repo': 'https://pagure.io/releng/fedora-module-defaults.git',
    'branch': 'main',
    'dir': '.'
}
# Optional module obsoletes configuration which is merged
# into the module index and gets resolved
module_obsoletes_dir = {
    'scm': 'git',
    'repo': 'https://pagure.io/releng/fedora-module-defaults.git',
    'branch': 'main',
    'dir': 'obsoletes'
}

variants_file='variants-fedora.xml'
sigkeys = ['12C944D0']

# Put packages into subdirectories hashed by their initial letter.
hashed_directories = True

# There is a special profile for use with compose. It makes Pungi
# authenticate automatically as rel-eng user.
koji_profile = 'compose_koji'

# RUNROOT settings
runroot = True
runroot_channel = 'compose'
runroot_tag = 'f32-build'

# PKGSET
pkgset_source = 'koji'
pkgset_koji_tag = 'f32'
pkgset_koji_inherit = False

filter_system_release_packages = False

# GATHER
gather_method = {
    '^.*': {                # For all variants
        'comps': 'deps',    # resolve dependencies for packages from comps file
        'module': 'nodeps', # but not for packages from modules
    }
}
gather_backend = 'dnf'
gather_profiler = True
check_deps = False
greedy_method = 'build'

repoclosure_backend = 'dnf'

# CREATEREPO
createrepo_deltas = False
createrepo_database = True
createrepo_use_xz = True
createrepo_extra_args = ['--zck', '--zck-dict-dir=/usr/share/fedora-repo-zdicts/rawhide']

...and then it just keeps going. Bestie, this config is a whole personality and we're still in it. 💅 Let's keep reading the example, chunk by chunk, because the comments in here are basically the manpage explaining itself.

          # CHECKSUMS
          media_checksums = ['sha256']
          media_checksum_one_file = True
          media_checksum_base_filename = '%(release_short)s-%(variant)s-%(version)s-%(arch)s-%(date)s%(type_suffix)s.%(respin)s'

          # CREATEISO
          iso_hfs_ppc64le_compatible = False

          # BUILDINSTALL
          buildinstall_method = 'lorax'
          buildinstall_skip = [
              # No installer for Modular variant
              ('^Modular$', {'*': True}),
              # No 32 bit installer for Everything.
              ('^Everything$', {'i386': True}),
          ]

Ok so media_checksums says ['sha256'], media_checksum_one_file is True, and the base filename template is '%(release_short)s-%(variant)s-%(version)s-%(arch)s-%(date)s%(type_suffix)s.%(respin)s'. Then under the # CREATEISO comment, iso_hfs_ppc64le_compatible is False.

For # BUILDINSTALL: buildinstall_method is 'lorax', and buildinstall_skip skips things by pattern — no installer for the ^Modular$ variant (everything, {'*': True}) and no 32-bit installer for ^Everything$ ({'i386': True}). Slay.

Next up, the lorax corner of the config:

          # Enables macboot on x86_64 for all variants and disables upgrade image building
          # everywhere.
          lorax_options = [
            ('^.*$', {
               'x86_64': {
                   'nomacboot': False
               },
               'ppc64le': {
                   # Use 3GB image size for ppc64le.
                   'rootfs_size': 3
               },
               '*': {
                   'noupgrade': True
               }
            })
          ]

So lorax_options here matches every variant ('^.*$') and, per the comment, enables macboot on x86_64 for all variants and disables upgrade image building everywhere. For x86_64 it sets 'nomacboot': False, for ppc64le it sets 'rootfs_size': 3 (use 3GB image size for ppc64le), and for '*' it sets 'noupgrade': True.

And the package-tweaking block, which is so much:

          additional_packages = [
              ('^(Server|Everything)$', {
                  '*': [
                      # Add all architectures of dracut package.
                      'dracut.*',
                      # All all packages matching this pattern
                      'autocorr-*',
                  ],
              }),

              ('^Everything$', {
                  # Everything should include all packages from the tag. This only
                  # applies to the native arch. Multilib will still be pulled in
                  # according to multilib rules.
                  '*': ['*'],
              }),
          ]

additional_packages: for ^(Server|Everything)$, on '*' arches it adds 'dracut.*' (add all architectures of dracut package) and 'autocorr-*' (all all packages matching this pattern — that's the original's wording, we keep it fr fr). For ^Everything$, '*': ['*'] means Everything should include all packages from the tag, and per the comment this only applies to the native arch. Multilib will still be pulled in according to multilib rules.

Then the filtering, multilib rules, and blacklist/whitelist:

          filter_packages = [
              ("^.*$", {"*": ["glibc32", "libgcc32"]}),
              ('(Server)$', {
                  '*': [
                      'kernel*debug*',
                      'kernel-kdump*',
                  ]
              }),
          ]

          multilib = [
              ('^Everything$', {
                  'x86_64': ['devel', 'runtime'],
              })
          ]

          # These packages should never be multilib on any arch.
          multilib_blacklist = {
              '*': [
                  'kernel', 'kernel-PAE*', 'kernel*debug*', 'java-*', 'php*', 'mod_*', 'ghc-*'
              ],
          }

          # These should be multilib even if they don't match the rules defined above.
          multilib_whitelist = {
              '*': ['wine', '*-static'],
          }

filter_packages filters "glibc32" and "libgcc32" for every variant ("^.*$" on "*"), and for '(Server)$' filters out 'kernel*debug*' and 'kernel-kdump*'. multilib gives ^Everything$ the 'x86_64' groups ['devel', 'runtime'].

multilib_blacklist is packages that should never be multilib on any arch: 'kernel', 'kernel-PAE*', 'kernel*debug*', 'java-*', 'php*', 'mod_*', 'ghc-*' for '*'. multilib_whitelist is the ones that should be multilib even if they don't match the rules defined above: 'wine' and '*-static' for '*'. Obsessed with how tidy that is. ✨

          createiso_skip = [
              # Keep binary ISOs for Server, but not source ones.
              ('^Server$', {'src': True}),

              # Remove all other ISOs.
              ('^Everything$', {'*': True, 'src': True}),
              ('^Modular$', {'*': True, 'src': True}),
          ]

createiso_skip: keep binary ISOs for Server, but not source ones (('^Server$', {'src': True})), then remove all other ISOs — ('^Everything$', {'*': True, 'src': True}) and ('^Modular$', {'*': True, 'src': True}).

Now the naming block, where the comment spills the whole tea about volume IDs:

          # Image name respecting Fedora's image naming policy
          image_name_format = '%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s-%(date)s%(type_suffix)s.%(respin)s.iso'
          # Use the same format for volume id
          image_volid_formats = [
              '%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s'
          ]
          # Used by Pungi to replace 'Cloud' with 'C' (etc.) in ISO volume IDs.
          # There is a hard 32-character limit on ISO volume IDs, so we use
          # these to try and produce short enough but legible IDs. Note this is
          # duplicated in Koji for live images, as livemedia-creator does not
          # allow Pungi to tell it what volume ID to use. Note:
          # https://fedoraproject.org/wiki/User:Adamwill/Draft_fedora_image_naming_policy
          volume_id_substitutions = {
                           'Beta': 'B',
                        'Rawhide': 'rawh',
                     'Silverblue': 'SB',
                       'Cinnamon': 'Cinn',
                          'Cloud': 'C',
                   'Design_suite': 'Dsgn',
                 'Electronic_Lab': 'Elec',
                     'Everything': 'E',
                 'Scientific_KDE': 'SciK',
                       'Security': 'Sec',
                         'Server': 'S',
                    'Workstation': 'WS',
          }

          disc_types = {
              'boot': 'netinst',
              'live': 'Live',
          }

          translate_paths = [
             ('/mnt/koji/compose/', 'https://kojipkgs.fedoraproject.org/compose/'),
          ]

image_name_format is the image name respecting Fedora's image naming policy: '%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s-%(date)s%(type_suffix)s.%(respin)s.iso'. image_volid_formats uses the same format for volume id: ['%(release_short)s-%(variant)s-%(disc_type)s-%(arch)s-%(version)s'].

volume_id_substitutions is used by Pungi to replace 'Cloud' with 'C' (etc.) in ISO volume IDs. Per the comment: there is a hard 32-character limit on ISO volume IDs, so we use these to try and produce short enough but legible IDs. Note this is duplicated in Koji for live images, as livemedia-creator does not allow Pungi to tell it what volume ID to use. And there's a link note: https://fedoraproject.org/wiki/User:Adamwill/Draft_fedora_image_naming_policy. The substitutions are: 'Beta': 'B', 'Rawhide': 'rawh', 'Silverblue': 'SB', 'Cinnamon': 'Cinn', 'Cloud': 'C', 'Design_suite': 'Dsgn', 'Electronic_Lab': 'Elec', 'Everything': 'E', 'Scientific_KDE': 'SciK', 'Security': 'Sec', 'Server': 'S', 'Workstation': 'WS'.

disc_types maps 'boot': 'netinst' and 'live': 'Live'. translate_paths maps ('/mnt/koji/compose/', 'https://kojipkgs.fedoraproject.org/compose/'). Iconic.

The globals that get inherited everywhere (iykyk):

          # These will be inherited by live_media, live_images and image_build
          global_ksurl = 'git+https://pagure.io/fedora-kickstarts.git?#HEAD'
          global_release = '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN'
          global_version = 'Rawhide'
          # live_images ignores this in favor of live_target
          global_target = 'f32'

These will be inherited by live_media, live_images and image_build: global_ksurl is 'git+https://pagure.io/fedora-kickstarts.git?#HEAD', global_release is '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN', global_version is 'Rawhide', and global_target is 'f32' — with the note that live_images ignores this in favor of live_target.

Container image building time:

          image_build = {
              '^Container$': [
                  {
                      'image-build': {
                              'format': [('docker', 'tar.xz')],
                              'name': 'Fedora-Container-Base',
                              'kickstart': 'fedora-container-base.ks',
                              'distro': 'Fedora-22',
                              'disk_size': 5,
                              'arches': ['armhfp', 'aarch64', 'ppc64le', 's390x', 'x86_64'],
                              'repo': 'Everything',
                              'install_tree_from': 'Everything',
                              'subvariant': 'Container_Base',
                              'failable': ['*'],
                              },
                      'factory-parameters': {
                          'dockerversion': "1.10.1",
                          'docker_cmd':  '[ "/bin/bash" ]',
                          'docker_env': '[ "DISTTAG=f32container", "FGC=f32", "container=oci" ]',
                          'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "32"}',
                      },
                  },
              ],
          }

For '^Container$', image_build has an 'image-build' dict: 'format': [('docker', 'tar.xz')], 'name': 'Fedora-Container-Base', 'kickstart': 'fedora-container-base.ks', 'distro': 'Fedora-22', 'disk_size': 5, 'arches': ['armhfp', 'aarch64', 'ppc64le', 's390x', 'x86_64'], 'repo': 'Everything', 'install_tree_from': 'Everything', 'subvariant': 'Container_Base', 'failable': ['*']. Plus 'factory-parameters': 'dockerversion': "1.10.1", 'docker_cmd': '[ "/bin/bash" ]', 'docker_env': '[ "DISTTAG=f32container", "FGC=f32", "container=oci" ]', 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "32"}'. Ate, no crumbs. 🎀

Live media, where the comments really spill:

          live_media = {
              '^Workstation$': [
                      {
                          'name': 'Fedora-Workstation-Live',
                          'kickstart': 'fedora-live-workstation.ks',
                          # Variants.xml also contains aarch64 and armhfp, but there
                          # should be no live media for those arches.
                          'arches': ['x86_64', 'ppc64le'],
                          'failable': ['ppc64le'],
                          # Take packages and install tree from Everything repo.
                          'repo': 'Everything',
                          'install_tree_from': 'Everything',
                      }
                  ],
              '^Spins': [
                  # There are multiple media for Spins variant. They use subvariant
                  # field so that they can be identified in the metadata.
                  {
                      'name': 'Fedora-KDE-Live',
                      'kickstart': 'fedora-live-kde.ks',
                      'arches': ['x86_64'],
                      'repo': 'Everything',
                      'install_tree_from': 'Everything',
                      'subvariant': 'KDE'

                  },
                  {
                      'name': 'Fedora-Xfce-Live',
                      'kickstart': 'fedora-live-xfce.ks',
                      'arches': ['x86_64'],
                      'failable': ['*'],
                      'repo': 'Everything',
                      'install_tree_from': 'Everything',
                      'subvariant': 'Xfce'
                  },
              ],
          }

For '^Workstation$': 'name': 'Fedora-Workstation-Live', 'kickstart': 'fedora-live-workstation.ks', and per the comment Variants.xml also contains aarch64 and armhfp, but there should be no live media for those arches — so 'arches': ['x86_64', 'ppc64le'], 'failable': ['ppc64le'], and (take packages and install tree from Everything repo) 'repo': 'Everything' and 'install_tree_from': 'Everything'.

For '^Spins' — per the comment there are multiple media for Spins variant. They use subvariant field so that they can be identified in the metadata. — there's 'Fedora-KDE-Live' with 'kickstart': 'fedora-live-kde.ks', 'arches': ['x86_64'], 'repo': 'Everything', 'install_tree_from': 'Everything', 'subvariant': 'KDE'. And 'Fedora-Xfce-Live' with 'kickstart': 'fedora-live-xfce.ks', 'arches': ['x86_64'], 'failable': ['*'], 'repo': 'Everything', 'install_tree_from': 'Everything', 'subvariant': 'Xfce'. The range bestie, the range. 💜

          failable_deliverables = [
              # Installer and ISOs for server failing do not abort the compose.
              ('^Server$', {
                  '*': ['buildinstall', 'iso'],
              }),
              ('^.*$', {
                  # Buildinstall is not blocking
                  'src': ['buildinstall'],
                  # Nothing on i386, ppc64le blocks the compose
                  'i386': ['buildinstall', 'iso'],
                  'ppc64le': ['buildinstall', 'iso'],
                  's390x': ['buildinstall', 'iso'],
              })
          ]

failable_deliverables: for '^Server$', '*': ['buildinstall', 'iso'] — installer and ISOs for server failing do not abort the compose. For '^.*$': 'src': ['buildinstall'] (buildinstall is not blocking), and 'i386': ['buildinstall', 'iso'], 'ppc64le': ['buildinstall', 'iso'], 's390x': ['buildinstall', 'iso'] — nothing on i386, ppc64le blocks the compose.

Now the ostree trio, and honestly it's giving main character energy:

          ostree = {
              "^Silverblue$": {
                  "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
                  # To get config, clone master branch from this repo and take
                  # treefile from there.
                  "treefile": "fedora-silverblue.yaml",
                  "config_url": "https://pagure.io/workstation-ostree-config.git",
                  "config_branch": "master",
                  # Consume packages from Everything
                  "repo": "Everything",
                  # Don't create a reference in the ostree repo (signing automation does that).
                  "tag_ref": False,
                  # Don't use change detection in ostree.
                  "force_new_commit": True,
                  # Use unified core mode for rpm-ostree composes
                  "unified_core": True,
                  # This is the location for the repo where new commit will be
                  # created. Note that this is outside of the compose dir.
                  "ostree_repo": "/mnt/koji/compose/ostree/repo/",
                  "ostree_ref": "fedora/rawhide/${basearch}/silverblue",
                  "arches": ["x86_64", "ppc64le", "aarch64"],
                  "failable": ['*'],
              }
          }

ostree for "^Silverblue$": "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN"; "treefile": "fedora-silverblue.yaml" (to get config, clone master branch from this repo and take treefile from there); "config_url": "https://pagure.io/workstation-ostree-config.git"; "config_branch": "master"; "repo": "Everything" (consume packages from Everything); "tag_ref": False (don't create a reference in the ostree repo (signing automation does that)); "force_new_commit": True (don't use change detection in ostree); "unified_core": True (use unified core mode for rpm-ostree composes); "ostree_repo": "/mnt/koji/compose/ostree/repo/" (this is the location for the repo where new commit will be created. Note that this is outside of the compose dir.); "ostree_ref": "fedora/rawhide/${basearch}/silverblue"; "arches": ["x86_64", "ppc64le", "aarch64"]; "failable": ['*'].

          ostree_container = {
              "^Sagano$": {
                  "treefile": "fedora-tier-0-38.yaml",
                  "config_url": "https://gitlab.com/CentOS/cloud/sagano.git",
                  "config_branch": "main",
                  # Consume packages from Everything
                  "repo": "Everything",
                  # Automatically generate a reasonable version
                  "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN",
                  # Only run this for x86_64 even if Sagano has more arches
                  "arches": ["x86_64"],
              }
          }

ostree_container for "^Sagano$": "treefile": "fedora-tier-0-38.yaml", "config_url": "https://gitlab.com/CentOS/cloud/sagano.git", "config_branch": "main", "repo": "Everything" (consume packages from Everything), "version": "!OSTREE_VERSION_FROM_LABEL_DATE_TYPE_RESPIN" (automatically generate a reasonable version), and "arches": ["x86_64"] (only run this for x86_64 even if Sagano has more arches).

And the ostree installer, which is the longest of them, of course it is:

          ostree_installer = [
              ("^Silverblue$", {
                  "x86_64": {
                      "repo": "Everything",
                      "release": None,
                      "rootfs_size": "8",
                      # Take templates from this repository.
                      'template_repo': 'https://pagure.io/fedora-lorax-templates.git',
                      'template_branch': 'master',
                      # Use following templates.
                      "add_template": ["ostree-based-installer/lorax-configure-repo.tmpl",
                                       "ostree-based-installer/lorax-embed-repo.tmpl",
                                       "ostree-based-installer/lorax-embed-flatpaks.tmpl"],
                      # And add these variables for the templates.
                      "add_template_var": [
                          "ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/ostree/repo/",
                          "ostree_update_repo=https://ostree.fedoraproject.org",
                          "ostree_osname=fedora",
                          "ostree_oskey=fedora-32-primary",
                          "ostree_contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist",
                          "ostree_install_ref=fedora/rawhide/x86_64/silverblue",
                          "ostree_update_ref=fedora/rawhide/x86_64/silverblue",
                          "flatpak_remote_name=fedora",
                          "flatpak_remote_url=oci+https://registry.fedoraproject.org",
                          "flatpak_remote_refs=runtime/org.fedoraproject.Platform/x86_64/f30 app/org.gnome.Baobab/x86_64/stable",
                      ],
                      'failable': ['*'],
                  },
              })
          ]

ostree_installer is a list, and for "^Silverblue$" under "x86_64": "repo": "Everything", "release": None, "rootfs_size": "8"; 'template_repo': 'https://pagure.io/fedora-lorax-templates.git' (take templates from this repository) with 'template_branch': 'master'; "add_template" (use following templates) = ["ostree-based-installer/lorax-configure-repo.tmpl", "ostree-based-installer/lorax-embed-repo.tmpl", "ostree-based-installer/lorax-embed-flatpaks.tmpl"]; and "add_template_var" (and add these variables for the templates) = "ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/ostree/repo/", "ostree_update_repo=https://ostree.fedoraproject.org", "ostree_osname=fedora", "ostree_oskey=fedora-32-primary", "ostree_contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist", "ostree_install_ref=fedora/rawhide/x86_64/silverblue", "ostree_update_ref=fedora/rawhide/x86_64/silverblue", "flatpak_remote_name=fedora", "flatpak_remote_url=oci+https://registry.fedoraproject.org", "flatpak_remote_refs=runtime/org.fedoraproject.Platform/x86_64/f30 app/org.gnome.Baobab/x86_64/stable". Plus 'failable': ['*'].

Whew. And we are still not done, bestie — keep scrolling, more is coming. 🦋

RCM Tools compose 🎀

This one's a lil compose used to deliver packages to Red Hat internal users, and the config is split across two files, okay?

# rcmtools-common.conf

release_name = "RCM Tools"
release_short = "RCMTOOLS"
release_version = "2.0"
release_type = "updates"
release_is_layered = True
createrepo_c = True
createrepo_checksum = "sha256"

# PKGSET
pkgset_source = "koji"
koji_profile = "brew"
pkgset_koji_inherit = True

# GENERAL SETTINGS
bootable = False
comps_file = "rcmtools-comps.xml"
variants_file = "rcmtools-variants.xml"
sigkeys = ["3A3A33A3"]

# RUNROOT settings
runroot = False

# GATHER
gather_method = "deps"
check_deps = True

additional_packages = [
    ('.*', {
        '*': ['puddle', 'rcm-nexus'],
        }
    ),
]

# Set repoclosure_strictness to fatal to avoid installation dependency
# issues in production composes
repoclosure_strictness = [
    ("^.*$", {
        "*": "fatal"
    })
]

And then the config that's specific to different base products gets split into its own separate files, like so:

# rcmtools-common.conf
from rcmtools-common import *

# BASE PRODUCT
base_product_name = "Red Hat Enterprise Linux"
base_product_short = "RHEL"
base_product_version = "7"

# PKGSET
pkgset_koji_tag = "rcmtools-rhel-7-compose"

# remove i386 arch on rhel7
tree_arches = ["aarch64", "ppc64le", "s390x", "x86_64"]

check_deps = False

# Packages in these repos are available to satisfy dependencies inside the
# compose, but will not be pulled in.
gather_lookaside_repos = [
    ("^Client|Client-optional$", {
        "x86_64": [
            "http://example.redhat.com/rhel/7/Client/x86_64/os/",
            "http://example.redhat.com/rhel/7/Client/x86_64/optional/os/",
        ],
    }),
     ("^Workstation|Workstation-optional$", {
        "x86_64": [
            "http://example.redhat.com/rhel/7/Workstation/x86_64/os/",
            "http://example.redhat.com/rhel/7/Workstation/x86_64/optional/os/",
        ],
    }),
    ("^Server|Server-optional$", {
        "aarch64": [
            "http://example.redhat.com/rhel/7/Server/aarch64/os/",
            "http://example.redhat.com/rhel/7/Server/aarch64/optional/os/",
        ],
        "ppc64": [
            "http://example.redhat.com/rhel/7/Server/ppc64/os/",
            "http://example.redhat.com/rhel/7/Server/ppc64/optional/os/",
        ],
        "ppc64le": [
            "http://example.redhat.com/rhel/7/Server/ppc64le/os/",
            "http://example.redhat.com/rhel/7/Server/ppc64le/optional/os/",
        ],
        "s390x": [
            "http://example.redhat.com/rhel/7/Server/s390x/os/",
            "http://example.redhat.com/rhel/7/Server/s390x/optional/os/",
        ],
        "x86_64": [
            "http://example.redhat.com/rhel/7/Server/x86_64/os/",
            "http://example.redhat.com/rhel/7/Server/x86_64/optional/os/",
        ],
    })
]

🦋 EXPORTING FILES FROM SCM 🦋

Okay bestie, listen — multiple places in Pungi can pull files from external storage. The config stays basically the same no matter which backend you're using, even though some features might differ a lil between them.

The so-called scm_dict always gets put into the configuration as a dictionary, and it can hold these keys:

scm — tells you which SCM system is in play. This one is always required, no skips. Allowed values are:
- file — copies files from the local filesystem
- git — copies files from a Git repository
- cvs — copies files from a CVS repository
- rpm — copies files from a package in the compose
- koji — downloads archives from a given build in the Koji build system
- container-image — downloads an artifact from a container registry
repo
- for Git and CVS backends this should be the URL to the repository
- for the RPM backend this should be a shell style glob matching package names (or a list of such globs)
- for the file backend this should be empty
- for the Koji backend this should be an NVR or package name
branch
- branch name for Git and CVS backends, with master and HEAD as defaults
- Koji tag for the koji backend if only a package name is given
- otherwise it should not be specified
file — a list of files that should be exported.
dir — a directory that should be exported. All its contents will be exported. This option is mutually exclusive with file.
command — defines a shell command to run after Git clone to generate the needed file (for example, to run make). Only supported in the Git backend.
options — a dictionary of additional configuration options. These are specific to the different backends.

Currently supported values for Git:

credential_helper — path to a credential helper used to supply username/password for remotes that require authentication.

Koji examples ✨

There are two different ways to configure the Koji backend, and honestly both are giving.

{
    # Download all *.tar files from build my-image-1.0-1.
    "scm": "koji",
    "repo": "my-image-1.0-1",
    "file": "*.tar",
}

{
    # Find latest build of my-image in tag my-tag and take files from
    # there.
    "scm": "koji",
    "repo": "my-image",
    "branch": "my-tag",
    "file": "*.tar",
}

Real talk though: using both a tag name and an exact NVR will result in an error. The NVR would get interpreted as a package name, and it would not match anything. So don't do that. 💀

file vs. dir 🌸

Exactly one of these two options has to be specified. No both, no neither, pick one. The docs for each configuration option should tell you whether it expects a file or a dir.

For the extra_files phase, either key is valid, so just choose based on what your actual use case is.

container-image example 🎀

Here's an example of pulling a container image into the compose:

{
    # Pull a container into an oci-archive tar file
    "scm": "container-image",
    # This is the pull spec including tag. It is passed directly to skopeo
    # copy with no modification.
    "repo": "docker://registry.access.redhat.com/ubi9/ubi-minimal:latest",
    # Key `file` is required, but the value is ignored.
    "file": "",
    # Optional subdirectory under Server/<arch>/os
    "target": "containers",
}

Caveats 🙈

The rpm backend can only be used in phases that would extract the files after the pkgset phase finished. You can't get a comps file from a package, bestie. It's just not the vibe.
Depending on the Git repository URL configuration, Pungi can only export the requested content using git archive. When a command should run, this isn't possible, and a clone is always needed.
When using the koji backend, it's required that you provide configuration for the Koji profile to be used (koji_profile). It is not possible to contact multiple different Koji instances.

💅 PROGRESS NOTIFICATION 💅

Pungi has the ability to emit notification messages about the progress and general status of the compose. These can be used to, like, send messages to fedmsg. This is implemented by actually calling a separate script.

The script will be called with one argument describing the action that just happened. A JSON-encoded object gets passed to standard input to give more info about the event. At the very least, the object will contain a compose_id key.

The notification script inherits the working directory from the parent process, and it can be called from the same directory pungi-koji is called from. That working directory is listed at the start of the main log. 📝

Currently these messages are sent:

status-change — when composing starts, finishes or fails; a status key is provided to indicate details
phase-start — on start of a phase
phase-stop — when a phase is finished
createiso-targets — with a list of images to be created
createiso-imagedone — when any single image is finished
createiso-imagefail — when any single image fails to create
fail-to-start — when there are incorrect CLI options or errors in the configuration file; this message does not contain compose_id, nor is it started in the compose directory (which does not exist yet)
ostree — when a new commit is created, this message will announce its hash and the name of the ref it is meant for.

For phase related messages, a phase_name key is provided as well. 👑

A pungi-fedmsg-notification script is provided and understands this interface.

Setting it up 🌈

The script should be provided as a command line argument --notification-script, like this:

--notification-script=pungi-fedmsg-notification

💖 GATHERING PACKAGES 💖

okay so a compose that Pungi makes? it's made of one or more variants. and a variant is just a lil subset of the content, aimed at one specific use case. that's it, that's the vibe ✨

there are different types of variants, and the type literally changes how packages get gathered into the variant. so pay attention bestie, this part matters 🫶

the inputs for gathering are defined by various gather sources. packages from all the sources get collected together to build one big list of:

package names
comps group names
a list of packages that should be filtered out

NOTE: the inputs for both the explicit package list and the comps file are interpreted as RPM names. not any arbitrary provides, and not source package names. iykyk 💅

next, gather_method decides how that list gets processed:

for nodeps, the results from source are used pretty much as-is <sup>[1]</sup>
for the deps method, a whole process gets launched to figure out which dependencies are needed, and those get pulled in too

[1] the lists are filtered based on what packages are actually available in the package set, but nothing else will be pulled in. periodt.

🌈 Variant types 🌈

Variant — is the base type. it has no special behaviour, it's just vibing.
Addon — is built on top of a regular variant. any packages that should go to both the addon and its parent get removed from the addon. and packages that are only in the addon but got pulled in because of the gather_fulltree option? those get moved to the parent.
Integrated Layered Product — works similarly to addon. additionally, all packages from addons on the same parent variant are removed from integrated layered products.

the main difference between an addon and an integrated layered product is that an integrated layered product has its own identity in the metadata (defined with product name and version).

NOTE: there's also Layered Product as a term, but that is not related to variants. it's used to describe a product that is not a standalone operating system and is instead meant to be used on some other base system. don't mix them up bestie 😭

Optional — contains packages that complete the base variant's package set. it always has fulltree and selfhosting enabled, so it contains build dependencies and packages which were not specifically requested for the base variant.

some configuration options get overridden for particular variant types. slay 💅

⭐ Depsolving configuration ⭐

here's how the overrides shake out, per variant type:

base — Fulltree: configurable · Selfhosting: configurable
addon/ILP — Fulltree: enabled · Selfhosting: disabled
optional — Fulltree: enabled · Selfhosting: enabled

🔥 Profiling 🔥

profiling data on the pungi-gather tool can be enabled by setting the gather_profiler configuration option to True. go off, measure your performance king 👑

🦋 Modular compose 🦋

a compose with gather_source set to module is called modular. the package list is determined by a list of modules.

the list of modules that will be put into a variant is defined in the variants.xml file. that file can contain either Name:Stream or Name:Stream:Version references. see Module Naming Policy for details. when Version is missing from the specification, Pungi will ask PDC for the latest one.

the module metadata in PDC contains a list of RPMs in the module, as well as the Koji tag from which the packages can be retrieved. iconic.

Restrictions

a modular compose must always use Koji as a package set source.

🌸 GETTING DATA FROM KOJI 🌸

when Pungi is configured to get packages from a Koji tag, it somehow needs to access the actual RPM files. (yes, somehow. the drama 😭)

historically, this required the storage used by Koji to be directly available on the host where Pungi was running. this was usually achieved by using NFS for the Koji volume, and mounting it on the compose host.

the compose could be created directly on the same volume. in that case the packages would be hardlinked, significantly reducing space consumption. big brain energy.

the compose could also be created on a different storage, in which case the packages would either need to be copied over or symlinked. using symlinks requires that anything that accesses the compose (e.g. a download server) would also need to mount the Koji volume in the same location. so like… coordinate, besties 🫶

there is also a risk with symlinks that the package in Koji can change (due to being resigned for example), which would invalidate composes linking to it. 💀

🎀 Using Koji without direct mount 🎀

it is possible now to run a compose from a Koji tag without direct access to Koji storage. we love to see it ✨

Pungi can download the packages over HTTP protocol, store them in a local cache, and consume them from there. to enable this behavior, set the koji_cache option in the compose configuration.

the local cache has a similar structure to what's on the Koji volume.

when Pungi needs some package, it has a path on Koji volume. it will replace the topdir with the cache location. if such file exists, it will be used. if it doesn't exist, it will be downloaded from Koji (by replacing the topdir with topurl).

Koji path     /mnt/koji/packages/foo/1/1.fc38/data/signed/abcdef/noarch/foo-1-1.fc38.noarch.rpm
Koji URL      https://kojipkgs.fedoraproject.org/packages/foo/1/1.fc38/data/signed/abcdef/noarch/foo-1-1.fc38.noarch.rpm
Local path    /mnt/compose/cache/packages/foo/1/1.fc38/data/signed/abcdef/noarch/foo-1-1.fc38.noarch.rpm

the packages can be hard- or softlinked from this cache directory (/mnt/compose/cache in the example).

🧚 Cleanup 🧚

while the approach above allows each RPM to be downloaded only once, it will eventually result in the Koji volume being mirrored locally. most of the packages will however no longer be needed. classic hoarding behaviour ngl 🙈

there is a script pungi-cache-cleanup that can help with that. it can find and remove files from the cache that are no longer needed.

a file is no longer needed if:

it has a single link (meaning it is only in the cache, not in any compose), and
it has mtime older than a given threshold.

it doesn't make sense to delete files that are hardlinked in an existing compose, as it would not save any space anyway. no cap.

the mtime check is meant to preserve files that are downloaded but not actually used in a compose — like a subpackage that is not included in any variant. every time its existence in the local cache is checked, the mtime is updated. cute lil timestamp refresh 💖

👑 Race conditions? 👑

it should be safe to have multiple compose hosts share the same storage volume for generated composes and local cache. yes queen, share the storage.

if a cache file is accessed and it exists, there's no risk of race condition. we're chilling.

if two composes need the same file at the same time and it is not present yet, one of them will take a lock on it and start downloading. the other will wait until the download is finished.

the lock is only valid for a set amount of time (5 minutes) to avoid issues where the downloading process is killed in a way that blocks it from releasing the lock.

if the file is large and network slow, the limit may not be enough to finish downloading. in that case the second process will steal the lock while the first process is still downloading. this will result in the same file being downloaded twice. 💀 not ideal but here we are.

when the first process finishes the download, it will put the file into the local cache location. when the second process finishes, it will atomically replace it, but since it's the same file it will be the same file. (the way that's just… the same file 😭)

if the first compose already managed to hardlink the file before it gets replaced, there will be two copies of the file present locally.

💜 Integrity checking 💜

there is minimal integrity checking. RPM packages belonging to real builds will be checked to match the checksum provided by Koji hub.

there is no checking for scratch builds or any images. fr fr, don't expect it.

✨ PROCESSING COMPS FILES ✨

the comps file that Pungi takes as input is not really pure comps as used by tools like DNF. there are extensions used to customize how the file is processed. it's giving comps but make it extra 💅

step one: Pungi retrieves the actual file. this can use anything that Exporting files from SCM supports.

the Pungi extensions are the arch attribute on packageref, group and environment tags. the value of this attribute is a comma separated list of architectures.

step two: Pungi creates a file for each architecture. this is done by removing all elements with an incompatible arch attribute. no additional clean up is performed on this file. the resulting file is only used internally for the rest of the compose process.

step three (and final): create a comps file for each Variant.Arch combination. this is the actual file that will be included in the compose. the start file is the original input file, from which all elements with incompatible architecture are removed. then clean up is performed by:

removing all empty groups,
removing non-existing groups from environments and categories,
and finally removing empty environments and categories.

as a last step, groups not listed in the variants file are removed. and that's the whole glow-up routine 🌈✨

💖 CONTRIBUTING TO PUNGI 💖

okay bestie, you wanna actually work on Pungi? iconic. let's get you set up so you can slay. 🫶

🌸 Set up development environment

To work on Pungi, you should install a recent version of Fedora.

🐍 Python2

Fedora 29 is recommended, because some packages are just not available in newer Fedora releases — like python2-libcomps. (the struggle is real 😭)

Install the required packages:

$ sudo dnf install -y krb5-devel gcc make libcurl-devel python2-devel python2-createrepo_c kobo-rpmlib yum python2-libcomps python2-libselinx

🐍 Python3

Install the required packages:

$ sudo dnf install -y krb5-devel gcc make libcurl-devel python3-devel python3-createrepo_c python3-libcomps

✨ Developing

Currently the development workflow for Pungi is on the master branch. here's the whole vibe:

Make your own fork at https://forge.fedoraproject.org/pungi/pungi
Clone your fork locally
cd into your local clone and add the remote upstream so you can rebase:

cd pungi
git remote add upstream https://forge.fedoraproject.org/pungi/pungi.git

NOTE: this workflow assumes you never git commit directly to the master branch of your fork. it'll make way more sense once we get to the rebasing part below, promise. 💅

create a topic branch based on master:

git branch my_topic_branch master
git checkout my_topic_branch

make edits, changes, add new features, etc., and then make sure to pull from upstream master and rebase before submitting a pull request:

# lets just say you edited setup.py for sake of argument
git checkout my_topic_branch

# make changes to setup.py
black setup.py
tox
git add setup.py
git commit -s -m "added awesome feature to setup.py"

# now we rebase
git checkout master
git pull --rebase upstream master
git push origin master
git push origin --tags
git checkout my_topic_branch
git rebase master

# resolve merge conflicts if any as a result of your development in
# your topic branch
git push origin my_topic_branch

NOTE: in order for your commit to be merged: - you must sign-off on it. use the -s option when running git commit. - the code must be well formatted via black and pass flake8 checking. run tox -e black,flake8 to do the check.

create a pull request in the Forge web UI
for convenience, here's a bash shell function you can drop into your ~/.bashrc and call like pullupstream pungi-4-devel — it automates a large portion of the rebase steps from above:

pullupstream () {
  if [[ -z "$1" ]]; then
    printf "Error: must specify a branch name (e.g. - master, devel)\n"
  else
    pullup_startbranch=$(git describe --contains --all HEAD)
    git checkout $1
    git pull --rebase upstream master
    git push origin $1
    git push origin --tags
    git checkout ${pullup_startbranch}
  fi
}

🧪 Testing

You must write unit tests for any new code — unless it's a trivial change, no exceptions. any code without sufficient test coverage may not be merged. periodt. 👑

To run all the existing tests, the suggested method is to use tox.

$ sudo dnf install python3-tox -y

$ tox -e py3
$ tox -e py27

Alternatively, you could create a vitualenv, install deps and run tests manually if you don't want to use tox.

$ sudo dnf install python3-virtualenvwrapper -y
$ mkvirtualenv --system-site-packages py3
$ workon py3
$ pip install -r requirements.txt -r test-requirements.txt
$ make test

# or with coverage
$ make test-coverage

If you need to run specified tests, pytest is recommended.

# Activate virtualenv first

# Run tests
$ pytest tests/test_config.py
$ pytest tests/test_config.py -k test_pkgset_mismatch_repos

In the tests/ directory there's a shell script test_compose.sh that you can use to try and create a miniature compose on dummy data. the actual data gets created by running make test-data in the project root. 🌈

$ sudo dnf -y install rpm-build createrepo_c isomd5sum genisoimage syslinux

# Activate virtualenv (the one created by tox could be used)
$ source .tox/py3/bin/activate

$ python setup.py develop
$ make test-data
$ make test-compose

This testing compose does not actually use all the phases that are available, and there is no checking that the result is correct. it only tells you whether it crashed or not. fr fr. 💀

NOTE: even when it finishes successfully, it may print errors about repoclosure on Server-Gluster.x86_64 in the test phase. this is not a bug. don't come for it.

📝 Documenting

You must write documentation for any new features and functional changes. any code without sufficient documentation may not be merged. same energy as the tests rule, bestie. ✍️

To generate the documentation, run make doc in the project root.

🔥 TESTING PUNGI 🔥

🍓 Test Data

Tests require test data and not all of it is available in git. you must create test repositories before running the tests:

make test-data

Requirements: createrepo_c, rpmbuild

⭐ Unit Tests

Unit tests cover functionality of Pungi python modules. you can run all of them at once:

make test

which is a shortcut to:

python2 setup.py test
python3 setup.py test

You can alternatively run individual tests:

cd tests
./<test>.py [<class>[.<test>]]

🦋 Functional Tests

Because a compose is quite a complex process and not everything is covered with unit tests yet, the easiest way to test whether your changes broke anything badly is to start a compose on a relatively small and well-defined package set:

cd tests
./test_compose.sh

👑 AUTHOR

Pungi maintainers

💜 COPYRIGHT

2016, Red Hat, Inc.

and that's a wrap on the whole page, bestie!! 🎀 the source/version line at the bottom is Aug 31, 2026 and the version tag is PUNGI(1) — so yeah, go forth, write your tests, sign off your commits, and slay. the maintainers are watching and they're rooting for you. mwah. 💖✨🦋

📦 where this page comes from

fedora 43: pungi 4.14.0-2.fc43 · GPL-2.0-only
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-10-18: a language model's rewrite of the original, under its terms