autopkgtest(1)

📦
version 6.0 · 7 versions
, in ubuntu stonking

✨ autopkgtest — the manual but make it cute ✨

okay bestie, so autopkgtest(1) is the General Commands Manual page for autopkgtest, and it is literally the tool that tests an installed binary package using the source package's tests. yes she is that girl. let's get into it 💅

🌸 NAME 🌸

autopkgtest — test an installed binary package using the source package's tests

🦋 SYNOPSIS 🦋

autopkgtest [options...]  [testbinary...]  testsrc -- virt-server
[virt-server-arg...]

💖 DESCRIPTION 💖

so what does she actually do?? autopkgtest runs tests on binary Debian packages, as installed on a system — and that system is called the "testbed". the tests themselves come from the source package. no cap.

she runs each test that a particular package supplies, and then reports the results back to you. she also drives whatever virtualisation regime you told her to use, parses the test description metadata, and arranges for data to be copied to and from the testbed as needed. multitasking queen 👑

if you want an actual introduction to how to use autopkgtest, go read /usr/share/doc/autopkgtest/README.running-tests.rst.gz. it's giving tutorial energy ✨

🎀 TESTING A DEBIAN PACKAGE 🎀

the positional (non-option) arguments specify exactly one source package (that's the one with the test code in it) and optionally some binary packages to test. one source package, bestie. not two. iykyk.

testsrc can be one of these:

  • .dsc file — run tests from a Debian .dsc source package. by default the package will also get built, and the resulting binaries get used to satisfy test dependencies 💅 to turn that off, pass the -B/--no-built-binaries option.
  • source package directory — run tests from a Debian source tree directory. if it's an unbuilt tree, this is very similar to specifying a .dsc. if it's a built tree, all test dependencies get satisfied by archive packages — unless you explicitly also specify locally built .debs.
⚠️ attention bestie: if you just specify a bare directory name that happens to be a legal Debian source package name, it will be interpreted as the latter (the source package name, see below). in that case, prefix the directory name with ./. do not forget the ./, i am begging 😭
  • current directory — if no source package is specified on the command line and the current directory is a Debian source package, then that's what gets tested. main character behavior honestly.
  • source package name — downloads the given source package name with apt-get source in the testbed, and runs its tests. this is similar to specifying a .dsc, but it avoids copying the source from the host to the testbed 🌈. possibly built binaries (if the test specifies build-needed) will not be used to satisfy dependencies — because usually in this mode you want to test binaries from a real archive.
  • git URL or URL#branch — git-clones the given URL (which must contain an unbuilt Debian source tree) and runs the tests from that 🧚. if branch is given, that branch gets checked out instead of the default (usually "master"). this can also be a more general refspec such as #refs/pull/123/head for a GitHub pull request.

this is very similar to cloning manually and specifying the checkout directory as test; i.e. this is commonly used with --no-built-binaries. the git package will be installed if necessary.

  • .changes file — run tests from the .dsc source package in the given .changes file. if the .changes contains .deb packages, they will be used for the test. acts as if you had specified the .debs and .dsc from the .changes file as explicit arguments. 📝 note that if the .changes contains only debs, the corresponding .dsc still needs to be specified alongside, or the current directory must be the source package.

all other positional arguments must be .deb binary packages. they get used for both build and test dependencies of the source package. and heads up: if any binary package is given, then --no-built-binaries is implied. periodt. 💀

👑 TEST OPTIONS 👑

  • -a ARCH, --test-architecture=ARCH — make the testbed a multiarch system, set up the environment for cross-building (possibly installing extra build dependencies), and run tests for the specified foreign architecture rather than for the testbed native architecture. she assumes that ARCH binaries can be run on the testbed, possibly via qemu-user-binfmt or qemu-user. cross-compiling icon behavior ✨
  • -B, --no-built-binaries — binaries from unbuilt source packages (see above) will not be built or will be ignored, and dependencies get satisfied with packages from the archive. 🫶 note that the source package still gets built if a test requires build-needed.
  • --override-control=PATH — read the test metadata from PATH instead of debian/tests/control. going off-script, we love to see it.
  • --test-name=TEST — run only the given test name (from the test control file). if this option is used more than once, all the named tests are run. cherry-picking queen 🍓
  • --skip-test=TEST — skip the given test name (from the test control file). if this option is used more than once, all the named tests are skipped. not me avoiding my problems… 🙈

💜 LOGGING OPTIONS 💜

if you don't specify any option at all, autopkgtest only writes its output/results to stderr. that's it. that's the default, lowkey minimalist 💫

  • -o dir, --output-dir=dir — specifies that test artifacts (stderr and stdout from the tests, the log file, built binary packages etc.) should be placed in the given directory. dir must not exist yet or be empty — otherwise autopkgtest will refuse to use it. she is picky and honestly? fair. 💅
  • -l logfile, --log-file=logfile — specifies that the trace log should be written to logfile instead of to output-dir.
  • --summary=summary — specifies that a summary of the outcome should be written to summary. the events in the summary are written to the log in any case, so you're never missing out 📋
  • -q, --quiet — do not send a copy of autopkgtest's trace logstream to stderr. this option does not affect the copy sent to logfile or output-dir.
bestie, real talk: without the trace logstream it can be very hard to diagnose problems. i'm just saying. 😭

more parts coming, stay tuned 🌟

✨ TEST BED SETUP OPTIONS ✨

  • --setup-commands=commands — run these commands before any test happens, but after we've added and updated the APT sources you asked for with --add-apt-source, --add-apt-release, or --apt-pocket. Basically it's your "do literally anything autopkgtest doesn't already have an option for" card 🎀. If commands is the name of a file that actually exists, the commands get read out of that file; otherwise it's just treated as a plain string of commands and run exactly as-is. Fun fact: if you give a file name with no directory in it, autopkgtest will look in both the current directory and in /usr/share/autopkgtest/setup-commands/, so you don't have to type out the whole path for the setup scripts that ship with autopkgtest 🫶.

Usually, if the setup commands fail, autopkgtest treats it as a transient testbed error (exit code 16). BUT — plot twist — if your setup commands exit with code 100, autopkgtest goes "oh this is an erroneous package" (exit code 12) instead. So you can use this to, like, detect upgrade errors to a new version. And yeah, apt does exit with 100 in those cases, iykyk 💅.

This option can be specified multiple times.

If --user is given, or the test bed has a suggested-normal-user capability, then the $AUTOPKGTEST_NORMAL_USER environment variable gets set to that user.

If your setup commands touch anything in the boot directories (think /boot or /lib/systemd/system) and the testbed supports rebooting, then the testbed gets rebooted after the setup commands. Want to block that? Just create a file at /run/autopkgtest_no_reboot.stamp and it's suppressed.

Warning: the way autopkgtest uses this user means it's gonna be completely trusted by root inside the testbed. Like, fully. No cap. 👑
  • --setup-commands-boot=commands — run commands after --setup-commands, and also after every reboot. Good example: adding files inside a tmpfs ✨.

These commands will never cause the testbed to reboot (because that could loop forever, and we're not doing that 😭). Other than that, they behave just like --setup-commands.

This option can be specified multiple times.

  • --add-apt-source='deb|deb-src [ options ] MIRROR SUITE COMPONENT...'
  • --add-apt-source=ppa:[USER:TOKEN@]OWNER/PPA[:FINGERPRINT]
  • --add-apt-source=FILENAME

Adds the given apt source to the APT sources.list. Specifying this option triggers an apt-get update. You can shove multiple sources in at once by separating them with a newline, but honestly, specifying the option multiple times is the preferred vibe 🎀.

When you use the ppa: syntax, both the binary ("deb") and source ("deb-src") entries get added. Also, if USER and TOKEN are supplied, then the PPA is assumed to be private and FINGERPRINT becomes a mandatory field, periodt. Heads up though: autopkgtest can leak TOKEN in so many ways, especially when running in debug mode. So if your credentials are private, run autopkgtest on a trusted platform, fr fr.

When you use the FILENAME syntax, the contents of the file you named get configured as sources on the testbed. FILENAME should end with .list or .sources.

  • --add-apt-release=RELEASE — adds the given RELEASE to the APT sources.list. The mirrors, components and options all get taken from the suite that was found to provide base-files before APT sources got extended by autopkgtest. All the "deb" sources.list entries for this release are used as a template for adding RELEASE. Both binary ("deb") and source ("deb-src") entries get added. Specifying this option triggers an apt-get update. This option can be specified multiple times.

Note that autopkgtest ignores the "NotAutomatic: yes" setting that added releases might declare — like Debian "experimental", the Debian "backports" pocket, or the Ubuntu "proposed" pocket as of 24.04 LTS (Noble Numbat). 🙈

  • --apt-pocket=POCKET[=PACKAGELIST] — adds the given POCKET to the APT sources.list. The pocket "base" release is assumed to be the suite found providing base-files before APT sources got extended by autopkgtest. Specifying this option triggers an apt-get update.

If you specify PACKAGELIST, you can pull only specific packages from the pocket. It's equivalent to --pin-packages=BASE-POCKET=PACKAGELIST, where BASE is the pocket base release as defined above. For more on the PACKAGELIST syntax, go peep --pin-packages 👀.

  • --pin-packages=RELEASE=PACKAGE[,PACKAGE2,...] — sets up apt pinning so only the specified packages come from RELEASE. This is for minimizing the dependencies you take from RELEASE, so package updates in that release can be tested independently from each other and stay nicely isolated ✨. If PACKAGE is given as src:SRCPACKAGE, then the pinning will cover all binary packages built by source package SRCPACKAGE.
Attention: this does not currently resolve some situations where the dependencies of the given packages can only be resolved in RELEASE. In that case the apt pinning gets removed and package installation is retried with the entirety of RELEASE — unless you specified --no-apt-fallback.
  • --no-apt-fallback — disables the apt-get fallback that's used with --pin-packages or --apt-pocket when dependency installation fails because of strict pinning.
  • --apt-upgrade | -U — runs apt-get update, upgrade, autopurge in the testbed before running the tests. Any --add-apt-source or --apt-pocket options take effect first, so this will upgrade packages from those sources if that's the right move 🔥.
  • --apt-default-release=SUITE — picks the release that should provide the test dependencies that aren't explicitly pinned via --apt-pocket or --pin-packages. Specifying this option usually isn't necessary at all: suitable pinning is configured automatically. This option is equivalent to setting APT::Default-Release.
  • --copy=HOSTPATH:TESTBEDPATH — copies a file or directory from the host into the testbed right after opening it. This happens before --setup-commands, which means you can actually use these files in your setup commands 🎀.
  • --env=VAR=value — sets an arbitrary environment variable in the build and the test. Can be specified multiple times.
  • --ignore-restrictions=RESTRICTION,RESTRICTION... — if a test would normally get skipped because it has Restrictions: RESTRICTION, you run it anyway. Can be specified multiple times.

For example, you might ignore the isolation-machine restriction when you're using the null virtualization server, if you know autopkgtest itself is running on an expendable virtual machine 💀. These options also work for unknown restrictions, so they're great for experimenting with brand-new restrictions.

👑 USER/PRIVILEGE HANDLING OPTIONS 👑

  • -u user | --user=user — run builds and tests as user on the testbed. This needs root on the testbed; if root on the testbed isn't available, then builds and tests run as whatever user you got.
  • --gain-root=gain-root — prefixes debian/rules binary with gain-root. The default is to not use anything at all, except that if --user is supplied, or root on the testbed isn't available, the default becomes fakeroot.
  • --insecure — relaxes file permissions and stuff, trying to make things not blow up with permission errors 🙈.

Passing this option might allow tests to corrupt the testbed (if it doesn't support resetting), or to escape from it, or it might let untrusted processes on the host or the testbed take over the account running autopkgtest. So, like, yeah.

It is safe to pass --insecure on a dedicated throwaway host. ✨

This partially restores the behaviour of autopkgtest 5.47 and earlier, which had several world-writable directories.

🔥 DEBUGGING OPTIONS 🔥

  • --debug | -d — includes extra debugging information in the trace log. Each additional -d bumps the debugging level; the current maximum is -ddd. If you like to actually see what's going on, -d or -dd is the recommendation 💅.
  • --shell-fail | -s — runs an interactive shell in the testbed after a failed build, test, or dependency installation.
  • --shell — runs an interactive shell in the testbed after every test.

⏰ TIMEOUT OPTIONS ⏰

  • --timeout=seconds — defines the global timeout, which limits the time allowed to perform the build (if one is required) and to run the tests. If the global timeout is reached during the build, the run gets aborted exactly the same way as if the build had failed. If the global timeout is exceeded while running a test, the test is aborted and any remaining tests get skipped. (default: 0s, which means no global timeout is in effect). The value must be specified as an integer number of seconds.
  • --timeout-which=seconds — uses a different timeout for operations on or with the testbed. There are five timeouts affected by five values of which: short: supposedly short operations like setting up the testbed's apt and checking the state (default: 100s); install: installation of packages including dependencies (default: 3,000s); test: test runs (default: 10,000s); copy: copy files/directories between host and testbed (default: 300s); and build: builds (default: 100,000s). The value must be specified as an integer number of seconds.
  • --timeout-factor=double — multiplies all of the default timeouts by the specified factor (see --timeout-which above 👆). Only the defaults are affected; explicit timeout settings get used exactly as specified. Periodt.

🌈 LOCALE OPTIONS 🌈

  • --set-lang=langval — when running commands on the testbed, sets the LANG environment variable to langval. The default in autopkgtest is to set it to C.UTF-8 🌸.

🌈 OTHER OPTIONS 🌈

  • --no-auto-control — bestie, this shuts OFF the automatic test generation that autodep8 would normally do, even if autodep8 is sitting right there installed on the system. So what happens to packages with no tests? They just exit with code 8 ("No tests in this package"), exactly like they would if autodep8 wasn't there at all. no cap 💅
  • --build-parallel=N — sets parallel=N as a DEB_BUILD_OPTION while building packages. Default is just the number of available processors. Why would you even want this? Mostly for containers where you can cap the available RAM but can't cap the number of CPUs — so you dial the parallelism down by hand. iconic little workaround ✨
  • --needs-internet=run|try|skip — this one decides how we treat the needs-internet restriction:
    • try — tests with needs-internet restrictions get run, but if they fail, they're treated as flaky. forgiving queen 🎀
    • skip — these tests get skipped immediately, not run at all. bye 👋
    • run — the restriction is basically ignored. this is the default, bestie.
  • -V, --validate — validate the test control file and dip out without running a single test. vibe check only 🌸
  • -h, --help — show the command line help and exit. the classic.

🖥️ VIRTUALIZATION SERVER 🖥️

-- virt-server virt-server-arg...

This is where you tell autopkgtest which virtualisation regime server to use — given as a command plus arguments to invoke.

In the autopkgtest invocation itself, literally everything after -- gets passed straight to the virtualisation server. all yours, go off 💫

The autopkgtest package ships several virtualisation servers, named autopkgtest-virt-* (think autopkgtest-virt-qemu, autopkgtest-virt-unshare, ...), and each one has its own manpage. When you want to use one of these, you can drop the autopkgtest-virt- prefix from the command. so convenient, obsessed 🫶

Which virtualisation server to pick depends on a bunch of stuff — like how isolated you want the testbed to be, and what tools you personally vibe with. If you're unsure: the qemu virt server gives you VM isolation (meaning tests with the isolation-machine restriction are allowed) and only needs QEMU to be available. Go read autopkgtest-virt-qemu(1) for the details. 👑

📋 OUTPUT FORMAT 📋

During a normal test run, autopkgtest prints one line per test. Each line is: a short string identifying the test, some horizontal whitespace, then one of PASS, PASS details, FAIL reason, SKIP reason, or FLAKY reason — where the pass/fail indication and the reason or details are separated by some horizontal whitespace too. (Horizontal whitespace = spaces/tabs, not newlines. 💅)

That identifying string is a short alphanumeric thing invented by autopkgtest to tell different command-line arguments apart — the argid — followed by a hyphen and then the test name.

  • SKIP means a test wasn't run, OR that the test code started but figured out it couldn't complete — for example because a required resource just wasn't available. not every test gets its moment 😭
  • FLAKY means a test would ordinarily have failed, but because this particular test is known to be unreliable, the failure got ignored. we move 🦋

Sometimes you'll see a SKIP reported when the test's name isn't known or isn't applicable — like when there are no tests in the package at all, or there's a test stanza containing features this version of autopkgtest doesn't understand. In that case, * appears where the test name should be. yes, literally a star ⭐

If autopkgtest detects that erroneous package(s) are involved, it prints these two lines:

blame: blamed-thing...
badpkg: message

Here, each whitespace-separated blamed-thing is one of arg:argument (representing a pathname found in a command line argument), dsc:package (a source package name), deb:package (a binary package name), or possibly other strings to be determined. This tells you which arguments and/or packages might have contributed to the problem. And plot twist — the ones processed most recently, which are therefore most likely to be the cause, are listed LAST. 🕵️♀️

Other possible errors you might run into: erroneous package: message, testbed failure: message, or quitting: message. 💀

⚙️ CONFIGURATION FILES ⚙️

If you're the type who uses a lot of options, or your virt server arguments are nontrivial (respect), you can dump any part of the command line into a text file — one line per option. easy 🎀

So like, you could make a file called sid.cfg with contents that look like:

-s
--output-dir=/tmp/testout
--apt-upgrade
--
qemu
/path/to/sid.img

and then run:

autopkgtest foo_1_amd64.changes @sid.cfg

The contents of the config file get expanded in-place, as if you'd typed them right there on the command line. ✨

⚠️ Careful though bestie: do not put spaces between short options and their values — that space would become part of the argument value. so -s value NOT -s value, periodt.

🚪 EXIT STATUS 🚪

  • 0 — all tests passed 🎉
  • 2 — at least one test was skipped (or at least one flaky test failed)
  • 4 — at least one test failed
  • 6 — at least one test failed and at least one test skipped
  • 8 — no tests in this package, or all non-superficial tests were skipped
  • 12 — erroneous package
  • 14 — erroneous package and at least one test skipped
  • 16 — failure to open, close, or communicate with testbed
  • 20 — other unexpected failures including bad usage 💀

👀 SEE ALSO 👀

  • /usr/share/doc/autopkgtest/README.running-tests.rst.gz
  • /usr/share/doc/autopkgtest/README.package-tests.rst.gz

go read them, iykyk 📚

This manpage is part of autopkgtest, a tool for testing Debian binary packages. autopkgtest is Copyright (C) 2006-2014 Canonical Ltd.

See /usr/share/doc/autopkgtest/CREDITS for the list of contributors and the full copying conditions. 🫶

and that's the whole page bestie!! this one's from autopkgtest, dated 2014, and it IS the autopkgtest(1) manpage — you made it to the end, main character energy fr fr 💅✨🌸

📦 where this page comes from