LIT(1)

📦
version 23.1.4 · 11 versions
, in arch rolling

💅 lit — LLVM Integrated Tester

Bestie, welcome to LIT(1) — the manual for lit, the LLVM Integrated Tester. She's the cutest little test runner in the whole toolchain and she's about to tell you everything. Sit down, grab a matcha, let's go 🍓

✨ SYNOPSIS ✨

lit [options] [tests]

🎀 DESCRIPTION 🎀

Okay so first things first: lit is a portable tool for executing LLVM and Clang style test suites, summarizing their results, and giving you a heads-up when things fail. That's literally her whole job. She's designed to be a lightweight testing tool with a user interface that's as simple as possible — no bloat, no drama, no cap. 👑

You should run lit with one or more tests specified right there on the command line. Those tests can be either individual test files or directories that she'll search for tests (peep TEST DISCOVERY for the deets). 🕵️‍♀️

Each test you specify gets executed (potentially concurrently, because she multitasks 💅) and once all of them have run, lit prints summary info on how many tests passed or failed (that's TEST STATUS RESULTS, iykyk). And heads up: lit will exit with a non-zero exit code if any tests fail. So if your CI is red, that's why bestie. 😭

By default lit uses a succinct progress display and only prints summary information for test failures. If you wanna control the progress display and output, that's all in OUTPUT OPTIONS. 🎛️

She also ships with a bunch of options for controlling how tests are executed — though ngl, specific features may depend on the particular test format you're using. See EXECUTION OPTIONS for more. 🔥

Finally, lit supports additional options for only running a subset of the tests specified on the command line. That's SELECTION OPTIONS, and it's giving "main character energy" for real. ✨

💡 Important pipeline tea: lit parses options from the environment variable LIT_OPTS after parsing options from the command line. LIT_OPTS is primarily useful for supplementing or overriding the command-line options supplied to lit by check targets defined by a project's build system.

She can also read options from response files, which you specify as inputs using the @path/to/file.rsp syntax. Arguments read from a file must be one per line, and they're treated as if they were in the exact same place as the original file-referencing argument on the command line. And yes — a response file can reference other response files. It's response files all the way down, bestie. 🪆

If you're the kind of girlie who's interested in the lit architecture or designing a lit testing implementation, go look at LIT INFRASTRUCTURE. That's your lane. 🧚

🌸 GENERAL OPTIONS 🌸

  • -h, --help — Show the lit help message and exit. The classic "hello, what do you do" move. 🫶
  • --version — Show lit's version number and exit. Gotta know which era of lit you're in. ⭐
  • -j N, --workers N — Run N tests in parallel. By default this is automatically chosen to match the number of detected available CPUs — she'll figure out your whole vibe on her own. 💅
  • --config-prefix NAME — Search for NAME.cfg and NAME.site.cfg when searching for test suites, instead of lit.cfg and lit.site.cfg. Custom naming, custom queen. 👑
  • -D NAME[=VALUE], --param NAME[=VALUE] — Add a user defined parameter NAME with the given VALUE (or the empty string if not given). What these parameters actually mean and how they get used is totally test suite dependent, so ask your config, not me. 🙈

🦋 OUTPUT OPTIONS 🦋

  • -q, --quiet — Suppress any output except for test failures. Silence is golden, unless something breaks, then it's loud. 💀
  • -s, --succinct — Show less output — for example, don't show information on tests that pass. Also shows a progress bar, unless --no-progress-bar is specified. Subtle. Understated. Iconic. ✨
  • -v, --verbose — Show more information on test failures, for example the entire test output instead of just the test result.

Each command is printed before it is executed. This can be valuable for debugging test failures, since the last printed command is the one that failed. Moreover, lit inserts 'RUN: at line N' after each command pipeline in the output to help you locate the source line of the failed command. It's giving detective energy. 🔍

  • -vv, --echo-all-commands — Deprecated alias for -v. She's still here, but she's deprecated bestie, we don't use her anymore. 🥲
  • -a, --show-all — Enable -v, but for all tests, not just failed ones. All the receipts, all the time. 🧾
  • -o PATH, --output PATH — Write test results to the provided path. File her away. 📁
  • --no-progress-bar — Do not use the curses based progress bar. No curses today, thank you. 🚫
  • --min-output-interval INTERVAL — Only output updates to the progress bar and status line at most once per INTERVAL seconds. Has no effect if the curses based progress bar is not used. Rate limiting for your eyeballs. 🎚️
  • --show-excluded — Show excluded tests. Let the exes be seen. 👀
  • --show-skipped — Show skipped tests. 🦘
  • --show-unsupported — Show unsupported tests. 🚧
  • --show-pass — Show passed tests. Celebrate the wins!! 🎉
  • --show-flakypass — Show passed with retry tests. She passed, but she made us nervous first. 😅
  • --show-xfail — Show expectedly failed tests. Failing on purpose is still failing, but make it fashion. 💅

🔥 EXECUTION OPTIONS 🔥

  • --gtest-sharding — Enable sharding for GoogleTest format. 🍰
  • --no-gtest-sharding — Disable sharding for GoogleTest format. 🚫🍰
  • --path PATH — Specify an additional PATH to use when searching for executables in tests. Extend her search radius. 🧭
  • --vg — Run individual tests under valgrind (using the memcheck tool). The --error-exitcode argument for valgrind is used so that valgrind failures will cause the program to exit with a non-zero status.

When this option is enabled, lit will also automatically provide a "valgrind" feature that can be used to conditionally disable (or expect failure in) certain tests. Valgrind said "let me check her" and honestly she did. 🔬

  • --vg-leak — When --vg is used, enable memory leak checks. When this option is enabled, lit will also automatically provide a "vg_leak" feature that can be used to conditionally disable (or expect failure in) certain tests. Drip check, but for memory. 💧
  • --vg-arg ARG — When --vg is used, specify an additional argument to pass to valgrind itself. Give valgrind a little extra something. 🎁
  • --no-execute — Don't execute any tests (assume that they pass). Manifesting a green run, no cap. 🌈
  • --xunit-xml-output XUNIT_XML_OUTPUT — Write XUnit-compatible XML test reports to the specified file. 📝
  • --report-failures-only — Only include failures (see TEST STATUS RESULTS) in the report. We don't talk about the passes. 🙈
  • --resultdb-output RESULTDB_OUTPUT — Write LuCI ResultDB compatible JSON to the specified file. 🧾
  • --time-trace-output TIME_TRACE_OUTPUT — Write Chrome tracing compatible JSON to the specified file. Chrome girlies rise. 🌐
  • --timeout MAXINDIVIDUALTESTTIME — Maximum time to spend running a single test (in seconds). 0 means no time limit. [Default: 0] ⏰
  • --timeout N — Spend at most N seconds (approximately) running each individual test. 0 means no time limit, and 0 is the default. Note that this is not an alias for --max-time — the two are different kinds of maximums. Don't confuse your queens. ⚠️
  • --max-failures MAX_FAILURES — Stop execution after the given number of failures. Set your limit, protect your peace. 🛑
  • --max-retries-per-test N — Retry running failed tests at most N times. Out of the following options to rerun failed tests, the --max-retries-per-test is the only one that doesn't require a change in the test scripts or the test config:
    1. --max-retries-per-test lit option
    2. config.test_retry_attempts test suite option
    3. ALLOW_RETRIES: annotation in test script

Any option in the list above overrules its predecessor. Ranked list, top wins, periodt. 👑

  • --allow-empty-runs — Do not fail the run if all tests are filtered out. Sometimes the vibe check filters everyone out and that's okay. 🫶
  • --per-test-coverage — Emit the necessary test coverage data, divided per test case (involves setting a unique value to LLVM_PROFILE_FILE for each RUN). The coverage data files will be emitted in the directory specified by config.test_exec_root. Coverage queen behaviour fr fr. 📊
  • --ignore-fail — Exit with status zero even if some tests fail. Delulu is the solulu. 💀
  • --skip-test-time-recording — Do not track elapsed wall time for each test. ⏱️
  • --time-tests[=N|all] — Track the wall time individual tests take to execute and include the results in the summary output. This is useful for determining which tests in a test suite take the most time to execute. When enabled, lit prints a slowest-test list and a histogram over all timed tests. The slowest-test list defaults to the 20 slowest tests, but can be limited with =N or expanded to every timed test with =all. The headings report how many tests are listed, for example Slowest Tests (N of M): and Test Times (M):. Name and shame the slow ones. 🐌🔥

🔍 SELECTION OPTIONS 🔍

Okay bestie, so out of the box lit is smart about this: it runs the failing tests first, then does the rest in descending execution-time order so the concurrency gets optimized. But you can absolutely change the execution order with --order if you want to be a different kind of main character.

The timing data lives in test_exec_root in a file named .lit_test_times.txt. If that file isn't there, lit checks test_source_root for it, so clean builds can optionally get a little speed boost. No cap, that's the whole vibe.

  • --max-tests N — Run at most N tests and then bail. That's it, that's the whole thing. 💅
  • --max-time N — Spend at most N seconds (approximately) running tests and then terminate. And real talk: this is not an alias for --timeout — those are two completely different kinds of maximums, don't get it twisted. 🫶
  • --order {lexical,random,smart} — Define the order in which tests are run. The supported values are:
    • lexical — tests get run in lexical order according to the test file path. This option is useful when predictable test order is desired (iykyk, sometimes you just want to know exactly what's coming next).
    • random — tests get run in random order. Chaos, but make it deliberate. 🎲
    • smart — tests that failed previously get run first, then the remaining tests, all in descending execution-time order. This is the default, because it optimizes concurrency. Iconic behavior honestly. 👑
  • --shuffle — Run the tests in a random order, not failing/slowest first. Deprecated, use --order instead. We're retiring her, she served her time. 🙈
  • -i, --incremental — Run failed tests first. (DEPRECATED: use --order=smart — same energy, newer packaging).
  • --filter REGEXP — Run only those tests whose name matches the regular expression specified in REGEXP. The environment variable LIT_FILTER can also be used in place of this option, which is especially useful in environments where the call to lit is issued indirectly (like, when something else is doing the dialing and you can't pass flags). ✨
  • --filter-out REGEXP — Filter out those tests whose name matches the regular expression specified in REGEXP. The environment variable LIT_FILTER_OUT can also be used in place of this option, which is especially useful in environments where the call to lit is issued indirectly. Bye bestie, not you. 🦋
  • --filter-failed — Run only those tests that previously failed. Tests that have been newly added but not yet run are not included. Only the certified ex-failures get a callback. 💀
  • --xfail LIST — Treat those tests whose name is in the semicolon separated list LIST as XFAIL. This can be helpful when one does not want to modify the test suite. The environment variable LIT_XFAIL can also be used in place of this option, which is especially useful in environments where the call to lit is issued indirectly.

A test name can be specified as a file name relative to the test suite directory. For example:

LIT_XFAIL="affinity/kmp-hw-subset.c;offloading/memory_manager.cpp"

In this case, all of the following tests are treated as XFAIL:

libomp :: affinity/kmp-hw-subset.c
libomptarget :: nvptx64-nvidia-cuda :: offloading/memory_manager.cpp
libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp

Alternatively, a test name can be specified as the full test name reported in LIT output. For example, we can adjust the previous example not to treat the nvptx64-nvidia-cuda version of offloading/memory_manager.cpp as XFAIL:

LIT_XFAIL="affinity/kmp-hw-subset.c;libomptarget :: x86_64-pc-linux-gnu :: offloading/memory_manager.cpp"

See, you can be surgical about it. Bougie. 💅

  • --xfail-not LIST — Do not treat the specified tests as XFAIL. The environment variable LIT_XFAIL_NOT can also be used in place of this option. The syntax is the same as for --xfail and LIT_XFAIL. And listen: --xfail-not and LIT_XFAIL_NOT always override all other XFAIL specifications, including an --xfail appearing later on the command line. The primary purpose is to suppress an XPASS result without modifying a test case that uses the XFAIL directive. She has final say, periodt. 👑
  • --unsupported LIST — Treat those tests whose name is in the semicolon separated list LIST as UNSUPPORTED. This can be helpful when one does not want to modify the test suite. The environment variable LIT_UNSUPPORTED can also be used in place of this option, which is especially useful in environments where the call to lit is issued indirectly.

The syntax for specifying test names is the same as for --xfail and LIT_XFAIL. A test name can be specified as a file name relative to the test suite directory or as the full test name reported in LIT output.

  • --unsupported-not LIST — Do not treat the specified tests as UNSUPPORTED. The environment variable LIT_UNSUPPORTED_NOT can also be used in place of this option. The syntax is the same as for --unsupported and LIT_UNSUPPORTED. And same deal as before: --unsupported-not and LIT_UNSUPPORTED_NOT always override all other UNSUPPORTED specifications, including an --unsupported appearing later on the command line. 💜
  • --exclude-xfail — XFAIL tests won't be run, unless they are listed in the --xfail-not (or LIT_XFAIL_NOT) lists. Simple as that, bestie.
  • --num-shards M — Divide the set of selected tests into M equal-sized subsets or "shards", and run only one of them. Must be used with the --run-shard=N option, which selects the shard to run. The environment variable LIT_NUM_SHARDS can also be used in place of this option. These two options provide a coarse mechanism for partitioning large testsuites, for parallel execution on separate machines (say, in a large testing farm — very farm-core, very distributed slay). 🌈
  • --run-shard N — Select which shard to run, assuming the --num-shards=M option was provided. The two options must be used together, and the value of N must be in the range 1..M. The environment variable LIT_RUN_SHARD can also be used in place of this option. Teamwork makes the dream work. 🫶

🛠️ ADDITIONAL OPTIONS 🛠️

  • --debug — Run lit in debug mode, for debugging configuration issues and lit itself. Turning the camera on herself, honestly. 📸
  • --show-suites — List the discovered test suites and exit. Show and tell, but make it infrastructure.
  • --show-tests — List all of the discovered tests and exit. Every single one, no crumbs left behind. ✨
  • --show-used-features — Show all features used in the test suite (in XFAIL, UNSUPPORTED and REQUIRES) and exit. Receipts, basically.
  • --update-tests — Pass failing tests to functions in the lit_config.test_updaters list to check whether any of them know how to update the test to make it pass. Let the updaters try to fix her up, bestie. 🎀

🚪 EXIT STATUS 🚪

lit will exit with an exit code of 1 if there are any failures (see TEST STATUS RESULTS) and --ignore-fail has not been passed. Otherwise, it will exit with the status 0. Other exit codes are used for non-test related failures (for example, a user error or an internal program error). So like — 1 means the tests said no, 0 means we're chilling, and anything else means something else went sideways. 💀

🔎 TEST DISCOVERY 🔎

The inputs passed to lit can be either individual tests, or entire directories or hierarchies of tests to run. When lit starts up, the first thing it does is convert the inputs into a complete list of tests to run as part of test discovery. Like making a guest list before the party, except everyone's invited by default. 📋

In the lit model, every test must exist inside some test suite. lit resolves the inputs specified on the command line to test suites by searching upwards from the input path until it finds a lit.cfg or lit.site.cfg file. These files serve as both a marker of test suites and as configuration files which lit loads in order to understand how to find and run the tests inside the test suite. Two jobs, one outfit. 💅

Once lit has mapped the inputs into test suites it traverses the list of inputs, adding tests for individual files and recursively searching for tests in directories. Very thorough, very main character.

This behavior makes it easy to specify a subset of tests to run, while still allowing the test suite configuration to control exactly how tests are interpreted. In addition, lit always identifies tests by the test suite they are in, and their relative path inside the test suite. For appropriately configured projects, this allows lit to provide convenient and flexible support for out-of-tree builds. We love a flexible icon. 🌸

🏁 TEST STATUS RESULTS 🏁

Each test ultimately produces one of the following eight results. Eight! A whole little zodiac of outcomes:

  • PASS — The test succeeded. The moment. The slay. ✨
  • FLAKYPASS — The test succeeded after being re-run more than once. This only applies to tests containing an ALLOW_RETRIES: annotation. She's a little inconsistent but she got there in the end. 😭
  • XFAIL — The test failed, but that is expected. This is used for test formats which allow specifying that a test does not currently work, but wish to leave it in the test suite. It's giving "known issue, on purpose." 🙈
  • XPASS — The test succeeded, but it was expected to fail. This is used for tests which were specified as expected to fail, but are now succeeding (generally because the feature they test was broken and has been fixed). Plot twist: the failure was the imposter. 👑
  • FAIL — The test failed. No notes. 💀
  • UNRESOLVED — The test result could not be determined. For example, this occurs when the test could not be run, the test itself is invalid, or the test was interrupted. The mystery of it all. 🕵️
  • UNSUPPORTED — The test is not supported in this environment. This is used by test formats which can report unsupported tests. Not her era, not this machine's era. 🦋
  • TIMEOUT — The test was run, but it timed out before it was able to complete. It simply ran out of time, bestie. ⏰

Unresolved (UNRESOLVED), timed out (TIMEOUT), failed (FAIL) and unexpectedly passed (XPASS) tests are considered failures. Yep, even the XPASS — because it was supposed to fail, and now it doesn't, and that's a whole thing.

Depending on the test format, tests may produce additional information about their status (generally only for failures). See the OUTPUT OPTIONS section for more information. 🫶

🏗️ LIT INFRASTRUCTURE 🏗️

This section describes the lit testing architecture for users interested in creating a new lit testing implementation, or extending an existing one. So like, if you want to build your own, this is your blueprint. 🔧

lit proper is primarily an infrastructure for discovering and running arbitrary tests, and to expose a single convenient interface to these tests. lit itself doesn't know how to run tests — rather, this logic is defined by test suites. She's the manager, not the performer. 🎤

📁 TEST SUITES 📁

As described in TEST DISCOVERY, tests are always located inside a test suite. Test suites serve to define the format of the tests they contain, the logic for finding those tests, and any additional information to run the tests.

lit identifies test suites as directories containing lit.cfg or lit.site.cfg files (see also --config-prefix). Test suites are initially discovered by recursively searching up the directory hierarchy for all the input files passed on the command line. You can use --show-suites to display the discovered test suites at startup.

Once a test suite is discovered, its config file is loaded. Config files themselves are Python modules which will be executed. When the config file is executed, two important global variables are predefined. (Well — here's where this part picks up, and lit_config is the one we're meeting right now:)

lit_config

The global lit configuration object (a LitConfig instance), which defines the builtin test formats, global configuration parameters, and other helper routines for implementing test configurations. She's the one holding everything together, fr fr. 👑

🎀 config 🎀

Okay so this one is the config object — it's a TestingConfig instance for the whole test suite, and the config file is literally expected to fill it in. Obsessed. 💅

On top of that, these variables also live on the config object: some of them have to be set by the config, and the rest are either optional or already predefined for you. No cap:

  • name [required] — the name of the test suite, bestie. It shows up in reports and diagnostics, so like… make it good? 👑
  • test_format [required] — the test format object that lit will use to discover and run the tests in this suite. Generally this is gonna be a builtin test format you can grab from the lit.formats module. 🧚
  • test_source_root — the filesystem path to the test suite root. For out-of-dir builds, this is the directory that gets scanned for tests. 🦋
  • test_exec_root — for out-of-dir builds, the path to the test suite root inside the object directory. This is where tests actually get run and where temporary output files get dropped. 🌸
  • environment — a dictionary representing the environment to use when executing tests in the suite. ✨
  • standalone_tests — when true, mark a directory with tests that are expected to be run standalone. Test discovery is disabled for that directory. And heads up: lit.suffixes and lit.excludes must be empty when this variable is true. Fr fr, it's a rule.
  • suffixes — for lit test formats that scan directories for tests, this is a list of suffixes used to identify test files. Used by: ShTest. 🍓
  • substitutions — for lit test formats that substitute variables into a test script, this is the list of substitutions to perform. Used by: ShTest. 💜
  • unsupported — marks a directory as unsupported, and all the tests inside it get reported as unsupported. Used by: ShTest. 😭
  • parent — the parent configuration, i.e. the config object for the directory containing the test suite, or None. 🌈
  • root — the root configuration. This is the top-most lit configuration in the project. Main character energy honestly. 👑
  • pipefail — normally a test using a shell pipe fails if any of the commands on the pipe fail. If that's not what you want, set this variable to false and the test will fail only if the last command in the pipe fails. 🫶
  • available_features — a set of features you can use in XFAIL, REQUIRES, and UNSUPPORTED directives. 💖

✨ TEST DISCOVERY ✨

So once the test suites are located, lit goes full detective mode and recursively traverses the source directory (following test_source_root) looking for tests. 🔍

When lit enters a sub-directory, the first thing it does is check whether a nested test suite is defined in that directory. If there is one — slay — it loads that test suite recursively. If not, it instantiates a local test config for that directory instead (peep LOCAL CONFIGURATION FILES below 👀).

Tests are identified by the test suite they're contained within plus the relative path inside that suite. And ngl this part is kinda important: that relative path might not refer to an actual file on disk. Some test formats (looking at you, GoogleTest) define "virtual tests" whose path contains both the path to the actual test file and a subpath identifying the virtual test. Iconic behavior, honestly. 🦋

🎀 LOCAL CONFIGURATION FILES 🎀

When lit loads a subdirectory in a test suite, it instantiates a local test configuration by cloning the configuration for the parent directory — and the root of this configuration chain will always be a test suite. Periodt.

Once the test configuration is cloned, lit checks for a lit.local.cfg file in the subdirectory. If it's there, that file gets loaded and you can use it to specialize the configuration for each individual directory. 💅

So what's this facility even for? Two big ones:

  • defining subdirectories of optional tests 🍓
  • changing other configuration parameters — for example, changing the test format, or changing the suffixes that identify test files ✨

🎀 SUBSTITUTIONS 🎀

okay so this is where it gets fun — lit lets you substitute patterns right inside your RUN commands, and it also hands you a whole starter pack of base substitutions, all defined in TestRunner.py. bestie, these are your little magic words ✨

Here's the full roster, no skips:

  • %s — source path (the path to the file that's currently being run)
  • %S — source dir (the directory of the file currently being run)
  • %p — same as %S
  • %{pathsep} — path separator
  • %{fs-src-root} — the root component of file system paths that point to the LLVM checkout
  • %{fs-tmp-root} — the root component of file system paths that point to the test's temporary directory
  • %{fs-sep} — file system path separator
  • %t — a path unique to the test (which may be used to make files or directories) 🫶
  • %basename_t — the last path component of %t but without the .tmp extension (deprecated, use %{t:stem} instead) ⚠️
  • %% — literally just %
  • %/s — %s but \ is replaced by /
  • %/S — %S but \ is replaced by /
  • %/p — %p but \ is replaced by /
  • %/t — %t but \ is replaced by /
  • %{s:basename} — the last path component of %s
  • %{s:stem} — the last path component of %s but with the last extension removed
  • %{t:stem} — the last path component of %t but without the .tmp extension (an alias for %basename_t)
  • %{s:real} — %s after expanding all symbolic links and substitute drives
  • %{S:real} — %S after expanding all symbolic links and substitute drives
  • %{p:real} — %p after expanding all symbolic links and substitute drives
  • %{t:real} — %t after expanding all symbolic links and substitute drives
  • %{/s:real} — %/s after expanding all symbolic links and substitute drives
  • %{/S:real} — %/S after expanding all symbolic links and substitute drives
  • %{/p:real} — %/p after expanding all symbolic links and substitute drives
  • %{/t:real} — %/t after expanding all symbolic links and substitute drives
  • %{/s:regex_replacement} — %/s but escaped for use in the replacement of a s@@@ command in sed
  • %{/S:regex_replacement} — %/S but escaped for use in the replacement of a s@@@ command in sed
  • %{/p:regex_replacement} — %/p but escaped for use in the replacement of a s@@@ command in sed
  • %{/t:regex_replacement} — %/t but escaped for use in the replacement of a s@@@ command in sed
  • %:s — on Windows, %/s but a : is removed if it's the second character. Otherwise, %s
  • %:S — on Windows it's %/S but with a : yeeted off if it happens to be the second character 🪟 otherwise it's %S with one single leading / removed
  • %:p — on Windows, %/p with a : removed if it's the second character; otherwise %p but with a single leading / removed
  • %:t — on Windows, %/t with a : removed if it's the second character; otherwise %t but with a single leading / removed
  • %{readfile:<filename>} — literally just reads the file you point it at 📄

Other substitutions exist too, and they're all little variations on this base set, and ngl each test module can define even more substitution patterns on top of that. Peep the modules LOCAL CONFIGURATION FILES section for that tea 🫖

Want the full lore on substitutions? It lives in the LLVM Testing Infrastructure Guide <>.

🦋 TEST RUN OUTPUT FORMAT 🦋

Okay bestie, the lit output for a test run follows this schema — and it's the same in both short and verbose modes (tho in short mode you won't see any PASS lines at all, so don't come for me when they're missing 💀). This schema was picked so a machine can reliably parse it (think buildbot log scraping, iykyk) and so other tools can generate it too. It's giving machine-readable and we love that for us ✨

Every test result is expected to show up on a line that matches:

<result code>: <test name> (<progress info>)

where <result-code> is a standard test result like PASS, FAIL, XFAIL, XPASS, UNRESOLVED, or UNSUPPORTED. And yes, the performance result codes IMPROVED and REGRESSED are allowed too 👑

The <test name> field can be literally any arbitrary string as long as it has no newline in it.

The <progress info> field is there to report progress info like (1/300), or it can just be empty — but even when it's empty the parentheses are required, so do not skip them, I'm begging 🙈

Now, if a test is allowed retries (that's the ALLOW_RETRIES: annotation, go read it) and it needed more than one attempt to succeed, then <progress info> gets extended with this:

, <num_attempts_made> of <max_allowed_attempts> attempts

Each test result can also include extra (multiline) log info, formatted like this:

<log delineator> TEST '(<test name>)' <trailing delineator>
... log message ...
<log delineator>

where <test name> should be the name of a test that was reported earlier, <log delineator> is a string of * characters at least four characters long (the recommended length is 20), and <trailing delineator> is an arbitrary string that doesn't get parsed at all 💅

Here's an example test run output with four tests — A, B, C, and D — plus a log message for the failing test C:

PASS: A (1 of 4)
PASS: B (2 of 4)
FAIL: C (3 of 4)
******************** TEST 'C' FAILED ********************
Test 'C' failed as a result of exit code 1.
********************
PASS: D (4 of 4)

🌸 DEFAULT FEATURES 🌸

For convenience (because lit is a considerate icon), lit automatically adds available_features for a bunch of common use cases 🎀

lit adds a feature based on the operating system it's being built on, for example: system-darwin, system-linux, etc. It also automatically adds a feature based on the current architecture, for example target-x86_64, target-aarch64, etc. 🖥️

And when you're building with sanitizers enabled, lit automatically adds the short name of the sanitizer, for example: asan, tsan, etc.

Wanna see the full list of features that can be added? Go look at llvm/utils/lit/lit/llvm/config.py — it's all in there fr fr.

🧚 LIT EXAMPLE TESTS 🧚

The lit distribution ships with several example implementations of test suites chilling in the ExampleTests directory. Perfect for when you're like "how does any of this work" 😭 — go read them, they ate 💅

👑 SEE ALSO 👑

💜 Author 💜

Maintained by the LLVM Team (https://llvm.org/).

2003-2026, LLVM Project

That's the whole page bestie, you made it to the end 🫶 this was LIT(1), dated 2026-10-20 — now go write some tests, slay, and remember: no cap, the schema is there for a reason 💖✨

📦 where this page comes from