FUTHARK-TEST(1)

📦
version 0.25.32 · 5 versions
, in ubuntu resolute, ubuntu stonking

💅 FUTHARK-TEST(1) — the Futhark manual, bestie edition 🎀

okay so. this is futhark-test, from the Futhark manual. it's a whole vibe. let's go ✨

✨ NAME ✨

futhark-test — test Futhark programs 🧪

🌸 SYNOPSIS 🌸

futhark test [options...] infiles...

🦋 DESCRIPTION 🦋

so basically, futhark-test tests Futhark programs based on input/output datasets. every single .fut file inside a directory you point it at? considered, bestie. all of them. 💖 by default, tests run against compiled code. you can switch that up with the -i option if you want.

a Futhark test program is just an ordinary Futhark program, except it has at least one test block describing input/output test cases and possibly other options. the last line must end in a newline (not me being picky, it's literally in the spec 🙈). a test block is commented-out text with this overall format:

description
==
cases...

the description is an arbitrary (possibly multiline) human-readable explanation of the test program. it gets separated from the test cases by a line containing just ==. any comment starting at the beginning of the line and containing a line consisting of just == counts as a test block. the format of a test case is:

[tags { tags... }]
[entry: names...]
["name..."] [ tags... ]
random input { types ... } | script input { ... } | input ({ values... } | @ filename)
output { values... } | auto output | error: regex

if a test case begins with a quoted string, that string is reported as the dataset name, including in the JSON file produced by futhark-bench. if no name is provided, one gets automatically generated. and the name must be unique across all test cases — no dupe names, periodt. 👑

if compiled is present before the input keyword, this test case will never be passed to the interpreter. super useful for test cases that are annoyingly slow to interpret (we've all been there 😭). the nobench keyword is for data sets that are too small to be worth benchmarking, and it only has meaning to futhark-bench.

if input is preceded by random, the text between the curly braces must be a sequence of Futhark types, including sizes in the case of arrays. when futhark test is run, a file located in a data/ subdirectory containing values of the indicated types and shapes is automatically constructed with futhark-dataset. apart from sizes, integer constants (with or without type suffix) and floating-point constants (always with type suffix) are also permitted. cute, right? 💅

if input is preceded by script, the text between the curly braces is interpreted as a FutharkScript expression (peep futhark-literate), which is executed to generate the input. it must use only functions explicitly declared as entry points. if the expression produces an n-element tuple, it will be unpacked and its components passed as n distinct arguments to the test function. the only builtin functions supported are $loaddata and $loadbytes. that's the whole list, don't come for me 🫶

if input is followed by an @ and a file name (which must not contain any whitespace) instead of curly braces, values will be read from the indicated file. this is recommended for large data sets. this notation cannot be used with random input. with script input, the file contents will be interpreted as a FutharkScript expression.

after the input block, the expected result of the test case is written as either output followed by another block of values, or error: followed by a regex indicating an expected run-time error. if neither output nor error is given, the program is expected to execute successfully, but its output will not be validated. (free real estate, no checks 🍓)

if output is preceded by auto (as in auto output), the expected values are automatically generated by compiling the program with futhark c and recording its result for the given input — which must not fail. this is usually only useful for testing or benchmarking alternative compilers, and not for testing the correctness of Futhark programs. also it currently does not work for script inputs. noted. 📝

alternatively, instead of input-output pairs, the test cases can simply be a description of an expected compile time type error:

error: regex

this is used to test the type checker. ✨

an optional tags specification is permitted in the first test block. this section can contain arbitrary tags that classify the benchmark:

tags { names... }

tags are sequences of alphanumeric characters, dashes, and underscores, with each tag separated by whitespace. any program with the disable tag is ignored by futhark test. bye bestie 👋

another optional directive is entry, which specifies the entry point to be used for testing. this is useful for writing programs that test libraries with multiple entry points. multiple entry points can be specified on the same line by separating them with space, and they will all be tested with the same input/output pairs. the entry directive affects subsequent input-output pairs in the same comment block, and may only be present immediately preceding these input-output pairs. if no entry is given, main is assumed. see below for an example 👇

for many usage examples, see the tests directory in the Futhark source directory. a simple example can be found in EXAMPLES below.

💜 VALUES 💜

the notation used for specifying values in input and output blocks is a small subset of Futhark syntax.

tuple syntax is not supported. instead, you can write an N-tuple as its constituent N values. beware of syntax errors in the values — the errors reported by futhark test are very poor 😭. when comparing the output of an entry point with the values in an output block, the tuple is implicitly unpacked and compared to the expected values.

the element type of an array value is determined by the first primitive element of the array. you therefore only need to provide a type suffix on the first element. if no suffix is provided, integer arrays have element type i32 and decimal arrays have type f64. iykyk 💅

🔥 OPTIONS 🔥

  • --backend=program — the backend used when compiling Futhark programs (without leading futhark, e.g. just opencl).
  • --cache-extension=EXTENSION — for a program foo.fut, pass --cache-file foo.fut.EXTENSION. by default, --cache-file is not passed.
  • -c — only run compiled code, do not run the interpreter. this is the default, so like, it's the main character. 👑
  • -C — compile the programs, but do not run them.
  • --concurrency=NUM — the number of tests to run concurrently. defaults to the number of (hyper-)cores available.
  • --exclude=tag — do not run test cases that contain the given tag. cases marked with "notest", "disable", or "no_foo" (where foo is the backend used) are ignored by default.
  • -i — test with the interpreter.
  • -I — pass the program through the compiler frontend, but do not run them. this is only useful for testing the Futhark compiler itself.
  • -t — type-check the programs, but do not run them.
  • -s — run structure tests. these are not run by default. when this option is passed, no other testing is done.
  • --futhark=program — the program used to perform operations (eg. compilation). defaults to the binary running futhark test itself.
  • --no-terminal — change the output format to be suitable for noninteractive terminals. prints a status message roughly every minute. 🕐
  • --no-tuning — do not look for tuning files.
  • --pass-option=opt — pass an option to benchmark programs that are being run. for example, we might want to run OpenCL programs on a specific device:
futhark test prog.fut --backend=opencl --pass-option=-dHawaii
  • --pass-compiler-option=opt — pass an extra option to the compiler when compiling the programs.
  • --runner=program — if set to a non-empty string, compiled programs are not run directly, but instead the indicated program is run with its first argument being the path to the compiled Futhark program. this is useful for compilation targets that cannot be executed directly (as with futhark-pyopencl on some platforms), or when you wish to run the program on a remote machine. 🌈
  • --tuning=EXTENSION — for each program being run, look for a tuning file with this extension, which is suffixed to the name of the program. for example, given --tuning=tuning (the default), the program foo.fut will be passed the tuning file foo.fut.tuning if it exists.

🌈 ENVIRONMENT VARIABLES 🌈

  • TMPDIR — directory used for temporary files such as gunzipped datasets and log files.

🍓 EXAMPLES 🍓

the following program tests simple indexing and bounds checking:

-- Test simple indexing of an array.
-- ==
-- tags { firsttag secondtag }
-- input { [4,3,2,1] 1i64 }
-- output { 3 }
-- input { [4,3,2,1] 5i64 }
-- error: Error*

let main (a: []i32) (i: i64): i32 =
  a[i]

the following program contains two entry points, both of which are tested:

let add (x: i32) (y: i32): i32 = x + y

-- Test the add1 function.
-- ==
-- entry: add1
-- input { 1 } output { 2 }

entry add1 (x: i32): i32 = add x 1

-- Test the sub1 function.
-- ==
-- entry: sub1
-- input { 1 } output { 0 }

entry sub1 (x: i32): i32 = add x (-1)

the following program contains an entry point that is tested with randomly generated data:

-- ==
-- random input { [100]i32 [100]i32 } auto output
-- random input { [1000]i32 [1000]i32 } auto output

let main xs ys = i32.product (map2 (*) xs ys)

👀 SEE ALSO 👀

futhark-bench, futhark-repl

2013-2020, DIKU, University of Copenhagen

okay that's the whole page, bestie, you're officially a futhark-test expert now 💖 this was version 0.25.32, dated Mar 11, 2026, so if you're reading this from the future... hi 🫶 slay, go test some Futhark 💅✨

📦 where this page comes from