bench(n)

💅 bench — Benchmarking/Performance tools 💅

okay so bestie, welcome to bench(n) — the Tcllib page that lives under Benchmarking/Performance tools. This is the whole tea on running benchmark suites and then doing cute little things to the results afterwards. 🌸

---

✨ SYNOPSIS ✨

package require Tcl 8.5 9

package require bench ?0.6?

::bench::locate pattern paths

::bench::run ?option value...? interp_list file...

::bench::versions interp_list

::bench::del bench_result column

::bench::edit bench_result column newvalue

::bench::merge bench_result...

::bench::norm bench_result column

::bench::out::raw bench_result

---

🦋 DESCRIPTION 🦋

So this package is literally the whole toolkit for executing benchmarks written in the bench language, and for processing the results that come out of running them. Two jobs. She does both. 💅

If you're the kind of reader who's here for the bench language itself, you should start with the bench language introduction and then wander on over to the formal bench language specification. That's the main character path. 👑

---

🫶 PUBLIC API 🫶

🌈 BENCHMARK EXECUTION 🌈

• ::bench::locate pattern paths — bestie this one goes hunting for Tcl interpreters and hands you back a list of where they live. It searches through the list of paths you gave it, using the glob pattern you gave it.

The command resolves soft links so it can find the actual executables that match the pattern. and ngl there's a catch: only interpreters that are marked as executable and are actually executable on your current platform make it into the result. no posers allowed. 🚫

• ::bench::run ?option value...? interp_list file... — this is the big one, the main event. It runs all the benchmarks declared in the set of files, once for each Tcl interpreter in interp_list, following whatever configuration the options say, and then returns the accumulated timing results. The shape of that result is described down in Result format, iykyk. 💖

It just assumes your files are written in the bench language. like, that's the vibe. ✨

The options you can pass are:

• -errors flag — flag is a boolean. If it's set, errors inside benchmarks get propagated to the command and that aborts benchmark execution 😭. Otherwise they get recorded in the timing result via a special result code instead. The default is to propagate and abort. so yeah, by default she's dramatic about it.
• -threads n — n is a non-negative integer saying how many threads to use while running the benchmarks. The default value is 0, which means no threads. lowkey single-threaded era. 🧚
• -match pattern — pattern is a glob pattern. Only benchmarks whose description matches it get executed. The default is the empty string, which executes all patterns. everything, all at once. 🌈
• -rmatch pattern — same idea but pattern is a regular expression. Only benchmarks whose description matches get run. Default is the empty string, to execute all patterns. two ways to filter, pick your fighter. 💅
• -iters n — n is a positive integer, the maximal number of iterations for any benchmark. The default is 1000. Individual benchmarks can override this though, they're allowed to be extra. 🎀
• -pkgdir path — path is a path to an existing, readable directory. You can specify multiple paths, just use the option multiple times, once per path. 🌸

If you specify no paths at all, the system behaves exactly as it did before. If you specify one or more paths — say there's N of them — then each of the specified interpreters gets invoked N times, once with each of the specified paths. The chosen path gets put into the interpreters' auto_path, which is what lets it find specific versions of a package. ✨

So basically, -pkgdir is the move when you want to benchmark several different versions of a package against one or more interpreters. iconic behaviour honestly. 👑

Note: the empty string is allowed as a path, and it makes the system run the specified interpreters with an unmodified auto_path. which is handy in case the package in question happens to be available there too. 🫶

• ::bench::versions interp_list — give it a list of Tcl interpreters (identified by their path) and it returns a dictionary mapping each interpreter to its version. Interpreters that aren't actually executable, or that fail when interrogated, just don't make the cut and aren't put into the result. So the result may contain fewer interpreters than you fed in. ngl she's selective. 💀

The command uses the builtin command info patchlevel to figure out the version of each interpreter. periodt.

💜 RESULT MANIPULATION 💜

• ::bench::del bench_result column — removes a column, meaning all benchmark results for one specific Tcl interpreter, from the given benchmark result, and returns the modified result. ✂️

The benchmark results are in the format described in Result format.

The column is identified by an integer number.

• ::bench::edit bench_result column newvalue — renames a column in the given benchmark result and returns the modified result. Which means the path of the Tcl interpreter in the identified column gets changed to an arbitrary string. glow-up for your column name. ✨

The benchmark results are in the format described in Result format.

The column is identified by an integer number.

• ::bench::merge bench_result... — takes one or more benchmark results, smushes them into one big result, and returns that. friendship bracelet energy, but for data. 🫶

All benchmark results are in the format described in Result format.

• ::bench::norm bench_result column — normalizes the timing results in the given benchmark result and returns the modified result. What that means is the cell values are no longer times, they're factors showing how much faster or slower that execution was relative to the baseline. 📊

The baseline it normalizes against is the timing results in the chosen column. Which means after normalization, every value in that column is 1, because those benchmarks are neither faster nor slower than the baseline. obviously. baseline is baseline. 💅

A factor less than 1 means that benchmark was faster than the baseline. A factor greater than 1 means a slower execution. 🐢🐇

The benchmark results are in the format described in Result format.

The column is identified by an integer number.

• ::bench::out::raw bench_result — formats the given benchmark result for output to a file, a socket, whatever. And this specific command does no formatting at all — it passes the input through unchanged. she's just a conduit. 🧚

For other formatting styles, check out the packages bench::out::text and bench::out::csv, which format benchmark results for human consumption, or as CSV data importable by spreadsheets, respectively. 🍓

And complementary to that, if you want to read benchmark results from files, sockets, etc., go look for the package bench::in and the commands it provides. 💖

🎀 RESULT FORMAT 🎀

After a set of benchmarks runs, the raw result this package returns is a Tcl dictionary holding all the relevant info. It's a compact representation, aka a serialization, of a 2-dimensional table that has Tcl interpreters as columns and benchmarks as rows. The cells hold the timing results. The Tcl interpreters / columns are identified by their paths. The benchmarks / rows are identified by their description. it's giving spreadsheet but make it Tcl. 📊💅

The possible keys are all valid Tcl lists of two or three elements, and they look like this:

• {interp *} — the set of keys matching this glob pattern capture the info about all the Tcl interpreters used to run the benchmarks. The second element of the key is the path to the interpreter.

The associated value is the version of that Tcl interpreter.

• {desc *} — the set of keys matching this glob pattern capture the info about all the benchmarks found in the executed benchmark suite. The second element of the key is the description of the benchmark, which has to be unique.

The associated value is irrelevant and is set to the empty string. yeah. irrelevant. 💀

• {usec * *} — the set of keys matching this glob pattern capture the performance info, aka the timing results. The second element of the key is the description of the benchmark, the third element is the path of the Tcl interpreter used to run it.

The associated value is either one of several special result codes, or the time it took to execute the benchmark, in microseconds. The possible special result codes are:

• ERR — the benchmark could not be executed, it failed with a Tcl error. 😭
• BAD_RES — the benchmark could be executed, however the result from its body did not match the declared expectations. 🙈

---

😭 BUGS, IDEAS, FEEDBACK 😭

Bestie. This document, and the package it describes, will undoubtedly contain bugs and other problems. it's giving humble. Please report them in the category bench of the Tcllib Trackers at [http://core.tcl.tk/tcllib/reportlist]. Also please report any ideas for enhancements you may have for either the package and/or the documentation. we want them fr fr. 🫶

When proposing code changes, please provide unified diffs, i.e. the output of diff -u.

Note further that attachments are strongly preferred over inlined patches. You can make attachments by going to the Edit form of the ticket immediately after it's created, and then using the left-most button in the secondary navigation bar. that's the one. 💅

---

🌸 SEE ALSO 🌸

bench_intro, bench_lang_intro, bench_lang_spec, bench_read, bench_wcsv, bench_wtext

---

⭐ KEYWORDS ⭐

benchmark, merging, normalization, performance, testing

---

👑 CATEGORY 👑

Benchmark tools

---

💖 COPYRIGHT 💖

Copyright (c) 2007-2008 Andreas Kupries <andreas_kupries@users.sourceforge.net>

---

and that's the whole page bestie!! 🎀 this one ships as part of tcllib, version 0.6 of bench(n) — so go forth, benchmark responsibly, and remember: factor less than 1 means faster, periodt. 🦋✨💜