UBENCH(1)

UBENCH(1) 💅✨

okay bestie, welcome to UBENCH(1) — the manual page for the Unix benchmark suite that's about to give your system a full-on vibe check. All the tests are executed using the "ubench" script in the top−level directory.

💖 DESCRIPTION 💖

The simplest way to generate results is with the commmand:

ubench

This will run a standard "index" test (see "The BYTE Index" below), and save the report in the "results" directory, with a filename like hostname−2007−09−23−01. An HTML version is also saved.

If you want to generate both the basic system index and the graphics index, then do:

ubench gindex

If your system has more than one CPU, the tests will be run twice −− once with a single copy of each test running at once, and once with N copies, where N is the number of CPUs. Some categories of tests, however (currently the graphics tests) will only run with a single copy. Since the tests are based on constant time (variable work), a "system" run usually takes about 29 minutes; the "graphics" part about 18 minutes. A "gindex" run on a dual−core machine will do 2 "system" passes (single-processing and dual-processing) and one "graphics" run, for a total around one and a quarter hours.

✨ DETAILED USAGE ✨

The ubench script takes a number of options which you can use to customise a test, and you can specify the names of the tests to run. The full usage is:

ubench [ −q | −b | −v ] [−i <n> ] [−c <n> [−c <n> ...]] [test ...]

The option flags are:

• −q — ubench in quiet mode.
• −b — ubench in brief mode, do not display mass of cpu info details.
• −v — ubench in verbose mode.
• −i <count> — ubench <count> iterations for each test −− slower tests use <count> / 3, but at least 1. Defaults to 10 (3 for slow tests).
• −c <n> — ubench <n> copies of each test in parallel. The −c option can be given multiple times; for example: ubench −c 1 −c 4 will run a single−streamed pass, then a 4−streamed pass. Note that some tests (currently the graphics tests) will only run in a single−streamed pass.

The remaining non−flag arguments are taken to be the names of tests to run. The default is to run "index". See "Tests" below.

When running the tests, I do not recommend switching to single−user mode ("init 1"). This seems to change the results in ways I don't understand, and it's not realistic (unless your system will actually be running in this mode, of course). However, if using a windowing system, you may want to switch to a minimal window setup (for example, log in to a "twm" session), so that randomly−churning background processes don't randomise the results too much. This is particularly true for the graphics tests.

Output can be specified by setting the following environment variables:

• UB_RESULTDIR — Absolute path of output directory of result files.
• UB_TMPDIR — Absolute path of temporary files for IO tests.
• UB_OUTPUT_FILE_NAME — Output file name. If exists it will be overwritten.
• UB_OUTPUT_CSV — If set "true", output results(score only) to .csv.

🧸 TESTS 🧸

The available tests are organised into categories; when generating index scores (see "The BYTE Index" below) the results for each category are produced separately. The categories are:

• system — The original Unix system tests (not all are actually in the index)
• 2d — 2D graphics tests (not all are actually in the index)
• 3d — 3D graphics tests
• misc — Various non−indexed tests

The following individual tests are available:

system

• dhry2reg — Dhrystone 2 using register variables
• whetstone−double — Double−Precision Whetstone
• syscall — System Call Overhead
• pipe — Pipe Throughput
• context1 — Pipe−based Context Switching
• spawn — Process Creation
• execl — Execl Throughput
• fstime−w — File Write 1024 bufsize 2000 maxblocks
• fstime−r — File Read 1024 bufsize 2000 maxblocks
• fstime — File Copy 1024 bufsize 2000 maxblocks
• fsbuffer−w — File Write 256 bufsize 500 maxblocks
• fsbuffer−r — File Read 256 bufsize 500 maxblocks
• fsbuffer — File Copy 256 bufsize 500 maxblocks
• fsdisk−w — File Write 4096 bufsize 8000 maxblocks
• fsdisk−r — File Read 4096 bufsize 8000 maxblocks
• fsdisk — File Copy 4096 bufsize 8000 maxblocks
• shell1 — Shell Scripts (1 concurrent) (runs "looper 60 multi.sh 1")
• shell8 — Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 8")
• shell16 — Shell Scripts (16 concurrent)(runs "looper 60 multi.sh 16")

Environment variable MULTI_SH_WORK_FACTOR (default 1) can be set to multiply the size of test input data (~8k). Note: changing MULTI_SH_WORK_FACTOR modifies the test. However, modifying the user/kernel workload balance may be useful for comparison with other systems on which the benchmark was run using the same MULTI_SH_WORK_FACTOR.

2d

• 2d−rects — 2D graphics: rectangles
• 2d−lines — 2D graphics: lines
• 2d−circle — 2D graphics: circles
• 2d−ellipse — 2D graphics: ellipses
• 2d−shapes — 2D graphics: polygons
• 2d−aashapes — 2D graphics: aa polygons
• 2d−polys — 2D graphics: complex polygons
• 2d−text — 2D graphics: text
• 2d−blit — 2D graphics: images and blits
• 2d−window — 2D graphics: windows

3d

• ubgears — 3D graphics: gears

misc

• C — C Compiler Throughput ("looper 60 $cCompiler cctest.c")
• arithoh — Arithoh (huh?)
• short — Arithmetic Test (short) (this is arith.c configured for "short" variables; ditto for the ones below)
• int — Arithmetic Test (int)
• long — Arithmetic Test (long)
• float — Arithmetic Test (float)
• double — Arithmetic Test (double)
• dc — Dc: sqrt(2) to 99 decimal places (runs "looper 30 dc < dc.dat", using your system's copy of "dc")
• hanoi — Recursion Test −− Tower of Hanoi
• grep — Grep for a string in a large file, using your system's copy of "grep"
• sysexec — Exercise fork() and exec().

The following pseudo−test names are aliases for combinations of other tests:

• arithmetic — Runs arithoh, short, int, long, float, double, and whetstone−double
• dhry — Alias for dhry2reg
• dhrystone — Alias for dhry2reg
• whets — Alias for whetstone−double
• whetstone — Alias for whetstone−double
• load — Runs shell1, shell8, and shell16
• misc — Runs C, dc, and hanoi
• speed — Runs the arithmetic and system groups
• oldsystem — Runs execl, fstime, fsbuffer, fsdisk, pipe, context1, spawn, and syscall
• system — Runs oldsystem plus shell1, shell8, and shell16
• fs — Runs fstime−w, fstime−r, fstime, fsbuffer−w, fsbuffer−r, fsbuffer, fsdisk−w, fsdisk−r, and fsdisk
• shell — Runs shell1, shell8, and shell16
• index — Runs the tests which constitute the official index: the oldsystem group, plus dhry2reg, whetstone−double, shell1, and shell8 See "The BYTE Index" below for more information.
• graphics — Runs the tests which constitute the graphics index: 2d−rects, 2d−ellipse, 2d−aashapes, 2d−text, 2d−blit, 2d−window, and ubgears
• gindex — Runs the index and graphics groups, to generate both sets of index results
• all — Runs all tests

👑 The BYTE Index 👑

The purpose of this test is to provide a basic indicator of the performance of a Unix−like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system.

Since 1995, the baseline system has been "George", a SPARCstation 20−61 with 128 MB RAM, a SPARC Storage Array, and Solaris 2.3, whose ratings were set at 10.0. (So a system which scores 520 is 52 times faster than this machine.) Since the numbers are really only useful in a relative sense, there's no particular reason to update the base system, so for the sake of consistency it's probably best to leave it alone. George's scores are in the file "pgms/index.base"; this file is used to calculate the index scores for any particular run.

Over the years, various changes have been made to the set of tests in the index. Although there is a desire for a consistent baseline, various tests have been determined to be misleading, and have been removed; and a few alternatives have been added. These changes are detailed in the README, and should be born in mind when looking at old scores.

A number of tests are included in the benchmark suite which are not part of the index, for various reasons; these tests can of course be run manually. See "Tests" above.

🌈 Graphics Benchmarks 🌈

As of version 5.1, UnixBench now contains some graphics benchmarks. These are intended to give a rough idea of the general graphics performance of a system. The graphics tests are in categories "2d" and "3d", so the index scores for these tests are separate from the basic system index. This seems like a sensible division, since the graphics performance of a system depends largely on the graphics adaptor.

The tests currently consist of some 2D "x11perf" tests and "ubgears". The 2D tests are a selection of the x11perf tests, using the host system's x11perf command (which must be installed and in the search path). Only a few of the x11perf tests are used, in the interests of completing a test run in a reasonable time; if you want to do detailed diagnosis of an X server or graphics chip, then use x11perf directly.

The 3D test is "ubgears", a modified version of the familiar "glxgears". This version runs for 5 seconds to "warm up", then performs a timed run and displays the average frames−per−second.

On multi−CPU systems, the graphics tests will only run in single−processing mode. This is because the meaning of running two copies of a test at once is dubious; and the test windows tend to overlay each other, meaning that the window behind isn't actually doing any work.

💜 Multiple CPUs 💜

If your system has multiple CPUs, the default behaviour is to run the selected tests twice −− once with one copy of each test program running at a time, and once with N copies, where N is the number of CPUs. (You can override this with the "−c" option; see "Detailed Usage" above.) This is designed to allow you to assess:

• the performance of your system when running a single task
• the performance of your system when running multiple tasks
• the gain from your system's implementation of parallel processing

The results, however, need to be handled with care. Here are the results of two runs on a dual−processor system, one in single−processing mode, one dual−processing. The columns are Test, Single, Dual, and Gain.

• Dhrystone 2 — Single 562.5, Dual 1110.3, Gain 97%
• Double Whetstone — Single 320.0, Dual 640.4, Gain 100%
• Execl Throughput — Single 450.4, Dual 880.3, Gain 95%
• File Copy 1024 — Single 759.4, Dual 595.9, Gain −22%
• File Copy 256 — Single 535.8, Dual 438.8, Gain −18%
• File Copy 4096 — Single 1261.8, Dual 1043.4, Gain −17%
• Pipe Throughput — Single 481.0, Dual 979.3, Gain 104%
• Pipe−based Switching — Single 326.8, Dual 1229.0, Gain 276%
• Process Creation — Single 917.2, Dual 1714.1, Gain 87%
• Shell Scripts (1) — Single 1064.9, Dual 1566.3, Gain 47%
• Shell Scripts (8) — Single 1567.7, Dual 1709.9, Gain 9%
• System Call Overhead — Single 944.2, Dual 1445.5, Gain 53%
• Index Score: — Single 678.2, Dual 1026.2, Gain 51%

As expected, the heavily CPU−dependent tasks −− dhrystone, whetstone, execl, pipe throughput, process creation −− show close to 100% gain when running 2 copies in parallel. The Pipe−based Context Switching test measures context switching overhead by sending messages back and forth between 2 processes. I don't know why it shows such a huge gain with 2 copies (ie. 4 processes total) running, but it seems to be consistent on my system. I think this may be an issue with the SMP implementation. The System Call Overhead shows a lesser gain, presumably because it uses a lot of CPU time in single−threaded kernel code. The shell scripts test with 8 concur-

so like, the shell8 test with 8 concurrent processes shows, like, zero gain — because the test itself already runs 8 scripts in parallel, so it's already eating both CPUs even when the benchmark runs in single-stream mode. The same test with one process per copy? Real gain. Slay. 💅

The filesystem throughput tests actually show a loss instead of a gain when multi-processing. And that's honestly to be expected — those tests are presumably constrained by the throughput of the I/O subsystem and the disk drive itself. The drop in performance is presumably down to the increased contention for resources, and maybe greater disk head movement. Not the vibe, but physics is physics. 💀

---

So what tests should you use, how many copies should you run, and how should you interpret the results? Bestie, that's up to you — it totally depends on what it is you're trying to measure. 🫶

The multi-processing mode is implemented at the level of test iterations. During each iteration of a test, N slave processes are started using fork(). Each of these slaves executes the test program using fork() and exec(), reads and stores the entire output, times the run, and prints all the results to a pipe. The ubench script reads the pipes for each of the slaves in turn to get the results and times. The scores are added, and the times averaged. 🌸

The result? Each test program has N copies running at once. They should all finish at around the same time, since they run for constant time. And if a test program itself starts off K multiple processes (as with the shell8 test), then the effect will be that there are N * K processes running at once. Which is probably not very useful for testing multi-CPU performance. 🙈

---

---

The $LANG environment variable determines how programs and library routines interpret text. This can have a big impact on the test results. 👑

• If $LANG is set to POSIX, or is left unset, text is treated as ASCII.
• If it is set to en_US.UTF-8, for example, then text is treated as being encoded in UTF-8, which is more complex and therefore slower.
• Setting it to other languages can have varying results.

To ensure consistency between test runs, the ubench script now (as of version 5.1.1) sets $LANG to "en_US.utf8". This setting is configured with the variable "$language". You should not change this if you want to share your results to allow comparisons between systems — however, you may want to change it to see how different language settings affect performance. Each test report now includes the language settings in use. 📝

Lowkey important: the reported language is what is set in $LANG, and is not necessarily supported by the system. But we also report the character mapping and collation order which are actually in use (as reported by "locale").

---

---

Interpreting the results of these tests is tricky, and totally depends on what you're trying to measure. Like, are you trying to measure how fast your CPU is? Or how good your compiler is? 🤔

Because these tests are all recompiled using your host system's compiler, the performance of the compiler will inevitably impact the performance of the tests. Is this a problem? Well — if you're choosing a system, you probably care about its overall speed, which may well depend on how good its compiler is; so including that in the test results may be the right answer. But you may want to ensure that the right compiler is used to build the tests. 🧚

On the other hand, with the vast majority of Unix systems being x86 / PC compatibles, running Linux and the GNU C compiler, the results will tend to be more dependent on the hardware; but the versions of the compiler and OS can make a big difference. (I measured a 50% gain between SUSE 10.1 and OpenSUSE 10.2 on the same machine.) So you may want to make sure that all your test systems are running the same version of the OS; or at least publish the OS and compiler versions with your results. Then again, it may be compiler performance that you're interested in. ✨

The C test is very dubious — it tests the speed of compilation. If you're running the exact same compiler on each system, OK; but otherwise, the results should probably be discarded. A slower compilation doesn't say anything about the speed of your system, since the compiler may simply be spending more time to super-optimise the code, which would actually make it faster. 🚀

This will be particularly true on architectures like IA-64 (Itanium etc.) where the compiler spends huge amounts of effort scheduling instructions to run in parallel, with a resultant significant gain in execution speed.

Some tests are even more dubious in terms of host-dependency — for example, the "dc" test uses the host's version of dc (a calculator program). The version of this which is available can make a huge difference to the score, which is why it's not in the index group. Read through the release notes for more on these kinds of issues. 📚

Another age-old issue is that of the benchmarks being too trivial to be meaningful. With compilers getting ever smarter, and performing more wide-ranging flow path analyses, the danger of parts of the benchmarks simply being optimised out of existence is always present. 😭

All in all, the "index" and "gindex" tests (see above) are designed to give a reasonable measure of overall system performance; but the results of any test run should always be used with care. 💖

---

okay that's the whole ubench(1) page, bestie — the source/version and date live down in the footer of the real page, so go peep those there before you cite anything. mwah 💋 you're gonna crush those benchmarks fr fr 🌈👑