kcbench(1)

📦 version 0.9.15, in fedora 43, fedora 44

💖 kcbench — the Linux kernel compile benchmark, speed edition 💖

okay bestie, welcome to kcbench(1), the manual page for kcbench ✨ — literally the "Linux kernel compile benchmark, speed edition". no cap, that's the whole vibe. 🏎️🔥

✨ SYNOPSIS ✨

kcbench [options]

🌸 DESCRIPTION 🌸

So like, kcbench tries to compile a Linux kernel really quickly to test how fast (or how stable) a system is. that's it. that's the job. 🦋

💡 Note: the number of compile jobs (-j) that gives you the best result depends on the machine being tested. full deets are in ON THE DEFAULT NUMBER OF JOBS below, don't skip it bestie.

now, to get results you can actually compare between different machines, you gotta use the exact same operating system on all of them. periodt. there's a bunch of reasons but the main one is: the Linux version compiled by default depends on the operating system's default compiler, and both of those heavily influence the result. it's giving chaos if you don't. 😭

if you choose to ignore that recommendation (bold move, but okay 🙈), at least hard code the Linux version you're compiling with -s 5.4, because for example compiling 5.7 will take longer than 5.4 or 4.19 and thus lead to results you can't compare. also make sure the compiler used on the systems you want to compare is from similar ones — because like, gcc10 will try harder to optimize the code than gcc8 or gcc9 and thus take more time for its work. the compiler ate, but it also took longer. 💅

kcbench is accompanied by kcbenchrate. both are quite similar but work slightly differently:

  • kcbench tries to build one kernel as fast as possible. this approach is called a 'speed run' and it makes kcbench start multiple jobs in parallel by using make -j #. that way kcbench will use multiple CPU cores most of the time, except during those few phases where the Linux kernel build process is single threaded and thus utilizes just one CPU core. that for example is the case when vmlinux is linked. 😔 single-threaded moments, we hate to see it.
  • kcbenchrate tries to keep all CPU cores busy constantly by starting workers on all of them, where each builds one kernel with just one job (make -j 1). this approach is called a 'rate run'. it takes a lot longer to generate a result than kcbench and also needs a lot more storage space — but it will keep the CPU cores busy all the time. no idle cores, no crumbs. 👑

🎀 Options 🎀

  • -b, --bypass — skip the initial kernel compile that fills caches; saves time, but the first result might be slightly lower than the following ones. 🫶
  • -d, --detailedresults — print more detailed results. give us the lore. 📊
  • -h, --help — show usage.
  • -i, --iterations int — determines the number of kernels that kcbench will compile sequentially with different values of jobs (-j). Default: 2
  • -j, --jobs int(,int, int, ...) — number of jobs to use when compiling a kernel (make -j #).

this option can be given multiple times (-j 2 -j 4 -j 8) or int can be a list (-j "2 4 8"). the default depends on the number of cores in the system and if its processor uses SMT. run --help to query the default on the particular machine. ✨

⚠️ Important note: kcbench on machines with SMTs will do runs which do not utilize all available CPU cores; this might look odd, but there are reasons for this behavior. see ON THE DEFAULT NUMBER OF JOBS below for details. 👀
  • -m, --modconfig — instead of using a config generated with defconfig, use one built by allmodconfig and compile modules as well. takes a lot longer to compile, which is more suitable for machines with a lot of fast CPU cores. 💀 it's giving "beefy machine only".
  • -k, --kconfig file — instead of using a config generated with defconfig or allmodconfig, use file as a base and complete it with olddefconfig. this option overrides a possible -m/--modconfig. 👑
  • -o, --outputdir dir — use path to compile Linux. passes O=dir/kcbench-worker/ to make when calling it to compile a kernel. without this, kcbench will use a temporary directory. 🗂️
  • -s, --src path|version — look for sources in path, ~/.cache/kcbench/linux-version or /usr/share/kcbench/linux-version. if not found, try to download version automatically unless --no-download was specified. 🌐
  • -v, --verbose — increase verbose level; option can be given multiple times. 🗣️
  • -V, --version — output program version.
  • --cc exec — use exec as target compiler. 🔧
  • --cross-compile arch — EXPERIMENTAL: cross compile the Linux kernel. cross compilers for this task are packaged in some Linux distribution. there are also pre-compiled compilers available on the internet, for example here: https://mirrors.edge.kernel.org/pub/tools/crosstool/

values of arch that kcbench understands: arm arm64 aarch64 riscv riscv64 powerpc powerpc64 x86_64

building for archs not directly supported by kcbench should work, too: just export ARCH= and CROSS_COMPILE= just like you would when normally cross compiling a Linux kernel. do not use --cross-compile in that case.

be aware there is a bigger risk running into compile errors (see below) when cross compiling. 😬

keep in mind that unless you provide a build configuration using --kconfig, kcbench will configure the compiled Linux kernel with the make target defconfig (or allmodconfig, if you specify -m). that .config might be unusual for the architecture in question, but good enough for benchmarking purposes. 💁♀️

  • --crosscomp-scheme scheme — on Linux distributions that are known to ship cross compilers, kcbench will assume you want to use those. this parameter allows to specify one of the various different naming schemes in cases this automatic detection fails, or work you want kcbench/kcbenchrate to find them using a generic scheme that should work with compilers from various sources, which is the default on unknown distributions.

valid values of scheme: debian fedora generic redhat ubuntu

  • --hostcc exec — use exec as host compiler. 🛠️
  • --infinite — run endlessly to create system load. 🔥 go off, i guess.
  • --llvm — set LLVM=1 to use clang as compiler and LLVM utilities as GNU binutils substitute. ✨
  • --add-make-args string — pass additional flags found in string to make when creating the config or building the kernel. this option is meant for experts that want to try unusual things, like specifying a special linker (--add-make-args 'LD=ld.lld').
⚠️ use with caution! 🚨
  • --no-download — never download Linux kernel sources from the web automatically. 🙅♀️
  • --savefailedlogs path — save log of failed compile runs to path. 💾

🌸 ON THE DEFAULT NUMBER OF JOBS 🌸

ok so here's the tea bestie: the perfect number of compile jobs (-j) for the best result? it depends entirely on the machine you're benching. 💅 on most systems you'll get the best result when the job count matches the number of CPU cores. like, that's literally the case on this 4-core Intel processor with no SMT:

[cttest@localhost ~]$ bash kcbench -s 5.3 -i 1
Processor:            Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz [4 threads]
Cpufreq; Memory:      Unknown; 15934 MByte RAM
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 4):         250.03 seconds / 14.40 kernels/hour
Run 2 (-j 6):         255.88 seconds / 14.07 kernels/hour

see how the run with 6 jobs was slower here? so kcbench trying that setting by default looks like it's just wasting your time on this box — but ngl it's actually the wise move, because other machines give their best result when they're oversubscribed a lil bit. 🦋 that's exactly what happens on this 6 core/12 threads processor, which hit its best result with 15 jobs:

[cttest@localhost ~]$ bash kcbench -s 5.3 -i 1
Processor:            Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz [12 threads]
Cpufreq; Memory:      Unknown; 15934 MByte RAM
Linux running:        5.6.0-0.rc2.git0.1.vanilla.knurd.2.fc31.x86_64
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 12):        92.55 seconds / 38.90 kernels/hour
Run 2 (-j 15):        91.91 seconds / 39.17 kernels/hour
Run 3 (-j 6):         113.66 seconds / 31.67 kernels/hour
Run 4 (-j 9):         101.32 seconds / 35.53 kernels/hour

you'll clock the attempts that only used the real cores (-j 6) and oversubscribed them a little (-j 9) — which on this machine looks like a total waste of time. but on some machines with SMT capable processors those settings are literally the winners, like on this AMD Threadripper processor with 64 core/128 threads:

$ kcbench
[cttest@localhost ~]$ bash kcbench -s 5.3 -i 1
Processor:            AMD Ryzen Threadripper 3990X 64-Core Processor [128 threads]
Cpufreq; Memory:      Unknown; 15934 MByte RAM
Linux running:        5.6.0-0.rc2.git0.1.vanilla.knurd.2.fc31.x86_64
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 128):       26.16 seconds / 137.61 kernels/hour
Run 2 (-j 136):       26.19 seconds / 137.46 kernels/hour
Run 3 (-j 64):        21.45 seconds / 167.83 kernels/hour
Run 4 (-j 72):        22.68 seconds / 158.73 kernels/hour

and it gets even more obvious when you're compiling an allmodconfig configuration:

[cttest@localhost ~]$ bash kcbench -s 5.3 -i 1 -m
Processor:            AMD Ryzen Threadripper 3990X 64-Core Processor [128 threads]
Cpufreq; Memory:      Unknown; 63736 MByte RAM
Linux running:        5.6.0-0.rc2.git0.1.vanilla.knurd.2.fc31.x86_64
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 128):       260.43 seconds / 13.82 kernels/hour
Run 2 (-j 136):       262.67 seconds / 13.71 kernels/hour
Run 3 (-j 64):        215.54 seconds / 16.70 kernels/hour
Run 4 (-j 72):        215.97 seconds / 16.67 kernels/hour

this can happen if the SMT implementation is bad or if something else becomes a bottleneck. 🔥 a few tests on the machine above indicated the memory interface was the limiting factor. an AMD Epyc from the same processor generation did not show this effect and delivered its best results when the number of jobs matched the number of CPUs:

[cttest@localhost ~]$ bash kcbench -s 5.3 -i 1 -m
Processor:            AMD EPYC 7742 64-Core Processor [256 threads]
Cpufreq; Memory:      Unknown; 63736 MByte RAM
Linux running:        5.6.0-0.rc2.git0.1.vanilla.knurd.2.fc31.x86_64
Compiler used:        gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)
Linux compiled:       5.3.0 [/home/cttest/.cache/kcbench/linux-5.3/]
Config; Environment:  defconfig; CCACHE_DISABLE="1"
Build command:        make vmlinux
Run 1 (-j 256):       128.24 seconds / 28.07 kernels/hour
Run 2 (-j 268):       128.87 seconds / 27.94 kernels/hour
Run 3 (-j 128):       141.83 seconds / 25.38 kernels/hour
Run 4 (-j 140):       137.46 seconds / 26.19 kernels/hour

this table will tell you how many jobs kcbench will use by default:

#                             Cores: Default # of jobs
#                             1 CPU: 1 2
#           2 CPUs (    no SMT    ): 2 3
#           2 CPUs (2 threads/core): 2 3 1
#           4 CPUs (    no SMT    ): 4 6
#           4 CPUs (2 threads/core): 4 6 2
#           6 CPUs (    no SMT    ): 6 9
#           6 CPUs (2 threads/core): 6 9 3
#           8 CPUs (    no SMT    ): 8 11
#           8 CPUs (2 threads/core): 8 11 4 6
#          12 CPUs (    no SMT    ): 12 16
#          12 CPUs (2 threads/core): 12 16 6 9
#          16 CPUs (    no SMT    ): 16 20
#          16 CPUs (2 threads/core): 16 20 8 11
#          20 CPUs (    no SMT    ): 20 25
#          20 CPUs (2 threads/core): 20 25 10 14
#          24 CPUs (    no SMT    ): 24 29
#          24 CPUs (2 threads/core): 24 29 12 16
#          28 CPUs (    no SMT    ): 28 34
#          28 CPUs (2 threads/core): 28 34 14 18
#          32 CPUs (    no SMT    ): 32 38
#          32 CPUs (2 threads/core): 32 38 16 20
#          32 CPUs (4 threads/core): 32 38 8 11
#          48 CPUs (    no SMT    ): 48 55
#          48 CPUs (2 threads/core): 48 55 24 29
#          48 CPUs (4 threads/core): 48 55 12 16
#          64 CPUs (    no SMT    ): 64 72
#          64 CPUs (2 threads/core): 64 72 32 38
#          64 CPUs (4 threads/core): 64 72 16 20
#         128 CPUs (    no SMT    ): 128 140
#         128 CPUs (2 threads/core): 128 140 64 72
#         128 CPUs (4 threads/core): 128 140 32 38
#         256 CPUs (    no SMT    ): 256 272
#         256 CPUs (2 threads/core): 256 272 128 140
#         256 CPUs (4 threads/core): 256 272 64 72

💀 ON FAILED RUNS DUE TO COMPILATION ERRORS 💀

so like, the compilation is unlikely to fail, as long as you're using a settled GCC version to natively compile the source of a current Linux kernel for popular architectures like ARM, ARM64/Aarch64, or x86_64. 🫶 for other cases there's a bigger risk that compilation will fail due to factors outside of kcbench control. they nevertheless try to catch a few common problems and warn, but they can not catch them all, because there are just too many factors involved:

  • brand new compiler generations are sometimes stricter than their predecessors and thus might fail to compile even the latest Linux kernel version. you might need to use a pre-release version of the next Linux kernel release to make it work, or simply need to wait until the compiler or kernel developers solve the problem. 😭
  • distributions enable different compiler features that might have an impact on the kernel compilation. for example GCC9 was capable of compiling Linux 4.19 on many distributions, but started to fail on Ubuntu 19.10 due to a GCC feature the distribution enabled. try compiling a newer Linux kernel version in such a case.
  • cross compilation increases the risk of running into compile problems in general, as there are many compilers and architectures out there. that for example is why compiling the Linux kernel for an unpopular architecture is more likely to fail due to bugs in the compiler or the Linux kernel sources that nobody had noticed before when the compiler or kernel was released. this is even more likely to happen if you start kcbench with -m/--allmodconfig to build a more complex kernel. 🧚

✨ HINTS ✨

running benchmarks so you can compare the results fairly can be tricky, bestie. here are a few of the aspects you should keep in mind when trying to do so:

  • do not compare results from two different architectures (like ARM64 and x86_64): kcbench compiles different code in that case, as they will compile a native kernel on each of those architectures. you can avoid this by cross compiling for a third architecture that is not related to any of the architectures compared (say RISC-V when comparing ARM64 and x86_64). 💅
  • unless you want to benchmark compiler effects, do not compare results from different compiler generations. for example do not compare results from GCC7 and GCC9, as the latter optimizes harder and thus will take more time. newer compiler generations often are also stricter, which is why they often uncover bugs in the Linux kernel source that need to be fixed for compiling to succeed — which is why the Linux version kcbench compiles by default depends on the compiler. 👑
  • compiling a Linux kernel scales very well and thus can utilize processors quite well. but be aware that some parts of the Linux compile process will only use one thread (and thus one CPU core), for example when linking vmlinuz; the other cores idle meanwhile. the effect on the result will grow with the number of CPU cores. 🙈

if you want to work against that, consider using -m to build an allmodconfig configuration with modules; compiling a newer, more complex Linux kernel version can also help. but the best way to avoid this effect is by using kcbenchrate. 🔥

  • kcbench by default sets CCACHE_DISABLE=1 when calling make to avoid interference from ccache.

🎀 EXAMPLES 🎀

to let kcbench decide everything automatically, simply run:

$ kcbench

on a four core processor without SMT, kcbench by default will compile two kernels with 4 jobs and two with 6 jobs. you can specify a setting like this manually: .

:

$ kcbench -s 5.4 --iterations 3 --jobs 2 --jobs 4

this will compile Linux 5.4 first three times with 2 jobs and then as often with 4 jobs. ✨

✨ RESULTS ✨

okay bestie, this is the part where the tea gets spilled — like, the actual numbers you've been waiting for 💅

By default, the lines you're looking for look like this:

Run 1 (-j 4): 230.30 sec / 15.63 kernels/hour [P:389%, 24 maj. pagefaults]

So here it took 230.30 seconds to compile the Linux kernel image. With a speed like this, the machine can compile 15.63 kernels per hour (that's 60*60/230.30, no cap 🧮). The results from this 4 core machine also show the CPU usage (P) was 389 percent, and 24 major page faults occurred during this run — and bestie, that number should be small, because processing those page faults takes some time and thus slows down the build 😭

🫶 heads up: this info is omitted if less than 20 major page faults happen. So if you don't see it, it's not a glitch, it's a vibe.

For details on how the CPU usage is calculated and how major page faults are detected, go read the man page for GNU time — because kcbench literally relies on it for its measurements. iykyk ✨

When you run with -d|--detailedresults you'll get a way more detailed result, like this:

Run 1 (-j 4): 230.30 sec / 15.63 kernels/hour [P:389%]
Elapsed Time(E): 2:30.10 (150.10 seconds)
Kernel time (S): 36.38 seconds
User time (U): 259.51 seconds
CPU usage (P): 197%
Major page faults (F): 0
Minor page faults (R): 9441809
Context switches involuntarily (c): 69031
Context switches voluntarily (w): 46955

look at all those little letters doing their thing 🎀 it's giving data. it's giving transparency. we love to see it.

🦋 MISSING FEATURES 🦋

bestie, here's what this tool doesn't do yet — manifesting it for a future update fr fr 🙈

  • some math to detect the fastest setting and do one more run with it before sanity checking the result and printing the best one, including standard deviation.

🌈 SEE ALSO 🌈

go check those out, they're part of the lore 💜

👑 AUTHOR 👑

Thorsten Leemhuis <linux [AT] leemhuis [DOT] info>

periodt. thank you for your service, icon 🫡💖

and that's the whole page, bestie!! 🎀 the footer is just kcbench(1) — no version, no date, just the name sitting there with main character energy. slay. you're now fully caught up on kcbench(1) and honestly? obsessed. go compile something 🌸💅✨

📦 where this page comes from