ZSTD(1)

📦
version 1.5.7 · 11 versions
, in opensuse-leap 16.0

💖 NAME 💖

Okay bestie, this is ZSTD(1) from the User Commands section, and it's giving pure compression energy 🎀

zstd - zstd, zstdmt, unzstd, zstdcat - Compress or decompress .zst files ✨

🎀 SYNOPSIS 🎀

So there are a few lil aliases here, and they're all just zstd in disguise 🧚

zstdmt is equivalent to zstd -T0
unzstd is equivalent to zstd -d
zstdcat is equivalent to zstd -dcf

✨ DESCRIPTION ✨

okay so zstd is a fast lossless compression algorithm and data compression tool 💅 Its command line vibes are super similar to gzip(1) and xz(1). It's based on the LZ77 family, with extra FSE & huff0 entropy stages on top 🤓 It's also highly configurable, so you can go from fast modes at > 200 MB/s per core all the way up to strong modes with excellent compression ratios. And the decoder is literally so fast, with speeds > 500 MB/s per core, that it stays roughly stable no matter what compression settings you picked 😭🔥

zstd's command line is generally like gzip's, but it has a few lowkey differences 👑

  • Source files are preserved by default. If you want them removed automatically, use --rm 🗑️
  • When compressing a single file, zstd shows progress notifications and a result summary by default. Add -q to turn them off 🤫
  • zstd shows a short help page when your command line is an error. Use -q if you don't want that 🙈
  • zstd does not accept input from the console, but it does accept stdin as long as stdin isn't the console 🎧
  • zstd does not store the input's filename or attributes, only its contents 💜

zstd processes each file based on the operation mode you picked. If no files are given, or the file is -, zstd reads from standard input and writes the processed data to standard output 🦋 Now here's the big rule: zstd will refuse to write compressed data to standard output if it's a terminal. It'll show an error message and skip that file. Same deal for reading compressed data from standard input: if it's a terminal, zstd won't do it 🙅‍♀️

Unless you use --stdout or -o, output files get written to a new file whose name is built from the source file name:

  • When compressing, the suffix .zst gets added to the source filename to make the target filename ✨
  • When decompressing, the .zst suffix gets removed from the source filename to make the target filename 🍓

🧵 Concatenation with .zst Files

It's possible to concatenate multiple .zst files together 💖 zstd will decompress that agglomerated file as if it were one single .zst file. Iconic 😭

✨ OPTIONS ✨

Integer Suffixes and Special Values

Wherever an integer argument is expected, you can add an optional suffix so you don't have to type out huge numbers 🧮 Just make sure there's no space between the integer and the suffix.

  • KiB — multiplies the integer by 1,024 (2\^10). Ki, K, and KB work as synonyms for KiB 🌸
  • MiB — multiplies the integer by 1,048,576 (2\^20). Mi, M, and MB are accepted as synonyms for MiB 💅

Operation Mode

If you give multiple operation mode options, the last one wins 👑

  • -z, --compress — compress! This is the default when you don't pick an operation mode and the command name doesn't imply one. (For example, unzstd implies --decompress, so it's already got that covered 🧚)
  • -d, --decompress, --uncompress — decompress, obviously 🔓
  • -t, --test — tests the integrity of compressed files. This is equivalent to --decompress --stdout > /dev/null, so the decompressed data gets thrown away but still gets checksummed for errors. No files get created or removed, so it's a totally safe vibe check 🩺
  • -b# — benchmarks file(s) using compression level #. Check BENCHMARK below for the full scoop 📊
  • --train FILES — uses FILES as a training set to build a dictionary. Your training set should have a lot of small files (> 100) 🧠 Check DICTIONARY BUILDER below for how this works.
  • -l, --list — displays info about a zstd compressed file, like its size, ratio, and checksum. Some of these fields might not be available, and you can add the -v modifier to get even more info 🔍

Operation Modifiers

  • -# — picks the compression level #, from [1-19] (default: 3) 🎚️ Higher levels usually give you a higher compression ratio, but it costs speed and memory. A rough rule of thumb is that compression speed gets divided by 2 about every 2 levels 📉 Under the hood, each level maps to a set of advanced parameters (which you can also tweak one by one, see below). Because the compressor's behavior depends a lot on the content it's squishing, you shouldn't expect a perfectly smooth progression from one level to the next 🤷‍♀️
  • --ultra — unlocks the high compression levels 20+ (max 22) 🚀 These use a lot more memory, and decompression also needs more memory when you use them. Plan accordingly, bestie 💀
  • --fast[=#] — switches to ultra-fast compression levels ⚡ If =# isn't there, it defaults to 1. The higher the value, the faster it goes, but you pay for it with some compression ratio. This overwrites a compression level if you set one earlier, and if you set a compression level after --fast, that one overrides it instead 🔄
  • -T#, --threads=# — compresses using # working threads (default: 1) 🧵 If # is 0, zstd tries to detect and use your number of physical CPU cores. In every case, the thread count gets capped to ZSTDMT_NBWORKERS_MAX, which is 64 in 32-bit mode or 256 for 64-bit environments. This does nothing if zstd was compiled without multithread support 🙃
  • --single-thread — uses one single thread for both I/O and compression 🧍‍♀️ Since compression is serialized with I/O, this can be slightly slower. The upside is that single-thread mode uses significantly less memory, which is great for systems with limited memory, like 32-bit systems 🌙
    • Note 1: this mode is the only one available when multithread support is disabled.
    • Note 2: this mode is different from -T1, which spawns 1 compression thread running in parallel with I/O. The final compressed result is also slightly different from -T1 👀
  • --auto-threads={physical,logical} (default: physical) — when you use the default thread amount via -T0, this picks the default based on the number of detected physical or logical cores 🧠
  • --adapt[=min=#,max=#] — zstd dynamically adapts the compression level to the I/O conditions it's perceiving 🦎 You can watch the adaptation happen live with -v. You can also constrain adaptation between a supplied min and max level. The feature works when combined with multi-threading and --long mode, but it does not work with --single-thread. By default it sets the window size to 8 MiB (you can change that manually, see wlog). Because dynamic adaptation is chaotic by nature, the compressed result is not reproducible ⚠️
    • Note: at the time of writing, --adapt can get stuck at low speed when combined with multiple worker threads (>=2) 😩
  • --long[=#] — enables long distance matching with # windowLog. If # isn't present, it defaults to 27 🛣️ This increases the window size (windowLog) and memory usage for both the compressor and the decompressor. It's designed to improve the compression ratio for files that have long matches at a large distance 💜
    • Note: if windowLog is set larger than 27, you need to pass --long=windowLog or --memory=windowSize to the decompressor too 🫶
  • -D DICT — uses DICT as a dictionary to compress or decompress FILE(s) 📖
  • --patch-from FILE — specifies the file to be used as a reference point for zstd's diff engine 🩹 This is basically dictionary compression with some convenient parameter selection, namely that windowSize > srcSize.
    • Note: you can't use this and -D together 🚫
    • Note: --long mode gets automatically activated if chainLog < fileLog (fileLog being the windowLog required to cover the whole file). You can also force it manually 💪
    • Note: up to level 15, you can use --patch-from in --single-thread mode to marginally improve the compression ratio, at the cost of speed. Using '--single-thread' above level 15 leads to lower compression ratios 📉
    • Note: for level 19, you can get an increased compression ratio at the cost of speed by setting --zstd=targetLength= to something large (i.e. 4096), and by setting a large --zstd=chainLog= 🔥
  • --rsyncable — zstd will periodically synchronize the compression state so the compressed file is more rsync-friendly 🔁 There's a negligible impact on compression ratio, and a potential impact on compression speed that's perceptible at higher speeds (like when you combine --rsyncable with lots of parallel worker threads). This feature does not work with --single-thread. You probably don't want to use it with long range mode, since it decreases the effectiveness of the synchronization points, but your mileage may vary 🤷‍♀️
  • -C, --[no-]check — adds an integrity check computed from the uncompressed data (default: enabled) ✅
  • --[no-]content-size — enables or disables whether the original size of the file gets placed in the header of the compressed file 📏 The default is --content-size, which means the original size will be placed in the header.
  • --no-dictID — doesn't store the dictionary ID within the frame header (dictionary compression) 🙈 The decoder will have to rely on implicit knowledge about which dictionary to use, and it won't be able to check if it's the correct one.
  • -M#, --memory=# — sets a memory usage limit 🧠 By default, zstd uses 128 MiB for decompression as the maximum amount of memory the decompressor is allowed to use. You can override this if needed, in either direction (you can increase or decrease it).
    • This is also used during compression when you use it with --patch-from=. In that case, this parameter overrides the maximum size allowed for a dictionary. (128 MiB).
    • It can also limit memory for dictionary training. This parameter overrides the default limit of 2 GiB. zstd will load training samples up to the memory limit and ignore the rest 🧁
  • --stream-size=# — sets the pledged source size of input coming from a stream 🌊 This value must be exact, because it gets included in the produced frame header. Incorrect stream sizes will cause an error ❌ This info is used to better optimize compression parameters, which can give you better and potentially faster compression, especially for smaller source sizes 🦋
  • --size-hint=# — ok so when zstd is reading from a stream, it has to guess how big the source is before it tunes the compression settings, and if the stream is small that guess can be kinda wrong 🙈 This flag lets you set the guess yourself. Exact guesses give the best compression ratios ✨ Overestimates make the ratio slightly worse, but underestimates can tank it hard 💀
  • --target-compressed-block-size=# — tries to make compressed blocks around this size 🎀 It splits bigger blocks so they land closer to the target. This is super useful for latency, because the receiver can start working with early, incomplete data 🦋 It's a loose target, so blocks hit it "on average" but some can still be bigger or smaller. Turning it on can slow compression down by up to ~10% at level 1 😭 Higher levels take a smaller speed hit, and at the higher settings it's basically invisible 💅
  • -f, --force — the "I know what I'm doing" button 🔥 It turns off the input and output checks, so you can overwrite existing files, read from the console, write to stdout, and operate on links, block devices, etc. When decompressing, or when the output goes to stdout, unrecognized formats get passed through as-is 🫶
  • -c, --stdout — writes to standard output (even if that's the console) and keeps your original files, aka it disables --rm 💖
  • -o FILE — saves the result into FILE 📁 Heads up, this clashes with -c. If you pass both on the command line, whichever one you wrote last wins 💜
  • --[no-]sparse — turns sparse file support on or off 🌸 Sparse mode makes files with lots of zeroes take up less space on disk, and can speed up decompression since there's less disk I/O ✨ Default is enabled when the output goes to a file, and disabled when it goes to stdout. This setting overrides the default, so you can force sparse mode over stdout too 🧚
  • --[no-]pass-through — turns passing uncompressed files through as-is on or off 🦋 During decompression with pass-through enabled, unrecognized formats get copied straight from input to output. By default, pass-through happens when the output is stdout and you've set the force (-f) option 💅
  • --rm — removes the source file(s) after a successful compression or decompression 😳 It's silently ignored if the output is stdout. If you combine it with -o, zstd will ask you to confirm first (you can silence that with -f), because deleting stuff is a destructive move 💀
  • -k, --keep — keeps the source file(s) after a successful compression or decompression 🌈 This is the default behavior, so you're safe unless you go out of your way 🫶
  • -r — operates recursively on directories 📂 It selects every file in the named directory and all its subdirectories. This is clutch for cutting down on command line typing, and for dodging shell expansion limits when there are tons of files and the names blow past the max command line size 🙌
  • --filelist FILE — reads the list of files to process from FILE 📝 The format is compatible with ls output, one file per line 🎀
  • --output-dir-flat DIR — stores the resulting files in the target DIR directory instead of the same directory as the original file 🗂️ Be careful though, this can cause name collisions if files from different directories end up with the same name. Collision resolution keeps the first file with a given name in DIR, but if you combine it with -f, the last file ends up there instead 😬
  • --output-dir-mirror DIR — basically the same vibe as --output-dir-flat, the output files go underneath DIR, but this one copies your input directory hierarchy into the output DIR 🌿

If the input directory contains "..", the files in that directory get ignored 🙅‍♀️ If the input directory is an absolute directory (like "/var/tmp/abc"), it gets stored into "output-dir/var/tmp/abc". If there are multiple input files or directories, collision resolution follows the same rules as --output-dir-flat 💜

  • --format=FORMAT — compresses and decompresses in other formats 🍓 If zstd was compiled with support for them, it can compress to or decompress from other compression algorithm formats. The possible options are zstd, gzip, xz, lzma, and lz4. If you don't pass a format, zstd is the default ✨
  • -h/-H, --help — displays help (or long help) and exits 🙋‍♀️
  • -V, --version — displays the version number and exits immediately 🏷️ Since it exits right away, any flags you put after -V are basically ignored 🙈 Advanced move: -vV also shows supported formats, -vvV also shows POSIX support, and -qV will only print the version number, which is perfect for machines to read 🤖
  • -v, --verbose — verbose mode, gives you more info 📣
  • -q, --quiet — suppresses warnings, interactivity, and notifications 🤫 Specify it twice to suppress errors too 😶
  • --no-progress — hides the progress bar but keeps all the other messages 📊
  • --show-default-cparams — shows the default compression parameters that'll be used for a particular input file, based on your compression level and the input size 🔍 If the file isn't a regular file (like a pipe), this flag outputs the parameters used for inputs of unknown size 🧠
  • --exclude-compressed — only compresses files that aren't already compressed 🙅‍♀️
  • -- — everything after -- is treated as a file, even if it looks like a flag 🫣

gzip Operation Modifiers

okay so when zstd gets invoked through a gzip symlink, it picks up a few extra options that mimic gzip's behavior 🎀 Here they are:

  • -n, --no-name — doesn't store the original filename and timestamps when compressing a file 🙈 This is the default behavior, so it's technically a no-op 🤷‍♀️
  • --best — alias to the option -9 👑

Environment Variables

okay this is a big deal bestie: setting parameters with environment variables comes with security implications, so this avenue is intentionally limited 🔒 Only ZSTD_CLEVEL and ZSTD_NBTHREADS are supported right now 💅 They set the default compression level and the number of threads to use during compression, respectively ✨

ZSTD_CLEVEL sets the level between 1 and 19 (the "normal" range) 🌈 If its value isn't a valid integer, it gets ignored with a warning message 😬 ZSTD_CLEVEL just replaces the default compression level, which is (3) 🧚

ZSTD_NBTHREADS sets how many threads zstd will try to use during compression 🧵 If its value isn't a valid unsigned integer, it gets ignored with a warning message 🙃 ZSTD_NBTHREADS defaults to (1), and it's capped at ZSTDMT_NBWORKERS_MAX==200 💀 zstd has to be compiled with multithread support for this variable to do anything at all 🫠

Both of these can be overridden by their command line equivalents: -# for the compression level and -T# for the number of compression threads 💖

✨ ADVANCED COMPRESSION OPTIONS ✨

okay so zstd comes with 22 predefined regular compression levels plus the fast levels 💖 each level secretly gets translated into a bunch of advanced parameters that control how the compressor behaves. if you're curious about what your level turns into, run --show-default-cparams and see the receipts 🧾 and if you want to tweak things yourself, you can override those parameters with advanced compression options. it's giving full control 💅

🎀 --zstd[=options] 🎀

you pass the options as a comma-separated list. bestie, you only need to name the ones you want to change, and everything else just gets pulled from your selected or default compression level 🫶 here's the full list of what you can play with:

🦋 strategy=strat, strat=strat

this picks the strategy the match finder uses 🔍

there are 9 strategies, numbered 1 to 9, going from fastest to strongest: 1=ZSTD_fast, 2=ZSTD_dfast, 3=ZSTD_greedy, 4=ZSTD_lazy, 5=ZSTD_lazy2, 6=ZSTD_btlazy2, 7=ZSTD_btopt, 8=ZSTD_btultra, 9=ZSTD_btultra2.

🌸 windowLog=wlog, wlog=wlog

this sets the maximum number of bits for a match distance 📏

higher number = more chances to find a match, which usually makes the compression ratio better 📈 but the catch is it also eats more memory for both the compressor and the decompressor 😭 the minimum wlog is 10 (1 KiB) and the maximum is 30 (1 GiB) on 32-bit platforms, and 31 (2 GiB) on 64-bit platforms.

Heads up, babe: if you set windowLog to something larger than 27, you have to pass --long=windowLog or --memory=windowSize to the decompressor too, or it won't be able to read the file 💀

🧚 hashLog=hlog, hlog=hlog

this sets the maximum number of bits for a hash table 🪣

bigger hash tables mean fewer collisions, which usually makes compression faster ⚡ but it needs more memory while compressing. the minimum hlog is 6 (64 entries / 256 B) and the maximum is 30 (1B entries / 4 GiB).

💜 chainLog=clog, clog=clog

this sets the maximum number of bits for the secondary search structure, and what that structure looks like depends on the strategy you picked 🕸️

higher numbers = more chances to find a match, so the ratio usually improves 📈 but compression slows down and needs more memory. this option gets ignored for the ZSTD_fast strategy, since that one only has the primary hash table, no cap 🙅‍♀️ the minimum clog is 6 (64 entries / 256 B), and the maximum is 29 (512M entries / 2 GiB) on 32-bit platforms or 30 (1B entries / 4 GiB) on 64-bit platforms.

🔎 searchLog=slog, slog=slog

this sets the maximum number of searches in a hash chain or binary tree, using a logarithmic scale 🌀

more searches = better chance of finding a match, which usually boosts the compression ratio, but compression speed takes a hit 🐢 the minimum slog is 1 and the maximum is 'windowLog' - 1.

🍓 minMatch=mml, mml=mml

this sets the minimum searched length of a match in a hash table 📐

longer search lengths usually make the compression ratio worse, but decompression gets faster 🚀 the minimum mml is 3 and the maximum is 7.

🌈 targetLength=tlen, tlen=tlen

this one's personality depends entirely on which strategy you're using 🎭

for ZSTD_btopt, ZSTD_btultra and ZSTD_btultra2, it's the minimum match length that tells the match finder to stop searching. a bigger targetLength usually improves the ratio but slows compression down 🐌

for ZSTD_fast, it turns on ultra-fast mode whenever it's greater than 0 ⚡ the value is how much data gets skipped between match sampling. the vibe is reversed here: a bigger targetLength makes compression faster but the ratio worse 🙃

for all the other strategies, this field does absolutely nothing 🤷‍♀️

the minimum tlen is 0 and the maximum is 128 KiB.

🫧 overlapLog=ovlog, ovlog=ovlog

this decides the overlapSize, which is how much data gets reloaded from the previous job 🔁 this option only shows up when multithreading is enabled. reloading more data makes the ratio better but the speed worse ⚖️

the minimum ovlog is 0 and the maximum is 9. 1 means "no overlap", so every job is completely independent 🏝️ 9 means "full overlap", meaning up to windowSize gets reloaded from the previous job. every time you drop ovlog by 1, the reloaded amount gets cut in half, so 8 means windowSize/2 and 6 means windowSize/8 🧮 the value 0 is special and means "default", where zstd picks the ovlog automatically. in that case it'll land anywhere from 6 to 9, depending on the strategy you're using.

🌊 ldmHashLog=lhlog, lhlog=lhlog

this sets the maximum size of the hash table used for long distance matching 🛰️

heads up, this one is ignored unless long distance matching is turned on. bigger hash tables usually make the ratio better, but they cost more memory while compressing and slow things down a bit 🐢 the minimum lhlog is 6 and the maximum is 30 (default: 20).

🐝 ldmMinMatch=lmml, lmml=lmml

this sets the minimum searched length of a match for long distance matching 📏

also ignored unless long distance matching is enabled ✋ bigger or very tiny values usually make the ratio worse 📉 the minimum lmml is 4 and the maximum is 4096 (default: 64).

🪺 ldmBucketSizeLog=lblog, lblog=lblog

this sets the size of each bucket for the long distance matching hash table 🧺

ignored unless long distance matching is on, obviously 😌 bigger buckets resolve collisions better, but compression speed drops 📉 the minimum lblog is 1 and the maximum is 8 (default: 3).

🎈 ldmHashRateLog=lhrlog, lhrlog=lhrlog

this controls how often entries get inserted into the long distance matching hash table ⏰

also ignored unless long distance matching is enabled 🙅‍♀️ bigger values make compression faster, but if you stray too far from the default, the ratio will probably suffer 📉 the default value is wlog - lhlog.

🌷 Example 🌷

okay here's a little recipe 🍰 the following parameters set advanced compression options to something similar to predefined level 19, for files bigger than 256 KB:

--zstd=wlog=23,clog=23,hlog=22,slog=6,mml=3,tlen=48,strat=6

🧩 -B# 🧩

this lets you set the size of each compression job 🧁 it's only available when multi-threading is enabled. each job runs in parallel, so this value indirectly decides how many threads are active at once 🧵 the default job size changes depending on the compression level (generally 4 * windowSize), but with -B# you can pick your own custom size 🎯

just know that the job size has to respect a minimum, which gets enforced automatically for you 🤖 that minimum is either 512 KB or overlapSize, whichever one is biggest. and fair warning: different job sizes will give you non-identical compressed frames 👀

📚 DICTIONARY BUILDER 📚

okay so zstd has this thing called dictionary compression and it is SO good for tiny files and messages 🥺✨. you train zstd on a bunch of samples and it saves the result into a file called a dictionary 📖. then when you compress AND decompress, you point at the same one with -D dictionaryFileName. small files that look like your samples get a total glow-up 💅🔥.

🚂 --train FILEs

Use FILEs as your training set to make a dictionary. ideally it has a LOT of samples (> 100) and weighs about 100x your target dictionary size, so like ~10 MB for a 100 KB dictionary 🫶. you can combine --train with -r to point at a directory instead of listing every single file, which is clutch when the shell is being dramatic about expansion limits 🙄.

since dictionary compression is mostly for small files, the training set should only have small files. if some samples happen to be big, only the first 128 KiB of them get used for training 🤏.

--train supports multithreading if zstd was compiled with threading support (which it is by default) 🧵. extra advanced params go with --train-fastcover, the legacy builder is --train-legacy, and the slower cover builder is --train-cover 🐢. the default --train is the same as --train-fastcover=d=8,steps=4 💜.

📖 -o FILE

Saves the dictionary into FILE (default name: dictionary) 🗂️.

🧢 --maxdict=#

Caps the dictionary at the size you give (default: 112640 bytes). quantities are bytes unless you say otherwise, but you can use suffixes like KB or MB for bigger numbers 📏.

🎚️ -#

Use # compression level during training (optional). this makes the stats more tuned to that level, so you get a small ratio boost for that exact level 📈.

✂️ -B#

Splits input files into blocks of size # (default: no split) 🍰.

🧠 -M#, --memory=#

Limits how much sample data gets loaded for training (default: 2 GB). the default 2 GB is also the max, so no going higher 🙅. this is a lifesaver when your training set size is kinda out of control and could be HUGE. training speed is directly tied to sample set size, so smaller = faster 🏎️.

if the training set is bigger than the max memory, the CLI randomly picks samples from what's available, up to the memory budget 🎲. this is so your dictionary isn't biased by clustering, like only grabbing files from the start of a list sorted by modification date or alphabetical order. the randomness is deterministic though, so the same list of files with the same params makes the same dictionary every single time 🔁.

🪪 --dictID=#

A dictionary ID is a locally unique ID. the decoder uses it to check it's using the right dictionary 🕵️. by default zstd makes a random 4-byte ID, but you can give an explicit number instead. the dictionary manager needs to make sure it never hands the same ID to two different dictionaries 🫡. short numbers are a flex: an ID < 256 only needs 1 byte in the compressed frame header, and an ID < 65536 only needs 2 bytes. both beat the 4-byte default 💪.

RFC8878 reserves IDs less than 32768 and greater than or equal to 2\^31, so please don't use those in public 🚫.

🏋️ --train-cover[=k#,d=#,steps=#,split=#,shrink[=#]]

This picks the params for the default dictionary builder algorithm, which is called cover. here's the tea ☕:

  • if d isn't specified, it tries d = 6 and d = 8
  • if k isn't specified, it tries steps values in the range [50, 2000]
  • if steps isn't specified, the default is 40
  • if split isn't specified, or split <= 0, the default is 100
  • it requires that d <= k
  • if the shrink flag isn't used, the default shrinkDict value is 0
  • if shrink isn't specified, the default shrinkDictMaxRegression value is 1

the vibe is: it selects segments of size k with the highest score to put in the dictionary 🏆. a segment's score is the sum of the frequencies of all its subsegments of size d. generally d should be in the range [6, 8], sometimes up to 16, but the algorithm runs faster with d <= 8 ⚡. good values for k vary a lot based on your input data, but a safe range is [2 * d, 2000]. if split is 100, all input samples get used for both training and testing so it can find the optimal d and k 🎯. it supports multithreading if zstd is compiled with threading support 🧵.

with shrink enabled, it takes a truncated dictionary of minimum size and doubles in size until its compression ratio is at most shrinkDictMaxRegression% worse than the compression ratio of the largest dictionary 📐.

examples, bestie:

zstd --train-cover FILEs
zstd --train-cover=k=50,d=8 FILEs
zstd --train-cover=d=8,steps=500 FILEs
zstd --train-cover=k=50 FILEs
zstd --train-cover=k=50,split=60 FILEs
zstd --train-cover=shrink FILEs
zstd --train-cover=shrink=2 FILEs

⚡ --train-fastcover[=k#,d=#,f=#,steps=#,split=#,accel=#]

this is the same as cover, but with extra f and accel params and a different default split 🏃‍♀️. if split isn't specified, it tries split = 75. if f isn't specified, it tries f = 20, and it requires that 0 < f < 32. if accel isn't specified, it tries accel = 1, and it requires that 0 < accel <= 10. it also requires that d = 6 or d = 8 🫠.

f is the log of the size of the array that keeps track of how often each subsegment of size d shows up 📊. each subsegment gets hashed to an index in the range [0,2^f - 1]. two different subsegments can hash to the same index, and then they're treated as the same subsegment when computing frequency 👯. a higher f means fewer collisions, but it takes longer ⏳.

examples:

zstd --train-fastcover FILEs
zstd --train-fastcover=d=8,f=15,accel=2 FILEs

🕰️ --train-legacy[=selectivity=#]

uses the legacy dictionary builder algorithm with the given dictionary selectivity (default: 9) 📜. the smaller the selectivity value, the denser the dictionary, which makes it more efficient but lowers its achievable maximum size. --train-legacy=s=# is also accepted 💅.

examples:

zstd --train-legacy FILEs
zstd --train-legacy=selectivity=8 FILEs

🏁 BENCHMARK 🏁

the zstd CLI has a benchmarking mode that's perfect for finding good compression params, or just flexing your computer's performance 💪. zstd -b [FILE(s)] benchmarks zstd for both compression and decompression at the default compression level 🎯. results depend A LOT on the content being compressed, so keep that in mind 🙈. you can pass multiple files to the benchmark, and even a directory with -r DIRECTORY 📂. if you don't give it a FILE, the benchmark uses a procedurally generated lorem ipsum text 📝.

here are the bench options:

  • -b# — benchmark file(s) using compression level #
  • -e# — benchmark file(s) across multiple compression levels, from -b# to -e# (inclusive)
  • -d — benchmark decompression speed only (you need to provide zstd-compressed content for this one)
  • -i# — minimum evaluation time in seconds (default: 3s), benchmark mode only
  • -B#, --block-size=# — cut file(s) into independent chunks of size # (default: no chunking)
  • -S — output one benchmark result per input file (default: one consolidated result)
  • -D dictionary — benchmark using a dictionary
  • --priority=rt — set process priority to real-time (Windows)

beyond compression levels, benchmarking plays nice with other params too 🫶. that means number of threads (-T#), advanced compression params (--zstd=###), dictionary compression (-D dictionary), or even disabling checksum verification, for example ✨.

the output looks like this:

CompressionLevel#Filename: InputSize -> OutputSize (CompressionRatio), CompressionSpeed, DecompressionSpeed

and the methodology is: for speed measurement, the entire input gets compressed/decompressed in memory 🧠. a run lasts at least 1 sec, so when files are tiny, they get compressed/decompressed several times per run to keep the measurements accurate 🎀.

🔗 SEE ALSO 🔗

okay bestie, go check these out 👀: zstdgrep(1), zstdless(1), gzip(1), xz(1) 🌸.

the zstandard format is specified in Y. Collet, "Zstandard Compression and the 'application/zstd' Media Type", https://www.ietf.org/rfc/rfc8878.txt, Internet RFC 8878 (February 2021) 📜💜.

🐛 BUGS 🐛

if something's broken, report it at https://github.com/facebook/zstd/issues 🙏🦋.

👩‍💻 AUTHOR 👩‍💻

Yann Collet 👑.

bestie, that's the whole thing 🥹💖. this was zstd 1.5.6, October 2024, and you are now officially a dictionary-building, benchmark-flexing legend 🧚✨. love you, mwah 😘💅

📦 where this page comes from