zstd(3)
💖 zstd(3) — Erlang Module Definition ✨
okay bestie, sit down, hydrate, because we're about to talk about zstd — the Erlang module, manual section 3, and honestly? she's iconic.
🎀 NAME 🎀
zstd — the Zstandard compression interface. that's it, that's the vibe. 🦋
🌸 DESCRIPTION 🌸
so like, this module hands you an API for the Zstandard library (www.zstd.net). you use it to compress and decompress data, and the way it's giving is: same compression ratio as zlib but at a lower CPU cost. no cap, bestie, that's the whole sales pitch and it slaps. 💅
here's the lil demo, look at her go:
1> Data = ~"my data to be compressed".
2> Compressed = zstd:compress(Data).
3> zstd:decompress(Compressed).
[~"my data to be compressed"]now. if you're compressing or decompressing possibly large amounts of data, you don't have to do it all in one shot — you can also do streamed compression/decompression. main character energy fr fr:
1> Compress = fun F(Ctx, D) ->
case file:read(D, 5) of
{ok, Data} ->
{continue, C} = zstd:stream(Ctx, Data),
[C|F(Ctx, D)];
eof ->
{done, C} = zstd:finish(Ctx, ""),
C
end
end.
2> {ok, Ctx} = zstd:context(compress).
3> {ok, D} = file:open(File,[read,binary]).
4> Compressed = iolist_to_binary(Compress(Ctx, D)).
<<40,181,47,253,0,88,89,0,0,108,111,114,101,109,32,105,112,115,117,109>>
5> zstd:decompress(Compressed).
[~"lorem ipsum"]lowkey important heads up: in all functions, errors can be thrown, and Reason is the thing that tells you what went wrong. 👀
k so the typical Reasons you might bump into:
badarg— bad argument. literally just that. 🙈zstd_error— an error generated by the Zstandard library itself. not us, it's the library. 💀not_on_controlling_process— bestie. the context was used by a process that did not create it. only the creator gets to touch it, that's the rule. 💅
🦋 DATA TYPES 🦋
compress_parameters()
compress_parameters() ::
#{dictionary => binary() | dict(),
pledgedSrcSize => non_neg_integer(),
compressionLevel => compressionLevel(),
windowLog => non_neg_integer(),
hashLog => non_neg_integer(),
chainLog => non_neg_integer(),
searchLog => non_neg_integer(),
minMatch => non_neg_integer(),
targetLength => non_neg_integer(),
strategy => strategy(),
targetCBlockSize => non_neg_integer(),
enableLongDistanceMatching => boolean(),
ldmHashLog => non_neg_integer(),
ldmMinMatch => non_neg_integer(),
ldmBucketSizeLog => non_neg_integer(),
ldmHashRateLog => non_neg_integer(),
contentSizeFlag => boolean(),
checksumFlag => boolean(),
dictIDFlag => boolean()}.compression parameters ✨ (affectionately: the knobs)
so Zstandard has so many parameters you can tune. real talk though: setting some of them will fail if you set them to an incorrect value, while others get silently adjusted to the closest valid value. the library just quietly fixes you up, no drama. 😌
here's the rundown:
dictionary— sets the compression dictionary for the context. it can be either abinary()representing the dictionary, or adict/0. when adict/0is attached to a context it will be kept alive until either the context is closed or it gets replaced by another dictionary. 🫶- wanna reset the context to not use any dictionary at all? use the empty dictionary, that is
<<>>. boom, clean slate.
- wanna reset the context to not use any dictionary at all? use the empty dictionary, that is
pledgedSrcSize— when you're usingstream/2to do streaming compression, the decompressed size is not known at the moment the header of the Zstandard frame gets emitted. setting this parameter on the context lets the compressor know the expected size of the data it's about to compress. and if that size isn't correct whenfinish/2is called, an exception will be generated — so get it right, bestie. 😭 (good news: usingcompress/1,2will automatically set this value for you, so you don't gotta.)compressionLevel— sets thecompressionLevel/0. obviously. 💅windowLog|hashLog|chainLog|searchLog|minMatch|targetLength|targetCBlockSize— set the corresponding parameter. see the Zstandard documentation for more details, iykyk. 📚strategy— sets the compressionstrategy/0. 🌈enableLongDistanceMatching— whether to enable Long Distance Matching or not. LDM is useful when compressing large datasets, and it's enabled by using a highcompressionLevel. (long distance relationship? no. long distance matching. different thing. 💀)ldmHashLog|ldmMinMatch|ldmBucketSizeLog|ldmHashRateLog— set the corresponding LDM parameter. see the Zstandard documentation for more details. ✨contentSizeFlag— whether to include the contentSize or not.checksumFlag— whether to include the checksum or not.dictIDFlag— whether to include the dictionary ID or not.
and if you want the full tea on every single one of these, the Zstandard documentation contains more details about each parameter. go read it, it's good for you. 🧚
compressionLevel()
compressionLevel() :: -22..22.the compression level, periodt. 📈
higher values mean better compression ratio at the sacrifice of performance. a negative value sacrifices compression ratio in favor of performance. so like, it's a tradeoff, pick your fighter.
and 0? that one's special — it represents the default compression level. slay. 👑
-opaque context()
a compression or decompression context that you can use for streaming compression or decompression. 🫧
⚠️ only the process that created the context can use it. do not pass it around like a snack, bestie, it will not go well.
decompress_parameters()
decompress_parameters() ::
#{dictionary => binary() | dict(),
windowLogMax => non_neg_integer()}.decompression parameters 💜
Zstandard has many parameters that can be tuned. and again, for real: setting some parameters will fail when set to an incorrect value, while others will be silently adjusted to the closest valid value. 🌸
dictionary— sets the decompression dictionary for the context. the dictionary can be either abinary()representing the dictionary, or a decompressiondict/0. when adict/0is attached to a context it will be kept alive until either the context is closed or it's replaced by another dictionary.- to reset the context to not use any dictionary, use the empty dictionary, that is
<<>>. 🧼
- to reset the context to not use any dictionary, use the empty dictionary, that is
windowLogMax— set the corresponding parameter. see the Zstandard documentation for more details. 🔍
-opaque dict()
a compression or decompression dictionary. that's her. that's the whole description. 💖
strategy()
strategy() ::
default | fast | dfast | greedy | lazy | lazy2 | btlazy2 |
btopt | btultra | btultra2.the compression strategy — a whole roster of queens. 👑
these are listed in order of the compression ratio they give. so: the fast strategy is the fastest but also has the worst compression ratio, while btultra2 is the slowest but has the best compression ratio. pick your fave, no wrong answers, just different vibes. ⚡
default is a special strategy representing the current default strategy. (a strategy that's just… the default strategy. she contains multitudes. 💅)
and for the details on each one? you already know — see the Zstandard documentation. 📖
✨ FUNCTIONS ✨
okay bestie, this is the whole function menu 💅 every single one, no skips, periodt.
💖 close/1 — bye bye context, release the resources
close(Ctx :: context()) -> ok.so this one closes a context/0 and releases all the resources it was holding onto. once it's closed, you literally cannot use it anymore, no cap. a context/0 also gets auto-closed when it's GC:ed, so the only reason you'd call this is to make those resources release before the next GC. proactive queen behavior ✨
💖 compress/2 — squish that data
compress(Data :: iodata(), Options :: compress_parameters()) ->
iodata();
(Data :: iodata(), Ctx :: context()) -> iodata().compress Data using either the given compress_parameters/0 or a context/0. that's it, that's the vibe.
example:
1> zstd:compress("abc").
2> zstd:compress("abc", #{ compressionLevel => 20 }).💖 context/2 — make a streaming context
context(compress, Options :: compress_parameters()) ->
{ok, context()};
(decompress, Options :: decompress_parameters()) ->
{ok, context()}.create a compression or decompression context/0. a context lets you do streaming compression/decompression and lets you reuse parameters for multiple compressions/decompressions. it's giving efficiency, periodt. 👑
💖 decompress/2 — unsquish
decompress(Data :: iodata(), Options :: decompress_parameters()) ->
iodata();
(Data :: iodata(), Ctx :: context()) -> iodata().decompress Data using the given decompress_parameters/0 or a context/0. example:
1> Compressed = zstd:compress("abc").
2> zstd:decompress(Compressed).
[~"abc"]💖 dict/3 — dictionary bestie
dict(compress,
Dict :: binary(),
#{compressionLevel => compressionLevel()}) ->
{ok, dict()};
(decompress, Dict :: binary(), #{}) -> {ok, dict()}.create a compression or decompression dictionary. 💫
- a compression dictionary can be used as a
compress_parameters/0to use a dictionary for compression. dictionaries allow good compression ratios even for small amounts of data. tiny data? still slay. - a decompression dictionary can be used as a
decompress_parameters/0to use a dictionary for decompression. the same dictionary has to be used for compression as decompression. to verify that the same dictionary is used you can useget_dict_id/1on the dictionary and compressed data, or just try to decompress, because decompression will raise an exception if an incorrect dictionary is given. - the
compressionLevelset on a dictionary will override thecompressionLevelset in thecontext/0.
example:
1> {ok, CDict} = zstd:dict(compress, Dict).
2> Data = lists:duplicate(100, 1).
[1, 1, 1 | _]
3> iolist_size(zstd:compress(Data)).
17
4> iolist_size(zstd:compress(Data, #{ dictionary => CDict, dictIDFlag => false })).
16as loading a dictionary can be a heavy operation, it is possible to create only a single dict/0 and provide it to multiple context/0.
there is no API exposed in zstd to create a dictionary, instead use the zstd command line tool. iykyk. 🛠️
💖 finish/2 — wrap it up
finish(Ctx :: context(), Data :: iodata()) -> Result
when Result :: {done, erlang:iovec()}.finish compressing/decompressing data. this flushes all output buffers and resets the context/0 so it can be used for compressing/decompressing again. example:
1> {ok, DCtx} = zstd:context(decompress).
2> {continue, D1} = zstd:stream(DCtx, <<40,181,47,253,32>>).
3> {done, D2} = zstd:finish(DCtx, <<2,17,0,0,97,98>>).
4> iolist_to_binary([D1,D2]).
<<"ab">>💖 flush/1 — flush it, but don't end the frame
flush(Ctx :: context()) -> Result
when Result :: {continue, Output :: binary()}.flush the compression stream. this flushes all pending compressed data without ending the frame, so the compressed data can be read immediately while keeping the context open for further compression. iconic. example:
1> {ok, CCtx} = zstd:context(compress).
2> {continue, C1} = zstd:stream(CCtx, ~"hello").
3> {continue, C2} = zstd:flush(CCtx).
4> zstd:decompress([C1, C2]).
[<<"hello">>]💖 get_dict_id/1 — who dis dictionary?
get_dict_id(DictOrFrame :: dict() | binary()) -> non_neg_integer().get the dictionary ID of a dictionary or a frame. the dictionary ID 0 represents no dictionary. example:
1> {ok, CDict} = zstd:dict(compress, Dict).
2> zstd:get_dict_id(CDict).
1850243626
3> zstd:get_dict_id(zstd:compress("abc")).
0💖 get_frame_header/1 — metadata serve
get_frame_header(Frame :: iodata()) ->
{ok,
#{blockSizeMax => non_neg_integer(),
checksumFlag => boolean(),
dictID => non_neg_integer(),
frameContentSize => non_neg_integer(),
frameType =>
'ZSTD_frame' | 'ZSTD_skippableFrame',
headerSize => non_neg_integer(),
windowSize => non_neg_integer()}} |
{error, unicode:chardata()}.get the header of a Zstandard compressed frame. a compressed Zstandard stream can consist of multiple frames. this function will read metadata from the first frame. this info can be useful when debugging corrupted Zstandard streams. example:
1> Compressed = zstd:compress(~"abc").
2> zstd:get_frame_header(Compressed).
{ok,#{frameContentSize => 3,windowSize => 3,blockSizeMax => 3,
frameType => 'ZSTD_frame',headerSize => 6,
dictID => 0, checksumFlag => false}}💖 get_parameter/2 — read a setting
get_parameter(Ctx :: context(), Key :: term()) -> Value :: term().get a parameter from a context/0. see compress_parameters/0 and decompress_parameters/0 for details on which parameters are available and what each parameter does.
note that it is not possible to get the dictionary and pledgedSrcSize parameters using this API. instead you can use get_dict_id/1 on the context/0 to get the id of the dictionary used. there is no way to get the pledgedSrcSize.
returns ok on success, raises an error on failure. example:
1> {ok, CCtx} = zstd:context(compress).
{ok, _}
2> zstd:get_parameter(CCtx, compressionLevel).
3
3> zstd:set_parameter(CCtx, compressionLevel, 15).
ok
4> zstd:get_parameter(CCtx, compressionLevel).
15💖 reset/1 — reset but keep the settings
reset(Ctx :: context()) -> ok.reset a context while streaming data, returning it to its original state but keeping all parameters set. by resetting the state, the context can be re-used for other operations even if it's in the middle of a (de)compression stream. example:
1> {ok, CCtx} = zstd:context(compress).
2> zstd:stream(CCtx, "a").
{continue, _}
3> zstd:reset(CCtx).
ok
4> {done, Compressed} = zstd:finish(CCtx, "b").
5> zstd:decompress(Compressed).
[~"b"]💖 set_parameter/3 — set a setting
set_parameter(Ctx :: context(), Key :: term(), Value :: term()) ->
ok.set a parameter on a context/0. see compress_parameters/0 and decompress_parameters/0 for details on which parameters are available and what each parameter does.
returns ok on success, raises an error on failure. example:
1> {ok, CCtx} = zstd:context(compress).
{ok, _}
2> ok = zstd:set_parameter(CCtx, compressionLevel, 15).
ok
3> zstd:stream(CCtx, "abc").
{continue, _}
4> catch zstd:set_parameter(CCtx, dictionary, "abc").
{'EXIT', {{zstd_error, <<"Operation not authorized at current processing stage">>}, _}}💖 stream/2 — stream it up
stream(Ctx :: context(), Data :: iodata()) -> Result
when
Result ::
{continue,
Remainder :: erlang:iovec(),
Output :: binary()} |
{continue, Output :: binary()}.compress or decompress a stream of data. the last stream of data should be called with finish/2 to complete the compression/decompression. example:
1> {ok, CCtx} = zstd:context(compress).
2> {continue, C1} = zstd:stream(CCtx, ~"a").
3> {done, C2} = zstd:finish(CCtx, ~"b").
4> Compressed = iolist_to_binary([C1, C2]).
<<40,181,47,253,0,88,17,0,0,97,98>>
5> zstd:decompress(Compressed).
[<<"ab">>]that's the whole FUNCTIONS section, bestie 💖 Ericsson AB, stdlib 8.0.4, zstd(3) — you ate that, periodt. 🎀✨
📦 where this page comes from
- ubuntu stonking: erlang-doc 1:29.0.6+dfsg-1 · Apache-2.0
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms