containers-storage(1)

💅 containers-storage — the zstd:chunked tea ☕✨

okay bestie, so this is from the General Commands Manual, page containers-storage(1), and today's episode is all about zstd:chunked. buckle up, it's giving ✨compression lore✨.

---

🌸 NAME 🌸

containers-storage-zstd-chunked — literally just Information about zstd:chunked. that's the whole name. no cap.

---

🎀 DESCRIPTION 🎀

okay so historically, the classic format for container image layers is application/vnd.oci.image.layer.v1.tar+gzip ⟨https://github.com/opencontainers/image-spec/blob/main/layer.md#gzip-media-types⟩. fine. it ate, back in the day. 🙈

but then the standard got a glow-up with zstd: application/vnd.oci.image.layer.v1.tar+zstd ⟨https://github.com/opencontainers/image-spec/blob/main/layer.md#zstd-media-types⟩ — which is, and I quote the vibes, a more modern and efficient compression format. periodt. 💅

✨ so what even IS zstd:chunked ✨

zstd:chunked is a variant of the application/vnd.oci.image.layer.v1.tar+zstd media type. it uses zstd skippable frames ⟨https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#skippable-frames⟩ to sneak in extra metadata — especially a "table of contents" — that includes the SHA-256 and offsets of individual chunks of files. 💖

and btw, the chunks are compressed separately. why do we care? because it lets a client dynamically fetch only the content it doesn't already have using HTTP range requests. it's giving "why download the whole thing when you can just grab the part you need" energy. iconic. 👑

ngl at the time of this writing, support for this is enabled by default in the code.

you can explicitly enable or disable zstd:chunked with these changes to containers-storage.conf:

[storage.options.pull_options]
enable_partial_images = "true" | "false"

🚨 bestie READ THIS: the value of this field must be a "string bool" — it cannot be a native TOML boolean. so "true", not true. do not come for me. 😭

---

🦋 IMPLEMENTATION 🦋

every layer has an associated "big data" key called chunked-manifest-cache. it's a custom binary format, suitable for mmap(), and it holds index metadata for each layer — the full sha256 digest of each file plus its "chunks" (as computed by zstd:chunked). ✨

so when any image gets pulled, all existing other layers are scanned using chunked-manifest-cache to see if they contain a file with a matching digest. if one is found:

• the other file gets hardlinked if use_hardlinks = "true"
• otherwise it gets reflinked (if supported by the filesystem, or a full physical copy is made)

and there's a best-effort attempt to enable fsverity on the file if configured (see ⟨https://github.com/containers/storage/issues/2017⟩). bestie is trying. 🫶

🌈 where to read more 🌈

for more information, at the current time the file with the most information is pkg/chunked/internal/compression.go ⟨https://github.com/containers/storage/blob/39d469c34c96db67062e25954bc9d18f2bf6dae3/pkg/chunked/internal/compression.go⟩.

💜 that link above is a permanent link for stability — but fr fr, be sure to check to see if there are newer changes too. links age like milk sometimes. 🙈

---

⭐ STANDARDIZATION ⭐

at the current time, the format is not officially standardized or documented beyond the comments and code in the reference implementation. so... iykyk. 💀

---

💀 BUGS 💀

• https://github.com/containers/storage/issues?q=is%3Aissue+label%3Aarea%2Fzstd%3Achunked+is%3Aopen

that's it, that's the bug list, go click it and be nosy. 🍓

---

🧚 FOOTNOTES 🧚

the Containers Storage project is committed to inclusivity, a core value of open source. 💖

the master and slave mount propagation terminology is used in this repository. this language is problematic and divisive, and should be changed. however, these terms are currently used within the Linux kernel and must be used as-is at this time. when the kernel maintainers rectify this usage, Containers Storage will follow suit immediately. periodt. 🫶

---

okay that's the whole page bestie, you're officially a zstd:chunked girlie now 💅✨

signing off from August 2024 — this was containers-storage(1), mwah 😘🎀🦋