/etc/contai.../config.toml(5)

📦
version 2.4.2 · 11 versions
, in arch rolling, fedora 44, nixpkgs unstable

✨ containerd-config.toml — the /etc/containerd/config.toml(5) glow-up ✨

okay bestie, so this is the File Formats Manual page for /etc/containerd/config.toml(5), and it's genuinely the blueprint of your whole containerd situation. it's long, so buckle up — we're doing this in parts 💅

🌸 NAME 🌸

containerd-config.toml — the configuration file for containerd. that's it, that's the whole name, and honestly? iconic.

🦋 SYNOPSIS 🦋

so the config.toml file is, like, the configuration file for the containerd daemon. for the daemon to actually pick it up, the file has to live at /etc/containerd/config.toml, or you have to point at it with the --config option of containerd. those are your two vibes.

and if the file isn't there at the right spot, or you didn't pass it via --config? containerd just shrugs and uses its default configuration settings. and you can peek at those defaults with the containerd config(1) command. so cute of it to show its work 🫶

💖 DESCRIPTION 💖

the TOML file that configures containerd's daemon settings is built like this: a short little list of global settings first, and then a whole series of sections for specific areas of daemon configuration. and then — the best part — there's a section for plugins, which gives every containerd plugin its own lil area for plugin-specific config and settings. main character energy for each plugin, periodt.

👑 FORMAT 👑

version

the version field in the config file says what version the config is. if you don't put a version number in there, it's assumed to be a version 1 config and gets parsed as such. version 4 is the latest config version. older configs get automatically migrated on startup, so no stress 💜

root

the root directory for containerd metadata. (Default: "/var/lib/containerd")

state

the state directory for containerd. (Default: "/run/containerd")

plugin_dir

the directory where dynamic plugins get stored.

[grpc] — deprecated in version 4

heads up bestie, this one's deprecated in version 4! don't build new stuff around it.

section for gRPC socket listener settings. in version 4, you use the server plugins io.containerd.server.v1.grpc and io.containerd.server.v1.grpc-tcp instead. existing configs get migrated automatically, so it's not a crisis, but still. contains these properties:

  • address (Default: "/run/containerd/containerd.sock")
  • tcp_address
  • tcp_tls_cert
  • tcp_tls_key
  • uid (Default: 0)
  • gid (Default: 0)
  • max_recv_message_size
  • max_send_message_size

[ttrpc] — deprecated in version 4

also deprecated in version 4 — same deal, migrate along with everything else ✨

section for TTRPC settings. in version 4, use the server plugin io.containerd.server.v1.ttrpc instead. in prior versions, when the TTRPC address wasn't explicitly set, it was derived from the GRPC address (grpcAddress + ".ttrpc") and inherited GRPC's UID/GID. in version 4, each server plugin is configured independently; the TTRPC plugin uses its own defaults when its configuration block is omitted. contains these properties:

  • address (Default: "")
  • uid (Default: 0)
  • gid (Default: 0)

[debug] — deprecated in version 4

section to enable and configure a debug socket listener. in version 4, use the server plugin io.containerd.server.v1.debug instead. contains these properties:

  • address (Default: "") — the debug endpoint does not listen by default
  • uid (Default: 0)
  • gid (Default: 0)
  • level (Default: "info") — sets the debug log level. supported levels are: "trace", "debug", "info", "warn", "error", "fatal", "panic"
  • format (Default: "text") — sets the log format. supported formats are "text" and "json"

[metrics] — deprecated in version 4

section to enable and configure a metrics listener. in version 4, use the server plugin io.containerd.server.v1.metrics instead. contains these properties:

  • address (Default: "") — the metrics endpoint does not listen by default
  • grpc_histogram (Default: false) — turn gRPC histogram metrics on or off

disabled_plugins

disabled plugins are IDs of plugins to disable. disabled plugins won't be initialized and started. ghosted, basically 💀

required_plugins

required plugins are IDs of required plugins. containerd exits if any required plugin doesn't exist, or fails to be initialized or started. no chill, no negotiation.

[plugins]

the plugins section holds configuration options exposed from installed plugins. the following plugins are enabled by default and their settings are shown below. plugins that are not enabled by default will provide their own documentation for configuration values. (yes, you have to go read their docs, sorry babe 🙈)

  • [plugins."io.containerd.server.v1.grpc"] — configures the main gRPC server listener (version 4):
    • address (Default: "/run/containerd/containerd.sock")
    • uid (Default: effective UID)
    • gid (Default: effective GID)
    • max_recv_message_size (Default: 16777216)
    • max_send_message_size (Default: 16777216)
  • [plugins."io.containerd.server.v1.grpc-tcp"] — configures the TCP gRPC server listener (version 4). skipped if address is empty:
    • address (Default: "")
    • tls_cert, tls_key, tls_ca, tls_common_name
    • max_recv_message_size (Default: 16777216)
    • max_send_message_size (Default: 16777216)
  • [plugins."io.containerd.server.v1.ttrpc"] — configures the TTRPC server listener (version 4). in version 4 this plugin is configured independently from the gRPC plugin. if the plugin block is omitted, the TTRPC server binds to its own default address rather than deriving one from the gRPC address:
    • address (Default: "/run/containerd/containerd.sock.ttrpc")
    • uid (Default: effective UID)
    • gid (Default: effective GID)
  • [plugins."io.containerd.server.v1.debug"] — configures the debug server listener (version 4). skipped if address is empty:
    • address (Default: "")
    • uid (Default: 0)
    • gid (Default: 0)
  • [plugins."io.containerd.server.v1.metrics"] — configures the metrics HTTP listener (version 4). skipped if address is empty:
    • address (Default: "")
  • [plugins."io.containerd.monitor.v1.cgroups"] — has one option: no_prometheus (Default: false)
  • [plugins."io.containerd.service.v1.diff-service"] — has one option: default, a list by default set to ["walking"]
  • [plugins."io.containerd.gc.v1.scheduler"] — has several options that do advanced tuning for the scheduler:
    • pause_threshold is the maximum amount of time GC should be scheduled (Default: 0.02),
    • deletion_threshold guarantees GC is scheduled after n number of deletions (Default: 0 [not triggered]),
    • mutation_threshold guarantees GC is scheduled after n number of database mutations (Default: 100),
    • schedule_delay defines the delay after the trigger event before scheduling a GC (Default "0ms" [immediate]),
    • startup_delay defines the delay after startup before scheduling a GC (Default "100ms")
  • [plugins."io.containerd.runtime.v2.task"] — specifies options for configuring the runtime shim:
    • platforms specifies the list of supported platforms
    • sched_core — core scheduling is a feature that allows only trusted tasks to run concurrently on cpus sharing compute resources (eg: hyperthreads on a core). (Default: false)
  • [plugins."io.containerd.service.v1.tasks-service"] — has performance options:
  • [plugins."io.containerd.grpc.v1.cri".containerd] — contains options for the CRI plugin, plus child nodes for CRI options:
    • default_runtime_name (Default: "runc") specifies the default runtime name
  • [plugins."io.containerd.grpc.v1.cri".containerd.runtimes] — one or more container runtimes, each with a unique name
  • [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.<runtime>] — a runtime named <runtime>
  • [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.<runtime>.options] — options for the named <runtime>, most important:
    • BinaryName specifies the path to the actual runtime to be invoked by the shim, e.g. "/usr/bin/runc"

oom_score

the out of memory (OOM) score applied to the containerd daemon process. (Default: 0)

[cgroup]

section for Linux cgroup specific settings 🐧

  • path (Default: "") — specify a custom cgroup path for created containers

[proxy_plugins]

proxy plugins configures plugins which are communicated to over gRPC.

  • type (Default: "")
  • address (Default: "")

timeouts

timeouts specified as a duration.

imports

imports is a list of additional configuration files to include. this lets you split up the main configuration file and keep some sections separate — for example, vendors may keep a custom runtime configuration in a separate file without modifying the main config.toml. imported files will overwrite simple fields like int or string (if not empty), and will append array and map fields. imported files are also versioned, and the version can't be higher than the main config. (no sneaking newer versions past the bouncer, bestie 🚫)

stream_processors

  • accepts (Default: "[]") — accepts specific media-types
  • returns (Default: "") — returns the media-type
  • path (Default: "") — path or name of the binary
  • args (Default: "[]") — args to the binary

and that's part 1 wrapped, but we are so not done — there's more coming, so don't go anywhere 💖

✨ EXAMPLES ✨

ok bestie this is the part where it all clicks, i promise 🫶

🎀 Version 4 Configuration 🎀

so check it — here's a config.toml example on version 4, where the server settings are all set up as plugins. iconic behaviour fr:

version = 4

root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0
imports = ["/etc/containerd/runtime_*.toml", "./debug.toml"]

[plugins."io.containerd.server.v1.grpc"]
  address = "/run/containerd/containerd.sock"

[plugins."io.containerd.server.v1.ttrpc"]
  address = "/run/containerd/containerd.sock.ttrpc"

[plugins."io.containerd.server.v1.debug"]
  address = "/run/containerd/debug.sock"
  level = "info"

[cgroup]
  path = ""

[plugins]
  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false
  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]
  [plugins."io.containerd.gc.v1.scheduler"]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = 0
    startup_delay = "100ms"
  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]
    sched_core = true
  [plugins."io.containerd.service.v1.tasks-service"]
    blockio_config_file = ""
    rdt_config_file = ""

🦋 Multiple Runtimes 🦋

now THIS one? this is the glow-up. here's an example of a partial config with two runtimes living their best lives side by side:

[plugins]

  [plugins."io.containerd.grpc.v1.cri"]

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          privileged_without_host_devices = false
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = "/usr/bin/runc"

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.other]
          privileged_without_host_devices = false
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.other.options]
            BinaryName = "/usr/bin/path-to-runtime"

so what did we just do 👀 the above creates two named runtime configurations — one called runc and one called other — and it sets the default runtime to runc. and ngl, these two are used solely for runtimes that get invoked via CRI. so to actually use the non-default other runtime from this example, a spec has to include the runtime handler named other, which is how you say "hey, i want the named runtime config please".

<div></div>

the CRI specification includes a ⟨https://github.com/kubernetes/cri-api/blob/de5f1318aede866435308f39cb432618a15f104e/pkg/apis/runtime/v1/api.proto#L476⟩, which will reference the named runtime. ✨

<div></div>

and this part is important, ok? the naming convention is not a suggestion, it's a whole lifestyle. runtimes go under

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

with every runtime getting its own unique name, like

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]

and on top of that, each runtime can have shim-specific options tucked under

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.<runtime>.options]

for example

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

<div></div>

the io.containerd.runc.v2 runtime is the one used to run OCI-compatible runtimes on Linux, like runc. in the example up there, the runtime_type field is what says which shim to use (io.containerd.runc.v2), while the BinaryName field is a shim-specific option that says where the OCI runtime lives. 🌈

<div></div>

so for the example config named runc, the shim will launch /usr/bin/runc as the OCI runtime. and for the example config named other, the shim will launch /usr/bin/path-to-runtime instead. two runtimes, two paths, zero confusion, we love to see it 💅

💀 BUGS 💀

if you hit any specific issues, pls go file them at

https://github.com/containerd/containerd

and not in my dms bestie 😭 the containerd folks are the ones who can actually fix it.

👑 AUTHOR 👑

Phil Estes estesp@gmail.com ⟨mailto:estesp@gmail.com⟩

🍓 SEE ALSO 🍓

and that's a wrap on part 2, bestie 🎀💖 this whole page is from the File Formats Manual, dated 04/05/2022, so shoutout to that version of the docs for coming through with the receipts 🫶 go forth and configure responsibly, ok? periodt. ✨🦋

📦 where this page comes from