CH-RUN(1)

📦

version 0.44 · 10 versions

👑 CH-RUN(1) — Charliecloud 👑

okay bestie, welcome to ch-run, the main character of the Charliecloud manual pages 💅 it runs a command in a completely unprivileged container, and yes, it's as iconic as it sounds.

✨ NAME ✨

ch-run — run a command in a Charliecloud container 🫶

✨ SYNOPSIS ✨

$ ch-run [OPTION...] IMAGE -- COMMAND [ARG1 [ARG2 [...]]]

✨ DESCRIPTION ✨

So here's the deal: ch-run runs the command COMMAND inside a container that is fully unprivileged — no privilege escalation anywhere, not even a crumb — using the image you point it at with IMAGE.

And IMAGE can be literally one of these three things:

a path to a directory 📁
the name of an image in ch-image storage, like example.com:5050/foo
a SquashFS archive — if the proper support is enabled 🗜️

fr fr, ch-run does not use any setuid or setcap helpers. Not even when it's mounting SquashFS images with FUSE. No setuid glow-up required ✨

Now, COMMAND is the name of the program you want to run inside the container, and it gets zero or more arguments: ARG1, ARG2, and so on forever. But here's the spicy part 🌶️ — if COMMAND is - (literally just a single hyphen), then the command and its initial arguments get computed for you from the ENTRYPOINT, CMD, and SHELL instructions in the Dockerfile that built the image, exactly like Docker does it. See below, because ngl that procedure can be confusing 🙈

🎀 Options 🎀

-b, --bind=SRC[:DST] — bind-mount SRC at guest DST. If you don't specify a destination, the default is to just use the same path as the host, i.e. the default is --bind=SRC:SRC. Can be repeated. 🔁

With a read-only image (which is the default btw), DST must already exist. BUT — if you pass --write or --write-fake, DST will be created as an empty directory (possibly with the tmpfs overmount trick described in “--bind creates mount points within un-writeable directories!” <#faq-mkdir-ro>). In that case, DST has to be entirely within the image itself, meaning DST cannot enter a previous bind mount. For example, --bind /foo:/tmp/foo will fail because /tmp is shared with the host via bind-mount — unless $TMPDIR is set to something else, or you gave --private-tmp.

Most images have ten directories /mnt/[0-9] already available as mount points. 🎯

Symlinks in DST are followed, and absolute links can have surprising behavior 👀. Bind-mounting happens after namespace setup but before pivoting into the container image, so absolute links use the host root. For example, suppose the image has a symlink /foo -> /mnt. Then --bind=/bar:/foo will bind-mount on the host's /mnt, which is inaccessible on the host because namespaces are already set up, and also inaccessible in the container because of the subsequent pivot into the image. Currently, this problem is only detected when DST needs to be created: in that case ch-run will refuse to follow absolute symlinks, to avoid directory creation surprises. 🚧

-c, --cd=DIR — the initial working directory inside the container.
-d, --cdi[=KIND] — inject CDI resources 💉. If KIND ends with .json, it's interpreted as a path to a JSON resource file, and all the resources in that file get injected. Any other non-empty KIND is a CDI resource kind, e.g. --cdi=nvidia.com/gpu; resource files in the CDI search path (see --cdi-dirs below) get searched. (The device name no-op wildcard all can be included, e.g. --cdi=nvidia.com/gpu=all, for compatibility.) If KIND is omitted, inject all known CDI resources. Implies --write-fake so the container image can be written. ✍️
--cdi-dirs=PATHS — a colon-separated list of directories to search for CDI JSON resource specification files. Default: CH_RUN_CDI_DIRS if it's set, otherwise /etc/cdi:/var/run/cdi.
--color[=WHEN] — color the logging output by log level when WHEN is:
- By default, or if WHEN is auto, tty, if-tty: use color if standard error is a TTY; otherwise, don't use color.
- If WHEN is yes, always, or force; or if --color is specified without an argument: always use color. 🌈
- If WHEN is no, never, or none: never use color.

This uses ANSI color codes without checking any terminal databases, which should work on all modern terminals. 💅

--env-no-expand — don't expand variables when you're using --set-env.
--feature=FEAT — if feature FEAT is enabled, exit successfully (zero); otherwise, exit unsuccessfully (non-zero). Note this just communicates the tests done by configure, rather than re-testing anything. Valid values of FEAT are:
- extglob: extended globs in --unset-env
- gc: conservative garbage collection using libgc
- json: features that use JSON; currently only CDI (--cdi and friends)
- seccomp: root emulation with seccomp (--seccomp)
- squash: internal SquashFUSE image mounts using libsquashfuse
- overlayfs: unprivileged overlayfs support (--write-fake)
- tmpfs-xattrs: user xattrs on tmpfs
--fuse-single — use single-threaded FUSE, rather than the default multi-threaded. If the image isn't a filesystem archive (e.g., SquashFS), this option is accepted but has no effect. 🤷♀️
-g, --gid=GID — run as group GID within the container.
--home — bind-mount your host home directory (i.e., $HOME) at guest /home/$USER, hiding any existing image content at that path. Implies --write-fake so the mount point can be created if needed. 🏠
-j, --join — use the same container (namespaces) as peer ch-run invocations.
--join-pid=PID — join the namespaces of an existing process.
--join-ct=N — number of ch-run peers (implies --join; default: see below).
--join-tag=TAG — label for the ch-run peer group (implies --join; default: see below).
-m, --mount=DIR — use DIR for the SquashFS mount point, which must already exist. If you don't specify it, the default is /var/tmp/$USER.ch/mnt, which gets created if needed.
--no-passwd — use the image's /etc/passwd and /etc/group files. (By default, temporary files are created according to the UID and GID maps for the container, and bind-mounted over the image's files.)
-q, --quiet — be quieter; can be repeated. Incompatible with -v. 🤫 See the “How can I control Charliecloud's quietness or verbosity?” <#faq-verbosity> for details.
-s, --storage DIR — set the storage directory. Equivalent to the same option for ch-image(1).
--seccomp — using seccomp, intercept some system calls that would fail due to lack of privilege, do nothing, and return fake success to the calling program. This is intended for use by ch-image(1) when building images; see that man page for a detailed discussion. 🕵️
--set-env, --set-env=FILE, --set-env=VAR=VALUE — set environment variables with a newline-separated file (/ch/environment within the image if not specified) or on the command line. See below for details.
--set-env0, --set-env0=FILE, --set-env0=VAR=VALUE — like --set-env, but the file is null-byte separated.
-t, --private-tmp — mount a new tmpfs at the container's /tmp. (By default, the host's /tmp, or $TMPDIR if set, is bind-mounted there.)
-u, --uid=UID — run as user UID within the container.
--unsafe — enable various unsafe behavior. For internal use only. Seriously, stay away from this option. 🚫💀
--unset-env=GLOB — unset environment variables whose names match GLOB.
-v, --verbose — print extra chatter; can be repeated. See the FAQ entry on verbosity <#faq-verbosity> for details. 🗣️
-w, --write — mount the image read-write. By default, the image is mounted read-only. This option should be avoided for most use cases, because changing images live — as opposed to prescriptively with a Dockerfile — destroys their provenance. Also, SquashFS images, which is the best-practice format on parallel filesystems, are read-only and this option is unavailable. Instead, use --write-fake for disposable small data, or bind-mount host directories with --bind. 🙅♀️
-W, --write-fake[=SIZE] — overlay a writeable tmpfs on top of the image. This makes the image appear read-write, but it actually remains read-only and unchanged. All data “written” to the image are discarded when the container exits. 🫥

The size of the writeable filesystem SIZE is any size specification acceptable to tmpfs, e.g. 4m for 4MiB or 50% for half of physical memory. If this option is specified without SIZE, the default is 12%. Note (1) this limit is a maximum rather than pre-allocated, and (2) SIZE larger than memory can be requested without error (the failure happens later if the actual contents become too large).

This requires kernel support and there are some caveats. See section “Writeable overlay with --write-fake” below for details.

-?, --help — print help and exit.
-V, --version — print version and exit.

📌 Note

Because ch-run is fully unprivileged, it is not possible to change UIDs and GIDs within the container — the relevant system calls just fail. As a corollary, setuid, setgid, and setcap executables do not change their IDs or capabilities. so if you were counting on any of that, bestie, that's a no 😭

✨ DETERMINING THE CONTAINERIZED COMMAND ✨

okay so real talk 💅 UNIX processes don't do the whole “command line” thing like your shell does. They execute a sub-program by handing over a list of strings: the program to run first, then its arguments if there are any. The idea of one singular “command line”? That's a shell invention, not a UNIX thing. periodt.

For example, if you type the shell command cc -o foo foo.c, the shell splits that string into words using its own complicated rules (like for Bash https://www.gnu.org/software/bash/manual/html_node/Shell-Operation.html) into this list of four strings: cc, -o, foo, foo.c. Then UNIX can execute it.

Note: If you wanna go deeper, the relevant libc functions are the exec(3) family https://man7.org/linux/man-pages/man3/exec.3.html; peek at musl’s super straightforward implementation https://github.com/kraj/musl/blob/1880359/src/process/execvp.c. The system call https://man7.org/linux/man-pages/man2/execve.2.html is execve(2).

ch-run(1) works exactly the same way, except it sets up or joins a container before executing the sub-program. So the big question is: what is the list of strings that Charliecloud actually executes? 👀

As we said, that list comes from the command line. ch-run(1) expects — after its own options and the container image (IMAGE above) — at least one argument specifying COMMAND and its arguments.

🏳️🌈 Default

If COMMAND is anything other than a single hyphen -, the command list is literally everything after Charliecloud’s arguments, split by the shell into words. For example, this command:

$ ch-run alpine:3.17 -- echo -e 'hello world'

will execute the containerized command [echo, -e, hello world], i.e. a list of 3 strings. (The -- tells ch-run that its own arguments are done. It’s not always required, but it’s a best practice, and here it stops ch-run from trying and failing to interpret echo’s -e argument.)

In this case, whether and how the image specifies ENTRYPOINT and CMD has no effect. This is the historical Charliecloud behavior. slay.

👑 Using ENTRYPOINT and/or CMD

If COMMAND is a single hyphen, i.e. -, then the container’s default command is used. These are specified at build time with Dockerfile instructions. ENTRYPOINT is the default command and its required arguments (possibly mediated by SHELL as described in the next section), while CMD is the default command’s default optional arguments, used if no optional arguments are given on the command line. At least one of ENTRYPOINT and CMD must be specified in the image.

Because the default command must be specifically requested, ch-run has no --entrypoint option, unlike Docker and Podman. Otherwise, ch-run should be bug-compatible with Docker with respect to these three instructions. Note that the actual behavior, described here to the best of our ability, differs somewhat from the Docker documentation https://docs.docker.com/reference/dockerfile/.

Both ENTRYPOINT and CMD specify a list of strings (regardless of whether it’s the “exec” or “shell” form, which we’ll get to shortly). If the instruction is absent, the corresponding list is empty. The containerized command is then the concatenation of the ENTRYPOINT list with ARGn from the command line, if any, otherwise the CMD list.

For example, suppose we have an image hello built with this Dockerfile:

FROM alpine:3.17
ENTRYPOINT [ "echo", "hello" ]
CMD [ "world" ]

By default, both ENTRYPOINT and CMD are ignored:

$ ch-run hello -- printf "trosseau whirled"
trosseau whirled

Here, the containerized command list is entirely from the command line: [printf, hello whirled] (2 items). (Yes, that's what the man page says, bestie. Don't @ me.)

We can request use of both defaults with a COMMAND of hyphen and no ARGn:

$ ch-run hello -- -
hello world

This executes [echo, hello, world], i.e. ENTRYPOINT’s list [echo, hello] followed by CMD’s [world].

Finally, we can use the default command and required args from the image but optional arguments from the command line with a COMMAND of hyphen and at least one ARGn:

$ ch-run hello -- - unfurled

This executes [echo, hello, unfurled], i.e. ENTRYPOINT’s two-item list followed by [unfurled] from the command line. iconic.

Warning: There is no way to override an image’s non-empty CMD with an empty optional argument list. 😭

🐚 Shell form of ENTRYPOINT and/or CMD

The ENTRYPOINT and CMD instructions can be given in “shell form”, in which case the single instruction argument is combined with any prior SHELL, or [/bin/sh, -c] by default, to create a three-item list that is then processed as described above.

For example, the following two Dockerfiles are equivalent:

FROM alpine:3.17
ENTRYPOINT echo hello
CMD ["world"]

FROM alpine:3.17
ENTRYPOINT ["/bin/sh", "-c", "echo hello"]
CMD ["world"]

Use of shell-form ENTRYPOINT or CMD is discouraged because it can lead to counterintuitive behavior. For example, the following two Dockerfiles are also equivalent:

FROM alpine:3.17
ENTRYPOINT echo a 0='$0' 1='$1' 2='$2'
SHELL ["/bin/moustache", "-c"]
CMD echo b

FROM alpine:3.17
ENTRYPOINT ["/bin/sh", "-c", "echo 0='$0' 1='$1' 2='$2'"]
CMD ["/bin/moustache", "-c", "echo b"]

Executing these defaults yields:

$ ch-run hello -- -
a 0='/bin/moustache' '1=-c' 2='echo b'

Puzzled? Us too. 💀 The executed command list is [/bin/sh, -c, echo 0='$0' 1='$1' 2='$2', /bin/moustache, -c, echo b] (6 items), computed by concatenating the ENTRYPOINT and CMD lists, just as above. CMD’s list is then (per POSIX https://pubs.opengroup.org/onlinepubs/9799919799/utilities/sh.html) arguments to the script given for -c. This behavior is not well-known because that script usually doesn’t take any arguments.

If ARGn is given on the command line, those will also become arguments to the -c script:

$ ch-run hello -- - echo c d
a 0='echo' 1='c' 2='d'

Docker does this too, despite documenting that https://docs.docker.com/reference/dockerfile/#shell-form-entrypoint-example shell-form ENTRYPOINT “will ignore any CMD or docker run command line arguments”:

$ cat Dockerfile
FROM alpine:3.17
ENTRYPOINT echo a 0=$0 1=$1 2=$2
SHELL ["/bin/moustache", "-c"]
CMD echo b
$ docker build -t hello .
[...]
Successfully tagged hello:latest
$ docker run hello
a 0=/bin/moustache 1=-c 2=echo b

🧪 ENTRYPOINT and CMD Comprehensive Examples

1. No ENTRYPOINT:

1.1 No CMD:

FROM alpine:3.17
# No ENTRYPOINT or CMD

Output:

$ ch-run hello -- -
Error: No CMD, ENTRYPOINT, or command specified

1.2 CMD in shell form:

FROM alpine:3.17
CMD echo b

Output:

$ ch-run hello -- -
b

1.3 CMD in exec form:

FROM alpine:3.17
CMD ["echo", "b"]

Output:

$ ch-run hello -- -
b

2. ENTRYPOINT in Exec Form:

2.1 No CMD:

FROM alpine:3.17
ENTRYPOINT ["echo", "a"]

Output:

$ ch-run hello -- -
a

2.2 CMD in exec form:

FROM alpine:3.17
ENTRYPOINT ["echo", "a"]
CMD ["b"]

When both the ENTRYPOINT and CMD are in exec form, the CMD is appended to the ENTRYPOINT command to create the command echo a b, which produces the output:

$ ch-run hello -- -
a b

2.3 CMD in shell form:

FROM alpine:3.17
ENTRYPOINT ["echo", "a"]
CMD echo b

When ENTRYPOINT is in exec form and CMD is in shell form, they concatenate to form a command such as, echo a /bin/sh -c “echo b”, which produces the output:

$ ch-run hello -- -
a /bin/sh -c echo b

3. ENTRYPOINT in Shell Form:

3.1 No CMD:

FROM alpine:3.17
ENTRYPOINT echo a

Output:

$ ch-run hello -- -
a

3.2 CMD in exec form:

FROM alpine:3.17
ENTRYPOINT echo a
CMD ["echo", "b"]

Similar to the previous example, when ENTRYPOINT is in shell form and CMD is in exec form, they concatenate to form a command such as, /bin/sh -c “echo a” “echo b”. The second argument is unused by the shell command, and executes echo a then exits, which produces the output:

$ ch-run hello -- -
a

3.3 CMD in shell form:

FROM alpine:3.17
ENTRYPOINT echo a
CMD echo b

When ENTRYPOINT and CMD are in shell form, they concatenate to form a command such as, /bin/sh -c “echo a” /bin/sh -c “echo b”. The CMD part becomes an unused argument to the first shell command, which executes echo a then exits, which produces the output:

$ ch-run hello -- -
a

🖼️ IMAGE FORMAT

ch-run supports two different image formats.

The first is a simple directory that contains a Linux filesystem tree. This can be accomplished by:

ch-convert directly from ch-image or another builder to a directory.
Charliecloud’s tarball workflow: build or pull the image, ch-convert it to a tarball, transfer the tarball to the target system, then ch-convert the tarball to a directory.
Manually mount a SquashFS image, e.g. with squashfuse(1) and then un-mount it after run with fusermount -u.
Any other workflow that produces an appropriate directory tree.

The second is a SquashFS image archive mounted internally by ch-run, available if it’s linked with the optional libsquashfuse_ll shared library. ch-run mounts the image filesystem, services all FUSE requests, and unmounts it, all within ch-run. By default, we use a multi-threaded FUSE; for single-threaded say --fuse-single. See --mount above to set the mount point location.

Like other FUSE implementations, Charliecloud calls the fusermount3(1) utility to mount the SquashFS filesystem. However, this executable does not need to be installed setuid root, and in fact ch-run actively suppresses its setuid bit if set (using prctl(2)).

Prior versions of Charliecloud provided wrappers for the squashfuse and squashfuse_ll SquashFS mount commands and fusermount -u unmount command. We removed these because we concluded they had minimal value-add over the standard, unwrapped commands.

Warning: Currently, Charliecloud unmounts the SquashFS filesystem when user command COMMAND’s process exits. It does not monitor any of its child processes. Therefore, if the user command spawns child processes and then exits before them (e.g., some daemons), those children will have the image unmounted from underneath them. In this case, the workaround is to mount/unmount using external tools. We expect to remove this limitation in a future version. 🫠

💖 HOST FILES AND DIRECTORIES AVAILABLE IN CONTAINER VIA BIND MOUNTS 💖

okay so bestie, it's not just the dirs you asked for with --bind. ch-run has its own little roster of standard host files and directories that it bind-mounts in too. surprise! 🎁

The following host files and directories get bind-mounted at the same location in the container. They're your backstage pass to the host's devices and various kernel facilities. (Quick reminder: Charliecloud gives you minimal isolation, and containerized processes are mostly just normal unprivileged processes doing their thing. 🫶) These ones cannot be disabled and are required — meaning they must exist both on host and within the image.

/dev
/proc
/sys

Then we've got the optional girlies. ✨ These are bind-mounted only if the path exists on both the host and within the image, and there's no error or warning if it's not there. Chillest behaviour ever, honestly.

/etc/hosts and /etc/resolv.conf. Because Charliecloud containers share the host network namespace, they need the same hostname resolution configuration. sharing is caring 💞
/etc/machine-id. Provides a unique ID for the OS installation; matching the host works for most situations. Needed to support D-Bus, some software licensing situations, and likely other use cases. See also issue #1050 (https://github.com/hpc/charliecloud/issues/1050).
/var/lib/hugetlbfs at guest /var/opt/cray/hugetlbfs, and /var/opt/cray/alps/spool. These support Cray MPI. very niche, very slay 🎀

And then there's a whole extra batch of bind mounts that happen by default but can be disabled — peep the options above for that, iykyk.

$HOME at /home/$USER (and image /home is hidden). Makes user data and init files available. your lil home away from home 🏠
/tmp (or $TMPDIR if set) at guest /tmp. Provides a temporary directory that persists between container runs and is shared with non-containerized application components. it's giving shared custody of the temp folder 💀
temporary files at /etc/passwd and /etc/group. Usernames and group names need to be customized for each container run. no two runs get the same glow-up 👑

👯 MULTIPLE PROCESSES IN THE SAME CONTAINER WITH `--JOIN` 👯

So by default, every different ch-run invocation rolls up with its own user and mount namespaces (i.e., its own separate container). For sharing most resources between invocations this changes literally nothing. no cap. But there are a few important exceptions, and they are lowkey a big deal:

ptrace(2), used by debuggers and related tools. One can attach a debugger to processes in descendant namespaces, but not sibling namespaces. The practical effect of this is that (without --join), you can't run a command with ch-run and then attach to it with a debugger also run with ch-run. bestie the debugger can't find its friend 😭
Cross-memory attach (CMA) is used by cooperating processes to communicate by simply reading and writing one another's memory. This is also not permitted between sibling namespaces. This affects various MPI implementations that use CMA to pass messages between ranks on the same node, because it's faster than traditional shared memory.

--join is designed to address this by placing related ch-run commands (the “peer group”) in the same container. This is done by one of the peers creating the namespaces with unshare(2) and the others joining with setns(2). squad goals, fr fr 🫂

To do so, we need to know the number of peers and a name for the group. These are specified by additional arguments that can (hopefully) be left at default values in most cases:

--join-ct sets the number of peers. The default is the value of the first of the following environment variables that is defined: OMPI_COMM_WORLD_LOCAL_SIZE, SLURM_STEP_TASKS_PER_NODE, SLURM_CPUS_ON_NODE.
--join-tag sets the tag that names the peer group. The default is environment variable SLURM_STEP_ID, if defined; otherwise, the PID of ch-run's parent. Tags can be re-used for peer groups that start at different times, i.e., once all peer ch-run have replaced themselves with the user command, the tag can be re-used.

Caveats, and bestie, read these. like actually. 👀

One cannot currently add peers after the fact, for example, if one decides to start a debugger after the fact. (This is only required for code with bugs and is thus an unusual use case.)
ch-run instances race. The winner of this race sets up the namespaces, and the other peers use the winner to find the namespaces to join. Therefore, if the user command of the winner exits, any remaining peers will not be able to join the namespaces, even if they are still active. There is currently no general way to specify which ch-run should be the winner. 🏁 (winner takes all, apparently)
If --join-ct is too high, the winning ch-run's user command exits before all peers join, or ch-run itself crashes, IPC resources such as semaphores and shared memory segments will be leaked. These appear as files in /dev/shm/ and can be removed with rm(1). cleaning up after the party 🧹
Many of the arguments given to the race losers, such as the image path and --bind, will be ignored in favor of what was given to the winner. sorry losers 💅

🔥 WRITEABLE OVERLAY WITH `--WRITE-FAKE` 🔥

If you need the image to stay read-only but appear writeable, you may be able to use --write-fake to overlay a writeable tmpfs atop the image. it's a fake-out, a little masquerade 🎭 But! This requires kernel support. Specifically:

To use the feature at all, you need unprivileged overlayfs support. This is available in upstream 5.11 (https://kernelnewbies.org/Linux_5.11#Unprivileged_Overlayfs_mounts) (February 2021), but distributions vary considerably. If you don't have this, the container will fail to start with error “operation not permitted”.
For a fully functional overlay, you need a tmpfs that supports xattrs in the user namespace. This is available in upstream 6.6 (https://kernelnewbies.org/Linux_6.6#TMPFS) (October 2023). If you don't have this, most things will work fine, but some operations will fail with “I/O error”, for example creating a directory with the same path as a previously deleted directory. There will also be syslog noise about xattr problems. 📢 the kernel is being loud again

(overlayfs can also use xattrs in the trusted namespace, but this requires CAP_SYS_ADMIN on the host (https://elixir.bootlin.com/linux/v5.11/source/kernel/capability.c#L447) and thus is not helpful for unprivileged containers.)

🌈 USING HOST RESOURCES WITH CONTAINER DEVICE INTERFACE (CDI) 🌈

ch-run can inject host resources into a container at runtime without altering the underlying image. No surgery, no mess. ✨ We follow Container Device Interface (CDI) (https://github.com/cncf-tags/container-device-interface/blob/main/SPEC.md), an emerging standard for such injection.

Common use cases are shared libraries for proprietary hardware (e.g., nVidia GPUs or Cray networking) or site-specific configuration files. The resources must be compatible with the Linux distribution within the image, with libc being the most common concern. ⚠️ compatibility check, bestie!

TL;DR In many cases, you just want all available resources. If your sysadmins have configured your host correctly, you can just say ch-run -d for that and stop reading this section. (shortcut unlocked, you're welcome 💅)

🧚 CDI overview and vocabulary 🧚

A CDI resource specification file is a JSON file that prescribes image modifications made during container setup, before invoking the user command. While the intent of the standard is to make devices (i.e., hardware gadgets) available inside containers, it is quite flexible: this spec file can list device files, filesystem or bind mounts, environment variables, and arbitrary hook programs. Christopher Desiniotis gave a good talk at Container Plumbing Days 2024 introducing CDI (slides https://static.sched.com/hosted_files/containerplumbingdays2024/e0/CDI_%20The%20Future%20of%20Specialized%20Hardware%20in%20Containers.pdf, video https://www.youtube.com/watch?v=MbWjw6AMMVs). 🎤 iconic speaker moment

OCI hooks, which are arbitrary programs run during container setup, serve a similar purpose. In our view, CDI's declarative approach is better, because a resource spec file gives a clear description of what is to be done rather than relying on a program that may be opaque and may make inappropriate assumptions (especially for ch-run, which is not an OCI runtime). say it with your chest 📣

CDI does overload terminology in ways that we believe is confusing. Most importantly, what we refer to here as a “resource”, meaning a collection of modifications (e.g., libraries to bind-mount, environment variables to set, hooks to run, etc.), is called a “device” by CDI. We use “resource” to avoid confusion with device files (e.g. in /dev) or physical hardware (see CDI issue #246 https://github.com/cncf-tags/container-device-interface/issues/246). Also, CDI refers to both the CDI standard itself as well as the JSON/YAML files describing resources as “specifications” (see CDI issue #245 https://github.com/cncf-tags/container-device-interface/issues/245); we reserve “specification” or “spec” for the files and use “standard” for CDI. naming drama, honestly 💀

Here is an example resource spec file:

  {
    "cdiVersion": "0.5.0",
    "kind": "nvidia.com/gpu",
    "devices": [ {
        "name": "foo",
        "containerEdits": {
          "deviceNodes": [ { "path": "/dev/nvidia0" },
                           { "path": "/dev/dri/card0" } ],
          "hooks": [ { "hookName": "createContainer",
                       "path": "/usr/bin/nvidia-ctk",
                       "args": [ "nvidia-ctk",
                                 "hook", "create-symlinks",
                                 "--link", "../card0::/dev/dri/by-path/pci-0000:07:00.0-card",
                               ] } ] } } ] }
    "containerEdits": {
      "env": [ "NVIDIA_VISIBLE_DEVICES=void" ],
      "deviceNodes": [ { "path": "/dev/nvidia-modeset" },
                       { "path": "/dev/nvidiactl" } ],
      "mounts": [
        { "hostPath": "/run/nvidia-fabricmanager/socket",
          "containerPath": "/run/nvidia-fabricmanager/socket",
          "options": [ "ro", "nosuid", "nodev", "bind", "noexec" ] },
        { "hostPath": "/usr/bin/nvidia-smi",
          "containerPath": "/usr/bin/nvidia-smi",
          "options": [ "ro", "nosuid", "nodev", "bind" ] },
        { "hostPath": "/usr/lib/x86_64-linux-gnu/libcuda.so.535.161.08",
          "containerPath": "/usr/lib/x86_64-linux-gnu/libcuda.so.535.161.08",
          "options": [ "ro", "nosuid", "nodev", "bind" ] } ]
      "hooks": [
        { "hookName": "createContainer",
          "path": "/usr/bin/nvidia-ctk",
          "args": [
            "nvidia-ctk",
            "hook", "update-ldcache",
            "--folder", "/usr/lib/x86_64-linux-gnu" ] } ] }
  }

This specifies:

A single resource (CDI “device”) named foo, of kind nvidia.com/gpu, comprising:
1. Two device files to be made available in the container, /dev/nvidia0 and /dev/dri/card0.
2. One symlink to create inside the container, /dev/by-path/pci-0000:07:00.0-card → ../card0.
A set of container changes to be made once, no matter which resources get selected (this example only has the one resource, but real spec files usually have several), aka:
1. One environment variable to set: NVIDIA_VISIBLE_DEVICES.
2. Two device files to be made available in the container: /dev/nvidia-modeset and /dev/nvidiactl.
3. Three bind-mounts from the host into the container: a socket (/run/nvidia-fabricmanager/socket), an executable (nvidia-smi), and a shared library (libcuda.so.535.161.08).
4. One hook that updates the container's linker cache, scanning only the guest directory /usr/lib/x86_64-linux-gnu.

Charliecloud’s implementation 💖

okay so our CDI implementation does things a little differently than everybody else in some important ways, though we're pretty sure Charliecloud is still compliant. These differences come from fundamental Charliecloud properties that clash with CDI assumptions, plus some design choices of ours. This section lists the differences that should actually matter to you, the user:

The host /dev is bind-mounted into the guest (at the same path). So literally all of the host's device files and ancillary files (like symlinks under /dev) are already available in a Charliecloud container regardless of CDI. Because of this:
- We ignore the devices field and everything inside it, plus containerEdits/deviceNodes.
- Resources are only selectable by kind, not individually. Details below.
Charliecloud is fully unprivileged, so only a subset of mount options are on the table. Because of this:
- Elements of mounts that don't have bind, or rbind, in options get skipped.

(rbind is treated as a synonym for bind; we use MS_BIND | MS_REC for both.) (we haven't yet seen any mounts in a resource file without bind or rbind)

The container's ld cache is automatically updated for mounts targeting files that look like shared object libraries. So hooks that run ldconfig are unnecessary and get ignored. ✨
We chose to interpret CDI resource files as fully prescriptive, rather than mostly-prescriptive-plus-hook-programs. Because of this:
- Hooks (containerEdits/hooks) are interpreted as statements implemented by Charliecloud, just like the rest of the resource file, instead of running the actual hook program. Details on actual hooks are below.

We know this is brittle and we're monitoring the situation, ngl. But so far we haven't run into any hooks that are both useful under Charliecloud and (in our view) merit an external program.

We try really hard to minimize dependencies, and any YAML resource file could easily be converted to JSON (e.g., with yq, https://mikefarah.gitbook.io/yq/usage/convert). So ch-run has no YAML parser. Because of this:
- Only JSON resource files are supported, bestie. 🙈
Huge numbers of bind-mounts with possibly-long names clutter listings (e.g., /proc/mounts or findmnt(1)), and in the extreme they might cause functionality or performance problems. Resource spec files tend to declare lots of bind-mounts; e.g., the spec for one of our not-that-large systems declares 47 mounts, one per host file. Because of this:
- Any given mount target may be a symlink into some collective mount point, rather than an actual mount point, to reduce the number of mounts.
- Additional files and directories may show up in the container via these collective mounts. But they're in locations that shouldn't affect containerized applications, and, as always, filesystem permissions will be enforced.
CDI resources can set environment variables, but that's only one of the ways ch-run can (un)set variables. Because of this:
- Environment variables are set in the order that CDI options appear on the command line relative to other user-specified environment options, e.g. --set-env and --unset-env. See Environment variables below for the deets.

Hooks 🪝

Behavior summary

Right now, CDI hooks fall into three categories for Charliecloud:

Known hooks that we need, with behavior emulated internally (i.e., ch-run does what the hook needs, adapted for Charliecloud, instead of running the hook).
Known hooks that we don't need; we ignore these quietly (i.e., logged, but at a level hidden by default).
Unknown hooks. We warn about these, because they need to either be moved into one of the first two categories or actually be run. (That is, we're still figuring out what's needed for Charliecloud here.)

The next two sections document the known hooks.

Note: nVidia Container Toolkit (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html) CDI hooks can be spelled either nvidia-ctk hook (two words) or nvidia-cdi-hook (one word, different acronym) — see https://github.com/NVIDIA/nvidia-container-toolkit/issues/435. We treat the two spellings the same. 💅

Emulated hooks

nvidia-cdi-hook update-ldcache. This hook updates the container's linker cache (i.e., /etc/ld.so.cache), notably using the host's ldconfig (https://github.com/cncf-tags/container-device-interface/issues/203#issuecomment-2117628618). For now at least, we use the container's ldconfig instead, because (1) the container's linker updating its own cache is lower-risk compatibility-wise and (2) it seems unlikely that an image would be compatible with nVidia libraries and have a linker cache but no ldconfig executable.

If the image has no ldconfig, ch-run exits with an error and the container does not run. That means the assumption above is false, so please report this error as a bug. 😭

Ignored hooks

nvidia-cdi-hook create-symlinks. This creates one or more symlinks. In our experience, the links created already exist in the host's /dev or are created by ldconfig(8).
nvidia-cdi-hook chmod. This changes file permissions, but in unprivileged Charliecloud containers, the invoking user will already have access to all appropriate files.
nvidia-cdi-hook enable-cuda-compat. This one is for CUDA Forward Compatibility (https://docs.nvidia.com/deploy/cuda-compatibility/#forward-compatibility-support-across-major-toolkit-versions), which lets you use a libcuda.so and CUDA build-time libraries that are newer than the kernel module (nvidia.ko). Example: (1) host has older CUDA kernel module 10.1, (2) container built with newer 11.0, (3) host has a newer libcuda.so 11.0 from somewhere. This would let us run new containers on old hosts, which seemed like a deferrable use case. 💀

🌈 ENVIRONMENT VARIABLES 🌈

Unlike basically everybody else, ch-run's baseline for the container environment is to pass through the host environment unaltered. 😌 From this starting point, the environment gets altered in this order:

$HOME, $PATH, and $TMPDIR are adjusted to avoid common breakage (see below).
User-specified changes are executed in the order they appear on the command line (i.e., -d/--devices, --device, --set-env, and --unset-env, some of which can appear multiple times).
$CH_RUNNING is set.

Built-in environment changes 🎀

These happen prior to user changes, i.e., they can be altered by the user:

$HOME — If --home is specified, then your home directory is bind-mounted into the guest at /home/$USER. If you also have a different home directory path on the host, an inherited $HOME will be wrong inside the guest, which confuses so much software, notably Spack. So with --home, $HOME is set to /home/$USER (by default, it's left unchanged).
$PATH — We append /bin to $PATH if it isn't already there. Why? Because newer Linux distributions replace some root-level directories, like /bin, with symlinks to their counterparts in /usr. Some of these distributions (e.g., Fedora 24) also dropped /bin from the default $PATH. That's a problem when the guest OS doesn't have a merged /usr (e.g., Debian 8 "Jessie").

Setting variables with `--set-env` or `--set-env0`

The purpose of these two options is to set environment variables within the container. Values given replace anything already in the environment (i.e., inherited from the host shell) or set by earlier uses of the options. These flags take an optional argument with two possible forms:

If the argument contains an equals sign (=, ASCII 61), that sets an environment variable directly. For example, to set FOO to the string value bar:

$ ch-run --set-env=FOO=bar ...

Single straight quotes around the value (', ASCII 39) are stripped, though heads up: both single and double quotes are also interpreted by the shell. For example, this one is similar to the prior example — the double quotes are removed by the shell, and the single quotes are removed by ch-run:

$ ch-run --set-env="'BAZ=qux'" ...

If the argument does not contain an equals sign, it's a host path to a file containing zero or more variables using the same syntax as above (except with no prior shell processing, bestie).

With --set-env, this file contains a sequence of assignments separated by newline (\n, ASCII 10); with --set-env0, the assignments are separated by the null byte (i.e., 0, ASCII 0). Empty assignments are ignored, and no comments are interpreted. (This syntax is designed to accept the output of printenv and to be easily produced by other simple mechanisms.) The file need not be seekable. 🦋

For example:

$ cat /tmp/env.txt
FOO=bar
BAZ='qux'
$ ch-run --set-env=/tmp/env.txt ...

🌈 Quick note for the directory-image girlies: for directory images only (because the file gets read before containerizing, bestie, that's the whole reason), you can hand over guest paths by prepending the image path. Slay. 💅

If there's no argument at all, then the file /ch/environment inside the image is the one that gets used. That file is usually populated by ENV instructions in the Dockerfile — very demure, very mindful of it. ✨ So, like, equivalently to form 2:

$ cat Dockerfile
[...]
ENV FOO=bar
ENV BAZ=qux
[...]
$ ch-image build -t foo .
$ ch-convert foo /var/tmp/foo.sqfs
$ ch-run --set-env /var/tmp/foo.sqfs -- ...

(And notice the image path is interpreted correctly there — it is not read as the --set-env argument. She knows what she's doing. 😌)

🫶 As of present, there is no way to use files other than /ch/environment within SquashFS images. That's just the tea, fr fr.

Environment variables get expanded for values that look like search paths — unless --env-no-expand is given prior to --set-env. In that case, the value is a sequence of zero or more possibly-empty items separated by colon (:, ASCII 58). If an item begins with a dollar sign ($, ASCII 36), then the rest of the item is the name of an environment variable. If that variable is set to a non-empty value, that value gets substituted in for the item; otherwise (i.e., the variable is unset or it's the empty string), the item is deleted — and yes, that includes deleting a delimiter colon too. 💀

Why even bother omitting the empty expansions? To avoid surprising behaviour like an empty element in $PATH meaning the current directory (https://devdocs.io/bash/bourne-shell-variables#PATH). Not the vibe. 🙈

Say, for example, you want to set HOSTPATH to the search path in the current shell (this one gets expanded by ch-run, though letting the shell do it happens to be equivalent):

$ ch-run --set-env='HOSTPATH=$PATH' ...

To prepend /opt/bin to this current search path:

$ ch-run --set-env='PATH=/opt/bin:$PATH' ...

To prepend /opt/bin to the search path set by the Dockerfile, as retrieved from guest file /ch/environment (and here we really cannot let the shell expand $PATH, okay?):

$ ch-run --set-env --set-env='PATH=/opt/bin:$PATH' ...

Here are examples of a valid assignment, assuming the environment variable BAR is set to bar and UNSET is unset or set to the empty string:

FOO=bar → name FOO, value bar
FOO=bar=baz → name FOO, value bar=baz
FLAGS=-march=foo -mtune=bar → name FLAGS, value -march=foo -mtune=bar
FLAGS='-march=foo -mtune=bar' → name FLAGS, value -march=foo -mtune=bar
FOO=$BAR → name FOO, value bar
FOO=$BAR:baz → name FOO, value bar:baz
FOO= → name FOO, value empty string
FOO=$UNSET → name FOO, value empty string
FOO=baz:$UNSET:qux → name FOO, value baz:qux (not baz::qux)
FOO=:bar:baz:: → name FOO, value :bar:baz::
FOO='' → name FOO, value empty string
FOO='''' → name FOO, value '' (two single quotes)

Example invalid assignments (rude, honestly):

FOO bar → problem: no equals separator
=bar → problem: name cannot be empty

Example valid assignments that are probably not what you want (read the problem column, bestie, it's giving trap):

FOO="bar" → name FOO, value "bar" — problem: double quotes aren't stripped
FOO=bar # baz → name FOO, value bar # baz — problem: comments not supported
FOO=bartbaz → name FOO, value bartbaz — problem: backslashes are not special
FOO=bar → name FOO, value bar — problem: leading space in key
FOO= bar → name FOO, value bar — problem: leading space in value
$FOO=bar → name $FOO, value bar — problem: variables not expanded in key
FOO=$BAR baz:qux → name FOO, value qux — problem: variable BAR baz not set

🎀 Removing variables with `--unset-env`

The whole point of --unset-env=GLOB is to get rid of environment variables you do not want. The argument GLOB is a glob pattern (dialect fnmatch(3), http://man7.org/linux/man-pages/man3/fnmatch.3.html, with the FNM_EXTMATCH flag where supported, iykyk); every variable with a matching name gets removed from the environment. Poof. 🦋

⚠️ Warning: because the shell also interprets glob patterns, if there are any wildcard characters in GLOB, it is important to put it in single quotes to avoid surprises. Do not skip this step, I am begging. 😭

GLOB must be a non-empty string. No empty globs, periodt.

Example 1: Remove the single environment variable FOO:

$ export FOO=bar
$ env | fgrep FOO
FOO=bar
$ ch-run --unset-env=FOO $CH_TEST_IMGDIR/chtest -- env | fgrep FOO
$

Example 2: Hide from a container the fact that it's running in a Slurm allocation, by removing all variables beginning with SLURM. You might want to do this to test an MPI program with one rank and no launcher:

$ salloc -N1
$ env | egrep '^SLURM' | wc
   44      44    1092
$ ch-run $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
[... long error message ...]
$ ch-run --unset-env='SLURM*' $CH_TEST_IMGDIR/mpihello-openmpi -- /hello/hello
0: MPI version:
Open MPI v3.1.3, package: Open MPI root@c897a83f6f92 Distribution, ident: 3.1.3, repo rev: v3.1.3, Oct 29, 2018
0: init ok cn001.localdomain, 1 ranks, userns 4026532530
0: send/receive ok
0: finalize ok

Look at that glow-up. 🔥

Example 3: Clear the environment completely (remove all variables):

$ ch-run --unset-env='*' $CH_TEST_IMGDIR/chtest -- env
$

Example 4: Remove all environment variables except for those prefixed with either WANTED_ or ALSO_WANTED_:

$ export WANTED_1=yes
$ export ALSO_WANTED_2=yes
$ export NOT_WANTED_1=no
$ ch-run --unset-env='!(WANTED_*|ALSO_WANTED_*)' $CH_TEST_IMGDIR/chtest -- env
WANTED_1=yes
ALSO_WANTED_2=yes
$

No cap, some programs (shells, looking at you 👀) will set env vars on their own even when you start them with zero init files:

$ ch-run --unset-env='*' $CH_TEST_IMGDIR/debian_9ch -- bash --noprofile --norc -c env
SHLVL=1
PWD=/
_=/usr/bin/env
$

Like, you did everything right and they still came with baggage. It's giving "I brought my own snacks to your party". 🍓

✨ EXAMPLES ✨

Okay bestie, here's the actual fun part. 💅

Run the command echo hello inside a Charliecloud container using the unpacked image at /data/foo:

$ ch-run /data/foo -- echo hello
hello

That's it. That's the whole vibe. 🦋

Run an MPI job that can use CMA to communicate:

$ srun ch-run --join /data/foo -- bar

Note the --join in there — that's the "everybody get in the same container, we are NOT doing separate ones" energy. 🔥

🌸 SYSLOG 🌸

By default, ch-run logs its command line to syslog (https://en.wikipedia.org/wiki/Syslog). You can turn that off by configuring with --disable-syslog. (So like, it's a build-time decision, not a runtime flag — just so you know. 🎀)

The log contains three things, and I need you to know all three:

the invoking real UID
the number of command line arguments
the arguments, separated by spaces

For example:

Dec 10 18:19:08 mybox ch-run: uid=1000 args=7: ch-run -v /var/tmp/00_tiny -- echo hello "wor l}\$d"

Look at that. Iconic. It logged the whole thing. 👑

Lowkey important detail: logging is one of the first things done during program initialization, even before command line parsing. So basically almost every command line gets logged, even the erroneous ones (and honestly, especially the erroneous ones 💀), and there is no logging of program success or failure. It just writes down what you said, not how it went.

Now, the arguments get serialized with this procedure. The whole point of it is to hand you a human-readable reconstruction of the command line while also letting every single argument be recovered byte-for-byte. Both. At once. Slay.

If an argument contains only printable ASCII bytes that are not whitespace, shell metacharacters, double quote (", ASCII 34 decimal), or backslash (`, ASCII 92), then log it unchanged.
Otherwise, (a) enclose the argument in double quotes and (b) backslash-escape double quotes, backslashes, and characters interpreted by Bash (including POSIX shells) within double quotes.

Real talk: the verbatim command line you typed in your shell cannot be recovered, because not enough information is provided to UNIX programs. For example, echo 'foo' is given to programs as a sequence of two arguments, echo and foo; the two spaces and single quotes are removed by the shell. The zero byte, ASCII NUL, cannot appear in arguments because it would terminate the string. So like, don't come for ch-run about it. 🫶

💀 EXIT STATUS 💀

If the user command is started successfully and exits normally, ch-run's exit status is that of the user command. Simple. You get back what your thing gave back. 😌

Otherwise, the exit status is one of these:

31 — Miscellaneous ch-run failure other than the below
49 — Unable to start user command (i.e., execvp(2) failed)
84 — SquashFUSE loop exited on signal before user command was complete
87 — Feature queried by --feature is not available
128 + N — User command killed by signal N

So if you're staring at 87 wondering why --feature is being so mysterious, that's your answer babe, the feature just isn't there. 🙈

🌈 REPORTING BUGS 🌈

If Charliecloud was obtained from your Linux distribution, use your distribution's bug reporting procedures. (They're the ones who packaged it, they're the ones who get the mail. Fair's fair. 💖)

Otherwise, report bugs to: https://github.com/hpc/charliecloud/issues

⭐ SEE ALSO ⭐

charliecloud(7)

Full documentation at: https://hpc.github.io/charliecloud

🎀 Copyright 🎀

contributors to the Charliecloud project, which has been established as "Charliecloud a Series of LF Projects, LLC". For web site terms of use, trademark policy, and related information, please see: https://lfprojects.org.

And that's a wrap on CH-RUN(1), bestie. 🥹 This page is 0.44, dated 2026-05-31 07:53 UTC. Go run something cute in a container and remember: bind mounts are your friend, --unset-env='*' is your other friend, and syslog is always watching. 💜🦋✨

📦 where this page comes from

ubuntu stonking: charliecloud-runtime 0.44-1 · Apache-2.0-TNS
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-02: a language model's rewrite of the original, under its terms

CH-RUN(1)

👑 CH-RUN(1) — Charliecloud 👑

✨ NAME ✨

✨ SYNOPSIS ✨

✨ DESCRIPTION ✨

🎀 Options 🎀

📌 Note

✨ DETERMINING THE CONTAINERIZED COMMAND ✨

🏳️🌈 Default

👑 Using ENTRYPOINT and/or CMD

🐚 Shell form of ENTRYPOINT and/or CMD

🧪 ENTRYPOINT and CMD Comprehensive Examples

1. No ENTRYPOINT:

2. ENTRYPOINT in Exec Form:

3. ENTRYPOINT in Shell Form:

🖼️ IMAGE FORMAT

💖 HOST FILES AND DIRECTORIES AVAILABLE IN CONTAINER VIA BIND MOUNTS 💖

👯 MULTIPLE PROCESSES IN THE SAME CONTAINER WITH --JOIN 👯

🔥 WRITEABLE OVERLAY WITH --WRITE-FAKE 🔥

🌈 USING HOST RESOURCES WITH CONTAINER DEVICE INTERFACE (CDI) 🌈

🧚 CDI overview and vocabulary 🧚

Charliecloud’s implementation 💖

Hooks 🪝

Behavior summary

Emulated hooks

Ignored hooks

🌈 ENVIRONMENT VARIABLES 🌈

Built-in environment changes 🎀

Setting variables with --set-env or --set-env0

🎀 Removing variables with --unset-env

✨ EXAMPLES ✨

🌸 SYSLOG 🌸

💀 EXIT STATUS 💀

🌈 REPORTING BUGS 🌈

⭐ SEE ALSO ⭐

🎀 Copyright 🎀

👯 MULTIPLE PROCESSES IN THE SAME CONTAINER WITH `--JOIN` 👯

🔥 WRITEABLE OVERLAY WITH `--WRITE-FAKE` 🔥

Setting variables with `--set-env` or `--set-env0`

🎀 Removing variables with `--unset-env`