CH-FROMHOST(1)

📦
version 0.44 · 10 versions
, in ubuntu stonking

💖 NAME 💖

ch-fromhost — inject files from the host into an image directory, with various ✨magic✨ (that's in the actual manual, bestie, I'm not even kidding)

🚀 SYNOPSIS 🚀

$ ch-fromhost [OPTION ...] [FILE_OPTION ...] IMGDIR

✨ DESCRIPTION ✨

Note: this command is experimental. Features may be incomplete and/or buggy. Please report any issues you find, so we can fix them! 🙏

Okay so what this does: it injects files from the host into the Charliecloud image directory IMGDIR. 💫

The whole point of this command is to inject arbitrary host files into a container — the stuff you need to reach host-specific resources. We're talking GPU things, proprietary interconnects, that whole situation. It is not a general copy-to-image tool, okay? There's a whole discussion of use cases further down, don't skip it. 👀

Run it after ch-convert and before ch-run. Once you invoke it, that image is no longer portable to other hosts. Periodt. 💅

Injection is not atomic — if an error happens partway through, the image is left in an undefined state and you should re-unpack it from storage. 😭 Also, injection is currently implemented as a simple file copy, but that may change in the future.

Arbitrary file injection and libfabric injection are handled differently. Let's get into it.

📁 Arbitrary files 📁

Arbitrary file paths that contain the strings /bin or /sbin are assumed to be executables and get placed in /usr/bin inside the container. 🫶

Paths that are not loadable libfabric providers, and that contain the strings /lib or .so, are assumed to be shared libraries and get placed in the first-priority directory reported by ldconfig (peep --lib-path below ✨).

Other files go in the directory specified by --dest.

If any shared libraries get injected, run ldconfig inside the container (using ch-run -w) after injection. Don't forget, bestie.

🧵 Libfabric 🧵

MPI implementations have, like, a million ways of passing messages over interconnects. We use libfabric (OFI), an OpenFabric framework that exports fabric communication services to applications, to manage these communications with built-in (or loadable) fabric providers. 🌈

Using OFI, we can (a) uniformly manage fabric communication services for both OpenMPI and MPICH, and (b) use simplified methods of accessing proprietary host hardware, e.g. Cray's Gemini/Aries and Slingshot (CXI). 🔥

OFI providers implement the application-facing software interfaces you need to access network-specific protocols, drivers, and hardware. Loadable providers — i.e. compiled OFI libraries that end in -fi.so, for example Cray's libgnix-fi.so — can be copied into, and used by, an image with an MPI configured against OFI. Alternatively, the image's libfabric.so can be overwritten with the host's. See the details and quirks below. 💀

🎀 OPTIONS 🎀

🌸 To specify which files to inject 🌸

  • -c, --cmd CMD — inject files listed in the standard output of command CMD.
  • -f, --file FILE — inject files listed in the file FILE.
  • -p, --path PATH — inject the file at PATH.
  • --cray-cxi — inject cray-libfabric for slingshot. This is equivalent to --path $CH_FROMHOST_OFI_CXI, where $CH_FROMHOST_OFI_CXI is the path to the Cray host libfabric libfabric.so.
  • --cray-gni — inject the cray gemini/aries GNI libfabric provider libgnix-fi.so. This is equivalent to --fi-provider $CH_FROMHOST_OFI_GNI, where CH_FROMHOST_OFI_GNI is the path to the Cray host ugni provider libgnix-fi.so.
  • --nvidia — use nvidia-container-cli list (from libnvidia-container) to find executables and libraries to inject.

These can be repeated, and at least one must be specified. 🫡

🦋 To specify the destination within the image 🦋

  • -d, --dest DST — place files specified later in directory IMGDIR/DST, overriding the inferred destination, if any. If a file's destination can't be inferred and --dest hasn't been specified, exit with an error. This can be repeated to place files in varying destinations.

⭐ Additional arguments ⭐

  • --print-cray-fi — print inferred destination for libfabric replacement.
  • --print-fi — print the guest destination path for libfabric provider(s).
  • --print-lib — print the guest destination path for shared libraries inferred as described above.
  • --no-ldconfig — don't run ldconfig even if we appear to have injected shared libraries.
  • -h, --help — print help and exit.
  • -v, --verbose — be more verbose about what is going on. Can be repeated.
  • --version — print version and exit.
Warning: ldconfig often prints scary-looking warnings on stderr even when everything is going fine. By default we suppress these, but you can see them with sufficient verbosity. For example:
$ ch-fromhost --print-lib /var/tmp/bullseye
/usr/local/lib
$ ch-fromhost -v --print-lib /var/tmp/bullseye
asking ldconfig for inferred shared library destination
inferred shared library destination: /var/tmp/bullseye//usr/local/lib
/usr/local/lib
$ ch-fromhost -v -v --print-lib /var/tmp/bullseye
asking ldconfig for inferred shared library destination
/sbin/ldconfig: Can't stat /usr/local/lib/x86_64-linux-gnu: No such file or directory
/sbin/ldconfig: Path `/lib/x86_64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib/x86_64-linux-gnu' given more than once
/sbin/ldconfig: /lib/x86_64-linux-gnu/ld-2.31.so is the dynamic linker, ignoring
inferred shared library destination: /var/tmp/bullseye//usr/local/lib
/usr/local/lib
See issue #732 https://github.com/hpc/charliecloud/issues/732 for an example of how this was confusing for users. 😵‍💫 Been there.

💅 WHEN TO USE CH-FROMHOST 💅

This command does a lot of heuristic magic, and while it can copy arbitrary files into an image, that usage is discouraged and prone to error. Here are some use cases and the recommended approach, bestie:

  1. I have some files on my build host that I want to include in the image. Use the COPY instruction within your Dockerfile. Note that it's OK to build an image that meets your specific needs but isn't generally portable, e.g. only runs on specific micro-architectures you're using. ✨
  2. I have an already built image and want to install a program I compiled separately into the image. Consider whether building a new derived image with a Dockerfile is appropriate. Another good option is to bind-mount the directory containing your program at run time. A less good option is to cp(1) the program into your image, because this permanently alters the image in a non-reproducible way. 🙈
  3. I have some shared libraries that I need in the image for functionality or performance, and they aren't available in a place where I can use COPY. This is the intended use case of ch-fromhost. You can use --cmd, --file, --ofi, and/or --path to put together a custom solution. But please consider filing an issue so we can package your functionality with a tidy option like --nvidia. 🫶

🌈 LIBFABRIC USAGE AND QUIRKS 🌈

The implementation of libfabric provider injection and replacement is experimental and has a couple quirks. Sit down, this is a long one. 😮‍💨

  1. Containers must have the following software installed:
    1. libfabric (https://ofiwg.github.io/libfabric/). See charliecloud/examples/Dockerfile.libfabric.
    2. Corresponding open source MPI implementation configured and built against the container libfabric, e.g.

See charliecloud/examples/Dockerfile.mpich and charliecloud/examples/Dockerfile.openmpi.

  1. At run time, a libfabric provider can be specified with the variable FI_PROVIDER. The path to search for shared providers can be specified with FI_PROVIDER_PATH. These variables can be inherited from the host or explicitly set with the container's environment file /ch/environment via --set-env.

To avoid issues and reduce complexity, the inferred injection destination for libfabric providers and replacement will always be at the path in the image where libfabric.so is found. 📍

  1. The Cray GNI loadable provider, libgnix-fi.so, will link to compiler(s) in the programming environment by default. For example, if it is built under the PrgEnv-intel programming environment, it will have links to files at paths /opt/gcc and /opt/intel that ch-run will not bind automatically.

Managing all possible bind mount paths is untenable. Thus, this experimental implementation injects libraries linked to a libgnix-fi.so built with the minimal modules necessary to compile, i.e.:

  • modules
  • craype-network-aries
  • eproxy
  • slurm
  • cray-mpich
  • craype-haswell
  • craype-hugepages2M

A Cray GNI provider linked against more complicated PE's will still work, assuming 1) the user explicitly bind-mounts missing libraries listed from its ldd output, and 2) all such libraries do not conflict with container functionality, e.g. glibc.so, etc. 🧠

  1. At the time of this writing, a Cray Slingshot optimized provider is not available; however, recent libfabric source activity indicates there may be at some point, see: https://github.com/ofiwg/libfabric/pull/7839

For now, on Cray systems with Slingshot, CXI, we need to overwrite the container's libfabric.so with the host's using --path. See examples for details.

  1. Tested only for C programs compiled with GCC. Additional bind mount or kludging may be needed for untested use cases. If you'd like to use another compiler or programming environment, please get in touch so we can implement the necessary support. 💜

Please file a bug if we missed anything above or if you know how to make the code better. 🙏

NOTES 💭

Okay bestie, listen up, this part is important:

  • Symbolic links are dereferenced, i.e., the files pointed to are injected, not the links themselves. No cap.
  • As a corollary 👑, do not include symlinks to shared libraries. These will be re-created by ldconfig.

There are two alternate approaches for nVidia GPU libraries, and here they are, unedited, because you deserve the full lore:

  1. Link libnvidia-containers into ch-run and call the library functions directly. However, this would mean that Charliecloud would either (a) need to be compiled differently on machines with and without nVidia GPUs or (b) have libnvidia-containers available even on machines without nVidia GPUs. Neither of these is consistent with Charliecloud's philosophies of simplicity and minimal dependencies. 💅
  2. Use nvidia-container-cli configure to do the injecting. This would require that containers have a half-started state, where the namespaces are active and everything is mounted but pivot_root(2) has not been performed. This is not feasible because Charliecloud has no notion of a half-started container. 😭

Further, while these alternate approaches would simplify or eliminate this script for nVidia GPUs, they would not solve the problem for other situations. Fr fr.

BUGS 🐛

  • File paths may not contain colons or newlines. Periodt.
  • ldconfig tends to print stat errors; these are typically non-fatal and occur when trying to probe common library paths. See issue #732 <https://github.com/hpc/charliecloud/issues/732>. So if you see them, breathe, bestie, it's giving normal. 🫶

EXAMPLES ✨

libfabric 🦋

Cray Slingshot CXI injection.

Replace image libabfric, i.e., libfabric.so, with Cray host's libfabric at host path /opt/cray-libfabric/lib64/libfabric.so.

$ ch-fromhost -v --path /opt/cray-libfabric/lib64/libfabric.so /tmp/ompi
[ debug ] queueing files
[ debug ]    cray libfabric: /opt/cray-libfabric/lib64/libfabric.so
[ debug ] searching image for inferred libfabric destination
[ debug ]    found /tmp/ompi/usr/local/lib/libfabric.so
[ debug ] adding cray libfabric libraries
[ debug ]    skipping /lib64/libcom_err.so.2
[...]
[ debug ] queueing files
[ debug ]    shared library: /usr/lib64/libcxi.so.1
[ debug ] queueing files
[ debug ]    shared library: /usr/lib64/libcxi.so.1.2.1
[ debug ] queueing files
[ debug ]    shared library: /usr/lib64/libjson-c.so.3
[ debug ] queueing files
[ debug ]    shared library: /usr/lib64/libjson-c.so.3.0.1
[...]
[ debug ] queueing files
[ debug ]    shared library: /usr/lib64/libssh.so.4
[ debug ] queueing files
[ debug ]    shared library: /usr/lib64/libssh.so.4.7.4
[...]
[ debug ] inferred shared library destination: /tmp/ompi//usr/local/lib
[ debug ] injecting into image: /tmp/ompi/
[ debug ]    mkdir -p /tmp/ompi//var/lib/hugetlbfs
[ debug ]    mkdir -p /tmp/ompi//var/spool/slurmd
[ debug ]    echo '/usr/lib64' >> /tmp/ompi//etc/ld.so.conf.d/ch-ofi.conf
[ debug ]    /opt/cray-libfabric/lib64/libfabric.so -> /usr/local/lib (inferred)
[ debug ]    /usr/lib64/libcxi.so.1 -> /usr/local/lib (inferred)
[ debug ]    /usr/lib64/libcxi.so.1.2.1 -> /usr/local/lib (inferred)
[ debug ]    /usr/lib64/libjson-c.so.3 -> /usr/local/lib (inferred)
[ debug ]    /usr/lib64/libjson-c.so.3.0.1 -> /usr/local/lib (inferred)
[ debug ]    /usr/lib64/libssh.so.4 -> /usr/local/lib (inferred)
[ debug ]    /usr/lib64/libssh.so.4.7.4 -> /usr/local/lib (inferred)
[ debug ] running ldconfig
[ debug ]    ch-run -w /tmp/ompi/ -- /sbin/ldconfig
[ debug ] validating ldconfig cache
done

Same as above, except also inject Cray's fi_info to verify Slingshot provider access. slay

$ ch-fromhost -v --path /opt/cray/libfabric/1.15.0.0/lib64/libfabric.so \
              -d /usr/local/bin \
              --path /opt/cray/libfabric/1.15.0.0/lib64/libfabric.so \
              /tmp/ompi
[...]
$ ch-run /tmp/ompi/ -- fi_info -p cxi
provider: cxi
  fabric: cxi
  [...]
  type: FI_EP_RDM
  protocol: FI_PROTO_CXI

Cray GNI shared provider injection.

Add Cray host built GNI provider libgnix-fi.so to the image and verify with fi_info. 🎀

$ ch-fromhost -v --path /home/ofi/libgnix-fi.so /tmp/ompi
[ debug ] queueing files
[ debug ]    libfabric shared provider: /home/ofi/libgnix-fi.so
[ debug ] searching /tmp/ompi for libfabric shared provider destination
[ debug ]    found: /tmp/ompi/usr/local/lib/libfabric.so
[ debug ] inferred provider destination: //usr/local/lib/libfabric
[ debug ] injecting into image: /tmp/ompi
[ debug ]    mkdir -p /tmp/ompi//usr/local/lib/libfabric
[ debug ]    mkdir -p /tmp/ompi/var/lib/hugetlbfs
[ debug ]    mkdir -p /tmp/ompi/var/opt/cray/alps/spool
[ debug ]    mkdir -p /tmp/ompi/opt/cray/wlm_detect
[ debug ]    mkdir -p /tmp/ompi/etc/opt/cray/wlm_detect
[ debug ]    mkdir -p /tmp/ompi/opt/cray/udreg
[ debug ]    mkdir -p /tmp/ompi/opt/cray/xpmem
[ debug ]    mkdir -p /tmp/ompi/opt/cray/ugni
[ debug ]    mkdir -p /tmp/ompi/opt/cray/alps
[ debug ]    echo '/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf
[ debug ]    echo '/opt/cray/alps/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf
[ debug ]    echo '/opt/cray/udreg/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf
[ debug ]    echo '/opt/cray/ugni/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf
[ debug ]    echo '/opt/cray/wlm_detect/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf
[ debug ]    echo '/opt/cray/xpmem/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf
[ debug ]    echo '/usr/lib64' >> /tmp/ompi/etc/ld.so.conf.d/ch-ofi.conf
[ debug ]    /home/ofi/libgnix-fi.so -> //usr/local/lib/libfabric (inferred)
[ debug ] running ldconfig
[ debug ]    ch-run -w /tmp/ompi -- /sbin/ldconfig
[ debug ] validating ldconfig cache
done

$ ch-run /tmp/ompi -- fi_info -p gni
provider: gni
  fabric: gni
  [...]
  type: FI_EP_RDM
  protocol: FI_PROTO_GNI

Arbitrary 🌈

Place shared library /usr/lib64/libfoo.so at path /usr/lib/libfoo.so (assuming /usr/lib is the first directory searched by the dynamic loader in the image), within the image /var/tmp/baz and executable /bin/bar at path /usr/bin/bar. Then, create appropriate symlinks to libfoo and update the ld.so cache.

$ cat qux.txt
/bin/bar
/usr/lib64/libfoo.so
$ ch-fromhost --file qux.txt /var/tmp/baz

Same as above:

$ ch-fromhost --cmd 'cat qux.txt' /var/tmp/baz

Same as above:

$ ch-fromhost --path /bin/bar --path /usr/lib64/libfoo.so /var/tmp/baz

Same as above, but place the files into /corge instead (and the shared library will not be found by ldconfig):

$ ch-fromhost --dest /corge --file qux.txt /var/tmp/baz

Same as above, and also place file /etc/quux at /etc/quux within the container:

$ ch-fromhost --file qux.txt --dest /etc --path /etc/quux /var/tmp/baz

Inject the executables and libraries recommended by nVidia into the image, and then run ldconfig:

$ ch-fromhost --nvidia /var/tmp/baz
asking ldconfig for shared library destination
/sbin/ldconfig: Can’t stat /libx32: No such file or directory
/sbin/ldconfig: Can’t stat /usr/libx32: No such file or directory
shared library destination: /usr/lib64//bind9-export
injecting into image: /var/tmp/baz
  /usr/bin/nvidia-smi -> /usr/bin (inferred)
  /usr/bin/nvidia-debugdump -> /usr/bin (inferred)
  /usr/bin/nvidia-persistenced -> /usr/bin (inferred)
  /usr/bin/nvidia-cuda-mps-control -> /usr/bin (inferred)
  /usr/bin/nvidia-cuda-mps-server -> /usr/bin (inferred)
  /usr/lib64/libnvidia-ml.so.460.32.03 -> /usr/lib64//bind9-export (inferred)
  /usr/lib64/libnvidia-cfg.so.460.32.03 -> /usr/lib64//bind9-export (inferred)
[...]
  /usr/lib64/libGLESv2_nvidia.so.460.32.03 -> /usr/lib64//bind9-export (inferred)
  /usr/lib64/libGLESv1_CM_nvidia.so.460.32.03 -> /usr/lib64//bind9-export (inferred)
running ldconfig

ACKNOWLEDGEMENTS 🫶

This command was inspired by the similar Shifter <http://www.nersc.gov/research-and-development/user-defined-images/> feature that allows Shifter containers to use the Cray Aries network. We particularly appreciate the help provided by Shane Canon and Doug Jacobsen during our implementation of --cray-mpi. 👑

We appreciate the advice of Ryan Olson at nVidia on implementing --nvidia. 💜

REPORTING BUGS 📣

If Charliecloud was obtained from your Linux distribution, use your distribution's bug reporting procedures. That's the vibe. 🎀

Otherwise, report bugs to: <https://github.com/hpc/charliecloud/issues> — they wanna hear from you, bestie.

SEE ALSO 👀

Full documentation at: <<https://hpc.github.io/charliecloud>>

contributors to the Charliecloud project, which has been established as “Charliecloud a Series of LF Projects, LLC”. For web site terms of use, trademark policy, and related information, please see: <https://lfprojects.org>.

okay that's literally the whole page, you ate it 💅 this is version 0.44, dated 2026-05-31 07:53 UTC, and it's CH-FROMHOST(1) — go forth and inject responsibly, ily 🦋✨💖

📦 where this page comes from