DMTCP(1)

📦 version 4.2.0, in arch rolling, fedora 43, fedora 44, nixos 26.05, nixpkgs unstable

🌸 NAME 🌸

okay so bestie, sit down, hydrate, because today we're talking about dmtcp — Distributed MultiThreaded Checkpointing 💖 it's literally the tool that freezes your programs mid-sentence like a screenshot and then brings them back to life later. no cap.

✨ SYNOPSIS ✨

like, the whole vibe in four lines:

dmtcp_coordinator [port]

dmtcp_launch command [args...]

dmtcp_restart ckpt_FILE1.dmtcp [ckpt_FILE2.dmtcp...]

dmtcp_command coordinatorCommand

🦋 DESCRIPTION 🦋

so DMTCP is a tool for transparently checkpointing the state of an arbitrary group of programs, spread across many machines, all connected by sockets 🔌. it does not modify your program, and it does not modify your operating system. it just… does it. main character energy, honestly.

and MTCP? that's a standalone component of DMTCP, available as a checkpointing library for a single process. one process, one little legend 🫶

🎀 OPTIONS 🎀

for each command, the --help or -h flag will show you the command-line options, bestie. and plot twist: most command line options can also be controlled through environment variables ✨ you can set those in bash with export NAME=value, or in tcsh with setenv NAME value. pick your fighter.

  • DMTCP_CHECKPOINT_INTERVAL=integer — time in seconds between automatic checkpoints. you can also start one manually by typing 'c' into the coordinator, iconic. (default: 0, disabled; dmtcp_coordinator only)
  • DMTCP_HOST=string — the hostname where the cluster-wide coordinator is running. (default: localhost; dmtcp_launch, dmtcp_restart only)
  • DMTCP_PORT=integer — the port the cluster-wide coordinator listens on. (default: 7779)
  • DMTCP_GZIP=(1|0) — set it to "0" to disable compression of checkpoint images. (default: 1, compression enabled; dmtcp_launch only)
⚠️ WARNING: gzip adds seconds 😭 without gzip, ckpt/restart is often less than 1 s.
  • DMTCP_CHECKPOINT_DIR=path — the directory to store checkpoint images in. (default: ./)
  • DMTCP_SIGCKPT=integer — internal signal number to use for checkpointing. it must not be used by the user program, okay? like, don't touch it. (default: SIGUSR2; dmtcp_launch only)

👑 DMTCP_COORDINATOR 👑

every computation you wanna checkpoint has to include a DMTCP coordinator process. you can either start one explicitly with dmtcp_coordinator, or just let one get started implicitly in the background by dmtcp_launch or dmtcp_restart — they need it to operate, so they'll handle it. the address of the unique coordinator should be specified by dmtcp_launch, dmtcp_restart, and dmtcp_command either through the --host and --port command-line flags or through the DMTCP_HOST and DMTCP_PORT environment variables. if you give neither, the host-port pair defaults to localhost-7779. the host-port pair associated with a particular coordinator is given by the command-line flags used in the dmtcp_coordinator command, or the environment variables then in effect, or the default of localhost-7779.

the coordinator is stateless and is not checkpointed. so on restart, you can use an existing coordinator or a brand new one, whichever. multiple computations under DMTCP control can coexist just fine, as long as you give each one a unique coordinator (with a unique host-port pair) 🫶

the coordinator initiates a checkpoint for all processes in its computation group. checkpoints can be:

  • performed automatically on an interval (see DMTCP_CHECKPOINT_INTERVAL above)
  • initiated manually on the standard input of the coordinator (see the next paragraph)
  • initiated directly under program control by the computation through the plugin API (see below)

the coordinator accepts the following commands on its standard input. each command should be followed by the <return> key 🔑 the commands are:

  • l — list connected nodes
  • s — print status message
  • c — checkpoint all nodes
  • f — force a restart even if there are missing nodes (debugging)
  • k — kill all nodes
  • q — kill all nodes and quit
  • ? — show this message

coordinator commands can also be issued remotely using dmtcp_command. slay 💅

🌈 EXAMPLE USAGE 🌈

  1. in a separate terminal window, start the dmtcp_coordinator. (see previous section.)
dmtcp_coordinator
  1. in separate terminal(s), replace each command(s) with "dmtcp_launch [command]". the checkpointed program will connect to the coordinator specified by DMTCP_HOST and DMTCP_PORT. new threads will be checkpointed as part of the process. child processes will automatically be checkpointed. remote processes started via ssh will automatically checkpointed. (internally, DMTCP modifies the ssh command line to call dmtcp_launch on the remote host.)
dmtcp_launch ./myprogram
  1. to manually initiate a checkpoint, either run the command below or type "c" followed by <return> into the coordinator. checkpoint files for each process will be written to DMTCP_CHECKPOINT_DIR. the dmtcp_coordinator will write "dmtcp_restart_script.sh" to its working directory. this script contains the necessary calls to dmtcp_restart to restart the entire computation, including remote processes created via ssh.
dmtcp_command -c
OR:  dmtcp_command --checkpoint
  1. to restart, one should execute dmtcp_restart_script.sh, which is created by the dmtcp_coordinator in its working directory at the time of checkpoint. one can optionally edit this script to migrate processes to different hosts. by default, only one restarted process will be restarted in the foreground and receive the standard input. the script may be edited to choose which process will be restarted in the foreground.
./dmtcp_restart_script.sh

🧚 DMTCP PLUGIN 🧚

the source distribution includes a top-level plugin directory, with examples of how to write a plugin for DMTCP 👑 plugins allow a checkpointed application to disconnect from an external resource, and then reconnect to it at the time of restart. further examples are in the test/plugin directory. in particular, test/plugin/applic-initiated demonstrates how an application can request a checkpoint under program control. cute, right?

the plugin feature adds three new user-programmable capabilities. a plugin may:

  • add wrappers around system calls
  • take special actions at during certain events (e.g. pre-checkpoint, resume/post-checkpoint, restart)
  • and may insert key-value pairs into a database at restart time that is then available to be queried by the restarted processes of a computation.

(the events available to the plugin feature form a superset of the events available with the dmtcpaware interface.) one or more plugins are invoked via a list of colon-separated absolute pathnames.

dmtcp_launch --with-plugin PLUGIN1[:PLUGIN2]...

💜 DMTCPAWARE API 💜

okay so this API is now deprecated in favor of DMTCP plugins (see above) 🙈 DMTCP provides a programming interface to allow checkpointed applications to interact with dmtcp. in the source distribution, see dmtcpaware/dmtcpaware.h for the functions available. see test/dmtcpaware[123].c for three example applications. for an example of its usage, try:

cd test; rm dmtcpaware1; make dmtcpaware1; ./autotest -v dmtcpaware1

the user application should link with libdmtcpaware.so (-ldmtcpaware) and use the header file dmtcp/dmtcpaware.h.

⭐ RETURN CODE ⭐

a target program under DMTCP control normally returns the same return code as if it were executed without DMTCP. however, if DMTCP fails (as opposed to the target program failing), DMTCP returns a DMTCP-specific return code, rc (or rc+1, rc+2 for two special cases), where rc is the integer value of the environment variable DMTCP_FAIL_RC if set, or else the default value, 99.

so yeah. 99. remember it. iykyk 💀

💖 SEE ALSO 💖

full documentation is available at the DMTCP home page:

🌸 AUTHORS 🌸

DMTCP and its standalone single-process component MTCP (MultiThreaded CheckPointing) were created by the long-term contributors, Jason Ansel, Kapil Arya, Gene Cooperman, Rohan Garg, Artem Y. Polyakov, Mike Rieker, and a series of additional contributors including Alex Brick, Tyler Denniston, William Enright, Gregory Kerr, Ana-Maria Visan, and others 🫶 for support, see the DMTCP home page.

and that's the whole thing, bestie ✨ this was written by the DMTCP team and dated June 17, 2008, living forever as DMTCP(1) in section 1 of the manual 💅 go checkpoint something and eat, fr fr 🎀

📦 where this page comes from