scrun(1)

📦

version 26.05.5.1 · 5 versions

26.05.5.1 · arch rolling, epel 10, nixpkgs unstable, ubuntu stonking (latest)
24.05.2 · fedora 43, fedora 44
23.11.4 · ubuntu noble
25.11.2 · nixos 26.05, ubuntu resolute
24.11.5 · debian trixie, opensuse-leap 16.0, ubuntu questing

, in arch rolling, epel 10, nixpkgs unstable, ubuntu stonking

✨ scrun(1) — Slurm Commands ✨

okay bestie, sit down, hydrate, because this is scrun from the Slurm Commands corner of the universe and she is a lot 💅

🌸 NAME 🌸

scrun — an OCI runtime proxy for Slurm. no cap, that's the whole job description.

🎀 SYNOPSIS 🎀

So there's not one single way to call her, there's a whole little family of operations. Let's go one by one, mwah 😘

💖 Create Operation

scrun [GLOBAL OPTIONS...] create [CREATE OPTIONS] <container-id>

Prepares a new container with container-id in current working directory.

🦋 Start Operation

scrun [GLOBAL OPTIONS...] start <container-id>

Request to start and run container in job.

⭐ Query State Operation

scrun [GLOBAL OPTIONS...] state <container-id>

Output OCI defined JSON state of container.

💀 Kill Operation

scrun [GLOBAL OPTIONS...] kill <container-id> [signal]

Send signal (default: SIGTERM) to container.

🎀 Delete Operation

scrun [GLOBAL OPTIONS...] delete [DELETE OPTIONS] <container-id>

Release any resources held by container locally and remotely.

Perform OCI runtime operations against container-id per: https://github.com/opencontainers/runtime-spec/blob/main/runtime.md

scrun attempts to mimic the commandline behavior as closely as possible to crun and runc in order to maintain in place replacement compatibility with DOCKER and podman. All commandline arguments for crun and runc will be accepted for compatibility but may be ignored depending on their applicability. (iykyk — she's being polite, she's nodding along, she's not always listening 🙈)

💅 DESCRIPTION 💅

scrun is an OCI runtime proxy for Slurm. It acts as a common interface to DOCKER or podman to allow container operations to be executed under Slurm as jobs. scrun will accept all commands as an OCI compliant runtime but will proxy the container and all STDIO to Slurm for scheduling and execution. The containers will be executed remotely on Slurm compute nodes according to settings in oci.conf(5). lowkey iconic behavior honestly.

scrun requires all containers to be OCI image compliant per: https://github.com/opencontainers/image-spec/blob/main/spec.md

🌈 RETURN VALUE 🌈

On successful operation, scrun will return 0. For any other condition scrun will return any non-zero number to denote a error. simple as that, bestie.

✨ GLOBAL OPTIONS ✨

the big-girl flags. they go before the operation name, remember that 🫶

--cgroup-manager — Ignored. she's literally just standing there, menacingly 🧍
--debug — Activate debug level logging.
-f <slurm_conf_path> — Use specified slurm.conf for configuration. Default: sysconfdir from configure during compilation
--usage — Show quick help on how to call scrun
--log-format=<json|text> — Optional select format for logging. May be "json" or "text". Default: text
--root=<root_path> — Path to spool directory to communication sockets and temporary directories and files. This should be a tmpfs and should be cleared on reboot. Default: /run/user/{user_id}/scrun/
--rootless — Ignored. All scrun commands are always rootless. 👑
--systemd-cgroup — Ignored.
-v — Increase logging verbosity. Multiple -v's increase verbosity. stack 'em like bracelets.
-V, --version — Print version information and exit.

🍓 CREATE OPTIONS 🍓

these only hit different when you're doing the create operation, bestie.

-b <bundle_path>, --bundle=<bundle_path> — Path to the root of the bundle directory. Default: caller's working directory
--console-socket=<console_socket_path> — Optional path to an AF_UNIX socket which will receive a file descriptor referencing the master end of the console's pseudoterminal. Default: ignored
--no-pivot — Ignored.
--no-new-keyring — Ignored.
--pid-file=<pid_file_path> — Specify the file to lock and populate with process ID. Default: ignored
--preserve-fds — Ignored.

🔥 DELETE OPTIONS 🔥

--force — Ignored. All delete requests are forced and will kill any running jobs. yeah. she's not asking 🫢

💜 INPUT ENVIRONMENT VARIABLES 💜

env vars she reads on the way in. each one takes one of these levels:

SCRUN_DEBUG=<quiet|fatal|error|info|verbose|debug|debug2|debug3|debug4|debug5> — Set logging level.
SCRUN_STDERR_DEBUG=<quiet|fatal|error|info|verbose|debug|debug2|debug3|debug4|debug5> — Set logging level for standard error output only.
SCRUN_SYSLOG_DEBUG=<quiet|fatal|error|info|verbose|debug|debug2|debug3|debug4|debug5> — Set logging level for syslogging only.
SCRUN_FILE_DEBUG=<quiet|fatal|error|info|verbose|debug|debug2|debug3|debug4|debug5> — Set logging level for log file only.

🧚 JOB INPUT ENVIRONMENT VARIABLES 🧚

okay so here's the thing — this list is LONG. but every single one of these is just scrun being like "yeah I read that one, go check srun(1)". so take a breath, it's a vibe, not a quiz 📚✨

SCRUN_ACCOUNT — See SLURM_ACCOUNT from srun(1).
SCRUN_ACCTG_FREQ — See SLURM_ACCTG_FREQ from srun(1).
SCRUN_BURST_BUFFER — See SLURM_BURST_BUFFER from srun(1).
SCRUN_CLUSTER_CONSTRAINT — See SLURM_CLUSTER_CONSTRAINT from srun(1).
SCRUN_CLUSTERS — See SLURM_CLUSTERS from srun(1).
SCRUN_CONSTRAINT — See SLURM_CONSTRAINT from srun(1).
SLURM_CORE_SPEC — See SLURM_ACCOUNT from srun(1).
SCRUN_CPU_BIND — See SLURM_CPU_BIND from srun(1).
SCRUN_CPU_FREQ_REQ — See SLURM_CPU_FREQ_REQ from srun(1).
SCRUN_CPUS_PER_GPU — See SLURM_CPUS_PER_GPU from srun(1).
SCRUN_CPUS_PER_TASK — See SRUN_CPUS_PER_TASK from srun(1).
SCRUN_DELAY_BOOT — See SLURM_DELAY_BOOT from srun(1).
SCRUN_DEPENDENCY — See SLURM_DEPENDENCY from srun(1).
SCRUN_DISTRIBUTION — See SLURM_DISTRIBUTION from srun(1).
SCRUN_EPILOG — See SLURM_EPILOG from srun(1).
SCRUN_EXACT — See SLURM_EXACT from srun(1).
SCRUN_EXCLUSIVE — See SLURM_EXCLUSIVE from srun(1).
SCRUN_GPU_BIND — See SLURM_GPU_BIND from srun(1).
SCRUN_GPU_FREQ — See SLURM_GPU_FREQ from srun(1).
SCRUN_GPUS — See SLURM_GPUS from srun(1).
SCRUN_GPUS_PER_NODE — See SLURM_GPUS_PER_NODE from srun(1).
SCRUN_GPUS_PER_SOCKET — See SLURM_GPUS_PER_SOCKET from salloc(1).
SCRUN_GPUS_PER_TASK — See SLURM_GPUS_PER_TASK from srun(1).
SCRUN_GRES_FLAGS — See SLURM_GRES_FLAGS from srun(1).
SCRUN_GRES — See SLURM_GRES from srun(1).
SCRUN_HINT — See SLURM_HIST from srun(1).
SCRUN_JOB_NAME — See SLURM_JOB_NAME from srun(1).
SCRUN_JOB_NODELIST — See SLURM_JOB_NODELIST from srun(1).
SCRUN_JOB_NUM_NODES — See SLURM_JOB_NUM_NODES from srun(1).
SCRUN_LABELIO — See SLURM_LABELIO from srun(1).
SCRUN_MEM_BIND — See SLURM_MEM_BIND from srun(1).
SCRUN_MEM_PER_CPU — See SLURM_MEM_PER_CPU from srun(1).
SCRUN_MEM_PER_GPU — See SLURM_MEM_PER_GPU from srun(1).
SCRUN_MEM_PER_NODE — See SLURM_MEM_PER_NODE from srun(1).
SCRUN_MPI_TYPE — See SLURM_MPI_TYPE from srun(1).
SCRUN_NCORES_PER_SOCKET — See SLURM_NCORES_PER_SOCKET from srun(1).
SCRUN_NETWORK — See SLURM_NETWORK from srun(1).
SCRUN_NSOCKETS_PER_NODE — See SLURM_NSOCKETS_PER_NODE from srun(1).
SCRUN_NTASKS — See SLURM_NTASKS from srun(1).
SCRUN_NTASKS_PER_CORE — See SLURM_NTASKS_PER_CORE from srun(1).
SCRUN_NTASKS_PER_GPU — See SLURM_NTASKS_PER_GPU from srun(1).
SCRUN_NTASKS_PER_NODE — See SLURM_NTASKS_PER_NODE from srun(1).
SCRUN_NTASKS_PER_TRES — See SLURM_NTASKS_PER_TRES from srun(1).
SCRUN_OPEN_MODE — See SLURM_MODE from srun(1).
SCRUN_OVERCOMMIT — See SLURM_OVERCOMMIT from srun(1).
SCRUN_OVERLAP — See SLURM_OVERLAP from srun(1).
SCRUN_PARTITION — See SLURM_PARTITION from srun(1).
SCRUN_POWER — See SLURM_POWER from srun(1).
SCRUN_PROFILE — See SLURM_PROFILE from srun(1).
SCRUN_PROLOG — See SLURM_PROLOG from srun(1).
SCRUN_QOS — See SLURM_QOS from srun(1).
SCRUN_REMOTE_CWD — See SLURM_REMOTE_CWD from srun(1).
SCRUN_REQ_SWITCH — See SLURM_REQ_SWITCH from srun(1).
SCRUN_RESERVATION — See SLURM_RESERVATION from srun(1).
SCRUN_SIGNAL — See SLURM_SIGNAL from srun(1).
SCRUN_SLURMD_DEBUG — See SLURMD_DEBUG from srun(1).
SCRUN_SPREAD_JOB — See SLURM_SPREAD_JOB from srun(1).
SCRUN_TASK_EPILOG — See SLURM_TASK_EPILOG from srun(1).
SCRUN_TASK_PROLOG — See SLURM_TASK_PROLOG from srun(1).
SCRUN_THREAD_SPEC — See SLURM_THREAD_SPEC from srun(1).
SCRUN_THREADS_PER_CORE — See SLURM_THREADS_PER_CORE from srun(1).
SCRUN_THREADS — See SLURM_THREADS from srun(1).
SCRUN_TIMELIMIT — See SLURM_TIMELIMIT from srun(1).
SCRUN_TRES_BIND — Same as --tres-bind
SCRUN_TRES_PER_TASK — See SLURM_TRES_PER_TASK from srun(1).
SCRUN_UNBUFFEREDIO — See SLURM_UNBUFFEREDIO from srun(1).
SCRUN_USE_MIN_NODES — See SLURM_USE_MIN_NODES from srun(1).
SCRUN_WAIT4SWITCH — See SLURM_WAIT4SWITCH from srun(1).
SCRUN_WCKEY — See SLURM_WCKEY from srun(1).
SCRUN_WORKING_DIR — See SLURM_WORKING_DIR from srun(1).

we survived. hydrating rn 🍓💧

🦋 OUTPUT ENVIRONMENT VARIABLES 🦋

these are the ones she sets, so your container gets the tea when it wakes up:

SCRUN_OCI_VERSION — Advertised version of OCI compliance of container.
SCRUN_CONTAINER_ID — Value based as container_id during create operation.
SCRUN_PID — PID of process used to monitor and control container on allocation node.
SCRUN_BUNDLE — Path to container bundle directory.
SCRUN_SUBMISSION_BUNDLE — Path to container bundle directory before modification by Lua script.
SCRUN_ANNOTATION_* — List of annotations from container's config.json.
SCRUN_PID_FILE — Path to pid file that is locked and populated with PID of scrun.
SCRUN_SOCKET — Path to control socket for scrun.
SCRUN_SPOOL_DIR — Path to workspace for all temporary files for current container. Purged by deletion operation.
SCRUN_SUBMISSION_CONFIG_FILE — Path to container's config.json file at time of submission.
SCRUN_USER — Name of user that called create operation.
SCRUN_USER_ID — Numeric ID of user that called create operation.
SCRUN_GROUP — Name of user's primary group that called create operation.
SCRUN_GROUP_ID — Numeric ID of user primary group that called create operation.
SCRUN_ROOT — See --root.
SCRUN_ROOTFS_PATH — Path to container's root directory.
SCRUN_SUBMISSION_ROOTFS_PATH — Path to container's root directory at submission time.
SCRUN_LOG_FILE — Path to scrun's log file during create operation.
SCRUN_LOG_FORMAT — Log format type during create operation.

🫶 JOB OUTPUT ENVIRONMENT VARIABLES 🫶

okay so when your job actually runs, these are the little env vars slurm drops into its environment so the job knows what it got. per-component edition for het jobs, bestie 💖

SLURM_*_HET_GROUP_# — for a heterogeneous job allocation, these environment variables get set separately for each component. every component gets its own lil copy, no sharing 💅
SLURM_CLUSTER_NAME — the name of the cluster where the job is actually executing. the one it's living its life on ✨
SLURM_CONTAINER — the OCI Bundle for the job.
SLURM_CONTAINER_ID — the OCI id for the job.
SLURM_CPUS_PER_GPU — how many CPUs you asked for per allocated GPU.
SLURM_CPUS_PER_TASK — how many CPUs you requested per task.
SLURM_DIST_PLANESIZE — the plane distribution size. ngl it's only set for plane distributions, so if you're not doing that, don't come looking for it 🙈
SLURM_DISTRIBUTION — the distribution type used for the allocated jobs.
SLURM_GPU_BIND — the requested binding of tasks to GPU.
SLURM_GPU_FREQ — the requested GPU frequency.
SLURM_GPUS — how many GPUs were requested. the count, the whole count 🔥
SLURM_GPUS_PER_NODE — requested GPU count per allocated node.
SLURM_GPUS_PER_SOCKET — requested GPU count per allocated socket.
SLURM_GPUS_PER_TASK — requested GPU count per allocated task.
SLURM_HET_SIZE — set to the count of components in a heterogeneous job.
SLURM_JOB_ACCOUNT — the account name associated with the job allocation.
SLURM_JOB_CPUS_PER_NODE — the count of CPUs available to the job, across the nodes in the allocation, using the format CPU_count[(xnumber_of_nodes)][,CPU_count [(xnumber_of_nodes)] ...].

for example: SLURM_JOB_CPUS_PER_NODE='72(x2),36' means that on the first and second nodes (as listed by SLURM_JOB_NODELIST) the allocation has 72 CPUs, while the third node has 36 CPUs.

NOTE: the select/linear plugin allocates entire nodes to jobs, so the value indicates the total count of CPUs on the allocated nodes. the select/cons_tres plugin allocates individual CPUs to jobs, so this number indicates the number of CPUs allocated to the job. two different vibes, two different meanings, read carefully bestie 😭

SLURM_JOB_END_TIME — the UNIX timestamp for a job's projected end time.
SLURM_JOB_GPUS — the global GPU IDs of the GPUs allocated to this job. these IDs are not relative to any device cgroup, even if devices are constrained with task/cgroup. it's only set in batch and interactive jobs. ✨
SLURM_JOB_ID — the ID of the job allocation.
SLURM_JOB_NODELIST — the list of nodes allocated to the job.
SLURM_JOB_NUM_NODES — the total number of nodes in the job allocation.
SLURM_JOB_PARTITION — the name of the partition where the job is running.
SLURM_JOB_QOS — the Quality Of Service (QOS) of the job allocation.
SLURM_JOB_RESERVATION — the advanced reservation containing the job allocation, if there is one.
SLURM_JOB_START_TIME — the UNIX timestamp for a job's start time.
SLURM_MEM_BIND — bind tasks to memory.
SLURM_MEM_BIND_LIST — set to the bit mask used for memory binding.
SLURM_MEM_BIND_PREFER — set to "prefer" if the SLURM_MEM_BIND option includes the prefer option.
SLURM_MEM_BIND_TYPE — set to the memory binding type specified with the SLURM_MEM_BIND option. possible values are "none", "rank", "map_mem:", "mask_mem:" and "local".
SLURM_MEM_BIND_VERBOSE — set to "verbose" if the SLURM_MEM_BIND option includes the verbose option. set to "quiet" otherwise.
SLURM_MEM_PER_CPU — the minimum memory required per usable allocated CPU.
SLURM_MEM_PER_GPU — the requested memory per allocated GPU.
SLURM_MEM_PER_NODE — specifies the real memory required per node.
SLURM_NTASKS — specifies the number of tasks to run.
SLURM_NTASKS_PER_CORE — requests the maximum ntasks be invoked on each core.
SLURM_NTASKS_PER_GPU — requests that there are ntasks tasks invoked for every GPU.
SLURM_NTASKS_PER_NODE — requests that ntasks be invoked on each node.
SLURM_NTASKS_PER_SOCKET — requests the maximum ntasks be invoked on each socket.
SLURM_OVERCOMMIT — overcommit resources.
SLURM_PROFILE — enables detailed data collection by the acct_gather_profile plugin.
SLURM_SHARDS_ON_NODE — the number of GPU Shards available to the step on this node.
SLURM_SUBMIT_HOST — the hostname of the computer from which scrun was invoked. hi bestie, this is you 👋
SLURM_TASKS_PER_NODE — the number of tasks to be initiated on each node. values are comma separated and in the same order as SLURM_JOB_NODELIST. if two or more consecutive nodes are to have the same task count, that count is followed by "(x#)" where "#" is the repetition count.

for example, SLURM_TASKS_PER_NODE=2(x3),1 means the first three nodes will each execute two tasks and the fourth node will execute one task. slay, compression 🎀

SLURM_THREADS_PER_CORE — this is only set if --threads-per-core or SCRUN_THREADS_PER_CORE were specified. the value will be set to the value specified by --threads-per-core or SCRUN_THREADS_PER_CORE. this is used by subsequent srun calls within the job allocation. ✨
SLURM_TRES_PER_TASK — set to the value of --tres-per-task. if --cpus-per-task or --gpus-per-task is specified, it is also set in SLURM_TRES_PER_TASK as if it were specified in --tres-per-task.

🧚 SCRUN.LUA 🧚

so /etc/slurm/scrun.lua must be present on every single node where scrun will be invoked. no file, no vibes. and scrun.lua has to be a compliant lua script — the good kind 💅

🌸 Required functions 🌸

these ones are non-negotiable, they must be defined:

function slurm_scrun_stage_in(id, bundle, spool_dir, config_file, job_id, user_id, group_id, job_env)

called right after job allocation to stage the container into the job node(s). it must return SLURM.success or the job will be cancelled 😭 it is required that the function will prepare the container for execution on job node(s) as required to run as configured in oci.conf(1). the function may block as long as required until the container has been fully prepared (up to the job's max wall time). so yeah, take your time, but not more than that ⏳

id — Container ID
bundle — OCI bundle path
spool_dir — temporary working directory for container
config_file — path to config.json for container
job_id — jobid of job allocation
user_id — resolved numeric user id of job allocation. it is generally expected that the lua script will be executed inside of a user namespace running under the root(0) user.
group_id — resolved numeric group id of job allocation. it is generally expected that the lua script will be executed inside of a user namespace running under the root(0) group.
job_env — table with each entry of Key=Value or Value of each environment variable of the job.

function slurm_scrun_stage_out(id, bundle, orig_bundle, root_path, orig_root_path, spool_dir, config_file, jobid, user_id, group_id)

called right after container step completes to stage out files from job nodes. it must return SLURM.success or the job will be cancelled 🔥 it is required that the function will pull back any changes and clean up the container on job node(s). the function may block as long as required until the container has been fully prepared (up to the job's max wall time).

id — Container ID
bundle — OCI bundle path
orig_bundle — originally submitted OCI bundle path before modification by set_bundle_path().
root_path — path to directory root of container contents.
orig_root_path — original path to directory root of container contents before modification by set_root_path().
spool_dir — temporary working directory for container
config_file — path to config.json for container
job_id — jobid of job allocation
user_id — resolved numeric user id of job allocation. it is generally expected that the lua script will be executed inside of a user namespace running under the root(0) user.
group_id — resolved numeric group id of job allocation. it is generally expected that the lua script will be executed inside of a user namespace running under the root(0) group.

🦋 Provided functions 🦋

these are handed to you already, so any Lua function can just call them as needed. free real estate 💜

slurm.set_bundle_path(PATH) — call this to notify scrun to use PATH as the new OCI container bundle path. depending on the filesystem layout, cloning the container bundle may be required to allow execution on job nodes.
slurm.set_root_path(PATH) — call this to notify scrun to use PATH as the new container root filesystem path. depending on the filesystem layout, cloning the container bundle may be required to allow execution on job nodes. the script must also update #/root/path in config.json when changing root path. don't forget that part ok 🙈
STATUS,OUTPUT = slurm.remote_command(SCRIPT) — run SCRIPT in a new job step on all job nodes. returns numeric job status as STATUS and job stdio as OUTPUT. blocks until SCRIPT exits. it will simply wait for you, patiently 🫶
STATUS,OUTPUT = slurm.allocator_command(SCRIPT) — run SCRIPT as a forked child process of scrun. returns numeric job status as STATUS and job stdio as OUTPUT. blocks until SCRIPT exits.
slurm.log(MSG, LEVEL) — log MSG at log LEVEL. valid range of values for LEVEL is [0, 4].
slurm.error(MSG) — log error MSG.
slurm.log_error(MSG) — log error MSG.
slurm.log_info(MSG) — log MSG at log level INFO.
slurm.log_verbose(MSG) — log MSG at log level VERBOSE.
slurm.log_verbose(MSG) — log MSG at log level VERBOSE. (yes bestie, it's listed twice in the original. we keep both. no notes 💅)
slurm.log_debug(MSG) — log MSG at log level DEBUG.
slurm.log_debug2(MSG) — log MSG at log level DEBUG2.
slurm.log_debug3(MSG) — log MSG at log level DEBUG3.
slurm.log_debug4(MSG) — log MSG at log level DEBUG4.
MINUTES = slurm.time_str2mins(TIME_STRING) — parse TIME_STRING into a number of minutes as MINUTES. valid formats:
- days-[hours[:minutes[:seconds]]]
- hours:minutes:seconds
- minutes[:seconds]
- -1
- INFINITE
- UNLIMITED

🧪 Example scrun.lua scripts 🧪

okay bestie, this is the part where it gets juicy — actual working-ish example scripts. 💖

📦 Full Container staging example using rsync 📦

This full example will stage a container as given by docker or podman. No cap, it takes the container's config.json and modifies it to remove unwanted functions that may cause the container run to under crun or runc. Then it uses rsync to move the container to a shared filesystem under the scratch_path variable. it's giving "relocating your whole vibe to a different address" 💅

NOTE: Support for JSON in liblua must generally be installed before Slurm is compiled. scrun.lua's syntax and ability to load JSON support should be tested by directly calling the script using lua outside of Slurm. fr fr, test it outside first or you're gonna have a bad time 😭

local open = io.open
local scratch_path = "/run/user"

local function read_file(path)
     local file = open(path, "rb")
     if not file then return nil end
     local content = file:read "*all"
     file:close()
     return content
end

local function write_file(path, contents)
     local file = open(path, "wb")
     if not file then return nil end
     file:write(contents)
     file:close()
     return
end

function slurm_scrun_stage_in(id, bundle, spool_dir, config_file, job_id, user_id, group_id, job_env)
     slurm.log_debug(string.format("stage_in(%s, %s, %s, %s, %d, %d, %d)",
                 id, bundle, spool_dir, config_file, job_id, user_id, group_id))

     local status, output, user, rc
     local config = slurm.from_json(read_file(config_file))
     local src_rootfs = config["root"]["path"]
     rc, user = slurm.allocator_command(string.format("id -un %d", user_id))
     user = string.gsub(user, "%s+", "")
     local root = scratch_path.."/"..math.floor(user_id).."/slurm/scrun/"
     local dst_bundle = root.."/"..id.."/"
     local dst_config = root.."/"..id.."/config.json"
     local dst_rootfs = root.."/"..id.."/rootfs/"

     if string.sub(src_rootfs, 1, 1) ~= "/"
     then
          -- always use absolute path
          src_rootfs = string.format("%s/%s", bundle, src_rootfs)
     end

     status, output = slurm.allocator_command("mkdir -p "..dst_rootfs)
     if (status ~= 0)
     then
          slurm.log_info(string.format("mkdir(%s) failed %u: %s",
                      dst_rootfs, status, output))
          return slurm.ERROR
     end

     status, output = slurm.allocator_command(string.format("/usr/bin/env rsync --exclude sys --exclude proc --numeric-ids --delete-after --ignore-errors --stats -a -- %s/ %s/", src_rootfs, dst_rootfs))
     if (status ~= 0)
     then
          -- rsync can fail due to permissions which may not matter
          slurm.log_info(string.format("WARNING: rsync failed: %s", output))
     end

     slurm.set_bundle_path(dst_bundle)
     slurm.set_root_path(dst_rootfs)

     config["root"]["path"] = dst_rootfs

     -- Always force user namespace support in container or runc will reject
     local process_user_id = 0
     local process_group_id = 0

     if ((config["process"] ~= nil) and (config["process"]["user"] ~= nil))
     then
          -- resolve out user in the container
          if (config["process"]["user"]["uid"] ~= nil)
          then
               process_user_id=config["process"]["user"]["uid"]
          else
               process_user_id=0
          end

          -- resolve out group in the container
          if (config["process"]["user"]["gid"] ~= nil)
          then
               process_group_id=config["process"]["user"]["gid"]
          else
               process_group_id=0
          end

          -- purge additionalGids as they are not supported in rootless
          if (config["process"]["user"]["additionalGids"] ~= nil)
          then
               config["process"]["user"]["additionalGids"] = nil
          end
     end

     if (config["linux"] ~= nil)
     then
          -- force user namespace to always be defined for rootless mode
          local found = false
          if (config["linux"]["namespaces"] == nil)
          then
               config["linux"]["namespaces"] = {}
          else
               for _, namespace in ipairs(config["linux"]["namespaces"]) do
                    if (namespace["type"] == "user")
                    then
                         found=true
                         break
                    end
               end
          end
          if (found == false)
          then
               table.insert(config["linux"]["namespaces"], {type= "user"})
          end

          -- Provide default user map as root if one not provided
          if (true or config["linux"]["uidMappings"] == nil)
          then
               config["linux"]["uidMappings"] =
                    {{containerID=process_user_id, hostID=math.floor(user_id), size=1}}
          end

          -- Provide default group map as root if one not provided
          -- mappings fail with build???
          if (true or config["linux"]["gidMappings"] == nil)
          then
               config["linux"]["gidMappings"] =
                    {{containerID=process_group_id, hostID=math.floor(group_id), size=1}}
          end

          -- disable trying to use a specific cgroup
          config["linux"]["cgroupsPath"] = nil
     end

     if (config["mounts"] ~= nil)
     then
          -- Find and remove any user/group settings in mounts
          for _, mount in ipairs(config["mounts"]) do
               local opts = {}

               if (mount["options"] ~= nil)
               then
                    for _, opt in ipairs(mount["options"]) do
                         if ((string.sub(opt, 1, 4) ~= "gid=") and (string.sub(opt, 1, 4) ~= "uid="))
                         then
                              table.insert(opts, opt)
                         end
                    end
               end

               if (opts ~= nil and #opts > 0)
               then
                    mount["options"] = opts
               else
                    mount["options"] = nil
               end
          end

          -- Remove all bind mounts by copying files into rootfs
          local mounts = {}
          for i, mount in ipairs(config["mounts"]) do
               if ((mount["type"] ~= nil) and (mount["type"] == "bind") and (string.sub(mount["source"], 1, 4) ~= "/sys") and (string.sub(mount["source"], 1, 5) ~= "/proc"))
               then
                    status, output = slurm.allocator_command(string.format("/usr/bin/env rsync --numeric-ids --ignore-errors --stats -a -- %s %s", mount["source"], dst_rootfs..mount["destination"]))
                    if (status ~= 0)
                    then
                         -- rsync can fail due to permissions which may not matter
                         slurm.log_info("rsync failed")
                    end
               else
                    table.insert(mounts, mount)
               end
          end
          config["mounts"] = mounts
     end

     -- Force version to one compatible with older runc/crun at risk of new features silently failing
     config["ociVersion"] = "1.0.0"

     -- Merge in Job environment into container -- this is optional!
     if (config["process"]["env"] == nil)
     then
          config["process"]["env"] = {}
     end
     for _, env in ipairs(job_env) do
          table.insert(config["process"]["env"], env)
     end

     -- Remove all prestart hooks to squash any networking attempts
     if ((config["hooks"] ~= nil) and (config["hooks"]["prestart"] ~= nil))
     then
          config["hooks"]["prestart"] = nil
     end

     -- Remove all rlimits
     if ((config["process"] ~= nil) and (config["process"]["rlimits"] ~= nil))
     then
          config["process"]["rlimits"] = nil
     end

     write_file(dst_config, slurm.to_json(config))
     slurm.log_info("created: "..dst_config)

     return slurm.SUCCESS
end

function slurm_scrun_stage_out(id, bundle, orig_bundle, root_path, orig_root_path, spool_dir, config_file, jobid, user_id, group_id)
     if (root_path == nil)
     then
          root_path = ""
     end

     slurm.log_debug(string.format("stage_out(%s, %s, %s, %s, %s, %s, %s, %d, %d, %d)",
                 id, bundle, orig_bundle, root_path, orig_root_path, spool_dir, config_file, jobid, user_id, group_id))

     if (bundle == orig_bundle)
     then
          slurm.log_info(string.format("skipping stage_out as bundle=orig_bundle=%s", bundle))
          return slurm.SUCCESS
     end

     status, output = slurm.allocator_command(string.format("/usr/bin/env rsync --numeric-ids --delete-after --ignore-errors --stats -a -- %s/ %s/", root_path, orig_root_path))
     if (status ~= 0)
     then
          -- rsync can fail due to permissions which may not matter
          slurm.log_info("rsync failed")
     else
          -- cleanup temporary after they have been synced backed to source
          slurm.allocator_command(string.format("/usr/bin/rm --preserve-root=all --one-file-system -dr -- %s", bundle))
     end

     return slurm.SUCCESS
end

slurm.log_info("initialized scrun.lua")

return slurm.SUCCESS

💥 SIGNALS 💥

the ways you can tell it to wrap it up 🫶

SIGINT — attempt to gracefully cancel any related jobs (if any) and cleanup. polite exit energy, love that for us ✨
SIGCHLD — wait for all children, cleanup anchor and gracefully shutdown. it's giving "let everyone finish their sentence before the group chat dies" 💀🌸

👑 COPYING (the legal tea, bestie, don't skip it)

This file is part of Slurm, a resource management program. For details, see https://slurm.schedmd.com/. 🌈

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. 💖

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. 😭

ngl the all-caps part is not me being dramatic for once, that's literally how it's written and it literally means it. she's free, she's open, she's got zero promises about working for your specific use case. read the license, bestie, it's giving responsible queen. 💅

🦋 SEE ALSO (your little reading list, mwah)

slurm(1) — the mothership, the blueprint, the origin story 🫶
oci.conf(5) — how scrun talks to your container runtime, iconic config file energy
srun(1) — launching jobs, the classic, we stan
crun
runc
DOCKER and podman 👑

okay so that's literally the whole page bestie, you made it to the end, slay 🎀 this is Slurm 26.05 from the Slurm Commands collection, and it's all about scrun(1) — go containerize something cute, run your OCI bundles, and remember: no cap, the manpage was right the whole time. 💜🌸✨

📦 where this page comes from

arch rolling: slurm-llnl 26.05.5.1-1 · GPL-2.0-or-later
epel 10: slurm 26.05.5-1.el10_3 · GPL-2.0-or-later AND BSD-2-Clause AND BSD-1-Clause
nixpkgs unstable: slurm 26-05-5-1
ubuntu stonking: slurm-client 26.05.4-1ubuntu1 · GPL-2.0-or-later WITH OpenSSL exception
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-03: a language model's rewrite of the original, under its terms