scancel(1)

📦
version 26.05.5.1 · 8 versions
, in arch rolling, epel 10, nixpkgs unstable, ubuntu stonking

💖 scancel(1) — Slurm Commands 💖

okay bestie, sit down, hydrate, and let me tell you about scancel, a.k.a. the Slurm command that says "girl, you're done" to jobs and job steps. this is part of the Slurm Commands family, and it's lowkey one of the most useful ones in the whole toolkit. 🎀

🌸 NAME 🌸

scancel — used to signal jobs or job steps that are under the control of Slurm.

✨ SYNOPSIS ✨

scancel [OPTIONS...] [job_id[_array_id][.step_id]]
[job_id[_array_id][.step_id]...]

🦋 DESCRIPTION 🦋

so scancel is the one that signals or cancels jobs, job arrays, or job steps. you can signal literally an arbitrary number of them, either by using job specification filters or by handing it a space separated list of specific job and/or job step IDs. no cap. 💅

job arrays are where it gets spicy, so listen up:

  • if you specify the job ID of a job array with an array ID value, and the job associated with that array ID value has been split from the array, then only that one job array element gets cancelled.
  • if you specify the job ID of a job array without an array ID value, or the array ID value corresponds to a job that has not been split from the array, then all job array elements get cancelled.

now, heterogeneous jobs (het jobs, the main characters with many components):

  • while a heterogeneous job is in a PENDING state, you can only cancel the entire job — not its individual components. a request to cancel an individual component while it's PENDING will return an error. like, that's not a vibe, that's just how it is. 😭
  • after the job has begun execution, an individual component can be cancelled — except for component zero. and if component zero is cancelled, the whole het job is cancelled. periodt.

step IDs are unique across the whole heterogeneous job. 🌈

  • cancelling a step through the job leader kills that step in every component that has it, and leaves the component allocations running.
  • cancelling a step through an individual component signals only that component's step. it is not an error for a component not to have that step; only a step ID that no component has gets reported as an invalid job ID.
  • signals other than the default SIGKILL are sent only to the step of the component whose job ID was given.

(and yes bestie, the original page says the leader thing twice in slightly different words — component zero is the heterogeneous job leader, and cancelling a step through it kills that step in every component that has it while leaving the component allocations running. also, it's still not an error if some components don't have that step; only a step ID that no component has is reported as an invalid job ID. and cancelling a step through an individual component signals only that component's step — if that component doesn't have the step, the request is reported as an invalid job ID. two ways of saying one truth, we love a consistent queen 👑)

one more thing, and it's important:

a job or job step can only be signaled by the owner of that job or by user root. if an unauthorized user tries it, an error message gets printed and the job will not be signaled. bye. 🙈

💅 OPTIONS 💅

  • -A, --account=account — restrict the scancel operation to jobs under this charge account.
  • --admin-comment=comment — set the AdminComment on the job while canceling jobs. user must have Administrator privileges on the system.
  • -b, --batch — by default, signals other than SIGKILL are not sent to the batch step (the shell script). with this option scancel signals only the batch step, but not any other steps. this is useful when the shell script has to trap the signal and take some application defined action. most shells cannot handle signals while a command is running (i.e. is a child process of the batch step), so the shell needs to wait until the command ends to then handle the signal. children of the batch step are not signaled with this option — if that's what you want, use -f, --full instead. NOTE: if used with -f, --full, this option is ignored. NOTE: this option is not applicable if step_id is specified. NOTE: the shell itself may exit upon receipt of many signals. you may avoid this by explicitly trap signals within the shell script (e.g. "trap <arg> <signals>"). see the shell documentation for details.
  • -M, --clusters=<string> — cluster to issue commands to. implies --ctld. note that the slurmdbd must be up for this option to work properly, unless running in a federation with FederationParameters=fed_display configured.
  • --ctld — if this option is not used with --interactive, --sibling, federated job ids, or specific step ids, then this issues a single request to the slurmctld to signal all jobs matching the specified filters. this greatly improves the performance of slurmctld and scancel. otherwise, this option causes scancel to send each job signal request to the slurmctld daemon rather than directly to the slurmd daemons, which increases overhead, but offers better fault tolerance. --ctld is the default behavior on when the --clusters option is used.
  • -c, --cron — confirm request to cancel a job submitted by scrontab. this option only has effect with the "explicit_scancel" option is set in ScronParameters.
  • -f, --full — by default, signals other than SIGKILL are not sent to the batch step (the shell script). with this option scancel also signals the batch script and its children processes. most shells cannot handle signals while a command is running (i.e. is a child process of the batch step), so the shell needs to wait until the command ends to then handle the signal. unlike -b, --batch, children of the batch step are also signaled with this option. NOTE: srun steps are also children of the batch step, so steps are also signaled with this option.
  • --help — print a help message describing all scancel options. 🫶
  • -H, --hurry — do not stage out any burst buffer data.
  • -i, --interactive — interactive mode. confirm each job_id.step_id before performing the cancel operation.
  • -n, --jobname=job_name, --name=job_name — restrict the scancel operation to jobs with this job name.
  • --me — restrict the scancel operation to jobs owned by the current user. (it's giving main character energy, honestly ✨)
  • -w, --nodelist=host1,host2,... — cancel any jobs using any of the given hosts. the list may be specified as a comma-separated list of hosts, a range of hosts (host[1-5,7,...] for example), or a filename. the host list will be assumed to be a filename only if it contains a "/" character.
  • -p, --partition=partition_name — restrict the scancel operation to jobs in this partition.
  • -q, --qos=qos — restrict the scancel operation to jobs with this quality of service.
  • -Q, --quiet — do not report an error if the specified job is already completed. this option is incompatible with the --verbose option.
  • -R, --reservation=reservation_name — restrict the scancel operation to jobs with this reservation name.
  • --sibling=cluster_name — remove an active sibling job from a federated job.
  • -s, --signal=signal_name — the name or number of the signal to send. if this option is not used the specified job or step will be terminated.
  • -t, --state=job_state_name — restrict the scancel operation to jobs in this state. job_state_name may have a value of either "PENDING", "RUNNING" or "SUSPENDED".
  • --usage — print a brief help message listing the scancel options.
  • -u, --user=user_name — restrict the scancel operation to jobs owned by the given user.
  • -v, --verbose — print additional logging. multiple v's increase logging detail. this option is incompatible with the --quiet option.
  • -V, --version — print the version number of the scancel command.
  • --wckey=wckey — restrict the scancel operation to jobs using this workload characterization key.

🎀 ARGUMENTS 🎀

🌸 job_id

the Slurm job ID to be signaled. a SLUID may also be used in place of a numeric job ID. iconic, honestly.

🌸 step_id

the step ID of the job step to be signaled. if not specified, the operation is performed at the level of a job.

  • if neither --batch nor --signal are used, the entire job will be terminated.
  • when --batch is used, the batch shell processes will be signaled. the child processes of the shell will not be signaled by Slurm, but the shell may forward the signal.
  • when --batch is not used but --signal is used, then all job steps will be signaled, but the batch script itself will not be signaled.

🔥 PERFORMANCE 🔥

okay this part matters, bestie, don't skip it. 🫶

when executing scancel without the --ctld option; or with the --ctld option and --interactive, --sibling, or specific step ids — a remote procedure call is sent to slurmctld to get all the jobs. scancel then sends a signal job remote procedure call for each job that matches the requested filters. that's a lot of little calls, one per job.

when executing scancel with the --ctld option and without --interactive, --sibling, or specific step ids — a single remote procedure call is sent to slurmctld to signal all jobs matching the requested filters. it is therefore recommended to use the --ctld option in order to reduce the number of remote procedure calls sent to the slurmctld. periodt. 👑

if enough calls from scancel or other Slurm client commands that send remote procedure calls to the slurmctld daemon come in at once, it can result in a degradation of performance of the slurmctld daemon, possibly resulting in a denial of service. 💀

do not run scancel or other Slurm client commands that send remote procedure calls to slurmctld from loops in shell scripts or other programs. ensure that programs limit calls to scancel to the minimum necessary for the information you are trying to gather. we do not spam the daemon in this house. 🙅‍♀️

🌈 ENVIRONMENT VARIABLES 🌈

some scancel options may be set via environment variables. these environment variables, along with their corresponding options, are listed below. (Note: command line options will always override these settings.)

  • SCANCEL_ACCOUNT — -A, --account=account
  • SCANCEL_BATCH — -b, --batch
  • SCANCEL_CTLD — --ctld
  • SCANCEL_CRON — -c, --cron
  • SCANCEL_FULL — -f, --full
  • SCANCEL_HURRY — -H, --hurry
  • SCANCEL_INTERACTIVE — -i, --interactive
  • SCANCEL_NAME — -n, --name=job_name
  • SCANCEL_PARTITION — -p, --partition=partition_name
  • SCANCEL_QOS — -q, --qos=qos
  • SCANCEL_STATE — -t, --state=job_state_name
  • SCANCEL_USER — -u, --user=user_name
  • SCANCEL_VERBOSE — -v, --verbose
  • SCANCEL_WCKEY — --wckey=wckey
  • SLURM_CONF — the location of the Slurm configuration file.
  • SLURM_CLUSTERS — -M, --clusters
  • SLURM_DEBUG_FLAGS — specify debug flags for scancel to use. see DebugFlags in the slurm.conf(5) man page for a full list of flags. the environment variable takes precedence over the setting in the slurm.conf.

💅 NOTES 💅

okay bestie, listen up, there's some lore here that you need:

  • if you hand over multiple filters at once (like --partition and --name together, iykyk), then only the jobs that satisfy all of those filtering options get signaled. it's giving "and", not "or", periodt. 🎀
  • cancelling a job step will not terminate the job itself. like, the job is still there, bestie. if you want that resource allocation back, you have to cancel the job. no cap. 😭
  • to cancel a job, just invoke scancel without the --signal option. here's the little choreography it does, and honestly it's kind of iconic:
    1. sends a SIGCONT to all steps first, to eventually wake them up 🌞
    2. then a SIGTERM
    3. then it waits the KillWait duration defined in the slurm.conf file ⏳
    4. and finally, if they still haven't terminated, it sends a SIGKILL 💀

so yeah, it's literally giving your running job/step(s) time to clean up. so thoughtful of it fr fr.

  • oh and heads up: if you send a signal value of "KILL" to an entire job, it will cancel the active job steps but not cancel the job itself. the way that's a whole trap 🙈

👑 AUTHORIZATION 👑

when you're using SlurmDBD, here's who gets to boss other people's jobs around:

  • users who have an AdminLevel defined (Operator or Admin) ✨
  • and users who are account coordinators 🫶

those people are given the authority to invoke scancel on other users' jobs. everyone else? stay in your lane bestie 💅

🌸 EXAMPLES 🌸

okay these are the main character examples, run them exactly like this:

Send SIGTERM to steps 1 and 3 of job 1234:

$ scancel --signal=TERM 1234.1 1234.3

Cancel job 1234 along with all of its steps:

$ scancel 1234

Send SIGKILL to all steps of job 1235, but do not cancel the job itself:

$ scancel --signal=KILL 1235

Send SIGUSR1 to the batch shell processes of job 1236:

$ scancel --signal=USR1 --batch 1236

Cancel all pending jobs belonging to user "bob" in partition "debug":

$ scancel --state=PENDING --user=bob --partition=debug

Cancel only array ID 4 of job array 1237

$ scancel 1237_4

like, look at that. --state=PENDING --user=bob --partition=debug with zero positional args? that's the vibe check passing. 🔥

📜 COPYING 📜

okay now the legal bit, and yes we read it, we don't skip it:

Copyright (C) 2002-2007 The Regents of the University of California. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). Copyright (C) 2008-2011 Lawrence Livermore National Security. Copyright (C) 2010-2022 SchedMD LLC.

This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

bestie the all-caps in that warranty section is not me being dramatic, that's literally how it's written. respect the caps. 😤

🦋 SEE ALSO 🦋

go read these next, they're the friends of this page:

  • slurm_kill_job (3)
  • slurm_kill_job_step (3)

and that's a wrap on scancel(1) from the Slurm Commands collection, page version Slurm 26.05 💖 go forth and cancel responsibly, bestie. slay. 👑✨🎀

📦 where this page comes from