HTCONDOR-JDL(1)

📦

version 25.14.1 · 5 versions

💖 HTCONDOR-JDL(1) — HTCondor Manual 💖

okay bestie, welcome to the HTCondor Job Description Language page, aka the htcondor-jdl manual. this is the whole entire manual, and she is long, so buckle up 🎀

✨ NAME ✨

htcondor-jdl — the HTCondor Manual, but make it a job-language diva 💅

🌸 DESCRIPTION 🌸

so the HTCondor Job Description Language (JDL) is the thing that defines all the commands you can put inside submit description files — yep, the ones you hand over to condor_submit. if you wanna know about the condor_submit tool itself and its command-line options, go read condor_submit(1), she's got her own page 🫶

🦋 SUBMIT DESCRIPTION FILE COMMANDS 🦋

quick note from the group chat: there's more tea on submitting HTCondor jobs at /users-manual/submitting-a-job 👑

the condor_submit language is lowkey flexible and supports multi-line values in commands. the syntax is literally the same as the configuration language (more deets here: admin-manual/introduction-to-configuration:multi-line values) ✨

every single submit description file describes one or more clusters of jobs headed for the HTCondor execution pool. here's the rule: all jobs in a cluster must share the same executable, but they can totally have different input and output files, and different program arguments. no cap.

the submit description file is generally the last command-line argument you give to condor_submit. and if you just… don't pass one? condor_submit will read the submit description from standard input. iconic behaviour honestly.

the submit description file must contain at least one executable command and at least one queue command. everything else has default actions, she's chill like that 💅

NOTE: a submit file that contains more than one executable command will produce multiple clusters when submitted. this is not generally recommended 🚩 and it is not allowed for submit files that are run as DAG node jobs by condor_dagman.

the commands that can show up in the submit description file are numerous — like, so many — and they're all listed here in alphabetical order by category. let's go 🌈

🌟 BASIC COMMANDS 🌟

arguments = <argument_list> — the list of arguments that get supplied to the executable as part of the command line.

in the java universe, the first argument must be the name of the class containing main ☕

there are two permissible formats for specifying arguments, and they go by the old syntax and the new syntax. the old syntax only supports white space characters within arguments in special circumstances; when you use it, the command line arguments get represented in the job ClassAd attribute Args. the new syntax supports uniform quoting of white space characters within arguments; when you use it, the command line arguments get represented in the job ClassAd attribute Arguments. ✨

💜 Old Syntax 💜

in the old syntax, individual command line arguments are delimited (separated) by space characters. to allow a double quote mark inside an argument, you escape it with a backslash — meaning the two character sequence \" becomes a single double quote mark within an argument.

further interpretation of the argument string differs depending on the operating system 👀 on Windows, the entire argument string is passed verbatim (other than that backslash in front of double quote marks) to the Windows application. most Windows applications will allow spaces within an argument value by surrounding the argument with double quote marks. in all other cases, there is no further interpretation of the arguments.

Example:

arguments = one \"two\" 'three'

Produces in Unix vanilla universe:

argument 1: one
argument 2: "two"
argument 3: 'three'

💜 New Syntax 💜

here are the rules for using the new syntax, bestie, read them carefully:

the entire string representing the command line arguments is surrounded by double quote marks. this permits the white space characters of spaces and tabs to potentially be embedded within a single argument. putting the double quote mark within the arguments is accomplished by escaping it with another double quote mark.
the white space characters of spaces or tabs delimit arguments.
to embed white space characters of spaces or tabs within a single argument, surround the entire argument with single quote marks.
to insert a literal single quote mark, escape it within an argument already delimited by single quote marks by adding another single quote mark.

Example:

arguments = "3 simple arguments"

Produces:

argument 1: 3
argument 2: simple
argument 3: arguments

Another example:

arguments = "one 'two with spaces' 3"

Produces:

argument 1: one
argument 2: two with spaces
argument 3: 3

And yet another example:

arguments = "one ""two"" 'spacey ''quoted'' argument'"

Produces:

argument 1: one
argument 2: "two"
argument 3: spacey 'quoted' argument

notice that in the new syntax, the backslash has no special meaning 😭 this is for the convenience of Windows users, okay, she's just being nice.

environment = <parameter_list> — the list of environment variables 🌱

there are two different formats for specifying the environment variables: the old format and the new format. the old format is retained for backward-compatibility. it suffers from a platform-dependent syntax and the inability to insert some special characters into the environment. rip 😔

the new syntax for specifying environment values, here we go:

put double quote marks around the entire argument string. this distinguishes the new syntax from the old. the old syntax does not have double quote marks around it. any literal double quote marks within the string must be escaped by repeating the double quote mark.
each environment entry has the form

<name>=<value>

use white space (space or tab characters) to separate environment entries.
to put any white space in an environment entry, surround the space and as much of the surrounding entry as desired with single quote marks.
to insert a literal single quote mark, repeat the single quote mark anywhere inside of a section surrounded by single quote marks.

Example:

environment = "one=1 two=""2"" three='spacey ''quoted'' value'"

Produces the following environment entries:

one=1
two="2"
three=spacey 'quoted' value

under the old syntax, there are no double quote marks surrounding the environment specification. each environment entry remains of the form

<name>=<value>

under Unix, you list multiple environment entries by separating them with a semicolon (;). under Windows, you separate multiple entries with a vertical bar (|). there is no way to insert a literal semicolon under Unix or a literal vertical bar under Windows 😬 note that spaces are accepted, but rarely desired, characters within parameter names and values, because they are treated as literal characters, not separators or ignored white space. place spaces within the parameter list only if required. fr fr.

A Unix example:

environment = one=1;two=2;three="quotes have no 'special' meaning"

This produces the following:

one=1
two=2
three="quotes have no 'special' meaning"

if the environment is set with the environment command and getenv is also set, values specified with environment override values in the submitter's environment (regardless of the order of the environment and getenv commands). she said what she said 💅

error = <pathname> — a path and file name that HTCondor uses to capture any error messages the program would normally write to the screen (so basically, this file becomes stderr) 🫣 a path is given with respect to the file system of the machine on which the job is submitted. the file is written (by the job) in the remote scratch directory of the machine where the job is executed. when the job exits, the resulting file is transferred back to the machine where the job was submitted, and the path is utilized for file placement. if you specify a relative path, the final path will be relative to the job's initial working directory, and HTCondor will create directories as necessary to transfer the file. if not specified, the default value of /dev/null is used for submission to a Unix machine. if not specified, error messages are ignored for submission to a Windows machine.

⚠️ more than one job should not use the same error file, since this will cause one job to overwrite the errors of another 💀

if HTCondor detects that the error and output files for a job are the same, it will run the job such that the output and error data is merged. two files become one, it's giving fusion cuisine ✨

executable = <pathname> — an optional path and a required file name of the executable file for this job cluster 🧚 only one executable command within a submit description file is guaranteed to work properly. more than one often works. (note the "often", bestie, that's not a promise 🙈)

if no path or a relative path is used, then the executable file is presumed to be relative to the current working directory of the user as the condor_submit command is issued.

shell = <shell command to run> — on Unix systems, instead of specifying executable and args, the shell command can give a full shell command to be run under /bin/sh, including shell special characters like pipes and redirects 🐚🔥

heads up though: if any executables run by this shell need to be transferred to the job, HTCondor will not automatically add those to the transfer_input list — you will manually need to add those programs. no shortcuts, sorry babe 😭

batch_name = <batch_name> — bestie this is just a lil label for your submit ✨ It sets the batch name for this submit, and it's what shows up when you run condor_q -batch. It's made for you, the user, to give your jobs actual meaningful names and to nudge how condor_q groups stuff for display. And heads up 🫶 — whatever you put here in the submit file can be overridden if you pass the -batch-name argument on the condor_submit command line. So the command line wins, periodt.
getenv = <<matchlist> | True | False> — the environment-variable situation, and it is giving options. 💅

If getenv is set to True, then condor_submit will copy all of your current shell environment variables at the time of job submission into the job ClassAd. So the job executes with the exact same set of environment variables you had at submit time. It defaults to False.

Ngl, a wholesale import of your environment is very likely to lead to problems executing the job on a remote machine — unless there's a shared file system for users' home directories between the access point and the execute machine. So instead of setting getenv to True, it is much better to set it to a list of environment variables to import. Do that. Trust. 🙈

Matchlist is a comma, semicolon or space separated list of environment variable names and name patterns that match or reject names. Matchlist members are matched case-insensitively to each name in the environment, and the ones that match get imported. Matchlist members can contain * as a wildcard character which matches anything at that position. Members can have two * characters if one of them is at the end. Members can be prefixed with ! to force a matching environment variable to not be imported. The order of members in the Matchlist has no effect on the result. And getenv = true is equivalent to getenv = *. 🌈

Prior to HTCondor 8.9.7, getenv allows only True or False as values. (Old lore, iykyk.)

Examples, and yes these go hard:

# import everything except PATH and INCLUDE (also path, include and other case-variants)
getenv = !PATH, !INCLUDE

# import everything with CUDA in the name
getenv = *cuda*

# Import every environment variable that starts with P or Q, except PATH
getenv = !path, P*, Q*

One more thing, and it matters: if the environment is set with the environment command and getenv is also set, then the values specified with environment override the values in the submitter's environment — regardless of the order of the environment and getenv commands. 💖

input = <pathname> — so here's the tea: HTCondor assumes its jobs are long-running, and that you, the user, are not gonna sit at the terminal waiting for them to finish. Because of that, the standard files that normally access the terminal — stdin, stdout, and stderr — must refer to files. 🎀

So the file name you give with input should contain any keyboard input the program requires; that is, this file becomes stdin. A path is given with respect to the file system of the machine on which the job is submitted. The file is transferred before execution to the remote scratch directory of the machine where the job is executed. If not specified, the default value of /dev/null is used for submission to a Unix machine. If not specified, input is ignored for submission to a Windows machine.

And don't get it twisted 😭 — this command does not refer to the command-line arguments of the program. The command-line arguments are specified by the arguments command. Different thing entirely, bestie.

log = <pathname> — use log to name a file where HTCondor will write a log file of what's happening with this job cluster, called a job event log 📝 For example, HTCondor will drop a log entry into this file when and where the job begins running, when it transfers files, if the job is evicted, and when the job completes. Most users find specifying a log file to be handy; its use is recommended. If no log entry is specified, HTCondor does not create a log for this cluster. If you give a relative path, it's relative to the current working directory as the job is submitted, or to the directory specified by the submit command initialdir on the access point. ✨
notification = <Always | Complete | Start | Error | Never> — owners of HTCondor jobs get notified by e-mail when certain events happen, according to the value of this command. Here's the breakdown, bestie 👑
- Always — e-mail is sent when the job terminates.
- Complete — e-mail is sent when the job terminates.
- Start — e-mail is sent on first job start.
- Error — e-mail is sent only if the job terminates abnormally (as defined by JobSuccessExitCode, if defined), or if the job is placed on hold because of a failure and not by user request.
- Never — never (the default); the owner will not receive e-mail regardless of what happens to the job. 💀

The HTCondor User's manual documents statistics included in the e-mail. 📬

notify_user = <email-address> — used to specify the e-mail address to use when HTCondor sends e-mail about a job. If it's not specified, HTCondor defaults to using the e-mail address defined by

job-owner@UID_DOMAIN

where the configuration variable UID_DOMAIN is specified by the HTCondor site administrator. And if UID_DOMAIN has not been specified, HTCondor sends the e-mail to:

job-owner@submit-machine-name

There you go, that's the whole fallback chain 🫶

output = <pathname> — the output file captures any information the program would ordinarily write to the screen; that is, this file becomes stdout. 🌸 A path is given with respect to the file system of the machine on which the job is submitted. The file is written (by the job) in the remote scratch directory of the machine where the job is executed. When the job exits, the resulting file is transferred back to the machine where the job was submitted, and the path is utilized for file placement.

If you specify a relative path, the final path will be relative to the job's initial working directory, and HTCondor will create directories as necessary to transfer the file. If not specified, the default value of /dev/null is used for submission to a Unix machine. If not specified, output is ignored for submission to a Windows machine.

Multiple jobs should not use the same output file, since this will cause one job to overwrite the output of another. The audacity 💀

If HTCondor detects that the error and output files for a job are the same, it will run the job such that the output and error data is merged. Slay, convenient.

And note that if a program explicitly opens and writes to a file, that file should not be specified as the output file. Okay? Okay. 💅

priority = <integer> — an HTCondor job priority can be any integer, with 0 being the default. Jobs with higher numerical priority will run before jobs with lower numerical priority. 🔥

But here's the catch: this priority is on a per user basis. One user with many jobs may use this command to order their own jobs, and this will have no effect on whether or not these jobs will run ahead of another user's jobs. It's your own personal queue drama, not a global one. 👑

Also note that the priority setting in an HTCondor submit file will be overridden by condor_dagman if the submit file is used for a node in a DAG, and the priority of the node within the DAG is non-zero (see DAG Node Priorities for more details).

queue [<int expr>] — places zero or more copies of the job into the HTCondor queue. Classic. The moment you've all been waiting for. ✨
queue [<int expr>] [<varname>] in [slice] <list of items> — places zero or more copies of the job in the queue based on items in a <list of items>. 🦋
queue [<int expr>] [<varname>] matching [files | dirs] [slice] <list of items with file globbing> — places zero or more copies of the job in the queue based on files that match a <list of items with file globbing>. File globbing, bestie, it eats. 🍓
queue [<int expr>] [<list of varnames>] from [slice] <file name> | <list of items> — places zero or more copies of the job in the queue based on lines from the submit file or from <file name>. Very main character energy. 👑

Okay and now the part that ties all these forms together, listen up:

The optional argument <int expr> specifies how many times to repeat the job submission for a given set of arguments. It may be an integer or an expression that evaluates to an integer, and it defaults to 1. All but the first form of this command are various ways of specifying a list of items. When these forms are used, <int expr> jobs will be queued for each item in the list. The in, matching and from keyword indicates how the list will be specified. And that's the whole vibe check on queue. 💜

in — the list of items is just an explicit comma and/or space separated <list of items>. 💫 And plot twist: if that <list of items> starts with an open paren, and the close paren is not on the same line as the open one, then the list just keeps going — it does not stop until a line that begins with a close paren gets read from the submit file. iykyk.
matching — every item in the <list of items with file globbing> gets matched against the names of files and directories relative to the current directory, and the set of matching names is the resulting list of items. lowkey the most satisfying form fr.
- files — only filenames get matched.
- dirs — only directory names get matched.
from <file name> | <list of items> — each line from <file name> or <list of items> is a single item, and this is the one that lets you set multiple variables for each item. ✨ Lines from <file name> or <list of items> get split on comma and/or space until there are values for each of the variables specified in <list of varnames> — and the last variable gets the remainder of the line. When you use the <list of items> form, the list keeps going until the first line that begins with a close paren, and lines beginning with pound sign ('#') get skipped. 🙈 When you use the <file name> form, if the <file name> ends with |, then it gets executed as a script, and whatever the script writes to stdout will be the list of items. not me thinking that's iconic.

The optional argument <varname> or <list of varnames> is the name (or the names) of variables that get set to the value of the current item when queuing the job. If no <varname> is specified, the variable ITEM gets used. Leading and trailing whitespace gets trimmed. The optional argument <slice> is a python style slice that selects only some of the items in the list of items. Negative step values are not supported. ⚠️

A submit file may contain more than one queue statement, and if you want, any commands can sit between subsequent queue commands — stuff like new input, output, error, initialdir, or arguments commands. 🫶 This is so handy when you're submitting multiple runs into one cluster with one submit description file.

universe = <vanilla | scheduler | local | grid | java | vm | parallel | docker | container> — says which HTCondor universe to use when running this job. A HTCondor universe is an HTCondor execution environment, bestie. Here's the full roster: 👑
- vanilla — the default (except where the configuration variable DEFAULT_UNIVERSE defines it otherwise).
- scheduler — for a job that is to run on the machine where the job is submitted. Intended for a job that acts as a metascheduler and will not be preempted.
- local — for a job that is to run on the machine where the job is submitted. Runs the job immediately and will not preempt the job.
- grid — forwards the job to an external job management system. Further specification is done with the grid_resource command.
- java — for programs written to the Java Virtual Machine.
- vm — facilitates the execution of a virtual machine.
- parallel — for parallel jobs (e.g. MPI) that require multiple machines in order to run.
- docker — runs a docker container as an HTCondor job.
- container — runs a container as an HTCondor job using a supported container runtime system on the Execution Point.
max_materialize = <limit> — submits jobs as a late materialization factory and tells the condor_schedd to keep the given number of jobs materialized. Use this one to reduce the load on the condor_schedd when you're submitting a large number of jobs. The limit can be an expression, but it must evaluate to a constant at submit time. A limit less than 1 gets treated as unlimited. The condor_schedd can be configured to have a materialization limit too — and the lower of the two limits is the one that gets used. (see users-manual/submitting-a-job:submitting lots of jobs for more details).
max_idle = <limit> — submits jobs as a late materialization factory and tells the condor_schedd to keep the given number of non-running jobs materialized. Again, use this to reduce the load on the condor_schedd when submitting a huge number of jobs. The limit may be an expression, but it must evaluate to a constant at submit time. Jobs in the Held state count as Idle for this limit. A limit of less than 1 will stop jobs from being materialized — although the factory will still be submitted to the condor_schedd. (see users-manual/submitting-a-job:submitting lots of jobs for more details).

🔥 COMMANDS FOR MATCHMAKING 🔥

rank = <ClassAd Float Expression> — a ClassAd Floating-Point expression that states how to rank machines that have already met the requirements expression. Basically, rank expresses preference. A higher numeric value equals a better rank. HTCondor gives the job the machine with the highest rank. 🌈 For example:

request_memory = max({60, Target.TotalSlotMemory})
rank = Memory

This asks HTCondor to find all available machines with more than 60 megabytes of memory and give the job the machine with the most amount of memory. The HTCondor User's Manual has the complete info on the syntax and the available attributes that can be used in the ClassAd expression.

request_cpus = <num-cpus> — a requested number of CPUs (cores). If you don't specify it, the number requested will be 1. If you do specify it, the expression

&& (RequestCpus <= Target.Cpus)

gets appended to the requirements expression for the job.

For pools that enable dynamic condor_startd provisioning, this specifies the minimum number of CPUs requested for this job, which results in a dynamic slot being created with this many cores. ⭐

request_disk = <quantity> — the requested amount of disk space, in KiB, requested for this job. If you don't specify it, it will be set to the job ClassAd attribute DiskUsage. The expression

&& (RequestDisk <= Target.Disk)

gets appended to the requirements expression for the job.

For pools that enable dynamic condor_startd provisioning, a dynamic slot will be created with at least this much disk space.

Characters may be appended to a numerical value to indicate units. 💅 K or KB indicates KiB, 2^10 numbers of bytes. M or MB indicates MiB, 2^20 numbers of bytes. G or GB indicates GiB, 2^30 numbers of bytes. T or TB indicates TiB, 2^40 numbers of bytes.

retry_request_disk = <quantity> [, <quantity2> [...]] — the amount of disk in Kb the job should request if it gets evicted from a slot for using more than the original request_disk value.

The value can be a single quantity, or a comma separated list of quantities. If more than one quantity is specified, each one must be larger than the one before it. 📈

Okay so units first, bestie: you can append characters to a numerical value to indicate units. K or KB indicates KiB, 2^ 10 numbers of bytes. M or MB indicates MiB, 2^ 20 numbers of bytes. G or GB indicates GiB, 2^ 30 numbers of bytes. T or TB indicates TiB, 2^ 40 numbers of bytes. Memorize it, it comes back like five times. 💅

💾 `request_memory = <quantity>`

This is the amount of memory your job needs, in Mb. If you don't specify it, the value is set by the configuration variable JOB_DEFAULT_REQUESTMEMORY. The actual amount of memory used by a job is represented by the job ClassAd attribute MemoryUsage. Facts.

Characters may be appended to a numerical value to indicate units. K or KB indicates KiB, 2^ 10 numbers of bytes. M or MB indicates MiB, 2^ 20 numbers of bytes. G or GB indicates GiB, 2^ 30 numbers of bytes. T or TB indicates TiB, 2^ 40 numbers of bytes.

The expression

&& (RequestMemory <= Target.Memory)

is appended to the requirements expression for the job. So it's not just a vibe, it's a literal requirement.

🔁 `retry_request_memory = <quantity> [, <quantity2> [...]]`

The amount of memory in Mb the job should request if it is evicted from a slot for using more than the original request_memory value.

The value can be a single quantity, or a comma separated list of quantities. If more than one quantity is specified, each must be larger than the previous one. No downgrades, only glow-ups.

When retry_request_memory is specified, when the job is evicted from a slot for using too much memory, the memory request of the job will be adjusted to the next quantity and the job will run again rather than being held. When the job is already requesting the last quantity in the list, it will be held if it uses too much memory. Caught lacking at the max? Held. Periodt.

In the case where a cluster of jobs usually uses a smaller amount of memory, but occasionally needs a larger amount of memory, this command will allow the majority of jobs to match with smaller memory slots, and perhaps allow more of them to match and start, without wasting the larger amount of memory only seldomly needed. So, if most jobs in a cluster can run with 1 GB of memory, but some need 4 GB, then a submit file with:

request_memory = 1 GB
retry_request_memory = 4 GB

will start every job with 1 Gb of memory, and those that fail due to exceeding the memory will be automatically restarted with 4 GB of memory. Iconic adaptive behavior, honestly.

🔁➕ `retry_request_memory_increase = <quantity or expression>`

The amount of memory in Mb that should be added to the current memory request when the job is evicted from a slot for using too much memory. When used along with retry_request_memory_max, the memory request will continue to increase util it hits the max value. If a job that is at the max value is evicted for using too much memory it will be put on hold. If this submit command is used without retry_request_memory_max, the memory request will only be increased once. One shot, then no more.

The increase can be either a quantity to add to the memory request, or an expression that uses RequestMemory. For example, and note that because RequestMemory * 4 is a classad expression, a unit suffix is not allowed after the 4:

request_memory = 1 GB
retry_request_memory_increase = RequestMemory * 4
retry_request_memory_max = 16 GB

🔝 `retry_request_memory_max = <quantity>`

The maximum amount of memory in Mb that should be requested by the job when a retry_request_memory_increase is specified. If this command is used without retry_request_memory_increase it behaves the same as retry_request_memory with a single quantity. Basically the ceiling, bestie.

🎮 `request_gpus = <num-gpus>`

A requested number of GPUs. If not specified, no GPUs will be requested. If specified one of the expressions below

&& (TARGET.GPUs >= RequestGPUs)
   or
&& (countMatches(MY.RequireGPUs,TARGET.AvailableGPUs) >= RequestGPUs)

is appended to the requirements expression for the job. The first expression above is used when there is no constraint on the GPU properties specified in the submit file. The second expression is used when require_gpus or one of the GPU property constraints such as gpus_minimum_memory is used.

For pools that enable dynamic condor_startd provisioning, request_gpus specifies the minimum number of GPUs requested for this job, resulting in a dynamic slot being created with this many GPUs with the required properties. GPU go brrr.

🎮🔒 `require_gpus = <constraint-expression>`

A constraint on the properties of GPUs when used with a non-zero request_gpus value. This expression will be combined with constraints generated by the use of one or more of gpus_minimum_capability, gpus_minimum_memory, gpus_minimum_runtime, or gpus_maximum_capability. If not specified, no constraint on GPUs will be added to the job. If specified and request_gpus is non-zero, the expression

&& (countMatches(MY.RequireGPUs, TARGET.AvailableGPUs) >= RequestGPUs)

is appended to the requirements expression for the job. This expression cannot be evaluated by HTCondor prior to version 9.8.0. A warning to this will effect will be printed when condor_submit detects this condition. Version check yourself before you wreck yourself.

For pools that enable dynamic condor_startd provisioning and are at least version 9.8.0, the constraint will be tested against the properties of AvailableGPUs and only those that match will be assigned to the dynamic slot.

Also there's literally just a request_GPUs line sitting here in the source. No description. No explanation. Just request_GPUs. We're including it because the rules say every detail stays. 💀

🧩 `request_<name> = <quantity>`

The required amount of the custom machine resource identified by <name> that this job needs. The custom machine resource is defined in the machine's configuration. Machines that have available GPUs will define <name> to be GPUs. <name> must be at least two characters, and must not begin with _. If <name> is either Cpu or Gpu a warning will be printed since these are common typos. Spell check your resource names, bestie.

🎚️ `gpus_minimum_capability = <version>` / `gpus_maximum_capability = <version>`

The mininum or maximum required Capability value of the GPU, inclusive. Specified as a floating point value (for example 8.5). Use of one or more of these commands will create or modify the require_gpus expression unless that expression already references the GPU property Capabilities. When request_gpus is not used, these commands are ignored. Ignored! Like your ex's texts.

🧠 `gpus_minimum_memory = <quantity>`

The mininum quantity of GPU memory in MiB that a GPU must have in order to run the job.

Use of this command will create or modify the require_gpus expression unless that expression already references the GPU property GlobalMemoryMB. When request_gpus is not used, this command is ignored. Again, ignored without request_gpus.

⏱️ `gpus_minimum_runtime = <version>`

The version of the GPU (usually CUDA) runtime used or required by this job, specified as <major>.<minor> (for example, 9.1). If the minor version number is zero, you may specify only the major version number. A single version number of 1000 or higher is assumed to be the integer-coded version number using the Nvida convention of (major * 1000 + (minor * 10)). Use of this command will create or modify the require_gpus expression unless that expression already references the GPU property MaxSupportedVersion. When request_gpus is not used, this command is ignored. Nvida. That's what it says. We don't make the rules.

🧪 `cuda_version = <version>`

The version of the CUDA runtime, if any, used or required by this job, specified as <major>.<minor> (for example, 9.1). If the minor version number is zero, you may specify only the major version number. A single version number of 1000 or higher is assumed to be the integer-coded version number (major * 1000 + (minor % 100)).

This command has been superseded by gpus_minimum_runtime, but it will still work if machines advertise the CUDAMaxSupportedVersion attribute. This does not arrange for the CUDA runtime to be present, only for the job to run on a machine whose driver supports the specified version. Superseded but still functional. A queen never truly leaves.

📜 `requirements = <ClassAd Boolean Expression>`

The requirements command is a boolean ClassAd expression which uses C-like operators. In order for any job in this cluster to run on a given machine, this requirements expression must evaluate to true on the given machine.

For scheduler and local universe jobs, the requirements expression is evaluated against the Scheduler ClassAd which represents the the condor_schedd daemon running on the access point, rather than a remote machine. Like all commands in the submit description file, if multiple requirements commands are present, all but the last one are ignored. By default, condor_submit appends the following clauses to the requirements expression:

Arch and OpSys are set equal to the Arch and OpSys of the submit machine. In other words: unless you request otherwise, HTCondor will give your job machines with the same architecture and operating system version as the machine running condor_submit.
Cpus >= RequestCpus, but only if the job ClassAd attribute RequestCpus is actually defined. No attribute, no clause, bestie 💅
Disk >= RequestDisk, but only if the job ClassAd attribute RequestDisk is defined. If it's not defined, then Disk >= DiskUsage gets appended to the requirements instead. And what is DiskUsage?? Okay so it gets initialized to the size of the executable plus the size of any files you named in a transfer_input_files command. It literally exists so there's enough disk space on the target machine for HTCondor to copy over both the executable and the input files it needs. The DiskUsage attribute is the maximum total disk space the job needs, measured in kilobytes. And HTCondor auto-updates DiskUsage roughly every 20 minutes while the job runs, with however much space the job is using on the execute machine. Iconic, honestly.
Memory >= RequestMemory, but only if the job ClassAd attribute RequestMemory is defined.
If Universe is set to Vanilla, then FileSystemDomain gets set equal to the access point's FileSystemDomain. Slay.

Wanna peek at the requirements of a job you already submitted (plus literally everything else about the job's ClassAd)? Run condor_q -l — go look at the command reference for condor_q. And if you want the complete tea on the syntax and all the attributes you can use inside a ClassAd expression, the HTCondor Users Manual has you. 🫶

🚚 FILE TRANSFER COMMANDS 🚚

dont_encrypt_input_files = < file1,file2,file... > — a comma- and/or space-separated list of input files that should NOT be network encrypted when they get transferred by the file transfer mechanism. Listing files here overrides any config that would have used encryption. Heads up: each input file also has to be in the list given by transfer_input_files, okay? When you specify a path to an input file or directory, that path is the path to the file on the submit side. And you can use a single wild card character (*) in each file name. ✨
dont_encrypt_output_files = < file1,file2,file... > — a comma- and/or space-separated list of output files that should NOT be network encrypted when they get transferred back by the file transfer mechanism. Same deal: listing them here overrides config that would have used encryption. The output file(s) also have to either be in the list given by transfer_output_files, or be discovered and headed back through the file transfer mechanism anyway. When you specify a path to an output file or directory, that path is the path to the file on the execute side. A single wild card character (*) is allowed in each file name. 🌸
encrypt_execute_directory = <True | False> — defaults to False. Set it to True and HTCondor encrypts the contents of the remote scratch directory on the machine where the job runs. The job itself can't even tell — the encryption is transparent to it — but it means any files left behind on the execute machine's local disk (say, because of a system crash 💀) stay private. Also, condor_submit appends this to the job's requirements expression:

&& (TARGET.HasEncryptExecuteDirectory)

so the job only matches a machine that can encrypt the contents of the execute directory. Fair warning: this support is limited to Windows platforms using the NTFS file system, and Linux platforms with the ecryptfs-utils package installed.

encrypt_input_files = < file1,file2,file... > — a comma- and/or space-separated list of input files that should be network encrypted when transferred by the file transfer mechanism. Listing them here overrides config that would not have used encryption. Each input file must also be in the list given by transfer_input_files. When you specify a path to an input file or directory, that path is the path to the file on the submit side. A single wild card character (*) may be used in each file name. The encryption method used is whatever got agreed on in security negotiation — and if that negotiation failed, then the file transfer mechanism must also fail for files to be network encrypted. No vibes-based crypto here 🔥
encrypt_output_files = < file1,file2,file... > — a comma- and/or space-separated list of output files that should be network encrypted when transferred back by the file transfer mechanism. Listing them here overrides config that would not have used encryption. The output file(s) must also either be in the list given by transfer_output_files, or be discovered and transferred back by the file transfer mechanism. When you specify a path to an output file or directory, that path is the path to the file on the execute side. A single wild card character (*) may be used in each file name. And again: the encryption method is whatever security negotiation agreed on; if that negotiation failed, then the file transfer mechanism must also fail for files to be network encrypted. Periodt. 💅
erase_output_and_error_on_restart = < true | false> — if false, and when_to_transfer_output is ON_EXIT_OR_EVICT, then HTCondor will append to the output and error logs instead of erasing (truncating) them when the job restarts. Bestie, keep those receipts 🧾
max_transfer_input_mb = <ClassAd Integer Expression> — this integer expression sets the maximum allowed total size, in MiB, of the input files transferred for a job. Ngl, it does NOT apply to grid universe, or to files transferred via file transfer plug-ins. The expression can refer to attributes of the job. The special value -1 means no limit. If it's not defined, the value set by the configuration variable MAX_TRANSFER_INPUT_MB is used. If the observed size of all input files at submit time is bigger than the limit, the job gets placed on hold immediately, with a HoldReasonCode value of 32. And if the job passes that first test but the input files grow, or the limit shrinks, so the limit gets violated — the job gets put on hold at the moment the file transfer is attempted. 😭
max_transfer_output_mb = <ClassAd Integer Expression> — this integer expression sets the maximum allowed total size, in MiB, of the output files transferred for a job. Also does NOT apply to grid universe or files transferred via file transfer plug-ins. The expression can refer to attributes of the job. The special value -1 means no limit. If it's not set, the value set by the configuration variable MAX_TRANSFER_OUTPUT_MB is used. If the total size of the job's output files to be transferred is bigger than the limit, the job gets placed on hold with a HoldReasonCode value of 33. And here's the messy part: the output is transferred up until the point the limit is hit — so some files may be fully transferred, some partially, and some not at all. 🫠
output_destination = <destination-URL> — when present, this defines a URL that specifies both a plug-in and a destination for transferring the entire output sandbox, or a subset of output files as specified by the submit command transfer_output_files. The plug-in handles the transfer of all the files in the sandbox — except for the standard output and standard error files. By default those two files go back To the access point. If you want those two sent to the output_destination as well, set output and/or error to the same value as the output_destination. You can only use one of this command or output_directory. Full details live in the HTCondor Administrator's manual. 📦
output_directory = <dir> — when present, this defines the directory where the entire output sandbox, or a subset of output files as specified by the submit command transfer_output_files, will be transferred to. You can only use one of this command or output_destination. Want a separate sub-directory for each job? Then <dir> can include $() expansions. For example:

output_directory = /path/to/output/$INT(ClusterId,%05d)/$(JobId)

will transfer the output into a separate directory for each submission, with a separate subdirectory for each job. $INT(ClusterId,%05d) prints the ClusterId with a minimum of 5 digits using leading 0's — so if the ClusterId of the submission is 95, the output directories will be

/path/to/output/00095/95.0/
/path/to/output/00095/95.1/
...

Obsessed with a tidy little directory tree, actually 🌈

should_transfer_files = <YES | NO | IF_NEEDED> — okay so bestie, this one decides whether HTCondor moves your files to and from the remote machine where the job actually runs. The whole file-transfer mechanism is what lets you run jobs on machines that don't share a file system with the submit machine. 💅

Here's the vibe check on the values:

YES — always transfer files for the job. no cap. ✨
NO — disables HTCondor's file transfer mechanism. 🚫
IF_NEEDED — does not transfer files for the job if it gets matched with a resource in the same FileSystemDomain as the access point (and therefore, on a machine with the same shared file system). If the job is matched with a remote resource in a different FileSystemDomain, HTCondor will transfer the necessary files. it's giving "only when we have to". 🦋

For more information about this and other settings related to transferring files, see the HTCondor User's manual section on the file transfer mechanism.

NOTE: should_transfer_files is not supported for jobs submitted to the grid universe. periodt.

skip_filechecks = <True | False> — When True, file permission checks for the submitted job are disabled. When False, file permissions are checked; this is the behavior when this command is not present in the submit description file. File permissions are checked for read permissions on all input files, such as those defined by commands input and transfer_input_files, and for write permission to output files, such as a log file defined by log and output files defined with output or transfer_output_files. 🔥
skip_if_dataflow = <True | False> — A boolean value that defaults to False. When True, it marks this job as a dataflow job. Dataflow jobs are marked as completed and skipped if all of their output files exist and have newer modification dates than all of the input files, similar to how the "make" program works. very make-coded, ngl. 🧚
stream_error = <True | False> — If True, then stderr is streamed back to the machine from which the job was submitted. If False, stderr is stored locally and transferred back when the job completes. This command is ignored if the job ClassAd attribute TransferErr is False. The default value is False. This command must be used in conjunction with error, otherwise stderr will be sent to /dev/null on Unix machines and ignored on Windows machines. 😭
stream_input = <True | False> — If True, then stdin is streamed from the machine on which the job was submitted. The default value is False. The command is only relevant for jobs submitted to the vanilla or java universes, and it is ignored by the grid universe. This command must be used in conjunction with input, otherwise stdin will be /dev/null on Unix machines and ignored on Windows machines. lowkey important. 🎀
stream_output = <True | False> — If True, then stdout is streamed back to the machine from which the job was submitted. If False, stdout is stored locally and transferred back when the job completes. This command is ignored if the job ClassAd attribute TransferOut is False. The default value is False. This command must be used in conjunction with output, otherwise stdout will be sent to /dev/null on Unix machines and ignored on Windows machines. 🫶
transfer_executable = <True | False> — This command is applicable to jobs submitted to the grid and vanilla universes. If transfer_executable is set to False, then HTCondor looks for the executable on the remote machine, and does not transfer the executable over. This is useful for an already pre-staged executable. The default value is True. iconic for pre-staged stuff. 👑
transfer_input_files = < file1,file2,file... > — A comma-delimited list of all the files and directories to be transferred into the working directory for the job, before the job is started. By default, the file specified in the executable command and any file specified in the input command (for example, stdin) are transferred. 🍓

Okay so when a path to an input file or directory is specified, this specifies the path to the file on the submit side. The file is placed in the job's temporary scratch directory on the execute side, and it is named using the base name of the original path. For example, /path/to/input_file becomes input_file in the job's scratch directory. no cap. ✨

When a directory is specified, the behavior depends on whether there is a trailing path separator character. When a directory is specified with a trailing path separator, it is as if each of the items within the directory were listed in the transfer list. Therefore, the contents are transferred, but the directory itself is not. When there is no trailing path separator, the directory itself is transferred with all of its contents inside it. On platforms such as Windows where the path separator is not a forward slash (/), a trailing forward slash is treated as equivalent to a trailing path separator. An example of an input directory specified with a trailing forward slash is input_data/. 💅

For grid universe jobs other than HTCondor-C, the transfer of directories is not currently supported. 🚫

Symbolic links to files are transferred as the files they point to. Transfer of symbolic links to directories is not currently supported. 🦋

For vanilla and vm universe jobs only, a file may be specified by giving a URL, instead of a file name. The implementation for URL transfers requires both configuration and available plug-in. 🔌

If you have a plugin which handles https:// URLs (and HTCondor ships with one enabled), HTCondor supports pre-signing S3 URLs. This allows you to specify S3 URLs for this command, for transfer_output_remaps, and for output_destination. By pre-signing the URLs on the submit node, HTCondor avoids transferring your S3 credentials to the execute node. You must specify aws_access_key_id_file and aws_secret_access_key_file; you may specify aws_region, if necessary; see below. To use the S3 service provided by AWS, use S3 URLs of the following forms:

# For older buckets that aren't region-specific.
s3://<bucket>/<key>

# For newer, region-specific buckets.
s3://<bucket>.s3.<region>.amazonaws.com/<key>

To use other S3 services, where <host> must contain a .:

s3://<host>/<key>

# If necessary
aws_region = <region>

You may specify the corresponding access key ID and secret access key with s3_access_key_id_file and s3_secret_access_key_file if you prefer (which may reduce confusion, if you're not using AWS). 👑

If you must access S3 using temporary credentials, you may specify the temporary credentials using aws_access_key_id_file and aws_secret_access_key_file for the files containing the corresponding temporary token, and +EC2SessionToken for the file containing the session token. 🔑

Temporary credentials have a limited lifetime. If you are using S3 only to download input files, the job must start before the credentials expire. If you are using S3 to upload output files, the job must finish before the credentials expire. HTCondor does not know when the credentials will expire; if they do so before they are needed, file transfer will fail. 💀

HTCondor does not presently support transferring entire buckets or directories from S3. 🚫

HTCondor supports Google Cloud Storage URLs -- gs:// -- via Google's "interoperability" API. You may specify gs:// URLs as if they were s3:// URLs, and they work the same way. You may specify the corresponding access key ID and secret access key with gs_access_key_id_file and gs_secret_access_key_file if you prefer (which may reduce confusion). 🌈

Note that (at present), you may not provide more than one set of credentials for s3:// or gs:// file transfer; this implies that all such URLs download from or upload to the same service. 🎀

public_input_files = <file, file2> — A list of files on the AP that HTCondor should use a pre-configured HTTP server on the AP to transfer. These files will not be encrypted, and will be publicly fetchable by anyone who knows their name. 🙈

💅 `transfer_output_files = < file1,file2,file... >`

okay bestie, this command forms an explicit list of output files and directories to be transferred back from the temporary working directory on the execute machine to the access point. If there are multiple files, they must be delimited with commas. Setting transfer_output_files to the empty string ("") means that no files are to be transferred.

For HTCondor-C jobs and all other non-grid universe jobs, if transfer_output_files is not specified, HTCondor will automatically transfer back all files in the job's temporary working directory which have been modified or created by the job. Subdirectories are not scanned for output, so if output from subdirectories is desired, the output list must be explicitly specified. For grid universe jobs other than HTCondor-C, desired output files must also be explicitly listed. Another reason to explicitly list output files is for a job that creates many files, and the user wants only a subset transferred back.
For grid universe jobs other than with grid type condor, to have files other than standard output and standard error transferred from the execute machine back to the access point, do use transfer_output_files, listing all files to be transferred. These files are found on the execute machine in the working directory of the job.
When a path to an output file or directory is specified, it specifies the path to the file on the execute side. As a destination on the submit side, the file is placed in the job's initial working directory, and it is named using the base name of the original path. For example, path/to/output_file becomes output_file in the job's initial working directory. The name and path of the file that is written on the submit side may be modified by using transfer_output_remaps. Note that this remap function only works with files but not with directories.
When a directory is specified, the behavior depends on whether there is a trailing path separator character. When a directory is specified with a trailing path separator, it is as if each of the items within the directory were listed in the transfer list. Therefore, the contents are transferred, but the directory itself is not. When there is no trailing path separator, the directory itself is transferred with all of its contents inside it. On platforms such as Windows where the path separator is not a forward slash (/), a trailing forward slash is treated as equivalent to a trailing path separator. An example of an input directory specified with a trailing forward slash is input_data/.
For grid universe jobs other than HTCondor-C, the transfer of directories is not currently supported.
Symbolic links to files are transferred as the files they point to. Transfer of symbolic links to directories is not currently supported.

🎀 `transfer_checkpoint_files = < file1,file2,file3... >`

If present, this command defines the list of files and/or directories which constitute the job's checkpoint. When the job successfully checkpoints -- see checkpoint_exit_code -- these files will be transferred to the submit node's spool.
If this command is absent, the output is transferred instead.
If no files or directories are specified, nothing will be transferred. This is generally not useful.
The list is interpreted like transfer_output_files, but there is no corresponding remaps command.

🌈 `checkpoint_destination = <destination-URL>`

When present, defines a URL that specifies both a plug-in and a destination for the transfer of the entire checkpoint of a job. The plug-in does the transfer of files, and no files are sent back to the access point.

🦋 `preserve_relative_paths = < True | False >`

For vanilla and Docker -universe jobs (and others that use the shadow), this command modifies the behavior of the file transfer commands. When set to true, the destination for an entry that is a relative path in a file transfer list becomes its relative path, not its basename. For example, input_data/b (and its contents, if it is a directory) will be transferred to input_data/b, not b. This applies to the input, output, and checkpoint lists.
Trailing slashes are ignored when preserve_relative_paths is set.

👑 `transfer_output_remaps = < " name = newname ; name2 = newname2 ... " >`

This specifies the name (and optionally path) to use when downloading output files from the completed job. Normally, output files are transferred back to the initial working directory with the same name they had in the execution directory. This gives you the option to save them with a different path or name. If you specify a relative path, the final path will be relative to the job's initial working directory, and HTCondor will create directories as necessary to transfer the file.
name describes an output file name produced by your job, and newname describes the file name it should be downloaded to. Multiple remaps can be specified by separating each with a semicolon. If you wish to remap file names that contain equals signs or semicolons, these special characters may be escaped with a backslash. You cannot specify directories to be remapped.
Note that whether an output file is transferred is controlled by transfer_output_files. Listing a file in transfer_output_remaps is not sufficient to cause it to be transferred.

⭐ `transfer_plugins = < tag=plugin ; tag2,tag3=plugin2 ... >`

Specifies the file transfer plugins (see ../admin-manual/file-and-cred-transfer) that should be transferred along with the input files prior to invoking file transfer plugins for files specified in transfer_input_files. tag should be a URL prefix that is used in transfer_input_files, and plugin is the path to a file transfer plugin that will handle that type of URL transfer.

💖 `when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT | ON_SUCCESS >`

Here's the tea on when those output files actually move, bestie:

ON_EXIT — When the job completes (exits on its own). If a job is evicted and started again, the subsequent execution will start with only the executable and input files in the scratch directory sandbox.
ON_EXIT_OR_EVICT — When the job completes (exits on its own) and when the job is evicted. When the job is evicted, HTCondor will transfer the output files to a temporary directory on the submit node (determined by the SPOOL configuration variable). When the job restarts, these files will be transferred instead of the input files.
ON_SUCCESS — Only when the job completes successfully. Success is defined by the success_exit_code command, which must be set, even if the successful value is the default 0. This prevents the job from going on hold if it does not produce all of the output files when it fails.

If transfer_output_files is not set, HTCondor considers all new files in the sandbox's top-level directory to be the output; subdirectories and their contents will not be transferred.

In all three cases, the job will go on hold if transfer_output_files specifies a file which does not exist at transfer time.

🍓 `aws_access_key_id_file = file_name OR s3_access_key_id_file = file_name`

One of these commands is required if you specify an s3:// URL; they specify the file containing the access key ID (and only the access key ID) used to pre-sign the URLs. Use only one.
aws_secret_access_key_file = file_name OR s3_secret_access_key_file = file_name — bestie, if you're rolling with an s3:// URL then one of these is non-negotiable 🫶. They point at the file that holds your secret access key — and literally nothing else but the secret access key — the one used to pre-sign the URLs. Pick exactly one of them, do not do both, that's the vibe check 💅
aws_region = <name of aws region> — optional if you're using an S3 URL (and straight-up ignored if you're not, no hard feelings 💀). It tells HTCondor which region to use when the URL itself doesn't already say. Region-free URL? This is your moment. ✨
gs_access_key_id_file = path to key file — required if you're using gs:// URLs, periodt. Give it the file containing the access key ID (only the access key ID, nothing extra) that's used to pre-sign the URLs 🔑
gs_secret_access_key_file = path to secret access key file — also required if you're using gs:// URLs 🌸. This one is the file with the secret access key (just the secret access key), and yeah, it's what gets used to pre-sign the URLs.

💖 POLICY COMMANDS 💖

allowed_execute_duration = <integer> — the longest stretch, in seconds, that your job is allowed to actually be executing. Blow past it and your job goes on hold with hold code 47. (peep codes-other-values/hold-reason-codes:Hold Reason Codes for the deets 👀)

Heads up tho: this clock does not count file-transfer time. And if your job self-checkpoints, it gets this long to write out each checkpoint 🎀

The whole point? Minimize the time wasted by jobs that are out here running forever by accident. We're saving your cycles, bestie. ⏳

allowed_job_duration = <integer> — the longest time, in seconds, that a job can continuously sit in the running state. Go over and boom — hold, with hold code 46 (again, codes-other-values/hold-reason-codes:Hold Reason Codes 🫶). Exiting the running state resets the job duration this command is measuring.

Same energy as above: it exists so jobs that erroneously run forever don't eat your whole day. Waste not, want not. ✨

max_retries = <integer> — the max number of retries your job gets (has to be non-negative, no sneaky negatives 🙈). If the job fails — meaning it doesn't exit with the success_exit_code exit code — it gets retried up to max_retries times, unless retries get stopped early by the retry_until command. If max_retries isn't defined but either retry_until or success_exit_code is, then the value of DEFAULT_JOB_MAX_RETRIES steps in as the max number of retries 👑

The holy trinity of max_retries, retry_until, and success_exit_code together auto-generates an appropriate OnExitRemove expression. And if you've defined both retry command(s) and on_exit_remove, the OnExitRemove expression gets made by OR'ing the expression you wrote in OnExitRemove with the one the retry commands generated. Teamwork, fr fr. 💅

retry_until = <Integer | ClassAd Boolean Expression> — an integer or boolean expression that stops any further retries, even if max_retries hasn't been used up yet 🛑. If retry_until is an integer, then the job exiting with that exit code makes retries cease. If retry_until is a ClassAd expression, the expression evaluating to True makes retries cease.

So like, say you only want to retry exit codes 17, 34, and 81:

max_retries = 5
retry_until = !member( ExitCode, {17, 34, 81} )

success_exit_code = <integer> — which exit code counts as a win for this job. If you don't define it, it defaults to 0.

Note: non-zero values of success_exit_code should generally not be used for DAG node jobs, unless when_to_transfer_output is set to ON_SUCCESS, and that's so you dodge failed jobs going on hold 😭

Right now, condor_dagman does not take the value of success_exit_code into account. Which means: if success_exit_code is set to a non-zero value, condor_dagman will call the job failed when it actually succeeded. For single-proc DAG node jobs, you can work around this with a POST script that takes the value of success_exit_code into account (though this is not recommended, bestie 🙈). For multi-proc DAG node jobs, there's currently no way around this limitation. 💀

checkpoint_exit_code = <integer> — the exit code that signals the executable exited after successfully taking a checkpoint 🎀. The checkpoint gets transferred and the executable restarted. Peep users-manual/self-checkpointing-applications:Self-Checkpointing Applications for the full story ✨
hold = <True | False> — set hold to True and your submitted job lands straight in the Hold state. Jobs chilling in Hold won't run until condor_release frees them. Defaults to False 🫶
keep_claim_idle = <integer> — a whole number of seconds where the job asks condor_schedd to wait before releasing its claim, either after the job exits or after the job gets removed ⏳

Okay so context, because this one slaps: claiming a condor_startd is kind of time-consuming. To amortize that cost, condor_schedd tries to reuse claims to run subsequent jobs once a job using a claim is done. But it can only do that if there's an idle job sitting in the queue at the exact moment the previous job completes. Sometimes — especially for node jobs with DAGMan — there's a subsequent job about to be submitted, but it hasn't arrived in the queue yet when the previous job finishes. So condor_schedd releases the claim, and the next job has to wait an entire negotiation cycle to start 😭

So: when this submit command is defined with a non-negative integer, and the job exits, condor_schedd tries as usual to reuse the claim. Can't? Then instead of releasing it, condor_schedd keeps the claim until either the number of seconds you gave as the parameter passes, or a new job that matches that claim arrives — whichever comes first 🏁 The condor_startd in question stays in the Claimed/Idle state, and the original job gets "charged" (in terms of priority) for the time spent in that state. Fair's fair.

leave_in_queue = <ClassAd Boolean Expression> — when this ClassAd expression evaluates to True, the job does not leave the queue on completion 🫶. This lets the user of a remotely spooled job retrieve output files in cases where HTCondor would've removed them as part of the cleanup that comes with completion. The job only leaves the queue once the expression evaluates to False.

leave_in_queue normally defaults to False. But when a job is submitted with spooled data files, leave_in_queue defaults to an expression that keeps the job in the queue until it's removed (like with condor_rm), or the spooled data files are retrieved, or the job has been in Completed state for 10 days ⏰

next_job_start_delay = <ClassAd Boolean Expression> — this expression says how many seconds to delay after starting up this job before the next job is started. The max allowed delay is set by the HTCondor configuration variable MAX_NEXT_JOB_START_DELAY, which defaults to 10 minutes. This command does not apply to scheduler or local universe jobs 🙅‍♀️

Historically this was used to implement a form of job start throttling from the submitter's perspective — useful when multiple job submission plus transferring extremely large input data sets to the execute machine made machine performance suffer. But this command is no longer useful, because throttling should be done through configuration of the condor_schedd daemon instead. Old tech, retiring gracefully ✨

on_exit_hold = <ClassAd Boolean Expression> — this ClassAd expression gets checked when the job exits, and if it's True, the job goes into the Hold state. If it's False (which is the default value when it's not defined), nothing happens and the on_exit_remove expression gets checked to figure out if that needs to be applied.

Example time 🎀 — suppose a job is known to run for a minimum of an hour. If it exits after less than an hour, the job should go on hold and an e-mail notification should be sent, instead of being allowed to leave the queue:

on_exit_hold = (time() - JobStartDate) < (60 * $(MINUTE))

This expression places the job on hold if it exits for any reason before running for an hour. An e-mail goes to the user explaining that the job was placed on hold because this expression became True 📧

Precedence tea ☕: periodic_* expressions take precedence over on_exit_* expressions, and *_hold expressions take precedence over *_remove expressions.

Only job ClassAd attributes will be defined for this ClassAd expression to use. This expression is available for the vanilla, java, parallel, grid, local and scheduler universes 👑

on_exit_hold_reason = <ClassAd String Expression> — when the job goes on hold because the on_exit_hold expression became True, this expression gets evaluated to set the value of HoldReason in the job ClassAd 🌸. If this expression is UNDEFINED, or it produces an empty or invalid string, then a default description gets used instead. No worrying about blanks, bestie.
on_exit_hold_subcode = <ClassAd Integer Expression> — when the job goes on hold because the on_exit_hold expression became True, this expression gets evaluated to set the value of HoldReasonSubCode in the job ClassAd. The default subcode is 0. The HoldReasonCode gets set to 3, which means the job went on hold due to a job policy expression 💅
on_exit_remove = <ClassAd Boolean Expression> — okay bestie, this ClassAd expression gets checked the literal second your job exits. If it comes back True (which is the default value when it's undefined, fyi), the job gets to leave the queue like normal, no drama. If it's False, the job gets yeeted right back into the Idle state. And heads up: if your job runs under the vanilla universe, it restarts from the very beginning — no save point, no resume, just vibes from zero. 🫠

So picture this: your job occasionally segfaults 💀 but honestly the odds are it'll finish successfully if you just run it again with the same data. The on_exit_remove expression can make it re-run all by itself, like so. Assume the signal identifier for the segmentation fault is 11 on the platform where the job will be running:

on_exit_remove = (ExitBySignal == False) || (ExitSignal != 11)

This expression lets the job leave the queue if it wasn't killed by a signal, OR if it was killed by any signal other than 11 (which is the segmentation fault in this example). So if it exited due to signal 11? It stays in the job queue. Any other kind of exit? It leaves the queue like it normally would've. Periodt. 💅

And another one — if you want the job to only leave the queue when it exited on its own with status 0, this expression is literally the one:

on_exit_remove = (ExitBySignal == False) && (ExitCode == 0)

If the job got killed by a signal or exited with a non-zero exit status, HTCondor would leave it in the queue to run again. 🔁

Precedence lore, get into it: periodic_* expressions take precedence over on_exit_* expressions, and *_hold expressions take precedence over a *_remove expressions.

Also, only job ClassAd attributes will be defined for use by this ClassAd expression.

periodic_hold = <ClassAd Boolean Expression> — this one gets checked periodically whenever the job is not in the Held state. The moment it flips to True, the job gets placed on hold. If you leave it unspecified, the default value is False.

Same precedence energy as before: periodic_* expressions take precedence over on_exit_* expressions, and *_hold expressions take precedence over a *_remove expressions.

Only job ClassAd attributes will be defined for use by this ClassAd expression. And ngl, by default this expression only gets checked once every 60 seconds. If you wanna mess with that, the period of these evaluations can be adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE configuration macros. ⏰

periodic_hold_reason = <ClassAd String Expression> — when the job goes on hold because the periodic_hold expression became True, this expression gets evaluated to set the value of HoldReason in the job ClassAd. If this expression is UNDEFINED or produces an empty or invalid string, a default description is used instead. 💬
periodic_hold_subcode = <ClassAd Integer Expression> — when the job goes on hold because periodic_hold became true, this expression gets evaluated to set the value of HoldReasonSubCode in the job ClassAd. The default subcode is 0. Meanwhile HoldReasonCode will be set to 3, which indicates that the job went on hold due to a job policy expression. 🔢
periodic_release = <ClassAd Boolean Expression> — checked periodically while the job is sitting in the Held state. If the expression becomes True, the job gets released. But plot twist: if the job was held via condor_hold (i.e. HoldReasonCode is 1), then this expression is ignored. 🙈

Only job ClassAd attributes will be defined for use by this ClassAd expression. And again — by default this expression is only checked once every 60 seconds. The period of these evaluations can be adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE configuration macros.

periodic_remove = <ClassAd Boolean Expression> — checked periodically, and if it becomes True, the job gets removed from the queue. If unspecified, the default value is False.

Peep the Examples section of this manual page for an example of a periodic_remove expression. 👀

Precedence, one more time: periodic_* expressions take precedence over on_exit_* expressions, and *_hold expressions take precedence over a *_remove expressions. So the periodic_remove expression takes precedent over the on_exit_remove expression if the two describe conflicting actions. It's giving main character energy, honestly.

Only job ClassAd attributes will be defined for use by this ClassAd expression. And yup — by default this expression is only checked once every 60 seconds. The period of these evaluations can be adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE configuration macros.

periodic_vacate = <ClassAd Boolean Expression> — checked periodically for running jobs. If it becomes True, the job gets evicted from the machine it's running on and returns to the queue in an Idle state. If unspecified, the default value is False. 🏃‍♀️✨

Only job ClassAd attributes will be defined for use by this ClassAd expression. And no cap — by default this expression is only checked once every 60 seconds. The period of these evaluations can be adjusted by setting the PERIODIC_EXPR_INTERVAL, MAX_PERIODIC_EXPR_INTERVAL, and PERIODIC_EXPR_TIMESLICE configuration macros.

🚀 COMMANDS FOR THE GRID 🌈

arc_application = <XML-string> — for grid universe jobs of type arc, this provides extra XML attributes under the <Application> section of the ARC ADL job description that aren't covered by the regular submit description file parameters. 🛰️
arc_data_staging = <XML-string> — for grid universe jobs of type arc, provides extra XML attributes under the <DataStaging> section of the ARC ADL job description which are not covered by regular submit description file parameters.
arc_resources = <XML-string> — for grid universe jobs of type arc, provides extra XML attributes under the <Resources> section of the ARC ADL job description which are not covered by regular submit description file parameters.
arc_rte = < rte1 option,rte2 > — for grid universe jobs of type arc, this hands over a list of Runtime Environment names that the job needs on the ARC system. The list is comma-delimited. If a Runtime Environment name supports options, those can be provided after the name, separated by spaces. Runtime Environment names are defined by the ARC server. 🧩
azure_admin_key = <pathname> — for grid type azure jobs, specifies the path and file name of a file that contains an SSH public key. This key can be used to log into the administrator account of the instance via SSH. 🔑
azure_admin_username = <account name> — for grid type azure jobs, specifies the name of an administrator account to be created in the instance. This account can be logged into via SSH.
azure_auth_file = <pathname> — for grid type azure jobs, specifies a path and file name of the authorization file that grants permission for HTCondor to use the Azure account. If it's not defined, then HTCondor will attempt to use the default credentials of the Azure CLI tools. 🪪
azure_image = <image id> — for grid type azure jobs, identifies the disk image to be used for the boot disk of the instance. This image must already be registered within Azure.
azure_location = <image id> — for grid type azure jobs, identifies the location within Azure where the instance should be run. As an example, one current location is centralus. 📍
azure_size = <machine type> — for grid type azure jobs, the hardware configuration that the virtual machine instance is to run on. 💪
batch_extra_submit_args = <command-line arguments> — used for batch grid universe jobs. Specifies additional command-line arguments to be given to the target batch system's job submission command. 📝
batch_project = <projectname> — used for batch grid universe jobs. Specifies the name of the PBS/LSF/SGE/SLURM project, account, or allocation that should be charged for the resources used by the job. 💸
batch_queue = <queuename> — used for batch grid universe jobs. Specifies the name of the PBS/LSF/SGE/SLURM job queue into which the job should be submitted. If not specified, the default queue is used. For a multi-cluster SLURM configuration, which cluster to use can be specified by supplying the name after an @ symbol. For example, to submit a job to the debug queue on cluster foo, you would use the value debug@foo. 🎯
batch_runtime = <seconds> — used for batch grid universe jobs. Specifies a limit in seconds on the execution time of the job. This limit is enforced by the PBS/LSF/SGE/SLURM scheduler. ⏳
cloud_label_names = <name0,name1,name...> — for grid type gce jobs, specifies the case of tag names that will be associated with the running instance. This is only necessary if a tag name case matters. By default the list will be automatically generated. 🏷️
cloud_label_<name> = <value> — for grid type gce jobs, specifies a label and value to be associated with the running instance. The label name will be lower-cased; use cloud_label_names to change the case. ✨
delegate_job_GSI_credentials_lifetime = <seconds> — sets the max number of seconds your delegated proxies are allowed to stay valid 💖. If you don't say anything, the vibe check comes from the config variable DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME, which defaults to one day. Setting it to 0 means the delegated proxy lives as long as the credential that made it will allow — no cap, living its best life 🌈. Heads up tho: this only applies to proxies delegated for non-grid jobs and for HTCondor-C jobs. And it has zero effect if DELEGATE_JOB_GSI_CREDENTIALS is False, because in that case the job proxy is copied, not delegated. Different mechanic entirely, bestie 🫶
ec2_access_key_id = <pathname> — for grid type ec2 jobs, this points at the file holding the access key 🔑.
ec2_ami_id = <EC2 xMI ID> — for grid type ec2 jobs, this names the machine image. Services that play nice with the EC2 Query API might call these by other abbreviations instead of AMI — for example EMI is totally valid for Eucalyptus 🌿.
ec2_availability_zone = <zone name> — for grid type ec2 jobs, tells it which Availability Zone the instance should run in. Optional, unless ec2_ebs_volumes is set 👀. One example of a current zone is us-east-1b.
ec2_block_device_mapping = <block-device>:<kernel-device>,<block-device>:<kernel-device>, ... — for grid type ec2 jobs, this spells out the block device → kernel device mapping. Optional, no pressure ✨.
ec2_ebs_volumes = <ebs name>:<device name>,<ebs name>:<device name>,... — for grid type ec2 jobs, optionally lists the Elastic Block Store (EBS) volumes to hand to the instance, plus the device names they should have once they're inside 💾.
ec2_elastic_ip = <elastic IP address> — for grid type ec2 jobs, optional specification of an Elastic IP address that should be assigned to this instance 📍.
ec2_iam_profile_arn = <IAM profile ARN> — for grid type ec2 jobs, an Amazon Resource Name (ARN) identifying which Identity and Access Management (IAM) (instance) profile should be associated with the instance 🪪.
ec2_iam_profile_name = <IAM profile name> — for grid type ec2 jobs, a name identifying which Identity and Access Management (IAM) (instance) profile should be associated with the instance 🪪.
ec2_instance_type = <instance type> — for grid type ec2 jobs, identifies the instance type. Different services may offer different instance types, so there's no default value set — you pick, queen 👑.
ec2_keypair = <ssh key-pair name> — for grid type ec2 jobs, the name of an SSH key-pair that's already registered with the EC2 service. The associated private key can be used to ssh into the virtual machine once it's running 🔐.
ec2_keypair_file = <pathname> — for grid type ec2 jobs, the complete path and file name of a file into which HTCondor will write an SSH key for use with ec2 jobs. That key can be used to ssh into the virtual machine once it's running 🔐. Oh and — if ec2_keypair is specified for a job, then ec2_keypair_file is ignored 🙈.
ec2_parameter_names = ParameterName1, ParameterName2, ... — for grid type ec2 jobs, a space- or comma-separated list of the names of additional parameters to pass when instantiating an instance 📝.
ec2_parameter_<name> = <value> — for grid type ec2 jobs, the value for the correspondingly named (instance instantiation) parameter. <name> is the parameter name you gave in the submit command ec2_parameter_names, except with any periods swapped out for underscores 💅.
ec2_secret_access_key = <pathname> — for grid type ec2 jobs, the path and file name containing the secret access key 🤫.
ec2_security_groups = group1, group2, ... — for grid type ec2 jobs, the list of EC2 security groups to associate with the job 🛡️.
ec2_security_ids = id1, id2, ... — for grid type ec2 jobs, the list of EC2 security group IDs to associate with the job 🛡️.
ec2_spot_price = <bid> — for grid type ec2 jobs, the spot instance bid — aka the absolute most the job submitter is willing to pay per hour to run this job 💸.
ec2_tag_names = <name0,name1,name...> — for grid type ec2 jobs, specifies the case of the tag names that get associated with the running instance. Only needed if the case of a tag name actually matters. By default the list is generated automatically ✨.
ec2_tag_<name> = <value> — for grid type ec2 jobs, a tag to associate with the running instance 🏷️. The tag name gets lower-cased — so use ec2_tag_names if you wanna change the case.
WantNameTag = <True | False> — for grid type ec2 jobs, a job may request that its 'name' tag be (or not be) set by HTCondor. If the job doesn't otherwise specify any tags, then not setting its name tag eliminates a call by the EC2 GAHP, which improves performance ⚡.
ec2_user_data = <data> — for grid type ec2 jobs, a block of data the virtual machine can access 📦. If both ec2_user_data and ec2_user_data_file are specified for a job, the two blocks of data get concatenated, with the data from this ec2_user_data submit command coming first 🥇.
ec2_user_data_file = <pathname> — for grid type ec2 jobs, a path and file name whose contents the virtual machine can access 📦. If both ec2_user_data and ec2_user_data_file are specified for a job, the two blocks of data get concatenated, with the data from that ec2_user_data submit command coming first 🥇.
ec2_vpc_ip = <a.b.c.d> — for grid type ec2 jobs that are part of a Virtual Private Cloud (VPC), an optional specification of the IP address this instance should have within the VPC 🌐.
ec2_vpc_subnet = <subnet specification string> — for grid type ec2 jobs, an optional specification of the Virtual Private Cloud (VPC) that this instance should be a part of 🌐.
gce_account = <account name> — for grid type gce jobs, the Google cloud services account to use. If you don't specify this submit command, then a random account from the authorization file given by gce_auth_file gets used 🎲.
gce_auth_file = <pathname> — for grid type gce jobs, a path and file name of the authorization file that grants permission for HTCondor to use the Google account 🎫. If this command isn't specified, then the default file of the Google command-line tools is used.
gce_image = <image id> — for grid type gce jobs, the identifier of the virtual machine image representing the HTCondor job to be run. This virtual machine image must already be registered with GCE and reside in Google's Cloud Storage service ☁️.
gce_json_file = <pathname> — for grid type gce jobs, a path and file name of a file containing JSON elements that should be added to the instance description submitted to the GCE service 📄.
gce_machine_type = <machine type> — for grid type gce jobs, the long form of the URL describing the machine configuration that the virtual machine instance is to run on 🖥️.
gce_metadata = <name=value,...,name=value> — for grid type gce jobs, a comma-separated list of name and value pairs defining metadata for a virtual machine instance that is an HTCondor job 🏷️.
gce_metadata_file = <pathname> — for grid type gce jobs, a path and file name of the file containing metadata for a virtual machine instance that is an HTCondor job 🏷️. Inside that file, each name and value pair sits on its own line — so the pairs are separated by the newline character, periodt.
gce_preemptible = <True | False> — for grid type gce jobs, whether the virtual machine instance should be preemptible. The default is for the instance to not be preemptible ✋.
grid_resource = <grid-type-string> <grid-specific-parameter-list> — okay so for every grid-type-string value there are further type-specific values that must be specified, and this submit description file command lets you give each of them in a space-separated list 🌈. The allowable grid-type-string values are arc, azure, batch, condor, ec2, and gce. The HTCondor manual chapter on Grid Computing details the whole variety of grid types, iykyk 📚.
- For a grid-type-string of batch, the single parameter is the name of the local batch system, and it'll be one of pbs, lsf, slurm, or sge.
- For a grid-type-string of condor, the first parameter is the name of the remote condor_schedd daemon. The second parameter is the name of the pool that the remote condor_schedd daemon belongs to.
- For a grid-type-string of ec2, one additional parameter specifies the EC2 URL.
- For a grid-type-string of arc, the single parameter is the name of the ARC resource to be used.
transfer_error = <True | False> — for jobs submitted to the grid universe only 🌌. If True, then the error output (from stderr) from the job is transferred from the remote machine back to the access point. The name of the file after transfer is given by the error command. If False, no transfer takes place (from the remote machine to the access point), and the name of the file is given by the error command. The default value is True 💖.
transfer_input = <True | False> — bestie, this one is for jobs submitted to the grid universe only. If True, then the job input (stdin) gets transferred from the machine where the job was submitted to the remote machine. The name of the file that is transferred is given by the input command. If False, then the job's input is taken from a pre-staged file on the remote machine, and the name of the file is given by the input command. The default value is True.

For transferring files other than stdin, see transfer_input_files.

transfer_output = <True | False> — grid universe only again, no cap. If True, then the output (from stdout) from the job is transferred from the remote machine back to the access point. The name of the file after transfer is given by the output command. If False, no transfer takes place (from the remote machine to access point), and the name of the file is given by the output command. The default value is True.

For transferring files other than stdout, see transfer_output_files,

use_x509userproxy = <True | False> — set this command to True to indicate that the job requires an X.509 user proxy. If x509userproxy is set, then that file is used for the proxy. Otherwise, the proxy is looked for in the standard locations. If x509userproxy is set or if the job is a grid universe job of grid type arc, then the value of use_x509userproxy is forced to True. Defaults to False.
x509userproxy = <full-pathname> — used to override the default path name for X.509 user certificates. The default location for X.509 proxies is the /tmp directory, which is generally a local file system. Setting this value would allow HTCondor to access the proxy in a shared file system (for example, AFS). HTCondor will use the proxy specified in the submit description file first. If nothing is specified in the submit description file, it will use the environment variable X509_USER_PROXY. If that variable is not present, it will search in the default location. Note that proxies are only valid for a limited time. Condor_submit will not submit a job with an expired proxy, it will return an error. Also, if the configuration parameter CRED_MIN_TIME_LEFT is set to some number of seconds, and if the proxy will expire before that many seconds, condor_submit will also refuse to submit the job. That is, if CRED_MIN_TIME_LEFT is set to 60, condor_submit will refuse to submit a job whose proxy will expire 60 seconds from the time of submission.

x509userproxy is relevant when the universe is vanilla, or when the universe is grid and the type of grid system is one of condor, or arc. Defining a value causes the proxy to be delegated to the execute machine. Further, VOMS attributes defined in the proxy will appear in the job ClassAd.

use_scitokens = <True | False | Auto> — set this command to True to indicate that the job requires a scitoken. If scitokens_file is set, then that file is used for the scitoken filename. Otherwise, the scitoken filename is looked for in the BEARER_TOKEN_FILE environment variable. If scitokens_file is set then the value of use_scitokens defaults to True. If the filename is not defined in on one of these two places, then condor_submit will fail with an error message. Set this command to Auto to indicate that the job will use a scitoken if scitokens_file or the BEARER_TOKEN_FILE environment variable is set, but it will not be an error if no file is specified.

This command is only useful for grid universe jobs. The scitoken will be used by the condor_gridmanager to authenticate to the remote CE; It has no effect on how any submit method authenticates to the condor_schedd to submit the initial grid universe job.

scitokens_file = <full-pathname> — used to set the path to the file containing the scitoken that the job needs, or to override the path to the scitoken contained in the BEARER_TOKEN_FILE environment variable.

scitokens_file is relevant when the universe grid and the type of grid system is one of condor, or arc. Defining a value causes authentication to the remote system to be made using the given scitoken. Unlike x509userproxy, no attributes from the scitoken other than the filename will be copied into the job. Note that neither this nor use_scitokens will have any effect on how any job submission method authenticates to the condor_schedd to place the grid universe job initially.

🌈 COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES 🌈

hold_kill_sig = <signal-number> — for the scheduler universe only, signal-number is the signal delivered to the job when the job is put on hold with condor_hold. signal-number may be either the platform-specific name or value of the signal. If this command is not present, the value of kill_sig is used.
jar_files = <file_list> — specifies a list of additional JAR files to include when using the Java universe. JAR files will be transferred along with the executable and automatically added to the classpath.
java_vm_args = <argument_list> — specifies a list of additional arguments to the Java VM itself. When HTCondor runs the Java program, these are the arguments that go before the class name. This can be used to set VM-specific arguments like stack size, garbage-collector arguments and initial property values.
machine_count = <max> — for the parallel universe, a single value (max) is required. It is neither a maximum or minimum, but the number of machines to be dedicated toward running the job.
remove_kill_sig = <signal-number> — for the scheduler universe only, signal-number is the signal delivered to the job when the job is removed with condor_rm. signal-number may be either the platform-specific name or value of the signal. This example shows it both ways for a Linux signal:

remove_kill_sig = SIGUSR1
remove_kill_sig = 10

If this command is not present, the value of kill_sig is used.

🦋 COMMANDS FOR THE VM UNIVERSE 🦋

vm_disk = file1:device1:permission1, file2:device2:permission2:format2, ... — a list of comma separated disk files. Each disk file is specified by 4 colon separated fields. The first field is the path and file name of the disk file. The second field specifies the device. The third field specifies permissions, and the optional fourth field specifies the image format. If a disk file will be transferred by HTCondor, then the first field should just be the simple file name (no path information).

An example that specifies two disk files:

vm_disk = /myxen/diskfile.img:sda1:w,/myxen/swap.img:sda2:w

vm_checkpoint = <True | False> — a boolean value specifying whether or not to take checkpoints. If not specified, the default value is False. In the current implementation, setting both vm_checkpoint and vm_networking to True does not yet work in all cases. Networking cannot be used if a vm universe job uses a checkpoint in order to continue execution after migration to another machine.
vm_macaddr = <MACAddr> — defines that MAC address that the virtual machine's network interface should have, in the standard format of six groups of two hexadecimal digits separated by colons.
vm_memory = <MBytes-of-memory> — the amount of memory in MBytes that a vm universe job requires.
vm_networking = <True | False> — specifies whether to use networking or not. In the current implementation, setting both vm_checkpoint and vm_networking to True does not yet work in all cases. Networking cannot be used if a vm universe job uses a checkpoint in order to continue execution after migration to another machine.
vm_networking_type = <nat | bridge | user > — when vm_networking is True, this definition augments the job's requirements to match only machines with the specified networking. If not specified, then any available networking type matches.
vm_no_output_vm = <True | False> — when True, prevents HTCondor from transferring output files back to the machine from which the vm universe job was submitted. If not specified, the default value is False.
vm_type = <xen | kvm> — specifies the underlying virtual machine software that this job expects.
xen_initrd = <image-file> — when xen_kernel gives a file name for the kernel image to use, this optional command may specify a path to a ramdisk (initrd) image file. If the image file will be transferred by HTCondor, then the value should just be the simple file name (no path information).
xen_kernel = <included | path-to-kernel> — a value of included specifies that the kernel is included in the disk file. If not one of these values, then the value is a path and file name of the kernel to be used. If a kernel file will be transferred by HTCondor, then the value should just be the simple file name (no path information).
xen_kernel_params = <string> — a string that is appended to the Xen kernel command line.
xen_root = <string> — a string that is appended to the Xen kernel command line to specify the root device. This string is required when xen_kernel gives a path to a kernel. Omission for this required case results in an error message during submission.

🐳 COMMANDS FOR THE DOCKER UNIVERSE 🐳

docker_image = < image-name > — defines the name of the Docker image that is the basis for the docker container.
docker_network_type = < host | none | custom_admin_defined_value> — okay so if you set this to the string host, your job runs on the host's network, fr. Set it to none and your job gets no network at all (unplugged, iconic, isolated 💅). Leave it unset and every job gets its own private network interface. And plot twist: some admins define site-specific docker networks on a given execution point, so if that's your situation, extra values might be valid here too. iykyk.
docker_pull_policy = < always > — set this to always and boom: whenever a docker universe job starts on an execution point, the option --pull always gets passed to the docker run command. Heads up tho, this only hits EPs that already have a locally cached version of the image. With this on, docker will always check with the repo to see if the cached version is out of date. That's more network connectivity required, and it may cause docker hub to throttle your future pull requests 😭. Genuinely recommended: never mutate a docker image tag name, and just avoid needing this option at all. periodt.
container_service_names = <service-name>[, <service-name>]* — a string- or comma-separated list of service names, bestie. Each service-name must have a matching <service-name>_container_port command specifying a port number (an integer from 0 to 65535). HTCondor will ask Docker to forward from a host port to the specified port inside the container. Once Docker does that, HTCondor adds an attribute to the job ad for each service, <service-name>HostPort, which contains the port number on the host forwarding to the corresponding service. so slay.
<service-name>_container_port = port_number — see above 👆
<service-name>_HostPort = port_number — see above 👆
docker_override_entrypoint = <True | False> — if you set this to True and executable isn't empty, the image entrypoint gets replaced with the executable. The default value (False) follows the exact same logic the docker engine uses with images (check docker run for the receipts):
- Without entrypoint, executable runs as main PID
- With entrypoint, it is launched with the executable as first argument

Any additional arguments will follow the executable. cute!

🧚 COMMANDS FOR THE CONTAINER UNIVERSE 🧚

container_image = < image-name > — defines the name of the container image, bestie. It can be a singularity .sif file, a singularity exploded directory, or a path to an image in a docker style repository. Now listen: images that are neither on shared filesystems nor transferred by the container service (think docker:// URLs) are considered common files by default. If you need to turn that off for some reason, go peek at container_is_common. 💖
transfer_container = < True | False > — a boolean value that defaults to True. Set it to false and .sif container images and expanded directories are assumed to be pre-staged on the EP, and HTCondor will not even attempt to transfer them. no cap.
container_target_dir = < path-to-directory-inside-container > — defines the working directory of the job inside the container. It will be mapped to the scratch directory on the execution point. iconic.
mount_under_scratch = < path-to-directory-inside-container > — binds a brand new, empty writeable directory inside the container image that the job will have permissions to write to. ✨ fresh space, just for you ✨
container_is_common = < True | False > — defaults to True. If you set it to False, HTCondor will not treat the container image as a common file, which means it's gotta be transferred to each job individually. one-by-one, like a lil lineup 🫶

👑 ADVANCED COMMANDS 👑

accounting_group = <accounting-group-name> — causes jobs to negotiate under the given accounting group. This value gets advertised in the job ClassAd as AcctGroup. The HTCondor Administrator's manual has the full lore on accounting groups if you're curious.
accounting_group_user = <accounting-group-user-name> — sets the name associated with this job, used for resource usage accounting purposes — like computation of fair-share priority and reporting via condor_userprio. If you don't set it, it defaults to the value of the job ClassAd attribute User. This value is advertised in the job ClassAd as AcctGroupUser. 💅
concurrency_limits = <string-list> — a list of resources that this job needs. The resources are presumed to have concurrency limits placed upon them, which limits the number of concurrent jobs in execution that need the named resource. Commas and space characters delimit the items in the list. Each item in the list is a string that identifies the limit, or it's a ClassAd expression that evaluates to a string — and that gets evaluated in the context of the machine ClassAd being considered as a match. Each item may also specify a numerical value identifying the integer number of resources required for the job. The syntax follows the resource name by a colon character (:) and the numerical value. Details on concurrency limits live in the HTCondor Administrator's manual.
concurrency_limits_expr = <ClassAd String Expression> — a ClassAd expression that represents the list of resources this job needs after evaluation. The ClassAd expression may specify machine ClassAd attributes that are evaluated against a matched machine. After evaluation, the list sets concurrency_limits. ✨ chain reaction ✨
coresize = <size> — should the user's program abort and produce a core file, coresize specifies the maximum size in bytes of the core file which the user wishes to keep. If coresize isn't specified in the command file, this is set to 0 (meaning no core will be generated). zero core, zero drama 💀
cron_day_of_month = <Cron-evaluated Day> — the set of days of the month for which a deferral time applies. The HTCondor User's manual section on Time Scheduling for Job Execution has further details.
cron_day_of_week = <Cron-evaluated Day> — the set of days of the week for which a deferral time applies. The HTCondor User's manual section on Time Scheduling for Job Execution has further details.
cron_hour = <Cron-evaluated Hour> — the set of hours of the day for which a deferral time applies. The HTCondor User's manual section on Time Scheduling for Job Execution has further details.
cron_minute = <Cron-evaluated Minute> — the set of minutes within an hour for which a deferral time applies. The HTCondor User's manual section on Time Scheduling for Job Execution has further details.
cron_month = <Cron-evaluated Month> — the set of months within a year for which a deferral time applies. The HTCondor User's manual section on Time Scheduling for Job Execution has further details.
cron_prep_time = <ClassAd Integer Expression> — analogous to deferral_prep_time. The number of seconds prior to a job's deferral time that the job may be matched and sent to an execution machine. 🕐
cron_window = <ClassAd Integer Expression> — analogous to the submit command deferral_window. It allows cron jobs that miss their deferral time to begin execution. The HTCondor User's manual section on Time Scheduling for Job Execution has further details.
dagman_log = <pathname> — DAGMan inserts this command to specify an event log that it watches to maintain the state of the DAG. If the log command is not specified in the submit file, DAGMan uses the log command to specify the event log. 👀
deferral_prep_time = <ClassAd Integer Expression> — the number of seconds prior to a job's deferral time that the job may be matched and sent to an execution machine. The HTCondor User's manual section on Time Scheduling for Job Execution has further details.
deferral_time = <ClassAd Integer Expression> — lets a job specify the time at which its execution is to begin, instead of beginning execution as soon as it arrives at the execution machine. The deferral time is an expression that evaluates to a Unix Epoch timestamp (the number of seconds elapsed since 00:00:00 on January 1, 1970, Coordinated Universal Time). Deferral time is evaluated with respect to the execution machine. This option delays the start of execution, but not the matching and claiming of a machine for the job. If the job is not available and ready to begin execution at the deferral time, it has missed its deferral time. A job that misses its deferral time will be put on hold in the queue. 🚨

The HTCondor User's manual section on Time Scheduling for Job Execution has further details.

Due to implementation details, a deferral time may not be used for scheduler universe jobs. sorry bestie, that one's off the table 😭

deferral_window = <ClassAd Integer Expression> — the deferral window is used in conjunction with the deferral_time command to allow jobs that miss their deferral time to begin execution. The HTCondor User's manual section on Time Scheduling for Job Execution has further details.
description = <string> — a string that sets the value of the job ClassAd attribute JobDescription. When it's set, tools which display the executable — like condor_q — will instead use this string. glow up for your queue view ✨
email_attributes = <list-of-job-ad-attributes> — a comma-separated list of attributes from the job ClassAd. These attributes and their values will be included in the e-mail notification of job completion. 📧
image_size = <size> — a lil heads-up for HTCondor about the maximum virtual image size your job is gonna balloon into while it runs. Tell it this and HTCondor will only place your job on machines with enough resources (like virtual memory) to actually hold it. Skip it, and HTCondor makes its own (reasonably accurate, ngl) estimate of your job's size and keeps adjusting it as the program runs. Heads up tho: if you specify it and underestimate, your job may crash because it can't grab more address space — like when malloc() fails. 😭 And if you overestimate, HTCondor may have a whole struggle finding machines with the resources you claimed you need. size is in KiB btw. So for an image size of 8 MiB, size should be 8000. ✨
initialdir = <directory-path> — gives your job a directory to do its file input and output relative to. It also hands over a directory (on the machine you submitted from) for the job event log, whenever you don't specify a full path.

Here's the vibe by universe, bestie:

For vanilla universe jobs on a shared file system, it's the current working directory on the machine where the job is executed.
For vanilla or grid universe jobs where file transfer mechanisms are in play (so, no shared file system), it's the directory on the machine you submitted from where the input files come from, and where your job's output files go to.
For scheduler universe jobs, it's the directory on the machine you submitted from where the job runs — the current working directory for file input and output with respect to relative path names.

🫶 Important: the path to the executable is not relative to initialdir if it's a relative path. It's relative to the directory in which the condor_submit command is run. Periodt.

job_ad_information_attrs = <attribute-list> — a comma-separated list of job ClassAd attribute names. Those named attributes and their values get written to the job event log every single time any event is written to the log. It does the exact same thing as the configuration variable EVENT_LOG_INFORMATION_ATTRS, except it applies to the job event log instead of the system event log. 💅
job_lease_duration = <number-of-seconds> — for vanilla, parallel, VM, and java universe jobs only, bb. This is the duration in seconds of a job lease. Default is 2,400, aka forty minutes. Don't want a job lease? Set it to 0 explicitly to disable the job lease semantics entirely. The value can also be a ClassAd expression that evaluates to an integer. More deets live in the HTCondor User's manual section on Special Environment Considerations. 🌸
job_machine_attrs = <attr1, attr2, ...> — a comma and/or space separated list of machine attribute names that should get recorded in the job ClassAd, on top of whatever the condor_schedd daemon's system configuration variable SYSTEM_JOB_MACHINE_ATTRS already specifies. When there are multiple run attempts, the history of machine attributes from previous run attempts may be kept. And you can stretch how many run attempts get stored beyond the system-specified history length by using the submit file command
job_machine_attrs_history_length — okay so a machine attribute named X gets inserted into the job ClassAd as an attribute named MachineAttrX0. The previous value of that attribute is named MachineAttrX1, the one before that is MachineAttrX2, and so on, all the way up to the history length you specified. A history of length 1 means only MachineAttrX0 gets recorded. The value recorded in the job ClassAd is the evaluation of the machine attribute in the context of the job ClassAd at the moment the condor_schedd daemon kicks off the job's start up. If that evaluation comes back Undefined or Error, then what's recorded in the job ad is Undefined or Error, respectively. Simple as. ✨
want_graceful_removal = <boolean expression> — if true, your job gets a chance to shut down cleanly when it's removed. 🥺 It'll be given as much time as the admin of the execute resource allows — which, lowkey, may be none. Default is false. For the full tea, see the configuration setting GRACEFULLY_REMOVE_JOBS.
kill_sig = <signal-number> — when HTCondor needs to kick a job off a machine, it sends that job the signal named by signal-number, which has to be an integer representing a valid signal on the execution machine. Default value is SIGTERM, the standard way to terminate a program in Unix. 🔥
kill_sig_timeout = <seconds> — this submit command should no longer be used. job_max_vacate_time instead. 🙈 If job_max_vacate_time is not defined, this defines the number of seconds HTCondor should wait after sending the kill signal defined by kill_sig before forcibly killing the job. The actual amount of time between sending the signal and forcibly killing the job is the smallest of this value and the configuration variable KILLING_TIMEOUT, as defined on the execute machine.
load_profile = <True | False> — when True, it loads the account profile of the dedicated run account for Windows jobs. May not be used with run_as_owner. 💀
log_xml = <True | False> — if log_xml is True, the job event log file gets written in ClassAd XML. Not specified? Then XML is not used. Careful tho: the file is an XML fragment — it's missing the file header and footer. Do not mix XML and non-XML within a single file. And if multiple jobs write to a single job event log file, make sure all of them specify this option the same way. 👑
match_list_length = <integer value> — defaults to the value zero (0). When match_list_length is defined with an integer value greater than zero (0), attributes get inserted into the job ClassAd. The max number of attributes defined is given by that integer value. The job ClassAds introduced are given as:

LastMatchName0 = "most-recent-Name"
LastMatchName1 = "next-most-recent-Name"

Each introduced ClassAd's value comes from the value of the Name attribute in the machine ClassAd of a previous execution (a match). As a job gets matched, those definitions roll: LastMatchName1 becomes LastMatchName2, LastMatchName0 becomes LastMatchName1, and LastMatchName0 gets set from the most recent value of the Name attribute. 🌈

An intended use of these job attributes is inside the requirements expression — so your requirements can let a job prefer a match with either the same or a different resource than a previous match. 🦋

job_max_vacate_time = <integer expression> — an integer-valued expression (in seconds) you can use to adjust how much time an evicted job gets for gracefully shutting down. If the job's setting is less than the machine's, the job's wins. If the job's setting is larger than the machine's, the result depends on whether the job has any excess retirement time. If the job has more retirement time left than the machine's max vacate time setting, then retirement time gets converted into vacating time, up to the amount the job requested.

Setting this expression does not affect the job's resource requirements or preferences — no cap. So if you want a job to only run on a machine with a minimum MachineMaxVacateTime, or to preferentially run on such machines, you have to say that explicitly in the requirements and/or rank expressions. 💅

manifest = <True | False> — for vanilla and Docker-universe jobs (and others that use the shadow), this specifies whether HTCondor (the starter) should produce a "manifest" — that's a directory containing three files: the list of files and directories in the sandbox when file transfer in completes (in), the same when file transfer out begins (out), and a dump of the environment set for the job (-Environment). 🎀

This feature is not presently available for Windows. 😭

manifest_dir = <directory name> — again for vanilla and Docker-universe jobs (and others that use the shadow), this specifies the directory in which to record the manifest. Specifying this enables the creation of a manifest. By default, the manifest directory is named <cluster>_<proc>_manifest, to avoid conflicts. 🌸

This feature is not presently available for Windows. 😭

max_job_retirement_time = <integer expression> — an integer-valued expression (in seconds) that does absolutely nothing unless the machine running the job has been configured to provide retirement time. Retirement time is a grace period handed to a job so it can finish when a resource claim is about to be preempted. The default behavior in many cases is to take as much retirement time as the machine offers, so this command will rarely appear in a submit description file. 🫶

Like, if a resource claim is about to get preempted 👀, this expression in your submit file says the maximum run time of the job (in seconds, counted since the job started, fr). It does absolutely nothing if it's bigger than the maximum retirement time the machine policy hands out. And if the resource claim isn't preempted at all? Then this expression and the machine retirement policy are just... irrelevant, they're not even in the group chat. But if the claim does get preempted, the job gets to run until the retirement time expires, and at that exact point it's hard-killed. 😭 The job also gets soft-killed once it's getting close to the end of retirement, so it has a hot second to shut down gracefully like the icon it is. How much lead-time you get for that soft-kill is decided by the maximum vacating time granted to the job.

Any jobs running with nice_user priority have a default max_job_retirement_time of 0, so lowkey no retirement time is utilized by default. In every other case, no default value is provided, so the maximum amount of retirement time is utilized by default. 🌸

Setting this expression does not affect the job's resource requirements or preferences, no cap. If you want a job to only run on a machine with a minimum MaxJobRetirementTime, or to preferentially run on such machines, you have to explicitly say so in the requirements and/or rank expressions. Periodt.

nice_user = <True | False> — Normally when a machine becomes available to HTCondor, HTCondor picks which job runs based on user and job priorities. Setting nice_user to True tells HTCondor like... don't use my regular user priority bestie, this job should have last priority among all users and all jobs. So jobs submitted this way only run on machines that no other non-nice_user job wants — a true bottom-feeder job, a little scavenger, we love her for it. 🙈 Super handy if you've got jobs you want to run but you don't want to hog resources that could instead run other people's HTCondor jobs. Jobs submitted this way have an accounting group, and that accounting group is configurable by setting NICE_USER_ACCOUNTING_GROUP_NAME, which defaults to nice-user. The default value is False.
noop_job = <ClassAd Boolean Expression> — When this boolean expression is True, the job is immediately removed from the queue and HTCondor makes zero attempts at running it. The job's log file will show a job submitted event and a job terminated event, along with an exit code of 0 — unless you specify a different signal or exit code, of course. 💅
noop_job_exit_code = <return value> — When noop_job is in the submit description file and evaluates to True, this command lets the job specify the return value shown in the job's log file job terminated event. If you don't specify it, the job shows up as having terminated with status 0. This overrides any value specified with noop_job_exit_signal.
noop_job_exit_signal = <signal number> — When noop_job is in the submit description file and evaluates to True, this command lets the job specify the signal number that the job's log event will show the job having terminated with.
primary_unix_group = group_name — On Linux systems, if the job runs with a supplemental group of group_name, install this group as the primary unix group. It's equivalent to the unix command newgrp. Lowkey useful when you're working with shared filesystems and the job needs to control which of its groups any files created in a shared filesystem should have. 🐧
remote_initialdir = <directory-path> — The path specifies the directory in which the job is to be executed on the remote machine.
rendezvousdir = <directory-path> — Used to specify the shared file system directory to be used for file system authentication when submitting to a remote scheduler. Should be a path to a preexisting directory, so like, don't make it up as you go babes.
want_job_networking = <True | False> — If false, then the job may be matched to a machine where the job has no access to the networking. This may allow the job to run in more places than it otherwise would. Default is true.
run_as_owner = <True | False> — A boolean value that causes the job to be run under the login of the submitter, if supported by the joint configuration of the submit and execute machines. On Unix platforms this defaults to True, and on Windows platforms it defaults to False. May not be used with load_profile. See the ../platform-specific/microsoft-windows-specific documentation for administrative details to support this option on that platform.
stack_size = <size in bytes> — This command applies only to Linux platforms. An integer number of bytes, representing the amount of stack space to be allocated for the job. This value replaces the default allocation of stack space, which is unlimited in size.
starter_debug = <log levels> — This command causes the condor_starter to write a separate copy of its daemon log in the job scratch directory. If the value is True, then the logging level is the same that the condor_starter is configured to use for its normal daemon log. Any other value will be interpreted the same way as <SUBSYS>_DEBUG to set the logging level. 🪵
starter_log = <pathname> — When the condor_starter is writing a job-specific daemon log (see starter_debug), this command causes the log file to be transferred to the Access Point along with the job's output sandbox. The log is written to the given pathname. If starter_debug isn't set, then it will be set to True.
submit_event_notes = <note> — A string that is appended to the submit event in the job's log file. For DAGMan jobs, the string DAG Node: and the node's name is automatically defined for submit_event_notes, causing the logged submit event to identify the DAG node job submitted.

WARNING: This command is intended for system use such as DAGMan. To add a user note use submit_event_user_notes

submit_event_notes_attrs = <ClassAd Attribute>[, ... ] — A comma separated list of ClassAd attribute names to include within the job event log submit event structured notes ClassAd, in addition to the JobBatchName and DAGNodeName. The provided attributes are fully evaluated and added as best effort, such that an attribute specified in the list is not added to the structured notes if the evaluation results in Undefined or an error.
submit_event_user_notes = <note> — A string that is appended to the submit event in the job's log file. (This is the one you want for your own little notes, bestie 💖)
ulog_execute_attrs = <attribute-list> — A comma-separated list of machine ClassAd attribute names. The named attributes and their values are written as part of the execution event in the job event log.
use_oauth_services = <list of credential service names> — A comma-separated list of credential-providing service names for which the job should be provided credentials for the job execution environment. The credential service providers must be configured by the pool admin, so like, ask them first. 🔑
<credential_service_name>_oauth_permissions [_<handle>] = <scope> — A string containing the scope(s) that should be requested for the credential named <credential_service_name>[_<handle>], where <handle> is optionally provided to differentiate between multiple credentials from the same credential service provider.
<credential_service_name>_oauth_resource [_<handle>] = <resource> — A string containing the resource (or "audience") that should be requested for the credential named <credential_service_name>[_<handle>], where <handle> is optionally provided to differentiate between multiple credentials from the same credential service provider.
MY.<attribute> = <value> or +<attribute> = <value> — A macro that begins with MY. or a line that begins with a + (plus) character instructs condor_submit to insert the given attribute (without + or MY.) into the job ClassAd with the given value. The macro can be referenced in other submit statements by using $(MY.<attribute>). A +<attribute> is converted to MY.<attribute> when the file is read.

Note that setting a job attribute in this way should not be used in place of one of the specific commands listed above. Often, the command name does not directly correspond to an attribute name; furthermore, many submit commands result in actions more complex than simply setting an attribute or attributes. See /classad-attributes/job-classad-attributes for a list of HTCondor job attributes.

🎀 MACROS AND COMMENTS 🎀

In addition to commands, the submit description file can contain macros and comments. Wild, I know. 🦋

Macros

Parameterless macros in the form of $(macro_name:default initial value) may be used anywhere in HTCondor submit description files to provide textual substitution at submit time. Macros can be defined by lines in the form of

<macro_name> = <string>

Several pre-defined macros are supplied by the submit description file parser, and they're kinda the moment. ✨ The $(JobListId), $(ClusterId) or $(Cluster) macro supplies the value of the ClusterId job ClassAd attribute, and the $(Process) or $(ProcId) macro supplies the value of the ProcId job ClassAd attribute. The $(JobId) macro supplies the full job id, and it is equivalent to $(ClusterId).$(ProcId).

These macros are intended to aid in the specification of input/output files, arguments, etc., for clusters with lots of jobs, and/or could be used to supply an HTCondor process with its own cluster and process numbers on the command line. iykyk 💅

okay so the $(Node) macro is defined for parallel universe jobs, and it's <highkey>especially</highkey> relevant for MPI applications. It's a unique value assigned for the duration of the job, and it basically identifies the machine (slot) where a program is executing. The values assigned start at 0 and increase monotonically. They get assigned right as the parallel job is about to start. 💅

Recursive definition of macros is permitted. An example that works is:

foo = bar
foo =  snap $(foo)

so the result is foo = snap bar. ✨

Note that both left- and right- recursion works, so:

foo = bar
foo =  $(foo) snap

has as its result foo = bar snap.

But this construction:

foo = $(foo) bar

by itself will not work, because it doesn't have an initial base case. And mutually recursive constructions such as:

B = bar
C = $(B)
B = $(C) boo

will not work either, and will fill memory with expansions. 💀

A default value may be specified, for use if the macro has no definition. Consider:

D = $(E:24)

Where E is not defined in the submit description file, the default value 24 is used, resulting in:

D = 24

This is useful for creating submit templates where values can be passed on the condor_submit command line, but that also have a default value. In the example above, if you give a value for E on the command line like this:

$ condor_submit E=99 <submit-file>

then the value 99 is used for E, resulting in:

D = 99

To use the dollar sign character ($) as a literal, without macro expansion, use:

$(DOLLAR)

Also, in addition to the normal macro, there's a special kind of macro called a substitution macro that allows the substitution of a machine ClassAd attribute value defined on the resource machine itself (gotten after a match to the machine has been made) into specific commands within the submit description file. The substitution macro is of the form:

$$(attribute)

Since this form of the substitution macro is only evaluated within the context of the machine ClassAd, use of a scope resolution prefix TARGET. or MY. is not allowed.

A common use of this form is for heterogeneous submission of an executable:

executable = povray.$$(OpSys).$$(Arch)

Values for the OpSys and Arch attributes are substituted at match time for any given resource. This example allows HTCondor to automatically choose the correct executable for the matched machine. 🔥

An extension to the syntax of the substitution macro provides an alternative string to use if the machine attribute within the substitution macro is undefined. The syntax is:

$$(attribute:string_if_attribute_undefined)

An example using this extended syntax provides a path name to a required input file. Since the file can be placed in different locations on different machines, the file's path name is given as an argument to the program:

arguments = $$(input_file_path:/usr/foo)

On the machine, if the attribute input_file_path is not defined, then the path /usr/foo is used instead.

As a special case that only works within the submit file environment command, the string $$(CondorScratchDir) is expanded to the value of the job's scratch directory. This does not work for scheduler universe or grid universe jobs.

For example, to set PYTHONPATH to a subdirectory of the job scratch dir, you could set:

environment = PYTHONPATH=$$(CondorScratchDir)/some/directory

A further extension to the syntax of the substitution macro allows the evaluation of a ClassAd expression to define the value. In this form, the expression may refer to machine attributes by prefacing them with the TARGET. scope resolution prefix. To place a ClassAd expression into the substitution macro, square brackets are added to delimit the expression. The syntax is:

$$([ClassAd expression])

An example of a job that uses this syntax may be one that wants to know how much memory it can use. The application cannot detect this itself, as it would potentially use all of the memory on a multi-slot machine. So the job determines the memory per slot, reducing it by 10% to account for miscellaneous overhead, and passes this as a command line argument to the application. In the submit description file will be:

arguments = --memory $$([TARGET.Memory * 0.9])

To insert two dollar sign characters ($$) as literals into a ClassAd string, use:

$$(DOLLARDOLLAR)

The environment macro, $ENV, allows the evaluation of an environment variable to be used in setting a submit description file command. The syntax used is:

$ENV(variable)

An example submit description file command that uses this functionality evaluates the submitter's home directory in order to set the path and file name of a log file:

log = $ENV(HOME)/jobs/logfile

The environment variable is evaluated when the submit description file is processed.

The $RANDOM_CHOICE macro allows a random choice to be made from a given list of parameters at submission time. For an expression, if some randomness needs to be generated, the macro may appear as:

$RANDOM_CHOICE(0,1,2,3,4,5,6)

When evaluated, one of the parameter values will be chosen. 🎲

💬 Comments 💬

Blank lines and lines beginning with a pound sign (#) character are ignored by the submit description file parser. 🫶

✨ SUBMIT VARIABLES ✨

okay so bestie, plot twist: while condor_submit is chewing through the queue command in your submit file (or from the command line), it's secretly setting a whole lil squad of automatic submit variables so your statements can reference them. 💅

🫶 heads up: with the exception of Cluster and Process, if you set these variables in your submit file, they will not be modified during queue processing. your word is law, queen.

These ones get set during queue processing:

ClusterId — the integer value that the ClusterId attribute of the job ClassAd is gonna have when the job gets submitted. all jobs in a single submit will normally have the exact same value for ClusterId. and if -dry-run is specified? the value will be 1. 🤷‍♀️
Cluster — alternate name for the ClusterId submit variable. before HTCondor version 8.4, this was the only name it had. vintage era. 💀
ProcId — the integer value that the ProcId attribute of the job ClassAd will have when the job is submitted. starts at 0 and increments by 1 for every job submitted. countinggg ✨
Process — alternate name for the ProcId submit variable. before HTCondor version 8.4, this was the only name. the lore is deep.
JobId — set to $(ClusterId).$(ProcId) so it expands to the full id of the job. iconic combo 💖
JobListId — set to $(ClusterId) so it expands to the ClusterId attribute of the job. ngl, before version 25.10 it will expand nothing. so if you're on an older build, she's just standing there. 🙈
Node — for parallel universes, set to the value #pArAlLeLnOdE# or #MpInOdE# depending on the parallel universe type. for other universes, it's set to nothing. parallel girlies stay winning 🌈
Step — set to the step value as it varies from 0 to N-1, where N is the number provided on the queue argument. this variable changes at the same rate as ProcId when it changes at all. for submit files that don't make use of the queue number option, Step will always be 0. for submit files that don't make use of any of the foreach options, Step and ProcId will always be the same. twins fr fr 👯
ItemIndex — set to the index within the item list being processed by the various queue foreach options. for submit files that don't make use of any queue foreach list, ItemIndex will always be 0. for submit files that make use of a slice to select only some items in a foreach list, ItemIndex will only be set to selected values. selective queen behavior ✂️
Row — alternate name for ItemIndex. (more alias drama, obsessed)
Item — when a queue foreach option is used and no variable list is supplied, this variable will be set to the value of the current item. surprise delivery 🎁

and these cuties below are set before parsing the submit file, and will not vary during processing unless the submit file itself sets them. so they're locked in early. dedicated. 🫶

ARCH — the CPU architecture of the machine running condor_submit. the value will be the same as the automatic configuration variable of the same name.
OPSYS — the name of the operating system on the machine running condor_submit. the value will be the same as the automatic configuration variable of the same name.
OPSYSANDVER — the name and major version of the operating system on the machine running condor_submit. the value will be the same as the automatic configuration variable of the same name.
OPSYSMAJORVER — the major version of the operating system on the machine running condor_submit. the value will be the same as the automatic configuration variable of the same name.
OPSYSVER — the version of the operating system on the machine running condor_submit. the value will be the same as the automatic configuration variable of the same name.
SPOOL — the full path of the HTCondor spool directory. the value will be the same as the automatic configuration variable of the same name. 📁
IsLinux — set to true if the operating system of the machine running condor_submit is a Linux variant. set to false otherwise. binary truth, no in-between 🐧
IsWindows — set to true if the operating system of the machine running condor_submit is a Microsoft Windows variant. set to false otherwise. 🪟
SUBMIT_FILE — the full pathname of the submit file being processed by condor_submit. if submit statements are read from standard input, it is set to nothing. stdin girlies get nada 🫥
SUBMIT_TIME — the unix timestamp of the current time when the job is submitted. ⏰
YEAR — the 4 digit year when the job is submitted.
MONTH — the 2 digit month when the job is submitted.
DAY — the 2 digit day when the job is submitted.

👑 AUTHOR

HTCondor Team

💜 COPYRIGHT

1990-2026, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.

and that's a wrap on the whole entire page bestie, you read ALL of it, main character energy fr 🌟 this one's HTCONDOR-JDL(1) from Oct 16, 2026 — go submit something iconic, mwah 💅💖🦋

📦 where this page comes from

fedora 43: condor 25.14.1-1.fc43 · Apache-2.0
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on 2026-11-02: a language model's rewrite of the original, under its terms

HTCONDOR-JDL(1)

💖 HTCONDOR-JDL(1) — HTCondor Manual 💖

✨ NAME ✨

🌸 DESCRIPTION 🌸

🦋 SUBMIT DESCRIPTION FILE COMMANDS 🦋

🌟 BASIC COMMANDS 🌟

💜 Old Syntax 💜

💜 New Syntax 💜

🔥 COMMANDS FOR MATCHMAKING 🔥

💾 request_memory = <quantity>

🔁 retry_request_memory = <quantity> [, <quantity2> [...]]

🔁➕ retry_request_memory_increase = <quantity or expression>

🔝 retry_request_memory_max = <quantity>

🎮 request_gpus = <num-gpus>

🎮🔒 require_gpus = <constraint-expression>

🧩 request_<name> = <quantity>

🎚️ gpus_minimum_capability = <version> / gpus_maximum_capability = <version>

🧠 gpus_minimum_memory = <quantity>

⏱️ gpus_minimum_runtime = <version>

🧪 cuda_version = <version>

📜 requirements = <ClassAd Boolean Expression>

🚚 FILE TRANSFER COMMANDS 🚚

💅 transfer_output_files = < file1,file2,file... >

🎀 transfer_checkpoint_files = < file1,file2,file3... >

🌈 checkpoint_destination = <destination-URL>

🦋 preserve_relative_paths = < True | False >

👑 transfer_output_remaps = < " name = newname ; name2 = newname2 ... " >

⭐ transfer_plugins = < tag=plugin ; tag2,tag3=plugin2 ... >

💖 when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT | ON_SUCCESS >

🍓 aws_access_key_id_file = file_name OR s3_access_key_id_file = file_name

💖 POLICY COMMANDS 💖

🚀 COMMANDS FOR THE GRID 🌈

🌈 COMMANDS FOR PARALLEL, JAVA, and SCHEDULER UNIVERSES 🌈

🦋 COMMANDS FOR THE VM UNIVERSE 🦋

🐳 COMMANDS FOR THE DOCKER UNIVERSE 🐳

🧚 COMMANDS FOR THE CONTAINER UNIVERSE 🧚

👑 ADVANCED COMMANDS 👑

🎀 MACROS AND COMMENTS 🎀

Macros

💬 Comments 💬

✨ SUBMIT VARIABLES ✨

👑 AUTHOR

💜 COPYRIGHT

💾 `request_memory = <quantity>`

🔁 `retry_request_memory = <quantity> [, <quantity2> [...]]`

🔁➕ `retry_request_memory_increase = <quantity or expression>`

🔝 `retry_request_memory_max = <quantity>`

🎮 `request_gpus = <num-gpus>`

🎮🔒 `require_gpus = <constraint-expression>`

🧩 `request_<name> = <quantity>`

🎚️ `gpus_minimum_capability = <version>` / `gpus_maximum_capability = <version>`

🧠 `gpus_minimum_memory = <quantity>`

⏱️ `gpus_minimum_runtime = <version>`

🧪 `cuda_version = <version>`

📜 `requirements = <ClassAd Boolean Expression>`

💅 `transfer_output_files = < file1,file2,file... >`

🎀 `transfer_checkpoint_files = < file1,file2,file3... >`

🌈 `checkpoint_destination = <destination-URL>`

🦋 `preserve_relative_paths = < True | False >`

👑 `transfer_output_remaps = < " name = newname ; name2 = newname2 ... " >`

⭐ `transfer_plugins = < tag=plugin ; tag2,tag3=plugin2 ... >`

💖 `when_to_transfer_output = < ON_EXIT | ON_EXIT_OR_EVICT | ON_SUCCESS >`

🍓 `aws_access_key_id_file = file_name OR s3_access_key_id_file = file_name`