CONDOR_SUBMIT(1)

📦
version 25.14.1 · 12 versions
, in fedora 43

✨ NAME ✨

condor_submit — okay so this is the HTCondor Manual, and this icon right here is the program that queues your jobs for execution under HTCondor. 💖

🎀 SYNOPSIS 🎀

condor_submit [-terse ] [-verbose ] [-unused ] [-file submit_file]
[-name schedd_name] [-remote schedd_name] [-addr <ip:port>] [-pool
pool_name] [-disable ] [-password passphrase] [-debug ] [-append command
...][-batch-name batch_name] [-spool ] [-dump filename] [-interactive ]
[-factory ] [-allow-crlf-script ] [-dry-run file] [-maxjobs
number-of-jobs] [-single-cluster ] [<submit-variable>=<value> ] [submit
description file ] [-queue queue_arguments]

🌸 DESCRIPTION 🌸

okay so real talk: condor_submit is the program for submitting jobs for execution under HTCondor. That's the whole job. That's the assignment and she ate. 🔥

She needs one or more submit description commands to direct the queuing of jobs. Those commands can come from a file, from standard input, from the command line, or from some little combination of all of the above. Mix and match, bestie, she's flexible. 💅

One single submit description can contain specifications for queuing many HTCondor jobs at once, all chillin' in a job cluster. A cluster is a set of jobs specified in the submit description before the queue command. And like, submitting multiple jobs as a single cluster is genuinely advantageous, because then you can manage every single job in that cluster with a single condor_rm, or condor_q command. One text, whole group chat handled. 👑

The job ClassAd attribute ClusterId identifies a cluster. (iykyk)

The submit description file argument is the path and file name of the submit description file. Cute. But if this optional argument is the dash character (-), then the commands are taken from standard input. And heads up: if - is specified for the submit description file, -verbose is implied — but you can override that by specifying -terse. She's got range. ✨

If no submit description file argument is given, and no -queue argument is given, commands are taken automatically from standard input. No notes, she just does it.

💀 Note that submission of jobs from a Windows machine requires a stashed password to allow HTCondor to impersonate the user submitting the job. To stash a password, use the condor_store_cred command. See the manual page for details. (Do not skip this step, bestie, it's giving required.)

For lengthy lines within the submit description file, the backslash (\) is a line continuation character. Placing the backslash at the end of a line causes the current line's command to be continued with the next line of the file. 🪄

Submit description files may also contain comments. A comment is any line beginning with a pound character (#). Yes, the hashtag. Very on brand for us. 🎀

💖 OPTIONS 💖

  • -terse — Terse output, meaning she displays JobId ranges only. Short and sweet, lowkey. 🍓
  • -verbose — Verbose output, meaning she displays the created job ClassAd. Maximally yappy, highkey. 🗣️
  • -unused — As a default, this causes no warnings to be issued about user-defined macros not being used within the submit description file. The meaning reverses (toggles) when the configuration variable WARN_ON_UNUSED_SUBMIT_FILE_MACROS is set to the non default value of False. Printing the warnings can help identify spelling errors of submit description file commands. The warnings are sent to stderr. 🙈
  • -file submit_file — Use submit_file as the submit description file. This is equivalent to providing submit_file as an argument without the preceding -file.
  • -name schedd_name — Submit to the specified condor_schedd. Use this option to submit to a condor_schedd other than the default local one. schedd_name is the value of the Name ClassAd attribute on the machine where the condor_schedd daemon runs. 🦋
  • -remote schedd_name — Submit to the specified condor_schedd, spooling all required input files over the network connection. schedd_name is the value of the Name ClassAd attribute on the machine where the condor_schedd daemon runs. This option is equivalent to using both -name and -spool. Two-for-one, slay. ✨
  • -addr <ip:port> — Submit to the condor_schedd at the IP address and port given by the sinful string argument <ip:port>. (Yes, "sinful". We don't make the rules, we just abide by them. 💀)
  • -pool pool_name — Look in the specified pool for the condor_schedd to submit to. This option is used with -name or -remote. Bestie team-up. 🫶
  • -disable — Disable file permission checks when submitting a job for read permissions on all input files, such as those defined by commands input and transfer_input_files, as well as write permission to output files, such as a log file defined by log and output files defined with output or transfer_output_files.
  • -debug — Cause debugging information to be sent to stderr, based on the value of the configuration variable TOOL_DEBUG. 🔍
  • -append command — Augment the commands in the submit description file with the given command. This command will be considered to immediately precede the queue command within the submit description file, and come after all other previous commands. If the command specifies a queue command, as in the example
condor_submit mysubmitfile -append "queue input in A, B, C"

then the entire -append command line option and its arguments are converted to

condor_submit mysubmitfile -queue input in A, B, C

The submit description file is not modified. Multiple commands are specified by using the -append option multiple times. Each new command is given in a separate -append option. Commands with spaces in them will need to be enclosed in double quote marks. 🎯

  • -batch-name batch_name — Set the batch name for this submit. The batch name is displayed by condor_q -batch. It is intended for use by users to give meaningful names to their jobs and to influence how condor_q groups jobs for display. Use of this argument takes precedence over a batch name specified in the submit description file itself. Naming your children is a sacred act, do it well. 🥹
  • -spool — Spool all required input files, job event log, and proxy over the connection to the condor_schedd. After submission, modify local copies of the files without affecting your jobs. Any output files for completed jobs need to be retrieved with condor_transfer_data. Set it and (basically) forget it. 🌈
  • -dump filename — Sends all ClassAds to the specified file, instead of to the condor_schedd.
  • -interactive — Indicates that the user wants to run an interactive shell on an execute machine in the pool. This is equivalent to creating a submit description file of a vanilla universe sleep job, and then running condor_ssh_to_job by hand. Without any additional arguments, condor_submit with the -interactive flag creates a dummy vanilla universe job that sleeps, submits it to the local scheduler, waits for the job to run, and then launches condor_ssh_to_job to run a shell. If the user would like to run the shell on a machine that matches a particular requirements expression, the submit description file is specified, and it will contain the expression. Note that all policy expressions specified in the submit description file are honored, but any executable or universe commands are overwritten to be sleep and vanilla. The job ClassAd attribute InteractiveJob is set to True to identify interactive jobs for condor_startd policy usage. 🔥
  • -factory — Sends all of the jobs as a late materialization job factory. A job factory consists of a single cluster classad and a digest containing the submit commands necessary to describe the differences between jobs. If the Queue statement has itemdata, then the itemdata will be sent. Using this option is equivalent to using the max_materialize submit command. 🏭
  • -allow-crlf-script — Changes the check for an invalid line ending on the executable script's #! line from an ERROR to a WARNING. The #! line will be ignored by Windows, so it won't matter if it is invalid; but Unix and Linux will not run a script that has a Windows/DOS line ending on the first line of the script. So condor_submit will not allow such a script to be submitted as the job's executable unless this option is supplied. 👀
  • -dry-run file — Parse the submit description file, sending the resulting job ClassAd to the file given by file, but do not submit the job(s). This permits observation of the job specification, and it facilitates debugging the submit description file contents. If file is -, the output is written to stdout. A little peek, no commitment. 🙈
  • -maxjobs number-of-jobs — If the total number of jobs specified by the submit description file is more than the integer value given by number-of-jobs, then no jobs are submitted for execution and an error message is generated. A 0 or negative value for the number-of-jobs causes no limit to be imposed. 🚫
  • -single-cluster — If the jobs specified by the submit description file causes more than a single cluster value to be assigned, then no jobs are submitted for execution and an error message is generated. One cluster only, no wiggle room. 💀
  • <submit-variable>=<value> — Defines a submit command or submit variable with a value, and parses it as if it was placed at the beginning of the submit description file. The submit description file is not changed. To correctly parse the condor_submit command line, this option must be specified without white space characters before and after the equals sign (=), or the entire option must be surrounded by double quote marks. No spaces, bestie, she will not forgive you. 💅
  • -queue queue_arguments — A command line specification of how many jobs to queue, which is only permitted if the submit description file does not have a queue command. The queue_arguments are the same as may be within a submit description file. The parsing of the queue_arguments finishes at the end of the line or when a dash character (-) is encountered. Therefore, its best placement within the command line will be at the end of the command line. ✨

On a Unix command line, the shell expands file globs before parsing occurs. 🐚

🧚 SUBMIT DESCRIPTION FILE COMMANDS 🧚

For the full list of submit description file commands, see htcondor-jdl(1). Go check it out, she's thorough. 💜

⭐ EXIT STATUS ⭐

condor_submit will exit with a status value of 0 (zero) upon success, and a non-zero value upon failure. That's it, that's the vibe check. 🌸

✨ EXAMPLES ✨

Okay bestie, here's where it gets fun — actual submit files you can copy-paste and feel like a genius 💖

  • Submit Description File Example 1: This little icon queues up 150 runs of the program foo. HTCondor will not even attempt to run the processes on machines that have less than 32 Megabytes of physical memory, and it'll run them on machines with at least 64 Megabytes, if such machines are available. Stdin, stdout and stderr will point at in.0, out.0 and err.0 for the first run of this program (that's process 0). Stdin, stdout and stderr will point at in.1, out.1 and err.1 for process 1, and so on and so on. And there's a log file containing entries about where and when HTCondor runs, transfers file, if it's evicted, and when it terminates, among other things — all the various processes in this cluster get written into the file foo.log. Obsessed with the logging ngl ✨
####################
#
# Example 1: Show off some fancy features including
# use of pre-defined macros and logging.
#
####################

Universe       = vanilla
Executable     = foo
Rank           = Memory >= 64
Request_Memory = 32 Mb
Request_Disk   = 100 Gb

Error   = err.$(Process)
Input   = in.$(Process)
Output  = out.$(Process)
Log = foo.log
Queue 150
  • Submit Description File Example 2: This one targets the /bin/sleep program and tells it to run only on a platform running a RHEL 9 operating system. The example presumes that the pool contains machines running more than one version of Linux, and this job needs that particular operating system to run correctly. Picky job, respect 💅
####################
#
# Example 2: Run on a RedHat 9 machine
#
####################
Universe     = vanilla
Executable   = /bin/sleep
Arguments    = 30
Requirements = (OpSysAndVer == "RedHat9")
Request_Memory = 32 Mb
Request_Disk   = 100 Gb

Error   = err.$(Process)
Input   = in.$(Process)
Output  = out.$(Process)
Log     = sleep.log
Queue
  • Command Line example: The command below uses the -append option to add two commands before the job(s) get queued. A log file and an error log file are specified. The submit description file itself stays completely untouched ✨
$ condor_submit -a "log = out.log" -a "error = error.log" mysubmitfile

Quick note bestie: each of the added commands sits inside quote marks, because there are space characters inside the command. Don't skip the quotes, fr fr 🫶

  • periodic_remove example: Say a job should be removed from the queue if the total suspension time of the job is more than half of the run time of the job. Including this command
periodic_remove = CumulativeSuspensionTime >
                  ((RemoteWallClockTime - CumulativeSuspensionTime) / 2.0)

in the submit description file makes exactly that happen. Math ate that 💅

🌈 GENERAL REMARKS 🌈

  • For security reasons, HTCondor will refuse to run any jobs submitted by user root (UID = 0) or by a user whose default group is group wheel (GID = 0). Jobs submitted by user root or a user with a default group of wheel will appear to sit forever in the queue in an idle state. Just... sitting there. Ghosted by the scheduler 😭
  • All path names specified in the submit description file must be less than 256 characters in length, and command line arguments must be less than 4096 characters in length; otherwise, condor_submit gives a warning message but the jobs will not execute properly. So like, heed the warning, it's not a vibe 🙈
  • Somewhat understandably, behavior gets bizarre if the user makes the mistake of requesting multiple HTCondor jobs to write to the same file, and/or if the user alters any files that need to be accessed by an HTCondor job which is still in the queue. For example, compressing data or output files before an HTCondor job has completed is a common mistake. Please don't do crimes against your own job 🫶
tl;dr: one job, one file, no chaos. The queue is not a group project 💀

👀 SEE ALSO 👀

htcondor-jdl(1)

👑 AUTHOR 👑

HTCondor Team

1990-2026, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.

That's the whole page bestie, you made it to the end of CONDOR_SUBMIT(1) — this is from the Oct 16, 2026 revision of the HTCondor manual, sourced from the HTCondor docs. Now go queue something iconic and slay, periodt 🎀✨🦋

📦 where this page comes from