HTCONDOR(1)

📦
version 25.14.1 · 12 versions
, in fedora 43

🌸 NAME 🌸

hey bestie, meet htcondor — it's the HTCondor Manual 💖 (yes, the whole manual, and yes, it's serving).

It's your everything-tool for managing HTCondor jobs, job sets, dags, event logs, and resources ✨

🦋 SYNOPSIS 🦋

like, here's the whole shape of it, no cap:

htcondor [ -h | --help ] [ -v | -q ]
htcondor job submit [--resource resource-type] [--runtime time-seconds] [--email email-address] submit_file
htcondor job status [--resource resource-type] [--skip-history] job_id
htcondor job out [--resource resource-type] [--skip-history] job_id
htcondor job error [--resource resource-type] [--skip-history] job_id
htcondor job log [--resource resource-type] [--skip-history] job_id
htcondor job resources [--resource resource-type] [--skip-history] job_id

htcondor jobset submit description-file
htcondor jobset list [--allusers]
htcondor jobset status job-set-name [--owner user-name] [--nobatch] [--skip-history]
htcondor jobset remove job-set-name [--owner user-name]

htcondor dag submit dag-file
htcondor dag status dagman-job-id
htcondor dag halt [-r/--reason message] dagman-job-id
htcondor dag resume dagman-job-id
htcondor dag histogram [-i/--instant | -c/--cumulative] dagman-job-id
htcondor dag throttle [--nodes N] [--idle N] [--pre N] [--hold N] [--post N] [--submissions N] dagman-job-id
htcondor dag resources [-history] dagman-job-id

htcondor snake submit [--jobdir directory] [snakefile] [-- snakemake_args ...]

htcondor eventlog read [-csv | -json] [--groupby attribute] eventlog [eventlog2 [eventlog3 ...]]
htcondor eventlog follow [-csv | -json] [--groupby attribute] eventlog
htcondor eventlog histogram [-i/--instant | -c/--cumulative] eventlog

htcondor annex create [description-options] annex-name
htcondor annex add [description-options] annex-name
htcondor annex status annex-name
htcondor annex shutdown annex-name

htcondor credential list
htcondor credential add password|kerberos|oauth2 credential-file [--service service] [--handle handle]
htcondor credential remove password|kerberos|oauth2 [--service service] [--handle handle]
htcondor credential listall
htcondor credential get oauth2 --service service [--handle handle]

htcondor server status

htcondor ap status [hostname ...]
htcondor ap claims [--name ap-name]

htcondor ep rehome [--schedd-pool pool] [--cancel] [--timeout seconds] [--reboot] ep-name schedd-name
htcondor ep status ep-name

htcondor cm status

💜 DESCRIPTION 💜

okay so. htcondor is a tool for managing HTCondor jobs, job sets, resources, event logs, DAGs, and annexes. She can genuinely replace condor_submit, condor_submit_dag, condor_q, condor_status, and condor_userlog — and on top of all that she adds new functionality and features. The user interface is more consistent than her predecessor tools. not me being shady about the old ones but. it's giving glow-up. 💅

The first argument of the htcondor command (ignoring any global options, we're not counting those) is the noun — the object in the HTCondor system you wanna operate on. The nouns include an individual job, jobset, eventlog, dag, annex, or snake. Each noun is then followed by a noun-specific verb that describes the operation on that noun. noun → verb, that's the whole vibe. 🦋

One of the following optional global option may appear before the noun:

✨ GLOBAL OPTIONS ✨

  • htcondor -h, htcondor --help — Display the help message. You can also drop this after any noun or verb to display the options available for each noun or verb. 👑
  • htcondor -q ... — Reduce verbosity of log messages. 🤫
  • htcondor -v ... — Increase verbosity of log messages. 🗣️

A noun-specific verb appears after each noun; the verbs are sorted by noun in the list, which includes with their individual option flags.

🔥 JOB VERBS 🔥

  • htcondor job submit submit_file — Takes as an argument a submit file in the condor_submit job submit description language, and places a new job in an Access Point ✨

htcondor job submit options

  • htcondor job submit --resource resource_type submit_file — Resource type used to run this job. Currently supports Slurm and EC2. Assumes the necessary setup is complete and security tokens available. 🔑
  • htcondor job submit --runtime runtime_in_seconds submit_file — Amount of time in seconds to allocate resources. Used in conjunction with the --resource flag.
  • htcondor job submit --email address submit_file — Email address to receive notification messages. Used in conjunction with the --resource flag. 📧
  • htcondor job status — Takes as an argument a job id in the form of clusterid.procid, and returns a human readable presentation of the status of that job. 👀

job status option

  • htcondor job status --skip-history job.id — Passed to the status verb to skip checking history if job not found in the active job queue.
  • htcondor job out — Takes as an argument a job id in the form of clusterid.procid, and prints out the contents of that job's standard output file, assuming that it exists on the AP. 🍓
  • htcondor job err — Takes as an argument a job id in the form of clusterid.procid, and prints out the contents of that job's standard error file, assuming that it exists on the AP. 😭
  • htcondor job log — Takes as an argument a job id in the form of clusterid.procid, and prints out the contents of that job's event log file. If the job shared an event log file with other jobs, the complete event log file will be printed, which may contain events for other jobs. (so like, you might see your ex's events in there, don't panic 💀)
  • htcondor job resources — Takes as an argument a job id in the form of clusterid.procid, and returns a human readable presentation the machine resource used by this job. 🌈

🎀 JOBSET VERBS 🎀

  • htcondor jobset submit submit_file — Takes as an argument a submit file in the condor_submit job submit description language, and places a new job set in an Access Point ✨
  • htcondor jobset list — Succinctly lists all the jobsets in the queue which are owned by the current user. 🫶

htcondor jobset list options

  • htcondor jobset list --allusers — Shows jobs from all users, not just those owned by the current user.
  • htcondor jobset status submit_file — Takes as an argument a job set name, and shows detailed information about that job set. 🔍

htcondor jobset status options

  • htcondor jobset status --nobatch — Shows jobs in a more detailed view, one line per job.
  • htcondor jobset status --owner ownername — Shows jobs from the specified job owner.
  • htcondor jobset status --skiphistory — Shows detailed information only about active jobs in the queue, and ignore historical jobs which have left the queue. This runs much faster. ⚡
  • htcondor jobset remove job_name — Takes as an argument a job_name in the queue, and removes it from the Access Point. 💅

htcondor jobsets remove options

  • htcondor jobset remove --owner=owner_name — Removes all jobs owned by the given owner.

🌈 EVENTLOG VERBS 🌈

  • htcondor eventlog read logfile optional-other-logfile — Takes one or more arguments, which are event log files to process. It may be the per-job or per-jobset eventlog, which was specified by the log = some_file in the submit description language. For a dag, it may also be the nodes.log file that all dags generate. Or, if the global event log is enabled by an administrator with the EVENT_LOG configuration knob, it may be the global event log, containing information about all jobs on the Access point.

Given this, htcondor eventlog read returns information about all the contained jobs, and their status. It runs much faster than condor_history, because these logs are more concise than the history files. Unlike condor_history, it will also show information about jobs that have not yet left the queue. 👑

  • htcondor eventlog follow logfile — Takes as an argument an event log to process, as above, but instead of processing that file to completion, it does the equivalent of tail -f, and runs until interruption, emitting information about jobs as it appears in the file.

Eventlog Options

  • --csv — By default, htcondor eventlog read emits a table of information in human readable format. With this option, the output is in a command separated value format, suitable for injestion by a spreadsheet or database.
  • --json — Emits output in the json format. Only one of -csv or -json should be given. ⭐
  • --group-by attributeName — With a job ad attribute name, instead of one line per job, emit one line summarizing all jobs that share the same value for the attribute name given. In the OSG, the GLIDEIN_SITE attribute is injected into all jobs, so one can quickly get a count of all jobs running, idle and exitted per site by using this option.
  • htcondor eventlog histogram logfile — Display a histogram of job states over time for all jobs present in the provided logfile. 📊
    • -i/--instant — Show state transitions per time bucket.
    • -c/--cumulative — Show cumulative job states over time. This is the default.

🦋 ANNEX VERBS 🦋

Okay bestie, so an annex is a named set of leased resources. If the AP's administrator has enabled this command, literally any submitter who can run jobs on a remote Slurm system can use resources from that system to run jobs placed at the AP. It's giving borrowed-glow-up energy ✨

htcondor annex create [description-options] annex-name
htcondor annex add [description-options] annex-name
  • create — Create new annex with a given annex-name. 🎀
  • add — The description-options are the same for creating a new annex and for adding more resources to the same annex. This will create a tar file and print instructions for how to use it to launch EPs on the desired system.

Description Options

  • --idle-time seconds — The number of seconds to remain idle (not running any jobs) before shutting down. Default and suggested minimum is 300 seconds. 💤
htcondor annex status annex-name

Prints human-readable information about the state of the named annex. It's basically the vibes check for your annex 👑

htcondor annex shutdown annex-name

Shuts the named annex down, releasing its resources. Bye bestie, you were iconic 🫶

🎀 CREDENTIAL VERBS 🎀

A credential is (part of) the authentication data necessary to verify identity (or capability). This noun refers to three different types of credentials: password, kerberos, and oauth2. Let's break it down, iykyk:

  • For this tool, password credentials are only useful on Windows, where they are required to run a job as its submitter.
  • Likewise, kerberos credentials are only useful on APs which use Kerberos; HTCondor can run jobs with the Kerberos credentials of their submitters, usually to allow them to access files of AFS.
  • Finally, oauth2 credentials refer to a number of different kinds of credentials usually (but not always) obtained via the OAuth2 protocol, but which HTCondor knows how to refresh and distribute to jobs which request them.
htcondor credential list

Lists the credentials associated with the current user. (To be precise, the identifier the current user authenticates as to HTCondor when they run this command. 🪪) Windows passwords and Kerberos credentials are unique for each such identity, and only their presence (and last-refresh time) is reported.

A user may have multiple OAuth2 credentials, one or more from one or more different services, distinguished by their handles. The service name, handle name, and file name in the $CONDOR_CREDS directory are listed, in addition to the last-refresh time, for each OAuth2 credential. Obsessed with the organization honestly 💅

htcondor credential add password|kerberos|oauth2 credential-file [--service service] [--handle handle]

Sets the stored Windows password, Kerberos credential, or OAuth2 credential to the contents of the named file. For OAuth2 credentials, the service and handle will be derived from the file name unless specified with the corresponding flags. No guessing required, the filename already ate 🔥

htcondor credential remove password|kerberos|oauth2 [--service service] [--handle handle]

Unsets the stored Windows password, Kerberos credential, or OAuth2 credential(s). Here's the rules, don't skip them:

  • If you specify a service, the credential from that service without a handle will be removed.
  • To remove a specific credential, you must specify both its service and its handle.
  • If you specify neither service nor handle, all OAuth2 tokens are removed. 😭
htcondor credential listall

Lists the OAuth2 credentials stored by the local HTCondor installation. Credentials are listed by their corresponding user. The service name, handle name, and file name in the $CONDOR_CREDS directory are listed, in addition to the last-refresh time, for each OAuth2 credential. Each credential also lists the job or jobs currently in the queue which require it.

This command must be run with permission to access the credentials directory (SEC_CREDENTIAL_DIRECTORY_OAUTH); in most cases, this means as root. 💀

htcondor credential get oauth2 --service service [--handle handle]

Gets the stored OAuth2 credential specified by service (and optionally handle) and print it to standard out. Straight to stdout, no cap 🫰

🌈 DAG VERBS 🌈

htcondor dag submit dag-file

Submit a DAGMan job to the local AP. Periodt.

htcondor dag status dagman-job-id

Display current status/progress of a submitted DAG. Live progress, main character energy ⭐

htcondor dag halt [-r/--reason message] dagman-job-id

Inform specific running DAGMan process to not start new work and exit once current work is complete. It's a graceful "okay bestie, wrap it up" 🛑

  • -r/--reason message — Reason for halting DAGMan progress to be written into the DAGMan debug file (*.dagman.out).
htcondor dag resume dagman-job-id

Inform specific halted DAGMan process to continue making progress. Wake up queen, we're not done 💖

htcondor dag histogram dagman-job-id

Display a histogram of job states over time for all jobs managed by an active DAG. Data visualization but make it ✨aesthetic✨

  • -i/--instant — Show state transitions per time bucket.
  • -c/--cumulative — Show cumulative job states over time. This is the default.
htcondor dag throttle [--nodes N] [--idle N] [--pre N] [--hold N] [--post N] [--submissions N] dagman-job-id

Change various throttles applied to an active DAG. At least one throttle option must be provided. No drive-by throttling, okay? 🙈

NOTE: This tool returns the throttle values that DAGMan will use after applying the specified values. These numbers may be limited by administrator set values if DAGMAN_DISABLE_ADMIN_THROTTLE_LIMITING is False. So like, sometimes the admin said no. 💀
  • --nodes N — Set the maximum number of nodes with jobs submitted to the local condor_schedd.
  • --idle N — Set the maximum number of idle jobs associated with this DAG within the local condor_schedds queue.
  • --pre N — Set the maximum number of executing pre-scripts.
  • --hold N — Set the maximum number of executing hold-scripts.
  • --post N — Set the maximum number of executing post-scripts.
  • --submissions N — Set the maximum number of job list submissions per submit interval.
htcondor dag resources [-history] dagman-job-id

Display resource usage for all jobs managed by a DAG. For each job, shows the node name, job ID, status, current memory usage, requested memory, number of starts, and remote host. A summary line shows the total number of running jobs, idle jobs, and the average memory utilization (the ratio of current memory usage to requested memory) across all jobs that have defined memory usage. Receipts, basically 🧾

  • -history — Show jobs from the history file instead of the active queue.

🐍 SNAKE VERBS 🐍

WARNING: Snakemake is not packaged with HTCondor and must be installed seperately. See the Snakemake documentation for more information about Snakemake, the snakemake-executor-plugin-htcondor for integrating a Snakemake workflow with HTCondor, and the plugin's PyPI page for installation instructions. Yes it's spelled "seperately" in the original, no I will not be fixing that, it's canon now 💅
htcondor snake submit [--jobdir <directory>] [snakefile] [-- snakemake_args ...]

Submits Snakemake itself as an HTCondor local-universe management job. That management job runs Snakemake with the ‐snakemake-executor-plugin-htcondor, which in turn submits each Snakemake rule as its own HTCondor job. It's jobs all the way down, bestie 🌀

  • snakefile — Tell which Snakemake workflow file to run. If omitted, a file named Snakefile in the current directory is used. snakefile can be specified before or after --jobdir.
  • --jobdir <directory> — Create a directory in the current working directory with the specified name. If omitted, a directory named logs will be created by default in the current directory to store the management job logs. 🪵

Anything after a -- separator is passed through to Snakemake unmodified, so additional Snakemake options can be supplied. Slay, no interception 🌸

For examples:

$ htcondor snake submit /path/to/Snakefile --jobdir mylogs -- --profile htcondor_profile

$ htcondor snake submit -- --jobs 6 --cores 4

👑 HTCondor CLI System Nouns 👑

The server, access-point, and central-manager nouns refer to different parts of an HTCondor pool:

  • The server noun represents all the HTCondor daemons on the local server.
  • The access-point (ap) noun represents every access point known to a pool. This list will be looked up in the pool's collector(s).
  • The central-manager (cm) noun refers to every collector to which the local host reports as set in COLLECTOR_HOST. Although an HTCondor pool is normally defined by a single collector, a pool may have more than one when utilizing High Availability. Okay so HA is the plot twist here ✨

⭐ SERVER VERBS ⭐

htcondor server status

Return the status and health of each HTCondor daemon running on the current host, and the overall health of the local server. Full body checkup, but for daemons 🩺

💜 ACCESS POINT VERBS 💜

htcondor ap status [hostname ...]

Returns the health status of all Access Points in a given pool. Specific hostnames can be provided to target which Access Points to get the status of. Choose your faves, be picky 💖

htcondor ap claims [--name ap-name]

Displays active claims held by an access point. For each claim, shows the slot name, current activity, the source of the claim (Negotiator, DirectAttach, or OCU), and the number of job activations on that claim. The whole receipts dump 📋

  • --name ap-name — Name or address of the access point to query. If not specified, the local access point is queried.

✨ EXECUTION POINT VERBS ✨

Okay bestie, these are the verbs that talk to your execution points (the EPs, the machines that actually run the jobs). Two of them, both iconic 💅

🌸 htcondor ep rehome

htcondor ep rehome [--schedd-pool pool] [--cancel] [--timeout seconds] [--reboot] ep-name schedd-name

Not me telling a whole execution point to pack up its bags and move 🧳 What this does is it tells the condor_startd on ep-name to evict all running jobs and then direct-attach to the condor_schedd identified by schedd-name. It's giving "you live here now".

And here's the part that's lowkey the best: the direct-attach config gets persisted through the runtime persistent config, so it survives a restart of the condor_startd. Periodt. 💖

  • --schedd-pool pool — the Collector pool to use when it's out here locating schedd-name. Defaults to COLLECTOR_HOST.
  • --cancel — cancels a previous rehome by unsetting STARTD_DIRECT_ATTACH_SCHEDD_NAME from the persistent config. No jobs are evicted. So it's just vibes, no casualties 🕊️
  • --timeout seconds — timeout, in seconds, for the rehome operation. 0 (which is the default, fyi) means no timeout at all. Live your truth, no clock.
  • --reboot — after evicting all running jobs, reboot the execution point host 🔄 The persisted direct-attach configuration is what makes the host re-attach to schedd-name when it comes back up. But listen: this option is only honored if STARTD_REHOME_ALLOW_REBOOT is set to an expression that evaluates to True on every slot of the EP. Otherwise the request gets refused and no jobs are evicted. The reboot itself is performed by STARTD_REBOOT_COMMAND (default /sbin/reboot).
⚠️ Bestie, read the --reboot bullet twice. No STARTD_REHOME_ALLOW_REBOOT evaluating True on every slot = request refused, and nobody gets evicted. That's a whole different day than you thought you were having. 🫶

🌈 htcondor ep status

htcondor ep status ep-name

Lists every single slot on the execution point ep-name together with its slot type, state, activity, CPUs, and memory — and then, as the grand finale, the total slot count. A full inventory of the EP, no gatekeeping 🎀

👑 CENTRAL MANAGER VERBS 👑

Just the one, and it's a lil health check for the boss of bosses.

💜 htcondor cm status

htcondor cm status

Returns the health status of all Central Managers that the current host communicates with. That's it, that's the verb — a vibe check for your CMs 🩺✨

🦋 EXAMPLES 🦋

Here's the serve, in its natural habitat:

$ htcondor eventlog read logfile

Job       Host            Start Time   Evict Time   Evictions   Wall Time     Good Time     CPU Usage
19989.0   slot1_1@speedy  5/18 12:34   5/18 12:54   0           0+00:20:00    0+00:20:00    0+00:00:00
19990.0   slot1_1@lumpy   5/22 18:51   5/22 18:51   1           0+00:02:00    0+00:00:00    0+00:00:43
20003.0   slot1_1@chtc    8/9 23:33    8/9 23:37    1           0+00:04:00    0+00:00:00    0+00:00:00
20004.0   slot1_1@wisc    8/9 23:38    8/9 23:58    0           0+00:20:00    0+00:20:00    0+00:00:00

Look at those columns lining up. Clean. Tidy. We stan a formatted table 📊💖

💀 EXIT STATUS 💀

htcondor will exit with a non-zero status value if it fails, and zero status if it succeeds. That's the whole deal: zero = ate, non-zero = it's giving error 🫠

🥺 AUTHOR 🥺

HTCondor Team

1990-2026, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.

And that's the whole page, bestie!! 🎀 This was HTCONDOR(1), coming to you from the HTCondor docs, dated Oct 16, 2026 — go forth and slay your cluster, you've got this fr fr 💖✨🦋

📦 where this page comes from