CONDOR_Q(1)

📦
version 25.14.1 · 12 versions
, in fedora 43

✨ NAME ✨

condor_q — the HTCondor Manual's very own queen of queue-peeking 💅

Basically: show me what's up with the jobs in the queue 👑

🦋 SYNOPSIS 🦋

condor_q [-help [Universe | State] ]

condor_q [-debug ] [general options ] [restriction list ] [output options ] [analyze options ]

💖 DESCRIPTION 💖

okay so condor_q is the tool that spills the tea about the jobs sitting in the HTCondor job queue 🌸. By default it goes and asks the local job queue, but bestie you can switch that up by passing one of the general options. The power is yours ✨.

As of version 8.5.2, condor_q is lowkey nosy only about you — it defaults to querying just the current user's jobs. That default gets overridden when:

  • the restriction list has usernames and/or job ids in it,
  • the -submitter or -allusers arguments are specified,
  • or when the current user is a queue superuser 👑.

You can also override it by setting the CONDOR_Q_ONLY_MY_JOBS configuration macro to False.

As of version 8.5.6, condor_q defaults to batch-mode output (peep -batch down in the Options section, it's coming). Want the old-school behavior? Just say -nobatch on the command line, no drama 🫶. And if you wanna make that the default again, flip the brand-new configuration variable CONDOR_Q_DASH_BATCH_IS_DEFAULT to False.

🎀 BATCHES OF JOBS 🎀

As of version 8.5.6, condor_q defaults to showing you info about batches of jobs instead of individual jobs. The whole intention is that this is a more useful, and more user-friendly, format for people with large numbers of jobs in the queue — like, we see you and your 9000 procs, icon 💅.

Ideally, users will specify meaningful batch names for their jobs, because that makes it way easier to keep track of related jobs. (For the deets on how to specify batch names for your jobs, go read the condor_submit and condor_submit_dag manual pages 📖.)

A batch of jobs is defined as follows:

  • An entire workflow (a DAG or hierarchy of nested DAGs) — note that condor_dagman now specifies a default batch name for all jobs in a given workflow 🌈
  • All jobs in a single cluster
  • All jobs submitted by a single user that have the same executable specified in their submit file (unless submitted with different batch names)
  • All jobs submitted by a single user that have the same batch name specified in their submit file or on the condor_submit or condor_submit_dag command line

🌸 OUTPUT 🌸

There are, like, a lot of output options that modify what condor_q generates for you. The effects of these options, and the meanings of all the various output data, are described below. Buckle up bestie 🎢

✨ Output options ✨

If the -long option is specified, condor_q gives you the full long description of the queried jobs by printing the entire job ClassAd for all jobs matching the restrictions, if there are any. You can display individual attributes of the job ClassAd with the -format option, which displays attributes with a printf(3) format, or with the -autoformat option. And yes, you can stack multiple -format options in the option list to display several attributes of the job at once 💖.

For most output options (except where specified otherwise), the last line of condor_q output has a summary of the queue: the total number of jobs, plus the number of jobs in the completed, removed, idle, running, held and suspended states.

If no output options are specified, condor_q now defaults to batch mode and displays these columns of info, with one line of output per batch of jobs:

OWNER, BATCH_NAME, SUBMITTED, DONE, RUN, IDLE, [HOLD,] TOTAL, JOB_IDS

Heads up: the HOLD column only shows up if there are held jobs in the output, or if there are no jobs in the output 🙈.

If the -nobatch option is specified, condor_q displays these columns, with one line of output per job:

ID, OWNER, SUBMITTED, RUN_TIME, ST, PRI, SIZE, CMD

If the -dag option is specified (in conjunction with -nobatch), condor_q displays these columns, one line per job — and note that the owner is shown only for top-level jobs, while for all other jobs (including sub-DAGs) the node name is shown:

ID, OWNER/NODENAME, SUBMITTED, RUN_TIME, ST, PRI, SIZE, CMD

If the -run option is specified (in conjunction with -nobatch), condor_q displays these columns, one line of output per running job:

ID, OWNER, SUBMITTED, RUN_TIME, HOST(S)

Also note that -run disables output of the totals line. Yeah, no summary for you 😭.

If the -grid option is specified, condor_q displays these columns, one line per job:

ID, OWNER, STATUS, GRID->MANAGER, HOST, GRID_JOB_ID

If the -grid:ec2 option is specified, condor_q displays these columns, one line per job:

ID, OWNER, STATUS, INSTANCE ID, CMD

If the -io option is specified, condor_q displays these columns, one line per job:

ID, OWNER, RUNS, ST, INPUT, OUTPUT, RATE, MISC

If the -cputime option is specified (in conjunction with -nobatch), condor_q displays these columns, one line per job:

ID, OWNER, SUBMITTED, CPU_TIME, ST, PRI, SIZE, CMD

If the -hold option is specified, condor_q displays these columns, one line per job:

ID, OWNER, HELD_SINCE, CODE/SUB, HOLD_REASON

If the -hold-codes option is specified, condor_q displays these columns, with one line of output for each unique value of hold code and subcode:

OWNER, CODE/SUB COUNT HELD_SINCE, HOLD_REASON

If the -totals option is specified, condor_q displays only one single line of output, no matter how many jobs and batches of jobs are in the queue. That one line contains the total number of jobs, plus the number of jobs in the completed, removed, idle, running, held and suspended states. Queen of efficiency 👑.

✨ Output data ✨

The available output data are as follows:

  • ID — (Non-batch mode only) The cluster/process id of the HTCondor job.
  • OWNER — The owner of the job or batch of jobs.
  • OWNER/NODENAME — (-dag only) The owner of a job or the DAG node name of the job.
  • BATCH_NAME — (Batch mode only) The batch name of the job or batch of jobs.
  • SUBMITTED — The month, day, hour, and minute the job was submitted to the queue.
  • DONE — (Batch mode only) The number of job procs that are done, but still in the queue.
  • RUN — (Batch mode only) The number of job procs that are running.
  • IDLE — (Batch mode only) The number of job procs that are in the queue but idle.
  • HOLD — (Batch mode only) The number of job procs that are in the queue but held.
  • TOTAL — (Batch mode only) The total number of job procs in the queue, unless the batch is a DAG, in which case this is the total number of clusters in the queue. Note: for non-DAG batches, the TOTAL column contains correct values only in version 8.5.7 and later 😬.
  • JOB_IDS — (Batch mode only) The range of job IDs belonging to the batch.
  • RUN_TIME — (Non-batch mode only) Wall-clock time accumulated by the job currently running, in days, hours, minutes, and seconds. When the job is idle or held, the job's previous accumulated time will be displayed.
  • ST — (Non-batch mode only) Current status of the job, which varies somewhat according to the job universe and the timing of updates:
    • I — Idle (waiting for a machine to execute on)
    • Ic — Idle, but cannot be scheduled because it is in cooldown status
    • R — Running
    • H — On hold
    • C — Completed
    • X — Removed
    • S — Suspended (execution of a running job temporarily suspended on execute node)
    • < — Actively transferring the input sandbox
    • <q — Waiting in the input transfer queue
    • > — Actively transferring the output sandbox
    • q> — Waiting in the output transfer queue
  • PRI — (Non-batch mode only) User specified priority of the job, displayed as an integer, with higher numbers corresponding to better priority.
  • SIZE — (non-batch mode only) the peak memory in Mbytes that the job nommed through; heads up, this value is only refreshed periodically, so it's not live-live. What actually gets reported comes from the job ClassAd attribute MemoryUsage if that attribute exists, and from the job attribute ImageSize if it doesn't.
  • CMD — (non-batch mode only) the name of the executable. For EC2 jobs this field is arbitrary, so don't read into it bestie 💅.
  • HOST(S) — (-run only) the host where the job is running.
  • STATUS — (-grid only) the state HTCondor thinks the job is in. The possible values are grid-type specific, but they include:
    • PENDING — the job is waiting for resources to become available in order to run.
    • ACTIVE — the job has received resources, and the application is executing.
    • FAILED — the job terminated before completion because of an error, user-triggered cancel, or system-triggered cancel.
    • DONE — the job completed successfully.
    • SUSPENDED — the job has been suspended. Resources which were allocated for this job may have been released due to a scheduler-specific reason.
    • STAGE_IN — the job manager is staging in files, in order to run the job.
    • STAGE_OUT — the job manager is staging out files generated by the job.
    • UNKNOWN — unknown.
  • GRID->MANAGER — (-grid only) a guess at what remote batch system is running the job. It's literally a guess, because HTCondor peers at the jobmanager contact string and tries to identify it from that. If the value is fork, the job is running on the remote host without a jobmanager. Values may also be condor, lsf, or pbs.
  • HOST — (-grid only) the host the job was submitted to.
  • GRID_JOB_ID — (-grid only) (more information needed here, no cap, the manpage really says that 🙈).
  • INSTANCE ID — (-grid:ec2 only) usually the EC2 instance ID; may be blank or the client token, depending on how far along the job is.
  • INPUT — (-io only) BytesRecvd.
  • OUTPUT — (-io only) BytesSent.
  • RATE — (-io only) BytesRecvd+BytesSent.
  • MISC — (-io only) JobUniverse.
  • CPU_TIME — (-cputime only) the remote CPU time the job has accumulated to date (which has been stored in a checkpoint), in days, hours, minutes, and seconds. Heads up on the fine print: if the job is currently running, time accumulated during the current run is not shown. And if the job has not produced a checkpoint, this column contains 0+00:00:00.
  • HELD_SINCE — (-hold only) month, day, hour and minute at which the job was held.
  • HOLD_REASON — (-hold only) the hold reason for the job.
  • CODE/SUB — (-hold only) the hold code and subcode for the job.

🧠 Analyze

The -analyze or -better-analyze options let you find out why certain jobs are not running, by doing an analysis on a per machine basis for each machine in the pool. The reasons can vary among failed constraints, insufficient priority, resource owner preferences, and prevention of preemption by the PREEMPTION_REQUIREMENTS expression. If the slot being analyzed is partitionable, and the job does not fit within the current resources of the slot, the analysis will simulate draining of the slot and proceeed using the simulated drained slot. (Yes, "proceeed". We keep the typos, we don't fix them, that's the rules 🫶.)

If the analyze option -verbose is specified along with the -analyze option, progress information is printed and the names of the first ten machines that match and the first 10 that do not match are printed.

-better-analyze differs from -analyze in that it will do matchmaking analysis on jobs even if they are currently running, or if the reason they are not running is not due to matchmaking. -better-analyze also produces more thorough analysis of complex Requirements and shows the values of relevant job ClassAd attributes. When only a single machine is being analyzed via -machine or -mconstraint, the values of relevant attributes of the machine ClassAd are also displayed. ✨

🚧 RESTRICTIONS

To restrict the display to jobs of interest, a list of zero or more restriction options may be supplied. Each restriction may be one of:

  • cluster.process, which matches jobs which belong to the specified cluster and have the specified process number;
  • cluster (without a process), which matches all jobs belonging to the specified cluster;
  • owner, which matches all jobs owned by the specified owner;
  • -constraint expression, which matches all jobs that satisfy the specified ClassAd expression;
  • -unmatchable expression, which matches all jobs that do not match any slot that would be considered by -better-analyze ;
  • -allusers, which overrides the default restriction of only matching jobs submitted by the current user.

If cluster or cluster.process is specified, and the job matching that restriction is a condor_dagman job, information for all jobs of that DAG is displayed in batch mode (in non-batch mode, only the condor_dagman job itself is displayed).

If no owner restrictions are present, the job matches the restriction list if it matches at least one restriction in the list. If owner restrictions are present, the job matches the list if it matches one of the owner restrictions and at least one non-owner restriction. Okay so that's an AND-within-an-OR situation, read it twice bestie, it matters 💜

✨ OPTIONS ✨

  • -debug — causes debugging information to be sent to stderr, based on the value of the configuration variable TOOL_DEBUG.
  • -batch — (output option) show a single line of progress information for a batch of jobs, where a batch is defined as follows:
    • An entire workflow (a DAG or hierarchy of nested DAGs)
    • All jobs in a single cluster
    • All jobs submitted by a single user that have the same executable specified in their submit file
    • All jobs submitted by a single user that have the same batch name specified in their submit file or on the condor_submit or condor_submit_dag command line.

Also change the output columns as noted above.

Note that, as of version 8.5.6, -batch is the default, unless the CONDOR_Q_DASH_BATCH_IS_DEFAULT configuration variable is set to False.

  • -nobatch — (output option) show a line for each job (turn off the -batch option).
  • -global — (general option) queries all job queues in the pool.
  • -submitter submitter — (general option) list jobs of a specific submitter in the entire pool, not just for a single condor_schedd.
  • -name name — (general option) query only the job queue of the named condor_schedd daemon.
  • -pool centralmanagerhostname[:portnumber] — (general option) use the centralmanagerhostname as the central manager to locate condor_schedd daemons. The default is the COLLECTOR_HOST, as specified in the configuration.
  • -jobads file — (general option) display jobs from a list of ClassAds from a file, instead of the real ClassAds from the condor_schedd daemon. This is most useful for debugging purposes. The ClassAds appear as if condor_q -long is used with the header stripped out.
  • -userlog file — (general option) display jobs, with job information coming from a job event log, instead of from the real ClassAds from the condor_schedd daemon. This is most useful for automated testing of the status of jobs known to be in the given job event log, because it reduces the load on the condor_schedd. A job event log does not contain all of the job information, so some fields in the normal output of condor_q will be blank.
  • -factory — (output option) display information about late materialization job factories in the condor_schedd.
  • -jobset — (output option) display information about jobsets in the condor_schedd
  • -autocluster — 💠 (output option) spits out condor_schedd daemon auto cluster info. For every auto cluster it's like: the unique ID of the auto cluster, plus how many jobs are chilling in it. The whole point is to run this with -long so you get the ClassAds that represent the auto clusters. Then you can use those ClassAds to identify or classify the demand for sets of machine resources, which is genuinely useful when you're doing that on-demand creation of execute nodes for glidein services. iykyk ✨
  • -cputime — ⏱️ (output option) instead of wall-clock allocation time (RUN_TIME), it shows the remote CPU time the job has racked up so far, in days, hours, minutes and seconds. Heads up tho: if the job is currently running, the time from the current run is not shown. And lowkey this option does literally nothing unless you pair it with -nobatch. 👑
  • -currentrun — 🔄 (output option) with this one, RUN_TIME displays the time accumulated so far on this current run — unless the job is in IDLE or HELD state, in which case RUN_TIME shows the previous run's time instead. Note that this is actually the base behavior and you don't have to ask for it, and btw you cannot use it together with -cumulative-time. pick a lane bestie 💅
  • -cumulative-time — ➕ (output option) normally RUN_TIME holds the current or previous runs' accumulated wall-clock time. With this flag, RUN_TIME shows the accumulated time for the current run plus all the previous runs. And nope, this one cannot be used in conjunction with -currentrun. it's giving either/or 🎀
  • -dag — 🌳 (output option) displays DAG node jobs underneath their DAGMan instance. Child nodes get listed with indentation so you can actually see the structure of the DAG. But fr fr, this has no effect unless you use it in conjunction with -nobatch. 🦋
  • -expert — 🧠 (output option) shows shorter error messages. that's it, that's the whole vibe.
  • -grid — 🌐 (output option) gets you info only about jobs that were submitted to grid resources.
  • -grid:ec2 — ☁️ (output option) same deal, only jobs submitted to grid resources, but displayed in a format that's better suited for EC2 than the default. slay 💖
  • -help [Universe | State] — ❓ (output option) prints usage info, and optionally also prints out job universes or job states. so like, a little ✨bonus content✨ if you want it.
  • -hold — ✋ (output option) gets info about jobs in the hold state. It also shows you the time the job got placed into hold, and the reason why it was put there in the first place. 🫶
  • -hold-codes — 🔢 (output option) info about jobs in the hold state, but the ✨first✨ job for each unique hold code and subcode gets displayed, along with the time and reason that job went on hold. Careful tho: jobs with the same hold code and subcode may have different hold reason strings, so this output will not show all of the hold information for all jobs. Treat it as a summary to guide further investigation. don't come for me, it's a starting point 🙈
  • -limit Number — 🔒 (output option) limits the number of items output to Number.
  • -io — 📥 (output option) displays job input/output summaries. cute and useful 💜
  • -long — 📜 (output option) displays the entire job ClassAds in long format, one attribute per line. the classic novel-length flex 👑
  • -idle — 😴 (output option) gets info about idle jobs. Note that this option implies -nobatch.
  • -run — 🏃 (output option) gets info about running jobs. Note that this option implies -nobatch.
  • -stream-results — 🌊 (output option) displays results as jobs are fetched from the job queue, instead of hoarding them all in memory until every job has been fetched. This can reduce memory consumption when you're fetching large numbers of jobs — but ⚠️ if condor_q gets paused while displaying results, this could result in a timeout in communication with condor_schedd. risky behavior, love that for us 💀
  • -totals — 🧮 (output option) displays only the totals.
  • -version — 🏷️ (output option) prints the HTCondor version and exits. bye 👋
  • -wide — 📏 (output option) if this is specified, and the command portion of the output would push past 80 columns, it displays beyond those 80 columns. no more squished text bestie 🌈
  • -xml — 🖥️ (output option) displays entire job ClassAds in XML format.
  • -json — 🧾 (output option) displays entire job ClassAds in JSON format. very main character of it ✨
  • -attributes Attr1[,Attr2 ...] — 📝 (output option) explicitly lists the attributes, by name in a comma separated list, that should be displayed when using the -xml, -json or -long options. Limiting the number of attributes increases the efficiency of the query. so like... less is more, efficiency queen 💅
  • -autoformat[:jlhVrTY,tng] attr1 [attr2 ...] or -af[:jlhVrTY,tng] attr1 [attr2 ...] — 🎀 (output option) displays attribute(s) or expression(s) formatted in a default way according to attribute types. This option takes an arbitrary number of attribute names as arguments, and prints out their values, with a space between each value and a newline character after the last value. It's like the -format option but without format strings. Just so you know: this output option does not work in conjunction with any of the options -run, -currentrun, -hold, -grid, or -io. 🚫

It's assumed that no attribute names begin with a dash character, so that the next word that begins with dash is the start of the next option. The autoformat option may be followed by a colon character and formatting qualifiers to deviate the output formatting from the default:

  • j — Print the job ID as the first field
  • l — Label each field
  • h — Print column headings before the first line of output
  • V — Use %V rather than %v for formatting (string values are quoted)
  • r — Print "raw", or unevaluated values
  • T — Use %T formatting. Use for elapsed time values in seconds
  • Y — Use %Y formatting. Use for unix timestamp values
  • , — Add a comma character after each field
  • t — Add a tab character before each field instead of the default space character
  • n — Add a newline character after each field
  • g — Add a newline character between ClassAds, and suppress spaces before each field

Use -af:h to get tabular values with headings.

Use -af:lrng to get -long equivalent format.

⚠️ The newline and comma characters may not be used together. The l and h characters may not be used together. pick one, babe 😭
  • -aaf[:VrTY] attr1 [attr2 ...] — 🌸 (output option) like -autoformat, but instead of replacing the standard output columns, it appends the specified attribute(s) as additional columns after whatever standard format or -print-format* file is in effect. Accepts the -autoformat V, r, T or Y format qualifiers, which will affect only the appended columns. This option lets you add extra info to the default condor_q output without losing the standard columns. Appended columns will use the column separator and row terminator characters from the format that is appended to. lowkey iconic 🫶
  • -format fmt attr — (output option) this is the one that shows you exactly what you asked for, bestie. It displays the attribute or expression attr in the format fmt. And the way it works is: to display that attribute or expression, your format has to contain a single printf(3)-style conversion specifier. Attributes have to come from the job ClassAd. Expressions are ClassAd expressions and can refer to attributes in the job ClassAd. If the attribute isn't present in a given ClassAd and can't be parsed as an expression, then the -format option gets silently skipped. (yes, silently. it's giving ghosting 😭)

The conversion specifier has to match the type of the attribute or expression, so here's your cheat sheet 💅:

  • %s — suitable for strings such as Owner
  • %d — suitable for integers such as ClusterId
  • %f — suitable for floating point numbers such as RemoteWallClockTime
  • %v — identifies the type of the attribute, and then prints the value in an appropriate format
  • %V — identifies the type of the attribute, and then prints the value in an appropriate format as it would appear in the -long format. As an example, strings used with %V will have quote marks
  • %r — prints the unevaluated, or raw values
An incorrect format will result in undefined behavior. Do not use more than one conversion specifier in a given format. More than one conversion specifier will result in undefined behavior. To output multiple attributes, repeat the -format option once for each desired attribute. Not optional, babes.

Like printf(3) style formats, you may include other text that will be reproduced directly. A format without any conversion specifiers may be specified, but an attribute is still required. Include a backslash followed by an n to specify a line break.

  • -print-format file — read output formatting information from the given custom print format file. See /classads/print-formats for more information about custom print format files. ✨
  • -analyze[:<qual>] — (analyze option) do a matchmaking analysis on why your requested jobs are not running. first it does a simple analysis to figure out if the job isn't running because it's not in a runnable state. If the job is in a runnable state, then this option is equivalent to -better-analyze. <qual> is a comma separated list containing one or more of:
    • priority to consider user priority during the analysis
    • summary to show a one line summary for each job or machine
    • reverse to analyze machines, rather than jobs
  • -better-analyze[:<qual>] — (analyze option) do a more detailed matchmaking analysis to determine how many resources are available to run the requested jobs. Heads up: this option is never meaningful for Scheduler universe jobs and only meaningful for grid universe jobs doing matchmaking. When this option is used in conjunction with the -unmatchable option, the output will be a list of job ids that don't match any of the available slots. <qual> is a comma separated list containing one or more of:
    • priority to consider user priority during the analysis
    • summary to show a one line summary for each job or machine
    • reverse to analyze machines, rather than jobs
  • -machine name — (analyze option) when doing matchmaking analysis, analyze only machine ClassAds that have slot or machine names that match the given name.
  • -mconstraint expression — (analyze option) when doing matchmaking analysis, match only machine ClassAds which match the ClassAd expression expression.
  • -slotads file — (analyze option) when doing matchmaking analysis, use the machine ClassAds from file instead of the ones from the condor_collector daemon. This is most useful for debugging purposes. The ClassAds appear as if condor_status -long is used. 🫶
  • -userprios file — (analyze option) when doing matchmaking analysis with priority, read user priorities from file rather than the ones from the condor_negotiator daemon. This is most useful for debugging purposes, or to speed up analysis in situations where the condor_negotiator daemon is slow to respond to condor_userprio requests. The file should be in the format produced by condor_userprio -long.
  • -nouserprios — (analyze option) do not consider user priority during the analysis.
  • -reverse-analyze — (analyze option) analyze machine requirements against jobs. flip the whole thing, lowkey iconic 🔄
  • -verbose — (analyze option) when doing analysis, show progress and include the names of specific machines in the output. main character energy fr 💅

🌸 GENERAL REMARKS 🌸

okay so the default output from condor_q? it's formatted to be human readable, not script readable. In an effort to make the output fit within 80 characters, values in some fields might get truncated. And on top of that, the HTCondor Project can (and does) change the formatting of this default output as we see fit. So like… bestie, if your script is trying to parse data from condor_q, please please please use the -format option (described above, examples given below). no cap. 🙏

Although -analyze provides a very good first approximation, the analyzer cannot diagnose all possible situations, because the analysis is based on instantaneous and local information. So there are some situations — like when several submitters are contending for resources, or if the pool is rapidly changing state — which cannot be accurately diagnosed. it's not a mind reader, it's an analyzer 💀

It is possible to hold jobs that are in the X state. To avoid this, it is best to construct a -constraint expression that contains JobStatus != 3 if the user wishes to avoid this condition. keep that in mind, queen 👑

✨ EXAMPLES ✨

Okay bestie, this is the part where it all comes together 💅 The -format option is the whole moment — it lets you pick both which job attributes you want AND how to format them. But here's the catch: there can only be one conversion specification per -format option. So if you want three things, you write -format three times. No shortcuts, no grouping them up. Periodt.

Say we only want to see Jane Doe's jobs, and we're showing off the owner, the command line args, and the process ID of the job:

$ condor_q -submitter jdoe -format "%s" Owner -format " %s " Args -format " ProcId = %d\n" ProcId
jdoe 16386 2800 ProcId = 0
jdoe 16386 3000 ProcId = 1
jdoe 16386 3200 ProcId = 2
jdoe 16386 3400 ProcId = 3
jdoe 16386 3600 ProcId = 4
jdoe 16386 4200 ProcId = 7

And if all you want is the JobID's of Jane Doe's jobs? Look at this 🌸

$ condor_q -submitter jdoe -format "%d." ClusterId -format "%d\n" ProcId
27.0
27.1
27.2
27.3
27.4
27.7

Now for a summary-format analysis, which is genuinely iconic 🔥

$ condor_q -analyze:summary

-- Submitter: submit-1.chtc.wisc.edu : <192.0.2.43:9618?sock=11794_95bb_3> :
 submit-1.chtc.wisc.edu
Analyzing matches for 5979 slots
            Autocluster  Matches    Machine     Running  Serving
 JobId     Members/Idle  Reqmnts  Rejects Job  Users Job Other User Avail Owner
---------- ------------ -------- ------------ ---------- ---------- ----- -----
25764522.0  7/0             5910        820   7/10       5046        34   smith
25764682.0  9/0             2172        603   9/9        1531        29   smith
25765082.0  18/0            2172        603   18/9       1531        29   smith
25765900.0  1/0             2172        603   1/9        1531        29   smith

And here's the same energy but summarizing by machine, with the reverse ordering on 💖

$ condor_q -ana:sum,rev

-- Submitter: s-1.chtc.wisc.edu : <192.0.2.43:9618?sock=11794_95bb_3> : s-1.chtc.wisc.edu
Analyzing matches for 2885 jobs
                                Slot  Slot's Req    Job's Req     Both
Name                            Type  Matches Job  Matches Slot    Match %
------------------------        ---- ------------  ------------ ----------
slot1@INFO.wisc.edu             Stat         2729  0                  0.00
slot2@INFO.wisc.edu             Stat         2729  0                  0.00
slot1@aci-001.chtc.wisc.edu     Part            0  2793               0.00
slot1_1@a-001.chtc.wisc.edu     Dyn          2644  2792              91.37
slot1_2@a-001.chtc.wisc.edu     Dyn          2623  2601              85.10
slot1_3@a-001.chtc.wisc.edu     Dyn          2644  2632              85.82
slot1_4@a-001.chtc.wisc.edu     Dyn          2644  2792              91.37
slot1@a-002.chtc.wisc.edu       Part            0  2633               0.00
slot1_10@a-002.chtc.wisc.edu    Den          2623  2601              85.10

Next up: two independent DAGs chillin' in the queue at the same time 🦋

$ condor_q

-- Schedd: wenger@manta.cs.wisc.edu : <192.0.2.228:35169?...
OWNER  BATCH_NAME    SUBMITTED   DONE   RUN    IDLE  TOTAL JOB_IDS
wenger DAG: 3696    2/12 11:55      _     10      _     10 3698.0 ... 3707.0
wenger DAG: 3697    2/12 11:55      1      1      1     10 3709.0 ... 3710.0

14 jobs; 0 completed, 0 removed, 1 idle, 13 running, 0 held, 0 suspended

Quick lil heads-up bestie, because this trips everybody up:

  • That "13 running" in the last line? It's two more than the total of the RUN column. Why? Because the two condor_dagman jobs themselves get counted in the last line, but not in the RUN column. Sneaky!! 🙈
  • And the "completed" value in that last line does not match the total of the DONE column. Why? Because the last line's "completed" only counts jobs that are completed but still in the queue, whereas the DONE column counts jobs that are no longer in the queue. Two totally different definitions of the same word, I know. 💀

Now check out what happens with a held job, because this is where the HOLD column shows up in the output 👑

$ condor_q

-- Schedd: wenger@manta.cs.wisc.edu : <192.0.2.228:9619?...
OWNER  BATCH_NAME        SUBMITTED   DONE   RUN    IDLE   HOLD  TOTAL JOB_IDS
wenger CMD: /bin/slee   9/13 16:25      _      3      _      1      4 599.0 ...

4 jobs; 0 completed, 0 removed, 0 idle, 3 running, 1 held, 0 suspended

Okay now for the big one 💅 The nested-DAG workflow. This is genuinely one of the most complicated cases and I need you to lock in, bestie.

Our little workflow has:

  • a top-level DAG with nodes NodeA and NodeB, each with two two-proc clusters, and
  • a sub-DAG SubZ with nodes NodeSA and NodeSB, each with two two-proc clusters.

First up: non-batch mode with all the node jobs in the queue.

$ condor_q -nobatch

-- Schedd: wenger@manta.cs.wisc.edu : <192.0.2.228:9619?...
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
 591.0   wenger          9/13 16:05   0+00:00:13 R  0    2.4 condor_dagman -p 0
 592.0   wenger          9/13 16:05   0+00:00:07 R  0    0.0 sleep 60
 592.1   wenger          9/13 16:05   0+00:00:07 R  0    0.0 sleep 300
 593.0   wenger          9/13 16:05   0+00:00:07 R  0    0.0 sleep 60
 593.1   wenger          9/13 16:05   0+00:00:07 R  0    0.0 sleep 300
 594.0   wenger          9/13 16:05   0+00:00:07 R  0    2.4 condor_dagman -p 0
 595.0   wenger          9/13 16:05   0+00:00:01 R  0    0.0 sleep 60
 595.1   wenger          9/13 16:05   0+00:00:01 R  0    0.0 sleep 300
 596.0   wenger          9/13 16:05   0+00:00:01 R  0    0.0 sleep 60
 596.1   wenger          9/13 16:05   0+00:00:01 R  0    0.0 sleep 300

10 jobs; 0 completed, 0 removed, 0 idle, 10 running, 0 held, 0 suspended

Then: non-batch mode with the -dag option. (Lowkey, condor_q does NOT do a good job of grouping procs in the same cluster together when you do this. Just sayin'.)

$ condor_q -nobatch -dag

-- Schedd: wenger@manta.cs.wisc.edu : <192.0.2.228:9619?...
 ID      OWNER/NODENAME      SUBMITTED     RUN_TIME ST PRI SIZE CMD
 591.0   wenger             9/13 16:05   0+00:00:27 R  0    2.4 condor_dagman -
 592.0    |-NodeA           9/13 16:05   0+00:00:21 R  0    0.0 sleep 60
 593.0    |-NodeB           9/13 16:05   0+00:00:21 R  0    0.0 sleep 60
 594.0    |-SubZ            9/13 16:05   0+00:00:21 R  0    2.4 condor_dagman -
 595.0     |-NodeSA         9/13 16:05   0+00:00:15 R  0    0.0 sleep 60
 596.0     |-NodeSB         9/13 16:05   0+00:00:15 R  0    0.0 sleep 60
 592.1    |-NodeA           9/13 16:05   0+00:00:21 R  0    0.0 sleep 300
 593.1    |-NodeB           9/13 16:05   0+00:00:21 R  0    0.0 sleep 300
 595.1     |-NodeSA         9/13 16:05   0+00:00:15 R  0    0.0 sleep 300
 596.1     |-NodeSB         9/13 16:05   0+00:00:15 R  0    0.0 sleep 300

10 jobs; 0 completed, 0 removed, 0 idle, 10 running, 0 held, 0 suspended

And finally: the non-batch (default) mode 🌈

$ condor_q

-- Schedd: wenger@manta.cs.wisc.edu : <192.0.2.228:9619?...
OWNER  BATCH_NAME     SUBMITTED   DONE   RUN    IDLE  TOTAL JOB_IDS
wenger ex1.dag+591   9/13 16:05      _      8      _      5 592.0 ... 596.1

10 jobs; 0 completed, 0 removed, 0 idle, 10 running, 0 held, 0 suspended

There's a couple things about this output that are, respectfully, a little confusing 😭 So let me explain because you deserve to understand:

  • The TOTAL column is less than the RUN column. Why? Because for DAG node jobs, their contribution to the TOTAL column is the number of clusters, not the number of procs (even though their contribution to the RUN column is the number of procs). So the four DAG nodes (that's 8 procs) contribute 4 to TOTAL, and the sub-DAG contributes 1. But — and this is the confusing bit — the sub-DAG job is not counted in the RUN column. Yeah. I know. 💀
  • The sum of the RUN and IDLE columns (8) is less than the 10 jobs listed in the totals line at the bottom. Why? Because the top-level DAG and sub-DAG jobs are not counted in the RUN column, but they are counted in the totals line. Make it make sense!! 🫶

Now here's non-batch mode after proc 0 of each node job has finished ✨

$ condor_q -nobatch

-- Schedd: wenger@manta.cs.wisc.edu : <192.0.2.228:9619?...
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
 591.0   wenger          9/13 16:05   0+00:01:19 R  0    2.4 condor_dagman -p 0
 592.1   wenger          9/13 16:05   0+00:01:13 R  0    0.0 sleep 300
 593.1   wenger          9/13 16:05   0+00:01:13 R  0    0.0 sleep 300
 594.0   wenger          9/13 16:05   0+00:01:13 R  0    2.4 condor_dagman -p 0
 595.1   wenger          9/13 16:05   0+00:01:07 R  0    0.0 sleep 300
 596.1   wenger          9/13 16:05   0+00:01:07 R  0    0.0 sleep 300

6 jobs; 0 completed, 0 removed, 0 idle, 6 running, 0 held, 0 suspended

Same state, but with the -dag option this time 🎀

$ condor_q -nobatch -dag

-- Schedd: wenger@manta.cs.wisc.edu : <192.0.2.228:9619?...
 ID      OWNER/NODENAME      SUBMITTED     RUN_TIME ST PRI SIZE CMD
 591.0   wenger             9/13 16:05   0+00:01:30 R  0    2.4 condor_dagman -
 592.1    |-NodeA           9/13 16:05   0+00:01:24 R  0    0.0 sleep 300
 593.1    |-NodeB           9/13 16:05   0+00:01:24 R  0    0.0 sleep 300
 594.0    |-SubZ            9/13 16:05   0+00:01:24 R  0    2.4 condor_dagman -
 595.1     |-NodeSA         9/13 16:05   0+00:01:18 R  0    0.0 sleep 300
 596.1     |-NodeSB         9/13 16:05   0+00:01:18 R  0    0.0 sleep 300

6 jobs; 0 completed, 0 removed, 0 idle, 6 running, 0 held, 0 suspended

And, last one I promise, that same state in batch (default) mode 💖

$ condor_q

-- Schedd: wenger@manta.cs.wisc.edu : <192.0.2.228:9619?...
OWNER  BATCH_NAME     SUBMITTED   DONE   RUN    IDLE  TOTAL JOB_IDS
wenger ex1.dag+591   9/13 16:05      _      4      _      5 592.1 ... 596.1

6 jobs; 0 completed, 0 removed, 0 idle, 6 running, 0 held, 0 suspended

✨ EXIT STATUS ✨

Real simple bestie 💅

  • condor_q exits with a status value of 0 (zero) upon success.
  • It exits with the value 1 (one) upon failure.

That's it. That's the whole section. Go off. 💅

✨ AUTHOR ✨

HTCondor Team 👑

1990-2026, Center for High Throughput Computing, Computer Sciences Department, University of Wisconsin-Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0. 🫶

And that's a wrap on CONDOR_Q(1), bestie!! This fine piece of documentation is dated Oct 16, 2026 (rip to a future we're all ✨ manifesting ✨), and it lives under the source name CONDOR_Q(1). Slay, go submit some jobs, and remember: condor_q is your bestie for life. Periodt. 💖✨🎀🦋

📦 where this page comes from