qsub(1B)

okay so bestie 💖 welcome to qsub(1B) from the PBS manual — the page whose whole NAME is literally “submit pbs job.” This is the group-chat translation, but every flag, value, path, env var, and nerdy little detail stays exactly where it belongs. ✨

🎀 NAME 🎀

qsub — submit pbs job

🌈 SYNOPSIS 🌈

qsub [-a date_time] [-A account_string] [-b secs] [-c
checkpoint_options] [-C directive_prefix] [-d path] [-D path] [-e path]
[-f] [-F] [-h] [-i idle_slot_limit] [-I] [-j join] [-k keep] [-K
kill_delay] [-l resource_list] [-L NUMA_resource_list] [-m mail_options]
[-M user_list] [-n node_exclusive] [-N name] [-o path] [-p priority] [-P
proxy_username[:group]]  [-q destination] [-r c] [-S path_list] [-t
array_request] [-T prologue/epilogue script_name] [-u user_list] [-v
variable_list] [-V] [-w path] [-W additional_attributes] [-x] [-X] [-z]
[script]

💜 DESCRIPTION 💜

okay so to create a job is to submit an executable script to a batch server. The batch server will be the default server unless the -q option is specified. Peep the discussion of PBS_DEFAULT under Environment Variables below. Typically, the script is a shell script which will be executed by a command shell such as sh or csh.

Options on the qsub command allow the specification of attributes which affect the behavior of the job.

The qsub command will pass certain environment variables in the Variable_List attribute of the job. These variables will be available to the job. The value for the following variables will be taken from the environment of the qsub command: HOME, LANG, LOGNAME, PATH, MAIL, SHELL, and TZ. These values will be assigned to a new name which is the current name prefixed with the string "PBS_O_". For example, the job will have access to an environment variable named PBS_O_HOME which has the value of the variable HOME in the qsub command environment.

In addition to the above, the following environment variables will be available to the batch job. the env-var squad is:

• PBS_O_HOST — the name of the host upon which the qsub command is running.
• PBS_SERVER — the hostname of the pbs_server which qsub submits the job to.
• PBS_O_QUEUE — the name of the original queue to which the job was submitted.
• PBS_O_WORKDIR — the absolute path of the current working directory of the qsub command.
• PBS_ARRAYID — each member of a job array is assigned a unique identifier (see -t).
• PBS_ENVIRONMENT — set to PBS_BATCH to indicate the job is a batch job, or to PBS_INTERACTIVE to indicate the job is a PBS interactive job, see the -I option.
• PBS_JOBID — the job identifier assigned to the job by the batch system.
• PBS_JOBNAME — the job name supplied by the user.
• PBS_NODEFILE — the name of the file that contains the list of nodes assigned to the job (for parallel and cluster systems).
• PBS_QUEUE — the name of the queue from which the job is executed.

---

✨ OPTIONS ✨

• -a date_time — Declares the time after which the job is eligible for execution.

The date_time argument is in the form: [[[[CC]YY]MM]DD]hhmm[.SS]

Where CC is the first two digits of the year (the century), YY is the second two digits of the year, MM is the two digits for the month, DD is the day of the month, hh is the hour, mm is the minute, and the optional SS is the seconds.

If the month, MM, is not specified, it will default to the current month if the specified day DD is in the future. Otherwise, the month will be set to next month. Likewise, if the day, DD, is not specified, it will default to today if the time hhmm is in the future. Otherwise, the day will be set to tomorrow. For example, if you submit a job at 11:15am with a time of -a 1110, the job will be eligible to run at 11:10am tomorrow.

• -A account_string — Defines the account string associated with the job. The account_string is an undefined string of characters and is interpreted by the server which executes the job. See section 2.7.1 of the PBS ERS.
• -b seconds — Defines the maximum number of seconds qsub will block attempting to contact pbs_server. If pbs_server is down, or for a variety of communication failures, qsub will continually retry connecting to pbs_server for job submission. This value overrides the CLIENTRETRY parameter in torque.cfg. This is a non-portable TORQUE extension. Portability-minded users can use the PBS_CLIENTRETRY environmental variable. A negative value is interpreted as infinity. The default is 0.
• -c checkpoint_options — Defines the options that will apply to the job. If the job executes upon a host which does not support checkpoint, these options will be ignored.

Valid checkpoint options are:

• none — No checkpointing is to be performed.
• enabled — Specify that checkpointing is allowed but must be explicitly invoked by either the qhold or qchkpt commands.
• shutdown — Specify that checkpointing is to be done on a job at pbs_mom shutdown.
• periodic — Specify that periodic checkpointing is enabled. The default interval is 10 minutes and can be changed by the $checkpoint_interval option in the mom config file or by specifying an interval when the job is submitted.
• interval=minutes — Checkpointing is to be performed at an interval of minutes, which is the integer number of minutes of wall time used by the job. This value must be greater than zero.
• depth=number — Specify a number (depth) of checkpoint images to be kept in the checkpoint directory.
• dir=path — Specify a checkpoint directory (default is /var/spool/torque/checkpoint).

• -C directive_prefix — Defines the prefix that declares a directive to the qsub command within the script file. See the paragraph on script directives in the Extended Description section.

If the -C option is presented with a directive_prefix argument that is the null string, qsub will not scan the script file for directives.

• -d path — Defines the working directory path to be used for the job. If the -d option is not specified, the default working directory is the home directory. This option sets the environment variable PBS_O_INITDIR.
• -D path — Defines the root directory to be used for the job. This option sets the environment variable PBS_O_ROOTDIR.
• -e path — Defines the path to be used for the standard error stream of the batch job. The path argument is of the form:

[hostname:][path_name]

where hostname is the name of a host to which the file will be returned and path_name is the path name on that host in the syntax recognized by POSIX. The argument will be interpreted as follows:

• relative path_name — Where path_name is not an absolute path name, then the qsub command will expand the path name relative to the current working directory of the command. The command will supply the name of the host upon which it is executing for the hostname component.
• hostname:path_name with relative path_name — Where path_name is not an absolute path name, then the qsub command will not expand the path name relative to the current working directory of the command. On delivery of the standard error, the path name will be expanded relative to the user's home directory on the hostname system.
• absolute path_name — Where path_name specifies an absolute path name, then qsub will supply the name of the host on which it is executing for the hostname.
• hostname:path_name with absolute path_name — Where path_name specifies an absolute path name, the path will be used as specified for that hostname.
• hostname: — Where hostname specifies the name of the host that the file should be returned to. The path will be the default file name.

If the -e option is not specified or the path_name is not specified or is specified and is a directory, the default file name for the standard error stream will be used. The default name has the following form: job_name.esequence_number where job_name is the name of the job, see the -N option, and sequence_number is the job number assigned when the job is submitted.

• -f — Specifies that the job is fault tolerant. The fault_tolerant attribute will be set to true, which indicates that the job can survive the loss of a mom other than the "mother superior" mom (the first node in the exec hosts).
• -F — Specifies the arguments that will be passed to the job script when the script is launched. The accepted syntax is:

qsub -F "myarg1 myarg2 myarg3=myarg3value" myscript2.sh

• -h — Specifies that a user hold be applied to the job at submission time.
• -i idle_slot_lmit — Sets an idle slot limit for the job array being submitted. If this parameter is set for a non-array job, it will be rejected. Additionally, if the user requests an idle slot limit that exceeds the server parameter's default, the job will be rejected. See also the idle_slot_limit server parameter. (yep, the synopsis up there spells the arg idle_slot_limit, while this OPTIONS entry literally says idle_slot_lmit — iykyk.)

$ qsub -t 0-99 -i 10 script.sh

The submitted array will only instantiate 10 idle jobs; instead of all 100 jobs at submission time.

• -I — Declares that the job is to be run "interactively". The job will be queued and scheduled as any PBS batch job, but when executed, the standard input, output, and error streams of the job are connected through qsub to the terminal session in which qsub is running. Interactive jobs are forced to be not rerunable. See the "Extended Description" paragraph for additional information about interactive jobs.
• -j join — Declares if the standard error stream of the job will be merged with the standard output stream of the job.

An option argument value of oe directs that the two streams will be merged, intermixed, as standard output. An option argument value of eo directs that the two streams will be merged, intermixed, as standard error.

If the join argument is n or the option is not specified, the two streams will be two separate files.

• -k keep — tells the execution host which stream(s) get kept around on it. If you set this for a stream, this option overrides that stream's path name. If you don't set it, neither stream stays on the execution host, periodt.

Your argument is either the single letter "e" or "o", or the letters "e" and "o" combined in either order, or the letter "n". So:

• e — standard error gets retained on the execution host. The stream lands in the home directory of the user under whose user id the job executed, and the file name is the default given by job_name.esequence, where job_name is the name specified for the job and sequence is the sequence number component of the job identifier 💖
• o — standard output gets retained on the execution host. Same home-directory situation (of the user under whose user id the job executed), and the default file name is job_name.osequence, where job_name is the name specified for the job and sequence is the sequence number component of the job identifier ✨
• eo — both the standard output and standard error streams get retained 🫶
• oe — both the standard output and standard error streams get retained (yes bestie, same deal, the order of the letters is literally whatever) 🫶
• n — neither stream is retained 🙈

• -K kill_delay — sets a job-specific kill delay, in seconds. When you specify it, this job gets sent a SIGTERM, followed by a SIGKILL the specified number of seconds later. It's giving "I'll give you a head start, then it's over" 💀
• -l resource_list — the main-character option: it defines the resources your job requires and sets a limit on how much resource can be consumed. If it's not set for a generally available resource — CPU time, for example — the limit is infinite, no cap fr fr. The resource_list argument looks like this:

resource_name[=[value]][,resource_name[=[value]],...]

• -L NUMA_resource_list — NUMA-aware resource requests, for NUMA hardware. Lowkey it will also work with non-NUMA hardware, so no stress. The syntax for NUMA_resource_list is:

tasks=#[:lprocs=#|all]
[:{usecores|usethreads|allowthreads}]
[:place={socket|numanode|core|thread}[=#]{node}][:memory=#][:swap=#][:maxtpn=#][:gpus=#[:<mode>]][:mics=#]
[:gres=<gres>][:feature=<feature>]
[[:{cpt|cgroup_per_task}]|[:{cph|cgroup_per_host}]]

• -m mail_options — defines the set of conditions under which the execution server will send a mail message about the job 📬 The mail_options argument is a string made of either the single character "n" or "p", or one or more of the characters "a", "b", "e", and "f".

If the character "n" is specified, no normal mail is sent — but mail for job cancels and other events outside of normal job processing still goes out. If the character "p" is specified, mail will never be sent for the job. Iconic silence.

And for the characters "a", "b", "e", and "f":

• a — mail is sent when the job is aborted by the batch system 😭
• b — mail is sent when the job begins execution 🌸
• e — mail is sent when the job terminates 🎀
• f — mail is sent when the job terminates with a non-zero exit code 💀

If the -m option isn't specified, mail will be sent if the job is aborted.

• -M user_list — declares the list of users that the execution server mails when it sends mail about the job. The user_list argument looks like:

user[@host][,user[@host],...]

If it's unset, the list defaults to the submitting user at the qsub host — i.e. the job owner 🫶

• -n node_exclusive — lets you request exclusive-node access/allocation for the job. This sets node_exclusive = True in the output of qstat -f <job ID>. By default this only applies for cpusets, and only for compatible schedulers.

For systems that use Moab and have cgroups enabled, the recommended way to assign all cores is to use NUMA syntax: "-L tasks=<count>:lprocs=all:place=node".

With cgroups, the (-l) syntax (lowercase L) will, by default, restrict to the number of cores requested, or to the resources_default.procs value (i.e., 1 core, typically). To override this behavior and have Moab assign all the cores on a node while using "-l...singlejob" and/or "-n" (in other words, without "-L ...lprocs=all..."), you must also set RMCFG[<torque>] FLAGS=MigrateAllJobAttributes in moab.cfg. No cap, that's just how it is 💅

• -N name — declares a name for the job, you know, so it doesn't feel anonymous 💖 The name can be up to and including 15 characters in length. It must consist of printable, non white space characters, and the first character must be alphabetic.

If -N isn't specified, the job name will be the base name of the job script file you specified on the command line. And if no script file name was specified and the script was read from the standard input, then the job name gets set to STDIN 🐚

• -o path — defines the path used for the standard output stream of the batch job. The path argument looks like:

[hostname:][path_name]

where hostname is the name of a host to which the file will be returned, and path_name is the path name on that host in the syntax recognized by POSIX. The argument gets interpreted like this:

• path_name — where path_name is not an absolute path name, the qsub command expands the path name relative to the current working directory of the command. The command supplies the name of the host upon which it's executing for the hostname component 🌸
• hostname:path_name — where path_name is not an absolute path name, the qsub command will not expand the path name relative to the current working directory of the command. On delivery of the standard output, the path name gets expanded relative to the user's home directory on the hostname system 🦋
• path_name — where path_name specifies an absolute path name, then qsub supplies the name of the host on which it's executing for the hostname
• hostname:path_name — where path_name specifies an absolute path name, the path is used as specified. hostname.
• hostname: — where hostname specifies the name of the host that the file should be returned to. The path will be the default file name 💅

If the -o option isn't specified, or path_name isn't specified, or it is specified and is a directory, then the default file name for the standard output stream gets used. That default name is:

job_name.osequence_number

where job_name is the name of the job (peep the -N option) and sequence_number is the job number assigned when the job was submitted 🎀

• -p priority — defines the priority of the job. The priority argument must be an integer between -1024 and +1023, inclusive. The default is no priority, which is equivalent to a priority of zero ⭐
• -P proxy_user[:group] — the proxy user the job should be submitted for. Heads up: this option is only available for the super user 👑
• -q destination — defines the destination of the job. The destination names a queue, a server, or a queue at a server.

The qsub command submits the script to the server defined by the destination argument. If the destination is a routing queue, the job may get routed by the server to a new destination 🔀

If -q isn't specified, qsub submits the script to the default server. Check PBS_DEFAULT under the Environment Variables section of this man page, and the PBS ERS section 2.7.4, "Default Server".

If -q is specified, it's in one of these three forms:

queue
@server
queue@server

• If the destination argument names a queue and doesn't name a server, the job gets submitted to the named queue at the default server.
• If the destination argument names a server and doesn't name a queue, the job gets submitted to the default queue at the named server.
• If the destination argument names both a queue and a server, the job gets submitted to the named queue at the named server 🎯

• -r y|n — declares whether the job is rerunable. See the qrerun command. The option argument is a single character, either y or n.

If the argument is "y", the job is rerunable. If the argument is "n", the job is not rerunable. The default value is 'y' — rerunable 🔁

• -S path_list — declares the shell that interprets the job script. The option argument path_list looks like:

path[@host][,path[@host],...]

Only one path may be specified for any host named. Only one path may be specified without the corresponding host name. The path that gets selected is the one with the host name matching the name of the execution host. If no matching host is found, then the path specified without a host gets selected, if present.

If the -S option isn't specified, the option argument is the null string, or no entry from the path_list is selected, then execution uses the user's login shell on the execution host 🐚

• -t array_request — specifies the task ids of a job array. Single task arrays are allowed, bestie, no gatekeeping 💜

The array_request argument is an integer id or a range of integers. Multiple ids or id ranges can be combined in a comma delimted list. Examples: -t 1-100 or -t 1,10,50-100 ✨

Also! Lowkey the most important part of the array_request: you can tack on an optional slot limit at the end, and it caps how many jobs from your array are allowed to run at the same time. Default is unlimited, no cap, run free 🦋

The slot limit has to be the last thing you specify in the array_request, and it's separated from the array by a percent sign (%).

qsub script.sh -t 0-299%5

That sets the slot limit to 5 — only 5 jobs from this array can run at the same time. iconic, honestly.

Note: you can use qalter to change slot limits on an existing array. And the server parameter max_slot_limit lets you set a global slot limit policy. 👑

---

• -T script_name — this one's for per-job prologue and epilogue scripts ✨ The full script name ends up being prologue.[name] or epilogue.[name] — so when you submit, you only ask for the name of the prologue/epilogue script, not the whole thing.

Example:

qsub -T prescript

That one says "use the script prologue.prescript" 💅

• -u user_list — sets the user name the job runs under on the execution system.

The user_list argument is of the form:

user[@host][,user[@host],...]

Rules, bestie, listen up: only one user name per specified host. Only one of your user specs can be supplied without a matching host spec — and that hostless user name is the one used for execution on any host that isn't named in the argument list. If you don't set it at all, the user list defaults to whoever is running qsub. 🫶

• -v variable_list — expands the list of environment variables exported to the job 🌈

On top of the variables described in the DESCRIPTION section above, variable_list names environment variables from the qsub command environment that get made available to the job when it executes. The variable_list is a comma-separated list of strings of the form variable or variable=value. Those variables and their values get passed to the job. 📦

• -V — declares that all environment variables in the qsub command's environment are exported to the batch job. all of them. every last one. go off 💀
• -w path — defines the working directory path to use for the job. If you don't pass -w, the default working directory is the current directory. This option sets the environment variable PBS_O_WORKDIR. 🗂️
• -W additional_attributes — the ✨extra credit✨ option, for specifying additional job attributes.

General syntax:

-W attr_name=attr_value[,attr_name=attr_value...]

Heads up: if white space shows up anywhere inside the option argument string, or an equal sign = occurs inside an attribute_value string, then you gotta wrap the string in either single or double quote marks. no cap, it'll break otherwise 😭

PBS currently supports these attributes inside -W:

• depend= dependency_list — defines the dependency between this job and other jobs. The dependency_list is in the form:

type[:argument[:argument...][,type:argument...].

The argument is either a numeric count or a PBS job id, depending on type. If it's a count, it must be greater than 0. If it's a job id and it's not fully specified as seq_number.server.name, it gets expanded according to the default server rules that apply to job IDs on most commands. If argument is null (the preceding colon need not be specified), the dependency of that type is cleared/unset. 🧹

• synccount: count — this job is the first in a set of jobs that execute at the same time. count is the number of additional jobs in the set.
• syncwith: jobid — this job is an additional member of a set of jobs that execute at the same time. In this and the following dependency types, jobid is the job identifier of the first job in the set.
• after: jobid[:jobid...] — this job may be scheduled for execution at any point after jobs jobid have started execution.
• afterok: jobid[:jobid...] — this job may be scheduled for execution only after jobs jobid have terminated with no errors. See the csh warning under "Extended Description".
• afternotok: jobid[:jobid...] — this job may be scheduled for execution only after jobs jobid have terminated with errors. See the csh warning under "Extended Description".
• afterany: jobid[:jobid...] — this job may be scheduled for execution after jobs jobid have terminated, with or without errors.
• on: count — this job may be scheduled for execution after count dependencies on other jobs have been satisfied. This form is used together with one of the before forms, see below 👇
• before: jobid[:jobid...] — once this job has begun execution, then jobs jobid... may begin.
• beforeok: jobid[:jobid...] — if this job terminates execution without errors, then jobs jobid... may begin. See the csh warning under "Extended Description".
• beforenotok: jobid[:jobid...] — if this job terminates execution with errors, then jobs jobid... may begin. See the csh warning under "Extended Description".
• beforeany: jobid[:jobid...] — when this job terminates execution, jobs jobid... may begin.

If any of the before forms are used, the jobs referenced by jobid must have been submitted with a dependency type of on. fr fr, that's the deal 🤝

Array Dependencies 🎀

It is now possible to have a job depend on an array. These dependencies are in the form depend=arraydep:arrayid[num]. If [num] isn't present, the dependency applies to the entire array. If [num] is present, then num means the number of jobs that must meet the condition for the dependency to be satisfied.

• afterstartarray: arrayid[count] — this job may be scheduled for execution only after jobs in arrayid have started execution.
• afterokarray: arrayid[count] — this job may be scheduled for execution only after jobs in arrayid have terminated with no errors.
• afternotok: arrayid[count] — this job may be scheduled for execution only after jobs in arrayid have terminated with errors.
• afteranyarray: arrayid[count] — this job may be scheduled for execution after jobs in array id have terminated, with or without errors.
• beforestartarray: arrayid[count] — this job may be scheduled for execution only before jobs in arrayid have started execution.
• beforeokarray: arrayid[count] — this job may be scheduled for execution only before jobs in arrayid have terminated with no errors.
• beforenotok: arrayid[count] — this job may be scheduled for execution only before jobs in arrayid have terminated with errors.
• beforeanyarray: arrayid[count] — this job may be scheduled for execution before jobs in array id have terminated, with or without errors.

If any of the before forms are used, the jobs referenced by jobid must have the same owner as the job being submitted. Otherwise, the dependency is just ignored 🙈

Error processing of the existence, state, or condition of the job that the newly submitted job depends on is a deferred service — like, the check happens after the job is queued. If an error is detected, the new job gets deleted by the server 😭 and mail is sent to the job submitter explaining the error.

Dependency examples, saved for you bestie 🍓:

qsub -W depend=afterok:123.big.iron.com /tmp/script
qsub -W depend=before:234.hunk1.com:235.hunk1.com /tmp/script
qsub -W depend=afterokarray:21.tom.com[] /tmp/script
qsub -W depend=beforenotokarray:22.tom.com[][5] /tmp/script

• group_list= g_list — defines the group name the job runs under on the execution system. The g_list argument is of the form:

group[@host][,group[@host],...]

Same energy as -u: only one group name per specified host, and only one of the group specs may be supplied without its matching host spec. That hostless group name is the one used for execution on any host not named in the argument list. If it's not set, group_list defaults to the primary group of the user under which the job will be run. 💜

• interactive=true — if the interactive attribute is specified, the job is an interactive job. (And psst, the -I option is an alternative way of specifying this same attribute 👑)

• stagein=file_list / stageout=file_list — okay so these two tell qsub which files get staged (copied) in before the job starts, or staged out after the job finishes. On completion of the job, all staged-in and staged-out files are removed from the execution system. The file_list is in the form:

local_file@hostname:remote_file[,...]

regardless of the direction of the copy. local_file is the name of the file on the system where the job executed. It may be an absolute path or relative to the home directory of the user. remote_file is the destination name on the host specified by hostname. That name may be absolute or relative to the user's home directory on the destination host. The use of wildcards in the file name is not recommended 🙈. The file names map to a remote copy program (rcp) call on the execution system in the follow manner:

For stagein:   rcp hostname:remote_file local_file
For stageout:  rcp local_file hostname:remote_file

Data staging examples, bestie:

-W stagein=/tmp/input.txt@headnode:/home/user/input.txt
-W stageout=/tmp/output.txt@headnode:/home/user/output.txt

If TORQUE has been compiled with wordexp support, then variables can be used in the specified paths. Currently only $PBS_JOBID, $HOME, and $TMPDIR are supported for stagein.

• umask=XXX — sets the umask used to create stdout and stderr spool files in the pbs_mom spool directory. Values starting with 0 are treated as octal values, otherwise the value is treated as a decimal umask value. 🎀
• -x — when you're running an interactive job, this flag makes it so that the script won't be parsed for PBS directives, but instead will be a command that is launched once the interactive job has started. The job will terminate at the completion of this command. No cap.
• -X — enables X11 forwarding. The DISPLAY environment variable must be set. 💅
• -z — directs that the qsub command is not to write the job identifier assigned to the job to the command's standard output. Sneaky lil quiet mode.

🦋 OPERANDS 🦋

The qsub command accepts a script operand that is the path to the script of the job. If the path is relative, it will be expanded relative to the working directory of the qsub command.

If the script operand is not provided or the operand is the single character -, the qsub command reads the script from standard input. When the script is being read from standard input, qsub will copy the file to a temporary file. This temporary file is passed to the library interface routine pbs_submit. The temporary file is removed by qsub after pbs_submit returns or upon the receipt of a signal which would cause qsub to terminate. Very responsible of it, honestly 🫶

🌸 STANDARD INPUT 🌸

The qsub command reads the script for the job from standard input if the script operand is missing or is the single character -.

🎀 INPUT FILES 🎀

The script file is read by the qsub command. Qsub acts upon any directives found in the script.

When the job is created, a copy of the script file is made and that copy cannot be modified. It's giving locked-in main character energy 🔥

⭐ STANDARD OUTPUT ⭐

Unless the -z option is set, the job identifier assigned to the job will be written to standard output if the job is successfully created.

🌈 STANDARD ERROR 🌈

The qsub command will write a diagnostic message to standard error for each error occurrence. So if something's cooked, it will tell you 😭

💜 ENVIRONMENT VARIABLES 💜

The values of some or all of the variables in the qsub command's environment are exported with the job, see the -v and -V options.

The environment variable PBS_DEFAULT defines the name of the default server. Typically, it corresponds to the system name of the host on which the server is running. If PBS_DEFAULT is not set, the default is defined by an administrator established file.

The environment variable PBS_DPREFIX determines the prefix string which identifies directives in the script.

The environment variable PBS_CLIENTRETRY defines the maximum number of seconds qsub will block. See the -b option above. Despite the name, currently qsub is the only client that supports this option. iykyk ✨

🧚 TORQUE.CFG 🧚

The torque.cfg file, located in PBS_SERVER_HOME (/var/spool/torque by default) controls the behavior of the qsub command. This file contains a list of parameters and values separated by whitespace.

QSUBSLEEP takes an integer operand which specifies time to sleep when running qsub command. Used to prevent users from overwhelming the scheduler. Respect the scheduler's nap time 💅

SUBMITFILTER specifies the path to the submit filter used to pre-process job submission. The default path is $(libexecdir)/qsub_filter, which falls back to /usr/local/sbin/torque_submitfilter for backwards compatibility. This torque.cfg parameter overrides this default.

SERVERHOST specifies the value for the PBS_SERVER environment variable.

QSUBHOST specifies the hostname for the jobs QSUB_O_HOST variable.

QSUBSENDUID specifies a uid to use for the jobs PBS_O_UID variable.

XAUTHPATH specifies the path to xauth.

CLIENTRETRY specifies the integer seconds between retry attempts to communicate with pbs_server.

VALIDATEGROUP set this parameter to force qsub to verify the submitter's group id.

DEFAULTCKPT specifies the default value for the jobs checkpoint attribute. The user overrides this with the -c qsub option.

VALIDATEPATH set this parameter to force qsub to validate local existence of a -d working directory.

RERUNNABLEBYDEFAULT this parameter specifies if a job is rerunnable by default. The default is true, setting this to false causes the rerunnable attribute value to be false unless the users specifies otherwise with the -r option.

FAULT_TOLERANT_BY_DEFAULT this parameter specifies if a job is fault tolerant by default. The default value for the fault_tolerant job attribute is false, setting this parameter to true causes the default value of the attribute to be true. The user can specify their preference with the -f qsub option.

For example:

QSUBSLEEP  2
RERUNNABLEBYDEFAULT  false

✨ EXTENDED DESCRIPTION ✨

📜 Script Processing

Okay so a job script can be made of three things: PBS directives, comments, and executable statements. A PBS directive is literally just a way to set job attributes in addition to the command line options — like, you get to be extra, bestie. Example time:

:
#PBS -N Job_name
#PBS -l walltime=10:30,mem=320kb
#PBS -m be
#
step1 arg1 arg2
step2 arg3 arg4

Here's how qsub reads it: it scans the lines of the script file looking for directives. If the very first line of the script starts with the characters #! or the character :, that line is ignored and scanning starts with the next line. 👑

Scanning keeps going until it hits the first executable line — meaning a line that is not blank, not a directive line, and not a line whose first non-white-space character is #. And heads up: if directives show up on any lines after that point, they get ignored. Sorry babe, the window closed. 🙈

A line in the script file gets processed as a directive to qsub if and only if the string of characters starting with the first non-white-space character on the line, of the same length as the directive prefix, matches the directive prefix. That's the whole vibe check.

The rest of the directive line is just the options to qsub in the exact same syntax as they'd appear on the command line. The option character gets preceded with the - character. ✨

Now the drama: if an option is in both a directive and on the command line, that option and its argument (if it has one) get ignored in the directive. Command line takes precedence, always, no cap. Command line is the main character.

But if an option is in a directive and not on the command line? Then it and its argument (if any) get processed as if it had occurred on the command line. Teamwork. 🫶

The directive prefix string gets determined in order of preference from:

1. The value of the -C option argument, if the option is specified on the command line.
2. The value of the environment variable PBS_DPREFIX if it is defined.
3. The four character string #PBS.

Lowkey important: if the -C option is found in a directive in the script file, it will be ignored. Periodt. 💅

---

🔐 User Authorization

So when the user submits a job from a system other than the one the PBS Server is running on, the name under which the job is executed gets selected according to the rules listed under the -u option. The user submitting the job must be authorized to run the job under the execution user name. 😤

This authorization is provided if:

1. The host on which qsub is run is trusted by the execution host (see /etc/hosts.equiv), or
2. The execution user has an .rhosts file naming the submitting user on the submitting host.

That's the whole deal — one of those two, or you're not getting in. 🚪

---

🐚 C-Shell .logout File

Okay this next one is a genuine warning and it applies to all the csh girlies out there. If the job is executed under the csh and a .logout file exists in the home directory in which the job executes, the exit status of the job is that of the .logout script — not the job script. 😭

And yeah, this may impact any inter-job dependencies. Which is bad. So to preserve the job exit status, you either remove the .logout file entirely, or place the following line as the first line in the .logout file:

set EXITVAL = $status

and then the following line as the last executable line in .logout:

exit $EXITVAL

There. Fixed. You're welcome. 🎀

---

💻 Interactive Jobs

If the -I option is specified on the command line or in a script directive, or if the "interactive" job attribute is declared true via the -W option — that's -W interactive=true — either on the command line or in a script directive, then congrats, the job is an interactive job, bestie. 🌈

The script will be processed for directives, but it will not be included with the job. When the job begins execution, all input to the job comes from the terminal session in which qsub is running. So it's very much a you and the terminal, together situation.

When an interactive job is submitted, qsub will not terminate just because the job got submitted. qsub stays running until the job terminates, gets aborted, or the user interrupts qsub with an SIGINT (the Ctrl+C key). If qsub is interrupted before the job starts, it will query whether the user wishes to exit — and if the user responds "yes", qsub exits and the job is aborted. 💀

Once the interactive job has started executing, input to and output from the job both pass through qsub. Keyboard-generated interrupts get passed along to the job too. And here's the cute part — lines you enter that begin with the tilde (~) character and contain special sequences get escaped by qsub. The recognized escape sequences are:

• ~. — qsub terminates execution. The batch job is also terminated.
• ~susp — Suspend the qsub program if running under the C shell. "susp" is the suspend character, usually CNTL-Z.
• ~asusp — Suspend the input half of qsub (terminal to job), but allow output to continue to be displayed. Only works under the C shell. "asusp" is the auxiliary suspend character, usually CNTL-Y.

---

💫 EXIT STATUS

Upon successful processing, the qsub exit status will be a value of zero. 🥳

If the qsub command fails, the command exits with a value greater than zero. Sad times, but at least you know. 😔

---

👀 SEE ALSO

Go follow the whole friend group, they're all iconic:

qalter(1B), qdel(1B), qhold(1B), qmove(1B), qmsg(1B), qrerun(1B), qrls(1B), qselect(1B), qsig(1B), qstat(1B), pbs_connect(3B), pbs_job_attributes(7B), pbs_queue_attributes(7B), pbs_resources_irix5(7B), pbs_resources_sp2(7B), pbs_resources_sunos4(7B), pbs_resources_unicos8(7B), pbs_server_attributes(7B), and pbs_server(8B).

---

And that's the whole page, babe! 💖 This one's brought to you by Local and it's the qsub(1B) era officially — go submit something iconic, and remember: command line always wins. 🦋🌸👑