qsub(1B)

📦 version 23.06.06-unstable-2026-04, in nixos 26.05, nixpkgs unstable

💅 qsub — PBS Professional 💅

okay bestie, welcome to qsub(1B) from the PBS Professional manual — yes, section 1B (she's fancy, she's a "B" section, we love her for it) 💖

This is the page where the queen herself shows up: qsub, the command that gets your job into PBS. Grab a snack, because this synopsis is a buffet and we're gonna walk it like a runway. 🎀

✨ NAME ✨

qsub — submit a job to PBS 🚀

Like, that's literally her whole thing. She submits jobs. To PBS. Main character energy from line one, no cap. 👑

🦋 SYNOPSIS 🦋

okay so here's the full shape of it — every bracket here is optional, which means the options are giving "choose your own adventure" and honestly? obsessed. 🌈

qsub [-a <date and time>] [-A <account string>] [-c <checkpoint spec>]
     [-C <directive prefix>] [-e <path>] [-f] [-h]
     [-I [-G [-- <GUI application/script>]] | [-X]] [-j <join>]
     [-J <range> [%<max subjobs]] [-k <discard>] [-l <resource list>]
     [-m <mail events>] [-M <user list>] [-N <name>] [-o <path>]
     [-p <priority>] [-P <project>] [-q <destination>] [-r <y|n>]
     [-R <remove options>] [-S <path list>] [-u <user list>]
     [-v <variable list>] [-V] [-W <additional attributes>] [-z]
     [- | <script> | -- <executable> [<arguments to executable>]]
qsub --version

a few lil things to clock about that, because the synopsis is lowkey telling a story 📖 ✨

  • the very last argument slot is a three-way vibe check: you can pipe from standard input with a bare -, hand over a <script>, or go full -- <executable> [<arguments to executable>] and pass the executable plus its arguments along. choose wisely, babe 🫶
  • there's also a whole standalone incantation, qsub --version, sitting on its own line like she doesn't even need the rest of the options. independent icon behavior. 💅 (the details on what she does are coming in the next parts — eyes peeled 👀)
  • notice [-I [-G [-- <GUI application/script>]] | [-X]] — that's a little nested situation with a | in the middle, so it's -I world or -X world, and inside the -I world there's a -G world. inception but make it options. 🌀
  • and [-J <range> [%<max subjobs]] is doing that same nested-bracket thing with a %. she's got layers. 🎂
just so we're clear, bestie — the synopsis is the shape of the command, not the full tea. Every single one of those flags has a whole personality (and rules, and defaults, and sometimes chaos) which is all coming up in the sections below. don't go running off with just the synopsis, iykyk 😭

that's the vibe check on qsub's identity and her whole bracket situation ✅ — more parts are coming, don't go anywhere, we're so not done 💜

💖 DESCRIPTION 💖

okay so this is the whole point of qsub: you use it to submit a batch job to PBS. And submitting a PBS job is basically three things at once — you're specifying a task, you're requesting resources, and you're setting job attributes. All in one go. Slay. ✨

The qsub command can read from a job script, from standard input, or from the command line. Once you've submitted, PBS hands you back the job identifier for that job, and for a regular job it's giving:

<sequence number>.<server name>

For an array job, it's giving:

<sequence number>[].<server name>

During execution, jobs can be interactive or non-interactive. Heads up though, bestie — interactive jobs are not rerunnable, and if they're blocking, you cannot use their exit status. 😭 That's just how it is.

Jobs are run as the user and group who submitted the job. No cap.

🌀 Background Process 🌀

By default, on the first invocation, qsub spawns a background process to manage communication with the PBS server. Later invocations of qsub attempt to communicate with this background process. Cute, right? 🎀

But — and this is the part where you go 🙈 — under certain circumstances, calls to qsub when it uses the background process can result in communication problems. You can prevent qsub from spawning a background process by using the -f option, although this can degrade performance. So like, pick your poison, iykyk.

📁 Where PBS Puts Job Files 📁

By default, PBS copies the stdout and stderr files from the job back to the current working directory where the qsub command is executed. That's the vibe. 💅

But you can absolutely specify the output paths using the -o and -e options. You can also specify which and whether these files should be kept on the execution host via the -k option, or deleted, using the -R option. See the -k, -o, -e, and -R options. 🫶

📜 Submitting Jobs By Using Scripts 📜

To submit a PBS job by using a script, you specify a job script on the command line:

qsub [<options>] <script name>

For example:

qsub myscript.sh

Job scripts are run as the user and group who submitted the job. And they can be written in Python, Linux shells such as csh and sh, the Windows command batch language, Perl, etc. So many options, we love versatility. 🌈

A PBS job script consists of the following:

  • Optional shell specification
  • Any PBS directives
  • The user's tasks: programs, commands, or applications
  • Optional comments (Under Windows, comments can contain only ASCII characters.)

🐚 Using Shells and Interpreters 🐚

By default, PBS uses your login shell to run your script. But you can optionally specify a different shell or interpreter to run your script — main character energy, honestly:

Via the -S option to qsub:

qsub -S <path to shell> <script name>

For example:

qsub -S /bin/bash myscript.sh

You can specify a different interpreter in the first line of your script. For example:

cat myscript.sh
#!/bin/bash
#PBS -N MyHelloJob
echo "Hello"

🐍 Python Job Scripts 🐍

You can use the same Python script under Linux or under Windows, if the script is written to be portable. PBS includes a Python package, allowing Python job scripts to run — you do not need to install Python. Obsessed. 💖

You can include PBS directives in a Python job script as you would in a Linux shell script. Python job scripts can access Win32 APIs, including the following modules:

  • Win32api
  • Win32con
  • Pywintypes

For example, we have a Python job script that includes PBS directives:

cat myjob.py
#!/usr/bin/python
#PBS -l select=1:ncpus=3:mem=1gb
#PBS -N HelloJob
print "Hello"

So long as the first line of the script is the #!/usr/bin/python line or similar, you don't need to do anything special to run a python script. Periodt.

qsub <script name>

To run a Python job script under Windows, use the path to the pbs_python executable on the execution host:

qsub -S <pbs_python path on execution host> <script name>

For example,

qsub -S %PBS_EXEC%\bin\pbs_python.exe <script name>

If the script pathname contains spaces, it must be quoted, for example:

qsub -S "C:\Program Files\PBS Pro\bin\pbs_python.exe" <script name>

🐧 Linux Shell Job Scripts 🐧

For example, we have a Linux job script named "weatherscript" for a job named "Weather1" which runs the executable "weathersim" on Linux:

#!/bin/sh
#PBS -N Weather1
#PBS -l walltime=1:00:00
/usr/local/weathersim

To submit the job, the user types:

qsub weatherscript <return>

🪟 Windows Command Job Scripts 🪟

For example, we have a script named "weather.exe" for a job named "Weather1" which runs under Windows:

#PBS -N Weather1
#PBS -l walltime=1:00:00
weathersim.exe

To submit the job, the user types:

qsub weather.exe <return>
⚠️ In Windows, if you use notepad to create a job script, the last line does not automatically get newline-terminated. Be sure to put one explicitly, otherwise, PBS job will get the following error message:
More?

when the Windows command interpreter tries to execute that last line. 😭 Not the More? jumpscare. Please add that newline, bestie.

⌨️ Submitting Jobs From Standard Input ⌨️

To submit a PBS job by typing job specifications at the command line, you type:

qsub [<options>] [-] <return>

then type any directives, then any tasks, followed by:

  • Linux: CTRL+D on a line by itself
  • Windows: CTRL+Z <return>

to terminate the input.

The qsub command behaves the same both with and without the dash operand. For example, on Linux:

qsub <return>
#PBS -N StdInJob
sleep 100
<CTRL-D>

🚀 Submitting Job Directly by Specifying Executable on Command Line 🚀

To submit a job directly, you specify the executable on the command line:

qsub [<options>] -- <executable> [<arguments to executable>] <return>

When you run qsub this way, it runs the executable directly. It does not start a shell, so no shell initialization scripts are run, and execution paths and other environment variables are not set. There is not an easy way to run your command in a different directory. You should make sure that environment variables are set correctly, and you will usually have to specify the full path to the command. 🫡

For example, to run myprog with the arguments a and b:

qsub -- myprog a b <return>

For example, to run myprog with the arguments a and b, naming the job "JobA":

qsub -N JobA -- myprog a b <return>

On Linux, you need to specify the path to myprog, so the previous example becomes:

qsub -N JobA -- /path/to/myprog a b <return>

🧚 Requesting Resources and Placing Jobs 🧚

Requesting resources includes setting limits on resource usage and controlling how the job is placed on vnodes. It's giving very organized.

Resources are requested by using the -l option, either in job-wide requests using <resource name>=<value> pairs, or in chunks inside of selection statements.

See the pbs_resources(7B) man page.

Job-wide <resource name>=<value> requests are of the form:

-l <resource name>=<value>[,<resource name>=<value> ...]

The selection statement is of the form:

-l select=[<N>:]<chunk>[+[<N>:]<chunk> ...]

where N specifies how many of that chunk, and a chunk is of the form:

<resource name>=<value>[:<resource name>=<value> ...]

You choose how your chunks are placed using the place statement. The place statement can contain the following elements, in any order:

-l place=[<arrangement>][:<sharing>][:<grouping>]

where:

  • arrangement — Whether this chunk is willing to share this vnode or host with other chunks from the same job. One of free | pack | scatter | vscatter
  • sharing — Whether this this chunk is willing to share this vnode or host with other jobs. One of excl | shared | exclhost
  • grouping — Whether the chunks from this job should be placed on vnodes that all have the same value for a resource. Can have only one instance of group=<resource name>

And here's what those arrangement values actually mean, bestie:

  • free — Place job on any vnode(s).
  • pack — All chunks are taken from one host.
  • scatter — Only one chunk with any MPI processes is taken from a host. A chunk with no MPI processes may be taken from the same vnode as another chunk.
  • vscatter — Only one chunk is taken from any vnode. Each chunk must fit on a vnode.
  • excl — Only this job uses the vnodes chosen.
  • shared — This job can share the vnodes chosen.
  • exclhost — The entire host is allocated to the job.
  • group=<resource name> — Chunks are grouped according to a resource. All vnodes in the group must have a common value for resource, which can be either the built-in resource host or a custom vnode-level resource.
    • resource name must be a string or a string array.
💀 The place statement cannot begin with a colon. Colons are delimiters; use them only to separate parts of a place statement, unless they are quoted inside resource values.

Note that vnodes can have sharing attributes that override job placement requests. See the pbs_node_attributes.7B man page.

🙈 Caveats for Requesting Resources 🙈

Do not mix old-style resource or vnode specifications with the new select and place statements. Do not use one in a job script and the other on the command line. Mixing the two will result in an error. You've been warned, bestie. 😭

You cannot submit a job requesting a custom resource which has been created to be invisible or read-only for unprivileged users, regardless of your privilege. A Manager or Operator can use the qalter command to change a job's request for this kind of custom resource. ikyk. 💅

For more on resource requests, usage limits and job placement, see pbs_resources(7B).

🎀 Setting attributes 🎀

The job submitter sets job attributes by giving options to the qsub command or by using PBS directives. Most qsub options set a job attribute, and have a corresponding PBS directive with the same syntax as the option. Attributes set via command-line options take precedence over those set using PBS directives. See pbs_job_attributes.7B.

🌸 Changing qsub Behavior 🌸

The behavior of the qsub command may be affected by the server's default_qsub_arguments attribute. This attribute can set the default for any job attribute. The default_qsub_arguments server attribute is settable by the administrator, and is overridden by command-line arguments and script directives. See the pbs_server_attributes(1B) man page.

The behavior of the qsub command may also be affected by any site hooks. Site hooks can modify the job's attributes, change its routing, etc. Sneaky but iconic. ✨

⭐ Options to qsub ⭐

okay bestie, here come the options — buckle up, there are a lot and they all matter. 🫶

  • -a <date and time> — Point in time after which the job is eligible for execution. Given in pairs of digits. Sets job's Execution_Time attribute to date and time. Format: datetime, expressed as
[[[[CC]YY]MM]DD]hhmm[.SS]

where CC is the century, YY is the year, MM is the month, DD is the day of the month, hh is the hour, mm is the minute, and SS is the seconds. 🕰️

Each portion of the date defaults to the current date, as long as the next-smaller portion is in the future. For example, if today is the 3rd of the month and the specified day DD is the 5th, the month MM is set to the current month.

If a specified portion has already passed, the next-larger portion is set to one after the current date. For example, if the day DD is not specified, but the hour hh is specified to be 10:00 a.m. and the current time is 11:00 a.m., the day DD is set to tomorrow. ⏰

Now for the flags themselves — no skips, no cap: 💅

  • -A <account string> — The accounting string associated with the job. It’s used for labeling accounting data, and it sets the job’s Account_Name attribute to account string. Format: String. 💸
  • -c <checkpoint spec> — Determines when the job will be checkpointed. It sets the job’s Checkpoint attribute to checkpoint spec. Heads up: an $action script is required to checkpoint the job. The checkpoint spec can take one of these values:
    • c — Checkpoint at intervals, measured in CPU time, set on the job’s execution queue. If there is no interval set at the queue, the job is not checkpointed.
    • c=<minutes of CPU time> — Checkpoint at intervals of the specified number of minutes of job CPU time. This value must be greater than zero. If the interval specified is less than that set on the job’s execution queue, the queue’s interval is used. Format: Integer.
    • w — Checkpoint at intervals, measured in walltime, set on the job’s execution queue. If there is no interval set at the queue, the job is not checkpointed.
    • w=<minutes of walltime> — Checkpoint at intervals of the specified number of minutes of job walltime. This value must be greater than zero. If the interval specified is less than that set on the job’s execution queue, the queue’s interval is used. Format: Integer.
    • n — No checkpointing.
    • s — Checkpoint only when the server is shut down.
    • u — Unset. Defaults to the behavior when the interval argument is set to s.

Default: u — Format: String.

  • -C <directive prefix> — Defines the prefix identifying a PBS directive. The default prefix is "#PBS". If the directive prefix argument is a null string, qsub does not scan the script file for directives. This overrides the PBS_DPREFIX environment variable and the default. The string "PBS_DPREFIX" cannot be used as a PBS directive. Length limit: 4096 characters. 🎀
  • -e <path> — The path used for the job’s standard error stream. It sets the job’s Error_Path attribute to path. The path argument is of the form [<hostname>:]<path>. The path is interpreted like this:
    • path — If path is relative, it is taken to be relative to the current working directory of the qsub command, where it is executing on the current host. If path is absolute, it is taken to be an absolute path on the current host where the qsub command is executing.
    • hostname:path — If path is relative, it is taken to be relative to the user’s home directory on the host named hostname. If path is absolute, it is an absolute path on the host named hostname.

If path does not include a filename, the default filename has the form <job ID>.ER.

If the -e option is not specified, PBS copies the standard error to the current working directory where the qsub command was executed, and writes standard error to the default filename, which has this form: <job name>.e<sequence number>.

If you use a UNC path for output or error files, the hostname is optional. If you use a non-UNC path, the hostname is required.

This option is overridden by the -k option. 🧚

  • -f — Prevents qsub from spawning a background process. By default, qsub spawns a background process to manage communication with the PBS server. When this option is specified, the qsub process connects directly to the server and no background process is created.
NOTE: Use of this option degrades performance of the qsub command when calls are made in rapid succession. ⚠️
  • -G [-- <GUI application>] — Starts a GUI session. When no application or script is provided, it starts a GUI-enabled interactive shell. When an application or script is provided, it starts the GUI application or script. Use the full path to the application or script unless the path is part of the user’s PATH environment variable on the execution host. When submission and execution hosts are different, this uses a remote viewer.

The session is terminated when the remote viewer, GUI application, or interactive shell is terminated, or when the job is deleted.

Can be used only with interactive jobs (the -I option). Available only under Windows. 🪟

  • -h — Applies a User hold to the job. Sets the job’s Hold_Types attribute to "u". ✋
  • -I — The job is to be run interactively. Sets the job’s interactive attribute to True. The job is queued and scheduled as any PBS batch job, but when executed, the standard input, output, and error streams of the job are connected to the terminal session in which qsub is running. If a job script is given, only its directives are processed. When the job begins execution, all input to the job is taken from the terminal session.

Interactive jobs are not rerunnable.

Job arrays cannot be interactive.

When used with -Wblock=true, no exit status is returned. 🖥️

  • -j <join> — Specifies whether and how to join the job’s standard error and standard output streams. Sets the job’s Join_Path attribute to join. Default: n, not merged. The join argument can take these values:
    • oe — Standard error and standard output are merged into standard output.
    • eo — Standard error and standard output are merged into standard error.
    • n — Standard error and standard output are not merged.
  • -J <range> [%<max subjobs>] — Makes this job an array job. Sets the job’s array attribute to True.

Use the range argument to specify the indices of the subjobs of the array. range is specified in the form X-Y[:Z] where X is the first index, Y is the upper bound on the indices, and Z is the stepping factor. For example, 2-7:2 will produce indices of 2, 4, and 6. If Z is not specified, it is taken to be 1. Indices must be greater than or equal to zero.

Use the optional %max subjobs argument to set a limit on the number of subjobs that can be running at one time. This sets the value of the max_run_subjobs job attribute to the specified maximum.

Job arrays are always rerunnable. 🧮

  • -k <discard> — Specifies whether and which of the standard output and standard error streams is left behind on the execution host, or written to their final destination. Sets the job’s Keep_Files attribute to discard. Overrides default path names for these streams. Overrides -o and -e options.

Default: n; neither is retained, and files are not written to their final destinations.

In the case where output and/or error is retained on the execution host in a job-specific staging and execution directory created by PBS, these files are deleted when PBS deletes the directory.

The discard argument can take these values:

  • e — The standard error stream is retained on the execution host, in the job’s staging and execution directory. The filename is <job name>.e<sequence number>.
  • o — The standard output stream is retained on the execution host, in the job’s staging and execution directory. The filename is <job name>.o<sequence number>.
  • eo, oe — Both standard output and standard error streams are retained on the execution host, in the job’s staging and execution directory.
  • d — Output and/or error are written directly to their final destination. Overrides action of leaving files on execution host.
  • n — Neither stream is retained. 🗂️
  • -l <resource list> — Allows the user to request resources and specify job placement. Sets the job’s Resource_list attribute to resource list. Requesting a resource places a limit on its usage.

For how to request resources and place jobs, see Requesting Resources and Placing Jobs above. 📋

  • -m <mail events> — Specifies the set of conditions under which mail about the job is sent. Sets the job’s Mail_Points attribute to mail events. The mail events argument can be one of:
    • The single character "n"
    • Any combination of "a", "b", and "e", with optional "j"

The sub-options to -m are:

  • n — No mail is sent.
  • a — Mail is sent when the job is aborted by the batch system.
  • b — Mail is sent when the job begins execution.
  • e — Mail is sent when the job terminates.
  • j — Mail is sent for subjobs. Must be combined with one or more of the a, b, or e sub-options.

Format: String — Syntax: n | [j](one or more of a, b, e) — Example: -m ja — Default: a. 📬

  • -M <user list> — List of users to whom mail about the job is sent. Sets the job’s Mail_Users attribute to user list.

The user list argument has the form: <username>[@<hostname>][,<username>[@<hostname>],...].

Default: Job owner. 👤

  • -N <name> — Sets the job’s Job_Name attribute and name to name.

Format: Job Name. 🏷️

  • -o <path> — where your job's standard output stream goes. It sets the job's Output_Path attribute to path. The path argument is shaped like this:
[<hostname>:]<path>

and it gets read like this:

  • path — if path is relative, it's relative to the current working directory of the qsub command, where it's executing on the current host. If path is absolute, it's an absolute path on the current host where qsub is executing.
  • hostname:path — if path is relative, it's relative to your home directory on the host named hostname. If path is absolute, it's an absolute path on the host named hostname.

If path doesn't include a filename, the default filename has the form <job ID>.OU.

If you don't pass -o, PBS copies standard output to the current working directory where the qsub command was executed and writes standard output to the default filename, which looks like <job name>.o<sequence number>.

If you're using a UNC path, the hostname is optional. If you're using a non-UNC path, the hostname is required — no hostname, no deal.

Heads up bestie: this option gets overridden by the -k option. -k is the main character here, -o is just vibing in its shadow. 💅
  • -p <priority> — the job's priority. Sets the job's Priority attribute to priority.
    • Format: host-dependent integer
    • Range: [-1024, +1023] inclusive
    • Default: Zero
  • -P <project> — gives your job a project. Sets the job's project attribute to project.
    • Format: Project Name
    • Default value: "_pbs_project_default"
  • -q <destination> — where the job gets sent when it's submitted. You can name a queue, a server, or a queue at a server. The destination argument can be any of these:
    • <queue name> — job is submitted to the specified queue at the default server.
    • @<server name> — job is submitted to the default queue at the specified server.
    • <queue name>@<server name> — job is submitted to the specified queue at the specified server.

Default: the default queue at the default server. ✨

  • -r <y|n> — declares whether the job is rerunnable. Sets the job's Rerunable attribute to the argument value. Ngl, this does not affect how the job is handled in the case where the job was unable to begin execution — don't get it twisted.
    • Format: single character, "y" or "n"
    • y — job is rerunnable.
    • n — job is not rerunnable.

Default: "y".

Interactive jobs are not rerunnable. Job arrays are automatically marked as rerunnable. See the qrerun.1B man page for the whole lore. 👑

  • -R <remove options> — says whether the standard output and/or standard error files get automatically removed (deleted) when the job finishes.

Sets the job's Remove_Files attribute to remove options. Overrides the default path names for these streams. Overrides the -o and -e options.

Fr fr: this attribute cannot be altered once the job has begun execution. Once it's in motion, it's in motion. 😭

Default: Unset — neither one is removed.

The remove options argument takes these values:

  • e — the standard error stream is removed (deleted) upon job completion.
  • o — the standard output stream is removed (deleted) upon job completion.
  • eo, oe — both standard output and standard error streams are removed (deleted) upon job completion.
  • unset — neither stream is removed.
  • -S <path list> — the interpreter or shell path for your job script. Sets the job's Shell_Path_List attribute to path list.

The path list argument is the full path to the interpreter or shell, including the executable name.

Only one path may be specified without a host name. Only one path may be specified per named host. The path that gets selected is the one whose host name is the host name of the server on which the job resides. Simple as that.

Format:

<path>[@<hostname>][,<path>@<hostname> ...]

Default: your login shell on the execution host.

Example of using bash via a directive:

#PBS -S /bin/bash@mars,/usr/bin/bash@jupiter

Example of running a Python script from the command line on Linux:

qsub -S $PBS_EXEC/bin/pbs_python <script name>

Example of running a Python script from the command line on Windows:

qsub -S %PBS_EXEC%\bin\pbs_python.exe <script name>
  • -u <user list> — a list of usernames, and the job runs under a username from this list. Sets the job's User_List attribute to user list.

Only one username may be specified without a hostname. Only one username may be specified per named host. The server on which the job resides will pick first the username whose hostname matches the server name. Failing that, the next pick is the username with no specified hostname. The usernames on the server and execution hosts must be the same. And the job owner has to have authorization to run as the specified user — no sneaky identity switching. 🕵️‍♀️

Format of user list:

<username>[@<hostname>][,<username>@<hostname> ...]

Default: the job owner (username on the submission host).

  • -v <variable list> — lists environment variables and shell functions to be exported to the job. This is the list of env vars that get added to the ones already automatically exported. These variables exist in your environment from which qsub is run. The job's Variable_List attribute gets appended with the variables in variable list and their values. Peep the ENVIRONMENT section of this man page for more. 💜

Format: comma-separated list of strings shaped like:

<variable>

or

<variable>=<value>

If a <variable>=<value> pair contains any commas, the value has to be enclosed in single or double quotes, and the <variable>=<value> pair has to be enclosed in the kind of quotes you did not use to enclose the value. For example:

qsub -v "var1='A,B,C,D'" job.sh
qsub -v "a=10,var2='A,B',c=20,d='Hello world'" job.sh

Default: no environment variables are added to the job's variable list.

  • -V — every environment variable and shell function in your environment where qsub is run gets exported to the job. The job's Variable_List attribute is appended with all of these environment variables and their values. It's giving "bring the whole group chat." 🫶
  • -W <additional attributes> — lets you specify some job attributes. Some job attributes must be specified using this option, and those are listed below.

Format:

-W <attribute name>=<value>[,<attribute name>=<value>...]

If white space occurs within the additional attributes argument, or the equal sign "=" occurs within a value string, it must be enclosed with single quotes or double quotes.

These are the attributes you can set using the -W option only:

💅 the -W squad

  • block=true — the qsub command waits for the job to terminate, then returns the job's exit value. Sets the job's block attribute to True. When used with X11 forwarding or interactive jobs, no exit value is returned. See the EXIT STATUS section in this man page.
  • create_resv_from_job=<value> — when this job starts, it immediately creates and confirms a job-specific start reservation on the same resources as the job (including resources inherited by the job), and places the job in the job-specific reservation queue. Sets the job's create_resv_from_job attribute to True. Sets the job-specific reservation's reserve_job attribute to the ID of the job from which the reservation was created. The new reservation's duration and start time are the same as the job's walltime and start time. If the job is peer scheduled, the job-specific reservation is created in the pulling complex.
    • Format: Boolean
    • Example: qsub myscript.sh -Wcreate_resv_from_job=1
Not the vibe for job arrays or jobs being submitted into a reservation — can't be used with those. 🙈
  • depend=<dependency list> — defines dependencies between this job and other jobs. Sets the job's depend attribute to dependency list. The dependency list has the form:
<type>:<arg list>[,<type>:<arg list> ...]

where, except for the on type, the arg list is one or more PBS job IDs, and has the form:

<job ID>[:<job ID> ...]

The type can be:

  • after: <arg list> — this job may be scheduled for execution at any point after all jobs in arg list have started execution.
  • afterok: <arg list> — this job may be scheduled for execution only after all jobs in arg list have terminated with no errors. See "Warning about exit status with csh" in EXIT STATUS.
  • afternotok: <arg list> — this job may be scheduled for execution only after all jobs in arg list have terminated with errors. See "Warning about exit status with csh" in EXIT STATUS.
  • afterany: <arg list> — this job may be scheduled for execution after all jobs in arg list have finished execution, with any exit status (with or without errors). This job will not run if a job in the arg list was deleted without ever having been run. 💀
  • before: <arg list> — jobs in arg list may begin execution once this job has begun execution. It's uncommon for users to specify a before condition. Rather, PBS adds before dependencies automatically to the targets of after dependencies.

okay so we're still deep in -W territory, bestie, and it's giving "dependency lore" 💫

  • beforeok: <arg list> — jobs in arg list may begin execution once this job terminates without errors. Peep "Warning about exit status with csh" in EXIT STATUS, okay?
  • beforenotok: <arg list> — if this job terminates execution with errors, jobs in arg list may begin. Also see "Warning about exit status with csh" in EXIT STATUS.
  • beforeany: <arg list> — jobs in arg list may begin execution once this job terminates execution, with or without errors.
  • on: count — this job may be scheduled for execution after count dependencies on other jobs have been satisfied. This type is used in conjunction with one of the before types listed. count is an integer greater than 0.

Job IDs in the arg list of before types must have been submitted with a type of on. ✨

To use the before types, the user must have the authority to alter the jobs in arg list. Otherwise, the dependency is rejected and the new job aborted. That's the rule, no cap.

Error processing of the existence, state, or condition of the job that the newly submitted job depends on is performed after the job is queued. If an error is detected, the new job is deleted by the server. Mail is sent to the job submitter stating the error. 😭

Dependency example: in this example, we save the output (the jobid) from the first qsub into shell variable "jobid" so we can supply it to the depend option on the second job.

jobid=`qsub first_step.sh`
qsub -W depend=afterok:$jobid second_step.sh
  • group_list=<group list> — list of group names. The job is run under a group name from this list. Sets the job's group_List attribute to group list.
    • Only one group name may be specified without a hostname. Only one group name may be specified per named host.
    • The server on which the job resides will select first the group name whose hostname is the same as the server name. Failing that, the next selection is the group name with no specified hostname.
    • The group names on the server and execution hosts must be the same.
    • Job submitter's primary group is automatically added to this list.
    • Under Windows, the primary group is the first group found for the user by PBS when it queries the accounts database.
    • Format of group list: <group name>[@<hostname>][,<group name>@<hostname> ...]
    • Default: Login group name of job owner
  • pwd, pwd='', pwd="" — these forms prompt the user for a password. A space between W and pwd is optional. Spaces between the quotes are optional. Examples:
qsub ... -Wpwd <return>
qsub ... -W pwd='' <return>
qsub ... -W pwd="  " <return>

Available on supported Linux platforms only.

  • release_nodes_on_stageout=<value> — when set to True, all of the job's vnodes not on the primary execution host are released when stageout begins.
    • Cannot be used with vnodes tied to Cray X* series systems.
    • When cgroups is enabled and this is used with some but not all vnodes from one MoM, resources on those vnodes that are part of a cgroup are not released until the entire cgroup is released.
    • The job's stageout attribute must be set for the release_nodes_on_stageout attribute to take effect.
    • Format: Boolean
    • Default: False
  • run_count=<value> — sets the number of times the server thinks it has run the job. Sets the value of the job's run_count attribute to value.
    • Format: Integer greater than or equal to zero
  • sandbox=<sandbox spec> — determines which directory PBS uses for the job's staging and execution. Sets job's sandbox attribute to sandbox spec.
    • Allowed values for sandbox spec:
      • PRIVATE — PBS creates a job-specific directory for staging and execution.
      • HOME or unset — PBS uses the user's home directory for staging and execution.
    • Format: String
  • stagein=<path list>, stageout=<path list> — specifies files or directories to be staged in before execution or staged out after execution is complete. Sets the job's stagein and stageout attributes to the specified path lists. On completion of the job, all staged-in and staged-out files and directories are removed from the execution host(s). The path list has the form:
<file spec>[,<file spec>]

where file spec is:

<execution path>@<hostname>:<storage path>

regardless of the direction of the copy. The name execution path is the name of the file or directory on the primary execution host. It can be relative to the staging and execution directory on the execution host, or it can be an absolute path.

The "@" character separates execution path from storage path. 🦋

The name storage_path is the path on <hostname>. The storage_path can be absolute, or it can be relative to the user's home directory on hostname.

If path list has more than one file spec, i.e. it contains commas, it must be enclosed in double quotes.

If you use a UNC path, the hostname is optional. If you use a non-UNC path, the hostname is required.

  • umask=<mask value> — the umask with which the job is started. Sets job's umask attribute to mask value. Controls umask of job's standard output and standard error.
    • The following example allows group and world read on the job's output: -W umask=33
    • Format: One to four octal digits; typically two
    • Default value: 077
  • -X — allows user to receive X output from interactive job.
    • DISPLAY variable in submission environment must be set to desired display.
    • Can be used with interactive jobs only: must be used with one of the following:
      • -I
      • -W interactive=true (deprecated)
    • Cannot be used with -v DISPLAY.
    • When used with -Wblock=true, no exit status is returned.
    • Can be used with -V option.
    • Not available under Windows.
  • -z — job identifier is not written to standard output.
  • --version — the qsub command returns its PBS version information and exits. This option can only be used alone.

🫶 OPERANDS 🫶

The qsub command accepts as operands one of the following:

  • (no operands) — same as with a dash. Any PBS directives and user tasks are read from the command line.
  • <script> — path to script. Can be absolute or relative to current directory where qsub is run. The script must be the last argument to qsub.
  • - — when you use a dash, any PBS directives and user tasks are read from the command line.
  • -- <executable> [<arguments to executable>] — a single executable (preceded by two dashes) and its arguments.

The executable, and any arguments to the executable, are given on the qsub command line. The executable is preceded by two dashes, "--". All qsub options must come before the "--".

When you run qsub this way, it runs the executable directly. It does not start a shell, so no shell initialization scripts are run, and execution paths and other environment variables are not set. You should make sure that environment variables are set correctly.

💖 STANDARD OUTPUT 💖

  • Job ID for submitted jobs — if the job is successfully created
  • (No output) — if the -z option is set

😭 STANDARD ERROR 😭

The qsub command writes a diagnostic message to standard error for each error occurrence.

🌈 ENVIRONMENT VARIABLES 🌈

The qsub command uses the following environment variables:

  • PBS_DEFAULT — name of default server.
  • PBS_DPREFIX — prefix string which identifies PBS directives.

PBS automatically exports the following environment variables to the job. Environment variables beginning with "PBS_O_" are created by qsub and included in the job's Variable_List. ✨

  • PBS_ENVIRONMENT — set to PBS_BATCH for a batch job. Set to PBS_INTERACTIVE for an interactive job.
  • PBS_JOBDIR — pathname of job's staging and execution directory on the primary execution host.
  • PBS_JOBID — job identifier given by PBS when the job is submitted.
  • PBS_JOBNAME — job name specified by submitter.
  • PBS_NODEFILE — name of file containing the list of vnodes assigned to the job when the job runs.
  • PBS_O_HOME — user's home directory. Value of HOME taken from user's submission environment.
  • PBS_O_HOST — name of submit host. Value taken from user's submission environment.
  • PBS_O_LANG — value of LANG taken from user's submission environment.
  • PBS_O_LOGNAME — user's login name. Value of LOGNAME taken from user's submission environment.
  • PBS_O_MAIL — value of MAIL taken from user's submission environment.
  • PBS_O_PATH — user's PATH. Value of PATH taken from user's submission environment.
  • PBS_O_QUEUE — name of the queue to which the job was submitted. Value taken from job submission, otherwise default queue.
  • PBS_O_SHELL — value of SHELL taken from user's submission environment.
  • PBS_O_SYSTEM — operating system, from uname -s, on submit host. Value taken from user's submission environment.
  • PBS_O_TZ — timezone. Value taken from user's submission environment.
  • PBS_O_WORKDIR — absolute path to directory where qsub is run. Value taken from user's submission environment.
  • PBS_QUEUE — name of the queue from which the job is executed.
  • TMPDIR — pathname of job's scratch directory. Set when PBS assigns it.

💖 EXIT STATUS 💖

🌸 For non-blocking jobs 🌸

  • Zero — bestie, this is the good one: it means successful processing of input ✨
  • Greater than zero — okay so this means qsub failed 😭 no cap

🦋 For blocking jobs 🦋

  • Exit value of job — when the job runs successfully, you get the job's exit value 💅
  • 3 — if the job is deleted without being run. Periodt 💀

💀 Warning About Exit Status with csh 💀

🚨 csh besties, listen up: if a job is run in csh and a .logout file exists in the user's home directory on the host where the job executes, the exit status of the job is that of the .logout script, not the job script. This may impact any inter-job dependencies. Like… literally be aware 🙈

✨ SEE ALSO ✨

Go peek at the rest of the friend group, iykyk 🔥

  • pbs_job_attributes(7B)
  • pbs_server_attributes(7B)
  • pbs_resources(7B)
  • qalter(1B)
  • qhold(1B)
  • qmove(1B)
  • qmsg(1B)
  • qrerun(1B)
  • qrls(1B)
  • qselect(1B)
  • qstat(1B)

That's the whole vibe, bestie 🫶 this page's source/version is Local / qsub(1B), and it's dated 25 January 2021 — iconic, honestly 💜🎀

📦 where this page comes from