pbs_sched(8B)

💖 NAME 💖

okay so bestie, this is pbs_sched(8B) from PBS Professional — the manual page for pbs_sched, aka run a PBS scheduler. 🫶 main character energy for your cluster, no cap.

---

✨ SYNOPSIS ✨

the whole invocation vibe, exactly:

pbs_sched [-a <alarm>] [-c <clientsfile>] [-d <home dir>]
          [-I <scheduler name>] [-L <logfile>] [-n] [-N]
          [-p <output file>] [-R <port number>] [-S <port number>]
          [-t <num threads>]

pbs_sched --version

🌸 DESCRIPTION 🌸

pbs_sched runs the default scheduler or a multisched. that's the whole job 💅

⚠️ important: pbs_sched must be executed with root permission. like, actually root. no cap.

💅 OPTIONS 💅

• -a <alarm> — Deprecated. Overwrites value of sched_cycle_length scheduler attribute. Time in seconds to wait for a scheduling cycle to finish. Format: Time, in seconds.
• -c <clientsfile> — Add clients to this scheduler's list of known clients. The clientsfile contains single-line entries of the form $clienthost <hostname>. Each hostname is added to the list of hosts allowed to connect to this scheduler. If clientsfile cannot be opened, this scheduler aborts. Path can be absolute or relative. If relative, it is relative to PBS_HOME/sched_priv.
• -d <home dir> — The directory in which this scheduler will run. Default: PBS_HOME/sched_priv.
• -I <scheduler name> — Name of scheduler to start. Required when starting a multisched.
• -L <logfile> — The absolute path and filename of the log file. A scheduler writes its PBS version and build information to the logfile whenever it starts up or the logfile is rolled to a new file. See the -d option. Default: A scheduler opens a file named for the current date in the PBS_HOME/sched_logs directory.
• -n — Tells this scheduler to not restart itself if it receives a sigsegv or a sigbus. A scheduler by default restarts itself if it receives either of these two signals more than five minutes after starting. A scheduler does not restart itself if it receives either one within five minutes of starting.
• -N — Runs the scheduler in standalone mode.
• -p <output file> — Any output which is written to standard out or standard error is written to output file. The pathname can be absolute or relative, in which case it is relative to PBS_HOME/sched_priv. See the -d option. Default: PBS_HOME/sched_priv/sched_out.
• -R <port number> — The port for MOM to use. If this option is not given, the port number is taken from PBS_MANAGER_SERVICE_PORT, in pbs.conf. Default: 15003.
• -S <port number> — The port for this scheduler to use. Required when starting a multisched. For the default scheduler, if this option is not specified the default port is taken from PBS_SCHEDULER_SERVICE_PORT, in pbs.conf. Default value for default scheduler: 15004. Default value for multisched: none.
• -t <num threads> — Specifies number of threads for this scheduler. Scheduler automatically caps number of threads at the number of cores (or hyperthreads if applicable), regardless of value of num threads. Overrides PBS_SCHED_THREADS environment variable and PBS_SCHED_THREADS parameter in pbs.conf. Valid values: >=1. Default: half the number of cores (or hyperthreads if applicable) on this host.
• --version — The pbs_sched command returns its PBS version information and exits. This option can only be used alone.

🧚 CONFIGURATION FILE 🧚

The file PBS_HOME/sched_priv/sched_config contains configuration parameters for this scheduler. Each entry must be a single unbroken line. Format: name: value [prime | non-prime | all | none] where

• name — Must not contain whitespace.
• value — Must be double-quoted if it contains whitespace. value can be True | False | <number> | <string>. True and False are not case-sensitive.
• [prime | non-prime | all | none] — Specifies when this setting applies: during primetime, non-primetime, all the time, or none of the time. A blank third field is equivalent to all which means that it applies to both primetime and non-primetime. Valid values: all, ALL, none, NONE, prime, PRIME, non_prime, NON_PRIME.

Any line starting with a hashmark, #, is a comment, and is ignored.

💫 Configuration Parameters 💫

• backfill — Deprecated. Use the backfill_depth queue/server attribute instead. Toggle that controls whether PBS uses backfilling. If this is set to True, this scheduler attempts to schedule smaller jobs around higher-priority jobs when using strict_ordering, as long as running the smaller jobs won't change the start time of the jobs they were scheduled around. A scheduler chooses jobs in the standard order, so other high-priority jobs will be considered first in the set to fit around the highest-priority job. When this parameter is True and help_starving_jobs is True, this scheduler backfills around starving jobs. Can be used with strict_ordering and help_starving_jobs. Format: Boolean. Default: True all.
• backfill_prime — This Scheduler will not run jobs which would overlap the boundary between primetime and non-primetime. This assures that jobs restricted to running in either primetime or non-primetime can start as soon as the time boundary happens. See also prime_spill, prime_exempt_anytime_queues. Format: Boolean. Default: False all.
• by_queue — If set to True, all jobs that can be run from the highest-priority queue are run, then any jobs that can be run from the next queue are run, and so on. If by_queue is set to False, all jobs are treated as if they are in one large queue. The by_queue parameter is overridden by the round_robin parameter when round_robin is set to True. Format: Boolean. Default: True all.
• cpus_per_ssinode — Obsolete. literally that's it.
• dedicated_prefix — Queue names with this prefix are treated as dedicated queues, meaning jobs in that queue are considered for execution only when the system is in dedicated time as specified in the configuration file PBS_HOME/sched_priv/dedicated_time. Format: String. Default: ded.
• fair_share — Enables the fairshare algorithm, and turns on usage collecting. Jobs will be selected based on a function of their recent usage and priority (shares). Not a prime option. Format: Boolean. Default: False all.
• fairshare_decay_factor — Decay multiplier for fairshare usage reduction. Each decay period, the usage is multiplied by this value. Valid values: between 0 and 1, not inclusive. Not a prime option. Format: Float. Default: 0.5.
• fairshare_decay_time — Time between fairshare usage decay operations. Not a prime option. Format: Duration. Default: 24:00:00.
• fairshare_entity — Specifies the entity for which fairshare usage data will be collected. Can be euser, egroup, Account_Name, queue, or egroup:euser. Not a prime option. Format: String. Default: euser.
• fairshare_enforce_no_shares — If this option is enabled, jobs whose entity has zero shares will never run. Requires fair_share parameter to be enabled. Format: Boolean. Default: False.
• fairshare_usage_res — Specifies the mathematical formula to use in fairshare calculations. Is composed of PBS resources as well as mathematical operations that are standard Python operators and/or those in the Python math module. When using a PBS resource, if resources_used.<resource name> exists, that value is used. Otherwise, the value is taken from Resource_List.<resource name>. Not a prime option. Format: String. Default: cput.
• half_life — Deprecated (as of 13.0). The half-life for fairshare usage; after the amount of time specified, the fairshare usage is halved. Requires that fair_share parameter be enabled. Not a prime option. Format: Duration. Default: 24:00:00.
• help_starving_jobs — Setting this option enables starving job support. Once jobs have waited for the amount of time given by max_starve they are considered starving. If a job is considered starving, no lower-priority jobs will run until the starving job can be run, unless backfilling is also specified. To use this option, the max_starve configuration parameter needs to be set as well. See also backfill, max_starve, and the server's eligible_time_enable attribute. At each scheduler iteration, PBS calculates estimated.start_time and estimated.exec_vnode for starving jobs being backfilled around. Format: Boolean. Default: True all.
• job_sort_key — Specifies how jobs should be sorted. job_sort_key can be used to sort either by resources or by special case sorting routines. Multiple job_sort_key entries can be used, one to a line, in which case the first entry will be the primary sort key, the second will be used to sort equivalent items from the first sort, etc. The HIGH option implies descending sorting, LOW implies ascending. See example for details.

okay so, continuing right in the middle of job_sort_key: also bestie, this attribute is overridden by the job_sort_formula attribute. If both are set, job_sort_key is ignored and an error message is printed. 💅

Syntax:

job_sort_key: "<resource name> HIGH|LOW"
job_sort_key: "fairshare_perc HIGH|LOW"
job_sort_key: "job_priority HIGH|LOW"

There are three special case sorting routines, which can be used instead of <resource name>:

• fairshare_perc HIGH — sort based on how much fairshare percentage the entity deserves, based on the values in the resource group file. If user A has more priority than user B, all of user A's jobs will always run first. Past history is not used. This should only be used if entity share (strict priority) sorting is needed. Incompatible with the fair_share scheduling parameter being True.
• job_priority HIGH | LOW — sort jobs by the job priority attribute regardless of job owner.
• sort_priority HIGH|LOW — deprecated. See job_priority above.

The following example shows how to sort jobs so that those with high CPU count come first:

job_sort_key: "ncpus HIGH" all

The following example shows how to sort jobs so that those with lower memory come first:

job_sort_key: "mem LOW" prime

Format: Quoted string Default: Not in force

💖 load_balancing

When set to True, this scheduler takes into account the load average on vnodes as well as the resources listed in the resources line in sched_config. Load balancing can result in overloaded CPUs. 😭

Format: Boolean Default: False all

🦋 load_balancing_rr

Deprecated. To duplicate this setting, enable load_balancing and set smp_cluster_dist to round_robin.

✨ log_filter

Obsolete. See the log_events scheduler attribute.

🌸 max_starve

The amount of time before a job is considered starving. This variable is used only if help_starving_jobs is set.

Upper limit: None Format: Duration Default: 24:00:00

🎀 mem_per_ssinode

Obsolete.

🧚 mom_resources

This option is used to query the MoMs to set the value of resources_available.<resource name> where resource name is a site-defined resource. Each MoM is queried with the resource name and the return value is used to replace resources_available.<resource name> on that vnode. On a multi-vnoded machine with a natural vnode, all vnodes share anything set in mom_resources.

Format: String

💀 node_sort_key

Defines sorting on resource or priority values on vnodes. Resource must be numerical, for example, long or float. Up to 20 node_sort_key entries can be used, in which case the first entry will be the primary sort key, the second will be used to sort equivalent items from the first sort, etc.

Syntax:

node_sort_key: <resource name>|sort_priority <HIGH | LOW>
node_sort_key: <resource name> <HIGH | LOW> <total | assigned | unused>

where:

• total — use the resources_available value.
• assigned — use the resources_assigned value.
• unused — use the value given by resources_available - resources_assigned.
• sort_priority — sort vnodes by the value of the vnode priority attribute.

When sorting on a resource, the default third field is total.

Format: String Default: node_sort_key: sort_priority HIGH all

🌈 nonprimetime_prefix

Queue names which start with this prefix are treated as non-primetime queues. Jobs in these queues run only during non-primetime. Primetime and non-primetime are defined in the holidays file.

Format: String Default: np_

👑 peer_queue

Defines the mapping of a pulling queue to a furnishing queue for peer scheduling. Maximum number is 50 peer queues per scheduler.

Format: String Default: Unset

🔥 preemptive_sched

Enables job preemption. See the preempt_order scheduler attribute.

Format: String Default: True all

🙈 preempt_order

Obsolete. See the preempt_order scheduler attribute.

💀 preempt_prio

Obsolete. See the preempt_prio scheduler attribute.

🫶 preempt_queue_prio

Obsolete. See the preempt_queue_prio scheduler attribute.

⭐ preempt_sort

Obsolete. See the preempt_sort scheduler attribute.

🎀 primetime_prefix

Queue names starting with this prefix are treated as primetime queues. Jobs in these queues run only during primetime. Primetime and non-primetime are defined in the holidays file.

Format: String Default: p_

🦋 prime_exempt_anytime_queues

Determines whether anytime queues are controlled by backfill_prime. If set to true, jobs in an anytime queue are not prevented from running across a primetime/nonprimetime or non-primetime/primetime boundary. If set to false, the jobs in an anytime queue may not cross this boundary, except for the amount specified by their prime_spill setting. See also backfill_prime, prime_spill.

Format: Boolean Default: False

🌸 prime_spill

Specifies the amount of time a job can spill over from non-primetime into primetime or from primetime into non-primetime. This option is only meaningful if backfill_prime is True. This option can be separately specified for primetime and non-primetime. See also backfill_prime, prime_exempt_anytime_queues.

For example, non-primetime jobs can spill into primetime by 1 hour:

prime_spill: 1:00:00 prime

For example, jobs in either prime/non-prime can spill into the other by 1 hour:

prime_spill: 1:00:00 all

Format: Duration Default: 00:00:00

✨ provision_policy

Specifies how vnodes are selected for provisioning. Can be set by Manager only; readable by all. Can be set to one of the following:

• avoid_provision — PBS first tries to satisfy the job's request from free vnodes that already have the requested AOE instantiated. PBS uses node_sort_key to sort these vnodes. If it cannot satisfy the job's request using vnodes that already have the requested AOE instantiated, PBS uses the server's node_sort_key to select the free vnodes that must be provisioned in order to run the job, choosing from any free vnodes, regardless of which AOE is instantiated on them. Of the selected vnodes, PBS provisions any that do not have the requested AOE instantiated on them.
• aggressive_provision — PBS selects vnodes to be provisioned without considering which AOE is currently instantiated. PBS uses the server's node_sort_key to select the vnodes on which to run the job, choosing from any free vnodes, regardless of which AOE is instantiated on them. Of the selected vnodes, PBS provisions any that do not have the requested AOE instantiated on them.

Format: string Default: aggressive_provision

💚 resources

Specifies those resources which are not to be over-allocated, or if Boolean, are to be honored, when scheduling jobs. Vnode-level Boolean resources are automatically enforced and do not need to be listed here. Limits are set by setting resources_available.<resource name> on vnodes, queues, and the server. A scheduler considers numeric (integer or float) items as consumable resources and ensures that no more are assigned than are available (e.g. ncpus or mem). Any string resources are compared using string comparisons (e.g. arch).

If "host" is not added to the resources line, then when the user submits a job requesting a specific vnode in the following syntax:

qsub -l select=host=vnodeName

the job will run on any host.

Format: String Default: ncpus, mem, arch, host, vnode, aoe

🍓 resource_unset_infinite

Resources in this list are treated as infinite if they are unset. Cannot be set differently for primetime and non-primetime.

Example: resource_unset_infinite: vmem, foo_licenses

Format: Comma-delimited list of resources Default: Empty list

🔄 round_robin

If set to True, this scheduler considers one job from the first queue, then one job from the second queue, and so on in a circular fashion. The queues are ordered with the highest-priority queue first. Each scheduling cycle starts with the same highest-priority queue, which will therefore get preferential treatment. 👑

...and if there are groups of queues that all share the same priority, and this parameter is set to True, this scheduler round-robins through each group of queues before it moves on to the next group. 🌀

If round_robin is set to False, then this scheduler considers jobs according to whatever the by_queue parameter says. And when it's True, it overrides by_queue — yeah, it said what it said. 💅

Format: Boolean Default: False all

server_dyn_res 🫶

This one directs the scheduler to replace the server's resources_available values with brand-new values that get returned by a site-specific external program. So like, your server's resources get a whole glow-up from outside. ✨

Format: String Default: Unset

smp_cluster_dist 💀

Deprecated as of 12.2, bestie, so don't come for me. It specifies how single-host jobs should be distributed to all hosts of the complex. The options are:

• pack — keep putting jobs onto one host until it's full, then move on to the next one.
• round_robin — put one job on each vnode in turn before cycling back to the first one.
• lowest_load — put the job on the lowest-loaded host.

Format: String Default: pack all

sort_queues 🧚

Obsolete. That's it, that's the whole thing. She's retired. 👋

strict_fifo

Deprecated. Use strict_ordering instead, okurrr? 🙈

strict_ordering 👑

This specifies that jobs must be run in the order determined by whatever sorting parameters are being used. Which means a job cannot be skipped just because the resources it needs aren't available — nope, no cutting the line. If the job that's due to run next can't run, then no job will run, unless backfilling is being used, in which case jobs can be backfilled around the job that's due to run next. Main character scheduling energy, honestly. 💅

Example line in PBS_HOME/sched_priv/sched_config:

strict_ordering: true ALL

Format: Boolean Default: False all

sync_time

Obsolete. Also retired, also living her best quiet life. 🌸

unknown_shares 🎀

The number of shares for the unknown group. These shares determine the portion of a resource that gets allotted to that group via fairshare. Heads up: this one requires fair_share to be enabled, so don't forget that part. 🔥

Format: Integer Default: The unknown group gets 0 shares

---

✨ FORMATS ✨

Okay so here's the lil cheat sheet of every format type the config file can throw at you. Save this, screenshot it, tattoo it, whatever. 🦋

• Boolean — allowable values (case insensitive): True|T|Y|1|False|F|N|0
• Duration — a period of time. Expressed either as:
  • integer seconds
  • or [[hours:]minutes:]seconds[.milliseconds]

Milliseconds are rounded to the nearest second. Don't @ me, that's just how it is. ⏱️

• Float — allowable values: [+-] 0-9 [[0-9] ...][.][[0-9] ...]
• Long — long integer. Allowable values: 0-9 [[0-9] ...], and + and -
• Size — number of bytes or words. The size of a word is 64 bits. 💾

Format: <integer>[<suffix>]

where suffix can be:

• b or w — bytes or words
• kb or kw — Kilobytes or kilowords (2 to the 10th, or 1024)
• mb or mw — Megabytes or megawords (2 to the 20th, or 1,048,576)
• gb or gw — Gigabytes or gigawords (2 to the 30th, or 1,073,741,824)
• tb or tw — Terabytes or terawords (2 to the 40th, or 1024 gigabytes)
• pb or pw — Petabytes or petawords (2 to the 50th, or 1,048,576 gigabytes)

Default: bytes

• String — (resource value) any character, including the space character. Only one of the two types of quote characters, " or ', may appear in any given value. 👀

Allowable values:

[_a-zA-Z0-9][[-_a-zA-Z0-9 !"#$%'
()*+,-./:;<=>?@[\]^_{|}~] ...]

String resource values are case-sensitive, ngl. Watch those caps. 🔠

---

🌈 FILES 🌈

• $PBS_HOME/sched_priv is the default directory for configuration files.
• $PBS_HOME/sched_priv/holidays is the holidays file.

---

💅 SIGNAL HANDLING 💅

So like, all signals are ignored until the end of the cycle. Most signals get handled in the standard UNIX fashion, periodt.

• SIGHUP — this scheduler closes and reopens its log file, and rereads its configuration file if one exists. It's giving "fresh start, new me". ✨
• SIGALRM, SIGBUS, etc. — ignored until end of scheduling cycle. Then this scheduler quits. Bye bestie. 💀
• SIGINT and SIGTERM — this scheduler closes its log file and shuts down. 🫶

---

⭐ EXIT STATUS ⭐

Zero upon normal termination. That's the whole vibe. 🎀

---

👑 SEE ALSO 👑

• pbs_server(8B)
• pbs_mom(8B)

---

okay that's the whole page bestie, you're officially a scheduling expert 💖 this little text came from <rainbow>Local</rainbow> and it's dated 6 May 2020, filed under pbs_sched(8B) like the icon it is. mwah, go schedule something gorgeous 🦋