pbs_mom(8B)

📦 version 23.06.06-unstable-2026-04, in nixos 26.05, nixpkgs unstable

pbs_mom(8B) 💖

okay bestie, welcome to the PBS Professional manual page for pbs_mom(8B) — the PBS job monitoring and execution daemon. This is the whole lore, no skips, no cap.

🌸 NAME 🌸

pbs_mom — run the PBS job monitoring and execution daemon

✨ SYNOPSIS ✨

pbs_mom [-a <alarm timeout>] [-C <checkpoint directory>]
        [-c <config file>] [-d <MoM home directory>] [-L <logfile>]
        [-M <MoM service port>] [-N] [-n <nice value>] [-p|-r]
        [-R <inter-MoM communication port>] [-S <server port>]
        [-s <options>]
pbs_mom --version

🫶 DESCRIPTION 🫶

The pbs_mom command starts the PBS job monitoring and execution daemon, called MoM. Yes, she has a name, and she is basically the main character of job execution energy.

The standard MoM starts jobs on the execution host, monitors and reports resource usage, enforces resource usage limits, and notifies the server when the job is finished. The MoM also runs any prologue scripts before the job runs, and runs any epilogue scripts after the job runs. Iconic behavior, honestly.

The MoM performs any communication with job tasks and with other MoMs. The MoM on the first vnode on which a job is running manages communication with the MoMs on the remaining vnodes on which the job runs.

The MoM manages one or more vnodes. PBS may treat a host as a set of virtual nodes, in which case one MoM manages all of the host's vnodes. One MoM, many vnodes, she contains multitudes.

📝 Logging

The MoM's log file is in PBS_HOME/mom_logs. The MoM writes an error message in its log file when it encounters any error. If it cannot write to its log file, it writes to standard error. The MoM writes events to its log file. The MoM writes its PBS version and build information to the logfile whenever it starts up or the logfile is rolled to a new file. Receipts? Logged. Version info? Also logged. Periodt.

🔐 Required Permission

The executable for pbs_mom is in PBS_EXEC/sbin, and can be run only by root on Linux, and Admin on Windows. Root on Linux, Admin on Windows — that's the whole permission vibe check.

💻 HPE machine Running Supported Versions of Performance Software - Message Passing Interface 💻

A PBS job can run across multiple machines that run supported versions of Performance Software - Message Passing Interface.

PBS can run using HPE's MPI (MPT) over InfiniBand.

💅 Effect on Jobs of Starting MoM 💅

When MoM is started or restarted, her default behavior is to leave any running processes running, but to tell the PBS server to requeue the jobs she manages. MoM tracks the process ID of jobs across restarts.

In order to have all jobs killed and requeued, use the -r option when starting or restarting MoM.

In order to leave any running processes running, and not to requeue any jobs, use the -p option when starting or restarting MoM.

🔥 OPTIONS 🔥

  • -a <alarm timeout> — Number of seconds before alarm timeout. Whenever a resource request is processed, an alarm is set for the given amount of time. If the request has not completed before alarm timeout, the OS generates an alarm signal and sends it to MoM. Default: 10 seconds. Format: integer.
  • -C <checkpoint directory> — Specifies the path of the directory where MoM creates job-specific subdirectories used to hold each job's restart files. MoM passes this path to checkpoint and restart scripts. Overrides other checkpoint path specification methods. Any directory specified with the -C option must be owned, readable, writable, and executable by root only (rwx,---,---, or 0700), to protect the security of the checkpoint files. See the -d option. Format: string. Default: PBS_HOME/spool/checkpoint.
  • -c <config file> — MoM will read this alternate default configuration file upon starting. If this is a relative file name it will be relative to PBS_HOME/mom_priv. If the specified file cannot be opened, pbs_mom will abort. See the -d option.

MoM's normal operation, when the -c option is not given, is to attempt to open the default configuration file PBS_HOME/mom_priv/config. If this file is not present, pbs_mom will log the fact and continue. Lowkey graceful about it.

  • -d <MoM home directory> — Specifies the path of the directory to be used in place of PBS_HOME by pbs_mom. The default directory is given by $PBS_HOME. Format: string.
  • -L <logfile> — Specifies an absolute path and filename for the log file. The default is a file named for the current date in PBS_HOME/mom_logs/. See the -d option. Format: string.
  • -M <MoM port> — Specifies the number of the port on which MoM will listen for server requests and instructions. Overrides PBS_MOM_SERVICE_PORT setting in pbs.conf and environment variable. Default: 15002. Format: integer port number.
  • -n <nice value> — Specifies the priority for the pbs_mom daemon. Format: integer.
  • -N — Specifies that when starting, MoM should not detach from the current session.
  • -p — Specifies that when starting, MoM should allow any running jobs to continue running, and not have them requeued. This option can be used for single-host jobs only; multi-host jobs cannot be preserved. Cannot be used with the -r option. MoM is not the parent of these jobs.
  • -r — Specifies that when starting, MoM should requeue any rerunnable jobs and kill any non-rerunnable jobs that she was tracking, and mark the jobs as terminated. Cannot be used with the -p option. MoM is not the parent of these jobs.
It is not recommended to use the -r option after a reboot, because process IDs of new, legitimate tasks may match those MoM was previously tracking. If they match and MoM is started with the -r option, MoM will kill the new tasks. That's a big yikes — process ID matchmaking is NOT cute here.
  • -R <inter-MoM communication port> — Specifies the number of the port on which MoM will listen for pings, resource information requests, communication from other MoMs, etc. Overrides PBS_MANAGER_SERVICE_PORT setting in pbs.conf and environment variable. Default: 15003. Format: integer port number.
  • -S <server port> — Specifies the port number on which pbs_mom initially contacts the server. Default: 15001. Format: integer port number.
  • -s <file options> — If you are running the cgroups hook, make sure that the vnode names in any Version 2 configuration file exactly match those in the output of pbsnodes -av. This option lets you add, delete, and display MoM Version 2 configuration files. See CONFIGURATION FILES. Run this command at the host you want to change. The file options are used this way:
    • -s insert <Version 2 filename> <inputfile> — Reads inputfile and copies it to a Version 2 pbs_mom configuration file with the filename Version 2 filename. For example, to create a Version 2 file named "Myhost_V2":
pbs_mom -s insert <Myhost_V2> <myhost_v2_input>

If a configuration file with the specified Version 2 filename already exists, the operation fails, and pbs_mom writes a diagnostic and exits with a nonzero status. Configuration files whose names begin with the prefix "PBS" are reserved. You cannot add a file whose name begins with "PBS"; pbs_mom will print a diagnostic message and exit with a nonzero status. Reserved prefix means hands off, bestie.

  • -s remove <Version 2 filename> — Removes the configuration file named Version 2 filename if it exists. Example:
pbs_mom -s remove <Version 2 filename>

If the file does not exist or if you try to remove a file with the reserved "PBS" prefix, the operation fails, and pbs_mom prints a diagnostic and exits with a nonzero status.

  • -s show <Version 2 filename> — Prints the contents of the named file to standard output. Example:
pbs_mom -s show <Version 2 filename>

If Version 2 filename does not exist, the operation fails, and pbs_mom writes a diagnostic and exits with a nonzero status.

  • -s list — MoM lists the PBS-prefixed and site-defined configuration files in the order in which they are executed. Example:
pbs_mom -s list

🚨 WINDOWS 🚨

Under Windows, use the -N option so that pbs_mom will start up as a standalone program. For example:

pbs_mom -N -s insert <Version 2 filename> <inputfile>

or

pbs_mom -N -s list
  • --version — The pbs_mom command returns its PBS version information and exits. This option can only be used alone. Just her little version reveal moment, and nothing else.

🎀 CONFIGURATION FILES 🎀

okay so MoM's configuration info can live in configuration files of three types: default, PBS-prefixed, and site-defined. The default configuration file is usually PBS_HOME/mom_priv/config. The "PBS" prefix is reserved for files created by PBS. Site-defined configuration files are those created by the site administrator. MoM reads the configuration files at startup and reinitialization. The files are processed in this order:

  • The default configuration file
  • PBS-prefixed configuration files
  • Site-defined configuration files

The contents of a file read later override the contents of a file read earlier. For example, to change flags, create a script "update_flags" then use the -s insert option:

pbs_mom -s insert update_script update_flags

This adds the configuration file "update_script". Configuration files can be added, deleted and displayed using the -s option.

MoM's configuration files can use either the syntax shown below under Default Syntax and Contents or the syntax for describing vnodes shown in Vnode Syntax.

📍 Location 📍

The default configuration file is in PBS_HOME/mom_priv/.

PBS places PBS-prefixed and site-defined configuration files in an area that is private to each installed instance of PBS. This area is relative to the default PBS_HOME. Note that the -d option changes where MoM looks for PBS_HOME.

The -c option will change which default configuration file MoM reads.

Site-defined configuration files can be moved from one installed instance of PBS to another.

🚨 Do not move PBS-prefixed configuration files.

To move a set of site-defined configuration files from one installed instance of PBS to another:

  1. Use the -s list directive with the "source" instance of PBS to enumerate the site-defined files.
  2. Use the -s show directive with each site-defined file of the "source" instance of PBS to save a copy of that file.
  3. Use the -s insert directive with each file at the "target" instance of PBS to create a copy of each site-defined configuration file.

🦋 Vnode Configuration File Syntax and Contents 🦋

Configuration files with the following syntax describe vnodes and the resources available on them. They do not contain initialization values for MoM.

PBS-prefixed configuration files use the following syntax. Other configuration files can use the following syntax.

Any configuration file containing vnode-specific assignments must begin with this line:

$configversion 2

The format a file containing vnode information is:

<ID> : <ATTRNAME> = <ATTRVAL>

where

  • <ID> — sequence of characters not including a colon (":")
  • <ATTRNAME> — sequence of characters beginning with alphabetics or numerics, which can contain underscore ("_") and dash ("-")
  • <ATTRVAL> — sequence of characters not including an equal sign ("=")

The colon and equal sign may be surrounded by spaces.

A vnode's ID is an identifier that will be unique across all vnodes known to a given pbs_server and will be be stable across reinitializations or invocations of pbs_mom. ID stability is of importance when a vnode's CPUs or memory might be expected to change over time and PBS is expected to adapt to such changes by resuming suspended jobs on the same vnodes to which they were originally assigned. Vnodes for which this is not a consideration may simply use IDs of the form "0", "1", etc. concatenated with some identifier that ensures uniqueness across the vnodes served by the pbs_server.

A natural vnode does not correspond to any actual hardware. It is used to define any placement set information that is invariant for a given host, such as pnames.

It is defined as follows:

  • The name of the natural vnode is, by convention, the MoM contact name, which is usually the hostname. The MoM contact name is the vnode's MoM attribute. See the pbs_node_attributes(7B) man page.
  • An attribute, "pnames", with value set to the list of resource names that define the placement sets' types for this machine.
  • An attribute, "sharing" is set to the value "force_shared"

The natural vnode is used to define any placement set information that is invariant for a given host (e.g. the placement set resource names themselves).

The order of the pnames attribute follows placement set organization. If name X appears to the left of name Y in this attribute's value, an entity of type X may be assumed to be smaller (that is, be capable of containing fewer vnodes) than one of type Y. No such guarantee is made for specific instances of the types.

For example, on an HPE machine named "HostA", with two vnodes, a natural vnode, four processors and two cbricks, the description would look like this:

HostA:  pnames = cbrick
HostA:  sharing = force_shared
HostA[001c02#0]:  sharing = default_excl
HostA[001c02#0]:  resources_available.ncpus = 2
HostA[001c02#0]:  resources_available.cbrick = cbrick-0
HostA[001c02#0]:  resources_available.mem = 1968448kb
HostA[001c04#0]:  sharing = default_excl
HostA[001c04#0]:  resources_available.ncpus = 2
HostA[001c04#0]:  resources_available.cbrick = cbrick-1
HostA[001c04#0]:  resources_available.mem = 1961328kb

The natural vnode is described in the first two lines. The first vnode uses cbrick-0, and the second one uses cbrick-1.

🌸 Default Syntax and Contents 🌸

Configuration files with this syntax list local resources and initialization values for MoM. Local resources are either static, listed by name and value, or externally-provided, listed by name and command path. See the -c option.

Each configuration item is listed on a single line, with its parts separated by white space. Comments begin with a hashmark ("#").

⚠️ The default configuration file must be secure. It must be owned by a user ID and group ID both less than 10 and must not be world-writable.

✨ Externally-provided Resources ✨

Externally-provided resources use a shell escape to run a command. These resources are described with a name and value, where the first character of the value is an exclamation mark ("!"). The remainder of the value is the path and command to execute.

Parameters in the command beginning with a percent sign ("%") can be replaced when the command is executed. For example, this line in a configuration file describes a resource named "escape":

escape     !echo 0xx %yyy

If a query for the "escape" resource is sent with no parameter replacements, the command executed would be "echo 0xx %yyy". If one parameter replacement is sent, "escape[xxx=hi there]", the command executed would be "echo hi there %yyy". If two parameter replacements are sent, "escape[xxx=hi][yyy=there]", the command executed would be "echo hi there". If a parameter replacement is sent with no matching token in the command line, "escape[zzz=snafu]", an error is reported.

🦋 Windows Notes 🦋

If the argument to a MoM option is a pathname containing a space, enclose it in double quotes as in the following:

hostn !"\Program Files\PBS Pro\exec\bin\hostn" host

When you edit any PBS configuration file, make sure that you put a newline at the end of the file. The Notepad application does not automatically add a newline at the end of a file; you must explicitly add the newline.

🔥 Replacing Actions 🔥

$action <default action> <timeout> <new action>

Replaces the default action for an event with the site-specified new action. timeout is the time allowed for new action to run. The default action can be one of:

  • checkpoint — Run new action in place of the periodic job checkpoint, after which the job continues to run.
  • checkpoint_abort — Run new action to checkpoint the job, after which the job must be terminated by the script.
  • multinodebusy <timeout> requeue — Used with cycle harvesting and multi-vnode jobs. Changes default behavior when a vnode becomes busy. Instead of allowing the job to run, the job is requeued. timeout is ignored. The only new action is requeue.
  • restart — Runs new action in place of restart.
  • terminate — Runs new action in place of SIGTERM or SIGKILL when MoM terminates a job.

👑 MoM Parameters 👑

  • $alps_client <path> — Cray only. MoM runs this command to get the ALPS inventory. Must be full path to command. Format: path to command. Default: None.
  • alps_release_jitter <maximum jitter> — Cray only. PBS sends requests to ALPS to release a finished job at intervals specified by the sum of $alps_release_wait_time and a randomly generated value between zero and maximum jitter, in seconds. Format: Float. Default: 0.12 seconds.
  • $alps_release_timeout <timeout> — Cray only. Specifies the amount of time that PBS tries to release an ALPS reservation before giving up. After this amount of time has passed, PBS stops trying to release the ALPS reservation, the job exits, and the job's rsources are released. PBS sends a HUP to the MoM so that she re-reads the ALPS inventory to get the current available ALPS resources. We recommend that the value for this parameter be greater than the value for suspectbegin. Format: Seconds, specified as positive integer. Default: 600 (10 minutes).
  • $alps_release_wait_time <wait time> — Cray only. PBS sends requests to ALPS to release a finished job at intervals specified by the sum of wait time and a randomly generated value between zero and the maximum specified in $alps_release_jitter, in seconds. Format: Float. Default: 0.4 seconds.
  • $checkpoint_path <path> — MoM passes this path to checkpoint and restart scripts. This path can be absolute or relative to PBS_HOME/mom_priv. Overrides default. Overridden by pbs_mom -C option and by PBS_CHECKPOINT_PATH environment variable.
  • $clienthost <hostname> — hostname is added to the list of hosts which are allowed to connect to MoM as long as they are using a privileged port. For example, this allows the hosts "fred" and "wilma" to connect to MoM:
"$clienthost      fred"
"$clienthost      wilma"

So the following hostnames are added to $clienthost automatically: the server, the localhost, and if configured, the secondary server. The server sends each MoM a list of the hosts in the nodes file, and those get added internally to $clienthost. None of these hostnames need to be listed in the configuration file — not even one, bestie 💅.

Two hostnames are always allowed to connect to pbs_mom: "localhost" and the name returned to MoM by the system call gethostname(). These hostnames do not need to be listed in the configuration file either ✨.

The hosts listed as "clienthosts" make up a "sisterhood" of machines. Any one of the sisterhood will accept connections from within the sisterhood. The sisterhood must all use the same port number 👑.

$cputmult <factor> 🎀

This sets a factor used to adjust CPU time used by each job. It lets you adjust the time charged and the limits enforced when jobs run on a system with different CPU performance. If MoM's system is faster than the reference system, set factor to a decimal value greater than 1.0. For example:

$cputmult 1.5

If MoM's system is slower, set factor to a value between 1.0 and 0.0. For example:

$cputmult 0.75

$dce_refresh_delta <delta> 🦋

Defines the number of seconds between successive refreshings of a job's DCE login context. For example:

$dce_refresh_delta 18000

$enforce <limit> 🔥

MoM will enforce the given limit. Some limits have associated values. Syntax:

$enforce <variable name> <value>
  • $enforce mem — MoM will enforce each job's memory limit.
  • $enforce cpuaverage — MoM will enforce ncpus when the average CPU usage over a job's lifetime usage is greater than the job's limit.
    • $enforce average_trialperiod <seconds> — Modifies cpuaverage. Minimum number of seconds of job walltime before enforcement begins. Default: 120. Integer.
    • $enforce average_percent_over <percentage> — Modifies cpuaverage. Gives percentage by which a job may exceed its ncpus limit. Default: 50. Integer.
    • $enforce average_cpufactor <factor> — Modifies cpuaverage. The ncpus limit is multiplied by factor to produce the actual limit. Default: 1.025. Float.
  • $enforce cpuburst — MoM will enforce the ncpus limit when CPU burst usage exceeds the job's limit.
    • $enforce delta_percent_over <percentage> — Modifies cpuburst. Gives percentage over limit to be allowed. Default: 50. Integer.
    • $enforce delta_cpufactor <factor> — Modifies cpuburst. The ncpus limit is multiplied by factor to produce the actual limit. Default: 1.5. Float.
    • $enforce delta_weightup <factor> — Modifies cpuburst. Weighting factor for smoothing burst usage when average is increasing. Default: 0.4. Float.
    • $enforce delta_weightdown <factor> — Modifies cpuburst. Weighting factor for smoothing burst usage when average is decreasing. Default: 0.4. Float.

$ideal_load <load> 🌸

Defines the load below which the vnode is not considered to be busy. Used with the $max_load directive. No default. Float.

Example:

$ideal_load 1.8

Use of $ideal_load adds a static resource to the vnode called "ideal_load", which is only internally visible 👀.

$jobdir_root <stage directory root> 🎀

Directory under which PBS creates job-specific staging and execution directories. PBS creates a job's staging and execution directory when the job's sandbox attribute is set to PRIVATE.

If $jobdir_root is unset, it defaults to the job owner's home directory. In this case the user's home directory must exist. If stage_directory_root does not exist when MoM starts up, MoM will abort. If stage directory root does not exist when MoM tries to run a job, MoM will kill the job 💀. Path must be owned by root, and permissions must be 1777. On Windows, this directory should have Full Control Permission for the local Administrators group.

When you set $jobdir_root to a shared (e.g. NFS) directory, tell MoM it is shared by setting the shared directive after the directory name:

$jobdir_root <directory name> shared

Otherwise sister MoMs can prematurely delete files and directories when nodes are released. This is because when a job's sandbox attribute is set to PRIVATE and $jobdir_root is set to a shared directory, PBS can use a shared location for job files. When sister nodes are released, those sister MoMs would normally clean up their own files upon release. So if $jobdir_root is set to a shared directory, you need to tell the sister MoMs not to do the cleanup, and let the primary execution host MoM clean up when the job is finished 🫶.

Example of a shared directory:

$jobdir_root /r/shared shared

Example of a non-shared directory:

$jobdir_root /scratch/foo

$job_launch_delay ⭐

When the primary MoM gets a job whose tolerate_node_failures attribute is set to all or job_start, the primary MoM can wait to start the job (running the job script or executable) for up to a configured number of seconds. During this time, execjob_prologue hooks can finish and the primary MoM can check for communication problems with sister MoMs. You configure the number of seconds for the primary MoM to wait for hooks via the job_launch_delay configuration parameter in MoM's config file:

$job_launch_delay <number of seconds to wait>

Default: the sum of the values of the alarm attributes of any enabled execjob_prologue hooks. If there are no enabled execjob_prologue hooks, the default value is 30 seconds. For example, if there are two enabled execjob_prologue hooks, one with alarm = 30 and one with alarm = 60, the default value of MoM's job_launch_delay is 90 seconds.

After all the execjob_prologue hooks have finished, or MoM has waited for the value of the job_launch_delay parameter, she starts the job 💅.

$kbd_idle <idle wait> <min use> <poll interval> 🦋

Declares that the vnode will be used for batch jobs during periods when the keyboard and mouse are not in use.

The vnode must be idle for a minimum of idle wait seconds before being considered available for batch jobs. No default. Integer.

The vnode must be in use for a minimum of min use seconds before it becomes unavailable for batch jobs. Default: 10. Integer.

Mom checks for activity every poll interval seconds. Default: 1. Integer.

Example:

$kbd_idle 1800 10 5

$logevent <mask> 🌈

Sets the mask that determines which event types are logged by pbs_mom. To include all debug events, use 0xffffffff.

Log events:

  • ERROR — hex 0001 — Internal errors
  • SYSTEM — hex 0002 — System errors
  • ADMIN — hex 0004 — Administrative events
  • JOB — hex 0008 — Job-related events
  • JOB_USAGE — hex 0010 — Job accounting info
  • SECURITY — hex 0020 — Security violations
  • SCHED — hex 0040 — Scheduler events
  • DEBUG — hex 0080 — Common debug messages
  • DEBUG2 — hex 0100 — Uncommon debug messages
  • RESV — hex 0200 — Reservation-related info
  • DEBUG3 — hex 0400 — Rare debug messages
  • DEBUG4 — hex 0800 — Limit-related messages

$max_check_poll <seconds> 🎀

Maximum time between polling cycles, in seconds. Minimum recommended value: 30 seconds.

The interval between each poll starts at $min_check_poll and increases with each cycle until it reaches $max_check_poll, after which it remains the same. The amount by which the cycle increases is 1/20 of the difference between $max_check_poll and $min_check_poll.

Format: Integer Minimum value: 1 second Default value: 120 seconds

$max_load <load> [suspend] 🔥

Defines the load above which the vnode is considered to be busy. Used with the $ideal_load directive. No new jobs are started on a busy vnode.

The optional suspend directive tells PBS to suspend jobs running on the vnode if the load average exceeds the $ max_load number, regardless of the source of the load (PBS and/or logged-in users). Without this directive, PBS will not suspend jobs due to load.

We recommend setting load to a value that is slightly higher than the number of CPUs, for example 0.25 + ncpus.

Default: number of CPUs on machine Format: Float Example:

$max_load 3.5

$max_poll_downtime 😭

When mother superior detects that a sister mom has lost connectivity (e.g. MoM went down or the network is having problems) it waits downtime seconds for the sister to reconnect before it gives up and kills the job.

Format: Integer Default: Five minutes

memreserved <megabytes> 💀

Deprecated. Bestie, this one is old-school.

The amount of per-vnode memory reserved for system overhead. This much memory is deducted from the value of resources_available.mem for each vnode managed by this MoM.

Example:

memreserved 16

Default: 0MB

$min_check_poll <seconds> 🌸

Minimum time between polling cycles, in seconds. Must be greater than zero and less than $max_check_poll. Minimum recommended value: 10 seconds.

Format: Integer. Minimum value: 1 second Default value: 10 seconds

pbs_accounting_workload_mgmt <value> 👑

Controls whether CSA accounting is enabled. Name does not start with dollar sign. If set to 1, on, or true, CSA accounting is enabled. If set to 0, off, or false, accounting is disabled. Cray only. Requires CLE 5.2.

Default: "true"; enabled

$prologalarm <timeout> 🫶

Defines the maximum number of seconds the prologue and epilogue may run before timing out. Default: 30 seconds. Integer. Example:

$prologalarm 30

okay so, final stretch of the MoM parameters — she’s still iconic, and these last ones are lowkey important. 💅

  • $reject_root_scripts <True | False> — when this is True, MoM is on a strict no-new-root-hooks vibe: she won’t acquire any new hook scripts, and she won’t run job scripts that would execute as root or Admin. However (plot twist 🫶), MoM will still run previously-acquired hooks that run as root.
    • Format: Boolean
    • Default: False
  • $restart_background <True | False> — controls how MoM runs a restart script after checkpointing a job. When this is True, MoM forks a child that runs the restart script. The child returns when all restarts for all the local tasks of the job are done. MoM does not block on the restart. When this is False, MoM runs the restart script and waits for the result. 😭
    • Format: Boolean
    • Default: False
  • $restart_transmogrify <True | False> — also controls how MoM runs a restart script after checkpointing a job. When this is True, MoM runs the restart script and replaces the session ID of the original task’s top process with the session ID of the script. ✨

When this is False, MoM runs the restart script and waits for the result. The restart script must restore the original session ID for all the processes of each task so that MoM can continue to track the job.

When this is False and the restart uses an external command, the configuration parameter restart_background is ignored and treated as if it were set to True, preventing MoM from blocking on the restart. 💀

  • Format: Boolean
  • Default: False
  • $restrict_user <True | False> — controls whether users who are not submitting jobs have access to this machine. If the value is True, restrictions are applied. See $restrict_user_exceptions and $restrict_user_maxsysid. Not supported on Windows. 🙈
    • Format: Boolean
    • Default: False
  • $restrict_user_exceptions <user list> — comma-separated list of users who are exempt from the access restrictions applied by $restrict_user. Leading spaces within each entry are allowed. Maximum of 10 names. 🎀
  • $restrict_user_maxsysid <value> — any user with a numeric user ID less than or equal to value is exempt from restrictions applied by $restrict_user. If $restrict_user is True and no value exists for $restrict_user_maxsysid, PBS looks in /etc/login.defs, if it exists, for the value. Otherwise the default is used.
    • Format: Integer
    • Default: 999
  • $restricted <hostname> — the hostname is added to the list of hosts which are allowed to connect to MoM without being required to use a privileged port. Hostnames can be wildcarded. For example, to allow queries from any host from the domain "xyz.com": $restricted *.xyz.com

Queries from the hosts in the restricted list are only allowed access to information internal to this host, such as load average, memory available, etc. They may not run shell commands. 🚫

  • $sister_join_job_alarm — when the primary MoM gets a job whose tolerate_node_failures attribute is set to all or job_start, the primary MoM can wait to start the job for up to a configured number of seconds if the sister MoMs do not immediately acknowledge joining the job. This gives the sister MoMs more time to join the job. You configure the number of seconds for the primary MoM to wait for sister MoMs via the sister_join_job_alarm configuration parameter in MoM’s config file: $sister_join_job_alarm <number of seconds to wait>

Default: the sum of the values of the alarm attributes of any enabled execjob_begin hooks. If there are no enabled execjob_begin hooks, the default value is 30 seconds. For example, if there are two enabled execjob_begin hooks, one with alarm = 30 and one with alarm = 20, the default value of MoM’s sister_join_job_alarm is 50 seconds. 🧮

After all the sister MoMs have joined the job, or MoM has waited for the value of the $sister_join_job_alarm parameter, she starts the job. 👑

  • $suspendsig <suspend signal> [resume signal] — alternate suspend signal is used to suspend jobs instead of SIGSTOP. Optional resume signal is used to resume jobs instead of SIGCONT. 🦋
  • $tmpdir <directory> — location where each job’s scratch directory will be created. PBS creates a temporary directory for use by the job, not by PBS. PBS creates the directory before the job is run and removes the directory and its contents when the job is finished. It is scratch space for use by the job. Permission must be 1777 on Linux, writable by Everyone on Windows. 🍓

Example: $tmpdir /memfs

  • Default on Linux: /var/tmp
  • Default on Windows: value of the TMP environment variable
  • $usecp <hostname:source prefix> <destination prefix> — MoM uses /bin/cp or the program specified by PBS_CP to deliver output files when the destination is a network mounted file system, or when the source and destination are both on the local host, or when the source prefix can be replaced with the destination prefix on hostname. Both source prefix and destination prefix are absolute pathnames of directories, not files. Overrides PBS_CP and PBS_RCP and PBS_SCP. Use trailing slashes on both source and destination. For example: $usecp HostA:/users/work/myproj/ /sharedwork/proj_results/ 💾
  • $vnodedef_additive — specifies whether MoM considers a vnode that appeared previously either in the inventory or in a vnode definition file, but that does not appear now, to be in her list of vnodes. When $vnodedef_additive is True, MoM treats missing vnodes as if they are still present, and continues to report them as if they are present. This means that the server does not mark missing vnodes as stale. When $vnodedef_additive is False, MoM does not list missing vnodes, the server’s information is brought up to date with the inventory and vnode definition files, and the server marks missing vnodes as stale. Visible in configuration file on Cray only. 🌈
    • Format: Boolean
    • Default for MoM on Cray login node: False
  • $wallmult <factor> — each job’s walltime usage is multiplied by factor. For example: $wallmult 1.5 ⏱️

🧸 Static Resources 🧸

Static resources local to the vnode are described one resource to a line, with a name and value separated by white space. For example, tape drives of different types could be specified by:

tape3480    4
tape3420    2
tapedat     1
tape8mm     1

📁 FILES AND DIRECTORIES 📁

  • $PBS_HOME/mom_priv — default directory for default configuration files.
  • $PBS_HOME/mom_priv/config — MoM’s default configuration file.
  • $PBS_HOME/mom_logs — default directory for log files written by MoM.
  • $PBS_HOME/mom_priv/prologue — file containing administrative script to be run before job execution.
  • $PBS_HOME/mom_priv/epilogue — file containing administrative script to be run after job execution.

🚨 SIGNAL HANDLING 🚨

pbs_mom handles the following signals:

  • SIGHUP — the pbs_mom daemon rereads its configuration files, closes and reopens the log file, and reinitializes resource structures. 🔄
  • SIGALRM — MoM writes a log file entry. See the -a alarm_timeout option.
  • SIGINT — the pbs_mom daemon exits, leaving all running jobs still running. See the -p option. 😭
  • SIGKILL — this signal is not caught. The pbs_mom daemon exits immediately. 💀
  • SIGTERM, SIGXCPU, SIGXFSZ, SIGCPULIM, SIGSHUTDN — the pbs_mom daemon terminates all running children and exits.
  • SIGPIPE, SIGUSR1, SIGUSR2, SIGINFO — these are ignored. 🙈
  • All other signals have their default behavior installed.

🚪 EXIT STATUS 🚪

Greater than zero — if any of these happen, MoM is not having a good time:

  • If the pbs_mom daemon fails to start
  • If the -s insert option is used with an existing Version 2 filename
  • If the administrator attempts to add a script whose name begins with "PBS"
  • If the administrator attempts to use the -s remove option on a nonexistent configuration file, or on a configuration file whose name begins with "PBS"
  • If the administrator attempts to use the -s show option on a nonexistent script

👀 SEE ALSO 👀

  • pbs_server(8B)
  • pbs_sched(8B)
  • qstat(1B)

that’s the whole pbs_mom(8B) page, bestie — you ate. 💅 the footer energy is: source Local, date 1 March 2021, and the page is pbs_mom(8B). periodt. 🎀✨

📦 where this page comes from