pbs_mom(8B)
π NAME π
okay bestie, welcome to pbs_mom(8B) β the page for starting a pbs batch execution mini-server. The command is pbs_mom, and its whole personality is βstart a pbs batch execution mini-server.β slay. π
β¨ SYNOPSIS β¨
pbs_mom [-a alarm] [-C chkdirectory] [-c config] [-D] [-d directory]
[-F] [-h help] [-H hostname] [-L logfile] [-M MOMport] [-R RPPport]
[-p|-q|-r] [-w] [-x]
πΈ DESCRIPTION πΈ
The pbs_mom command starts the operation of a batch Machine Oriented Mini-server, MOM, on the local host. Typically, this command will be in a local boot file such as /etc/rc.local . To insure that the pbs_mom command is not runnable by the general user community, the server will only execute if its real and effective uid is zero. π
One function of pbs_mom is to place jobs into execution as directed by the server, establish resource usage limits, monitor the job's usage, and notify the server when the job completes. If they exist, pbs_mom will execute a prologue script before executing a job and an epilogue script after executing the job. π
The next function of pbs_mom is to respond to resource monitor requests. This was done by a separate process in previous versions of PBS but has now been combined into one process. The resource monitor function is provided mainly for the PBS scheduler. It provides information about the status of running jobs, memory available etc. π
The next function of pbs_mom is to respond to task manager requests. This involves communicating with running tasks over a tcp socket as well as communicating with other MOMs within a job (aka a "sisterhood"). π«Ά
pbs_mom will record a diagnostic message in a log file for any error occurrence. The log files are maintained in the mom_logs directory below the home directory of the server. If the log file cannot be opened, the diagnostic message is written to the system console. π
β¨ OPTIONS β¨
-Aalias β Used with-m(multi-mom option) to give the alias name of this instance ofpbs_mom.-aalarm β Specifies the alarm timeout in seconds for computing a resource. Every time a resource request is processed, an alarm is set for the given amount of time. If the request has not completed before the given time, an alarm signal is generated. The default is5seconds. β°-Cchkdirectory β Specifies the path of the directory used to hold checkpoint files. [Currently this is only valid on Cray systems.] The default directory isPBS_HOME/spool/checkpoint, see the-doption. The directory specified with the-Coption must be owned by root and accessible (rwx) only by root to protect the security of the checkpoint files. π-cconfig β Specifies an alternative configuration file, see description below. If this is a relative file name it will be relative toPBS_HOME/mom_priv, see the-doption. If the specified file cannot be opened,pbs_momwill abort. If the-coption is not supplied,pbs_momwill attempt to open the default configuration file"config"inPBS_HOME/mom_priv. If this file is not present,pbs_momwill log the fact and continue. π-hhelp β Displays the help/usage message.-Hhostname β Sets the MOM's hostname. This can be useful on multi-homed networks. π¦-Dβ Debug mode. Do not fork.-ddirectory β Specifies the path of the directory which is the home of the servers working files,PBS_HOME. This option is typically used along with-Mwhen debugging MOM. The default directory is given by$PBS_SERVER_HOMEwhich is typically/usr/spool/PBS. π-Fβ Do not fork. Use when running under systemd.-Llogfile β Specifies an absolute path name for use as the log file. If not specified, MOM will open a file named for the current date in thePBS_HOME/mom_logsdirectory, see the-doption. π-mβ Directs the MOM to start in multi-mom mode. In addition to using-mthe-M,-Rand-Aoptions need to be used to properly start a MOM in multi-mom mode. For examplepbs_mom -m -M 30002 -R 30003 -A alias-hostwill startpbs_momwith the service port on port30002, the manager port at30003and with the namealias-host. π―-Mport β Specifies the port number on which the mini-server (MOM) will listen for batch requests.-Rport β Specifies the port number on which the mini-server (MOM) will listen for resource monitor requests, task manager requests and inter-MOM messages. π‘-pβ (Default after version 2.4.0) (Preserve running jobs) -- Specifies the impact on jobs which were in execution when the mini-server shut-down. The-poption tries to preserve any running jobs when the MOM restarts. The new mini-server will not be the parent of any running jobs, MOM has lost control of her offspring (not a new situation for a mother). The MOM will allow the jobs to continue to run and monitor them indirectly via polling. All recovered jobs will report an exit code of0when they are complete. The-poption is mutually exclusive with the-r,-Pand-qoptions. π-Pβ (Terminate all jobs and remove them from the queue) -- Specifies the impact on jobs which were in execution when the mini-server shut-down. With the-Poption, it is assumed that either the entire system has been restarted or the MOM has been down so long that it can no longer guarantee that the pid of any running process is the same as the recorded job process pid of a recovering job. Unlike the-poption no attempt is made to try and preserve or recover running jobs. All jobs are terminated and removed from the queue. The-qoption is mutually exclusive with the-p,-qand-roptions. (Yes bestie, that is exactly what it says:-qis listed as mutually exclusive with itself. We are not making the rules π.)-qβ (Requeue all jobs - This is the default behavior in versions prior to 2.4.0) -- Specifies the impact on jobs which were in execution when the mini-server shut-down. Do not terminate running processes. With the-qoption, it is assumed that either the entire system has been restarted or the MOM has been down so long that it can no longer guarantee that the pid of any running process is the same as the recorded job process pid of a recovering job. No attempt is made to kill job processes. The MOM will mark the jobs as terminated and notify the batch server which owns the job. Re-runnable jobs will be requeued. The-qoption is mutually exclusive with the-p,-Pand-roptions. π-rβ (Terminate running processes and requeue all jobs) -- Specifies the impact on jobs which were in execution when the mini-server shut-down. With the-roption, MOM will kill any processes belonging to running jobs, mark the jobs as terminated and notify the batch server that owns the job. Re-runnable jobs are reset to a queued state so they can be run again. The-roption is mutually exclusive with the-p,-Pand-qoptions. π If the-roption is used following a reboot, process IDs (pids) may be reused and MOM may kill a process that is not a batch session. β οΈ-Sport β Specifies the port number on which thepbs_serveris listening for requests. Ifpbs_serveris started with a-poption,pbs_momwill need to use the-Soption and match the port value which was used to startpbs_server. π-wβ When started with-w,pbs_moms wait until they get their MOM hierarchy file frompbs_serverto send their first update, or until10minutes pass. This reduces network traffic on startup and can bring up clusters faster. β³-xβ Disables the check for privileged port resource monitor connections. This is used mainly for testing since the privileged port is the only mechanism used to prevent any ordinary user from connecting. π
π CONFIGURATION FILE π
The configuration file may be specified on the command line at program start with the -c flag. The use of this file is to provide several types of run time information to pbs_mom: static resource names and values, external resources provided by a program to be run on request via a shell escape, and values to pass to internal set up functions at initialization (and re-initialization). π§
Each item type is on a single line with the component parts separated by white space. If the line starts with a hash mark (pound sign, #), the line is considered to be a comment and is skipped. π¬
πΌ Static Resources πΌ
For static resource names and values, the configuration file contains a list of resource names/values pairs, one pair per line and separated by white space. An Example of static resource names and values could be the number of tape drives of different types and could be specified by
tape3480 4
tape3420 2
tapedat 1
tape8mm 1
π Shell Commands π
If the first character of the value is an exclamation mark (!), the entire rest of the line is saved to be executed through the services of the system(3) standard library routine. π«Ά
The shell escape provides a means for the resource monitor to yield arbitrary information to the scheduler. Parameter substitution is done such that the value of any qualifier sent with the query, as explained below, replaces a token with a percent sign (%) followed by the name of the qualifier. For example, here is a configuration file line which gives a resource name of "escape":
escape !echo %xxx %yyy
Okay so back to escape β if a query for "escape" is sent with no qualifiers, the command that gets executed would be "echo %xxx %yyy". If one qualifier is sent, escape[xxx=hi there], the command executed would be "echo hi there %yyy". If two qualifiers are sent, escape[xxx=hi][yyy=there], the command executed would be "echo hi there". And if a qualifier is sent with no matching token in the command line, like escape[zzz=snafu], an error is reported. Periodt β¨
size[fs=<FS>]β says the available and configured disk space in the<FS>filesystem is to be reported to thepbs_serverand scheduler. NOTE: to request disk space on a per-job basis, specify the file resource like this:qsub -l nodes=1,file=1000kb. For example, the available and configured disk space in the/localscratchfilesystem will be reported withsize[fs=/localscratch]β iconic little filesystem check π¦
Initialization Value πΈ
An initialization value directive has a name which starts with a dollar sign ($) and must be known to MOM via an internal table. The entries in this table now are:
auto_ideal_loadβ if jobs are running, setsidea_loadbased on a simple expression. The expressions start with the variablet(total assigned CPUs) orc(existing CPUs), an operator (+ - / *), and followed by a float constant. Example:$auto_ideal_load t-0.2β and yes, the original saysidea_loadright there, so we're keeping that detail, no cap πauto_max_loadβ if jobs are running, setsmax_loadbased on a simple expression. The expressions start with the variablet(total assigned CPUs) orc(existing CPUs), an operator (+ - / *), and followed by a float constant.cputmultβ sets a factor used to adjust cpu time used by a job. This is provided to allow adjustment of time charged and limits enforced where the job might run on systems with different cpu performance. If Mom's system is faster than the reference system, setcputmultto a decimal value greater than1.0. If Mom's system is slower, setcputmultto a value between1.0and0.0. Examples:$cputmult 1.5and$cputmult 0.75. Slay for cross-system fairness πconfigversionβ specifies the version of the config file data, a string.check_poll_timeβ specifies the MOM interval in seconds. MOM checks each job for updated resource usages, exited processes, over-limit conditions, etc. once per interval. This value should be equal or lower topbs_server'sjob_stat_rate. High values result in stale information reported topbs_server. Low values result in increased system usage by MOM. Default is45seconds.down_on_errorβ causes MOM to report itself as state"down"topbs_serverin the event of a failed health check. This feature is EXPERIMENTAL and likely to be removed in the future. See HEALTH CHECK below.enablemomrestartβ enable automatic restarts of MOM. If enabled, MOM will check if its binary has been updated and restart itself at a safe point when no jobs are running; thus making upgrades easier. The check is made by comparing the mtime of thepbs_momexecutable. Command-line args, the process name, and thePATHenv variable are preserved across restarts. It is recommended that this not be enabled in the config file, but enabled when desired withmomctl(see RESOURCES for more information).ideal_loadβ ideal processor load. Represents a low water mark for the load average. Nodes that are currently busy will consider itself free after falling belowideal_load.igncputβ Ignore cpu time violations on this mom, meaning jobs will not be cancelled due to exceeding their limits for cpu time.ignmemβ Ignore memory violations on this mom, meaning jobs will not be cancelled due to exceeding their memory limits.ignvmemβ If set to true, thenpbs_momwill ignore vmem/pvmem limit enforcement.ignwalltimeβ If set to true, thenpbs_momwill ignore walltime limit enforcement.job_output_file_maskβ specifies a mask for creating job output and error files. Values can be specified in base 8, 10, or 16; leading0implies octal and leading0xor0Xhexadecimal. A value of"userdefault"will use the user's default umask. Example:$job_output_file_mask 027.log_directoryβ changes the log directory. Default is$TORQUEHOME/mom_logs/.$TORQUEHOMEdefault is/var/spool/torque/but can be changed in the./configurescript. The value is a string and should be the full path to the desired mom log directory. Example:$log_directory /opt/torque/mom_logs/.logeventβ sets the mask that determines which event types are logged bypbs_mom. Examples:$logevent 0x1fffand$logevent 255. The first example would set the log event mask to0x1ff(511) which enables logging of all events including debug events. The second example would set the mask to0x0ff(255) which enables all events except debug events. The0x1fffvs0x1ffmoment is very real β keeping both exactly, bestie πlog_file_suffixβ optional suffix to append to log file names. If%his the suffix,pbs_momappends the hostname for where the log files are stored if it knows it, otherwise it will append the hostname where the mom is running. Example:$log_file_suffix tom = 20100223.tom.log_keep_daysβ specifies how many days to keep log files.pbs_momdeletes log files older than the specified number of days. If not specified,pbs_momwon't delete log files based on their age.loglevelβ specifies the verbosity of logging with higher numbers specifying more verbose logging. Values may range between0and7.log_file_max_sizeβ if this is set to a value> 0thenpbs_momwill roll the current log file tolog-file-name.1when its size is greater than or equal to the value oflog_file_max_size. This value is interpreted as kilobytes.log_file_roll_depthβ if this is set to a value>=1andlog_file_max_sizeis set thenpbs_momwill continue rolling the log files tolog-file-name.log_file_roll_depth.max_loadβ maximum processor load. Nodes over this load average are considered busy (seeideal_loadabove).memory_pressure_thresholdβ the option is only available, ifpbs_momis enabled to use cpusets. If set to a value> 0, a job gets killed if its memory pressure exceeds this value, and if$memory_pressure_durationis set. The default is0(memory pressure recording is off). Seecpuset(7)for more information about memory pressure.memory_pressure_durationβ the option is only available, ifpbs_momis enabled to use cpusets. Specifies the number of subsequent MOM intervals a job's memory pressure must be above$memory_pressure_thresholdto get killed. The default is0(jobs are never killed due to memory pressure).set See cpuset(7)for more information about memory pressure β yes, the original has that littlesetin there, and we're keeping it because details are sacred πnode_check_scriptβ specifies the fully qualified pathname of the health check script to run (see HEALTH CHECK for more information).node_check_intervalβ specifies when to run the MOM health check. The check can be either periodic, event-driver, or both. The value starts with an integer specifying the number of MOM intervals between subsequent executions of the specified health check. After the integer is an optional comma-separated list of event names. Currently supported are"jobstart"and"jobend". This value defaults to1with no events indicating the check is run every MOM interval. (See HEALTH CHECK for more information.) Examples:
$node_check_interval 0 #Disabled.
$node_check_interval 0,jobstart #Only runs at job starts
$node_check_interval 10,jobstart,jobend
nodefile_suffixβ specifies the suffix to append to a host names to denote the data channel network adapter in a multihomed compute node. Example:$nodefile_suffix i. With the suffix of'i'and the control channel adapter with the namenode01, the data channel would have a hostname ofnode01i.nospool_dir_listβ if the job's output file should be in one of the paths specified here, then it will be spooled directly in that directory instead of the normal spool directory. Specified in the formatpath1, path2, etc.Example:$nospool_dir_list/home/mike/*,/var/tmp/spool/.pbsclientβ causes a host name to be added to the list of hosts which will be allowed to connect to MOM as long as they are using a privileged port for the purposes of resource monitor requests. For example, here are two configuration file lines which will allow the hosts"fred"and"wilma"to connect:
$pbsclient fred
$pbsclient wilma
Two host names are always allowed to connection to pbs_mom, "localhost" and the name returned to pbs_mom by the system call gethostname(). These names need not be specified in the configuration file. The hosts listed as "clients" can issue Resource Monitor (RM) requests. Other MOM nodes and servers do not need to be listed as clients.
pbsserverβ bestie, this one is the bouncer at the club π it lists the hostnames runningpbs_serverthat are allowed to submit jobs, issue Resource Monitor (RM) requests, and grab status updates. MOM will continually try to contact all the server hosts for node status and state updates β she's not ghosting anybody. Just like$PBS_SERVER_HOME/server_name, the hostname may be followed by a colon and a port number. This parameter replaces the oft-confused$clienthostparameter from TORQUE 2.0.0p0 and earlier (rip to a messy era π). Note that the hostname in$PBS_SERVER_HOME/server_nameis used if no$pbsserverparameters are found.prologalarmβ the max duration in seconds that MOM will wait for the job prolog or the job epilog to complete. Defaults to 300 seconds (5 minutes) β so yeah, a full five minutes of patience, she's literally so forgiving π«Άrcpcmdβ specifies the full path and argument to be used for remote file copies. This overrides the compile-time default found inconfigure. It must contain 2 words: the full path to the command, and the switches. The copy command has to be able to recursively copy files to the remote host and accept arguments of the form"user@host:files". For example:
$rcpcmd /usr/bin/rcp -rp
$rcpcmd /usr/bin/scp -rpB
restrictedβ adds a host name to the list of hosts allowed to connect to MOM without needing to use a privileged port. These names allow for wildcard matching. Example config file line that lets any host from the"ibm.com"domain query:
$restricted *.ibm.com
The restriction on these connections: only internal queries may be made. No resources from a config file will be found. This is to prevent any shell commands from being run by a non-root process π this parameter is generally not required except for some versions of OSX.
remote_checkpoint_dirsβ tells MOM which server checkpoint directories are remotely mounted, i.e. which directories are shared with the server. Using remote checkpoint directories eliminates the need to copy the checkpoint files back and forth between the MOM and the server β no more middleman, we love efficiency β¨ This parameter is available in 2.4.1 and later.
$remote_checkpoint_dirs /var/spool/torque/checkpoint
remote_reconfigβ enables the ability to remotely reconfigurepbs_momwith a new config file. Default is disabled. This parameter accepts various forms oftrue,yes, and1.source_login_batchβ specifies whether or not mom will source the/etc/profile, etc. type files for batch jobs. Parameter accepts various forms oftrue,false,yes,no,1and0. Default isTrue.source_login_interactiveβ specifies whether or not mom will source the/etc/profile, etc. type files for interactive jobs. Parameter accepts various forms oftrue,false,yes,no,1and0. Default isTrue.spool_as_final_nameβ if set to true, jobs will spool directly as their output files, with no intermediate locations or steps. This is mostly useful for shared filesystems with fast writing capability β that throughput is giving π₯status_update_timeβ specifies (in seconds) how often MOM updates its status information topbs_server. This value should correlate with the server's scheduling interval. High values increase the load ofpbs_serverand the network. Low values causepbs_serverto report stale information. Default is 45 seconds β±οΈ highkey a balance-y situation.tmpdirβ sets the directory basename for a per-job temporary directory. Before job launch, MOM will append the jobid to thetmpdirbasename and create the directory. After the job exits, MOM will recursively delete it. The env variableTMPDIRwill be set for all pro/epilog scripts, the job script, and TM tasks.
Directory creation and removal is done as the job owner and group, so the owner must have write permission to create the directory. If the directory already exists and is owned by the job owner, it will not be deleted after the job. If the directory already exists and is NOT owned by the job owner, the job start will be rejected π« no cap, that's a whole rejection.
timeoutβ specifies the number of seconds before TCP messages will time out. TCP messages include job obituaries, and TM requests if RPP is disabled. Default is 60 seconds.usecpβ specifies which directories should be staged withcpinstead ofrcp/scp. If a shared filesystem is available on all hosts in a cluster, this directive is used to make these filesystems known to MOM. For example, if/homeis NFS mounted on all nodes in a cluster:
$usecp *:/home /home
varattrβ similar to a shell escape above, but includes a TTL π The command will only be run every TTL seconds. A TTL of-1will cause the command to be executed only once. A TTL of0will cause the command to be run every timevarattris requested. This parameter may be used multiple times, but all output will be grouped into a single"varattr"attribute in the request and status output. The command should output data in the form ofvarattrname=va1ue1[+value2]...
$varattr 3600 /path/to/script [<ARGS>]...
use_smtβ this option is only available ifpbs_momis enabled to use cpusets. It only has an effect if there is more than one logical processor per physical core in the system (simultaneous multithreading or hyperthreading is enabled via BIOS settings). If set totrue, all logical processors of allocated cores are added to the cpuset of a job. If set tofalse, only the first logical processor per allocated core is contained in the cpuset of a job. The default istrueπ§΅β¨wallmultβ sets a factor used to adjust wall time usage by job to a common reference system. The factor is used for walltime calculations and limits the same ascputmultis used for cpu time.
The configuration file must be executable and "secure". It must be owned by a user id and group id less than 10 and not be world writable π (yes, she checks IDs, bestie, it's not that deep but it is). Output from this file must be in the format $VAR=$VAL, i.e.
dataset13=20070104
dataset22=20070202
viraltest=abdd3
xauthpathβ specifies the path to thexauthbinary to enable X11 forwarding π₯οΈβ¨mom_hostβ sets the local hostname as used bypbs_mom.
π¦ LAYOUT FILE π¦
There's also an optional layout file for creating multiple moms on one box in a specified layout π―ββοΈ Inside the file, each mom on the single box is given its own hostname, cpu indexes, memory nodes (a linux construct), and memory size. This is useful for NUMA systems. Each line in the file specifies one mom. The file follows the following format:
<hostname> cpus=<X> mem=<Y> memsize=<Z>
cpus and mem can be comma separated lists, while memsize should be a memory size in the format:
<number><units>
For example, a file could contain the following line:
foohost-1 cpus=1,2 mem=1,2,3,4 memsize=8GB
This would specify that foohost-1 has cpus 1 and 2, memory nodes 1-4, and a total of 8 GB of memory πΈ iconic, no notes.
β¨ RESOURCES β¨
Resource Monitor queries can be made with momctl's -q option to retrieve and set pbs_mom options π Any configured static resource may be retrieved with a request of the same name. These are resource requests not otherwise documented in the PBS ERS.
cycleβ forces an immediate MOM cycle πstatus_update_timeβ retrieve or set the$status_update_timeparametercheck_poll_timeβ retrieve or set the$check_poll_timeparameterconfigversionβ retrieve the config versionjobstartblocktimeβ retrieve or set the$jobstartblocktimeparameterenablemomrestartβ retrieve or set the$enablemomrestartparameterloglevelβ retrieve or set the$loglevelparameterdown_on_errorβ retrieve or set the EXPERIMENTAL$down_on_errorparameterdiag0-diag4β retrieves various diagnostic information πrcpcmdβ retrieve or set the$rcpcmdparameterversionβ retrieves thepbs_momversion
π©Ί HEALTH CHECK π©Ί
Okay so the health check script? It gets run directly by the pbs_mom daemon under the root user id β yes, root, bestie, so act accordingly π
. It has to be accessible from the compute node, and it can be either a script or a compiled executable program. It may make any needed system calls and execute any combination of system utilities, but it should not execute resource manager client commands. Also, as of TORQUE 1.0.1, the pbs_mom daemon blocks until the health check is completed and does not possess a built-in timeout. So yeah: keep the launch script execution time short, and verify that the script will not block even under failure conditions. No pressure or anything π.
β οΈ Root, blocking, no built-in timeout β this is not the vibe for a slow script, bestie. Keep it short and make sure it won't hang even when stuff fails. π«Ά
If the script detects a failure, it should return the keyword 'ERROR' to stdout followed by an error message. The message β up to 256 characters β immediately following the ERROR string will be assigned to the node attribute 'message' of the associated node β¨.
If the script detects a failure when run from "jobstart", then the job will be rejected. This should probably only be used with advanced schedulers like Moab so that the job can be routed to another node.
TORQUE currently ignores ERROR messages by default, but advanced schedulers like moab can be configured to react appropriately.
If the experimental $down_on_error MOM setting is enabled, MOM will set itself to state down and report to pbs_server; and pbs_server will report the node as "down". Additionally, the experimental "down_on_error" server attribute can be enabled which has the same effect but moves the decision to pbs_server. It is redundant to have MOM's $down_on_error and pbs_server's down_on_error features enabled. See "down_on_error" in pbs_server_attributes(7B).
π FILES π
Here's the file situation, bestie β know these paths, love these paths:
$PBS_SERVER_HOME/server_nameβ contains the hostname runningpbs_server.$PBS_SERVER_HOME/mom_privβ the default directory for configuration files, typically(/usr/spool/pbs)/mom_priv.$PBS_SERVER_HOME/mom_logsβ directory for log files recorded by the server.$PBS_SERVER_HOME/mom_priv/prologueβ the administrative script to be run before job execution.$PBS_SERVER_HOME/mom_priv/epilogueβ the administrative script to be run after job execution.
π¨ SIGNAL HANDLING π¨
pbs_mom handles the following signals, and each one has its own little personality:
SIGHUPβ causespbs_momto re-read its configuration file, close and reopen the log file, and reinitialize resource structures.SIGALRMβ results in a log file entry. The signal is used to limit the time taken by certain children processes, such as the prologue and epilogue.SIGINTandSIGTERMβ results inpbs_momexiting without terminating any running jobs. This is the action for the following signals as well:SIGXCPU,SIGXFSZ,SIGCPULIM, andSIGSHUTDN.SIGUSR1,SIGUSR2β causes MOM to increase and decrease logging levels, respectively.SIGPIPE,SIGINFOβ are ignored.SIGBUS,SIGFPE,SIGILL,SIGTRAP, andSIGSYSβ cause a core dump if thePBSCOREDUMPenvironmental variable is defined.- All other signals β have their default behavior installed.
πͺ EXIT STATUS πͺ
If the mini-server command fails to begin operation, the server exits with a value greater than zero. That's the whole exit-status tea β.
π SEE ALSO π
Go read these next, bestie:
pbs_server(8B)pbs_scheduler_basl(8B)pbs_scheduler_tcl(8B)- the PBS External Reference Specification
- the PBS Administrator's Guide
π signed off from the Local pbs_mom(8B) page β date on your local copy, bestie π«Άβ¨
π¦ where this page comes from
- epel 9: torque-mom 6.1.3-8.el9 Β· OpenPBS and TORQUEv1.1
- nixos 26.05: torque 6.1.3h2
- nixpkgs unstable: torque 6.1.3h2
- Copyright (c) 1999-2000 Veridian Information Solutions, Inc
- π girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on : a language model's rewrite of the original, under its terms