xpbsmon(1B)

📦 version 6.1.3, in epel 9, nixos 26.05, nixpkgs unstable

🌸 NAME 🌸

Bestie, this is xpbsmon(1B) — the PBS GUI that's literally here to display and monitor the nodes/execution hosts under PBS. It's giving "visual dashboard for your cluster," no cap. 👑

✨ SYNOPSIS ✨

xpbsmon

🦋 DESCRIPTION 🦋

The xpbsmon command is your way to graphically display all the various nodes that run jobs. A node or execution host can be running a pbs_mom daemon, or not running the daemon. For the latter case, it could just be a nodename that appears in a nodes file that is managed by a main pbs_server running on another host.

This utility also gives you the ability to monitor values of certain system resources by posting queries to the pbs_mom of a node. With this utility, you can see what job is running on what node, who owns the job, how many nodes are assigned to a job, the status of each node (color-coded, and the colors are user-modifiable), how many nodes are available, free, down, reserved, offline, of unknown status, in use running multiple jobs, or executing only 1 job.

Please see the sections below for a tour and tutorials of xpbsmon. Also, within every dialog box, a Help button can be found for assistance. 🫶

✨ GETTING STARTED ✨

Running xpbsmon will initialize the X resource database from various sources in the following order:

  1. The RESOURCE_MANAGER property on the root window (updated via xrdb) with settings usually defined in the .Xdefaults file
  2. Preference settings defined by the system administrator in the global xpbsmonrc file
  3. User's ~/.xpbsmonrc file — this file defines various X resources like fonts, colors, list of colors to use to represent the various status of the nodes, list of PBS sites to query, list of server hosts on each site, list of nodes/execution hosts on each server host, list of system resource queries to send to the nodes' pbs_mom, and various view states. See PREFERENCES section below for a list of resources that can be set.

💅 RUNNING XPBSMON 💅

xpbsmon can be run either as a regular user or superuser. If you run it with less privilege, you may not be able to see all the information for a node. If it is executed as a regular user, you should still be able to see what jobs are running on what nodes, possibly state and properties, as these information are obtained by xpbsmon talking directly to the specified server. If you want other system resource values, it may require special privilege since xpbsmon will have to talk directly to the pbs_mom of a node.

In addition, the host where xpbsmon was running must also have been given explicit access permission by the mom (unless the GUI is running on the same host where mom is running). This is done by updating the $clienthost and/or the $restricted parameter on the mom's configuration file.

To run xpbsmon, type:

setenv DISPLAY <display_host>:0
xpbsmon

If you are running the GUI and only interested in jobs data, then be sure to set all the nodes' type to NOMOM in the Pref dialog box. 💅

🌈 THE XPBSMON DISPLAY 🌈

This section describes the main parts of the xpbsmon display. The main window is composed of 3 distinct areas (subwindows) arranged vertically (one on top of another) in the following order:

  1. Menu
  2. Site Information
  3. Info

The Menu area is composed of a row of command buttons that signal some action with a click of the left mouse button. The buttons are:

  • Site.. — displays a popup menu containing the list of PBS sites that have been added using the Sites Preferences window. Simply drag your mouse and release to the site name whose servers/nodes information you would like to see.
  • Pref.. — brings up various dialog boxes for specifying the list of sites, servers on each site, nodes that are known to a server, and the system resource queries to be sent to a node's pbs_mom daemon.
  • Auto Update.. — brings up another window for specifying whether or not to do auto updates of nodes information, and also for specifying the interval number of minutes between updates.
  • Help — contains some help information.
  • About — tells who the author is and who to send comments, bugs, suggestions to.
  • Close — for exiting xpbsmon plus saving the current setup information (if anything had changed) in the user's $HOME/.xpbsmonrc file. Information saved include the specified list of sites, servers on each site, nodes known to each server, and system resource queries to send to node's pbs_mom.
  • Minimize Button — shows the iconized view of Site Information where nodes are represented as tiny boxes, where each box is colored according to status. In order to get more information about a node, you need to double click on the colored box.
  • Maximize button — shows the full view of Site Information where nodes are represented in bigger boxes, still colored depending on the status, and some information on it is displayed.

🦋 Site Information 🦋

Only one site at a time can be displayed. This area (shown as one huge box referred to as the site box) can be further sub-divided into 3 areas: the site name label at the top, server boxes in the middle, and the color status bar at the bottom. The site name label shows the name of the site as specified in the Pref.. window. At the middle of the site box shows a row of big boxes housing smaller boxes.

The big box is an abstraction of a server host (called a server box), showing its server display label at the top of the box, a grid of smaller boxes representing the nodes that the server knows about (where jobs are run), and summary status for the nodes under the server. Status information will show counters for the number of nodes used, available, reserved, offline, or of unknown status and even # of cpus assigned. For a cleaner display, some counters with a value of zero are not displayed. The server boxes are placed in a grid, with a new row being started when either *siteBoxMaxNumServerBoxesPerRow or *siteBoxMaxWidth limit has been reached.

The smaller boxes represent the nodes/execution hosts where jobs are run (referred to as node boxes). Each node box shows the name at the top, and a sub-box (a smaller square) that is colored according to the status of the node that it represents, and if the view type is FULL, it will display some node information according to the system resource queries specified on the Pref.. window. Clicking on the sub-box will show a much bigger box (called the MIRROR view) with bigger fonts containing nodes information. Another view is called ICON and this shows a tiny box with a colored area. The node boxes are arranged in a grid, where a new row is created if either the *serverBoxMaxNumNodeBoxesPerRow or *serverBoxMaxWidth limit has been reached. ICON view of the node boxes will be constrained by the *nodeBoxIconMaxHeight and *nodeBoxIconMaxWidth pixel values; FULL view of the node boxes will be bounded by *nodeBoxFullMaxWidth and *nodeBoxFullMaxHeight; the mirror view of the node boxes has its size be *nodeBoxMirrorMaxWidth, and *nodeBoxMirrorMaxHeight.

Horizontal and vertical scrollbars for the site box, server box, and node box will be displayed as needed.

Finally, the color bar information shows a color chart displaying what the various colors mean in terms of node status. The color-to-status mapping can be modified by setting the X resources:

  • *nodeColorNOINFO
  • *nodeColorFREE
  • *nodeColorINUSEshared
  • *nodeColorINUSEexclusive
  • *nodeColorDOWN
  • *nodeColorRSVD
  • *nodeColorOFFL

💬 Info 💬

The Info Area shows the progress of some of the background actions performed by xpbsmon. Look into this box for errors.

🦋 WIDGETS USED IN XPBSMON 🦋

okay so before you start poking at things — here's the tea on the widgets this GUI uses and how you actually manipulate them 🫶

  1. listbox — the ones in this GUI are only single-selectable, meaning one entry can be highlighted/selected at a time via a mouse click. one. that's it. don't try to be greedy 💅
  2. scrollbar — usually shows up either vertically or horizontally, and it's got 5 distinct areas you mouse-click for different effects:
    • top arrow — causes the view in the associated widget to shift up by one unit (i.e. the object appears to move down one unit in its window). If the button is held down, the action will auto-repeat. hold it and it just keeps going, iconic ✨
    • top gap — causes the view in the associated window to shift up by one less than the number of units in the window (i.e. the portion of the object that used to appear at the very top of the window will now appear at the very bottom). If the button is held down, the action will auto-repeat.
    • slider — pressing button 1 in this area has no immediate effect except to cause the slider to appear sunken rather than raised. However, if the mouse is moved with the button down, then the slider will be dragged, adjusting the view as the mouse is moved. it's giving "grab and go" 💫
    • bottom gap — causes the view in the associated window to shift down by one less than the number of units in the window (i.e. the portion of the object that used to appear at the very bottom of the window will now appear at the very top). If the button is held down, the action will auto-repeat.
    • bottom arrow — causes the view in the associated window to shift down by one unit (i.e. the object appears to move up one unit in its window). If the button is held down, the action will auto-repeat.
  3. entry — brought into focus with a click of the left mouse button. To manipulate this widget, simply type in the text value. Use of arrow keys, mouse selection of text for deletion or overwrite, copying and pasting with sole use of mouse buttons are permitted. This widget is usually accompanied by a scrollbar for horizontally scanning a long text entry string. yes you can mouse-only the whole thing, she's flexible 💖
  4. box — made up of 1 or more listboxes displayed adjacent to each other, giving the effect of a "matrix". Each row from the listboxes makes up an element of the box. In order to add items to the box, you need to manipulate the accompanying entry widgets, one for each listbox, and then click the add button. Removing items from the box is done by selecting an element, and then clicking delete. add, delete, add, delete — main character energy fr fr 🎀
  5. spinbox — a combination of an entry widget and a horizontal scrollbar. The entry widget will only accept values that fall within a defined list of valid values, and incrementing through the valid values is done by clicking on the up/down arrows. no free-typing chaos here, she's got standards 💅
  6. button — a rectangular region appearing either raised or pressed that invokes an action when clicked with the left mouse button. When the button appears pressed, then hitting the RETURN key will automatically select the button. lowkey the most convenient thing in the whole window ngl 🌸

🌈 UPDATING PREFERENCES 🌈

CASE 1: Time Sharing

so picture this: you've got a time-sharing environment where the front-end is called bower and you have 4 nodes: bower1, bower2, bower3, bower4. bower is the host that runs the server; jobs are submitted to host bower, where it enqueues them for future execution. Also, a pbs_mom daemon is running on each of the execution hosts. If the server bower also maintains a nodes list containing information like state, properties for the 4 nodes, then this will also be reported. Then to setup xpbsmon, do the following:

  1. Click the Pref.. button on the Menu section.
  2. On the Sites Preference dialog, enter any arbitrary site name, for example "Local". Then click the add button.
  3. On the Server_Host entry box, enter "bower", and on the DisplayLabel entry box, put an arbitrary label (as it would appear on the header of the server box) like "Bower", and then click add.
  4. Click the nodes.. button that is accompanying the Servers box. This would bring up the Server Preference dialog.
  5. Now add the entries "bower1", "bower2", "bower3", "bower4", specifying type MOM for each on the Nodes box.
  6. If you need to monitor certain system resource parameters for each of the nodes, you need to specify query expressions containing resource queries to be sent to the individual PBS moms. For example, if you want to obtain memory usage, then select a node from the Nodes list, click on the query.. button that accompanies the Nodes list, and this would bring up the Query Table dialog. Specify the following input:
Query_Expr:    (availmem/totmem) * 100
Display_Info:  Memory Usage:
Display_Type:  SCALE

The above says to display the result of the "Query_Expr" in a scale widget calibrated over 100. The queries "availmem" and "totmem" will be sent to the PBS mom, and the expression is evaluated upon receiving all results from the mom. If you want to display the result of another query, say "loadave", directly, then specify the following:

Query_Expr:    loadave
Display_Info:  Load Average:
Display_Type:  TEXT
NOTE: For a list of queries that can be sent to a pbs_mom, please click on the Help button on the Query table window. bestie the help button literally exists, use it 🙈

CASE 2: Jobs Exclusive Environment

Supposing you have a "space non-sharing" environment where the server maintains a list of nodes that it runs jobs on exclusively (one job at a time outstanding per node). Let's call this server b1. Simply update Preferences information as follows:

  1. Click the Pref.. button on the Menu section.
  2. On the Sites Preference dialog, enter a site name, for example "B System". Then click the add button.
  3. On the Server_Host entry box, enter "b1", DisplayLabel entry box type "B1" (or whatever label that you would like to appear on the header of the server box), and then click add.

and that's literally it, no extra steps, we love a low-effort slay ✨

CASE 3: Hybrid Time Sharing/Space Sharing Environment

A cluster of heterogeneous machines, time-sharing or jobs exclusive, could easily be represented in xpbsmon by combining steps in CASE 1 and CASE 2. mix and match, it's giving ✨versatility✨ 💜

🎀 LEAVING XPBSMON 🎀

Click on the Close button located in the Menu bar to leave xpbsmon. If anything had changed, it will bring up a dialog box asking for a confirmation in regards to saving preferences information about list of sites, their view types, list of servers on each site, the list of nodes known to each server, and the list of queries to be sent to the pbs_mom of each node. The information is saved in ~/.xpbsmonrc.

so yeah, if you made changes, she's gonna double-check with you before dipping. say yes, save your stuff, don't lose your whole setup 😭

💅 PREFERENCES 💅

The resources that can be set in the X resources file, ~/.xpbsmonrc, are described in the following:

Node Box Properties

Resource names beginning with *small or *node apply to the properties of the node boxes. A node box is made of an outer frame where the node label sits on top, the canvas (smaller box) is on the middle, and possibly some horizontal/vertical scrollbars.

  • nodeColorNOINFO — color of node box when information for the node it represents could not be obtained. (and yes bestie, this one really is written without the asterisk, i don't make the rules 🫠)
  • *nodeColorFREE — color of canvas when node it represents is up.
  • *nodeColorINUSEshared — color when node it represents has more than 1 job running on it, or when node has been marked by the server that manages it as "job-sharing".
  • *nodeColorINUSEexclusive — list of colors to assign to a node box when host it represents is running only 1 job, or when node has been marked by the server that manages it as "time-sharing". xpbsmon will use this list to assign 1 distinct color per job unless all the colors have been exhausted, in which case, colors will start getting assigned more than once in a round-robin fashion. so she recycles, eco queen 🌈
  • *nodeColorDOWN — color when node it represents is down.
  • *nodeColorRSVD — color when node it represents is reserved.
  • *nodeColorOFFL — color when node it represents is offline.
  • *smallForeground — applies to the color of text inside the canvas.
  • *smallBackground — applies to the color of the frame.
  • *smallBorderWidth — distance (in pixels) from other node boxes.
  • *smallRelief — how node box will visually appear (style).
  • *smallScrollBorderWidth — significant only in FULL mode, this is the distance of the horizontal/vertical scrollbars from the canvas and lower edge of the frame.
  • *smallScrollBackground — background color of the scrollbars.
  • *smallScrollRelief — how scrollbars would visually appear (style).
  • *smallCanvasBackground — color of the canvas (later overridden depending on status of the node it represents).
  • *smallCanvasBorderWidth — distance of the canvas from the frame and possibly the scrollbars.
  • *smallCanvasRelief — how the canvas is visually represented (style).
  • *smallLabelBorderWidth — the distance of the node label from the canvas and the topmost edge of the frame.
  • *smallLabelBackground — the background of the area of the node label that is not filled.
  • *smallLabelRelief — how the label would appear visually (style).
  • *smallLabelForeground — the color of node label text.
  • *smallLabelFont — the font to use for the node label text.
  • *smallLabelFontWidth — font width (in pixels) of *smallLabelFont.
  • *smallLabelFontHeight — font height (in pixels) of *smallLabelFont.
  • *smallTextFont — font to use for the text that appear inside a canvas.
  • *smallTextFontWidth — font width (in pixels) of *smallTextFont.
  • *smallTextFontHeight — font height (in pixels) of *smallTextFont.
  • *nodeColorTrough — color of trough part (the /100 portion) of a canvas scale item.
  • *nodeColorSlider — the color of the slider part (aka the value portion) of a canvas scale item. the little filled bit that moves, bestie 💅
  • *nodeColorExtendedTrough — the color of the extended trough, meaning the over-100 portion that shows up when the value exceeds max. yes it goes past the bar, and yes it has a whole color of its own ✨
  • *nodeScaleFactor — tells how much bigger you want the scale item on the canvas to appear. (1 means to keep size as is)
  • *nodeBoxFullMaxWidth
  • *nodeBoxFullMaxHeight — maximum width and height (in pixels) of a node box in FULL mode. these two are a matched set, don't @ me 🫶
  • *nodeBoxIconMaxWidth
  • *nodeBoxIconMaxHeight — maximum width and height (in pixels) of a node box in ICON mode. the compact era 💖
  • *nodeBoxMirrorMaxWidth
  • *nodeBoxMirrorMaxHeight — maximum width and height (in pixels) of a node box displayed on a separate window (after it has been clicked with the mouse to obtain a bigger view). click a node, get a whole extra window, main character behavior fr 👑
  • *nodeBoxMirrorScaleFactor — tells how much bigger you want the scale item on the canvas to appear while the node box is displayed on a separate window (1 means to keep size as is)

🌸 Server Box Properties 🌸

okay so resource names beginning with *medium apply to the properties of the server boxes 💖

a server box is made of: an outer frame where the server display label sits on top, a canvas filled with node boxes in the middle, possibly some horizontal/vertical scrollbars, and a status label at the bottom. that's the whole anatomy, no cap.

  • *mediumLabelForeground — color of text applied to the server display label and status label.
  • *mediumLabelBackground — background color of the unfilled portions of the server display label and status label.
  • *mediumLabelBorderWidth — distance of the server display label and status label from other parts of the server box.
  • *mediumLabelRelief — how the server display label and status label appear visually (style).
  • *mediumLabelFont — font used for the text of the server display label and status label.
  • *mediumLabelFontWidth — font width (in pixels) of *mediumLabelFont.
  • *mediumLabelFontHeight — font height (in pixels) of *mediumLabelFont.
  • *mediumCanvasBorderWidth — the distance of the server box's canvas from the label widgets.
  • *mediumCanvasBackground — the background color of the canvas.
  • *mediumCanvasRelief — how the canvas appear visually (style).
  • *mediumScrollBorderWidth — distance of the scrollbars from the other parts of the server box.
  • *mediumScrollBackground — the background color of the scrollbars
  • *mediumScrollRelief — how the scrollbars appear visually.
  • *mediumBackground — the color of the server box frame.
  • *mediumBorderWidth — the distance of the server box from other boxes.
  • *mediumRelief — how the server box appears visually (style).
  • *serverBoxMaxWidth
  • *serverBoxMaxHeight — maximum width and height (in pixels) of a server box.
  • *serverBoxMaxNumNodeBoxesPerRow — maximum # of node boxes to appear in a row within a canvas.

✨ Miscellaneous Properties ✨

resource names beginning with *big apply to the properties of a site box, as well as to widgets found outside of the server box and node box 🦋 this includes the dialog boxes that appear when the menu buttons of the main window are manipulated. the site box is the one that appears on the main region of xpbsmon. she's the moment.

  • *bigBackground — background color of the outer layer of the main window.
  • *bigForeground — color applied to regular text that appear outside of the node box and server box.
  • *bigBorderWidth — distance of the site box from the menu area and the color information area.
  • *bigRelief — how the site box is visually represented (style)
  • *bigActiveColor — the color applied to the background of a selection, a selected command button, or a selected scroll bar handle. the "you clicked me" glow-up 💅
  • *bigShadingColor — a color shading applied to some of the frames to emphasize focus as well as decoration. it's giving depth, it's giving drama
  • *bigSelectorColor — the color applied to the selector box of a radiobutton or checkbutton.
  • *bigDisabledColor — color applied to a disabled widget. sad little gray era 😭
  • *bigLabelBackground — color applied to the unfilled portions of label widgets.
  • *bigLabelBorderWidth — distance from other widgets of a label widget.
  • *bigLabelRelief — how label widgets appear visually (style)
  • *bigLabelFont — font to use for labels.
  • *bigLabelFontWidth — font width (in pixels) of *bigLabelFont.
  • *bigLabelFontHeight — font height (in pixels) of *bigLabelFont.
  • *bigLabelForeground — color applied to text that function as labels.
  • *bigCanvasBackground — the color of the main region.
  • *bigCanvasRelief — how the main region looks like visually (style)
  • *bigCanvasBorderWidth — distance of the main region from the menu and info regions.
  • *bigScrollBorderWidth — if the main region has a scrollbar, this is its distance from other widgets appearing on the region.
  • *bigScrollBackground — background color of the scrollbar appearing outside a server box and node box.
  • *bigScrollRelief — how the scrollbar that appears outside a server box and node box looks like visually (style)
  • *bigTextFontWidth — the font width (in pixels) of *bigTextFont
  • *bigTextFontHeight — the font height (in pixels) of *bigTextFont
  • *siteBoxMaxWidth — maximum width (in pixels) of the site box.
  • *siteBoxMaxHeight — maximum height (in pixels) of the site box.
  • *siteBoxMaxNumServerBoxesPerRow — maximum number of server boxes to appear in a row inside the site box.
  • *autoUpdate — if set to true, then information about nodes is periodically gathered. the lil background refresher 🔥
  • *autoUpdateMins — the # of minutes between polling for data regarding nodes when *autoUpdate is set.
  • *siteInView — the name of the site that should be in view.
  • *rcSiteInfoDelimeterChar — the separator character for each input within a curly-bracketed line of input of *siteInfo.
  • *sitesInfo — okay this one is the big one, sit down, hydrate 💀 the value looks like:
{<site1name><sep><site1-display-type><sep><server-name><sep><server-display-label><sep><nodename><sep><nodetype><sep><node-query-expr>}
 . . .
{<site2name><sep><site2-display-type><sep><server-name><sep><server-display-label><sep><nodename><sep><nodetype><sep><node-query-expr>}

that's information about a site where <site1-display-type> can be either {FULL,ICON}, <nodetype> can be {MOM, NOMOM}, and <node-query-expr> has the format:

{ {<expr>} {expr-label} <output-format>}

where <output-format> could be {TEXT, SCALE}. it's probably better to use the Pref dialog boxes in order to specify a value for this. like genuinely, use the dialogs, save yourself 🫶

Example:

*rcSiteInfoDelimeterChar ;
*sitesInfo:     {Mars;ICON;newton;Newton;newton3;NOMOM;}
{Langley;FULL;db;DB;db.OpenPBS.org;MOM;{{ ( availmem / totmem ) * 100} {Memory Usage:} SCALE} {{ ( loadave / ncpus ) * 100} {Cpu Usage:} SCALE} {ncpus {Number of Cpus:} TEXT} {physmem {Physical Memory:} TEXT} {idletime {Idle Time (s):} TEXT} {loadave {Load Avg:} TEXT}}
{Mars;ICON;newton;Newton;newton4;NOMOM;}
{Mars;ICON;newton;Newton;newton1;NOMOM;}
{Mars;ICON;newton;Newton;newton2;NOMOM;}
{Mars;ICON;b0101;DB;aspasia.OpenPBS.org;MOM;{{ ( availmem / totmem ) * 100} {Memory Usage:} SCALE} {{ ( loadave / ncpus ) * 100} {Cpu Usage:} SCALE} {ncpus {Number of Cpus:} TEXT} {physmem {Physical Memory:} TEXT} {idletime {Idle Time (s):} TEXT} {loadave {Load Avg:} TEXT}}
{Mars;ICON;newton;Newton;newton7;NOMOM;}

💀 EXIT STATUS 💀

  • upon successful processing, the xpbsmon exit status will be a value of zero 🎀
  • if the xpbsmon command fails, the command exits with a value greater than zero 😭
  • if xpbsmon is querying a host running a server with an incompatible version, you may see the following messages:
Internal error: pbsstatnode: End of File (15031)
okay so the above message can be safely ignored. she's just being dramatic, it's fine, breathe 🧚

👑 SEE ALSO 👑

  • pbs_sched_tcl(8B) and pbs_mom(8B). go read those next, iykyk 💜

and that's the whole page, bestie 🌈 she's filed under Local on the left and tagged xpbsmon(1B) on the right down there in the footer, so if you ever need to cite her, that's your receipt ✨ love that for us, periodt 💖

📦 where this page comes from