PQOS(8)

📦
version 26.06 · 5 versions
, in nixpkgs unstable, ubuntu stonking

💅 pqos(8) — System Manager's Manual 💅

okay bestie, welcome to the pqos(8) manual. if you're here, you're probably trying to make your CPU behave, and honestly? slay. this is the Intel(R) Resource Director Technology/AMD PQoS monitoring and control tool — aka the thing that lets you stalk and boss around your cache and memory bandwidth. let's get into it. ✨

✨ NAME ✨

pqos, pqos-msr, pqos-os — Intel(R) Resource Director Technology/AMD PQoS monitoring and control tool.

🌈 SYNOPSIS 🌈

pqos [OPTIONS]...

🦋 DESCRIPTION 🦋

Intel(R) Resource Director Technology/AMD PQoS is designed to monitor and manage cpu resources and improve performance of applications and virtual machines. no cap, it's the whole point. 💖

Intel(R) Resource Director Technology/AMD PQoS includes monitoring and control technologies. Monitoring technologies include CMT (Cache Monitoring Technology), which monitors occupancy of last level cache, and MBM (Memory Bandwidth Monitoring). Control technologies include CAT (Cache Allocation Technology), CDP (Code Data Prioritization), MBA (Memory Bandwidth Allocation) and SMBA (Slow Memory Bandwidth Allocation). so you're getting both the tea and the power to change it. iconic. 👑

pqos supports CMT and MBM on a per core or hardware thread basis. MBM supports two types of events reporting local and remote memory bandwidth. pqos-msr and pqos-os are simple pqos wrapper scripts that automatically select the MSR or OS/Kernel library interface to program the technologies. Please see the -I option below for more information.

For hardware information please refer to the README located on: https://github.com/intel/intel-cmt-cat/blob/master/README

👑 OPTIONS 👑

pqos options are as follow:

  • -h, --help — show help. the "help me bestie" button. 🆘
  • -v, --verbose — verbose mode. gives you the chatty version of everything. ☕
  • -V, --super-verbose — super-verbose mode. okay this is the "tell me EVERYTHING" mode. 💅
  • -l FILE, --log-file=FILE — log messages into selected log FILE. dump the receipts into FILE. 📝
  • -s, --show — show the current allocation and monitoring configuration. a little snapshot of what's going on. 📸
  • -d, --display — display supported Intel(R) Resource Director Technology/AMD PQoS capabilities. shows you what your CPU can actually do. ✨
  • -D, --display-verbose — display supported Intel(R) Resource Director Technology/AMD PQoS capabilities in verbose mode. same as above but with extra lore. 📖
  • -f FILE, --config-file=FILE — load commands from selected configuration FILE. read your pre-planned moves from FILE. 🎀
  • -e CLASSDEF, --alloc-class=CLASSDEF — bestie, this is where you define the allocation classes on all CPU sockets. lowkey the main event. CLASSDEF format is "TYPE:ID=DEFINITION;...". and if you want to be specific, define classes for selected CPU resources — CLASSDEF format is "TYPE[@RESOURCE_ID]:ID=DEFINITION;...". For CAT, TYPE is "llc" for the last level cache (aka l3) and "l2" for level 2 cache, ID is a CLOS number and DEFINITION is a bitmask. For MBA, TYPE is "mba", ID is a CLOS number and DEFINITION is a value between 1 and 100 representing the percentage of available memory bandwidth. For MBA CTRL, TYPE is "mba_max", ID is a CLOS number and DEFINITION is a value representing the requested memory bandwidth specified in MBps. For SMBA, TYPE is "smba", ID is a CLOS number and DEFINITION is a value between 1 and 100 representing the percentage of available slow memory bandwidth. RESOURCE_ID is a unique number that can represent a socket or l2/l3 cache identifier. The RESOURCE_ID for each logical CPU can be found using "pqos -s".
    • Note: When L2/L3 CDP is on, ID can be postfixed with 'D' for data or 'C' for code.
    • Note: L2/L3 CDP is available on selected CPUs only.
    • Note: MBA CTRL is supported only by the OS interface and requires Linux and kernel version 4.18 or newer.
    • Some examples:
      • "-e llc:0=0xffff;llc:1=0x00ff"
      • "-e llc@0-1:2=0xff00;l2:2=0x3f;l2@2:1=0xf"
      • "-e llc:0d=0xfff;llc:0c=0xfff00"
      • "-e l2:0d=0xf;l2:0c=0xc"
      • "-e mba:1=30;mba@1:3=80"
      • "-e smba:1=64;mba@1:3=128"
      • "-e mba_max:1=6000;mba_max@1:3=10000"
    • Note:
      • "-e l2:2=0x3f" means that COS2 for all L2 cache clusters is changed to 0x3f.
      • "-e l2@2:1=0xf" means that COS1 for L2 cache cluster 2 is changed to 0xf.
      • "-e mba:1=30" means that COS1, on all sockets, can utilize up to 30% of available memory bandwidth.
      • "-e smba:1=64" means that COS1, on all sockets, can utilize up to 64 units of available slow memory bandwidth.
      • "-e mba_max:1=6000" means that COS1, on all sockets, can utilize up to 6000 MBps of memory bandwidth.
  • -a CLASS2ID, --alloc-assoc=CLASS2ID — this one associates allocation classes with cores or processes — like matching your faves to their VIP sections. 💅 CLASS2ID format is "TYPE:ID=CORE_LIST;..." or "TYPE:ID=TASK_LIST;...". For COS association required for CAT or MBA, TYPE is "cos", "llc", "core" (for COS-core association) or "pid" (for COS-process association) and ID is a class number. CORE_LIST is comma or dash separated list of cores. TASK_LIST is comma or dash separated list of process/task ID's.
    • For example:
      • "-a cos:0=0,2,4,6-10;cos:1=1;" associates cores 0, 2, 4, 6, 7, 8, 9, 10 with CAT class 0 and core 1 with class 1.
      • "-a llc:0=0,2,4,6-10;llc:1=1;" associates cores 0, 2, 4, 6, 7, 8, 9, 10 with CAT class 0 and core 1 with class 1.
      • "-a core:0=0,2,4,6-10;core:1=1;" associates cores 0, 2, 4, 6, 7, 8, 9, 10 with CAT class 0 and core 1 with class 1.
      • "-I -a pid:0=3543,7643,4556;pid:1=7644;" associates process ID 3543, 7643, 4556 with CAT class 0 and process ID 7644 with class 1.
    • Notes:
      • "llc" TYPE label is considered deprecated, please use "cos" or "core" instead.
      • The -I option must be used for PID association.
  • -R [CONFIG[,CONFIG]], --alloc-reset[=CONFIG[,CONFIG]] — reset allocation setting (L3 CAT, L2 CAT, MBA) and reconfigure allocation — factory reset but make it fashion. CONFIG is one of the following options:
    • l3cdp-on sets L3 CDP on
    • l3cdp-off sets L3 CDP off
    • l3cdp-any keeps current L3 CDP setting (default)
    • l3iordt-on sets L3 I/O RDT on
    • l3iordt-off sets L3 I/O RDT off
    • l3iordt-any keeps current L3 I/O RDT setting (default)
    • l2cdp-on sets L2 CDP on
    • l2cdp-off sets L2 CDP off
    • l2cdp-any keeps current L2 CDP setting (default)
    • mbaCtrl-on sets MBA CTRL on
    • mbaCtrl-off sets MBA CTRL off
    • mbaCtrl-any keeps current MBA CTRL setting (default)
    • mba40-on enables MBA 4.0 extensions for all sockets
    • mba40-off disables MBA 4.0 extensions for all sockets
    • mba40-any keeps current MBA 4.0 setting (default)
  • -m EVTCORES, --mon-core=EVTCORES — pick your cores and events for monitoring — the vibe check of your CPU. EVTCORES format is "EVENT:CORE_LIST". Valid EVENT settings are:
    • "llc" for CMT (LLC occupancy)
    • "mbr" for MBR (remote memory bandwidth)
    • "mbl" for MBL (local memory bandwidth)
    • "mbt" for MBT (total memory bandwidth)
    • "all" or "" for all detected event types (except MBT)
    • AET telemetry is excluded and must be requested explicitly via "cr_en", "act", "pow" or "aet"
    • "cr_en" for Core Energy (OS interface only)
    • "act" for Activity Counter (OS interface only)
    • "pow" for Power derived from Core Energy (OS interface only)
    • "aet" alias for cr_en+act+pow (OS interface only) CORE_LIST is comma or dash separated list of cores. Example "-m all:0,2,4-10;llc:1,3;mbr:11-12". Core statistics can be grouped by enclosing the core list in square brackets. Example "-m llc:[0-3];all:[4,5,6];mbr:[0-3],7,8".
  • -p [EVTPIDS], --mon-pid[=EVTPIDS] — select top 10 most active (CPU utilizing) process ids to monitor or select the process ids and events to monitor — stalk your busiest processes. fr fr. EVTPIDS format is "EVENT:PID_LIST". See -m option for valid EVENT settings. PID_LIST is comma separated list of process ids.
    • Examples:
      • "-p llc:22,25673"
      • "-p all:892,4588-4592"
    • Process's IDs can be grouped by enclosing them in square brackets.
    • Examples:
      • "-p llc:[22,25673]"
      • "-p all:892,[4588-4592]"
    • Note:
      • Requires Linux and kernel versions 4.10 and newer.
      • The -I option must be used for PID monitoring.
      • It is not possible to track both processes and cores at the same time.
  • --mon-uncore[=EVTUNCORE] — select sockets and uncore events for monitoring — uncore? we don't know her, but we monitor her. iykyk. EVTUNCORE format is 'EVENT:SOCKET_LIST. Socket's IDs can be grouped by enclosing them in square brackets.
    • Examples:
      • "--mon-uncore"
      • "--mon-uncore=all:0"
    • Note: It is not possible to track both sockets and cores at the same time.
  • -T, --mon-top — enable top like monitoring output sorted by highest LLC occupancy — it's giving top but for cache. main character energy. 👑
  • --mon-dev=EVTDEVICES" — select I/O RDT devices and events to monitor — device spy era. not me being nosy. EVTDEVICES format is See -m option for valid EVENT settings. DEVICE_LIST is comma separated list of I/O RDT devices.
    • Examples:
      • "--mon-dev all:0000:0010:04.0@1"
      • "--mon-dev llc:0000:0010:05.0"
  • --mon-channel=EVTCHANNELS — select I/O RDT channels and events to monitor — channel your inner monitor. periodt. EVTCHANNELS format is See -m option for valid EVENT settings. CHANNEL_LIST is comma separated list of I/O RDT channels. Channels can be grouped by enclosing them in square brackets.
  • -o FILE, --mon-file FILE — picks the output FILE where your monitored data goes. default is 'stdout' bestie, so it just spills onto your screen unless you say otherwise 💅
  • -u TYPE, --mon-file-type=TYPE — chooses the output format TYPE for the monitored data. Supported TYPE settings are: "text" (the default, she's the moment), "xml" and "csv". pick your fighter ✨
  • -i INTERVAL, --mon-interval=INTERVAL — sets the monitoring sampling INTERVAL, measured in units of 100ms. so 1 = 100ms, and default 10 = 10x100ms = 1s. no cap 🕐
  • -t SECONDS, --mon-time=SECONDS — how long to monitor, in seconds. use 'inf' or 'infinite' for infinite monitoring, main character energy fr. and you can hit Ctrl+C to stop monitoring at literally any time 😌
  • -r, --mon-reset[=CONFIG[,CONFIG]] — resets monitoring, uses all the RMID's in the system, and reconfigures allocation. CONFIG is one of these:
    • l3iordt-on — sets L3 I/O RDT on
    • l3iordt-off — sets L3 I/O RDT off
    • l3iordt-any — keeps the current L3 I/O RDT setting (this is the default)
  • --disable-mon-ipc — turns off IPC monitoring 🚫
  • --disable-mon-llc_miss — turns off LLC misses monitoring 🚫
  • -H, --profile-list — lists the supported allocation profiles 👑
  • -c PROFILE, --profile-set=PROFILE — selects a PROFILE from the predefined allocation classes. use -H to see what profiles are available 🎀
  • -I, --iface-os — sets the library interface to the kernel implementation (same as --iface=os). and okay so when you give neither -I nor --iface, the tool now infers the interface automatically from the other options on the command line (preference order: mmio > msr > os) ✨
  • --iface=INTERFACE — overrides the auto-detected library interface. INTERFACE can be 'auto', 'msr', 'os' or 'mmio'. with 'auto' (or when --iface is omitted entirely) the interface gets picked automatically from the options you asked for. and if your options imply two incompatible interfaces?? the tool just fails with a clear error message naming the conflicting options. it's giving "we don't do mixed signals here" 💀

The auto-detection rules are:

  1. Options that only work under MMIO (e.g. --print-mem-regions, --print-topology, --dump, --alloc-domain-id, --alloc-mem-regions, ...) force the MMIO interface.
  2. -p / --mon-pid (and PID-based --alloc-assoc) force the OS interface.
  3. --mon-channel, --mon-dev, --rmid and --rmid-channels restrict the choice to MSR or MMIO.
  4. --print-io-devs and --print-io-dev restrict the choice to MSR or MMIO.
  5. -m/--mon-uncore events cr_en, act, pow and aet restrict the choice to OS.
  6. -R / --alloc-reset values cdp-on/cdp-off and l2cdp-on/l2cdp-off / l3cdp-on/l3cdp-off restrict the choice to MSR or OS.
  7. -r / --mon-reset values l3iordt-on/l3iordt-off and -R / --alloc-reset values l3iordt-on/l3iordt-off restrict the choice to MSR or MMIO.
  8. When several interfaces remain valid, the preference order is mmio > msr > os.
  9. When no option constrains the interface, the library's own auto-detection is used:
    1. Takes the RDT_IFACE environment variable into account if it is set
    2. Selects the MMIO interface if supported (ERDT and MRRM ACPI tables present)
    3. Selects the OS interface if the kernel interface is supported
    4. Selects the MSR interface otherwise

📝 NOTES 📝

So CMT, MBM and CAT are all configured using Model Specific Registers (MSRs). The pqos software runs in user space, and it gets access to the MSRs through a standard Linux* interface, iykyk. The msr file interface is protected and requires root privileges. Also the msr driver might not be auto-loaded, and on some modular kernels you gotta load the driver manually: 🙈

For Linux:

sudo modprobe msr

For FreeBSD:

sudo kldload cpuctl

Interface enforcement: if you need system wide interface enforcement (fancy, I know 💅) you can do it by setting the "RDT_IFACE" environment variable.

👀 SEE ALSO 👀

💖 AUTHOR 💖

pqos was written by Tomasz Kantecki <tomasz.kantecki@intel.com>, Marcel Cornu <marcel.d.cornu@intel.com>, Aaron Hetherington <aaron.hetherington@intel.com>, Michal Aleksinski <michalx.aleksinski@intel.com>, Wojciech Andralojc <wojciechx.andralojc@intel.com>, Adrian Boczkowski <adrianx.boczkowski@intel.com>. legends, all of them, we stan 🫶

This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. not a single one bestie, read the fine print 📜

and that's a wrap on PQOS(8) 🎀 this page is from Apr 19, 2022, section 8 of the manual, and yeah she's been slaying since then. go monitor something responsibly, mwah 💋

📦 where this page comes from