SAR(1)

๐Ÿ“ฆ
version 12.7.7 ยท 4 versions
, in ubuntu questing, ubuntu resolute, ubuntu stonking

๐Ÿ’– SAR(1) โ€” Linux User's Manual ๐Ÿ’–

okay bestie, this is the manual page for sar, and it's giving "system activity reporter" ๐Ÿฆ‹ it's part of the Linux User's Manual, so this is the official SAR(1) page, and I'm gonna walk you through every single bit of it โœจ

โœจ NAME โœจ

sar is the tool that collects, reports, or saves system activity information ๐Ÿ“Š literally all of it in one place ๐Ÿ’…

๐ŸŽ€ SYNOPSIS ๐ŸŽ€

sar [ -A ] [ -B ] [ -b ] [ -C ] [ -D ] [ -d ] [ -F [ MOUNT ] ] [ -H ] [
-h ] [ -p ] [ -r [ ALL ] ] [ -S ] [ -t ] [ -u [ ALL ] ] [ -V ] [ -v ] [
-W ] [ -w ] [ -x ] [ -y ] [ -z ] [ --dec={ 0 | 1 | 2 } ] [
--dev=dev_list ] [ --fs=fs_list ] [ --help ] [ --human ] [
--iface=iface_list ] [ --int=int_list ] [ --pretty ] [ --sadc ] [ -I [
SUM | ALL ] ] [ -P { cpu_list | ALL } ] [ -m { keyword[,...] | ALL } ] [
-n { keyword[,...] | ALL } ] [ -q [ keyword[,...] | ALL ] ] [ -j { SID |
ID | LABEL | PATH | UUID | ... } ] [ -f [ filename ] | -o [ filename ] |
-[0-9]+ ] [ -i interval ] [ -s [ start_time ] ] [ -e [ end_time ] ] ] [
interval [ count ] ]

๐ŸŒธ DESCRIPTION ๐ŸŒธ

okay so the sar command writes to standard output whatever is inside the selected cumulative activity counters in the operating system ๐Ÿ‘€ the accounting system uses the count and interval parameters to decide how many times it writes info and how far apart those writes are spaced, in seconds ๐Ÿซถ

if the interval parameter is set to zero, sar shows the average stats for the time since the system was started โฐ and if you give interval without count, the reports keep coming forever, continuously ๐Ÿ”

the collected data can also be saved to the file you specify with the -o filename flag, on top of showing up on your screen ๐Ÿ’พ if filename is left out, sar uses the standard system activity daily data file (more on that below) โœจ by default, all the data the kernel has available gets saved into the data file ๐Ÿ’…

the sar command can also extract records that were saved earlier and write them to standard output ๐Ÿ“‚ those records live in either the file you pick with the -f flag or, by default, the standard system activity daily data file ๐Ÿ’– you can also pass -1, -2 etc. as an argument to sar to display data from that many days ago ๐Ÿฆ‹ for example, -1 points at the standard system activity file of yesterday ๐ŸŒˆ

so the standard system activity daily data files are named saDD or saYYYYMMDD, where YYYY is the current year, MM is the current month, and DD is the current day ๐ŸŽ€ they're only the default files sar uses when no filename has been explicitly specified, so no need to stress ๐Ÿซถ when sar writes data to files (with the -o option), it uses saYYYYMMDD if the -D option was also specified, otherwise it uses saDD ๐Ÿ’… and when it's displaying records that were saved earlier, sar looks for the most recent of saDD and saYYYYMMDD and uses that one ๐Ÿ‘‘

the standard system activity daily data files live in /var/log/sysstat by default ๐Ÿ“ but you can totally point somewhere else for them โœจ if you use a directory (instead of a plain file) with the -f or -o options, then that directory gets treated as the directory holding the data files ๐Ÿงš

without the -P flag, sar reports system-wide stats (global across all processors) ๐Ÿ“Š those get calculated as averages for values written as percentages, and as sums for everything else ๐Ÿ’œ if you do use the -P flag, sar reports activity for the processor or processors you specify ๐Ÿ”ฅ and if you give -P ALL, it reports stats for each individual processor AND the global stats across all of them ๐Ÿคฏ offline processors are not displayed, so don't look for them ๐Ÿ™ˆ

you can select info about specific system activities using flags ๐ŸŽ€ if you don't specify any flags, sar only selects CPU activity ๐Ÿ’– specifying the -A flag selects all possible activities, so that's your everything button โœจ

the default version of sar (the CPU utilization report) is honestly probably one of the first things you'll run when you start investigating system activity ๐Ÿ” because it monitors the major system resources ๐Ÿ’… if CPU utilization is near 100 percent (user + nice + system), the sampled workload is CPU-bound ๐Ÿ”ฅ

if you want multiple samples and multiple reports, it's super convenient to specify an output file for sar ๐Ÿซถ run sar as a background process, and the syntax for that is:

sar -o datafile interval count >/dev/null 2>&1 &

all the data gets captured in binary form and saved to a file (datafile) ๐Ÿ’พ then you can selectively display it with sar using the -f option ๐Ÿฆ‹ set the interval and count parameters to select count records at interval second intervals โฑ๏ธ if count isn't set, all the records saved in the file get selected ๐Ÿ’… collecting data this way is hugely useful for characterizing system usage over a period of time and figuring out your peak usage hours ๐Ÿ“ˆ literally iconic ๐Ÿ‘‘

Note: the sar command only reports on local activities ๐Ÿ  so it won't be telling you about anything happening somewhere else ๐Ÿ™ˆ

๐Ÿ’œ OPTIONS ๐Ÿ’œ

  • -A โ€” this is the everything flag ๐ŸŒˆ it's equivalent to specifying -bBdFHISvwWy -m ALL -n ALL -q ALL -r ALL -u ALL. it also implies specifying -I ALL -P ALL, unless you've explicitly set those on the command line ๐Ÿ’–
  • -B โ€” reports paging statistics ๐Ÿ“„ these values get displayed:
    • pgpgin/s โ€” total number of kibibytes the system paged in from disk per second ๐Ÿ“ฅ
    • pgpgout/s โ€” total number of kibibytes the system paged out to disk per second ๐Ÿ“ค
    • fault/s โ€” number of page faults (major + minor) made by the system per second ๐Ÿ’€ this isn't a count of page faults that generate I/O, because some page faults can be resolved without I/O ๐Ÿ™ƒ
    • majflt/s โ€” number of major faults the system has made per second, meaning the ones that needed a memory page loaded from disk ๐Ÿซ 
    • pgfree/s โ€” number of pages placed on the free list by the system per second โœจ
    • pgscank/s โ€” number of pages scanned by the kswapd daemon per second ๐Ÿงš
    • pgscand/s โ€” number of pages scanned directly per second ๐Ÿ‘€
    • pgsteal/s โ€” number of pages the system has reclaimed from cache (pagecache and swapcache) per second to satisfy its memory demands ๐Ÿงน
    • pgprom/s โ€” number of pages promoted (i.e. migrated from slow to fast memory types) by the system per second ๐Ÿš€
    • pgdem/s โ€” number of pages demoted (i.e. migrated from fast to slow memory types) by the system per second ๐Ÿ‚
  • -b โ€” reports I/O and transfer rate statistics ๐Ÿ’ซ these values get displayed:
    • tps โ€” total number of transfers per second that were issued to physical devices ๐Ÿ”ฅ a transfer is an I/O request to a physical device, and multiple logical requests can be combined into a single I/O request to the device ๐Ÿซถ a transfer is of indeterminate size ๐Ÿคทโ€โ™€๏ธ
    • rtps โ€” total number of read requests per second issued to physical devices ๐Ÿ“–
    • wtps โ€” total number of write requests per second issued to physical devices โœ๏ธ
    • dtps โ€” total number of discard requests per second issued to physical devices ๐Ÿ—‘๏ธ
    • bread/s โ€” total amount of data read from the devices in blocks per second ๐Ÿ“š blocks are equivalent to sectors, so they have a size of 512 bytes ๐Ÿ’…
    • bwrtn/s โ€” total amount of data written to devices in blocks per second ๐Ÿ’พ
    • bdscd/s โ€” total amount of data discarded for devices in blocks per second ๐Ÿƒ
  • -C โ€” when reading data from a file, tells sar to display comments that have been inserted by sadc ๐Ÿ’ฌ
  • -D โ€” uses saYYYYMMDD instead of saDD as the standard system activity daily data file name ๐Ÿ“… this option only works when it's used alongside the -o option to save data to a file ๐Ÿซถ
  • -d โ€” reports activity for each block device ๐Ÿ’ฟ when data gets displayed, the device name is shown the way it (should) appear in /dev โœจ sar uses data in /sys to figure out the device name from its major and minor numbers ๐Ÿ”Ž if that name resolution fails, sar falls back to the name mapping controlled by the /etc/sysstat/sysstat.ioconf file ๐Ÿ—บ๏ธ persistent device names can also be printed if the -j option is used (see below) ๐Ÿท๏ธ stats for all devices are displayed unless you give a restricted list with the --dev= option (see its entry) ๐Ÿ“‹ heads up: disk activity depends on sadc's options -S DISK and -S XDISK being collected ๐Ÿšจ these values get displayed:
    • tps โ€” total number of transfers per second that were issued to physical devices ๐Ÿ”ฅ a transfer is an I/O request to a physical device, multiple logical requests can be merged into one, and a transfer is of indeterminate size ๐Ÿซถ
    • rkB/s โ€” number of kibibytes read from the device per second ๐Ÿ“–
    • wkB/s โ€” number of kibibytes written to the device per second โœ๏ธ
    • dkB/s โ€” number of kibibytes discarded for the device per second ๐Ÿ—‘๏ธ
    • areq-sz โ€” the average size (in kibibytes) of the I/O requests that were issued to the device ๐Ÿ“
      • Note: in previous versions, this field was known as avgrq-sz and was expressed in sectors ๐Ÿ™ˆ
    • aqu-sz โ€” the average queue length of the requests that were issued to the device ๐Ÿงโ€โ™€๏ธ๐Ÿงโ€โ™€๏ธ๐Ÿงโ€โ™€๏ธ
      • Note: in previous versions, this field was known as avgqu-sz ๐Ÿ•ฐ๏ธ
    • await โ€” the average time (in milliseconds) for I/O requests issued to the device to be served โณ this includes the time the requests spend in the queue AND the time spent servicing them ๐Ÿ’€
    • %util โ€” percentage of elapsed time during which I/O requests were issued to the device (the bandwidth utilization for the device) ๐Ÿ“Š device saturation happens when this value is close to 100% for devices serving requests serially ๐Ÿ”ฅ but for devices serving requests in parallel, like RAID arrays and modern SSDs, this number does NOT reflect their performance limits ๐Ÿ™…โ€โ™€๏ธ
  • --dec={ 0 | 1 | 2 } โ€” specifies the number of decimal places to use (0 to 2, default value is 2) ๐Ÿ”ข
  • --dev=dev_list โ€” specifies the block devices whose statistics should be displayed by sar ๐Ÿ’ฟ dev_list is a list of comma-separated device names ๐Ÿซถ

-e [ hh:mm[:ss] ] / -e [ seconds_since_the_epoch ] โœจ

  • -e [ hh:mm[:ss] ], -e [ seconds_since_the_epoch ] โ€” this one sets when the report ends ๐Ÿ•• The default ending time is 18:00:00. You can give hours in 24-hour format, or as seconds since the epoch (a 10 digit number) ๐Ÿงฎ Only really matters when you're reading from or writing to a file (-f or -o) ๐Ÿ“
  • -F [ MOUNT ] โ€” show stats for the filesystems that are currently mounted ๐Ÿ’พ Pseudo-filesystems get ignored, no drama ๐Ÿ™…โ€โ™€๏ธ At the very end, sar drops a summary of all of them. Pass the MOUNT keyword and it reports the mountpoint instead of the filesystem device ๐Ÿ—บ๏ธ You get stats for every filesystem unless you narrow it down with --fs= (check that option's entry ๐Ÿ‘€). Also, filesystem stats only show up if sadc was run with its -S XDISK option ๐Ÿ”ฅ

The following values are displayed:

  • MBfsfree โ€” total free space in mebibytes, including the stuff only the privileged user can touch ๐Ÿ”“
  • MBfsused โ€” total space used in mebibytes ๐Ÿ“ฆ
  • %fsused โ€” percentage of filesystem space used, as a privileged user sees it ๐Ÿ‘‘
  • %ufsused โ€” percentage of filesystem space used, as an unprivileged user sees it ๐Ÿ™‹โ€โ™€๏ธ
  • Ifree โ€” total number of free file nodes (inodes) in the filesystem ๐Ÿ†“
  • Iused โ€” total number of file nodes used in the filesystem ๐Ÿ“Š
  • %Iused โ€” percentage of file nodes used in the filesystem ๐Ÿ’ฏ
  • -f [ filename ] โ€” pull records out of filename (the one you made with -o filename) ๐Ÿ—‚๏ธ If you don't give a name, it defaults to the current standard system activity daily data file ๐Ÿ“… If filename is a directory instead of a plain file, sar treats it as the folder where the standard daily data files live ๐Ÿ—ƒ๏ธ Heads up: -f and -o can't be used together, they're exclusive ๐Ÿ™…โ€โ™€๏ธ
  • --fs=fs_list โ€” pick which filesystems sar should show stats for ๐ŸŽฏ fs_list is a comma-separated list of filesystem names or mountpoints ๐Ÿงพ
  • -H โ€” report hugepages utilization stats ๐Ÿง  The following values are displayed:
    • kbhugfree โ€” amount of hugepages memory in kibibytes that hasn't been allocated yet ๐Ÿ†—
    • kbhugused โ€” amount of hugepages memory in kibibytes that has been allocated ๐Ÿ™Œ
    • %hugused โ€” percentage of total hugepages memory that's been allocated ๐Ÿ“ˆ
    • kbhugrsvd โ€” amount of reserved hugepages memory in kibibytes ๐Ÿ”’
    • kbhugsurp โ€” amount of surplus hugepages memory in kibibytes ๐ŸŒŠ
  • -h โ€” this is the same as typing --pretty --human ๐Ÿซถ
  • --help โ€” shows a short help message, then exits ๐Ÿ†˜
  • --human โ€” prints sizes in a human readable format, like 1.0k or 1.2M ๐Ÿ“ The units this gives you override any other default units tied to the metrics, like kibibytes or sectors. Main character energy โœจ
  • -I [ SUM | ALL ] โ€” report interrupt stats ๐ŸŒฉ๏ธ You get the number of interrupts per second, either for a given processor or across all processors ๐Ÿ–ฅ๏ธ You can narrow it to a list of interrupts with --int= (go read that entry ๐Ÿ“–). The SUM keyword shows the total number of interrupts received per second ๐Ÿงฎ The ALL keyword reports stats from every interrupt, and it's the default ๐Ÿซง Note that this needs sadc's -S INT option to have been used to collect the data ๐Ÿ”ฅ
  • -i interval โ€” selects data records at seconds as close as possible to the number you give for interval ๐ŸŽฏ
  • --iface=iface_list โ€” choose which network interfaces sar shows stats for ๐ŸŒ iface_list is a comma-separated list of interface names ๐Ÿงต
  • --int=int_list โ€” choose which interrupt names sar shows stats for ๐ŸŽ€ int_list is a comma-separated list of values or ranges, like 0-16,35,40- ๐Ÿง
  • -j { SID | ID | LABEL | PATH | UUID | ... } โ€” displays persistent device names ๐Ÿ’Ž Use it together with -d ๐Ÿค The keywords ID, LABEL, and so on tell it which type of persistent name you want. The list isn't limited to these, the only real requirement is that the directory holding the persistent names exists in /dev/disk ๐Ÿ“‚ The SID keyword tries to get a stable identifier to use as the device name ๐Ÿชช A stable identifier won't change across reboots for the same physical device. When one exists, it's usually the WWN (World Wide Name) of the device, read from the /dev/disk/by-id directory ๐Ÿ”
  • -m { keyword[,...] | ALL } โ€” report power management stats โšก Heads up: these need sadc's -S POWER option to have been used when collecting ๐Ÿ”Œ

Possible keywords are BAT, CPU, FAN, FREQ, IN, TEMP and USB ๐ŸŒˆ

  • BAT โ€” battery capacity stats get reported ๐Ÿ”‹ The following values are displayed:
    • %cap โ€” battery capacity ๐Ÿ’ช
    • cap/min โ€” capacity the battery lost or gained per minute ๐Ÿ“‰๐Ÿ“ˆ
    • status โ€” charging status of the battery: โ†‘ (full), โ†— (charging), โ†’ (not charging), โ†˜ (discharging), ? (unknown) ๐Ÿ•น๏ธ
  • CPU โ€” CPU stats get reported ๐Ÿง  The following value is displayed:
    • MHz โ€” instantaneous CPU clock frequency in MHz ๐ŸŽ๏ธ
  • FAN โ€” fan speed stats get reported ๐ŸŒ€ The following values are displayed:
    • rpm โ€” fan speed in revolutions per minute ๐Ÿ”„
    • drpm โ€” calculated as the difference between the current fan speed (rpm) and its low limit (fan_min) ๐Ÿ“
    • DEVICE โ€” sensor device name ๐Ÿท๏ธ
  • FREQ โ€” CPU clock frequency stats get reported ๐ŸŽข The following value is displayed:
    • wghMHz โ€” weighted average CPU clock frequency in MHz ๐Ÿงฎ Also, the cpufreq-stats driver has to be compiled into the kernel for this to work, no cap ๐Ÿงข
  • IN โ€” voltage input stats get reported โšก The following values are displayed:
    • inV โ€” voltage input in Volts ๐Ÿ”‹
    • %in โ€” relative input value. 100% means the voltage input has hit its high limit (in_max), and 0% means it hit its low limit (in_min) ๐ŸŽš๏ธ
    • DEVICE โ€” sensor device name ๐Ÿท๏ธ
  • TEMP โ€” device temperature stats get reported ๐ŸŒก๏ธ The following values are displayed:
    • degC โ€” device temperature in degrees Celsius ๐Ÿฅต
    • %temp โ€” relative device temperature. 100% means it's reached its high limit (temp_max) ๐Ÿ”ฅ
    • DEVICE โ€” sensor device name ๐Ÿท๏ธ
  • USB โ€” sar takes a snapshot of every USB device plugged into the system right now ๐Ÿ”Œ At the end of the report, it shows a summary of all those USB devices ๐Ÿงพ The following values are displayed:
    • BUS โ€” root hub number of the USB device ๐ŸšŒ
    • idvendor โ€” vendor ID number (assigned by the USB organization) ๐Ÿฌ
    • idprod โ€” product ID number (assigned by the manufacturer) ๐Ÿ†”
    • maxpower โ€” maximum power consumption of the device, in mA โšก
    • manufact โ€” manufacturer name ๐Ÿญ
    • product โ€” product name ๐Ÿ›๏ธ

The ALL keyword is the same as listing every keyword above, so you get all the power management stats ๐Ÿ‘‘

  • -n { keyword[,...] | ALL } โ€” report network stats ๐ŸŒ

Possible keywords are DEV, EDEV, FC, ICMP, EICMP, ICMP6, EICMP6, IP, EIP, IP6, EIP6, NFS, NFSD, SOCK, SOCK6, SOFT, TCP, ETCP, UDP and UDP6 ๐Ÿงฉ

  • DEV โ€” stats from the network devices get reported ๐Ÿ“ก You get every network interface unless you narrow it down with --iface= (see that entry ๐Ÿ‘€). The following values are displayed:
    • IFACE โ€” name of the network interface being reported ๐Ÿท๏ธ
    • rxpck/s โ€” total number of packets received per second ๐Ÿ“ฅ
    • txpck/s โ€” total number of packets transmitted per second ๐Ÿ“ค
    • rxkB/s โ€” total number of kibibytes received per second ๐Ÿ’ง
    • txkB/s โ€” total number of kibibytes transmitted per second ๐ŸŒŠ
    • rxcmp/s โ€” number of compressed packets received per second (for cslip and friends) ๐Ÿ—œ๏ธ
    • txcmp/s โ€” number of compressed packets transmitted per second ๐Ÿ—œ๏ธ
    • rxmcst/s โ€” number of multicast packets received per second ๐Ÿ“ข
    • %ifutil โ€” utilization percentage of the network interface. For half-duplex interfaces, it's calculated using the sum of rxkB/s and txkB/s as a percentage of the interface speed. For full-duplex, it's the greater of rxkB/S or txkB/s ๐Ÿงฎ
  • EDEV โ€” stats on failures (errors) from the network devices get reported ๐Ÿšจ Again, you get every interface unless you narrow it with --iface= (see corresponding entry ๐Ÿ‘€). The following values are displayed:
    • IFACE โ€” name of the network interface being reported ๐Ÿท๏ธ
    • rxerr/s โ€” total number of bad packets received per second ๐Ÿ™…โ€โ™€๏ธ
    • txerr/s โ€” total number of errors that happened per second while transmitting packets ๐Ÿ’ฅ
    • coll/s โ€” number of collisions that happened per second while transmitting packets ๐Ÿš—๐Ÿ’ฅ๐Ÿš—
    • rxdrop/s โ€” number of received packets dropped per second because linux buffers ran out of space ๐Ÿชฃ
    • txdrop/s โ€” number of transmitted packets dropped per second because linux buffers ran out of space ๐Ÿซ—
    • txcarr/s โ€” number of carrier-errors that happened per second while transmitting packets ๐Ÿ“ก๐Ÿ’€
    • rxfram/s โ€” number of frame alignment errors that happened per second on received packets ๐Ÿงฑ
  • rxfifo/s โ€” bestie, this one counts the FIFO overrun errors per second on received packets ๐Ÿ“ฅ๐Ÿ’€
  • txfifo/s โ€” same chaotic energy, but for transmitted packets ๐Ÿ“คโœจ

๐Ÿ’พ FC

With the FC keyword, we get the fibre channel traffic stats ๐Ÿซถ Heads up: these depend on sadc's option -S DISK being collected, so make sure that's on. Here's what you get:

  • FCHOST โ€” the name of the fibre channel host bus adapter (HBA) interface the stats are for ๐Ÿงš
  • fch_rxf/s โ€” the total number of frames received per second ๐Ÿ“จ
  • fch_txf/s โ€” the total number of frames transmitted per second ๐Ÿ“ค
  • fch_rxw/s โ€” the total number of transmission words received per second ๐Ÿ’œ
  • fch_txw/s โ€” the total number of transmission words transmitted per second ๐ŸŒธ

๐ŸŒ ICMP

With the ICMP keyword, we're looking at ICMPv4 network traffic ๐Ÿฆ‹ Note that ICMPv4 stats depend on sadc's option -S SNMP being collected. The formal SNMP names are in square brackets, so you can match them up:

  • imsg/s โ€” the total number of ICMP messages the entity received per second [icmpInMsgs]. This counter includes everything counted by ierr/s, so it's a big-picture one ๐Ÿ‘‘
  • omsg/s โ€” the total number of ICMP messages this entity tried to send per second [icmpOutMsgs]. This one also includes everything counted by oerr/s ๐Ÿ’…
  • iech/s โ€” ICMP Echo (request) messages received per second [icmpInEchos] ๐Ÿ“
  • iechr/s โ€” ICMP Echo Reply messages received per second [icmpInEchoReps] ๐Ÿ”
  • oech/s โ€” ICMP Echo (request) messages sent per second [icmpOutEchos] ๐Ÿš€
  • oechr/s โ€” ICMP Echo Reply messages sent per second [icmpOutEchoReps] ๐ŸŽ€
  • itm/s โ€” ICMP Timestamp (request) messages received per second [icmpInTimestamps] โฐ
  • itmr/s โ€” ICMP Timestamp Reply messages received per second [icmpInTimestampReps] โฑ๏ธ
  • otm/s โ€” ICMP Timestamp (request) messages sent per second [icmpOutTimestamps] ๐Ÿ•ฐ๏ธ
  • otmr/s โ€” ICMP Timestamp Reply messages sent per second [icmpOutTimestampReps] ๐Ÿง
  • iadrmk/s โ€” ICMP Address Mask Request messages received per second [icmpInAddrMasks] ๐ŸŽญ
  • iadrmkr/s โ€” ICMP Address Mask Reply messages received per second [icmpInAddrMaskReps] ๐ŸŽญ
  • oadrmk/s โ€” ICMP Address Mask Request messages sent per second [icmpOutAddrMasks] ๐Ÿซง
  • oadrmkr/s โ€” ICMP Address Mask Reply messages sent per second [icmpOutAddrMaskReps] ๐Ÿซง

๐Ÿšจ EICMP

With the EICMP keyword, we're getting into the ICMPv4 error messages ๐Ÿ˜ญ Same deal: these depend on sadc's option -S SNMP. SNMP names are in square brackets again:

  • ierr/s โ€” ICMP messages per second that the entity received but flagged as having ICMP-specific errors (bad ICMP checksums, bad length, etc.) [icmpInErrors] ๐Ÿ™ˆ
  • oerr/s โ€” ICMP messages per second that this entity didn't send because of problems found within ICMP, like a lack of buffers [icmpOutErrors] ๐Ÿซ 
  • idstunr/s โ€” ICMP Destination Unreachable messages received per second [icmpInDestUnreachs] ๐Ÿšง
  • odstunr/s โ€” ICMP Destination Unreachable messages sent per second [icmpOutDestUnreachs] ๐Ÿšง
  • itmex/s โ€” ICMP Time Exceeded messages received per second [icmpInTimeExcds] โŒ›
  • otmex/s โ€” ICMP Time Exceeded messages sent per second [icmpOutTimeExcds] โŒ›
  • iparmpb/s โ€” ICMP Parameter Problem messages received per second [icmpInParmProbs] ๐Ÿงฉ
  • oparmpb/s โ€” ICMP Parameter Problem messages sent per second [icmpOutParmProbs] ๐Ÿงฉ
  • isrcq/s โ€” ICMP Source Quench messages received per second [icmpInSrcQuenchs] ๐Ÿซฅ
  • osrcq/s โ€” ICMP Source Quench messages sent per second [icmpOutSrcQuenchs] ๐Ÿซฅ
  • iredir/s โ€” ICMP Redirect messages received per second [icmpInRedirects] ๐Ÿ”€
  • oredir/s โ€” ICMP Redirect messages sent per second [icmpOutRedirects] ๐Ÿ”€

๐ŸŒ™ ICMP6

With the ICMP6 keyword, it's ICMPv6 network traffic time ๐ŸŒˆ These depend on sadc's option -S IPV6 being collected. Formal SNMP names are in square brackets:

  • imsg6/s โ€” the total number of ICMP messages the interface received per second, including everything counted by ierr6/s [ipv6IfIcmpInMsgs] ๐Ÿ‘‘
  • omsg6/s โ€” the total number of ICMP messages this interface tried to send per second [ipv6IfIcmpOutMsgs] ๐Ÿ’…
  • iech6/s โ€” ICMP Echo (request) messages the interface received per second [ipv6IfIcmpInEchos] ๐Ÿ“
  • iechr6/s โ€” ICMP Echo Reply messages the interface received per second [ipv6IfIcmpInEchoReplies] ๐Ÿ”
  • oechr6/s โ€” ICMP Echo Reply messages the interface sent per second [ipv6IfIcmpOutEchoReplies] ๐ŸŽ€
  • igmbq6/s โ€” ICMPv6 Group Membership Query messages the interface received per second [ipv6IfIcmpInGroupMembQueries] ๐Ÿซง
  • igmbr6/s โ€” ICMPv6 Group Membership Response messages the interface received per second [ipv6IfIcmpInGroupMembResponses] ๐Ÿซง
  • ogmbr6/s โ€” ICMPv6 Group Membership Response messages sent per second [ipv6IfIcmpOutGroupMembResponses] ๐Ÿซง
  • igmbrd6/s โ€” ICMPv6 Group Membership Reduction messages the interface received per second [ipv6IfIcmpInGroupMembReductions] ๐Ÿช„
  • ogmbrd6/s โ€” ICMPv6 Group Membership Reduction messages sent per second [ipv6IfIcmpOutGroupMembReductions] ๐Ÿช„
  • irtsol6/s โ€” ICMP Router Solicit messages the interface received per second [ipv6IfIcmpInRouterSolicits] ๐Ÿงญ
  • ortsol6/s โ€” ICMP Router Solicitation messages the interface sent per second [ipv6IfIcmpOutRouterSolicits] ๐Ÿงญ
  • irtad6/s โ€” ICMP Router Advertisement messages the interface received per second [ipv6IfIcmpInRouterAdvertisements] ๐Ÿ“ฃ
  • inbsol6/s โ€” ICMP Neighbor Solicit messages the interface received per second [ipv6IfIcmpInNeighborSolicits] ๐Ÿ˜๏ธ
  • onbsol6/s โ€” ICMP Neighbor Solicitation messages the interface sent per second [ipv6IfIcmpOutNeighborSolicits] ๐Ÿ˜๏ธ
  • inbad6/s โ€” ICMP Neighbor Advertisement messages the interface received per second [ipv6IfIcmpInNeighborAdvertisements] ๐Ÿ 
  • onbad6/s โ€” ICMP Neighbor Advertisement messages the interface sent per second [ipv6IfIcmpOutNeighborAdvertisements] ๐Ÿ 

๐Ÿ’ฅ EICMP6

With the EICMP6 keyword, we're checking the ICMPv6 error messages ๐Ÿฅฒ These depend on sadc's option -S IPV6. Formal SNMP names are in square brackets:

  • ierr6/s โ€” ICMP messages per second that the interface received but flagged as having ICMP-specific errors (bad ICMP checksums, bad length, etc.) [ipv6IfIcmpInErrors] ๐Ÿ™ˆ
  • idtunr6/s โ€” ICMP Destination Unreachable messages the interface received per second [ipv6IfIcmpInDestUnreachs] ๐Ÿšง
  • odtunr6/s โ€” ICMP Destination Unreachable messages the interface sent per second [ipv6IfIcmpOutDestUnreachs] ๐Ÿšง
  • itmex6/s โ€” ICMP Time Exceeded messages the interface received per second [ipv6IfIcmpInTimeExcds] โŒ›
  • otmex6/s โ€” ICMP Time Exceeded messages the interface sent per second [ipv6IfIcmpOutTimeExcds] โŒ›
  • iprmpb6/s โ€” ICMP Parameter Problem messages the interface received per second [ipv6IfIcmpInParmProblems] ๐Ÿงฉ
  • oprmpb6/s โ€” ICMP Parameter Problem messages the interface sent per second [ipv6IfIcmpOutParmProblems] ๐Ÿงฉ
  • iredir6/s โ€” Redirect messages the interface received per second [ipv6IfIcmpInRedirects] ๐Ÿ”€
  • oredir6/s โ€” Redirect messages the interface sent by second [ipv6IfIcmpOutRedirects] ๐Ÿ”€
  • ipck2b6/s โ€” ICMP Packet Too Big messages the interface received per second [ipv6IfIcmpInPktTooBigs] ๐Ÿ“ฆ
  • opck2b6/s โ€” ICMP Packet Too Big messages the interface sent per second [ipv6IfIcmpOutPktTooBigs] ๐Ÿ“ฆ

๐ŸŒ IP

With the IP keyword, we're getting the IPv4 network traffic stats ๐Ÿ’– Note that IPv4 stats depend on sadc's option -S SNMP being collected. Formal SNMP names are in square brackets, so keep an eye out for them:

  • irec/s โ€” Okay so this is the total number of input datagrams that came in from interfaces per second, and yes, that includes the ones that arrived broken ๐Ÿ’€ [ipInReceives].
  • fwddgm/s โ€” Datagrams per second that weren't meant for this entity at all, so it went hunting for a route to pass them along to their real final destination ๐Ÿฆ‹ [ipForwDatagrams].
  • idel/s โ€” The total number of input datagrams successfully delivered per second up to the IP user-protocols, ICMP included ๐Ÿ’Œ [ipInDelivers].
  • orq/s โ€” The total number of IP datagrams that local IP user-protocols (ICMP included) handed to IP per second as transmission requests ๐Ÿ“ฎ [ipOutRequests]. Heads up, this counter doesn't include anything already counted in fwddgm/s ๐Ÿ‘€.
  • asmrq/s โ€” The number of IP fragments received per second that had to be reassembled here ๐Ÿงฉ [ipReasmReqds].
  • asmok/s โ€” The number of IP datagrams successfully reassembled per second โœจ [ipReasmOKs].
  • fragok/s โ€” The number of IP datagrams successfully fragmented at this entity per second ๐Ÿฐ [ipFragOKs].
  • fragcrt/s โ€” The number of IP datagram fragments generated per second because of fragmentation at this entity ๐Ÿซถ [ipFragCreates].

With the EIP keyword, you get the IPv4 network error stats. Real talk, IPv4 stats need sadc's option -S SNMP to be collected, or they won't show up ๐Ÿ™ˆ. These are the values displayed (formal SNMP names in square brackets):

  • ihdrerr/s โ€” The number of input datagrams discarded per second because of errors in their IP headers. That covers bad checksums, version number mismatches, other format errors, time-to-live exceeded, errors found while processing their IP options, and so on ๐Ÿ’€ [ipInHdrErrors].
  • iadrerr/s โ€” The number of input datagrams discarded per second because the IP address in their header's destination field wasn't valid to receive at this entity. This includes invalid addresses (like 0.0.0.0) and addresses from unsupported Classes (like Class E). For entities that aren't IP routers, and therefore don't forward datagrams, it also counts datagrams dropped because the destination wasn't a local address ๐Ÿšซ [ipInAddrErrors].
  • iukwnpr/s โ€” The number of locally-addressed datagrams received successfully but discarded per second because the protocol was unknown or unsupported ๐Ÿคทโ€โ™€๏ธ [ipInUnknownProtos].
  • idisc/s โ€” The number of input IP datagrams per second that had no problems stopping them from being processed, but got discarded anyway (for example, for lack of buffer space) ๐Ÿ—‘๏ธ [ipInDiscards]. This counter doesn't include datagrams discarded while waiting for reassembly.
  • odisc/s โ€” The number of output IP datagrams per second that had no problem stopping them from reaching their destination, but got discarded anyway (for example, for lack of buffer space) ๐Ÿ“ค [ipOutDiscards]. Note that this counter would include datagrams counted in fwddgm/s if any of them met this (discretionary) discard criterion.
  • onort/s โ€” The number of IP datagrams discarded per second because no route could be found to get them to their destination ๐Ÿงญ [ipOutNoRoutes]. This counter includes any packets counted in fwddgm/s that meet this 'no-route' criterion. It also includes any datagrams a host can't route because all of its default routers are down.
  • asmf/s โ€” The number of failures per second detected by the IP reassembly algorithm, for whatever reason (timed out, errors, etc.) ๐Ÿ’” [ipReasmFails]. This isn't necessarily a count of discarded IP fragments, because some algorithms can lose track of how many fragments there are when they combine them as they arrive.
  • fragf/s โ€” The number of IP datagrams discarded per second because they needed to be fragmented at this entity but couldn't be, for example because their Don't Fragment flag was set ๐Ÿšง [ipFragFails].

With the IP6 keyword, you get the IPv6 network traffic stats. Same deal as before: IPv6 stats need sadc's option -S IPV6 to be collected ๐Ÿ’…. These are the values displayed (formal SNMP names in square brackets):

  • irec6/s โ€” The total number of input datagrams received from interfaces per second, including the ones received in error [ipv6IfStatsInReceives].
  • fwddgm6/s โ€” The number of output datagrams per second that this entity received and forwarded on to their final destinations ๐Ÿฆ‹ [ipv6IfStatsOutForwDatagrams].
  • idel6/s โ€” The total number of datagrams successfully delivered per second to the IPv6 user-protocols, ICMP included ๐Ÿ’Œ [ipv6IfStatsInDelivers].
  • orq6/s โ€” The total number of IPv6 datagrams that local IPv6 user-protocols (ICMP included) supplied per second to IPv6 as transmission requests ๐Ÿ“ฎ [ipv6IfStatsOutRequests]. This counter doesn't include any datagrams counted in fwddgm6/s.
  • asmrq6/s โ€” The number of IPv6 fragments received per second that needed to be reassembled at this interface ๐Ÿงฉ [ipv6IfStatsReasmReqds].
  • asmok6/s โ€” The number of IPv6 datagrams successfully reassembled per second โœจ [ipv6IfStatsReasmOKs].
  • imcpck6/s โ€” The number of multicast packets received per second by the interface ๐Ÿ“ก [ipv6IfStatsInMcastPkts].
  • omcpck6/s โ€” The number of multicast packets transmitted per second by the interface ๐Ÿ“ฃ [ipv6IfStatsOutMcastPkts].
  • fragok6/s โ€” The number of IPv6 datagrams successfully fragmented at this output interface per second ๐Ÿฐ [ipv6IfStatsOutFragOKs].
  • fragcr6/s โ€” The number of output datagram fragments generated per second because of fragmentation at this output interface ๐Ÿซถ [ipv6IfStatsOutFragCreates].

With the EIP6 keyword, you get the IPv6 network error stats. Once again, these depend on sadc's option -S IPV6 being collected ๐ŸŒธ. These are the values displayed (formal SNMP names in square brackets):

  • ihdrer6/s โ€” The number of input datagrams discarded per second because of errors in their IPv6 headers. That includes version number mismatches, other format errors, hop count exceeded, errors found while processing their IPv6 options, and so on ๐Ÿ’€ [ipv6IfStatsInHdrErrors]
  • iadrer6/s โ€” The number of input datagrams discarded per second because the IPv6 address in their header's destination field wasn't valid to receive at this entity. This includes invalid addresses (like ::0) and unsupported addresses (like addresses with unallocated prefixes). For entities that aren't IPv6 routers, and therefore don't forward datagrams, it also counts datagrams dropped because the destination wasn't a local address ๐Ÿšซ [ipv6IfStatsInAddrErrors].
  • iukwnp6/s โ€” The number of locally-addressed datagrams received successfully but discarded per second because the protocol was unknown or unsupported ๐Ÿคทโ€โ™€๏ธ [ipv6IfStatsInUnknownProtos].
  • i2big6/s โ€” The number of input datagrams per second that couldn't be forwarded because their size was bigger than the link MTU of the outgoing interface ๐Ÿ“ [ipv6IfStatsInTooBigErrors].
  • idisc6/s โ€” The number of input IPv6 datagrams per second that had no problems stopping them from being processed, but got discarded anyway (for example, for lack of buffer space) ๐Ÿ—‘๏ธ [ipv6IfStatsInDiscards]. This counter doesn't include datagrams discarded while waiting for reassembly.
  • odisc6/s โ€” The number of output IPv6 datagrams per second that had no problem stopping them from reaching their destination, but got discarded anyway (for example, for lack of buffer space) ๐Ÿ“ค [ipv6IfStatsOutDiscards]. This counter would include datagrams counted in fwddgm6/s if any of them met this (discretionary) discard criterion.
  • inort6/s โ€” The number of input datagrams discarded per second because no route could be found to get them to their destination ๐Ÿงญ [ipv6IfStatsInNoRoutes].
  • onort6/s โ€” The number of locally generated IP datagrams discarded per second because no route could be found to get them to their destination ๐Ÿงญ [unknown formal SNMP name].
  • asmf6/s โ€” The number of failures per second detected by the IPv6 reassembly algorithm, for whatever reason (timed out, errors, etc.) ๐Ÿ’” [ipv6IfStatsReasmFails]. This isn't necessarily a count of discarded IPv6 fragments, because some algorithms can lose track of how many fragments there are when they combine them as they arrive.
  • fragf6/s โ€” okay so this one counts the IPv6 datagrams that got tossed each second because they needed fragmenting at this output interface but couldn't be ๐Ÿ’€ [ipv6IfStatsOutFragFails].
  • itrpck6/s โ€” the input datagrams dropped each second because the datagram frame didn't carry enough data to be valid ๐Ÿ™ˆ [ipv6IfStatsInTruncatedPkts].

With the NFS keyword, we get the receipts on NFS client activity ๐Ÿงพ The following values are displayed:

  • call/s โ€” how many RPC requests the client made per second ๐Ÿ“ž
  • retrans/s โ€” how many of those RPC requests per second had to be sent again, like when the server timed out ๐Ÿ˜ค
  • read/s โ€” how many 'read' RPC calls were made per second ๐Ÿ“–
  • write/s โ€” how many 'write' RPC calls were made per second โœ๏ธ
  • access/s โ€” how many 'access' RPC calls were made per second ๐Ÿ”‘
  • getatt/s โ€” how many 'getattr' RPC calls were made per second ๐Ÿ‘€

With the NFSD keyword, we get the tea on NFS server activity โ˜• The following values are displayed:

  • scall/s โ€” how many RPC requests the server received per second ๐Ÿ“ฅ
  • badcall/s โ€” how many bad RPC requests came in per second, the ones whose processing ended up throwing an error ๐Ÿ’€
  • packet/s โ€” how many network packets were received per second ๐Ÿ“ฆ
  • udp/s โ€” how many UDP packets were received per second ๐Ÿฆ‹
  • tcp/s โ€” how many TCP packets were received per second ๐Ÿงต
  • hit/s โ€” how many reply cache hits happened per second ๐ŸŽฏ
  • miss/s โ€” how many reply cache misses happened per second ๐Ÿ™ƒ
  • sread/s โ€” how many 'read' RPC calls were received per second ๐Ÿ“–
  • swrite/s โ€” how many 'write' RPC calls were received per second โœ๏ธ
  • saccess/s โ€” how many 'access' RPC calls were received per second ๐Ÿ”‘
  • sgetatt/s โ€” how many 'getattr' RPC calls were received per second ๐Ÿ‘€

With the SOCK keyword, we check the socket situation that's in use (IPv4) ๐Ÿงฆ The following values are displayed:

  • totsck โ€” the total number of sockets the system is using, full stop ๐ŸŒธ
  • tcpsck โ€” how many TCP sockets are in use right now ๐Ÿงต
  • udpsck โ€” how many UDP sockets are in use right now ๐Ÿฆ‹
  • rawsck โ€” how many RAW sockets are in use right now ๐Ÿฅฉ
  • ip-frag โ€” how many IP fragments are sitting in the queue right now ๐Ÿงฉ
  • tcp-tw โ€” how many TCP sockets are in the TIME_WAIT state โณ

With the SOCK6 keyword, we check the socket situation in use (IPv6) ๐ŸŒ Heads up, IPv6 stats only show up if you collect them with sadc's option -S IPV6 โš ๏ธ The following values are displayed:

  • tcp6sck โ€” how many TCPv6 sockets are in use right now ๐Ÿงต
  • udp6sck โ€” how many UDPv6 sockets are in use right now ๐Ÿฆ‹
  • raw6sck โ€” how many RAWv6 sockets are in use right now ๐Ÿฅฉ
  • ip6-frag โ€” how many IPv6 fragments are currently in use ๐Ÿงฉ

With the SOFT keyword, we peek at the software-based network processing ๐Ÿง  The following values are displayed:

  • total/s โ€” the total number of network frames processed per second ๐Ÿ“Š
  • dropd/s โ€” the total number of network frames dropped per second because the processing queue had no room left ๐Ÿ—‘๏ธ
  • squeezd/s โ€” how many times per second the softirq handler function stopped because its budget ran out or it hit the time limit, even though there was still more work it could've done ๐Ÿ˜ฉ
  • rx_rps/s โ€” how many times per second the CPU got woken up to process packets via an inter-processor interrupt โฐ
  • flw_lim/s โ€” how many times per second the flow limit got reached ๐Ÿšง Flow limiting is an optional RPS feature that caps how many packets can be queued to the backlog for each flow, to a certain amount. It's giving fairness, because it helps make sure the smaller flows still get processed even when the much bigger flows are pushing packets in like crazy ๐Ÿซถ
  • blg_len โ€” the length of the network backlog ๐Ÿ“š

With the TCP keyword, we get the tea on TCPv4 network traffic ๐Ÿต Note that TCPv4 stats depend on sadc's option -S SNMP being collected โš ๏ธ The following values are displayed (formal SNMP names are in square brackets):

  • active/s โ€” how many times per second TCP connections went straight from CLOSED to SYN-SENT [tcpActiveOpens] ๐Ÿš€
  • passive/s โ€” how many times per second TCP connections went straight from LISTEN to SYN-RCVD [tcpPassiveOpens] ๐Ÿ‘‚
  • iseg/s โ€” the total number of segments received per second, errors included [tcpInSegs] ๐Ÿ“ฅ This count includes segments received on connections that are currently established ๐Ÿค
  • oseg/s โ€” the total number of segments sent per second [tcpOutSegs] ๐Ÿ“ค This includes the ones on current connections but leaves out the ones that only contained retransmitted octets ๐Ÿ™…

With the ETCP keyword, we check the TCPv4 network errors ๐Ÿšจ Note that TCPv4 stats depend on sadc's option -S SNMP being collected โš ๏ธ The following values are displayed (formal SNMP names are in square brackets):

  • atmptf/s โ€” how many times per second TCP connections went to CLOSED from SYN-SENT or SYN-RCVD, plus how many times per second they went to LISTEN from SYN-RCVD [tcpAttemptFails] ๐Ÿ’”
  • estres/s โ€” how many times per second TCP connections went to CLOSED from ESTABLISHED or CLOSE-WAIT [tcpEstabResets] ๐Ÿซ 
  • retrseg/s โ€” the total number of segments retransmitted per second, meaning TCP segments sent that contained one or more previously transmitted octets [tcpRetransSegs] ๐Ÿ”
  • isegerr/s โ€” the total number of segments received in error per second, like bad TCP checksums [tcpInErrs] ๐Ÿ™…โ€โ™€๏ธ
  • orsts/s โ€” how many TCP segments per second were sent with the RST flag set [tcpOutRsts] ๐Ÿ›‘

With the UDP keyword, we look at UDPv4 network traffic ๐Ÿฆ‹ Note that UDPv4 stats depend on sadc's option -S SNMP being collected โš ๏ธ The following values are displayed (formal SNMP names are in square brackets):

  • idgm/s โ€” the total number of UDP datagrams delivered per second to UDP users [udpInDatagrams] ๐Ÿ“ฌ
  • odgm/s โ€” the total number of UDP datagrams sent per second from this entity [udpOutDatagrams] ๐Ÿ“ค
  • noport/s โ€” the total number of received UDP datagrams per second that had no application at the destination port [udpNoPorts] ๐Ÿš๏ธ
  • idgmerr/s โ€” the number of received UDP datagrams per second that couldn't be delivered for reasons other than having no application at the destination port [udpInErrors] ๐Ÿ˜ต

With the UDP6 keyword, we look at UDPv6 network traffic ๐ŸŒ Note that UDPv6 stats depend on sadc's option -S IPV6 being collected โš ๏ธ The following values are displayed (formal SNMP names are in square brackets):

  • idgm6/s โ€” the total number of UDP datagrams delivered per second to UDP users [udpInDatagrams] ๐Ÿ“ฌ
  • odgm6/s โ€” the total number of UDP datagrams sent per second from this entity [udpOutDatagrams] ๐Ÿ“ค
  • noport6/s โ€” the total number of received UDP datagrams per second that had no application at the destination port [udpNoPorts] ๐Ÿš๏ธ
  • idgmer6/s โ€” the number of received UDP datagrams per second that couldn't be delivered for reasons other than having no application at the destination port [udpInErrors] ๐Ÿ˜ต

The ALL keyword is basically the whole main character lineup ๐Ÿ‘‘ It's the same as typing every keyword above, so all the network activities get reported ๐ŸŒˆ

  • -o [ *filename* ] โ€” okay this one saves your readings to a file in binary form ๐Ÿ’พ Each reading gets its own separate record. The default filename is the current standard system activity daily data file ๐Ÿ“ If filename is a directory instead of a plain file, then it's treated as the directory where the standard system activity daily data files live ๐Ÿ—‚๏ธ Heads up, -o can't be used together with -f ๐Ÿ™… All the data available from the kernel gets saved to the file, because sar calls its data collector sadc with the option -S ALL ๐Ÿง  (See the sadc(8) manual page for more.)
  • -P { *cpu_list* | ALL } โ€” reports per-processor stats for the processor or processors you pick ๐Ÿ–ฅ๏ธ cpu_list is a list of comma-separated values or a range of values, like 0,2,4-7,12- ๐ŸŽ€ Processor 0 is the first processor, and processor all is the global average across all processors โœจ Using the ALL keyword reports stats for each individual processor, plus the global stats for all processors together ๐Ÿ’… Offline processors don't get displayed ๐Ÿ™ˆ
  • -p, --pretty โ€” makes reports easier for humans to read ๐Ÿง This one is especially useful when you're looking at network interface or block device stats ๐Ÿ’–
  • -q [ *keyword*[,...] | ALL ] โ€” reports system load and pressure-stall stats ๐Ÿ“ˆ

Possible keywords are CPU, IO, LOAD, MEM and PSI ๐ŸŒธ

With the CPU keyword, we get the CPU pressure stats ๐Ÿ˜ค The following values are displayed:

  • %scpu-10 โ€” the percentage of time that at least some runnable tasks were delayed because the CPU wasn't available to them, measured over the last 10 second window โฑ๏ธ

Okay so the pressure stats are where it gets spicy ๐ŸŒถ๏ธ Each % field tells you how often tasks were stuck, and the number at the end is the time window.

  • %scpu-60 โ€” ๐Ÿ”ฅ Percentage of the time that at least some runnable tasks were delayed because the CPU was unavailable to them, over the last 60 second window.
  • %scpu-300 โ€” ๐Ÿ”ฅ Same vibe, but over the last 300 second window.
  • %scpu โ€” ๐Ÿ”ฅ Same vibe again, but over the last time interval.

With the IO keyword, you get I/O pressure stats ๐Ÿ’พ Here's what shows up:

  • %sio-10 โ€” Percentage of the time that at least some tasks lost waiting for I/O, over the last 10 second window.
  • %sio-60 โ€” Percentage of the time that at least some tasks lost waiting for I/O, over the last 60 second window.
  • %sio-300 โ€” Percentage of the time that at least some tasks lost waiting for I/O, over the last 300 second window.
  • %sio โ€” Percentage of the time that at least some tasks lost waiting for I/O, over the last time interval.
  • %fio-10 โ€” Percentage of the time during which all non-idle tasks were stalled waiting for I/O, over the last 10 second window. ๐Ÿ’€
  • %fio-60 โ€” Percentage of the time during which all non-idle tasks were stalled waiting for I/O, over the last 60 second window. ๐Ÿ’€
  • %fio-300 โ€” Percentage of the time during which all non-idle tasks were stalled waiting for I/O, over the last 300 second window. ๐Ÿ’€
  • %fio โ€” Percentage of the time during which all non-idle tasks were stalled waiting for I/O, over the last time interval. ๐Ÿ’€

With the LOAD keyword, you get queue length and load averages ๐Ÿ“ˆ These are the values:

  • runq-sz โ€” Run queue length (number of tasks running or waiting for run time).
  • plist-sz โ€” Number of tasks in the task list.
  • ldavg-1 โ€” System load average for the last minute. The load average is calculated as the average number of runnable or running tasks (R state), and the number of tasks in uninterruptible sleep (D state) over the specified interval.
  • ldavg-5 โ€” System load average for the past 5 minutes.
  • ldavg-15 โ€” System load average for the past 15 minutes.
  • blocked โ€” Number of tasks currently blocked, waiting for I/O to complete.

With the MEM keyword, you get memory pressure stats ๐Ÿง  Here's the lineup:

  • %smem-10 โ€” Percentage of the time during which at least some tasks were waiting for memory resources, over the last 10 second window.
  • %smem-60 โ€” Percentage of the time during which at least some tasks were waiting for memory resources, over the last 60 second window.
  • %smem-300 โ€” Percentage of the time during which at least some tasks were waiting for memory resources, over the last 300 second window.
  • %smem โ€” Percentage of the time during which at least some tasks were waiting for memory resources, over the last time interval.
  • %fmem-10 โ€” Percentage of the time during which all non-idle tasks were stalled waiting for memory resources, over the last 10 second window. ๐Ÿ’€
  • %fmem-60 โ€” Percentage of the time during which all non-idle tasks were stalled waiting for memory resources, over the last 60 second window. ๐Ÿ’€
  • %fmem-300 โ€” Percentage of the time during which all non-idle tasks were stalled waiting for memory resources, over the last 300 second window. ๐Ÿ’€
  • %fmem โ€” Percentage of the time during which all non-idle tasks were stalled waiting for memory resources, over the last time interval. ๐Ÿ’€

The PSI keyword is basically CPU, IO and MEM all bundled together ๐ŸŽ€ so you get every pressure-stall stat at once.

The ALL keyword is the whole buffet ๐Ÿ“ It's the same as naming every keyword above, so every single stat gets reported.


  • -r [ ALL ] โ€” ๐Ÿง  Reports memory utilization stats. The ALL keyword means every memory field gets displayed. These are the values that may show up:
    • kbmemfree โ€” Amount of free memory available in kibibytes.
    • kbavail โ€” Estimate of how much memory in kibibytes is available for starting new applications, without swapping. The estimate takes into account that the system needs some page cache to function well, and that not all reclaimable slab will be reclaimable, due to items being in use. The impact of those factors will vary from system to system.
    • kbmemused โ€” Amount of used memory in kibibytes (calculated as total installed memory - kbmemfree - kbbuffers - kbcached - kbslab).
    • %memused โ€” Percentage of used memory.
    • kbbuffers โ€” Amount of memory used as buffers by the kernel in kibibytes.
    • kbcached โ€” Amount of memory used to cache data by the kernel in kibibytes.
    • kbcommit โ€” Amount of memory in kibibytes needed for current workload. This is an estimate of how much RAM/swap is needed to guarantee that there never is out of memory.
    • %commit โ€” Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap). This number may be greater than 100% because the kernel usually overcommits memory.
    • kbactive โ€” Amount of active memory in kibibytes (memory that has been used more recently and usually not reclaimed unless absolutely necessary).
    • kbinact โ€” Amount of inactive memory in kibibytes (memory which has been less recently used. It is more eligible to be reclaimed for other purposes).
    • kbdirty โ€” Amount of memory in kibibytes waiting to get written back to the disk.
    • kbanonpg โ€” Amount of non-file backed pages in kibibytes mapped into userspace page tables.
    • kbslab โ€” Amount of memory in kibibytes used by the kernel to cache data structures for its own use.
    • kbkstack โ€” Amount of memory in kibibytes used for kernel stack space.
    • kbpgtbl โ€” Amount of memory in kibibytes dedicated to the lowest level of page tables.
    • kbvmused โ€” Amount of memory in kibibytes of used virtual address space.
  • -S โ€” ๐Ÿ”„ Reports swap space utilization stats. These are the values displayed:
    • kbswpfree โ€” Amount of free swap space in kibibytes.
    • kbswpused โ€” Amount of used swap space in kibibytes.
    • %swpused โ€” Percentage of used swap space.
    • kbswpcad โ€” Amount of cached swap memory in kibibytes. This is memory that once was swapped out, is swapped back in but still also is in the swap area (if memory is needed it doesn't need to be swapped out again because it is already in the swap area. This saves I/O).
    • %swpcad โ€” Percentage of cached swap memory in relation to the amount of used swap space.
  • -s [ hh:mm[:ss] ] and -s [ seconds_since_the_epoch ] โ€” โฐ Sets the starting time of the data, so sar pulls records time-tagged at, or following, the time you give it. The default starting time is 08:00:00. Hours have to be in 24-hour format, or you can give the number of seconds since the epoch (a 10 digit number). This option only works when data is read from a file (option -f).
  • --sadc โ€” ๐Ÿ” Tells you which data collector sar calls. If the data collector is hiding somewhere in PATH, type "which sadc" to find out where it lives.
  • -t โ€” ๐ŸŒ When reading data from a daily data file, this makes sar show timestamps in the original local time of the data file creator. Without it, sar shows timestamps in your own local time.
  • -u [ ALL ] โ€” ๐Ÿ’ป Reports CPU utilization. The ALL keyword means every CPU field gets displayed. The report may show these fields:
    • %user โ€” Percentage of CPU utilization that occurred while executing at the user level (application). Note that this field includes time spent running virtual processors.
    • %usr โ€” Percentage of CPU utilization that occurred while executing at the user level (application). Note that this field does NOT include time spent running virtual processors.
    • %nice โ€” Percentage of CPU utilization that occurred while executing at the user level with nice priority.
    • %system โ€” Percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this field includes time spent servicing hardware and software interrupts.
    • %sys โ€” Percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this field does NOT include time spent servicing hardware or software interrupts.
    • %iowait โ€” Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request. ๐Ÿซถ

%steal ๐Ÿซฃ Percentage of time the virtual CPU or CPUs spent in involuntary wait while the hypervisor was busy servicing another virtual processor. Basically, someone else's VM was hogging the host.

  • %irq โšก Percentage of time the CPU or CPUs spent handling hardware interrupts.
  • %soft ๐ŸŒ€ Percentage of time the CPU or CPUs spent handling software interrupts.
  • %guest ๐Ÿซถ Percentage of time the CPU or CPUs spent running a virtual processor.
  • %gnice ๐Ÿ˜‡ Percentage of time the CPU or CPUs spent running a niced guest.
  • %idle ๐Ÿ˜ด Percentage of time the CPU or CPUs were just chilling, idle, and the system had no outstanding disk I/O request waiting on it.
  • -V ๐Ÿท๏ธ Prints the version number and then bounces.
  • -v ๐Ÿ—‚๏ธ Reports the status of the inode, file, and other kernel tables. You'll see these values:
    • dentunusd ๐Ÿงน Number of unused cache entries in the directory cache.
    • file-nr ๐Ÿ“ Number of file handles the system is using.
    • inode-nr ๐Ÿ”ข Number of inode handlers the system is using.
    • pty-nr ๐Ÿ–ฅ๏ธ Number of pseudo-terminals the system is using.
  • -W ๐Ÿ”„ Reports swapping stats. You'll get these values:
    • pswpin/s ๐Ÿ“ฅ Total number of swap pages the system brought in per second.
    • pswpout/s ๐Ÿ“ค Total number of swap pages the system brought out per second.
  • -w ๐Ÿงต Reports task creation and system switching activity. You'll get these values:
    • proc/s ๐ŸŒฑ Total number of tasks created per second.
    • cswch/s ๐Ÿ”€ Total number of context switches per second.
  • -x ๐Ÿ“ˆ Extended reports! On top of the averages, you also get the minimum and maximum values at the end of the report.
  • -y ๐Ÿ“Ÿ Reports TTY device activity. You'll get these values:
    • rcvin/s ๐Ÿ“จ Number of receive interrupts per second for the current serial line. The serial line number shows up in the TTY column.
    • xmtin/s ๐Ÿ“ค Number of transmit interrupts per second for the current serial line.
    • framerr/s ๐Ÿ’ฅ Number of frame errors per second for the current serial line.
    • prtyerr/s โš ๏ธ Number of parity errors per second for the current serial line.
    • brk/s ๐Ÿ’” Number of breaks per second for the current serial line.
    • ovrun/s ๐ŸŒŠ Number of overrun errors per second for the current serial line.
  • -z ๐Ÿซง Tells sar to skip the output for any devices that had zero activity during the sample period. Tidy!

๐ŸŒ ENVIRONMENT ๐ŸŒ

The sar command is super aware of the following environment variables. ๐Ÿ‘€

  • S_COLORS ๐ŸŽจ By default, stats show up in color when the output goes to a terminal. Use this variable to change that. The possible values are never, always, or auto (and auto is the same as the default). Also, no cap, the color a value shows up in (red, yellow, or whatever) doesn't mean something is wrong. The colors just mark different ranges of values. ๐Ÿ™…โ€โ™€๏ธ
  • S_COLORS_SGR ๐ŸŒˆ Lets you pick the colors and other attributes used to display stats on the terminal. The value is a colon-separated list of capabilities. It defaults to C=33;22:I=32;22:N=34;1:R=31;22:W=35;1:X=31;1:Z=34;22. The supported capabilities are:
    • C= ๐Ÿ’ฌ SGR (Select Graphic Rendition) substring for comments inserted in the binary daily data files.
    • I= ๐Ÿท๏ธ SGR substring for item names or values (like network interfaces, the CPU number, and so on).
    • N= ๐Ÿ”ฅ SGR substring for non-zero statistics values.
    • R= ๐Ÿ” SGR substring for restart messages.
    • W= (or M=) โš ๏ธ SGR substring for percentage values in the range from 75% to 90% (or from 10% to 25%, depending on what the metric means). It's also used for negative values in the range from -10 to -5.
    • X= (or H=) ๐Ÿšจ SGR substring for percentage values greater than or equal to 90% (or less than or equal to 10%, depending on what the metric means). It's also used for negative values less than or equal to -10.
    • Z= ๐Ÿซฅ SGR substring for zero values.
  • S_REPEAT_HEADER ๐Ÿ“‹ This contains the maximum number of lines after which sar has to show a header again when the output is not a terminal.
  • S_TIME_DEF_TIME ๐Ÿ•ฐ๏ธ If this variable exists and its value is UTC, then sar saves its data in UTC time (the data still displays in local time). sar also uses UTC instead of local time to figure out which daily data file in the /var/log/sysstat directory is the current one. This can be handy for servers with users spread across several timezones. ๐ŸŒ
  • S_TIME_FORMAT ๐Ÿ“… If this variable exists and its value is ISO, then the current locale gets ignored when printing the date in the report header. sar uses the ISO 8601 format (YYYY-MM-DD) instead. The timestamps follow ISO 8601 too. Very organized of it. ๐Ÿ’…

๐Ÿงช EXAMPLES ๐Ÿงช

sar -u 2 5

Reports CPU utilization every 2 seconds. 5 lines are displayed. โฑ๏ธ

sar -I --int=14 -o int14.file 2 10

Reports stats on IRQ 14 every 2 seconds. 10 lines are displayed. The data gets saved in a file called int14.file. ๐Ÿ’พ

sar -r -n DEV -f /var/log/sysstat/sa16

Displays the memory and network stats saved in the daily data file sa16. ๐Ÿง ๐ŸŒ

sar -A

Displays all the stats saved in the current daily data file. Everything, all at once! ๐Ÿซ 


๐Ÿ› BUGS ๐Ÿ›

๐Ÿšจ The /proc filesystem must be mounted for sar to work. No /proc, no sar. Period. ๐Ÿ™…โ€โ™€๏ธ

Not every statistic is necessarily available. It depends on the kernel version you're using. sar assumes you're running at least a 2.6 kernel. ๐Ÿง

๐Ÿค“ Here's a fun little gotcha: even though sar shows units that look like kilobytes (kB), megabytes (MB)..., it actually uses kibibytes (kiB), mebibytes (MiB)... A kibibyte is 1024 bytes, and a mebibyte is 1024 kibibytes. Math is serving. ๐Ÿงฎ

๐Ÿ“ FILES ๐Ÿ“

  • /var/log/sysstat/saDD ๐Ÿ—“๏ธ
  • /var/log/sysstat/saYYYYMMDD ๐Ÿ—“๏ธ These are the standard system activity daily data files, plus their default location. YYYY stands for the current year, MM for the current month, and DD for the current day. โœจ
  • /proc and /sys ๐Ÿงพ These contain various files with system statistics. The good stuff lives here! ๐Ÿ”

๐Ÿ‘ฉโ€๐Ÿ’ป AUTHOR ๐Ÿ‘ฉโ€๐Ÿ’ป

Sebastien Godard (sysstat <at> orange.fr) ๐Ÿซถ


๐Ÿ”— SEE ALSO ๐Ÿ”—

sadc(8), sa1(8), sa2(8), sadf(1), sysstat(5), pidstat(1), mpstat(1), iostat(1), vmstat(8) ๐Ÿ’–

https://github.com/sysstat/sysstat ๐Ÿ™ https://sysstat.github.io/ ๐ŸŒธ


Bestie, that's the whole sar page! ๐ŸŒˆ๐Ÿ’… You now know everything about system activity reporting, and honestly? Iconic. Stay curious, stay caffeinated, and go watch your CPU chill for a sec. ๐Ÿฆ‹โœจ

๐Ÿ“ฆ where this page comes from