COLLECTL(1)

📦
version 4.3.20.3 · 2 versions
, in fedora 43, fedora 44, nixos 26.05, nixpkgs unstable

💖 COLLECTL(1) 💖

🌸 NAME 🌸

collectl — bestie, she collects data that describes the current system status. periodt. 💅

✨ SYNOPSIS ✨

Okay so there are two modes, bestie. Record Mode and Playback Mode. Let's go.

Record Mode — read data from live system and write to file or display on terminal

collectl [-f file] [options]

Playback Mode — read data from one or more raw data files and display on terminal

collectl -p file1 [file2 ...] [options]

💅 OPTIONS 💅

🎀 Record Mode 🎀

In this mode data is taken from a live system and either displayed on the terminal or written to one or more files or a socket. iconic.

  • --align — okay so if the HiRes module is present, collectl sample monitoring will be aligned such that a sample will always be taken at the top of a minute. But no, that does NOT mean the first sample will occur then. This means all instances of collectl running on any systems which have their clocks synchronized will all take samples at the same time. Furthermore, if you're doing process monitoring, those samples will also be taken at the top of the minute and so can delay the start of sampling up to 2 full process monitoring intervals. slay. ✨
  • --all — Collect summary data for ALL subsystems except slabs, since slab monitoring requires a different monitoring interval. This also means you won't get any detail data which also includes processes and environmementals. You can use this switch anywhere -s can be used but not both together. If the system supports lustre and/or interconnect monitoring those statistics will be provided but the warnings produced when they are not available you try to select them with -s will not be displayed. 💖
  • --ALL — This is actually a superset of --all by adding detail statistics as well with the exception of TCP details when displaying to a terminal since those are only available with -P or -f. iykyk. 😭
  • -A, --address address[:port[:timeout]] | server[:port] — In the first form, one specifies an address, optional port and timeout (the first colon is required to specify timeout for default port). All data is then written to that socket prefaced with the current host name at the named address and port until the socket is closed, at which time collectl will exit. In the second form one enters the text server and optional port. In this form, collectl runs as a server, waiting for a connection and once established writes data on that socket. The key difference here is if the client exists collectl keeps running and will again look for a new connection, allowing it to survive client restarts or crashes. The default port is set at 2655 but can be changed — see collectl.conf. In both forms, one can additionally request local data logging by specifying a combination of -P and -f. See man collectl-logging for more details. 💅
  • --comment string — Add the specified string to the end of the headers in the data files. If any embedded spaces be sure to quote it. This can be very useful when doing characterizations or benchmarking and you're frequently changing system/application parameters and restarting collectl between tests. obsessed. 🎀
  • -C, --config filename — Name/location of the collectl configuration file. If not specified, collectl searches for collectl.conf first in /etc (the default), then in the same directory the collectl executable is in, and finally the current working directory. fr fr. 🦋
  • -c, --count Samples — The number of samples to record. This is one way of 3 ways of describing how long collectl should run (see -r and -R). Note that these 3 switches are mutually exclusive. periodt. 💀
  • -D, --daemon — Run collectl as a daemon, primarily used when starting as a service. One caveat about this mode is you can only run one copy. not me. 🙈
  • --export file[,options] — This requests that collectl does not print anything on the terminal (or send it to a socket) using the standard brief/verbose/plot formats. Instead it executes a perl "require" on the named file, using an extension of ph if not specified. It first looks in the current directory and if not there the directory the executable is in. It then calls the function "file"Init(options) towards the beginning of collectl and again as simply "file"(@options) to generate the exported formatted output. See the online documentation on Exporting Custom Output and Logging for more details. main character energy. ✨
  • -f, --filename Filename — This is the name of a file to write the output to. For details on how the output files are named, see the File Naming section of the documentation on collectl.sourceforge.net OR /usr/share/doc/collectl/FileNaming.html. slay. 💖
  • -F, --flush seconds — Flush output buffers after this number of seconds. This is equivalent to issuing kill -s USR1 at the same frequency (but a lot easier!). If 0, a flush will occur every data collection interval. lowkey genius. 🌸
  • --grep pattern — The main purpose of this switch is for those users who have discovered there is some data in the raw files that never appears in any display and have taken to displaying it themselves with grep. Unfortunately this method does not include timestamps and so makes it difficult to interpret the results. Even if you include the timestamp from the file it is in UTC and so needs to be translated to be of any real value. This switch does just that and then some. Specifically, it allows you to playback a file and instead of processing it normally it simply searches for any entries that match the perl pattern and reports those lines prefaced with time stamps. You can optionally change the time format with the usual -o options and can even select the timeframe with --from and --thru. iconic. 😭
  • --home — Always start the display for the current interval at the top of the screen also known as the home position (non-plot format only). This generates a real-time, continously refreshing display when the data fits on a single screen. vibe check passed. 🎀
  • --import file1[,options][:file2[,options]...] — This loads the named files and executes callbacks to them, which is the API mechanism for importing additional metrics into collectl. See the webpage on the API for further detail. Since these files also include instructions for how to report the output in all the various forms, you will also need to include --import during playback. Finally, since the default is to seamlessly include imported data with everything else collectl reports, if you ONLY want to display imported data you much explicitly deselect all other subsystems either by including -s- (note the trailing minus sign) followed by all the subsystems were recorded OR simply say -s-all. iykyk. 💅
  • -i, --interval interval[:interval2[:interval3]] — This is the sampling interval in seconds. The default is 10 seconds when run as a daemon and 1 second otherwise. The process subsystem and slabs (-sY and -sZ) are sampled at the lower rate of interval2. Environmentals (-sE), which only apply to a subset of hardware, are sampled at interval3. Both interval2 and interval3, if specified, must be an even multiple of interval1. The daemon default is -i10:60:300 and all other modes are -i1:60:300. To sample only processes once every 10 seconds use -i:10. slay. 💖
  • --nohup — Whenever collectl finishes a data collection interval, it checks to see if the starting parent has exited. This is to prevent the case in which someone might start a copy of collectl and then the process dies and collectl keeps running. If that is the behavior someone actually intends, they should start collectl with --nohup. NOTE — when running as a daemon, --nohup is implied. fr fr. 🙈
  • --quiet — Whenever collectl wants to tell the user something, it assigns a category to it such as Informational, Warning, Error or Fatal. When run with -m, all messages are displayed for the user and if logging data to a file with -f, these messages are also sent to a log file which is in the data collection directory and has an extenion of "log". However, if -m is not specified Informational messages (such as collectl starting or stopping) are not reported on the terminal but the other 3 are. Sometimes the warnings can be annoying and one can suppress these with --quiet though they will still be written to the message log in -f. You cannot suppress Error or Fatal errors. periodt. 💀
  • -r, --rolllogs time[[,days[:months]][,minutes]] — When selected, collectl runs indefinately (or at least until the system reboots). The maximum number of raw and/or plot files that will be retained (older ones are automatically deleted) is controlled by the days field, the default is 7. When -m is also specified to direct collectl to write messages to a log file in the logging directory, the number of months to retain those logs is controlled by the months field and its default is 12. The increment field which is also optional (but is position dependent) specifies the duration of an individual collection file in minutes the default of which is 1440 or 1 day. obsessed. 🌈
  • --rawdskfilt — This switch overrides the DiskFilter setting in collectl.conf and explicitly defines a perl regx expression against which records from /prod/diskstats are selected for processing. When there are a lot of disks to process, this can be a handy way to reduce the amount of data collected and actually improve performance since there are less patterns to match each input record against. Just remember that unlike --dskfilt which only filters during display, records filtered with this switch are never even recorded and so lost forever. You can optionally specify your filter with a leading plus-sign which tells collectl to just add your filter to the default specification. Care should be taken here as longer filters will slightly increase overhead and with a lot of disks and/or shorter monitoring intervals can add up. no cap. 😭

As a side benefit of this switch, if you really want to look at partition level stats you can do so by leaving off the trailing space in the default pattern.

Also bestie, be SO careful picking the correct pattern because it's lowkey easy to get it wrong and then you may end up collecting the WRONG data! 😭 To verify you are collecting what you think you are, make a test run using -d4 to see the raw data being recorded in real-time. ✨

  • --rawdskignore — This is the opposite of the rawdskfilt switch. When specified, any disks listed are completely ignored and will not appear in the raw file. Typically this switch is useful when you're only interested in recording a subset of disk statistics. 🚫💽
  • --rawnetfilt — This works just like --rawdskfilt except it applies to networks. Unlike disk filtering which has an explicit default pattern, the default for network filtering is to simply record all network data from /proc/net/dev. 🌐 The -d4 switch also works here, as well as everywhere, to see the raw data as it is being collected.
  • --rawnetignore — This is the opposite of the rawnetfilt switch and works just like the rawdskignore switch. When specified, any networks listed are ignored and will not appear in the raw file. Typically this switch is useful when you're only interested in recording a subset of network statistics. 🚫🌐
  • --rawtoo — Only available in conjunction with -P, this switch causes the creation/logging of raw data in addition to plottable data. While this may seem excessive, keep in mind that unlike plottable data, raw data can be played back with different switches potentially providing more details. The overhead to write out this additional data is minimal, the only real cost being that of extra disk space. 💾✨
  • -R, --runas uid[:gid] — This switch only works when running in daemon mode and so must be specified in the DaemonCommands line. Its presence will cause collectl to write the collectl.pid file into the same directory as its other output files as specified by -f, since /var/run does not normally grant non-privileged users write access. Furthermore, the ownership of that directory must match the specified ownership since collectl needs to write ALL its files to that directory and can no longer assume global permissions when run as root. 🔐

This WILL also require manually modifying /etc/init.d/collectl to change the PIDFILE variable to point to the same directory which the -f switch in the DaemonCommands line of collectl.conf points to.

As a final note of caution, since this mechanism changes where collectl reads/writes its pid file, once you start using --runas, all calls to run collectl as a daemon must use it or it may be confused and exhibit unpredictable behavior. 😱

  • -R, --runtime duration — Specify the duration of data collection where the duration is a number followed by one of wdhms, indicating how many weeks, days, hours, minutes or seconds the collection is to be taken for. ⏳
  • --sep separator — Specify the plot format separator — default is a space. If this is a numeric field it is interpreted as the decimal value of the associated ASCII character code. Otherwise it is interpreted as the character itself. In other words, "--sep :" sets the separator character to a colon and "--sep 9" sets it to a horizontal tab. "--sep 58" would also set it to a colon. 🎀
  • --tworaw — The switches -G and --group have been replaced by --rawtoo, which is more rescriptive of its function. When specified, it tells collectl to treat process and slab data as an entirely separate group of raw files, named with the extension "rawp". These separate files can be played back and processed just like any other collectl raw files and in fact one can even play back both at the same time if that is what is desired. The only real purpose of this switch is that on some systems with many processes, it is possible to generate huge raw files (some have been observed to be >250MB!) and while collectl will happily play back/process these files it can take a long time. By using the --tworaw switch one still gets a huge rawp file, but the normal raw file is a much more manageable size and as a result will be faster to process then when all data is combined into the same file. 🐘💾

🎧 Playback Mode 🎧

In this mode, data is read from one or more data files that were generated in Record Mode.

  • --export Filename — When playing back a file, use this switch to create an identical raw file differing only in the timeframe being covered, so naturally one must also include --from, --thru or both. Further, since the resultant file will contain the exact same raw data you cannot select a subset using -s. This switch is actually intended for a support function for situations where someone is having problems playing back a file and a subset of the original raw file that covers the problem time has been requested, hopefully allowing a significantly file to be posted or emailed. 📤
  • --extract filename — If specified, rather than actually play back the file specified with -p, ALL raw data between the date ranges is selected and a subset of that raw file created. The rules for how to interpret the filename are the same as used for -f. 📂
  • -f, --filename filename — If specified, this is the name of a file or directory to write the output to (rather than the terminal). See the description for details on the format of this field. This requires the -P flag as well. 📝
  • --from time range — Play back data starting with this time, which may optionally include the ending time as well, which is of the format of [date:]time[-[date:]time]. The leading 0 of the hour is optional and if the seconds field is not specified is assumed to be 0. If no dates specified the time(s) apply to each file specified by -P. Otherwise the time(s) only apply to the first/last dates and any files between those dates will have all their data reported. ⏰
  • --full — Full mode is actually a superset of --verbose and if selected will force --verbose. It will also force the RECORD separator to be printed for every interval even if only a single subsystem was requested and to include the actual subsystems that follow following the utc timestamp as a parsing aid for those who may wish to parse the text output rather than the plot data. 🧐
  • --offsettime seconds — This field originally was used before collectl reported the timezone in the file headers and allowed one to compensate. Since then it is rarely needed except in two possible cases, one in which data on two systems is to be compared and they weren't synchronized with ntp. This allows all the times to be reported as shifted by some number of seconds. The other case (and this is very rare) is when a clock had changed in the middle of a sample and will not be converted correctly. When this happens one may have to play back the samples in pieces and manually set the time offset. ⏱️
  • --passwd filename — When reporting usernames associated with a UID, use this file for the mapping. This is particularly important on systems running NIS where there are no user names in /etc/passwd. 👤
  • -p, --playback Filename — Read data from the specified playback file(s), noting that one can use wildcards in the filename if quoted (if playing back multiple files to the terminal you probably want to include -m to see the filenames as they are processed). The filename must either end in raw or raw.gz. As an added feature, since people sometimes automate the running of this option and don't want to hard code a date, you can specify the string YESTERDAY or TODAY and they will be replaced in the filename string by the appropriate date. 🎬
  • --pname name — By default, collectl uses the file /var/run/collectl.pid to indicate the pid of the running instance of collectl and prevent multiple copies from being run. If you DO want to run a second copy, this switch will cause collectl to change its process name to collectl-name and use that name as the associated pid file as well. 🏷️
  • --procanalyze — When specified and there is process data in the raw file, a summary file will be generated with one entry unique process containing such things as the total cpu consumed for both user and system, min/max utilization of various memory types, total page faults and several others. 📊
  • --slabanalyze — When specified and there is slab data in the raw file, a summary file will be generated with one entry unique slab containing data on physical memory usage by that slab. 🧱
  • --thru time — Time thru which to play back a raw file. See --from for more. ⏳

🫶 Common Switches - both record and playback modes 🫶

  • -d, --debug debug — Control the level of debugging information, not typically used. For details see the source code. 🐛
  • -h, --help, -x, --helpext, -X, --helpall — Display standard, extended help message (which doesn't include the optional displays such as --showoptions, --showsubsys, --showsubopts, --showtopopts) or everything. 🙋‍♀️
  • --hr, --headerrepeat num — Sets the number of intervals to display data for before repeating the header. A value -1 will prevent any headers from being displayed and a value of 0 will cause only a single header to be displayed and never repeated. 📋
  • --iosize — In brief mode, include iosize with disk, infiniband and network data. 📏
  • -l, --limits limit — Override one or more default exception limits. If more than one limit they must be separated by hyphens. Current values are:
    • SVC:value — Report partition activity with Service times >= 30 msec
    • IOS:value — Report device activity with 10 or more reads or writes per second
  • LusKBS:value — flags any client or OSS activity that goes over limit. Only does anything for Client Summary or OSS Detail reporting. [default=100000]
  • LusReints:value — flags MDS activity where Reint is bigger than limit. Only for MDS Summary reporting. [default=1000]
  • AND — the device only gets reported once both the IOS and SCV limits have been hit. This is the default value, and honestly it's only listed here so the list feels complete.
  • OR — the device gets reported if either the IOS or SVC thresholds are reached.

💅 -L, --lustsvcs [c|m|o][:seconds]

This one narrows down which servics Lustre actually checks, plus how often it checks them. Want the full tea? Go read the man page collectl-lustre.

🫶 -m, --messages

Drops status into a monthly log file sitting in the same directory as your output file — so yeah, you gotta also pass -f or this does nothing. The file gets named collectl-yyyymm.log and it keeps track of all the assorted messages that might pop off during every single run of collectl. Very "receipts" of it.

✨ -N, --nice

Bumps your priority to a nicer one of 10. Be kind to your scheduler bestie.

👑 -o, --options Options

These control how output is displayed or written to a plot file. They do not change how data gets selected for recording. Most of these switches work in both record mode and playback mode. Not sure which? Just try it, live a little.

  • 1 — data in plotting format should use 1 decimal point of precision, as appropriate.
  • 2 — data in plotting format should use 2 decimal points of precision, as appropriate.
  • a — always append data to an existing plot file. By default, if a plot file already exists, the playback file gets skipped, which is a way of assuring it's tied to a single recorded file. This switch overrides that whole mechanism so multiple recorded files can be processed and written into one single plot file.
  • c — always open newly named plot fies in create mode, overwriting any old ones that may already exists. If you process multiple files for the same day in append mode multiple times, the same data gets appended to the same file mulitple times. This assures a fresh new file gets created at the start of processing.
  • d — for use with terminal output and brief mode. Preceed each line with a date/time stamp, with the date in mm/dd format. This option can also be applied to plot formatit, which makes the date portion display in this format too, as opposed to the D format.
  • D — for use with terminal output and brief mode. Preceed each line with a date/time stamp, the date being in yyyymmdd format.
  • g — for use with terminal output and brief mode. When displaying values of 1G or greater there's limited precision for 1 digit values. This option gives you a way to display extra digits for more granularity by substituting a "g" for the decimal point instead of the trailing "G".
  • G — for use with terminal output and brief mode. Similar to "g" but it preserves the trailing "G" by sacrificing a digit of granularity.
  • m — whenever times are reported in plot format, in the normal terminal reporting format at the bginning of each interval, or when when one of the time reporting options (d, D, T or U) is selected, append the milliseconds to the time.
  • n — where appropriate, data like disk KBs or transfers gets normalized to units per second by taking the change in a counter and dividing it by the number of seconds in that interval. For CPUs, utilization (calculated in jiffies) is normalized as a percentage of the interval.

Normalization can be disabled with this option, and the result is that reported values are not divided by the duration of the interval. This is particulary useful for reporting values that are < 1/2 the sampling, which would otherwise be rounded to 0.

  • T — for use with terminal output and brief mode, preceeds each line with a time stamp.
  • u — create plot files with unique names by including the starting time of a colletion in the name. This forces multiple collections taken the same day to be written out to multiple files.
  • -U or --utc — in plot format only, report timestamps in Coordinated Universal time, which is more commonly know as UTC.
  • x — report only exception records for selected subsystems. Exception reporting also requires --verbose. Currently this only applies to disk detail and Lustre server information, so you've gotta select at least -s D, l or L for this to apply. If you're writing to a detail file, this data goes into a separate file with the extension X appended to the regular detail file name.
  • X — report both exceptions and all details for selected subsystems, for -s D, l or L only.
  • z — if the compression library has been installed, all output files get compressed by default. This switch tells collectl not to compress any plottable files. And if collectl tries to compress but can't because the library isn't installed, it'll generate a warning — which you can suppress with this switch.

🔥 -P, --plot

Generates output in plot format. That's space separated data made of a header (prefaced with a # so an analysis program can spot it easy, and also so programs like gnuplot treat it as a comment, since they honor that convention). When it's written to disk — which is the typical way this option gets used — summary data elements go to the tab file and the detail elements go to one or more files, one per detail subsystem. If -f isn't specified, all output gets sent to the terminal. Output is always one line per sampling interval. Periodt.

🌸 --stats

This switch makes brief data get reported as both totals and averages after processing one or more files for the same day, or in playback mode.

🦋 --statopts option(s)

Controls the way brief stats get reported. The default is to report the totals once, at the end of a day's worth of raw files, if there's more than one.

  • a — include averages along with totals
  • i — include the interval data itself, which is the equivalent of -oA
  • s — print summary stats at the end of each file processed, even if there's more than one per day

💜 -s, --subsys subsystem

This field controls which subsystem data gets collected or played back. The default for collecting data is "cdn" — that's CPU, Disk and Network summary data — and the default for playback is everthing that was collected.

The rules for displaying results change depending on the type of data you picked. If you write data for CPUs and DISKs to a raw file and play it back with -sc, you'll only see CPU data. If you play it back with -scm you'll still only see CPU data, since memory data was never collected. But when it's used with -P, collectl will always honor the subsystems you specified with this switch, so in that previous example you'll see CPU data plus memory data that's all 0s. Wanna see the current set of default subsystems — which is a subset of this full list — use -h.

You can also use + or - to add or subtract subsystems to/from the default values. For example, -s-cdn+N removes cpu, disk and network monitoring from the defaults while adding network detail.

Refer to the data definitions on the sourceforge website OR in /usr/share/collectl/doc/collectl-xxx to see complete descriptions of the data returned.

🌈 SUMMARY SUBSYSTEMS 🌈

  • b — buddy info (memory fragmentation)
  • c — CPU
  • d — Disk
  • f — NFS V3 Data
  • i — Inode and File System
  • j — Interrupts
  • l — Lustre
  • m — Memory
  • n — Networks
  • s — Sockets
  • t — TCP
  • x — Interconnect
  • y — Slabs (system object caches)

💖 DETAIL SUBSYSTEMS 💖

This is the set of detail data that, in most cases, the corresponding summary data gets derived from. There are currently 2 types that don't have corresponding summary data, and those are "Environmental" and "Process". So if you've got 3 disks and you choose -sd, you'll only see a single total taken across all 3 disks. If you choose -sD, individual disk totals get reported but no totals. Choose -sdD and you get both. Bestie behavior.

  • C — CPU
  • D — Disk
  • E — Environmental data (fan, power, temp), via ipmitool
  • F — NFS Data
  • J — Interrupts
  • L — Lustre OST detail OR client Filesystem detail
  • M — Memory node data, which is also known as numa data
  • N — Networks
  • T — 65 TCP counters only available in plot format
  • X — Interconnect
  • Y — Slabs (system object caches)
  • Z — Processes

⭐ --showheader

In collectl mode this command makes the header that's normally written to a data file get displayed on the terminal, and then collectl exists. Handy little way to get a brief overview of your system configuration. Lowkey iconic.

🎀 --showoptions

Shows only the portion of the help text that desribes the -o and --options switches, so you can save yourself the time of wading through the entire help screen. You're welcome.

  • --showcolheaders — prints the first set of headers collectl would print, then dips out and exits. Ngl it lowkey doesn't make sense for multi-section output, like when there are several sets of verbose or detail data. Also bestie, heads up: since it needs one monitoring interval to build up some headers (and some of those are dynamic), it also forces the interval to 0.
  • --showsubopts — lists all the subsystem specific options. 🫶
  • --showtopopts — shows all the different values for the --top type field, which is the field(s) you sort the data by.
  • --showrootslabs — this one only works on systems using the new slab allocator, and it lists the root name (those entries in /sys/slab which are not soft links) along with all its alias names. If a name doesn't have an alias, it will not appear in this report. 💀
  • --showslabaliases — also only works on systems using the new slab allocator. Like --showrootslabs, it names a slab and all its aliases, but rather than showing the root slab name it shows one of the aliases so the name is more meaningful. If there are any slabs that only have a single alias (or no alias at all), they will not be included in this report.
  • --showsubopts — similar to --showoptions, this command summaries just the parameters associated with -O and --subopts.
  • --showsubsys — yet another way to summare a portion of the help text, this one only shows valid subsystems.
  • --top [type][,num[,v]] — include the top num consumers by resource for this interval. The default number is the height of the window if it can be determined, otherwise 24, and the default resource is the total cpu time, which is taken as the sum of SysT and UsrT. Peep --showtopopts for a list of other types of data you can sort on.

This switch can also be used with -s, in which case a portion of the window is reserved at the top to fill in the subsystem data, which is currently in verbose mode, though a brief format is contemplated for some time in the future.

In interactive mode, and if not specified, the process monitoring interval will be set to that for other subsystems. The screen will be cleared for each interval, giving a display similar to the top utility. In playback mode the screen will NOT be cleared. And you cannot use this switch in "record" mode. periodt.

Finally, if v is specified as the 3rd parameter, the output scrolls vertically (like playback mode) rather than clearing the screen between intervals.

  • --umask mask — sets collectl's umask to control output file permissions. Only root can set the umask. Run man umask for the details.
  • --utime mask — write periodic micro-timestamps into the raw file at different points in time, for fine grained measurements of operation times.
    • 1 - write timestamps when entering major sections
    • 2 - write timestamps for all /proc accesses except for process data
    • 4 - write timestamps for /proc data for all processes including threads
  • -v — show version, plus whether or not the Compression and/or HiResTime modules have been installed, and exit. ✨
  • -V — show default parmeter and control settings, all of which can be changed in /etc/collectl.conf.
  • --verbose — display output in verbose mode. This often displays more data than in the default mode. When displaying detail data, verbose mode is forced. Furthermore, if summary data for a single subsystem is to be displayed in verbose mode, the headers are only repeated occasionally, whereas if multiple subsystems are involved each one needs its own header.
  • -w — disply data in wide mode. When displaying data on the terminal, some data is formatted followed by a K, M or G as appropriate. Selecting this switch causes the full field to be displayed. Note that there is no attempt to align data with the column headings in this mode.

🧩 SUBSYSTEM OPTIONS 🧩

The following options are subsystem specific and typically filter data for collection and/or display as well as affect the output format:

  • --cpufilt[^]perl-regx[,perl-regx...] — works the same as dskfilt and netfilt, and lets you select a subset of CPUs. These filters are also honored by interrupt reporting as well. 💅
  • --cpuopts
    • z - only applies to cpu details, do not report any CPUs with no load. In other words all entries are zero except for IDLE.
  • --dskfilt [^]perl-regx[,perl-regx...] — NOTE - this does NOT effect data collection, and ALL disk data will always be collected, unless --rawdskfilt is specified too. However, only data for disk names that match the pattern(s) will be included in the summary totals and displayed when details are requested. Alternatively, if you preface the first expression with a caret, all names that match all strings will be excluded from the summary totals and detail displays rather than included. If you don't know perl, a partial string will usually work too.

Just remember, this only applies to collected data, so if for example you specify a parition, such as sda1, you'll never see the data since it was filtered out at the time of data collection. To see those stats you would need to say --rawdskfilt sda1.

  • --dskopts
    • f - report some columns as fractions for more precision on detail output
    • i - display the i/o sizes in brief mode just like with --iosize
    • o - exclude unused disks from new file headers and plot data
    • z - only applies to disk details, do not report any lines with values of all zeros.
  • --dskremap aaa:bbb,ccc:ddd... — this will cause disk names matching the perl pattern aaa to be replaced with the string bbb. In some cases you may simply want to remove the entire string, in which case the second string should be left empty. If you want to remove a string container a /, be sure to escape it with a backslash.
  • --envopts Environmental Options — the default is to display ALL data, but the following will cause a subset to be displayed:
    • f - display fan data
    • p - display current (power) data
    • t - display temperature data
    • C - convert temperature to Celcius if in Farenheit
    • F - convert temperature to Farenheit if in Celcius
    • M - display each type of data on separate line
    • T - display data truncated to whole integers (some implemenations displayed them with fractional components)
    • 9 - any number, will tell ipmitool to read on this device number
  • --envfilt regx — if specified, this regx is evaluated against each line of data returned by ipmitool and only those that match are retained. All other data is lost. 😭
  • --envremap perl-regx,... — if specified as a comma separated list of perl regular substitution expressions without the =~s portion, each expression is applied to each environmental field name, thereby allowing you to rename the column headers. This can be most useful when running on heterogeneuos systems and you want consistent column names.
  • --intfilt [^]perl-regx[,perl-regx...] — NOTE - this does NOT effect data collection, ALL interrupt data will always be collected. However, only data for interrupts that match the pattern(s) will be included in the summary totals and displayed when details are requested. Alternatively, if you preface the first expression with a caret, all names that match all strings will be excluded from the summary totals and detail displays rather than included. If you don't know perl, a partial string will usually work too.

NOTE - these expressions are applied to the entire line one sees in /proc/interrupts, including the interrupt number, name and even counters, so if you do want to include an interrupt number in the pattern be sure to include the trailing colon as well. 🔥

  • --lustopts Lustre Options
    • B - For clients and servers, show buffer stats
    • D - For MDSs and OSTs AND running earlier versions of HPSFS, collect disk block iostats
    • M - For clients, collect metadata
    • O - For OSTs, show detail level stats
    • R - For client, collect readahead stats
  • --memopts Memory Options
    • R - show memory values (including swap space) as rates of change as opposed to absolute values. One can also show absolute changes between intervals by including -on.
  • --netfilt [^]perl-regx[,perl-regx...] — NOTE - this does NOT effect data collection and ALL network data will always be collected, unless --rawnetfilt is specified too. Also note that by default only eth, ib, em and p1p networks when present are included in the summary. When this switch is specified, only data for network names that match the pattern(s) will be included in the summary and displayed when details are requested. This switch therefore also gives you the ability to add other, possibly new, network devices to the summary totals. 🦋

Alternatively, if you preface the first expression with a caret, all names that match all strings will be excluded from the summary totals and detail displays rather than included. If you don't know perl, a partial string will usually work too.

  • --netopts
    • e - include network error counts in brief and explicit error types elsewhere
    • E - only include lines with network errors in them
    • i - include i/o sizes in brief mode
    • o - exclude unused networks from new file headers and plot data
    • w - set width of network device name
  • --nfsfilt NFS Filters — specify one or more comma separated filters as a C/S followed by an nfs version number and only those will have data reported on. For example, C2 says to report data on V2 Clients. As a data collection performance optimization, if one or more client filters are specified, data will actually be collected for all clients as is also done for servers.
  • --nfsopts NFS Options q.RS
    • z - only display detail lines which have data
  • --procfilt filters — bestie, this one is the ✨process filter✨ and it decides which processes even get picked up for collection/display. Using it lowkey slaps because it significantly reduces the load of process data collection — collectl builds a blacklist of all the existing processes that don't pass your filter, so they're permanently excluded from all future processing. Bye forever 💅

The format is one character type followed by a match string. You can stack multiple filters if you separate them with commas.

  • c — a substring of the command being executed, read straight out of /proc/pid/stat. Heads up though, this can literally be a perl expression, so if you want a command that ends in a particular string you just append a $ to the end of the string. Without it, it'll match anything containing that string.
  • C — any command that starts with the specified string
  • f — the full path of the command, including arguments, as read from /proc/pid/cmdline. Like the c modifier, this too can be a perl expression.
  • p — pid
  • P — parent pid
  • u — any process owned by this user's UID, or in the range specified by uxxx-yyy
  • U — any process owned by this username
⚠️ caution bestie: the process names collectl tries to match with c and C come from the second field in /proc/pid/stat, which may not necessarily be what you think! eg the name for X emacs is actually emacs-x. It's giving chaos fr fr 😭
  • --procopts options — these control the way data is displayed, and they can also improve data collection performance. Iconic multitasking 🫶
    • c — include CPU time of children who have exited (same as ps -S)
    • f — use cumulative totals for page faults in process data instead of rates
    • i — show process I/O counters in display instead of the default format
    • I — disable collection of I/O counters, see note below
    • k — remove known shells from process names, making it possible to see the actual command
    • m — show breakdown of memory utilization instead of the default format
    • p — never look for new pids or threads during data collection
    • r — show root command name only (no directory) for a narrower display. Note that this is applied AFTER k, so if arg1 becomes the new command it will be truncated now, which is very handy when running in a virtual python environment
    • R — show ALL process priorities ('RT' currently displayed if realtime)
    • s — show process start time in hh:mm:ss format
    • S — show process start time in mmmdd-hh:mm:ss format
    • t — include ALL process threads (increases collection overhead)
    • u — report username as 12 chars instead of 8, noting uxx will cause column width to be xx but cannot be less than 8
    • w — widen display by including whole argument string, with optional max width
    • x — include extended process attributes (currently only for context switches)
    • z — exclude any processes with 0 in the sort field (in --top mode)

Process data is the most expensive type of data collected, costing as much as 3 times the CPU load as all other types of data combined. Collecting thread data makes it even more expensive. You can significantly reduce this load by over 25 percent by disabling the collection of I/O stats. But keep in mind that even if you don't try to optimize process data collection, the overall system load by collectl can still be on the order of about 0.2% when running as a daemon with default collection rates. See the online documentation on measuring performance for more info 📚

A security hole was identified that allowed non-privileged users to read /proc/pid/io and guess password lengths, and now many distros restrict access to the owner or root. As a result, non-privileged users will see all 0 I/O counts for processes that are not theirs when specifying --procopt i. So like, don't be surprised 🙈

  • --slabfilt Slab Filters — you can specify a list of slab names separated by commas, and only those slabs whose names start with those strings will be listed or summarized.
  • --slabopts Slab Options
    • s — exclude any slabs with an allocation of 0
    • S — only show those slabs whose allocations changed since last display
  • --tcpfilt — these filters actually control both what gets collected and what gets displayed. If you select non-collected filters, 0s will be reported. There is one special case and that is if you include T (tcp extended stats) in the filter string, there are no brief ones and therefore --verbose will be forced 👑
    • i — ip stats
    • t — tcp stats
    • u — udp stats
    • c — icmp stats
    • I — ip extended stats
    • T — tcp excented stats
  • --xopts
    • i — include i/o sizes in brief mode

💖 DESCRIPTION 💖

The collectl utility is a system monitoring tool that records or displays specific operating system data for one or more sets of subsystems. Any set of the subsystems — like CPU, Disks, Memory or Sockets — can be included in or excluded from data collection. Data can either be displayed back to the terminal, or stored in either a compressed or uncompressed data file. The data files themselves can either be in raw format (essentially a direct copy from the associated /proc structures) or in a space separated plottable format such that it can be easily plotted using tools such as gnuplot or excel. Data files can be read and manipulated from the command line, or through use of command scripts. It really does do it all bestie ✨

Upon startup, collectl.conf is read, which sets a number of default parameters and switch values. Collectl searches for this file first in /etc, then in the directory the collectl executable lives in (typically /usr/sbin) and finally the current directory. These locations can be overridden with the -C switch. Unless you're doing something really special, this file need never be touched — the only exception perhaps being when choosing to run collectl as a service and you wish to change its default behavior, which is set by the DaemonCommand entry.

🚧 RESTRICTIONS/PROBLEMS 🚧

Thread reporting currently only works with 2.6 kernels. Not me being picky, that's just the vibe 🌸

The pagesize has been hardcoded for perl 5.6 systems to 4096 for IA32 and 16384 for all others. If you are running 5.6 on a system with a different pagesize you will see incorrect SLAB allocation sizes and will need to scale the numbers you're seeing accordingly.

I have recently discovered there is a bug in /proc in that an extra line is occasionally read with the end of the previous buffer! When this occurs a message is written (if -m enabled) and always written to the terminal. Since this happens with a higher frequency with process data I silently ignore those as the output can get pretty noisey. If for any reason this is a problem, be sure to let me know 💀

Since collectl has no control over the frequency at which data gets written to /proc, one can get anomalous statistics as collectl is only reporting a snapshot of what is being recorded. For more information see http://collectl.sourceforge.net/TheMath.html.

At least one network card occasionally generates erroneous network stats, and to try to keep the data rational, collectl tries to detect this and when it does generates a message that bogus data has been detected. We're keeping it honest 😤

🔍 FILES, EXAMPLES AND MORE INFORMATION 🔍

http://collectl.sourceforge.net OR /opt/hp/collectl/docs

🎀 ACKNOWLEDGEMENTS 🎀

I would like to thank Rob Urban for his creation of the Tru64 Unix collect tool, which collectl is based on. Genuine icon behavior, we love to see it 🫶

👑 AUTHOR 👑

This program was written by Mark Seger (mjseger@gmail.com).

Copyright 2003-2016 Hewlett-Packard Development Company, LP

collectl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the source kit.

And that's the whole page bestie, you ate that 💅 This is the LOCAL version, dated APRIL 2003, and it lives in section COLLECTL(1) of your friendly neighborhood manual. Go monitor something gorgeous 🌈✨💖

📦 where this page comes from