SYSDIG(8)

📦
version 0.41.4 · 8 versions
, in arch rolling

💖 NAME 💖

okay so bestie, meet sysdig(8) — sysdig is the definitive system and process troubleshooting tool. Periodt. 🫶

🌸 SYNOPSIS 🌸

sysdig [option]... [filter]

✨ DESCRIPTION ✨

First note, bestie: if you’re interested in an easier to use interface for the sysdig functionality, use the csysdig command line utility. 💅

sysdig is a tool for system troubleshooting, analysis and exploration. It can capture, filter and decode system calls and other OS events. sysdig can be both used to inspect live systems, or to generate trace files that can be analyzed at a later stage.

It includes a powerful filtering language, has customizable output, and can be extended through Lua scripts, called chisels. iconic behavior, honestly. 👑

🎀 Output format 🎀

By default, sysdig prints the information for each captured event on a single line, with the following format:

*%evt.num %evt.time %evt.cpu %proc.name (%thread.tid) %evt.dir %evt.type %evt.info

where:

  • evt.num — the incremental event number
  • evt.time — the event timestamp
  • evt.cpu — the CPU number where the event was captured
  • proc.name — the name of the process that generated the event
  • thread.tid — the TID that generated the event, which corresponds to the PID for single thread processes
  • evt.dir — the event direction, > for enter events and < for exit events
  • evt.type — the name of the event, e.g. 'open' or 'read'
  • evt.args — the list of event arguments

The output format can be customized with the -p switch, using any of the fields listed by sysdig -l. ✨

Using -pc or -pcontainer, the default format will be changed to a container-friendly one:

*%evt.num %evt.time %evt.cpu %container.name (%container.id) %proc.name (%thread.tid:%thread.vtid) %evt.dir %evt.type %evt.info

🦋 Trace Files 🦋

A trace file can be created using the -w switch:

$ sysdig -w trace.scap

The -s switch can be used to specify how many bytes of each data buffer should be saved to disk. And filters can be used to save only certain events to disk:

$ sysdig -s 2000 -w trace.scap proc.name=cat

Trace files can be read using the -r switch:

$ sysdig -r trace.scap

💅 Filtering 💅

sysdig filters are specified at the end of the command line. The simplest filter is a basic field-value check:

$ sysdig proc.name=cat

The list of available fields can be obtained with sysdig -l. Filter expressions can use one of these comparison operators: =, !=, <, <=, >, >=, contains, icontains, in and exists. e.g.

$ sysdig fd.name contains /etc
$ sysdig "evt.type in ( 'select', 'poll' )"
$ sysdig proc.name exists

Multiple checks can be combined through brackets and the following boolean operators: and, or, not. e.g.

$ sysdig "not (fd.name contains /proc or fd.name contains /dev)"

🧚 Chisels 🧚

sysdig’s chisels are little scripts that analyze the sysdig event stream to perform useful actions. To get the list of available chisels, type

$ sysdig -cl

To get details about a specific chisel, type

$ sysdig -i spy_ip

To run one of the chisels, you use the -c flag, e.g.

$ sysdig -c topfiles_bytes

If a chisel needs arguments, you specify them after the chisel name:

$ sysdig -c spy_ip 192.168.1.157

If a chisel has more than one argument, specify them after the chisel name, enclosed in quotes:

$ sysdig -c chisel_name "arg1 arg2 arg3"

Chisels can be combined with filters:

$ sysdig -c topfiles_bytes "not fd.name contains /dev"

🔥 OPTIONS 🔥

okay bestie, here come the flags — every single one, no gatekeeping. 👑

  • -A, --print-ascii — Only print the text portion of data buffers, and echo end-of-lines. This is useful to only display human-readable data.
  • -b, --print-base64 — Print data buffers in base64. This is useful for encoding binary data that needs to be used over media designed to handle textual data (i.e., terminal or json).
  • -c chiselname chiselargs, --chisel=chiselname chiselargs — run the specified chisel. If the chisel require arguments, they must be specified in the command line after the name.
  • -C filesize — Break a capture into separate files, and limit the size of each file based on the specified number of megabytes. The units of filesize are millions of bytes (10^6, not 2^20). Use in conjunction with -W to enable automatic file rotation. Otherwise, new files will continue to be created until the capture is manually stopped.

Files will have the name specified by -w with a counter added starting at 0.

  • -cl, --list-chisels — lists the available chisels. Sysdig looks for chisels in the following directories: ./chisels, ~/.chisels and /usr/share/sysdig/chisels.
  • -d, --displayflt — Make the given filter a display one. Setting this option causes the events to be filtered after being parsed by the state system. Events are normally filtered before being analyzed, which is more efficient, but can cause state (e.g. FD names) to be lost.
  • -D, --debug — Capture events about sysdig itself, display internal events in addition to system events, and print additional logging on standard error.
  • -E, --exclude-users — Don’t create the user/group tables by querying the OS when sysdig starts. This also means that no user or group info will be written to the tracefile by the -w flag. The user/group tables are necessary to use filter fields like user.name or group.name. However, creating them can increase sysdig’s startup time. Moreover, they contain information that could be privacy sensitive.
  • -e numevents — Break a capture into separate files, and limit the size of each file based on the specified number of events. Use in conjunction with -W to enable automatic file rotation. Otherwise, new files will continue to be created until the capture is manually stopped.

Files will have the name specified by -w with a counter added starting at 0.

  • -F, --fatfile — Enable fatfile mode. When writing in fatfile mode, the output file will contain events that will be invisible when reading the file, but that are necessary to fully reconstruct the state. Fatfile mode is useful when saving events to disk with an aggressive filter. The filter could drop events that would cause the state to be updated (e.g. clone() or open()). With fatfile mode, those events are still saved to file, but 'hidden' so that they won’t appear when reading the file. Be aware that using this flag might generate substantially bigger traces files.
  • --filter-proclist — apply the filter to the process table. A full dump of /proc is typically included in any trace file to make sure all the state required to decode events is in the file. This could cause the file to contain unwanted or sensitive information. Using this flag causes the command line filter to be applied to the /proc dump as well.
  • -G numseconds — Break a capture into separate files, and limit the size of each file based on the specified number of seconds. Use in conjunction with -W to enable automatic file rotation. Otherwise, new files will continue to be created until the capture is manually stopped.

Files will have the name specified by -w which should include a time format as defined by strftime(3). If no time format is specified, a counter will be used.

  • -h, --help — Print this page.
  • -H pluginname[:initconfig], --plugin pluginname[:initconfig] — Registers a plugin, using the passed init config if present. A path can also be used as pluginname. The format of initconf is controlled by the plugin, refer to each plugin’s documentation to learn about it.
  • -I pluginname[:openparams], --input pluginname[:openparams] — Capture events using the plugin with name pluginname, passing to the plugin the openparams string as parameters. The format of inputargs is controller by the plugin, refer to each plugin’s documentation to learn about it. The event sources available for capture vary depending on which plugins have been installed. You can list the plugins that have been loaded by using the -Il flag.
  • -Il, --list-inputs — List the loaded plugins. Sysdig looks for plugins in the following directories: ./plugins, ~/.plugins, /usr/share/sysdig/plugins.
  • --plugin-config-file — Load the plugin configuration from a Falco-compatible yaml config file. Mixing this option with -H or -I is unsupported. See the plugin section in https://falco.org/docs/configuration/ for additional informations.
  • -i chiselname, --chisel-info=chiselname — Get a longer description and the arguments associated with a chisel found in the -cl option list.
  • -j, --json — Emit output as json, data buffer encoding will depend from the print format selected.
  • -k, --k8s-api — Enable Kubernetes support by connecting to the API server specified as argument. E.g. "http://admin:password@127.0.0.1:8080". The API server can also be specified via the environment variable SYSDIG_K8S_API.
  • -K btfile | certfile:keyfile[#password][:cacertfile], --k8s-api-cert=btfile | certfile:keyfile[#password][:cacertfile] — Use the provided files names to authenticate user and (optionally) verify the K8S API server identity. Each entry must specify full (absolute, or relative to the current directory) path to the respective file. Private key password is optional (needed only if key is password protected). CA certificate is optional. For all files, only PEM file format is supported. Specifying CA certificate only is obsoleted - when single entry is provided for this option, it will be interpreted as the name of a file containing bearer token. Note that the format of this command-line option prohibits use of files whose names contain ':' or '#' characters in the file name. Option can also be provided via the environment variable SYSDIG_K8S_API_CERT.
  • -L, --list-events — List the events that the engine supports.
  • -l, --list — List the fields that can be used for filtering and output formatting. Use -lv to get additional information for each field.
  • --list-markdown — Like -l, but produces markdown output.
  • -m url[,marathon-url], --mesos-api=_url[,marathon-url]_ — turns on Mesos support by connecting to the API server you pass in (like http://admin:password@127.0.0.1:5050, credentials and all ✨). The Mesos url is required, bestie, no skipping it. The Marathon url is optional and defaults to auto-follow — meaning if you don't hand over a Marathon API server, sysdig will try to fetch the location of the Marathon API server from the Mesos master, and then keep following it if it migrates. 🧚 Heads up though: with auto-follow, sysdig will likely get a cluster-internal IP address for the Marathon API server, so running sysdig with Marathon auto-follow from a node that isn't part of the Mesos cluster may just not work 😭. Also, running sysdig with Mesos support on a node that has no containers managed by Mesos is of limited use — you'll still collect cluster metadata, but there'll be no Mesos/Marathon filtering capability. The API servers can also be specified via the environment variable SYSDIG_MESOS_API 🫶
  • -M num_seconds — stop collecting after reaching... yeah, the sentence trails off in the original like a text you sent at 3am 💀 — that's literally how it reads.
  • -n num, --numevents=num — stop capturing after num events.
  • --page-faults — capture user/kernel major/minor page faults.
  • -P, --progress — print progress on stderr while processing trace files.
  • -p outputformat, --print=outputformat — pick the format used when printing the events 🎀. With -pc or -pcontainer you get a container-friendly format. With -pk or -pkubernetes you get a kubernetes-friendly format. With -pm or -pmesos you get a mesos-friendly format. And specifying -pp on the command line makes sysdig print the default command line format and exit — iconic little flex 💅
  • -q, --quiet — don't print events on the screen. Genuinely useful when you're dumping to disk (no one needs that visual noise ❤️).
  • -r readfile, --read=readfile — read the events from readfile.
  • -R, --resolve-ports — resolve port numbers to names.
  • -S, --summary — print the event summary (i.e. the list of the top events) when the capture ends.
  • -s len, --snaplen=len — capture the first len bytes of each I/O buffer. By default, the first 80 bytes are captured. Use this option with caution — it can generate huge trace files 😭🔥
  • -t timetype, --timetype=timetype — change the way event time is displayed ⏰. Accepted values:
    • h — human-readable string
    • a — absolute timestamp from epoch
    • r — relative time from the first displayed event
    • d — delta between event enter and exit
    • D — delta from the previous event
  • -T, --force-tracers-capture — tells the driver to make sure full buffers are captured from /dev/null, so tracers are completely captured 🦋. Ngl, sysdig will enable extended /dev/null capture by itself after it detects tracers are written there — but that could result in the truncation of some tracers at the beginning of the capture. This option lets you prevent that.
  • --unbuffered — turns off output buffering. Every single line sysdig emits gets flushed, which means higher CPU usage but it's chef's kiss when you're piping sysdig's output into another process or into a script 🍓
  • -v, --verbose — verbose output. This flag makes the full content of text and binary buffers print on screen instead of being truncated to 40 characters. Note the data buffers length is still limited by the snaplen (see the -s flag docs). Also, -v makes sysdig print some summary information at the end of the capture ✨
  • --version — print version number.
  • -w writefile, --write=writefile — write the captured events to writefile.
  • -W num — turns on file rotation for continuous capture and limits the number of files created to num. Once the cap is reached, older files get overwritten (ring buffer energy 🔄). Use it together with the -C / -G / -e options to limit the size of each file based on number of megabytes, seconds, and/or events (respectively).
  • -x, --print-hex — print data buffers in hex.
  • -X, --print-hex-ascii — print data buffers in hex and ASCII.
  • -z, --compress — used with -w, enables compression for tracefiles 🎀

🌈 EXAMPLES 🌈

Capture all the events from the live system and print them to screen

$ sysdig

Capture all the events from the live system and save them to disk

$ sysdig -w dumpfile.scap

Capture all the events in the latest 24 hours and save them to disk, organized in files containing 1 hour of system activity each

$ sysdig -G 3600 -W 24 -w dumpfile.scap

Read events from a file and print them to screen

$ sysdig -r dumpfile.scap

Prepare a sanitized version of a system capture

$ sysdig -r dumpfile.scap 'not evt.buffer contains foo' -w cleandump.scap

Print all the open system calls invoked by cat

$ sysdig proc.name=cat and evt.type=open

Print the name of the files opened by cat

$ sysdig -p"%evt.arg.name" proc.name=cat and evt.type=open

List the available chisels

$ sysdig -cl

Use the spy_ip chisel to look at the data exchanged with 192.168.1.157:

$ sysdig -c spy_ip 192.168.1.157

🎀 FILES 🎀

  • /usr/share/sysdig/chisels — the global chisels directory.
  • ~/.chisels — the personal chisels directory (yes, your own little chisel stash 💖).

💀 BUGS 💀

  • sysdig and its chisels are designed to be used with LuaJIT in Lua 5.1 mode. While it is possible to use sysdig with LuaJIT in Lua 5.2 mode or regular Lua, some chisels may not work as expected — so like, don't come for me if a chisel acts up 🙈

👑 AUTHOR 👑

Draios Inc. aka sysdig info@sysdigcloud.com

🌸 SEE ALSO 🌸

csysdig(8), strace(8), tcpdump(8), lsof(8)

That's the whole page, bestie — the footer at the very bottom is just the usual source/version and date line that every manpage wears like a little name tag, so consider this your cute sign-off instead: go forth, capture some syscalls, and slay 💅💜

📦 where this page comes from