STRACE(1)
๐ฆ strace(1) โ trace system calls and signals ๐ฆ
okay bestie, pull up a chair, we're reading the manual page for strace(1) and honestly? it's a whole vibe. it's a General Commands Manual situation and it's about to eat ๐
๐ธ NAME ๐ธ
strace โ trace system calls and signals โจ
๐ SYNOPSIS ๐
strace [-ACdffhikkqqrtttTvVwxxyyYzZ] [-a column] [-b execve]
[-e expr]... [-I n] [-o file] [-O overhead] [-p pid]...
[-P path]... [-s strsize] [-S sortby] [-U columns] [-X format]
[--seccomp-bpf] [--stack-trace-frame-limit=limit]
[--syscall-limit=limit] [--secontext[=format]] [--tips[=format]]
{ -p pid | [-DDD] [-E var[=val]]... [-u username] command [args]
}
strace -c [-dfwzZ] [-b execve] [-e expr]... [-I n] [-O overhead]
[-p pid]... [-P path]... [-S sortby] [-U columns] [--seccomp-bpf]
[--syscall-limit=limit] [--tips[=format]] { -p pid | [-DDD]
[-E var[=val]]... [-u username] command [args] }
strace --tips[=format]
like, that's a lot of letters, and every single one of them is doing something. no cap. ๐
๐ DESCRIPTION ๐
Okay so the simplest possible use case: strace runs the command you give it until that command dips out (exits). The whole time, it's intercepting and recording the system calls the process makes, plus the signals the process receives. Then it prints the name of every system call, its arguments, and its return value โ either to standard error, or to a file if you passed the -o option. ๐ซถ
strace is a genuinely useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians, and troubleshooters will find it invaluable for solving problems with programs where the source code just isn't available โ because bestie, recompilation is not required to trace. Students, hackers, and the overly-curious will discover that you can learn a great deal about a system and its system calls just by tracing even the most ordinary programs. And programmers? They'll find that since system calls and signals happen at the user/kernel interface, staring really hard at that boundary is super useful for bug isolation, sanity checking, and attempting to capture race conditions. ๐โจ
Every line in the trace has the system call name, then its arguments in parentheses, then its return value. Here's an example from tracing the command cat /dev/null:
open("/dev/null", O_RDONLY) = 3
Errors โ usually showing up as a return value of -1 โ get the errno symbol and error string tacked on. ๐ญ
open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)
Signals get printed as a signal symbol plus a decoded siginfo structure. Here's an excerpt from tracing and interrupting the command sleep 666:
sigsuspend([] <unfinished ...>
--- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=...} ---
+++ killed by SIGINT +++
If a system call is still executing while another gets called from a different thread or process, strace will attempt to preserve the order of these events and mark the ongoing call as unfinished. When it comes back, it gets marked as resumed. ๐
[pid 28772] select(4, [3], NULL, NULL, NULL <unfinished ...>
[pid 28779] clock_gettime(CLOCK_REALTIME, {tv_sec=1130322148, tv_nsec=3977000}) = 0
[pid 28772] <... select resumed> ) = 1 (in [3])
The interruption of a (restartable) system call by a signal delivery is handled differently, because the kernel terminates the system call and arranges for its immediate re-execution after the signal handler finishes. ๐
read(0, 0x7ffff72cf5cf, 1) = ? ERESTARTSYS (To be restarted)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
rt_sigreturn({mask=[]}) = 0
read(0, "", 1) = 0
Arguments get printed in symbolic form, with passion. This next one shows the shell doing a >>xyzzy output redirection:
open("xyzzy", O_WRONLY|O_APPEND|O_CREAT, 0666) = 3
See how the second and third arguments of open(2) get decoded โ the flag argument gets broken down into its three bitwise-OR constituents, and the mode value is printed in octal, following tradition. Where traditional or native usage differs from ANSI or POSIX, the latter forms win. In some cases, strace output has proven to be more readable than the source code itself. iconic behavior honestly. ๐
Structure pointers get dereferenced, and their members get displayed as appropriate. In most cases, arguments are formatted in the most C-like fashion possible. For example, the essence of the command ls -l /dev/null is captured as:
lstat("/dev/null", {st_mode=S_IFCHR|0666, st_rdev=makedev(0x1, 0x3), ...}) = 0
Notice how the struct stat argument is dereferenced, and how each member is displayed symbolically. In particular, ๐ observe how the st_mode member is carefully decoded into a bitwise-OR of symbolic and numeric values. Also note that in this example, the first argument to lstat(2) is an input to the system call, and the second argument is an output. Since output arguments don't get modified if the system call fails, arguments may not always be dereferenced. For example, retrying that ls -l example with a non-existent file produces this line:
lstat("/foo/bar", 0xb004) = -1 ENOENT (No such file or directory)
In this case, the porch light is on but nobody is home. ๐ The pointer's value is displayed because the structure it points to wasn't populated, on account of the error.
System calls that strace doesn't know about get printed in a raw format, with the hexadecimal system call number prefixed with "syscall_":
syscall_0xbad(0x1, 0x2, 0x3, 0x4, 0x5, 0x6) = -1 ENOSYS (Function not implemented)
Character pointers get dereferenced and printed as C strings. Non-printing characters inside strings are normally represented by standard C escape codes. Only the first strsize bytes of strings are printed (32 by default), and longer strings get an ellipsis appended after the closing quote. Here's a line from ls -l where the getpwuid(3) library routine is reading the password file:
read(3, "root::0:0:System Administrator:/"..., 1024) = 422
While structures are annotated using curly braces, pointers to basic types and arrays are printed using square brackets, with commas separating the elements. Here's an example from the command id(1) on a system with supplementary group IDs:
getgroups(32, [100, 0]) = 2
On the other hand, bit-sets are also shown using square brackets, but the set elements are separated only by a space. Here's the shell, preparing to execute an external command:
sigprocmask(SIG_BLOCK, [CHLD TTOU], []) = 0
Here, the second argument is a bit-set of two signals, SIGCHLD and SIGTTOU. In some cases the bit-set is so full that it's more valuable to print the unset elements instead. In that case, the bit-set is prefixed by a tilde, like this:
sigprocmask(SIG_UNBLOCK, ~[], NULL) = 0
Here, the second argument represents the full set of all signals. โจ
๐ OPTIONS ๐
๐ General ๐
-eexpr โ Modifies which events get traced or how they get traced, by specifying a qualifying expression. The format of the expression is:
[qualifier=][!]value[,value]...
where qualifier is one of trace (or t), trace-fds (or trace-fd or fd or fds), abbrev (or a), verbose (or v), raw (or x), signal (or signals or s), read (or reads or r), write (or writes or w), fault, inject, status, quiet (or silent or silence or q), secontext, decode-fds (or decode-fd), decode-pids (or decode-pid), or kvm, and value is a qualifier-dependent symbol or number. The default qualifier is trace. Using an exclamation mark negates the set of values. For example, -e open is equivalent to -e trace=open, which in turn means trace only the open system call. By contrast, -e trace=!open means trace every system call except open. And on top of all that, the special values all and none may be used to trace every event or no events, respectively. ๐
โ ๏ธ Heads up bestie: some shells use the exclamation mark for history expansion even inside quoted arguments. In that case, the exclamation mark must be escaped with a backslash. Don't let your shell eat your negation, that's so embarrassing for everyone. ๐ญ
๐ Startup ๐
-Evar=val--env=var=valโ Runs the command with the environment variable var=val set for execution. ๐ซถ-Evar--env=varโ Removes var from the inherited environment variables before executing the command. ๐จ-ppid--attach=pidโ Attaches to the process with the process ID pid and begins tracing. The trace may be terminated at any time by a keyboard interrupt signal (Ctrl+C).stracewill respond by detaching itself from the traced processes, leaving them to continue running. ๐โโ๏ธ
Multiple -p options can be used to attach to several processes in addition to the command, and the command is optional if at least one -p option is given.
A single -p option can accept multiple process IDs separated by a comma (โ,โ), space (โ โ), tab, or newline. Consequently, syntaxes like -p "$(pidof PROG)" and -p "$(pgrep PROG)" are supported. โจ
-uusername--user=usernameโ Runs command with the user ID, group ID, and supplementary groups of username. This option is only useful when running as root, since it enables the correct execution of setuid and/or setgid binaries. Unless this option is used, setuid and setgid programs are executed without their effective privileges. ๐-uUID:GID--user=UID:GIDโ Alternative syntax where the program is started with exactly the given user and group IDs, and an empty list of supplementary groups. In this case, user and group name lookups are not performed. ๐--argv0=nameโ Sets the executed command'sargv[0]to name. This is useful for tracing multi-call executables that interpretargv[0], such asbusyboxorkmod. ๐ฆ
๐ Tracing ๐
-bsyscall--detach-on=syscallโ Detaches from the traced process if the specified system call is reached. Currently, only theexecvekeyword is supported, which includes theexecve(2)andexecveat(2)system calls. This option is useful for tracing a multi-threaded process with-fwithout also tracing its (potentially very complex) child processes. ๐ง-D--daemonize--daemonize=grandchildโ Runs the tracer process as a grandchild of the tracee, not as its parent. This reduces the visible effect ofstraceby keeping the tracee a direct child of the calling process. ๐ธ-DD,--daemonize=pgroup,--daemonize=pgrpโ runs the tracer process as the tracee's grandchild, in its own separate process group. besides making strace way less visible to the thing it's tracing, this also means strace won't get merked by akill(2)signal that somebody sent to the whole process group. iconic protective behaviour honestly ๐-DDD,--daemonize=sessionโ runs the tracer process as the tracee's grandchild in a separate session โ that's the "true daemonisation" version, the real deal โจ. on top of the whole reduced-visibility thing, it also keeps strace from getting terminated when the session ends. she simply will not be ended ๐-f,--follow-forksโ traces child processes as they're created by processes you're already tracing, wheneverfork(2),vfork(2)orclone(2)happens. so you get the whole family tree, bestie ๐ธ note tho: if the processPIDis multi-threaded, then-f -p PIDattaches to all of its threads, not just the one withthread_id = PID. no leaving the siblings behind.--output-separatelyโ if--output=filenameis active, the trace for each process gets written to its own separatefilename.pidfile, wherepidis the process ID. one file per baby, organized queen ๐-ff,--follow-forks --output-separatelyโ the combo move: does what--follow-forksdoes and what--output-separatelydoes at the same time. heads up: this is incompatible with-c, because no per-process counts get kept. ๐ซ
to get one combined view of all those log files, use strace-log-merge(1). see also: SEE ALSO, she's everywhere ๐
-I interruptible,--interruptible=interruptibleโ controls when strace is allowed to be interrupted by signals (like when you smash Ctrl+C). pick your vibe:1,anywhereโ no signals are blocked, free-for-all โจ2,waitingโ fatal signals are blocked while decoding a system call. this is the default ๐3,neverโ fatal signals are always blocked. this is the default if you use-o FILE PROG4,never_tstpโ fatal signals andSIGTSTP(Ctrl+Z) are always blocked. super useful if you wantstrace -o FILE PROGto not stop on Ctrl+Z. this one's the default if you use-D๐
--syscall-limit=limitโ detaches all tracees oncelimitsystem calls have been captured. ngl this is your "okay that's enough babe" option ๐ซถ one lil detail: system calls filtered out by--trace,--trace-pathor--statusdon't count toward that number. irrelevant calls simply don't make the tally.--kill-on-exitโ applies thePTRACE_O_EXITKILLptrace option to every tracee, which sends aSIGKILLsignal to a tracee if the tracer exits. basically: nobody gets left running after the tracer dips, because tracees will not be detached on cleanup. > โ ๏ธ--kill-on-exitis not compatible with-p/--attachoptions. pick one bestie, you cannot have both ๐
๐ฆ Filtering ๐ฆ
-e trace=syscall_set,-e t=syscall_set,--trace=syscall_setโ traces only the system calls in the set you name.syscall_setis defined as[!]value[,value], and eachvaluecan be any of these cuties:syscallโ traces one specific system call, named by its name (peepsyscalls(2)for a reference, but also go read NOTES ๐).?valueโ a question mark in front of a qualification suppresses errors if no matching system calls are found. it's the "no worries if not" of strace ๐ธvalue@64โ limits the system call spec described byvalueto the 64-bit personality.value@32โ limits the system call spec described byvalueto the 32-bit personality.value@x32โ limits the system call spec described byvalueto the x32 personality.allโ traces every single system call. the whole menu ๐/regexโ traces only the system calls that match the regex. you can use POSIX Extended Regular Expression syntax here (seeregex(7)). regex girlies stay winning โจ%file,fileโ traces all system calls that take a file name as an argument. think of it as a lil abbreviation for--trace=open,stat,chmod,unlink,..., which is genuinely so useful for seeing what files a process is touching. and using the abbreviation means you won't accidentally forget to include a call likenewfstatat(2). btw the syntax without the leading percent sign (--trace=file) is deprecated, so use the%. ๐%process,processโ traces system calls tied to process lifecycle (creation, exec, termination). the no-percent version (--trace=process) is deprecated.%net,%network,networkโ traces all the network related system calls.--trace=networkwithout the percent sign is deprecated.%signal,signalโ traces all signal related system calls.--trace=signalwithout the percent sign is deprecated.%ipc,ipcโ traces all IPC related system calls.--trace=ipcwithout the percent sign is deprecated.%desc,descโ traces all file descriptor related system calls.--trace=descwithout the percent sign is deprecated.%memory,memoryโ traces all memory mapping related system calls.--trace=memorywithout the percent sign is deprecated.%credsโ traces system calls that read or modify user and group identifiers or capability sets. ๐%statโ tracesstatsystem call variants.%lstatโ traceslstatsystem call variants.%fstatโ tracesfstat,fstatat, andstatxsystem call variants.%%statโ traces system calls used for requesting file status (stat,lstat,fstat,fstatat,statx, and their variants). yes, that's a double percent, she's fancy ๐%statfsโ tracesstatfs,statfs64,statvfs,osf_statfs, andosf_statfs64system calls. same effect you'd get from the regular expression--trace=/^(.*_)?statv?fsโจ%fstatfsโ tracesfstatfs,fstatfs64,fstatvfs,osf_fstatfs, andosf_fstatfs64system calls. same effect as--trace=/fstatv?fs๐%%statfsโ traces system calls related to file system statistics (thestatfs-like,fstatfs-like, andustatones). same effect as--trace=/statv?fs|fsstat|ustat๐%clockโ traces system calls that read or modify system clocks. โฐ%pureโ traces system calls that always succeed and take no arguments. rn that list is:arc_gettls(2),getdtablesize(2),getegid(2),getegid32(2),geteuid(2),geteuid32(2),getgid(2),getgid32(2),getpagesize(2),getpgrp(2),getpid(2),getppid(2),get_thread_area(2)(on architectures other than x86),gettid(2),get_tls(2),getuid(2),getuid32(2),getxgid(2),getxpid(2),getxuid(2),kern_features(2), andmetag_get_tls(2)system calls. pure of heart, zero drama ๐ฑ
the -c option is genuinely useful for figuring out which system calls are worth tracing, like a lil vibe check. for example, --trace=open,close,read,write means only those four get traced. > โ ๏ธ but be careful when making inferences about the user/kernel boundary if only a subset of system calls is being monitored. the default is --trace=all.
-e trace-fd=set,-e trace-fds=set,-e fd=set,-e fds=set,--trace-fds=setโ traces only the system calls that operate on the specified subset of (non-negative) file descriptors. note that using this also filters out every system call that doesn't operate on file descriptors at all โ bye ๐
this filter gets combined with the --trace-path filter: a system call is traced if it matches either of them. it's a union, bestie ๐ซถ
-e signal=set,-e signals=set,-e s=set,--signal=setโ traces only the specified subset of signals. the default is--signal=all. for example,--signal=!SIGIO(or--signal=!io) makes it soSIGIOsignals don't get traced. the!is your "not this one" โ-e status=set,--status=setโ prints only system calls with the specified return status. the default is--status=all. > โ ๏ธ ngl there's a catch: when you use the status qualifier, the chronological order of events may not be preserved. why? because strace has to wait for a system call to complete before deciding whether to print it. so if two concurrent threads each run a system call, strace will print both the entry and exit of whichever one exits first, no matter what their entry times were โ and then the entry and exit of the second one to exit get printed afterwards. it's giving non-chronological, sorry ๐
here's an example where select(2) is called, but a different thread calls clock_gettime(2) before select(2) finishes:
[pid 28779] 1130322148.939977 clock_gettime(CLOCK_REALTIME, {1130322148, 939977000}) = 0
[pid 28772] 1130322148.438139 select(4, [3], NULL, NULL, NULL) = 1 (in [3])
set can include the following elements:
successfulโ only the syscalls that came back with no error code. Literally exactly what-zdoes, so-zis just--status=successfulwith a glow-up. โจfailedโ only the syscalls that came back with an error code. Same energy:-Zis--status=failed.unfinishedโ the ones that never returned at all ๐ญ. This can happen when, say, anexecvecall happens in a different thread of the same thread group.unavailableโ the syscall did return, but strace couldn't fetch the error status. rip.detachedโ syscalls where strace detached before the return happened.
-Ppath--trace-path=pathโ traces only the system calls that touch path. You can flex multiple-Poptions to name several paths. Heads up: this filter stacks with the--trace-fdsfilter, and a syscall gets traced if it matches either one. ๐ซถ-z--successful-onlyโ prints only the system calls that returned without an error code. clean, flawless, no drama ๐-Z--failed-onlyโ prints only the system calls that returned with an error code. the messy ones. we love them.
๐ Output format ๐
-acolumn--columns=columnโ lines the return values up in a specific column. default column is40. aesthetic โจ-e abbrev=syscall_set-e a=syscall_set--abbrev=syscall_set โ abbreviates the output so it stops printing every single member of big chonky structures. the syntax of the syscall_set spec is the same as in--trace. default is--abbrev=all. and bestie,-vhas the effect of--abbrev=none. ๐ฆ-e verbose=syscall_set-e v=syscall_set--verbose=syscall_set โ dereferences structures for the specified set of syscalls. syscall_set syntax is the same as in--trace. default is--verbose=all. we go deep. ๐ฅ-e raw=syscall_set-e x=syscall_set--raw=syscall_set โ prints raw, undecoded arguments for the specified set of syscalls. syscall_set syntax = same as--trace. this one makes all arguments print in hexadecimal. super useful if you don't trust the decoding, or if you genuinely need the actual numeric value of an argument. also see the-X rawoption. ๐-e read=set-e reads=set-e r=set--read=set โ does a full hexadecimal and ASCII dump of all the data read from the file descriptors listed in set. like, to see all input activity on fds 3 and 5, you'd do--read=3,5. ngl this is independent from the normal tracing of theread(2)system call, which is the thing controlled by--trace=read. ๐-e write=set-e writes=set-e w=set--write=set โ full hexadecimal and ASCII dump of all the data written to the file descriptors listed in set. e.g. to see all output activity on fds 3 and 5:--write=3,5. same deal as above โ independent from the normal tracing ofwrite(2), which--trace=writecontrols. ๐ธ-e quiet=set-e silent=set-e silence=set-e q=set--quiet=set--silent=set--silence=set โ suppresses a bunch of informational messages. default is--quiet=none. set can include these elements:attachโ suppresses the messages about attaching and detaching, aka"[ Process NNNN attached ]"and"[ Process NNNN detached ]".exitโ suppresses the messages about process exits, aka"+++ exited with SSS +++".path-resolutionโ suppresses the messages about resolution of paths handed over via the-Poption, aka"Requested path "..." resolved into "..."".personalityโ suppresses the messages about process personality changes, aka"[ Process PID=NNNN runs in PPP mode. ]".thread-execvesupersededโ suppresses the messages about a process being superseded byexecve(2)in another thread, aka"+++ superseded by execve in pid NNNN +++".
-e decode-fds=set--decode-fds=set โ decodes all sorts of info tied to file descriptors. default is--decode-fds=none. set can include these elements:pathโ prints file paths. it also switches on printing of the tracee's current working directory whenever theAT_FDCWDconstant is used.socketโ prints socket protocol-specific information.devโ prints character/block device numbers.eventfdโ prints eventfd object details associated with eventfd file descriptors.pidfdโ prints PIDs associated with pidfd file descriptors.signalfdโ prints signal masks associated with signalfd file descriptors.
-e decode-pids=set--decode-pids=set โ decodes info tied to process IDs (and thread IDs, process group IDs, and session IDs too, because why not). default is--decode-pids=none. set can include:commโ prints command names associated with thread or process IDs.pidnsโ prints thread, process, process group, and session IDs in strace's PID namespace when the tracee is living in a different PID namespace.
-e kvm=vcpu|vcpu+--kvm=vcpu|vcpu+โ prints the exit reason of a kvm vcpu (=vcpu), or prints the wholekvm_runstruct *including* the reason (=*vcpu*+). requires Linux kernel version4.16.0` or higher. โญ
The vcpu+ output uses the prefixes VCPU:CPU#< and VCPU:CPU#>. Lines starting with the former show the state of the structure before calling KVM_RUN, and the ones starting with the latter show the state after calling KVM_RUN.
-e namespace=new--namespace=newโ prints the new namespaces entered by the tracee. supported system calls:clone(2),clone3(2),setns(2), andunshare(2). ๐ง-i--instruction-pointerโ prints the instruction pointer at the moment of the system call.-n--syscall-numberโ prints the system call number.-N--arg-namesโ prints the system call argument names.-k--stack-trace[=symbol]โ prints the execution stack trace of the traced processes after every system call. ๐ซถ-kk--stack-trace=sourceโ prints the execution stack trace and source code info of the traced processes after every system call. this option expects your target program to be compiled with the appropriate debug options:"-g"(gcc), or"-g -gdwarf-aranges"(clang).--stack-trace-frame-limit=limitโ prints no more than this many stack trace frames when backtracing a system call. the default is256. use it together with the--stack-trace(or-k) option.-ofilename--output=filename โ writes the trace output to the file filename instead of to stderr. if-ffis supplied, thefilename.pidform is used. if the argument starts with'|'or'!', the rest of the argument is treated as a command and all the output gets piped to it โ super convenient for piping debug output into a program without messing with the redirections of the executed programs. ๐
ok important: piping output to a command is not currently compatible with the -ff option. noted, bestie.
-A--output-append-modeโ opens the file given in the-ooption in append mode.--color=when โ colorizes the trace output. default is--color=auto, which only turns color on when the trace output stream is a terminal (stderr by default; see-o) that supports at least 8 colors. if theNO_COLORenvironment variable is set, color is disabled unless--color=alwaysis specified. the when parameter can be one ofauto,always, ornever. custom color schemes can be specified with theSTRACE_COLORSenvironment variable, for example:
STRACE_COLORS="syscall=1;33:argname=39:argval=35:call=1;34:const=1;36:comment=36:punct=0:retval=32:error=31"
The format of this value is key=value pairs separated by :, with semicolon-separated SGR codes as values.
The default palette uses standard ANSI colors (SGR 30โ37) and bold attributes rather than bright colors (SGR 90โ97), so that terminal color themes can adapt them for both dark and light backgrounds. iconic, honestly. ๐
Recognized color keys are: syscall, argname, argval, call, const, comment, punct, retval, error.
-q--quiet--quiet=attach,personalityโ suppresses the messages about attaching, detaching, and personality changes. this happens automatically when output is redirected to a file and the command is run directly instead of attaching.-qq--quiet=attach,personality,exitโ suppresses the messages about attaching, detaching, personality changes, and process exit status.-qqq--quiet=allโ suppresses all suppressible messages (go check the--quietoption description for the full list of suppressible messages, bestie).-r,--relative-timestamps[=precision]โ stamps a relative timestamp on every syscall entry ๐. It's literally recording the time gap between the start of one syscall and the start of the next one.<italic>precision</italic>can bes(seconds),ms(milliseconds),us(microseconds) orns(nanoseconds), and it sets how precise that printed time value gets. Default isus(microseconds). Heads up bestie:-rrides the monotonic clock, so its numbers can disagree with the time differences-treports, because-tuses the wall clock. Different clocks, different vibes ๐ฐ๏ธ-s strsize,--string-limit=strsizeโ sets the max string size that gets printed (default is32) โ๏ธ. But ngl, filenames are not treated as strings, so they always get printed in full, no truncation, no exceptions ๐--absolute-timestamps[=[[format:]format],[[precision:]precision]],--timestamps[=[[format:]format],[[precision:]precision]]โ puts the wall clock time at the front of every line of the trace, in the<italic>format</italic>you picked with the<italic>precision</italic>you picked โจ.<italic>format</italic>can be any of these:noneโ no timestamp printed at all. Lowkey useful for overriding whatever you set before ๐timeโ wall clock time (thestrftime(3)format string is%T)unixโ number of seconds since the epoch (thestrftime(3)format string is%s)
<italic>precision</italic> can be s (seconds), ms (milliseconds), us (microseconds) or ns (nanoseconds). The default arguments for this option are format:time,precision:s ๐
-t,--absolute-timestampsโ prefixes every line of the trace with the wall clock time โฐ-tt,--absolute-timestamps=precision:usโ prints the wall clock time with microsecond precision, she's precise ๐ซ-ttt,--absolute-timestamps=format:unix,precision:usโ prints the wall clock time as seconds since the epoch, with microsecond precision ๐-T,--syscall-times[=precision]โ shows how long each system call actually took โฑ๏ธ. It records the time difference between the beginning and the end of each syscall.<italic>precision</italic>can bes(seconds),ms(milliseconds),us(microseconds) orns(nanoseconds), and it sets how precise that printed time value gets. Default isus(microseconds).-v,--no-abbrevโ prints unabbreviated versions of environment, stat, termios, etc. calls ๐. Those structures are super common, so by default strace shows only a reasonable subset of their members. Use this option when you want every single member, in full detail, no crumbs left behind ๐--strings-in-hex[=option]โ controls whether hexadecimal escape sequences get used when printing strings ๐ข. This option changes the default escaping behavior.
Normally (so when neither this option nor -x is used), strace brings in escape sequences in two situations: to represent non-printable and non-ASCII characters (meaning those with character codes less than 32 or greater than 127), or to disambiguate output โ for example by escaping the quotation marks that enclose a string, or the angle brackets used in file descriptor paths. When a character needs escaping, strace prioritizes the symbolic C-standard sequences if one exists: \t (tab), \n (newline), \v (vertical tab), \f (form feed), and \r (carriage return). For every other character that requires escaping, strace defaults to an octal representation of the character's byte value. This option lets you override that default and use hexadecimal escapes instead of octal ones.
<italic>option</italic> can be one of these:
noneโ hexadecimal numbers aren't used in the output at all. When there's a need to emit an escape sequence, octal numbers are used.non-ascii-charsโ hexadecimal numbers are used instead of octal in the escape sequences.non-asciiโ strings that contain non-ASCII characters are printed using escape sequences with hexadecimal numbers.allโ all strings are printed using escape sequences with hexadecimal numbers.
When the option is given with no argument, all is assumed ๐
-x,--strings-in-hex=non-asciiโ prints all non-ASCII strings in hexadecimal string format ๐ฅ-xx,--strings-in-hex[=all]โ prints all strings in hexadecimal string format, absolutely everything, no cap ๐-X format,--const-print-style=formatโ sets the format for printing named constants and flags. The supported<italic>format</italic>values are:rawโ raw number output, without decoding.abbrevโ outputs a named constant or a set of flags instead of the raw number, if they're found. This is the default strace behaviour ๐verboseโ outputs both the raw value and the decoded string (as a comment).
-y,--decode-fds,--decode-fds=pathโ prints paths associated with file descriptor arguments and with theAT_FDCWDconstant ๐-yy,--decode-fds=allโ prints all available info associated with file descriptors: protocol-specific info associated with socket file descriptors, block/character device number associated with device file descriptors, and PIDs associated with pidfd file descriptors. The full lore ๐--pidns-translation,--decode-pids=pidnsโ if strace and the tracee are in different PID namespaces, it prints PIDs in strace's namespace too ๐-Y,--decode-pids=commโ prints command names for PIDs ๐ท๏ธ--secontext[=format],-e secontext=formatโ when SELinux is available and not disabled, prints the SELinux contexts of processes, files, and descriptors inside square brackets ๐. The<italic>format</italic>argument is a comma-separated list of items, and each item is one of these:fullโ prints the full context (user, role, type level and category).mismatchโ also prints the context recorded by the SELinux database in case the current context differs. That one gets printed after two exclamation marks (!!).
The default value for --secontext is !full,mismatch, which prints only the type instead of the full context and doesn't check for context mismatches ๐
--always-show-pidโ shows the PID prefix also for the process started by strace ๐. It's implied when-fand-oare both specified.
โจ Statistics โจ
-c,--summary-onlyโ counts time, calls, and errors for each system call and reports a summary when the program exits, suppressing the regular output ๐. This shows system time (CPU time spent in the kernel), which is independent of wall clock time. And if-cis used with-f, only aggregate totals for all traced processes are kept. One big girlboss total ๐-C,--summaryโ like-c, but it also prints the regular output while processes are running ๐ซถ. Best of both worlds, honestly.-O overhead,--summary-syscall-overhead=overheadโ sets the overhead for tracing system calls to<italic>overhead</italic>โ๏ธ. This is useful for overriding the default heuristic, which estimates the time spent in the measurement process itself when timing system calls with the-coption. The same overhead gets subtracted from polymorphic time columns and fromwall-total,wall-min,wall-max, andwall-avg. You can gauge how accurate the heuristic is by timing a given program run without tracing (usingtime(1)) and comparing the accumulated system call time to the total you got from-c.
The format of the <italic>overhead</italic> specification is described in the section Time specification format description.
-S sortby,--summary-sort-by=sortbyโ sorts the output of the histogram printed by the-coption by whatever criterion you pick ๐. Valid values aretime(ortime-percentortime-totalortotal-time),min-time(orshortestortime-min),max-time(orlongestortime-max),avg-time(ortime-avg),wall-total(ortotal-wall),wall-min(ormin-wall),wall-max(ormax-wall),wall-avg(oravg-wall),calls(orcount),errors(orerror),name(orsyscallorsyscall-name), andnothing(ornone). Default istimeโณ-U columns,--summary-columns=columnsโ configures the set and the order of columns shown in the call summary ๐๏ธ. The<italic>columns</italic>argument is a comma-separated list containing one or more of the following values:
and here's the rest of the column menu, bestie โ pick your faves ๐
time-percent(ortime) โ the percentage of cumulative time that one specific syscall gobbled up. ๐ฝ๏ธtotal-time(ortime-total) โ total system time (or wall-clock time, if you passed-w) eaten by a specific syscall.min-time(orshortestortime-min) โ the smallest call duration we saw, in seconds.max-time(orlongestortime-max) โ the biggest call duration we saw, in seconds.avg-time(ortime-avg) โ average call duration, in microseconds per call.wall-total(ortotal-wall) โ total wall-clock time a specific syscall consumed. This one is always wall-clock no matter what, even if you didn't pass-wโ so you can show it right next tototal-timein one single run. ๐wall-min(ormin-wall) โ smallest observed wall-clock call duration, in seconds.wall-max(ormax-wall) โ biggest observed wall-clock call duration, in seconds.wall-avg(oravg-wall) โ average wall-clock call duration, in microseconds per call.calls(orcount) โ how many times it got called. ๐ขerrors(orerror) โ how many times it errored out. ๐name(orsyscallorsyscall-name) โ the system call's name.
The default value is time-percent,total-time,avg-time,calls,errors,name. And heads up: if you don't explicitly supply the name field, it just gets tacked on as the last column anyway. The strace girlies will not let you forget the name. ๐
-w,--summary-wall-clockโ summarizes the wall-clock time for each system call, measured from when it started to when it ended. The default, for reference, is to summarize the system time. โฑ๏ธ
๐ Tampering ๐
okay so this is the part where you start messing with syscalls directly, and it's giving mad scientist energy fr fr ๐งช
--inject=syscall_set[:error=errno|:retval=value][:signal=sig][:syscall=syscall][:delay_enter=delay][:delay_exit=delay][:poke_enter=@argN=DATAN,@argM=DATAM...][:poke_exit=@argN=DATAN,@argM=DATAM...][:when=expr]โ performs system call tampering for the specified set of system calls.
The syntax of the syscall_set specification is the same as in the --trace option.
At least one of error, retval, signal, delay_enter, delay_exit, poke_enter, or poke_exit action options must be specified. And error and retval are mutually exclusive โ you literally cannot pick both, bestie, the vibe check forbids it. ๐ซ
- If the
error=errnooption is specified, a fault gets injected into the system call. This is done by replacing the system call number with-1(which represents an invalid system call) and setting the error code to the specifiederrno. This behavior of replacing the syscall number with-1can be overridden using thesyscall=option. Theerrnocan be a symbolic name likeENOSYS, or a numeric value in the range1..4095. - If the
retval=valueoption is specified, a success value gets injected instead. The system call number is replaced the same way as with theerror=option, but instead of an error, the specified success value gets returned to the caller process. โจ - If the
signal=sigoption is specified with either a symbolic value likeSIGSEGVor a numeric value within the1..SIGRTMAXrange, that signal is delivered on entering every system call specified by thesyscall_set. ๐จ - If the
delay_enter=delayordelay_exit=delayoptions are specified, delay injection is performed: the tracee gets delayed by the time period specified bydelayon entering or exiting the system call, respectively. The format of the delay specification is described in section Time specification format description. ๐ - If the
poke_enter=@argN=DATAN,@argM=DATAM...orpoke_exit=@argN=DATAN,@argM=DATAM...options are specified, the tracee's memory at the locations pointed to by system call argumentsargNandargM(going fromarg1toarg7) is overwritten by the dataDATANandDATAM(specified in hexadecimal format; for examplepoke_enter=@arg1=0000DEAD0000BEEF). Thepoke_enteroption modifies memory on system call enter, whilepoke_exitdoes it on system call exit. ๐ช
The injection actions are independent. For example, specifying only signal= delivers a signal without altering the system call's outcome or delaying it. Similarly, specifying only error= injects a system call fault without adding a signal or delay.
If the signal=sig option is specified together with error=errno or retval=value, then both the injection of a fault or success and the signal delivery are performed.
If the syscall=syscall option is specified, the given syscall is injected instead of the default -1. The specified syscall must have no side effects; currently, only system calls from the %pure set are supported.
Unless the when=expr subexpression is specified, an injection is being made into every invocation of each system call from the syscall_set.
The format of the subexpression is:
first[..last][+[step]]
The number first stands for the first invocation number in the range, the number last stands for the last invocation number in the range, and step stands for the step between two consecutive invocations. The following combinations are useful:
firstโ injects into invocation numberfirstonly, for each system call in thesyscall_set.first..lastโ injects into invocations fromfirstthroughlast(inclusive), for each system call in thesyscall_set.first+โ injects into every invocation, starting with numberfirst, for each system call in thesyscall_set.first+stepโ injects into invocations numberfirst,first+step,first+step+step, and so on, for each system call in thesyscall_set.first..last+stepโ same as the previous one, but only considers invocations with numbers up tolast(inclusive).
For example, to fail each third and subsequent chdir system calls with ENOENT, use --inject=chdir:error=ENOENT:when=3+. ๐ฏ
The valid range for the numbers first and step is 1..65535, and for the number last it's 1..65534.
An injection expression can contain at most one fault or return value specification (i.e., either error= or retval=) and at most one signal= specification. If an injection expression contains multiple when= specifications, the last one takes precedence.
Accounting of system calls that are subject to injection is done per system call and per tracee.
Specification of system call injection can be combined with other system call filtering options โ for example, -P /dev/urandom --inject=file:error=ENOENT. ๐ซถ
-e inject=argsโ this is equivalent to--inject=args. Same thing, shorter keyboard journey. ๐--fault=syscall_set[:error=errno][:when=expr]โ performs system call fault injection for the specified set of system calls.
This is a shortcut for the more general --inject= option, using a default errno of ENOSYS.
-e fault=argsโ this is equivalent to--fault=args. โจ
๐ Miscellaneous ๐
the grab bag section, the little extras, the confetti on top ๐
-d,--debugโ shows some debugging output of strace itself on the standard error. Yes bestie, strace can be introspective too. ๐-Fโ this option is deprecated. It is retained for backward compatibility only and may be removed in future releases. Using multiple-Foptions is equivalent to a single-f. This option is ignored entirely if used in conjunction with one or more-foptions. ๐ป-h,--helpโ prints the help summary. Classic. ๐ซถ--seccomp-bpfโ attempts to use seccomp-bpf (seeseccomp(2)) to cause the kernel to stop the tracee only for the system calls that are being traced.
This option has no effect unless -f/--follow-forks is also specified. --seccomp-bpf is not compatible with --syscall-limit and -b/--detach-on options. It is also not applicable to processes attached using the -p/--attach option.
An attempt to enable system calls filtering using seccomp-bpf may fail for various reasons โ e.g. there are too many system calls to filter, the seccomp API is not available, or strace itself is being traced. If the seccomp-bpf filter setup fails, strace just proceeds as usual, stopping traced processes on every system call. No drama, it just keeps going. ๐
When --seccomp-bpf is activated and the -p/--attach option is not used, the --kill-on-exit option is activated as well. ๐
๐จ bestie listen up, this one's lowkey important: in cases where the tracee has another seccomp filter that returns an action value with a precedence greater thanSECCOMP_RET_TRACE,strace --seccomp-bpfwill not be notified. Like, the tea is: if another seccomp filter, say, disables the system call or just straight-up kills the tracee, thenstrace --seccomp-bpfwill not be aware of that system call invocation at all. Not even a little. ๐
--tips[=[[id:]id],[[format:]format]]โ shows strace tips, tricks, and tweaks before exit. ๐ Theidcan be a non-negative integer to print one specific tip (note: these IDs are not guaranteed to be stable, so don't get attached). It can also berandom(which is the default), and then you just get a random tip, like a little fortune cookie but for syscalls ๐.formatcan be one of these:noneโ no tip is printed. Can be used to override the previous setting.compactโ prints the tip just big enough to contain all the text.fullโ prints the tip in its full glory. ๐
Default is id:random,format:compact.
-V,--versionโ prints the version number of strace and the list of enabled optional features. Multiple instances of this option beyond specific threshold tend to increase der Strauss awareness. ๐
โฐ Time specification format description โฐ
Time values are specified as a decimal floating point number (in a format accepted by strtod(3)), optionally followed by a suffix to indicate the unit of time: s (seconds), ms (milliseconds), us (microseconds), or ns (nanoseconds). If no suffix is specified, the value defaults to microseconds. ๐ซถ
The described format is used for -O, --inject=delay_enter, and --inject=delay_exit options.
๐ DIAGNOSTICS ๐
When command exits, strace exits with the same exit status. If command is terminated by a signal, strace terminates itself with the same signal, so that strace can be used as a wrapper process transparent to the invoking parent process. Honestly? Iconic behaviour. โจ
โ ๏ธ Note though: the parent-child relationship (signal stop notifications, thegetppid(2)value, etc) between the traced process and its parent is not preserved unless-Dis used. So like, the vibes are preserved but the family tree is not. ๐ญ
When using -p without a command, the exit status of strace is zero unless no processes have been attached or an unexpected error occurred during tracing.
๐ SETUID INSTALLATION ๐
If strace is installed setuid to root, then the invoking user will be able to attach to and trace processes owned by any user. In addition, setuid and setgid programs will be executed and traced with the correct effective privileges. Big power. ๐
Since these capabilities should only be granted to users with full root privileges, installing strace as setuid to root is only appropriate when its use is restricted to such trusted users. For example, a special version of strace could be installed with mode rwsr-x---, user root, and group trace. In this configuration, only trusted users who are members of the trace group could execute it.
๐ If you use this feature, remember to also install a regular, non-setuid version of strace for ordinary users. The trusted besties get the special one, everyone else gets the normal one, and that's the whole point.
๐ฆ MULTIPLE PERSONALITIES SUPPORT ๐ฆ
On some architectures, strace can decode system calls for processes that use a different Application Binary Interface (ABI) from the one strace uses. Specifically, in addition to decoding native ABI, strace can decode the following ABIs on the following architectures:
x86_64โi386,x32[1];i386[2]AArch64โARM 32-bit EABIPowerPC 64-bit[3] โPowerPC 32-bits390xโs390SPARC 64-bitโSPARC 32-bitTILE 64-bitโTILE 32-bit
[1] When strace is built as an x86_64 application [2] When strace is built as an x32 application [3] Big endian only
This support is optional and depends on the ability to generate and parse structure definitions at build time. Refer to the output of the strace -V command to determine which ABIs are supported by your strace build. In this context, "non-native" refers to an ABI that differs from the one strace is using:
m32-mpersโ strace can trace and properly decode non-native 32-bit binaries.no-m32-mpersโ strace can trace, but cannot properly decode non-native 32-bit binaries. ๐mx32-mpersโ strace can trace and properly decode non-native 32-on-64-bit binaries.no-mx32-mpersโ strace can trace, but cannot properly decode non-native 32-on-64-bit binaries. ๐ญ
If the output contains neither m32-mpers nor no-m32-mpers, it means that support for decoding non-native 32-bit binaries is not applicable to the architecture.
Likewise, if the output contains neither mx32-mpers nor no-mx32-mpers, it means that support for decoding non-native 32-on-64-bit binaries is not applicable to the architecture.
๐ NOTES ๐
Systems that use shared libraries often produce a large amount of tracing output when loading them. So yeah, brace yourself, the wall of text is coming and it's not personal. ๐ซถ
It is instructive to think about system call inputs and outputs as data-flow across the user/kernel boundary. Because user-space and kernel-space are separate and address-protected, it is sometimes possible to make deductive inferences about process behavior using inputs and outputs as propositions. Detective mode: activated ๐ต๏ธ
In some cases, a system call will differ from the documented behavior or have a different name. For example, the underlying faccessat(2) system call does not have a flags argument, and the setrlimit(2) library function is implemented using prlimit64(2) system call on modern (2.6.38+) kernels. These discrepancies are normal characteristics of the system call interface and are handled by C library wrapper functions.
Some system calls have different names in different architectures and personalities. In these cases, system call filtering and printing uses the names that match corresponding __NR_* kernel macros of the tracee's architecture and personality. There are two exceptions from this general rule: arm_fadvise64_64(2) ARM system call and xtensa_fadvise64_64(2) Xtensa system call are filtered and printed as fadvise64_64(2).
On the x32 ABI, some system calls are intended for 64-bit processes but can be invoked from x32 by setting the __X32_SYSCALL_BIT flag. When this occurs, strace designates these calls with a #64 suffix. An example is readv(2), which is syscall number 19 on x86_64, whereas its distinct x32 counterpart is syscall number 515.
On some platforms, a process attached with the -p option may receive a spurious EINTR error from a non-restartable system call. This can have an unpredictable effect on the process if it does not attempt to restart the call. Ideally, all system calls should be restarted on strace attach, making the attach invisible to the traced process, but a few system calls aren't. Arguably, every instance of such behavior is a kernel bug. ๐
Since strace executes the specified command directly without a shell, scripts that lack a shebang line (e.g., #!/bin/sh) will fail with an ENOEXEC error, even if a shell could run them correctly. It is advisable to manually supply a shell as a command with the script as its argument.
๐ BUGS ๐
Programs that use the setuid bit do not have effective user ID privileges while being traced.
A traced process runs more slowly than a non-traced one. The performance impact can be mitigated by using the --seccomp-bpf option.
When tracing a command, its descendant processes may be left running after strace is terminated by an interrupt signal (such as CTRL-C). This can be prevented by using the --kill-on-exit option, or by using --seccomp-bpf option in a way that implies --kill-on-exit.
A traced process can use the CLONE_UNTRACED flag with the clone system call to create a child process that is not traced by strace. This breaks a guarantee of the --seccomp-bpf option, as this untraced child may be left with an active seccomp filter after strace terminates. ๐ฉ
๐ HISTORY ๐
okay so storytime bestie, because strace has LORE ๐ซถ
The original strace was written by Paul Kranenburg for SunOS, and it was inspired by its trace utility. The SunOS version of strace got ported to Linux and glow-up'd by Branko Lankester, who also wrote the Linux kernel support. So even though Paul released strace 2.5 in 1992, Branko's work was based on Paul's strace 1.5 release from 1991. baby strace era ๐
Then in 1993, Rick Sladkey took on the project โ main character move honestly ๐ He merged strace 2.5 for SunOS with the second release of strace for Linux, added many features from SVR4's truss(1), and produced a version of strace that worked on both platforms. In 1994, Rick ported strace to SVR4 and Solaris and wrote the automatic configuration support โจ. In 1995 he ported strace to Irix (and became tired of writing about himself in the third person) ๐ญ๐ญ iconic.
Beginning with 1996, strace was maintained by Wichert Akkerman. During his tenure, strace development migrated to CVS; and ports to FreeBSD and many architectures on Linux (including ARM, IA-64, MIPS, PA-RISC, PowerPC, s390, SPARC) were introduced. so many platforms bestie, it's giving multiverse ๐
In 2002, responsibility for strace maintenance was transferred to Roland McGrath. Since then, strace gained support for several new Linux architectures (AMD64, s390x, SuperH), bi-architecture support for some of them, and received numerous additions and improvements in system calls decoders on Linux; and strace development migrated to Git during that period ๐ง
Since 2009, strace has been actively maintained by Dmitry Levin. During this period, strace has gained support for the AArch64, ARC, AVR32, Blackfin, C-SKY, LoongArch, Meta, Nios II, OpenRISC 1000, RISC-V, Tile/TileGx, and Xtensa architectures. In 2012, unmaintained and apparently broken support for non-Linux operating systems was removed ๐๏ธ (rip but also slay). Also, in 2012 strace gained support for path tracing and file descriptor path decoding. In 2014, support for stack trace printing was added. In 2016, system call tampering was implemented ๐
For the additional information, please refer to the NEWS file and strace repository commit log. the receipts are all there fr fr ๐
๐ REPORTING BUGS ๐
Found a problem with strace? bestie don't suffer in silence ๐ค Problems with strace should be reported to the strace mailing list ๐ฌ
๐ SEE ALSO ๐
strace-log-merge(1)ltrace(1)perf-trace(1)trace-cmd(1)time(1)ptrace(2)seccomp(2)syscall(2)proc(5)signal(7)
also, the strace Home Page exists and is out there living its best life ๐ธ
๐ AUTHORS ๐
The complete list of strace contributors can be found in the CREDITS file. so many legends, we love a collaborative queen ๐
and that's the whole page bestie, you read all of it, I'm so proud of you ๐ญ๐ซถ
this beauty is strace 7.3, dated 2026-06-25, and yes it's still STRACE(1) โจ go forth and trace responsibly ๐
๐๐ฆ
๐ฆ where this page comes from
- fedora 43: strace 7.3-1.fc43 ยท LGPL-2.1-or-later AND GPL-2.0-or-later AND GPL-3.0-or-later AND BSD-2-Clause AND BSD-3-Clause AND (GPL-1.0-or-later WITH Linux-syscall-note) AND (GPL-2.0-or-later WITH Linux-syscall-note) AND (GPL-2.0-only WITH Linux-syscall-note) AND (LGPL-2.0-or-later WITH Linux-syscall-note) AND (LGPL-2.1-or-later WITH Linux-syscall-note) AND ((GPL-2.0-only WITH Linux-syscall-note) OR Linux-OpenIB) AND ((GPL-2.0-only WITH Linux-syscall-note) OR MIT) AND ((GPL-2.0-or-later WITH Linux-syscall-note) OR BSD-3-Clause) AND ISC AND X11 AND FSFAP AND FSFUL AND FSFULLR AND FSFULLRWD
- fedora 44: strace 7.3-1.fc44 ยท LGPL-2.1-or-later AND GPL-2.0-or-later AND GPL-3.0-or-later AND BSD-2-Clause AND BSD-3-Clause AND (GPL-1.0-or-later WITH Linux-syscall-note) AND (GPL-2.0-or-later WITH Linux-syscall-note) AND (GPL-2.0-only WITH Linux-syscall-note) AND (LGPL-2.0-or-later WITH Linux-syscall-note) AND (LGPL-2.1-or-later WITH Linux-syscall-note) AND ((GPL-2.0-only WITH Linux-syscall-note) OR Linux-OpenIB) AND ((GPL-2.0-only WITH Linux-syscall-note) OR MIT) AND ((GPL-2.0-or-later WITH Linux-syscall-note) OR BSD-3-Clause) AND ISC AND X11 AND FSFAP AND FSFUL AND FSFULLR AND FSFULLRWD
- ๐ the page's license: LGPL-2.1-or-later
- Copyright (c) 1991, 1992 Paul Kranenburg <pk@cs.few.eur.nl>
- Copyright (c) 1993 Branko Lankester <branko@hacktic.nl>
- Copyright (c) 1993, 1994, 1995, 1996 Rick Sladkey <jrs@world.std.com>
- Copyright (c) 1996-2025 The strace developers
- ๐ girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms