ttb(3)

📦
version 29.0.6 · 2 versions
, in nixos 26.05, nixpkgs unstable, ubuntu stonking

🦋 ttb — the Trace Tool Builder (manual page ttb(3), Erlang Module Definition) 🦋

✨ NAME ✨

ttb — a base for building trace tools for distributed systems. bestie it is literally the blueprint for tracing across a whole distributed system, okay? 💅

🌸 DESCRIPTION 🌸

The Trace Tool Builder, ttb, is a base for building trace tools for distributed systems. that's it, that's the whole vibe. she's a foundation, not a finished product, and she knows it. ✨

⚠️ real talk, listen up: when you're using ttb, do NOT use module dbg in application Runtime_Tools in parallel. like. don't. they will not play nice and it's giving chaos. 🙈

🎀 DATA TYPES 🎀

okay so before we get into the functions, here's the whole cast of characters. learn their faces, you'll see them everywhere. 💖

format_fun() ::
          fun((Fd :: standard_io | file:fd(),
               Trace :: tuple(),
               TraceInfo :: [{atom(), list()}],
               State :: term()) ->
                  NewState :: term()).

that's the custom formatter signature — you get the fd, the trace, the trace info, and your state, and you hand back a new state. main character energy fr. 👑

format_handler() :: {format_fun(), InitialState :: term()}.

a format handler is just that fun plus whatever state you wanna start it with. cute, simple. 🎀

format_opt() ::
          {out, standard_io | file:filename()} |
          {handler, format_handler()} |
          disable_sort.
format_opts() :: format_opt() | [format_opt()].

one format option, or a whole lil list of them. she's flexible like that. 💅

item() ::
          pid() |
          port() |
          atom() |
          {global, term()} |
          all | processes | ports | existing | existing_processes |
          existing_ports | new | new_processes | new_ports.

the "who are we even tracing" type. so many options, so little time. 🦋

match_desc() ::
          [{matched, node(), integer()} |
           {matched, node(), 0, term()} |
           {saved, integer()}].
match_spec() :: pos_integer() | x | c | cx | [] | dbg:match_spec().
mfas() :: {Module :: atom(), Function :: atom(), [term()]}.

module, function, args. you know her. 🌸

nodes() :: node() | [node()] | all.

one node, a list of nodes, or everybody. no cap. 🌈

stop_opt() ::
          nofetch |
          {fetch_dir, file:filename()} |
          format |
          {format, format_opts()} |
          return_fetch_dir.
stop_opts() :: stop_opt() | [stop_opt()].
tp_arity() :: arity() | '_'.
tp_function() :: atom() | '_'.
tp_module() :: module() | '_'.

the '_' is the wildcard, iykyk. ⭐

trace_flag() ::
          s | r | m | c | p | sos | sol | sofs | all | clear | send |
          'receive' | procs | ports | call | arity | return_to |
          silent | running | exiting | running_procs | running_ports |
          garbage_collection | timestamp | cpu_timestamp |
          monotonic_timestamp | strict_monotonic_timestamp |
          set_on_spawn | set_on_first_spawn | set_on_link |
          set_on_first_link |
          {tracer, pid() | port()} |
          {tracer, module(), term()}.

the full menu of trace flags. yes, all of them. we don't skip. 🔥

👑 FUNCTIONS 👑

💜 format(Files, Options) -> ok | {error, term()}

format(Files, Options) -> ok | {error, term()}
                when
                    Files :: [file:filename()] | file:filename(),
                    Options :: format_opts().

Reads the specified binary trace log(s). 📖 the logs get processed in the order of their time stamps, as long as option disable_sort is not specified. so if you want them in time order, don't touch that flag, bestie. ⏰

  • If FormatHandler = {Function,InitialState}, then Function is called for each trace message. every single one. she's thorough. 💅
  • If FormatHandler = get_et_handler(), then et_viewer in application ET is used for presenting the trace log graphically. 🎨 and ttb provides a few different filters that can be selected from menu Filters and scaling in the et_viewer.
  • If FormatHandler is not specified, a default handler is used, presenting each trace message as a text line. plain, simple, still slaying. ✨

The state returned from each call of Function is passed to the next call — even if the next call is to format a message from another log file. the state follows you around like a loyal little ghost. 👻

If Out is specified, FormatHandler gets the file descriptor to Out as the first parameter.

🙈 Out is ignored if the et format handler is used. like, she said no thanks to your file descriptor.

Wrap logs can be formatted one by one or all at once. 🎁

  • To format one of the wrap logs in a set, specify the exact file name.
  • To format the whole set of wrap logs, specify the name with * instead of the wrap count.

For examples, see the User's Guide. 📚

🦋 get_et_handler() -> {Fun, InitState}

get_et_handler() -> {Fun, InitState}
                        when Fun :: fun(), InitState :: term().

Returns the et handler, which can be used with format/2 or tracer/2. she's the plug-and-play graphical one. 🎀

Example: ttb:format(Dir, [{handler, ttb:get_et_handler()}]).

🌸 list_config(ConfigFile) -> Result

list_config(ConfigFile) -> Result
                     when
                         ConfigFile :: file:filename(),
                         Result :: Config | {error, term()},
                         Config :: [{integer(), mfas()}].

Lists all entries in the specified configuration file. no skipping, no summarizing, we list them all. 📋

⭐ list_history() -> History | {error, term()}

list_history() -> History | {error, term()}
                      when
                          History ::
                              [{N :: integer(),
                                Func :: function(),
                                Args :: integer()}].

Every single call to ttb gets stored in the history. this function returns the current content of that history. 🕰️ any entry can be reexecuted with run_history/1, or stored in a configuration file with write_config/2,3. it's giving "replay your greatest hits." 🔁

💅 p(Item, Flags) -> Result

p(Item, Flags) -> Result
           when
               Item :: item(),
               Flags :: trace_flag() | [trace_flag()],
               Result :: {ok, [{item(), match_desc()}]}.

Sets the specified trace flags on the specified processes or ports. 🎯 Flag timestamp is always turned on — you don't get a say, she's non-negotiable like that. ⏰

See the Reference Manual for module dbg for the possible trace flags. Parameter MatchDesc is the same as returned from dbg:p/2.

Processes can be specified as registered names, globally registered names, or process identifiers. Ports can be specified as registered names or port identifiers. If a registered name is specified, the flags are set on processes/ports with this name on all active nodes. 🌐

Issuing this command starts the timer for this trace if option timer is specified with tracer/2. ⏱️

🎀 run_config(ConfigFile) -> Result

run_config(ConfigFile) -> Result
                    when
                        ConfigFile :: file:filename(),
                        Result :: ok | {error, term()}.

Executes all entries in the specified configuration file. 💫

psst — notice that the history of the last trace is always available in file ttb_last_config. she's always there for you.

🌈 run_config(ConfigFile, NumList) -> Result

run_config(ConfigFile, NumList) -> Result
                    when
                        ConfigFile :: file:filename(),
                        NumList :: [integer()],
                        Result :: ok | {error, term()}.

Executes selected entries from the specified configuration file. NumList is a list of integers pointing out the entries to be executed. so you get to pick and choose, like a lil playlist. 🎧

To list the contents of a configuration file, use list_config/1.

and again, reminder reminder: the history of the last trace is always available in file ttb_last_config. 💖

🦋 run_history(Entries) -> ok | {error, term()}

run_history(Entries) -> ok | {error, term()}
                     when
                         Entries :: [Entry] | Entry | all | all_silent,
                         Entry :: integer().

Executes the specified entry or entries from the history list. to list history, use list_history/0. 📜

🔥 seq_trigger_ms(Flags) -> match_spec()

seq_trigger_ms(Flags) -> match_spec()
                        when
                            Flags :: all | SeqTraceFlag | [SeqTraceFlag],
                            SeqTraceFlag :: atom().

A match specification can turn on or off sequential tracing. this function returns a match specification which turns on sequential tracing with the specified Flags. ✨

This match specification can be specified as the last argument to tp or tpl. The activated Item then becomes a trigger for sequential tracing. That means: if the item is called on a process with trace flag call set, the process is "contaminated" with token seq_trace. 🦠 (in a cute way. a slay way.)

If Flags = all, all possible flags are set. go big or go home. 💅

The possible values for SeqTraceFlag are available in seq_trace.

For a description of the match_spec() syntax, see section Match Specifications in Erlang in ERTS, which explains the general match specification "language". 📖

📌 NOTE: the system tracer for sequential tracing is automatically initiated by ttb when a trace port is started with ttb:tracer/0,1,2.

An example of how to use function seq_trigger_ms/0,1 follows:

(tiger@durin)5> ttb:tracer().
{ok,[tiger@durin]}
(tiger@durin)6> ttb:p(all,call).
{ok,{[all],[call]}}
(tiger@durin)7> ttb:tp(mod,func,ttb:seq_trigger_ms()).
{ok,[{matched,1},{saved,1}]}
(tiger@durin)8>

Whenever mod:func(...) is called after this, token seq_trace is set on the executing process. 👑

💖 start_trace(Nodes, Patterns, FlagSpec, TracerOpts) -> Result

start_trace(Nodes, Patterns, FlagSpec, TracerOpts) -> Result
                     when
                         Nodes :: nodes(),
                         Patterns :: [tuple()],
                         FlagSpec ::
                             {item(), trace_flag() | [trace_flag()]},
                         TracerOpts :: term(),
                         Result :: {ok, [{item(), match_desc()}]}.

This function is a shortcut allowing you to start a trace with one command. one. command. periodt. 💅 each tuple in Patterns is converted to a list, which in turn is passed to ttb:tpl/2,3,4.

The call:

> ttb:start_trace([Node, OtherNode],
                  [{mod, foo, []}, {mod, bar, 2}],
                  {all, call},
                  [{file, File}, {handler,{fun myhandler/4, S}}]).

is equivalent to:

> ttb:start_trace([Node, OtherNode],
                  [{file, File}, {handler,{fun myhandler/4, S}}]),
ttb:tpl(mod, foo, []),
ttb:tpl(mod, bar, 2, []),
ttb:p(all, call).

same energy, fewer lines. you're welcome. 🫶

🌸 stop() -> stopped | {stopped, Dir :: file:filename()}

Equivalent to stop([]). that's it. short queen. 👑

🎀 stop(Opts :: stop_opts()) -> stopped | {stopped, Dir :: file:filename()}

Stops tracing on all nodes. 🛑

Logs and trace information files are sent to the trace control node and stored in a directory named ttb_upload_FileName-Timestamp, where Filename is the one provided with {file, File} during trace setup and Timestamp is of the form yyyymmdd-hhmmss. 🗂️

Even logs from nodes on the same machine as the trace control node are moved to this directory. yes, even those. no exceptions, bestie.

The history list is saved to a file named ttb_last_config for further reference — as it is no longer accessible through history and configuration management functions, like ttb:list_history/0. 😭 so save the receipts while you can.

Options:

  • nofetch — Indicates that trace logs are not to be collected after tracing is stopped. like, leave them where they are, we're done here. 🙅
  • {fetch, Dir} — Allows specification of the directory to fetch the data to. If the directory already exists, an error is thrown. ⚠️ so don't reuse dirs, okay? she will yell at you.

to be continued… 🦋

  • format — says bestie, format the trace logs after tracing stops ✨. Every single log hanging out in the fetch directory gets merged together, all of them.
  • return_fetch_dir — makes the return value come back as {stopped, Dir} instead of just plain stopped. Heads up: this one implies fetch.
tpl(tp_module(), tp_function(), tp_arity(), match_spec()) ->
             {ok, match_desc()} | {error, term()}.

okay so these functions are the ones you pair with the trace flags call, send, and 'receive' to set and clear trace patterns. 🎀

When the trace flag call is set on a process, function calls get traced on that process if a trace pattern is set for the function being called.

The send and 'receive' flags turn on tracing of every message sent and received by the process/port. Trace patterns set with tpe can limit which messages get traced, based on the message content, the sender, and/or the receiver.

Trace patterns specify how to trace a function or a message by using match specifications. Match specifications are described in the ERTS User's Guide. 📖

These functions are equivalent to the corresponding functions in module dbg, but all calls are stored in the history. The history buffer makes it easy to create configuration files — like, you can set up the same trace environment over and over, for example to compare two test runs. It also means less typing when you're using ttb from the Erlang shell. 👑

  • tp Sets trace patterns on global function calls.
  • tpl Sets trace patterns on local and global function calls.
  • tpe Sets trace patterns on messages.
  • ctp Clears trace patterns on local and global function calls.
  • ctpl Clears trace patterns on local function calls.
  • ctpg Clears trace patterns on global function calls.
  • ctpe Clears trace patterns on messages.

With tp and tpl, you can use one of the match specification shortcuts (for example, ttb:tp(foo_module, caller)).

The shortcuts are as follows: 💅

  • return for [{'_',[],[{return_trace}]}] (report the return value from a traced function)
  • caller for [{'_',[],[{message,{caller}}]}] (report the calling function)
  • {codestr, Str} for dbg:fun2ms/1 arguments passed as strings (example: "fun(_) -> return_trace() end")
tracer(shell | dbg | nodes()) -> {ok, [node()]} | {error, term()}.

Handy shortcuts for common tracing settings, no cap. 🫶

  • shell is equivalent to tracer(node(),[{file, {local, "ttb"}}, shell]).
  • dbg is equivalent to tracer(node(),[{shell, only}]).
  • Nodes is equivalent to tracer(Nodes,[]).
tracer(Nodes, Opts) -> Result
                when
                    Nodes :: nodes(),
                    Opts :: Opt | [Opt],
                    Opt ::
                        {file, Client} |
                        {handler, format_handler()} |
                        {process_info, boolean()} |
                        shell |
                        {shell, ShellSpec} |
                        {timer, TimerSpec} |
                        {overload_check, {MSec, Module, Function}} |
                        {flush, MSec} |
                        resume |
                        {resume, MSec} |
                        {queue_size, non_neg_integer()},
                    TimerSpec :: MSec | {MSec, stop_opts()},
                    MSec :: timer:time(),
                    Module :: atom(),
                    Function :: atom(),
                    Client :: File | {local, File},
                    File :: file:filename() | Wrap,
                    Wrap ::
                        {wrap, file:filename()} |
                        {wrap,
                         file:filename(),
                         Size :: integer(),
                         Count :: integer()},
                    ShellSpec :: true | false | only,
                    Result :: {ok, [node()]} | {error, term()}.

Starts a file trace port on all the specified nodes and points the system tracer for sequential tracing at the very same port. 🌈

Options, let's go:

  • Filename The specified Filename gets prefixed with the node name. Default Filename is ttb.
  • File={wrap,Filename,Size,Count} Use this one if the size of the trace logs has to be limited. Default values are Size=128*1024 and Count=8.
  • Client When you're tracing diskless nodes, ttb has to be started from an external "trace control node" that has disk access, and Client must be {local, File}. All trace information then gets sent to the trace control node where it's written to file.
  • queue_size When tracing to shell or {local,File}, an ip trace driver is used internally. The ip trace driver has a queue of maximum QueueSize messages waiting to be delivered. If the driver can't deliver messages as fast as they're produced, the queue size might be exceeded and messages get dropped. This parameter is optional, and it's only useful if lots of {drop,N} trace messages are being received by the trace handler. It has no meaning if shell or {local,File} isn't used. See dbg:trace_port/2 for more info about the ip trace driver.
  • process_info Says whether process information is to be collected. If PI = true (which is the default), each process identifier Pid gets replaced by a tuple {Pid,ProcessInfo,Node}, where ProcessInfo is the registered process name, its globally registered name, or its initial function. To turn this off, set PI = false.
  • {shell, ShellSpec} Says trace messages are to be printed on the console as they're received by the tracing process. This implies trace client {local, File}. If ShellSpec is only (instead of true), no trace logs are stored. 💖
  • shell Shortcut for {shell, true}.
  • timer Says the trace should be automatically stopped after MSec milliseconds. StopOpts are passed to command ttb:stop/1 if specified (default is []). Notice the timing is approximate, since delays from network communication are always hanging around. The timer starts after ttb:p/2 is issued, so you can set up your trace patterns before it kicks in.
  • overload_check Enables overload checking on the nodes under trace. Module:Function(check) is performed each MSec millisecond. If the check returns true, tracing gets disabled on a specified node.

Module:Function must be able to handle at least three atoms: init, check, and stop. init and stop let you initialize and clean up the check environment.

When a node gets overloaded, it's not possible to issue ttb:p/2 or any command from the ttb:tp/2,3,4 family — because that would lead to inconsistent tracing state (different trace specifications on different nodes). 😭

  • flush Periodically flushes all file trace port clients (see dbg:flush_trace_port/1). When enabled, the buffers are freed each MSec millisecond. This option is not allowed with {file, {local, File}} tracing.
  • {resume, FetchTimeout} Enables the autoresume feature. When enabled, remote nodes try to reconnect to the controlling node if they get restarted. The feature requires application Runtime_Tools to be started (so it has to be present in the .boot scripts if the traced nodes run with embedded Erlang). If that's not possible, resume can be performed manually by starting Runtime_Tools remotely using rpc:call/4.

ttb tries to fetch all logs from a reconnecting node before reinitializing the trace. This must finish within FetchTimeout milliseconds or it gets aborted.

By default, autostart information is stored in a file named ttb_autostart.bin on each node. If that's not what you want (say, on diskless nodes), a custom module handling autostart information storage and retrieval can be provided by specifying environment variable ttb_autostart_module for the application Runtime_Tools. The module must respond to the following API:

  • write_config(Data) -> ok Stores the provided data for further retrieval. It's important to realize that the data storage used must not be affected by the node crash. 💀
  • read_config() -> {ok, Data} | {error, Error} Retrieves configuration stored with write_config(Data).
  • delete_config() -> ok Deletes configuration stored with write_config(Data). Notice that after this call, any subsequent calls to read_config must return {error, Error}.

resume implies the default FetchTimeout, which is 10 seconds. ⏱️

write_config(ConfigFile, Config, Opts) -> Result
                      when
                          ConfigFile :: file:filename(),
                          Config :: all | [integer()] | [mfas()],
                          Opts :: Opt | [Opt],
                          Opt :: append,
                          Result :: ok | {error, term()}.

Creates or extends a configuration file, which you can use later to restore a specific configuration. ✨

The contents of the configuration file can either be fetched from the history or specified directly as a list of {Mod,Func,Args}.

If the complete history is to be stored in the configuration file, Config must be all. If only a selected number of entries from the history are to be stored, Config must be a list of integers pointing out the entries to be stored.

If Opts isn't specified, or if it's [], ConfigFile gets deleted and a new file is created. If Opts = [append], ConfigFile is not deleted — instead the new information gets appended at the end of the file.

write_trace_info(Key :: term(), Info) -> ok
                                 when
                                     Info ::
                                         Data ::
                                             term() | fun(() -> Data ::
term()).

okay bestie, last lil bite of this page, and it's giving 🎀 data-file lore 🎀

File .ti contains {Key,ValueList} tuples. This function adds Data to the ValueList associated with Key. And the important part, the part that ATE 💅 — all information written with this function is included in the call to the format handler. like, every single crumb you wrote gets passed along, nothing gets left behind, no cap 🫶✨

and that's the whole page, bestie, we did it 😭💖 this one comes to you from Ericsson AB, version observer 2.19, and it lives in the ttb(3) corner of the manual, fr fr. go write some traces and slay 👑🌸

bye babe, mwah 💋

📦 where this page comes from