LAMTRACE(1)

✨ NAME ✨

lamtrace — unload LAM trace data, that's it that's the whole job bestie 💅

---

🎀 SYNOPSIS 🎀

lamtrace [-hkvR] [-mpi] [-l listno] [-f #secs] [filename] [nodes]
       [processes]

---

🌸 OPTIONS 🌸

• -h — prints out the useful info about this command, so you don't have to guess 🫶
• -k — copy the trace data and do not remove it. keep it, keep it, keep it 💖
• -v — be verbose. yes queen, tell me everything 👑
• -R — delete all trace data from the nodes you specified. not unloading, deleting. big girl move 💀
• -l — unload only from the given list number.
• -mpi — unload trace data for an MPI application. special mode, we love her ✨
• -f #secs — signal the target processes to flush their trace data to the daemon, then wait #secs before actually unloading. patience is a virtue fr fr
• filename — put the trace data into this file (default: def.lamtr)

---

🦋 DESCRIPTION 🦋

okay so. the -t option of mpirun(1) and loadgo(1) is what lets an application generate execution traces in the first place. those traces get stored in a buffer inside each application process first. when the buffer fills up, and when the application terminates, that runtime buffer gets flushed to the trace daemon — which is a structural component living inside the LAM daemon, iconic architecture honestly 💅. the trace daemon will also collect data up to a pre-compiled limit. past that limit? the oldest traces get forgotten in favor of the newer ones. sorry to the old traces, they had their moment 🙈

so after an application finishes, the record of its execution lives in the trace daemons of each node that was running the application. the lamtrace command is how you retrieve those traces and store them all in one file, so a performance visualization tool like xmpi(1) can display them. and if the application was started by xmpi(1)? then you don't normally need lamtrace at all, because the equivalent functionality is invoked with a button. a whole button. slay 🎀

you can unload incomplete trace data while the application is still running, which is very main character of you. but heads up: the output file must not exist prior to invoking lamtrace. this is exactly the good situation to use the -k option, which preserves the trace daemon's contents after unloading. that way each reload gets the entire run's trace data up to the present time. obsessed 🌈

now, a running process is likely holding the most recent trace data in an internal buffer. there's a standard LAM signal, LAM_SIGTRACE (see doom(1)), that causes trace enabled processes to flush that internal trace buffer to the daemon. the -f option tells lamtrace to send this signal to all target processes before unloading trace data. but lowkey a race condition develops between the target process storing trace data to the daemon and the unloading procedure. and the problem is foisted upon you, the user, who gives a delay parameter after -f. not me having to babysit a race condition 😭

trace data are organized by node, process identifier and list number. a process can store traces on any node, although the local node is the obvious, least intrusive choice. the process can identify itself in any meaningful way (getpid(2) is a good idea, ngl). the list number is also chosen by the process. these values may be set by an instrumented library, such as libmpi(3), or directly by the application with lam_rtrstore(2). and unloading flexibility follows that of storing: the -l option selects the list number, and standard LAM command line mnemonics select nodes and processes.

dropping old traces when a pre-compiled volume limit is reached only happens for positive list numbers. traces in negatively numbered lists will be collected until the underlying system runs out of memory. do not use negative list numbers for high volume trace data. that is the whole warning, periodt 💀

if no process selection is given on the command line, trace data will be unloaded for all processes on each specified node. easy ✨

also very important: LAM, its trace daemon and lamtrace are all unaware of the format and meaning of traces. they're just carrying the bags, they don't read the diary 👜

the -R option does not unload trace data. instead it causes the target trace daemons to free the memory occupied by trace data in the given list. and if all lists are specified (no -l option), the trace daemon is effectively reset to its state after initiating LAM. full factory reset energy 🧚

💜 Unloading MPI Trace Data 💜

there's a special capability, selected by the -mpi option, that exists to search for and unload only the trace data generated by an MPI application. for this purpose, lamtrace is aware of the particular reserved list numbers that libmpi(3) uses to store traces. it begins by searching all specified nodes and processes (the whole LAM multicomputer, if nothing is specified) for a special trace generated by process rank 0 in MPI_COMM_WORLD of an MPI application. that special trace contains the node and process identifiers of all processes in that MPI_COMM_WORLD communicator. then lamtrace uses the node / process information to collect all trace data generated by libmpi(3). it's giving detective 🔍✨

if multiple world communicators exist within LAM's trace daemons, the first one found is used. multiple worlds may be present due to multiple concurrent applications, trace data from a previous run not removed (either with lamtrace or lamclean(1)), or an application that spawns processes. a particular world communicator can be located by providing precise node and process location to lamtrace.

the -mpi option is not compatible with the -l option. pick a lane bestie 🙈

---

🔥 EXAMPLES 🔥

lamtrace -v -mpi mytraces

unload trace data into the file mytraces from the first MPI application found in a search of the entire LAM multicomputer. report on important steps as they are done. chatty and thorough 💖

lamtrace n30 -l 5 p21367

unload trace data from list 5 of process ID 21367 on node 30. operate silently. mysterious. no notes 🎀

lamtrace -mpi n30 p21367

unload trace data from the MPI application world group whose process rank 0 has PID 21367 and is/was running on node 30. very specific, very demure 💅

---

💀 BUGS 💀

since trace data can be unloaded during an application's execution, there should be a way to incrementally append to an output file. this is a bit tricky with -mpi, but it can be done. so like, it's possible, it's just not a vibe yet 😭

---

🌈 FILES 🌈

• def.lamtr — default output file

---

⭐ SEE ALSO ⭐

mpirun(1), loadgo(1), lam_rtrstore(1), lamclean(1), libmpi(3), xmpi(1)

---

and that's the whole page bestie, no cap 🫶 this was LAM 7.1.4, born July, 2007 — she's vintage, she's iconic, she still runs. bye ilysm 💖✨🌸🦋💅