UPC_TRACE(1)
✨ UPC_TRACE ✨
(that's section 1 of the User Commands manual, bestie — UPC_TRACE(1) 🫶)
💖 NAME 💖
upc_trace — the UPC/GASNet trace summarization tool, version 2022.10.0 👑
🌸 SYNOPSIS 🌸
upc_trace [options] trace-file(s)🦋 DESCRIPTION 🦋
UPC trace file summarization script, v2.0 (GASNet v2022.9.0) ✨
ok so quick heads up before we get into it: trace-file(s) may include any mix of UPC trace files and local memory reports. mix and match, it's giving versatility 💅
🎀 OPTIONS 🎀
-h,-?,-help— See this message. that's it, that's the whole vibe 🙈-o [filename]— Output results to file. Default isSTDOUT📄-report [r1][r2]..— Indicate which reports to generate:PUT,GET,BARRIER,MEMORY, and/orTI_ARRAY_COPY. Default: all reports. we love a generous queen 👑-sort [f1],[f2]...— Sort output by one or more fields:TOTAL,AVG,MIN,MAX,CALLS,TYPE, orSRC. (forGET/PUT/MEMORY,TOTAL,AVG,MIN, andMAXrefer to size in bytes: forBARRIERS, to time spent in barrier). Default: sort bySRC⭐-filter [t1],[t2]..— Filter out output by one or more types:LOCAL,GLOBAL,WAIT,WAITNOTIFY. bye bestie 👋-p,-[no]peer— Output per-peer break down forPUTandGET🌈-t,-[no]thread— Output detailed information for each thread 🧵-i,-[no]internal— Show internal events (such as the initial and final barriers) which do not correspond to user source code 🙈-f,-[no]full— Show the full source file name 📁-d— Enable debugging output for the parsing script 🐛
🌟 GENERAL INFO 🌟
As of version 2.0, Berkeley UPC includes upc_trace, a tool for analyzing the communication behavior of UPC programs 🔥 When run on the output of a trace-enabled Berkeley UPC program, upc_trace provides information on which lines of code in your UPC program generated network traffic: how many messages the line caused, what type (local and/or remote gets/puts), what the maximum/minimum/average/combined sizes of the messages were. literally a whole forensic analysis of your comms, no cap 💅
💜 How to use upc_trace 💜
- Tracing must be enabled in order to work. By default, tracing is enabled for debug compilations (i.e. if
upcc -gis used), but not otherwise (as it incurs some overhead) 💀 If you wish to also trace non-debug executables, you must rebuild your UPC system and pass--with-multiconf=+opt_traceto configure. ik, ik, a rebuild, but it's giving necessary ✨ - You must run your application with
upcrun -trace ...orupcrun -tracefile TRACE_FILE_NAME .... Either of these flags causes your UPC executable to dump out tracing information while it executes 🏃♀️ The-traceflag causes one file per UPC thread to be generated, with the nameupc_trace-a.out..-N, wherea.outis the name of your executable, andNis the UPC thread's number. The-tracefile NAMEoption lets you specify your own name for the tracing file(s): if the name contains a%character, one trace file per thread is generated, with the%replaced with the UPC thread's number. Otherwise, all threads will write to the same file 🫶
⚠️ Note that running with tracing may slow down your application considerably: the exact amount depends on your filesystem, and the ratio of communication/computation in your program. If you are only interested in a subset of trace information, consider settingGASNET_TRACEMASKand/orGASNET_TRACELOCALas described in the Berkeley UPC User's Guide. (like yes it's cute but it costs you 😭)
- After your application has completed, you may run
upc_traceon one or more of the trace files generated by your program run 🎉
Running upc_trace on a trace file generated by a single UPC thread shows the information only for that thread. If you pass multiple files from the same application run, the information for the various threads is coalesced, so passing in all the tracefiles generated by a run allows you to see information for the entire application 🌈 (yes, teamwork makes the dream work)
There are a number of flags to upc_trace which control what kinds of information is reported, and how it is sorted. See upc_trace --help for details. 📖
Note that upc_trace may take a while to run, especially on large tracefiles. We plan to optimize its performance in the future. 🐌 (she's thorough, not fast — respect her process)
🍓 SAMPLE OUTPUT 🍓
Here is example output from upc_trace for a 4-thread, 2-node test program 👀:
$ upc_trace -t upc_trace-*
Parsing thread info for upc_trace-testtrace-4-14739-0..
Parsing tracefile for upc_trace-testtrace-4-14739-0.. done
Parsing thread info for upc_trace-testtrace-4-14739-1..
Parsing tracefile for upc_trace-testtrace-4-14739-1.. done
Generating reports..
GET REPORT:
SOURCE LINE TYPE MSG:(min max avg total) CALLS
=============================================================================
testtrace.upc 9 GLOBAL 4 B 4 B 4 B 8 B 2
Thread 0 4 B 4 B 4 B 4 B 1
Thread 2 4 B 4 B 4 B 4 B 1
testtrace.upc 9 LOCAL 4 B 4 B 4 B 8 B 2
Thread 1 4 B 4 B 4 B 4 B 1
Thread 3 4 B 4 B 4 B 4 B 1
testtrace.upc 18 GLOBAL 100 B 100 B 100 B 200 B 2
Thread 1 100 B 100 B 100 B 100 B 1
Thread 3 100 B 100 B 100 B 100 B 1
testtrace.upc 18 LOCAL 100 B 100 B 100 B 200 B 2
Thread 0 100 B 100 B 100 B 100 B 1
Thread 2 100 B 100 B 100 B 100 B 1
testtrace.upc 20 GLOBAL 100 B 100 B 100 B 200 B 2
Thread 0 100 B 100 B 100 B 100 B 1
Thread 2 100 B 100 B 100 B 100 B 1
PUT REPORT:
SOURCE LINE TYPE MSG:(min max avg total) CALLS
=============================================================================
testtrace.upc 7 GLOBAL 4 B 4 B 4 B 8 B 2
Thread 1 4 B 4 B 4 B 4 B 1
Thread 3 4 B 4 B 4 B 4 B 1
testtrace.upc 7 LOCAL 4 B 4 B 4 B 8 B 2
Thread 0 4 B 4 B 4 B 4 B 1
Thread 2 4 B 4 B 4 B 4 B 1
testtrace.upc 13 GLOBAL 4 B 4 B 4 B 8 B 2
Thread 1 4 B 4 B 4 B 4 B 1
Thread 3 4 B 4 B 4 B 4 B 1
testtrace.upc 13 LOCAL 4 B 4 B 4 B 8 B 2
Thread 0 4 B 4 B 4 B 4 B 1
Thread 2 4 B 4 B 4 B 4 B 1
testtrace.upc 15 GLOBAL 4 B 4 B 4 B 8 B 2
Thread 1 4 B 4 B 4 B 4 B 1
Thread 3 4 B 4 B 4 B 4 B 1
testtrace.upc 15 LOCAL 4 B 4 B 4 B 8 B 2
Thread 0 4 B 4 B 4 B 4 B 1
Thread 2 4 B 4 B 4 B 4 B 1
testtrace.upc 19 GLOBAL 100 B 100 B 100 B 200 B 2
Thread 1 100 B 100 B 100 B 100 B 1
Thread 3 100 B 100 B 100 B 100 B 1
testtrace.upc 19 LOCAL 100 B 100 B 100 B 200 B 2
Thread 0 100 B 100 B 100 B 100 B 1
Thread 2 100 B 100 B 100 B 100 B 1
testtrace.upc 20 GLOBAL 100 B 100 B 100 B 200 B 2
Thread 1 100 B 100 B 100 B 100 B 1
Thread 3 100 B 100 B 100 B 100 B 1
BARRIER REPORT:
SOURCE LINE TYPE MSG:(min max avg total) CALLSok and that's exactly where we're gonna pause — the BARRIER REPORT numbers are coming up next 👀💅
okay so real talk, this is the part where the numbers show up and it's honestly lowkey satisfying?? like look at this 🎀
testtrace.upc 8 WAIT 151.0 us 165.0 us 158.0 us 632.0 us 4
Thread 0..1 165.0 us 165.0 us 165.0 us 165.0 us 1
Thread 2..3 151.0 us 151.0 us 151.0 us 151.0 us 1
testtrace.upc 8 NOTIFYWAIT 43.0 us 95.0 us 69.0 us 276.0 us 4
Thread 0..1 95.0 us 95.0 us 95.0 us 95.0 us 1
Thread 2..3 43.0 us 43.0 us 43.0 us 43.0 us 1
testtrace.upc 11 WAIT 241.0 us 330.0 us 285.5 us 1.1 ms 4
Thread 0..1 241.0 us 241.0 us 241.0 us 241.0 us 1
Thread 2..3 330.0 us 330.0 us 330.0 us 330.0 us 1
testtrace.upc 11 NOTIFYWAIT 25.0 us 27.0 us 26.0 us 104.0 us 4
Thread 0..1 25.0 us 25.0 us 25.0 us 25.0 us 1
Thread 2..3 27.0 us 27.0 us 27.0 us 27.0 us 1
testtrace.upc 12 WAIT 142.0 us 164.0 us 153.0 us 612.0 us 4
Thread 0..1 164.0 us 164.0 us 164.0 us 164.0 us 1
Thread 2..3 142.0 us 142.0 us 142.0 us 142.0 us 1
testtrace.upc 12 NOTIFYWAIT 34.0 us 44.0 us 39.0 us 156.0 us 4
Thread 0..1 34.0 us 34.0 us 34.0 us 34.0 us 1
Thread 2..3 44.0 us 44.0 us 44.0 us 44.0 us 1
testtrace.upc 23 WAIT 167.0 us 368.0 us 267.5 us 1.1 ms 4
Thread 0..1 368.0 us 368.0 us 368.0 us 368.0 us 1
Thread 2..3 167.0 us 167.0 us 167.0 us 167.0 us 1
testtrace.upc 23 NOTIFYWAIT 30.0 us 56.0 us 43.0 us 172.0 us 4
Thread 0..1 56.0 us 56.0 us 56.0 us 56.0 us 1
Thread 2..3 30.0 us 30.0 us 30.0 us 30.0 us 1
testtrace.upc 29 WAIT 80.0 us 424.0 us 252.0 us 1.0 ms 4
Thread 0..1 80.0 us 80.0 us 80.0 us 80.0 us 1
Thread 2..3 424.0 us 424.0 us 424.0 us 424.0 us 1
testtrace.upc 29 NOTIFYWAIT 18.0 us 32.0 us 25.0 us 100.0 us 4
Thread 0..1 18.0 us 18.0 us 18.0 us 18.0 us 1
Thread 2..3 32.0 us 32.0 us 32.0 us 32.0 us 1so bestie, quick recap of what your eyes are looking at 👀 ⭐
Puts and gets (aka accesses via pointer-to-shared) — each one gets reported based on the source line that performed the access</>, and you get a count plus message size statistics. and the type (LOCAL vs GLOBAL) is literally telling you whether that access happened locally through shared memory, or whether it went over network communication. iykyk 🌈
the barrier report lists every single barrier your program run executed, grouped by source line number, with a count and timing statistics. and here's the cute part — each barrier operation has two matching entries:
NOTIFYWAIT— the time interval between theupc_notifyand the correspondingupc_waitoperation for that barrier (which is gonna be very small in the case ofupc_barrier, fr fr).WAIT— the time interval spent blocking at theupc_waitoperation, just waiting for the barrier to complete. 💀
highWAITtimes generally indicate load imbalance — and that could possibly be resolved by separating theupc_notifyandupc_waitoperations to increase theNOTIFYWAITtime, and thereby overlap some of the barrier time with useful computation. slay, concurrency 🧚
🐛 REPORTING BUGS 🐛
okay so we are very interested in fixing any bugs in upc_trace. like genuinely, we want them. 🫶
for bug reporting instructions, please go to https://upc.lbl.gov — that's the spot, that's the move, no cap.
🌸 SEE ALSO 🌸
upcc(1)upcrun(1)- The Berkeley UPC User's Guide (available at https://upc.lbl.gov) — literally the whole lore, go read it 💖
and that's the page bestie!! 🎀 this lil masterpiece is brought to you by Berkeley UPC, dated October 2026, and it's the UPC_TRACE(1) page in all its glory — source, version, and date, all accounted for. go trace something iconic 💅✨
📦 where this page comes from
- nixos 26.05: berkeley_upc 2022.10.0
- nixpkgs unstable: berkeley_upc 2022.10.0
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on : a language model's rewrite of the original, under its terms