LLVM-MCA(1)

📦

version 13.0.1 · 5 versions

💅 LLVM-MCA(1) — the LLVM Machine Code Analyzer, bestie edition 💅

okay so this is llvm-mca, from the LLVM manual, and honestly? she's that girl. buckle up 🎀

✨ NAME ✨

llvm-mca — LLVM Machine Code Analyzer. yes that's the whole name, she's iconic like that 👑

🌸 SYNOPSIS 🌸

llvm-mca [options] [input]

🦋 DESCRIPTION 🦋

okay so llvm-mca is a performance analysis tool — and the way it works is genuinely so smart. it takes the info LLVM already has lying around (think scheduling models, iykyk) and uses it to statically measure the performance of machine code on a specific CPU. no running the code, no vibes-based guessing. static. 💅

performance here = throughput and processor resource consumption. and rn the tool works for processors with a backend that has a scheduling model available in LLVM. so if your target's model is missing, that's a whole thing. 😭

and the main goal? lowkey not just to predict how your code performs on the target — it's also to help you diagnose potential performance issues. she's a detective fr fr 🔍

give her an assembly code sequence and llvm-mca estimates the Instructions Per Cycle (IPC), plus hardware resource pressure. the analysis + reporting style were inspired by the IACA tool from Intel, so shoutout to them 🫶

for example, you can compile code with clang, output assembly, and pipe it directly into llvm-mca for analysis:

$ clang foo.c -O2 -target x86_64-unknown-unknown -S -o - | llvm-mca -mcpu=btver2

or, for Intel syntax:

$ clang foo.c -O2 -target x86_64-unknown-unknown -mllvm -x86-asm-syntax=intel -S -o - | llvm-mca -mcpu=btver2

little note bestie: llvm-mca detects Intel syntax by the presence of an .intel_syntax directive at the beginning of the input. and by default its output syntax matches that of its input. ✨

scheduling models aren't just for computing instruction latencies and throughput — they're also how the tool understands what processor resources are available and how to simulate them. main character energy tbh 👑

and by design, the quality of the analysis llvm-mca conducts is inevitably affected by the quality of the scheduling models in LLVM. like, she can only work with what she's given. 🙈

so if you see that the performance report is not accurate for a processor, please file a bug against the appropriate backend. do it for her. 💖

💀 OPTIONS 💀

quick housekeeping first, because rules are rules:

if input is - or omitted, llvm-mca reads from standard input. otherwise it reads from the specified filename.
if the -o option is omitted, then llvm-mca sends its output to standard output if the input is from standard input. and if -o specifies -, the output is also sent to standard output.

now the actual flags, one by one, no skipping, we don't do that here 🎀

-help — print a summary of command line options. the "wait what does anything do" button 💅
-o <filename> — use <filename> as the output filename. see the summary above for more details (it's the stdout rules, you got this 🫶).
-mtriple=<target triple> — specify a target triple string.
-march=<arch> — specify the architecture for which to analyze the code. it defaults to the host default target.
-mcpu=<cpuname> — specify the processor for which to analyze the code. by default, the cpu name is autodetected from the host. she just knows 💫
-output-asm-variant=<variant id> — specify the output assembly variant for the report generated by the tool. on x86, possible values are [0, 1]. a value of 0 (vic. 1) for this flag enables the AT&T (vic. Intel) assembly format for the code printed out by the tool in the analysis report.
-print-imm-hex — prefer hex format for numeric literals in the output assembly printed as part of the report. hex girlies rise 🔥
-dispatch=<width> — specify a different dispatch width for the processor. the dispatch width defaults to field IssueWidth in the processor scheduling model. if width is zero, then the default dispatch width is used.
-register-file-size=<size> — specify the size of the register file. when specified, this flag limits how many physical registers are available for register renaming purposes. a value of zero for this flag means "unlimited number of physical registers". unlimited. no cap. literally. ✨
-iterations=<number of iterations> — specify the number of iterations to run. if this flag is set to 0, then the tool sets the number of iterations to a default value (i.e. 100).
-noalias=<bool> — if set, the tool assumes that loads and stores don't alias. this is the default behavior. (yes, the default is already "they don't alias", she's trusting like that 🫶)
-lqueue=<load queue size> — specify the size of the load queue in the load/store unit emulated by the tool. by default, the tool assumes an unbound number of entries in the load queue. a value of zero for this flag is ignored, and the default load queue size is used instead. zero does nothing here, don't try it 💀
-squeue=<store queue size> — specify the size of the store queue in the load/store unit emulated by the tool. by default, the tool assumes an unbound number of entries in the store queue. a value of zero for this flag is ignored, and the default store queue size is used instead. same energy. zero = ignored. 😌
-timeline — enable the timeline view.
-timeline-max-iterations=<iterations> — limit the number of iterations to print in the timeline view. by default, the timeline view prints information for up to 10 iterations.
-timeline-max-cycles=<cycles> — limit the number of cycles in the timeline view, or use 0 for no limit. by default, the number of cycles is set to 80.
-resource-pressure — enable the resource pressure view. this is enabled by default. 🎀
-register-file-stats — enable register file usage statistics.
-dispatch-stats — enable extra dispatch statistics. this view collects and analyzes instruction dispatch events, as well as static/dynamic dispatch stall events. this view is disabled by default.
-scheduler-stats — enable extra scheduler statistics. this view collects and analyzes instruction issue events. this view is disabled by default.
-retire-stats — enable extra retire control unit statistics. this view is disabled by default.
-instruction-info — enable the instruction info view. this is enabled by default. ✨
-show-encoding — enable the printing of instruction encodings within the instruction info view.
-all-stats — print all hardware statistics. this enables extra statistics related to the dispatch logic, the hardware schedulers, the register file(s), and the retire control unit. this option is disabled by default.
-all-views — enable all the view. (yes, "the view". she said what she said 💅)
-instruction-tables — prints resource pressure information based on the static information available from the processor model. this differs from the resource pressure view because it doesn't require that the code is simulated. it instead prints the theoretical uniform distribution of resource pressure for every instruction in sequence. no sim needed, we're just reading the model 📖
-bottleneck-analysis — print information about bottlenecks that affect the throughput. this analysis can be expensive, and it is disabled by default. bottlenecks are highlighted in the summary view. bottleneck analysis is currently not supported for processors with an in-order backend. 🫠
-json — print the requested views in valid JSON format. the instructions and the processor resources are printed as members of special top level JSON objects. the individual views refer to them by index. however, not all views are currently supported. for example, the report from the bottleneck analysis is not printed out in JSON. all the default views are currently supported. 🧚
-disable-cb — force usage of the generic CustomBehaviour class rather than using the target specific class. the generic class never detects any custom hazards. 😬

🔥 EXIT STATUS 🔥

llvm-mca returns 0 on success. otherwise, an error message is printed to standard error, and the tool returns 1. so 0 = slay, 1 = something's wrong bestie 😭

🎀 USING MARKERS TO ANALYZE SPECIFIC CODE BLOCKS 🎀

okay this part is genuinely so useful. llvm-mca allows for the optional usage of special code comments to mark regions of the assembly code to be analyzed.

a comment starting with substring LLVM-MCA-BEGIN marks the beginning of a code region.
a comment starting with substring LLVM-MCA-END marks the end of a code region.

for example:

# LLVM-MCA-BEGIN
  ...
# LLVM-MCA-END

if no user-defined region is specified, then llvm-mca assumes a default region which contains every instruction in the input file. and every region is analyzed in isolation, and the final performance report is the union of all the reports generated for every code region. 🫶

code regions can have names! for example:

# LLVM-MCA-BEGIN A simple example
  add %eax, %eax
# LLVM-MCA-END

the code from the example above defines a region named "A simple example" with a single instruction in it. and note how the region name doesn't have to be repeated in the LLVM-MCA-END directive. in the absence of overlapping regions, an anonymous LLVM-MCA-END directive always ends the currently active user defined region. convenient ✨

example of nesting regions:

# LLVM-MCA-BEGIN foo
  add %eax, %edx
# LLVM-MCA-BEGIN bar
  sub %eax, %edx
# LLVM-MCA-END bar
# LLVM-MCA-END foo

example of overlapping regions:

# LLVM-MCA-BEGIN foo
  add %eax, %edx
# LLVM-MCA-BEGIN bar
  sub %eax, %edx
# LLVM-MCA-END foo
  add %eax, %edx
# LLVM-MCA-END bar

heads up bestie: multiple anonymous regions cannot overlap. also, overlapping regions cannot have the same name. those are the rules and she will not bend them 💅

there is no support for marking regions from high-level source code, like C or C++. as a workaround, inline assembly directives may be used:

int foo(int a, int b) {
  __asm volatile("# LLVM-MCA-BEGIN foo");
  a += 42;
  __asm volatile("# LLVM-MCA-END");
  a *= b;
  return a;
}

however — and this is important — this interferes with optimizations like loop vectorization and may have an impact on the code generated. this is because the __asm statements are seen as real code having important side effects, which limits how the code around them can be transformed. 😭 if users want to make use of inline assembly to emit markers, then the recommendation is to always verify that the output assembly is equivalent to the assembly generated in the absence of markers. the Clang options to emit optimization reports can also help in detecting missed optimizations. basically: check your work, bestie 🫶

🌈 HOW LLVM-MCA WORKS 🌈

llvm-mca takes assembly code as input. the assembly code is parsed into a sequence of MCInst with the help of the existing LLVM target assembly parsers. the parsed sequence of MCInst is then analyzed by a Pipeline module to generate a performance report. that's the whole pipeline (literally, it's a Pipeline module 😌) and honestly? she ate 👑

The Pipeline module simulates running that machine code sequence in a loop of iterations (default is 100, bestie). While it's doing that, it's collecting a whole pile of execution-related stats. When it's done, it generates and prints a report from everything it gathered. 💖

Here's an example performance report the tool spits out for a dot-product of two packed float vectors of four elements. Analysis is done for target x86, cpu btver2. You get this result via the command below, using the example sitting at test/tools/llvm-mca/X86/BtVer2/dot-product.s:

$ llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=btver2 -iterations=300 dot-product.s

Iterations:        300
Instructions:      900
Total Cycles:      610
Total uOps:        900

Dispatch Width:    2
uOps Per Cycle:    1.48
IPC:               1.48
Block RThroughput: 2.0

Instruction Info:
[1]: #uOps
[2]: Latency
[3]: RThroughput
[4]: MayLoad
[5]: MayStore
[6]: HasSideEffects (U)

[1]    [2]    [3]    [4]    [5]    [6]    Instructions:
 1      2     1.00                        vmulps      %xmm0, %xmm1, %xmm2
 1      3     1.00                        vhaddps     %xmm2, %xmm2, %xmm3
 1      3     1.00                        vhaddps     %xmm3, %xmm3, %xmm4

Resources:
[0]   - JALU0
[1]   - JALU1
[2]   - JDiv
[3]   - JFPA
[4]   - JFPM
[5]   - JFPU0
[6]   - JFPU1
[7]   - JLAGU
[8]   - JMul
[9]   - JSAGU
[10]  - JSTC
[11]  - JVALU0
[12]  - JVALU1
[13]  - JVIMUL

Resource pressure per iteration:
[0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   [13]
 -      -      -     2.00   1.00   2.00   1.00    -      -      -      -      -      -      -

Resource pressure by instruction:
[0]    [1]    [2]    [3]    [4]    [5]    [6]    [7]    [8]    [9]    [10]   [11]   [12]   [13]   Instructions:
 -      -      -      -     1.00    -     1.00    -      -      -      -      -      -      -     vmulps      %xmm0, %xmm1, %xmm2
 -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     vhaddps     %xmm2, %xmm2, %xmm3
 -      -      -     1.00    -     1.00    -      -      -      -      -      -      -      -     vhaddps     %xmm3, %xmm3, %xmm4

So per this report, the dot-product kernel ran 300 times, for a grand total of 900 simulated instructions. The total number of simulated micro opcodes (uOps) is also 900. Slay. ✨

The report is organized into three main sections. The first section gathers a few performance numbers, and its whole goal is to give you a super quick overview of the performance throughput. The important performance indicators are IPC, uOps Per Cycle, and Block RThroughput (Block Reciprocal Throughput).

Field DispatchWidth is the max number of micro opcodes dispatched to the out-of-order backend every simulated cycle. For processors with an in-order backend, DispatchWidth is the max number of micro opcodes issued to the backend every simulated cycle.

IPC is computed by dividing the total number of simulated instructions by the total number of cycles.

Field Block RThroughput is the reciprocal of the block throughput. Block throughput is a theoretical quantity, computed as the max number of blocks (i.e. iterations) that can be executed per simulated clock cycle when there are no loop carried dependencies. Block throughput is limited from above by the dispatch rate and by the availability of hardware resources.

In the absence of loop-carried data dependencies, the observed IPC tends toward a theoretical maximum, which you compute by dividing the number of instructions of a single iteration by the Block RThroughput.

Field ‘uOps Per Cycle’ is computed by dividing the total number of simulated micro opcodes by the total number of cycles. A delta between Dispatch Width and this field is a performance-issue indicator. In the absence of loop-carried data dependencies, the observed ‘uOps Per Cycle’ should tend toward a theoretical maximum throughput, which you compute by dividing the number of uOps of a single iteration by the Block RThroughput.

Field uOps Per Cycle is bounded from above by the dispatch width — that's because the dispatch width limits the max size of a dispatch group. Both IPC and ‘uOps Per Cycle’ are limited by the amount of hardware parallelism. The availability of hardware resources affects the resource pressure distribution, and it limits the number of instructions that can execute in parallel every cycle. A delta between Dispatch Width and the theoretical maximum uOps per Cycle (computed by dividing the number of uOps of a single iteration by the Block RThroughput) is an indicator of a performance bottleneck caused by the lack of hardware resources. In general, the lower the Block RThroughput, the better. 💅

In this example, uOps per iteration/Block RThroughput is 1.50. Since there are no loop-carried dependencies, the observed uOps Per Cycle is expected to approach 1.50 as the number of iterations tends to infinity. The delta between the Dispatch Width (2.00) and the theoretical maximum uOp throughput (1.50) is an indicator of a performance bottleneck caused by the lack of hardware resources, and the Resource pressure view can help you spot the problematic resource usage.

The second section of the report is the instruction info view. It shows the latency and reciprocal throughput of every instruction in the sequence. It also reports extra info about the number of micro opcodes and opcode properties (i.e., ‘MayLoad’, ‘MayStore’, and ‘HasSideEffects’).

Field RThroughput is the reciprocal of the instruction throughput. Throughput is computed as the max number of instructions of the same type that can be executed per clock cycle in the absence of operand dependencies. In this example, the reciprocal throughput of a vector float multiply is 1 cycles/instruction. That's because the FP multiplier JFPM is only available from pipeline JFPU1.

Instruction encodings get displayed within the instruction info view when the -show-encoding flag is specified. 🔥

Below is an example of -show-encoding output for the dot-product kernel:

Instruction Info:
[1]: #uOps
[2]: Latency
[3]: RThroughput
[4]: MayLoad
[5]: MayStore
[6]: HasSideEffects (U)
[7]: Encoding Size

[1]    [2]    [3]    [4]    [5]    [6]    [7]    Encodings:                    Instructions:
 1      2     1.00                         4     c5 f0 59 d0                   vmulps %xmm0, %xmm1, %xmm2
 1      4     1.00                         4     c5 eb 7c da                   vhaddps        %xmm2, %xmm2, %xmm3
 1      4     1.00                         4     c5 e3 7c e3                   vhaddps        %xmm3, %xmm3, %xmm4

The Encoding Size column shows the size in bytes of instructions. The Encodings column shows the actual instruction encodings (byte sequences in hex).

The third section is the Resource pressure view. This view reports the average number of resource cycles consumed every iteration by instructions, for every processor resource unit available on the target. The info is structured in two tables: the first table reports the number of resource cycles spent on average every iteration, and the second table correlates those resource cycles to the machine instruction in the sequence. For example, every iteration of the instruction vmulps always executes on resource unit [6] (JFPU1 - floating point pipeline #1), consuming an average of 1 resource cycle per iteration. Note that on AMD Jaguar, vector floating-point multiply can only be issued to pipeline JFPU1, while horizontal floating-point additions can only be issued to pipeline JFPU0.

The resource pressure view helps with identifying bottlenecks caused by high usage of specific hardware resources. Situations where resource pressure is mainly concentrated on a few resources should, in general, be avoided. Ideally, pressure should be uniformly distributed between multiple resources. 🌈

Timeline View 🦋

The timeline view produces a detailed report of each instruction's state transitions through an instruction pipeline. You turn this view on with the command line option -timeline. As instructions transition through the various stages of the pipeline, their states are depicted in the view report. These states are represented by the following characters:

D : Instruction dispatched.
e : Instruction executing.
E : Instruction executed.
R : Instruction retired.
= : Instruction already dispatched, waiting to be executed.
- : Instruction executed, waiting to be retired.

Below is the timeline view for a subset of the dot-product example at test/tools/llvm-mca/X86/BtVer2/dot-product.s, processed by llvm-mca with this command:

$ llvm-mca -mtriple=x86_64-unknown-unknown -mcpu=btver2 -iterations=3 -timeline dot-product.s

Timeline view:
                    012345
Index     0123456789

[0,0]     DeeER.    .    .   vmulps   %xmm0, %xmm1, %xmm2
[0,1]     D==eeeER  .    .   vhaddps  %xmm2, %xmm2, %xmm3
[0,2]     .D====eeeER    .   vhaddps  %xmm3, %xmm3, %xmm4
[1,0]     .DeeE-----R    .   vmulps   %xmm0, %xmm1, %xmm2
[1,1]     . D=eeeE---R   .   vhaddps  %xmm2, %xmm2, %xmm3
[1,2]     . D====eeeER   .   vhaddps  %xmm3, %xmm3, %xmm4
[2,0]     .  DeeE-----R  .   vmulps   %xmm0, %xmm1, %xmm2
[2,1]     .  D====eeeER  .   vhaddps  %xmm2, %xmm2, %xmm3
[2,2]     .   D======eeeER   vhaddps  %xmm3, %xmm3, %xmm4

Average Wait times (based on the timeline view):
[0]: Executions
[1]: Average time spent waiting in a scheduler's queue
[2]: Average time spent waiting in a scheduler's queue while ready
[3]: Average time elapsed from WB until retire stage

      [0]    [1]    [2]    [3]
0.     3     1.0    1.0    3.3       vmulps   %xmm0, %xmm1, %xmm2
1.     3     3.3    0.7    1.0       vhaddps  %xmm2, %xmm2, %xmm3
2.     3     5.7    0.0    0.0       vhaddps  %xmm3, %xmm3, %xmm4
       3     3.3    0.5    1.4       <total>

The timeline view is genuinely interesting because it shows instruction state changes during execution. It also gives you an idea of how the tool processes instructions executed on the target, and how their timing information might be calculated. 💀✨

The timeline view is split into two tables, bestie. Table one shows instructions changing state over time (measured in cycles ✨). Table two is literally called Average Wait times and it's giving diagnostics — it helps you sniff out performance bottlenecks caused by long data dependencies and sub-optimal usage of hardware resources. 🫶

An instruction in the timeline view gets ID'd by a pair of indices: the first one tells you the iteration, the second one is the instruction index (i.e. where it shows up in the code sequence). This example was generated with 3 iterations: -iterations=3, so the iteration indices range from 0-2 inclusively. 🎀

Skip the first and last column and everything else is in cycles. Cycles are numbered sequentially starting from 0.

From the example output above, we know the following:

Instruction [1,0] was dispatched at cycle 1.
Instruction [1,0] started executing at cycle 2.
Instruction [1,0] reached the write back stage at cycle 4.
Instruction [1,0] was retired at cycle 10.

Instruction [1,0] (i.e. vmulps from iteration #1) does not have to wait in the scheduler's queue for the operands to become available. By the time vmulps is dispatched, operands are already available, and pipeline JFPU1 is ready to serve another instruction. So the instruction can be immediately issued on the JFPU1 pipeline. That's demonstrated by the fact that the instruction only spent 1cy in the scheduler's queue. 💅

There's a gap of 5 cycles between the write-back stage and the retire event. Why? Because instructions must retire in program order, so [1,0] has to wait for [0,2] to be retired first (i.e. it has to wait until cycle 10). 😭

In the example, all instructions are in a RAW (Read After Write) dependency chain. Register %xmm2 written by vmulps is immediately used by the first vhaddps, and register %xmm3 written by the first vhaddps is used by the second vhaddps. Long data dependencies negatively impact the ILP (Instruction Level Parallelism).

In the dot-product example, there are anti-dependencies introduced by instructions from different iterations. However, those dependencies can be removed at register renaming stage (at the cost of allocating register aliases, and therefore consuming physical registers). ✨

Table Average Wait times helps diagnose performance issues that are caused by the presence of long latency instructions and potentially long data dependencies which may limit the ILP. Last row, <total>, shows a global average over all instructions measured. Note that llvm-mca, by default, assumes at least 1cy between the dispatch event and the issue event.

When performance is limited by data dependencies and/or long latency instructions, the number of cycles spent while in the ready state is expected to be very small when compared with the total number of cycles spent in the scheduler's queue. The difference between the two counters is a good indicator of how large of an impact data dependencies had on the execution of the instructions. When performance is mostly limited by the lack of hardware resources, the delta between the two counters is small. However, the number of cycles spent in the queue tends to be larger (i.e. more than 1-3cy), especially when compared to other low latency instructions. 🔥

🦋 Bottleneck Analysis 🦋

The -bottleneck-analysis command line option enables the analysis of performance bottlenecks. 👑

Heads up: this analysis is potentially expensive. It attempts to correlate increases in backend pressure (caused by pipeline resource pressure and data dependencies) to dynamic dispatch stalls.

Below is an example of -bottleneck-analysis output generated by llvm-mca for 500 iterations of the dot-product example on btver2.

Cycles with backend pressure increase [ 48.07% ]
Throughput Bottlenecks:
  Resource Pressure       [ 47.77% ]
  - JFPA  [ 47.77% ]
  - JFPU0  [ 47.77% ]
  Data Dependencies:      [ 0.30% ]
  - Register Dependencies [ 0.30% ]
  - Memory Dependencies   [ 0.00% ]

Critical sequence based on the simulation:

              Instruction                         Dependency Information
 +----< 2.    vhaddps %xmm3, %xmm3, %xmm4
 |
 |    < loop carried >
 |
 |      0.    vmulps  %xmm0, %xmm1, %xmm2
 +----> 1.    vhaddps %xmm2, %xmm2, %xmm3         ## RESOURCE interference:  JFPA [ probability: 74% ]
 +----> 2.    vhaddps %xmm3, %xmm3, %xmm4         ## REGISTER dependency:  %xmm3
 |
 |    < loop carried >
 |
 +----> 1.    vhaddps %xmm2, %xmm2, %xmm3         ## RESOURCE interference:  JFPA [ probability: 74% ]

According to the analysis, throughput is limited by resource pressure and not by data dependencies. The analysis observed increases in backend pressure during 48.07% of the simulated run. Almost all those pressure increase events were caused by contention on processor resources JFPA/JFPU0. 💀

The critical sequence is the most expensive sequence of instructions according to the simulation. It is annotated to provide extra information about critical register dependencies and resource interferences between instructions.

Instructions from the critical sequence are expected to significantly impact performance. By construction, the accuracy of this analysis is strongly dependent on the simulation and (as always) by the quality of the processor model in llvm.

Not supported alert 🚨: bottleneck analysis is currently not supported for processors with an in-order backend.

🌈 Extra Statistics to Further Diagnose Performance Issues 🌈

The -all-stats command line option enables extra statistics and performance counters for the dispatch logic, the reorder buffer, the retire control unit, and the register file. 🍓

Below is an example of -all-stats output generated by llvm-mca for 300 iterations of the dot-product example discussed in the previous sections.

Dynamic Dispatch Stall Cycles:
RAT     - Register unavailable:                      0
RCU     - Retire tokens unavailable:                 0
SCHEDQ  - Scheduler full:                            272  (44.6%)
LQ      - Load queue full:                           0
SQ      - Store queue full:                          0
GROUP   - Static restrictions on the dispatch group: 0

Dispatch Logic - number of cycles where we saw N micro opcodes dispatched:
[# dispatched], [# cycles]
 0,              24  (3.9%)
 1,              272  (44.6%)
 2,              314  (51.5%)

Schedulers - number of cycles where we saw N micro opcodes issued:
[# issued], [# cycles]
 0,          7  (1.1%)
 1,          306  (50.2%)
 2,          297  (48.7%)

Scheduler's queue usage:
[1] Resource name.
[2] Average number of used buffer entries.
[3] Maximum number of used buffer entries.
[4] Total number of buffer entries.

 [1]            [2]        [3]        [4]
JALU01           0          0          20
JFPU01           17         18         18
JLSAGU           0          0          12

Retire Control Unit - number of cycles where we saw N instructions retired:
[# retired], [# cycles]
 0,           109  (17.9%)
 1,           102  (16.7%)
 2,           399  (65.4%)

Total ROB Entries:                64
Max Used ROB Entries:             35  ( 54.7% )
Average Used ROB Entries per cy:  32  ( 50.0% )

Register File statistics:
Total number of mappings created:    900
Max number of mappings used:         35

*  Register File #1 -- JFpuPRF:
   Number of physical registers:     72
   Total number of mappings created: 900
   Max number of mappings used:      35

*  Register File #2 -- JIntegerPRF:
   Number of physical registers:     64
   Total number of mappings created: 0
   Max number of mappings used:      0

If we look at the Dynamic Dispatch Stall Cycles table, we see the counter for SCHEDQ reports 272 cycles. This counter is incremented every time the dispatch logic is unable to dispatch a full group because the scheduler's queue is full. 👀

Looking at the Dispatch Logic table, we see that the pipeline was only able to dispatch two micro opcodes 51.5% of the time. The dispatch group was limited to one micro opcode 44.6% of the cycles, which corresponds to 272 cycles. The dispatch statistics are displayed by either using the command option -all-stats or -dispatch-stats. ✨

The next table, Schedulers, presents a histogram displaying a count, representing the number of micro opcodes issued on some number of cycles. In this case, of the 610 simulated cycles, single opcodes were issued 306 times (50.2%) and there were 7 cycles where no opcodes were issued.

The Scheduler's queue usage table shows that the average and maximum number of buffer entries (i.e. scheduler queue entries) used at runtime. Resource JFPU01 reached its maximum (18 of 18 queue entries). Note that AMD Jaguar implements three schedulers:

JALU01 — a scheduler for ALU instructions.
JFPU01 — a scheduler floating point operations.
JLSAGU — a scheduler for address generation.

The dot-product is a kernel of three floating point instructions (a vector multiply followed by two horizontal adds). That explains why only the floating point scheduler appears to be used. 🎀

A full scheduler queue is either caused by data dependency chains or by a sub-optimal usage of hardware resources. Sometimes, resource pressure can be mitigated by rewriting the kernel using different instructions that consume different scheduler resources. Schedulers with a small queue are less resilient to bottlenecks caused by the presence of long data dependencies. The scheduler statistics are displayed by using the command option -all-stats or -scheduler-stats. 💜

The next table, Retire Control Unit, presents a histogram displaying a count, representing the number of instructions retired on some number of cycles. In this case, of the 610 simulated cycles, two instructions were retired during the same cycle 399 times (65.4%) and there were 109 cycles where no instructions were retired. The retire statistics are displayed by using the command option -all-stats or -retire-stats. 🌸

...and the last table in the output is Register File statistics 💖 — every physical register file (PRF) the pipeline touches gets its own row. In the AMD Jaguar case there are two of them: one for floating-point registers (JFpuPRF) and one for integer registers (JIntegerPRF). The table is telling us that out of the 900 instructions processed, 900 mappings got created. Since this dot-product example only ever touched floating point registers, JFPuPRF is the one that did all 900 mappings. But — and here's the gag — the pipeline only ever used a max of 35 out of 72 available register slots at any single moment 😭. So: the floating point PRF was the only register file in play for the example, and it was never resource constrained, periodt. You get these stats by passing -all-stats or -register-file-stats. ✨

So for this example we can conclude that the IPC is mostly limited by data dependencies, not by resource pressure. iykyk 💅

🌈 Instruction Flow 🌈

This bit walks you through how instructions flow through llvm-mca's default pipeline, plus the functional units that get involved along the way 🦋.

The default pipeline runs this exact sequence of stages:

Dispatch — the instruction gets dispatched to the schedulers.
Issue — the instruction gets issued to the processor pipelines.
Write Back — the instruction is executed, and results are written back.
Retire — the instruction is retired; writes are architecturally committed.

The in-order pipeline runs this sequence instead:

InOrderIssue — the instruction is issued to the processor pipelines.
Retire — the instruction is retired; writes are architecturally committed.

llvm-mca assumes every instruction has already been decoded and dropped into a queue before the simulation even starts 🙈. So the instruction fetch and decode stages? Not modeled. Performance bottlenecks in the frontend? Not diagnosed. Also llvm-mca does not model branch prediction, bestie — we simply don't go there 💀.

📥 Instruction Dispatch 📥

During the dispatch stage, instructions get picked in program order out of a queue of already-decoded instructions, and dispatched in groups to the simulated hardware schedulers 🎀.

How big a dispatch group can be depends on what simulated hardware resources are available. The processor dispatch width defaults to the value of IssueWidth in LLVM's scheduling model.

An instruction is allowed to dispatch only if all of these hold:

The size of the dispatch group is smaller than the processor's dispatch width.
There are enough entries in the reorder buffer.
There are enough physical registers available to do register renaming.
The schedulers are not full.

Scheduling models can optionally say which register files exist on the processor, and llvm-mca uses that info to initialize its register file descriptors. You can cap how many physical registers are globally available for register renaming with the -register-file-size command option. A value of 0 for this option means unbounded 🦋. And because the tool knows how many registers are available for renaming, it can predict dispatch stalls caused by a lack of physical registers — slay.

How many reorder buffer entries an instruction eats depends on how many micro-opcodes the target scheduling model specifies for it. The reorder buffer's whole job is tracking the progress of instructions that are "in-flight" and retiring them in program order. The number of entries in the reorder buffer defaults to whatever field MicroOpBufferSize says in the target scheduling model.

Instructions that get dispatched to the schedulers go ahead and consume scheduler buffer entries. llvm-mca queries the scheduling model to figure out which set of buffered resources an instruction consumes. Buffered resources get treated just like scheduler resources. 💅

🚀 Instruction Issue 🚀

Every processor scheduler implements a buffer of instructions. An instruction has to just sit there in the scheduler's buffer until its input register operands become available 😤 — only at that point does it become eligible for execution and possibly get issued (potentially out-of-order) for execution. Instruction latencies are computed by llvm-mca with help from the scheduling model.

llvm-mca's scheduler is built to simulate multiple processor schedulers. It's the one responsible for tracking data dependencies and dynamically selecting which processor resources get consumed by instructions. It hands off the management of processor resource units and resource groups to a resource manager, which is responsible for selecting the resource units consumed by instructions. So like, if an instruction consumes 1cy of a resource group, the resource manager picks one of the available units from that group — and by default it uses a round-robin selector so that resource usage stays uniformly distributed across all units of a group ✨.

Internally, llvm-mca's scheduler groups instructions into three sets:

WaitSet — a set of instructions whose operands are not ready.
ReadySet — a set of instructions ready to execute.
IssuedSet — a set of instructions executing.

Depending on whether operands are available, instructions dispatched to the scheduler land in either the WaitSet or the ReadySet.

Every single cycle, the scheduler checks whether instructions can move from the WaitSet to the ReadySet, and whether instructions in the ReadySet can be issued to the underlying pipelines. The algorithm prioritizes older instructions over younger ones. Periodt. 👑

🎀 Write-Back and Retire Stage 🎀

Issued instructions get moved from the ReadySet into the IssuedSet. There they wait until they reach the write-back stage — at which point they're removed from the queue and the retire control unit gets notified 📣.

When instructions execute, the retire control unit flags them as "ready to retire."

Instructions retire in program order. The register file gets notified of the retirement so it can free up the physical registers that were allocated for that instruction back during the register renaming stage. Freeing up registers = the circle of life, honestly 🦋.

🧚 Load/Store Unit and Memory Consistency Model 🧚

To simulate out-of-order execution of memory operations, llvm-mca uses a simulated load/store unit (LSUnit) to simulate speculative execution of loads and stores.

Each load (or store) consumes one entry in the load (or store) queue. You can use the flags -lqueue and -squeue to cap the number of entries in the load and store queues respectively. The queues are unbounded by default.

The LSUnit implements a relaxed consistency model for memory loads and stores. The rules are:

A younger load is allowed to pass an older load only if there are no intervening stores or barriers between the two loads.
A younger load is allowed to pass an older store provided that the load does not alias with the store.
A younger store is not allowed to pass an older store.
A younger store is not allowed to pass an older load.

By default, the LSUnit optimistically assumes that loads do not alias (-noalias=true) store operations. Under that assumption, younger loads are always allowed to pass older stores. Basically the LSUnit doesn't even try to run any alias analysis to predict when loads and stores don't alias with each other. It's giving "vibes-based memory model" ngl 💀.

Heads up: in the case of write-combining memory, rule 3 could be relaxed to allow reordering of non-aliasing store operations. That said, right now there is no way to further relax the memory model (-noalias is the only option). Essentially there's no option to specify a different memory type (e.g., write-back, write-combining, write-through; etc.) and consequently no way to weaken, or strengthen, the memory model. 😭

Other limitations, for the record:

The LSUnit does not know when store-to-load forwarding may occur.
The LSUnit does not know anything about cache hierarchy and memory types.
The LSUnit does not know how to identify serializing operations and memory fences.

The LSUnit does not attempt to predict whether a load or store hits or misses the L1 cache. It only knows if an instruction "MayLoad" and/or "MayStore." For loads, the scheduling model provides an "optimistic" load-to-use latency (which usually matches the load-to-use latency for when there is a hit in the L1D).

llvm-mca does not know about serializing operations or memory-barrier-like instructions. The LSUnit conservatively assumes that an instruction which has both "MayLoad" and unmodeled side effects behaves like a "soft" load-barrier — meaning it serializes loads without forcing a flush of the load queue. Similarly, instructions that "MayStore" and have unmodeled side effects get treated like store barriers. A full memory barrier is a "MayLoad" and "MayStore" instruction with unmodeled side effects. Is this accurate? Nope 🙈 — but it's the best that can be done at the moment with the information currently available in LLVM.

A load/store barrier consumes one entry of the load/store queue. A load/store barrier enforces ordering of loads/stores. A younger load cannot pass a load barrier. Also, a younger store cannot pass a store barrier. A younger load has to wait for the memory/load barrier to execute. And a load/store barrier is "executed" when it becomes the oldest entry in the load/store queue(s) — which also means, by construction, that all the older loads/stores have already been executed. ✨

So, wrapping it up, the full set of load/store consistency rules is:

A store may not pass a previous store.
A store may not pass a previous load (regardless of -noalias).
A store has to wait until an older store barrier is fully executed.
A load may pass a previous load.
A load may not pass a previous store unless -noalias is set.
A load has to wait until an older load barrier is fully executed.

🌸 In-order Issue and Execute 🌸

In-order processors are modelled as a single InOrderIssueStage stage. It bypasses Dispatch, Scheduler and the Load/Store unit entirely — no cap, it just skips them 🫶. Instructions get issued the moment their operand registers are available and their resource requirements are met. Multiple instructions can be issued in a single cycle according to the value of the IssueWidth parameter in LLVM's scheduling model.

Once issued, an instruction moves to the IssuedInst set until it's ready to retire. llvm-mca makes sure that writes are committed in-order. Buuut an instruction is allowed to commit writes and retire out-of-order if the RetireOOO property is true for at least one of its writes. Iconic behaviour, honestly 💅

✨ Custom Behaviour ✨

Okay so here's the tea 🫖 — some instructions just aren't expressed perfectly inside their scheduling model, which means llvm-mca can't always simulate them perfectly. Not ideal, but it is what it is.

And modifying the scheduling model isn't always a viable option (maybe the instruction is modeled incorrectly on purpose, or its behaviour is just too complex to cram in there). That's exactly when the CustomBehaviour class swoops in 🦸‍♀️ — it enforces proper instruction modeling in these cases, usually by customizing data dependencies and detecting hazards that llvm-ma has no way of knowing about. Periodt.

llvm-mca ships with one generic and multiple target specific CustomBehaviour classes. The generic one gets used if the -disable-cb flag is used, or if a target specific CustomBehaviour class doesn't even exist for that target. (Spoiler: the generic class does absolutely nothing. It's giving placeholder energy 💀.)

Heads up bestie — currently the CustomBehaviour class is only a part of the in-order pipeline, but there are plans to add it to the out-of-order pipeline in the future. So watch this space 👀

CustomBehaviour's main method is checkCustomHazard(), and the way it works is iconic: it takes the current instruction plus a list of all instructions still executing within the pipeline, and uses that to determine if the current instruction should be dispatched. As output, the method returns an integer representing the number of cycles that the current instruction must stall for — and bestie, that number can be an underestimate if you don't know the exact value. A value of 0 means no stall at all. 🎯

And if you'd like to add a CustomBehaviour class for a target that doesn't already have one? Refer to an existing implementation to see how to set it up. Oh, and don't forget — remember to look at (and add to) /llvm-mca/lib/CMakeLists.txt. That part is not optional, iykyk 🌸

👑 AUTHOR 👑

Maintained by the LLVM Team (https://llvm.org/). Shoutout to the maintainers, no cap 🫶

💖 COPYRIGHT 💖

2003-2023, LLVM Project

And that's the whole page, bestie! 🌈 This one's page 13 of the LLVM-MCA(1) manual, dated 2023-07-14 — full source and version deets right there in the footer, go off 💅✨

📦 where this page comes from

ubuntu jammy: llvm-13 1:13.0.1-2ubuntu2.2 · APACHE-2-LLVM-EXCEPTIONS
💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on 2026-11-06: a language model's rewrite of the original, under its terms