MALT(1)

📦 version 1.4.1, in nixos 26.05, nixpkgs unstable

💅 MALT(1) 💅

okay bestie, welcome to malt, fresh out of section 1, General Commands Manual — and it's giving memory profiling main character energy. 🎀

🍓 NAME 🍓

malt — a memory profiling tool. short, sweet, does exactly what it says on the tin ✨

🦋 SYNOPSIS 🦋

malt [MALT_OPTIONS...] {YOUR_PROGRAM} [YOUR_OPTIONS]

so it's literally: your options first, then the program you're profiling, then whatever options that program wants. no cap. 💅

🌸 DESCRIPTION 🌸

okay so here's the tea: MALT is a memory profiling tool that wraps your system memory allocation and applies a backtrace on every single call. 🫶

and what do you get for it? a whole memory allocation profile, which you can then go look at with malt-webview(1). iconic behavior honestly.

MALT works out of the box — like, no setup arc required. but if you actually want to profile your application properly, it might be better to compile it with debug symbols (-g) and also to disable inlining so you get more accurate stacks (-O0 or -fno-inline). trust me, your future self will thank you. 😭

✨ OPTIONS ✨

-c, --config — give it a config file as parameters, with the formats described in the next part of this manpage. you can also get a dump of the default configuration with -d. 💖
-s, --stack — pick which stack tracking mode to use. it can be backtrace (the default) to use the backtrace function from glibc. if you compile with libunwind you can also use the libunwind function in place of glibc backtrace. it fixes some bugs with programs compiled with the Intel compiler or using Intel libraries (like Intel OpenMP) — but plot twist, it has a side effect, because libunwind has an internal memory leak. 🙈 the other approach is to instrument your code with -finstrument-function, and in that case you need to use enter-exit to use it.
-q, --quiet — make MALT silent so it doesn't mix with the default output of your program. super useful if you're using the output to feed a script. 🤫
-v, --verbose — add more infos about MALT running. 🗣️
-d, --dumpconfig — dump the current MALT configuration into a file with the same name as the profile file. it's a whole quick-start for configuring stuff, because you get all the options and their default value. 🔥
-o — lets you pass some specific options from the config file on the command line. the value is a list of options separated by commas, formatted like SECTION:NAME=VALUE. see the configuration file format (INI format) to find the available sections and names. 👑
--mpi — enable MPI support so the output files get named with rank IDs instead of process IDs. MALT will LD_PRELOAD the library bridge from ~/.malt/libmaltmpi.so. 🌈
--prep-mpi — build the MPI bridge library: ~/.malt/libmaltmpi.so. MALT requires it to get access to the MPI_Comm_rank function, depending on your MPI implementation. you need to do this command each time you switch your MPI implementation. each time, bestie. 💀
--wrap — lets you wrap and capture a custom memory allocator. the wrap option expects a list of functions to wrap separated by commas (or you can call it multiple times). the values should give the type of function (malloc, free, realloc, calloc...) and the symbol to wrap, on the form: {type}:{symbol}. 🧚
--wrap-prefix — automatically wrap all the memory management functions under the given prefix. you can list several prefixes separated by comma. for example, giving je_ will wrap all the je_malloc, je_free, je_realloc... functions. slay. 💜
--gdb — run the program inside GDB, and automatically configure GDB so it works properly with MALT. 🎀
-h, --help — display the help message with a list of available command line options. the classic. 💖

🌟 USING ON MPI APPLICATIONS 🌟

MALT natively provides a lightweight support of MPI applications. each process is tracked separately, but MALT can help by numbering the output files with the task rank instead of the process ID. to get this, you need to compile a small wrapper adapted to your current MPI implementation. you only need to do it once:

malt --prep-mpi [mpicxx]

then make sure to use the --mpi option on the profiling tool:

mpirun malt --mpi ./you_program

and that's the whole ritual. periodt. 🫶

🖼️ GET THE GUI 🖼️

with MALT you can use kcachegrind directly — but only if you enable the valgrind compatible format in the configuration (output:callgrind). ⭐

but by default? MALT generates a JSON file to be used with malt-webview(1). so yeah, the pretty pictures are one step away. 🌸

🎀 CONFIGURATION FILE 🎀

you can get an example of this config file with the -d option, and then set up the config file to use with the -c one. the config file uses INI format with sections and variables, as in this example:

[time]
enabled                        = true
points                         = 512
linear_index                   = false

[stack]
enabled                        = true
mode                           = backtrace
resolve                        = true
libunwind                      = false
skip                           = 4   sampling                       = false   samplingBw                     = 4093

[output]
name                           = malt-%1-%2.%3
lua                            = false
json                           = true
callgrind                      = false
indent                         = true
config                         = true
verbosity                      = default
stack-tree                     = false
loop-suppress                  = false

[max-stack]
enabled                        = true

[distr]
alloc-size                     = true
realloc-jump                   = true

[trace]
enabled                        = false

[info]
hidden                         = false

[filter]
exec                           =
childs                         = true
enabled                        = true   ranks                          =

[dump]
on-signal                      =
after-seconds                  = 0
on-sys-full-at                 =
on-app-using-rss               =
on-app-using-virt              =
on-app-using-req               =
on-thread-stack-using          =   watch-dog                      = false

[python] instru = true stack = enter-exit mix = false obj = true mem = true raw = true

[tools] nm = true nmMaxSize = 50M

⏰ the `time` section supports:

enabled — enable support of tracking state values over time, so you can build those cute time charts. 📈
points — define the number of points used to discretize the execution time of the application.
linear_index — don't use time to index data, but a linear value increased on each call. it might be interesting so you don't shrink intensive allocation steps on a long program that mostly doesn't do allocation over the run. 🦋

🥞 the `stack` section supports:

enabled: enable support of stack tracking.
mode: overridden by the -s option from the command line — it sets the stack tracking method to use. see the -s documentation for more details. the available values are enter-exit, backtrace, python, none. by default backtrace is used, because it works out of the box everywhere. it's slower than enter-exit, but this last one needs recompilation of the code with -finstrument-functions and only sees the libs recompiled with this option. python lets you project all the C allocation directly into the python stack domain, so it hides the C underwood — good to improve performance over python when you're not interested in the C details. 🐍
resolve: enable symbol resolution at the end of execution, to extract full names and source location if debug options are available. 🔍
libunwind: use the linunwind backtrace method instead of the one from glibc.
skip: in backtrace mode, the backtrace is called inside our instrumentation function, itself called by malloc/free... in order to skip our internal code we need to remove them. but on some compilers or distros, inlining and LTO configuration make wrong detection. this can be fixed by playing with this value to keep malloc/free/calloc as leaf for every calltree extracted by MALT. if you don't see malloc as leaf, you should increase this value. decrease it if you start to see MALT internal functions inside. notice that in some cases (Gentoo, Gcc-8) we start to see LTO totally trashing the malloc/free call in O3 mode, normally seen via backtrace. there is currently no other solution except recompiling in O0/O1 to avoid it, or maybe disabling LTO optimizations, or considering not having the exact location of the malloc itself. 😭
sampling: enable sampling mode for the stacks, by capturing only for a few mallocs. it's less accurate than a full profile, but it costs less in memory and CPU. in sampling mode, the stack is processed only when we've seen passed a few bytes allocated — otherwise we consider the last seen call stack. with python you should also enable the backtrace mode for solving stacks via python:stack=backtrace. 🎯
samplingBw: define the amount of data seen passed between two samples. ideally this should be a prime number to avoid some multiple base biases. 🔢

📦 the `output` section supports:

name: define the name of the profile file. %1 is replaced by the program name, %2 by the PID or MPI rank, and %3 by the extension. 💖
lua: enable output in LUA format (same structure as JSON files, but in LUA).
json: enable output of the default JSON file format.
callgrind: enable output of the compatibility format with callgrind/kcachegrind. it cannot contain all the data, but it can be used with compatible existing tools.
indent: enable indentations in the JSON/LUA files. useful for debugging, but it generates bigger files. 🫧
config: dump the config INI file.
verbosity: set the verbosity mode of MALT. by default it prints at start and end. you can use silent mode to disable any output — for example if you instrument a shell script parsing the output of child processes. you can also use verbose to get more debugging infos in case it doesn't work as expected, mostly at the symbol extraction step while dumping outputs. 🗣️
stack-tree: enable storage of the stacks as a tree inside the output file. it produces smaller files, but it requires conversion at storage time and loading time to stay compatible with the basic expected format. you can use this option to get smaller files — in one case it lowers a 600 MB file to 200 MB, just to give you an idea. 🌳
loop-suppress: enable recursive loop calls to be removed, providing a more simplified equivalent call stack. it helps reduce the size of profiles from applications using intensively this kind of call chain. in one case it lowers the file from 200 MB to 85 MB. it can help if nodejs failed to load the file because of the size. this parameter can also provide more readable stacks, since you don't care too much about how many times you cycle to call loops — you just want to see one of them. 🔁

🧱 the `max-stack` section support:

Continuing in the config file — the stack section has one more key:

enabled — flips tracking of stack size and memory used by functions on stacks on or off. Needs --finstrument-function on your code to actually provide data, bestie. ✨

The `distr` section support

alloc-size — generates a distribution of the allocated chunk size.
realloc-jump — generates a distribution of the realloc size jumps.

The `trace` section support

enabled — turns tracing on or off. Currently not used by the GUI — work in progress, iykyk. 🫠

The `info` section support

enabled — hides execution information. This removes some possibly sensitive information from the output file, like executable names, hostname and command options. It’s still recommended to take a look at the file, for example to replace paths with sed(1), since those might also need removing. This option targets some companies that might want to hide their internal applications when exchanging with external partners. 🕵️‍♀️

The `filter` section support

exe — enables filtering of executables so MALT is enabled for the ones you want and ignored otherwise. By default, an empty value enables MALT on all executables.
childs — whether to instrument children processes or not. By default it instruments all.
enabled — enables profiling by default. You can disable it so you can activate it via a C function call in the app whenever you want.
ranks — when running in MPI mode, instrument only the given ranks. The list is provided under the form 1,2-4,6.

The `dump` section — dump the profile whenever you want 🫶

on-signal — dumps on the given signal. Can be on or a comma-separated list of: SIGINT, SIGUSR1, SIGUSR2, ... Heads up: profiling will currently stop from this point, and the app will continue without profiling. To be fixed later (their “latter”, honestly same energy). You can get the list of available ones by using help or avail in place of the name.
after-seconds — dumps the profile after X seconds. Same caveat: profiling will currently stop from this point, and the app will continue without profiling. To be fixed later.
on-sys-full-at — dumps if the system memory is globally filled at x%. Use an empty string to disable. It considers free memory and cached memory as free. A good value in practice is more like 70% / 80% than going to 95%, because you need to leave room for cached memory and it will start swapping before that. Also consider that MALT itself adds memory on top of yours (and that’s considered in the % here). Values can be in %, K, M, G by ending with the corresponding character.
on-app-using-rss — dumps if the application reaches the given RSS limit. The value is given in % of the global memory available or in K, M, G. Empty to disable (default).
on-app-using-virt — dumps if the application reaches the given virtual memory limit. The value is given in % of the global memory available or in K, M, G. Empty to disable (default).
on-app-using-req — dumps if the application reaches the given requested memory limit. The value is given in % of the global memory available or in K, M, G. Empty to disable (default).
on-thread-stack-using — dumps if one of the thread stacks reaches the given limit. The value is given in % of the global memory available or in K, M, G. Empty to disable (default).
watch-dog — starts a thread that spies on the memory usage of the process and triggers the dump as soon as it sees it going high. This balances the fact that, normally, system and process memory is spied on only sometimes by MALT in normal conditions to keep the overhead low.

The `python` section — configure how to instrument Python 🐍

Notice that you need to build MALT with --enable-python for this to be effective.

instru — enable or disable Python instrumentation.
stack — select the stack instrumentation mode: either enter-exit, backtrace or none. By default enter-exit is faster, so you should use it. When enabling sampling, you need to use backtrace — the page says bactrace, and we love that for them — if you don’t want to pay unneeded overhead. You can also disable Python stack checking with none.
mix — by default, when disabled, the C and Python stacks are analysed independently, which means that if a Python function calls a C function you’ll see only the C call stack. mix lets you merge the two layers so you see that Python calls C. But it adds overhead on the analysis, of course, because of the extra work.
obj — analyse or ignore the object allocation domain of Python. This is interesting to ignore all the small allocs used by the language, as it would have been on the stack in C. It improves Python profiling performance a lot, but misses part of the memory consumption if your program stores lots of small objects for long times.
mem — same but for the mem allocation domain of Python. In principle you should let it stay enabled.
raw — same but for the raw allocation domain of Python, which is backed by the standard C malloc function. In principle you should let it stay enabled.

The `tools` section — configure some sub-tools MALT calls for analysis 🧰

nm: use to extract the source location of the global variables. If true (default) it is used, otherwise it is skipped.
nmMaxSize: by default it limits the size of the .so files on which to apply NM in order to keep a decent profile dumping time when running on large frameworks like PyTorch, which tends to load huge .so files in memory.

ENVIRONMENT 💖

You can also directly LD_PRELOAD the MALT library in your program if the wrapper script has some issues for you.

LD_PRELOAD={PREFIX}/lib/libmalt.so {YOUR_PROGRAM}

In that case you can set up some environment variables to configure MALT:

MALT_CONFIG — define the configuration file to use (equivalent of -c option).
MALT_STACK — define the stack mode to use (equivalent of -s option).
MALT_OPTIONS — list of options to transmit to MALT (equivalent of -o option, see its documentation for format).

ON SUB-PART OF YOUR PROGRAM 🎯

If you run on a really big program doing millions of allocations you might get a big overhead, and maybe you’re just interested in a sub-part of the program. You can do it by including malt/malt.h in your files and use maltEnable() and maltDisable() to control MALT on each thread. It’s also a nice way to detect leaks of sub-parts of your code.

#include <malt/malt.h>

int main()
{
     maltDisable();
     //ignored
     malloc(16);

     maltEnable();
     //tracked
     malloc(16);
}

You will need to link the libmalt-controler.so to get the default fake symbols when not using MALT. You can also just provide the two empty functions in your own dynamic library (not static).

If you have some allocation not under your control before your first call, you can disable MALT by default on threads using the filter:enabled option, then enable it by hand.

WRAPPING A CUSTOM ALLOCATOR 🧵

If your application uses a custom allocator with a different namespace than the default malloc, free... you can use the --wrap or --wrap-prefix options.

You can select in detail the function by doing:

malt --wrap malloc:je_malloc ./prgm
malt --wrap malloc:je_malloc,free:je_free,calloc:je_calloc,malloc:another_custom_malloc ./prgm

You can also simply use a common prefix for all by using (typically useful if you embed jemalloc with a custom symbol prefix):

malt --wrap-prefix je_
malt --wrap-prefix je_,another_custom_

VERSION

This is the manpage of MALT version 1.4.1.

MALT(1)

💅 MALT(1) 💅

🍓 NAME 🍓

🦋 SYNOPSIS 🦋

🌸 DESCRIPTION 🌸

✨ OPTIONS ✨

🌟 USING ON MPI APPLICATIONS 🌟

🖼️ GET THE GUI 🖼️

🎀 CONFIGURATION FILE 🎀

⏰ the time section supports:

🥞 the stack section supports:

📦 the output section supports:

🧱 the max-stack section support:

The distr section support

The trace section support

The info section support

The filter section support

The dump section — dump the profile whenever you want 🫶

The python section — configure how to instrument Python 🐍

The tools section — configure some sub-tools MALT calls for analysis 🧰