ramalama-bench(1)

📦
version 0.25.0 · 6 versions
, in fedora 43, fedora 44, nixpkgs unstable

💖 ramalama-bench 💖

okay bestie, welcome to the ramalama-bench manual page — that's the General Commands Manual one, and it's literally about benchmarking a specific AI Model. slay. let's get into it 🫶

✨ NAME ✨

ramalama-bench — benchmark specified AI Model

✨ SYNOPSIS ✨

ramalama bench [options] model [arg ...]

🌈 MODEL TRANSPORTS 🌈

okay so here's the tea on how you can point RamaLama at a model. every transport has a prefix you slap on, and there's a place on the web where it lives. here's the whole vibe check, one by one 💅

  • URL based — prefixes: https://, http://, file:// — web site: https://web.site/ai.model, file:///tmp/ai.model
  • HuggingFace — prefixes: huggingface://, hf://, hf.co/ — web site: huggingface.co
  • ModelScope — prefixes: modelscope://, ms:// — web site: modelscope.cn
  • Ollama — prefix: ollama:// — web site: ollama.com
  • rlcr — prefix: rlcr:// — web site: ramalama.com
  • OCI Container Registries — prefixes: oci://, docker:// — web site: opencontainers.org
  • Hosted API Providers (deprecated) — prefix: openai:// — web site: api.openai.com
🚨 not me having to break the bad news 🚨 the hosted API transports are deprecated and will be removed in a future release. so instead of doing ramalama run openai://<model>, you should do ramalama chat --url https://api.openai.com/v1 --api-key <key> --model <model> — that reaches the exact same endpoint without a transport. and heads up bestie: the deprecated transport calls the OpenAI Responses API, while ramalama chat calls the chat completions API. different APIs, okay? don't come for me, that's just what the page says 😭

Models can be specified using a shortname (e.g. tiny) which is resolved via shortnames.conf, or with an explicit transport prefix such as huggingface://, oci://, ollama://, https://, http://, or file://. and if you're giving a model in the <org>/<model> format with no prefix at all? it gets pulled from Hugging Face, periodt. 👑

The default transport can be overridden in the ramalama.conf file or via the RAMALAMA_TRANSPORT environment variable. for example, export RAMALAMA_TRANSPORT=huggingface changes RamaLama to use the Hugging Face transport for unqualified model names. literally one env var and the whole vibe shifts ✨

URL support means if a model is on a web site or even on your local system, you can run it directly. iconic behavior honestly 🦋

✨ OPTIONS ✨

okay bestie, this is the part where we go option by option, and there are a LOT of them, but every single one is here, no skips, no shortcuts, we're eating this whole menu 🍽️

  • --authfile=path — the path to that authentication file you need for OCI registries. ya know, the little key to the club 🔑
  • --backend=auto | vulkan | rocm | cuda | sycl | openvino | cann | musa — which GPU backend we're using for inference (default: auto).

Which backends you can even pick depends on the GPU hardware we detect. so like, it's not a free-for-all, it's curated ✨

auto (the default) does the choosing for you, based on your GPU:

  • AMD GPUs: vulkan (Linux/macOS) or rocm (WSL2)
  • NVIDIA GPUs: cuda; vulkan is available as an explicit option
  • Intel GPUs: vulkan (Linux/macOS) or sycl (WSL2); openvino is available as an explicit option
  • Ascend NPUs: cann
  • MUSA GPUs: musa
  • No GPU: vulkan (CPU fallback)

Platform-specific behavior, lowkey important:

  • On Linux/macOS, Vulkan gives broad compatibility and is preferred for AMD and Intel GPUs.
  • On WSL2, Vulkan is a poor default, so vendor-specific backends (rocm, sycl) are preferred. It remains available, and --backend=vulkan still selects it. This covers both Windows, where containers run in the WSL2-backed machine, and ramalama running inside a WSL2 distro, which otherwise looks like Linux.

Explicit backend selection, for when you know exactly what you want:

  • vulkan: Use Vulkan-based inference (compatible with AMD, NVIDIA, Intel, and CPU)
  • rocm: Use AMD ROCm backend (AMD GPUs only)
  • cuda: Use NVIDIA CUDA backend (NVIDIA GPUs only)
  • sycl: Use Intel SYCL/oneAPI backend (Intel GPUs only)
  • openvino: Use Intel OpenVINO backend (Intel GPUs only); uses quay.io/ramalama/openvino
  • cann: Use Huawei CANN backend (Ascend NPUs only); uses quay.io/ramalama/cann
  • musa: Use Moore Threads MUSA backend (MUSA GPUs only); uses quay.io/ramalama/musa

Available choices: the allowed values for --backend are dynamically determined based on your detected GPU hardware. So for example, on a system with an AMD GPU, only auto, vulkan, and rocm are available. it's giving bespoke 💅

Configuration: the default can be overridden in the ramalama.conf file or via the RAMALAMA_BACKEND environment variable.

Examples:

# Use auto-detection (default)
ramalama bench granite

# Force Vulkan backend
ramalama bench --backend vulkan granite

# Force ROCm backend on AMD GPU
ramalama bench --backend rocm granite
  • --device — adds a host device to the container. you can optionally pass a permissions parameter to specify device permissions by combining r for read, w for write, and m for mknod(2).

Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm

The device specification gets passed straight through to the underlying container engine, no filter, so go read the docs of whatever container engine you're using for the full tea.

Pass --device=none to explicitly add no device to the container, e.g., for running a CPU-only performance comparison. cute, minimal, focused 🎀

  • --engine-args="args" — add args to the podman or docker invocation (before the container image), after RamaLama-generated options and model bind mounts. The option may be specified multiple times; each value is shell-split and all tokens are passed to the engine in order. Use it for extra --mount flags (for example multimodal projector files) or other engine-specific options. Shell-quoting rules match --runtime-args. stacking them up like a little tower of flags 🏗️
  • --env= — set environment variables inside the container.

This option allows arbitrary environment variables that are available for the process to be launched inside the container. If an environment variable is specified without a value, the container engine checks the host environment for a value and sets the variable only if it is set on the host. sneaky but respectful 🙈

  • --format — set the output format of the benchmark results. Options include json and table (default: table). pick your aesthetic, bestie 📊
  • --help, -h — show this help message and exit. the classic escape hatch 🚪
  • --image=IMAGE — the OCI container image to run with the specified AI model. RamaLama defaults to using images based on the accelerator it discovers and the selected --backend. For example: quay.io/ramalama/ramalama. See the table below for all default images. The default image tag is based on the minor version of the RamaLama package. Version 0.25.0 of RamaLama pulls an image with a :0.25 tag from the quay.io/ramalama OCI repository. The --image option overrides this default.

The default can be overridden in the ramalama.conf file or via the RAMALAMA_IMAGE environment variable. export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2 tells RamaLama to use the quay.io/ramalama/aiimage:1.2 image.

Note: the --backend option provides a higher-level way to select the appropriate image based on GPU type. Use --backend to select vulkan, rocm, cuda, sycl, or openvino backends, which will automatically choose the correct image. Use --image only when you need to override the image selection entirely.

Accelerated images, the whole lineup:

  • CPU, Vulkan → quay.io/ramalama/ramalama
  • ROCm (AMD) → quay.io/ramalama/rocm
  • CUDA (NVIDIA) → quay.io/ramalama/cuda
  • Intel GPU (sycl) → quay.io/ramalama/intel-gpu
  • Intel GPU (openvino) → quay.io/ramalama/openvino
  • Asahi (Apple Silicon) → quay.io/ramalama/asahi
  • CANN (Ascend) → quay.io/ramalama/cann
  • MUSA (Moore Threads) → quay.io/ramalama/musa

Upstream llama.cpp "full" images from ghcr.io/ggml-org/llama.cpp are also supported. RamaLama automatically detects the image type and adjusts the container CLI accordingly.

ramalama bench --image ghcr.io/ggml-org/llama.cpp:full-vulkan MODEL
  • --keep-groups — pass --group-add keep-groups to podman (default: False). If the GPU device on the host system is accessible to the user via group access, this option leaks the groups into the container. sharing the group chat membership fr fr 🫂
  • --name, -n — the name of the container to run the Model in.
  • --ncmoe — keep the Mixture of Experts (MoE) weights of the first N layers in the CPU. letting the CPU hold onto a lil slice of the brain 🧠
  • --network=none — set the network mode for the container.
  • --ngl — number of layers to store in VRAM: a number, auto, or all. When omitted, llama-server defaults to auto. let the server decide, it's got main character energy 👑
  • --oci-runtime — override the default OCI runtime used to launch the container. Container engines like Podman and Docker have their own default OCI runtime that they use. Using this option, RamaLama will override these defaults.

On Nvidia based GPU systems, RamaLama defaults to using the nvidia-container-runtime. Use this option to override that selection.

  • --privileged — by default, RamaLama containers are unprivileged (=false) and cannot, for example, modify parts of the operating system. This is because by de‐fault a container is only allowed limited access to devices. A "privi‐leged" container is given the same access to devices as the user launch‐ing the container, with the exception of virtual consoles (/dev/tty\d+) when running in systemd mode (--systemd=always).

A privileged container turns off the security features that isolate the container from the host. Dropped Capabilities, limited devices, read-only mount points, Apparmor/SELinux separation, and Seccomp filters are all disabled. Due to the disabled security features, the privileged field should almost never be set as containers can easily break out of confinement.

Containers running in a user namespace (e.g., rootless containers) can‐not have more privileges than the user that launched them.

bestie, please hear me on this one: this is the almost never option. the walls come down. it's not cute, it's not worth it unless you truly know why you need it. 🚨
  • --pull=policy — pull image policy. The default is missing.
    • always: Always pull the image and throw an error if the pull fails.
    • missing: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
    • never: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
    • newer: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
  • --runtime-args="args" — add args to the runtime (llama.cpp or vllm) invocation. get in there, runtime 🛠️
  • --seed= — specify a seed rather than using a random seed for model interaction. lock in the chaos, make it reproducible 🎲
  • --selinux=true — enable SELinux container separation (default: true). keeping things safely separated, as they should be ✨
  • --threads, -t — maximum number of cpu threads to use. The default is to use half the cores when more than 4 cores are available; otherwise, the default is 4 threads. sharing the CPU workload, very mindful, very demure 💅
  • --tls-verify=true — require HTTPS and verify certificates when contacting OCI registries. safety first, always 🔒

📝 DESCRIPTION 📝

Benchmark specified AI Model.

that's it. that's the whole job. run the model, get the numbers, feel iconic 📈

🌈 EXAMPLES 🌈

ramalama bench granite3-moe

look at her go 🦋

👀 SEE ALSO 👀

📜 HISTORY 📜

Jan 2025, Originally compiled by Eric Curtin ecurtin@redhat.com ⟨mailto:ecurtin@redhat.com⟩

okay bestie that's a wrap on ramalama-bench(1) 💖 the footer down there just says ramalama-bench(1) (and the history stamps it Jan 2025), so if you ever forget where you are in the terminal, that little tag at the bottom is the vibe check. go benchmark something and slay 🎀✨

📦 where this page comes from