ramalama-perplexity(1)

📦
version 0.25.0 · 6 versions
, in fedora 43, fedora 44, nixpkgs unstable

🦋 ramalama-perplexity(1) — General Commands Manual 🦋

okay bestie, buckle up, because this is the manual page for ramalama-perplexity, and she is serving ✨

✨ NAME ✨

ramalama-perplexity — calculate the perplexity value of an AI Model 💅

no cap, that's literally the whole job. she takes a model, she gives you a perplexity number. icon behaviour.

🌸 SYNOPSIS 🌸

ramalama perplexity [options] model [arg ...]

that's the whole vibe. [options] first if you're feeling fancy, then your model, then whatever extra arg ... you wanna yeet in there. 🫶

🎀 MODEL TRANSPORTS 🎀

so like, how do you even point at a model? RamaLama speaks a bunch of different transport languages, and each one has a prefix that tells her where to look. here's the whole roster, no gatekeeping:

  • URL based — prefix: https://, http://, file:// — web site: https://web.site/ai.model, file:///tmp/ai.model
  • HuggingFace — prefix: huggingface://, hf://, hf.co/ — web site: huggingface.co
  • ModelScope — prefix: modelscope://, ms:// — web site: modelscope.cn
  • Ollama — prefix: ollama:// — web site: ollama.com
  • rlcr — prefix: rlcr:// — web site: ramalama.com
  • OCI Container Registries — prefix: oci://, docker:// — web site: opencontainers.org
  • Hosted API Providers (deprecated 😭) — prefix: openai:// — web site: api.openai.com
⚠️ okay so listen — the hosted API transports are deprecated and will be removed in a future release. do not get attached. instead of ramalama run openai://<model>, you use ramalama chat --url https://api.openai.com/v1 --api-key <key> --model <model>, which reaches the same endpoint without a transport. heads up tho: the deprecated transport calls the OpenAI Responses API, while ramalama chat calls the chat completions API. different APIs, same destination. iykyk 👑

so how do you actually name a model? two ways, bestie:

  1. shortname — like tiny, which gets resolved via shortnames.conf 🍓
  2. explicit transport prefix — like huggingface://, oci://, ollama://, https://, http://, or file://

and if you just write a model in the <org>/<model> format with no prefix at all? she pulls it from Hugging Face. that's the default, periodt.

you can also override the default transport in the ramalama.conf file or via the RAMALAMA_TRANSPORT environment variable. like, for example:

export RAMALAMA_TRANSPORT=huggingface

and boom 💥 RamaLama switches to using the Hugging Face transport for unqualified model names.

the way URL support means that if a model is on a web site — or literally just chillin' on your local system — you can run it directly. it's giving convenience. it's giving main character energy. 🌈

✨ OPTIONS ✨

okay bestie, this is the big one, so grab a snack 🍓 every single flag lives here, no skips, no cap.

  • --authfile=path — the path to your authentication file for OCI registries. iykyk, it's how you get in the door 🔑
  • --backend=auto | vulkan | rocm | cuda | sycl | openvino | cann | musa — which GPU backend to use for inference (default: auto). 💅

the ones you actually get to pick depend on the GPU hardware that got detected. so it's giving ✨bespoke✨.

auto (the default): picks the preferred backend based on your GPU, all on its own:

  • AMD GPUs: vulkan (Linux/macOS) or rocm (WSL2)
  • NVIDIA GPUs: cuda; vulkan available as explicit option
  • Intel GPUs: vulkan (Linux/macOS) or sycl (WSL2); openvino available as explicit option
  • Ascend NPUs: cann
  • MUSA GPUs: musa
  • No GPU: vulkan (CPU fallback)

Platform-specific behavior, bc it's complicated and we love that for it:

  • On Linux/macOS, Vulkan provides broad compatibility and is preferred for AMD and Intel GPUs
  • On WSL2, Vulkan is a poor default, so vendor-specific backends (rocm, sycl) are preferred. It remains available, and --backend=vulkan still selects it. This covers both Windows, where containers run in the WSL2-backed machine, and ramalama running inside a WSL2 distro, which otherwise looks like Linux

Explicit backend selection, when you wanna be the main character and just say it:

  • vulkan: Use Vulkan-based inference (compatible with AMD, NVIDIA, Intel, and CPU)
  • rocm: Use AMD ROCm backend (AMD GPUs only)
  • cuda: Use NVIDIA CUDA backend (NVIDIA GPUs only)
  • sycl: Use Intel SYCL/oneAPI backend (Intel GPUs only)
  • openvino: Use Intel OpenVINO backend (Intel GPUs only); uses quay.io/ramalama/openvino
  • cann: Use Huawei CANN backend (Ascend NPUs only); uses quay.io/ramalama/cann
  • musa: Use Moore Threads MUSA backend (MUSA GPUs only); uses quay.io/ramalama/musa

Available choices: the allowed values for --backend are dynamically determined based on your detected GPU hardware. like, on a system with an AMD GPU, only auto, vulkan, and rocm are available. that's it, that's the menu 🎀

Configuration: the default can be overridden in the ramalama.conf file or via the RAMALAMA_BACKEND environment variable.

Examples, bc examples are iconic:

# Use auto-detection (default)
ramalama perplexity granite

# Force Vulkan backend
ramalama perplexity --backend vulkan granite

# Force ROCm backend on AMD GPU
ramalama perplexity --backend rocm granite
  • --ctx-size, -c — size of the prompt context. this one is also available as --max-model-len (two names, one queen 👑). Applies to llama.cpp and vllm regardless of alias (default: 0, and 0 = loaded from model).
  • --device — add a host device to the container. there's an optional permissions parameter you can use to specify device permissions by combining r for read, w for write, and m for mknod(2).

Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm

the device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.

Pass --device=none to explicitly add no device to the container, e.g., for running a CPU-only performance comparison. 🧊

  • --engine-args="args" — add args to the podman or docker invocation (before the container image), after RamaLama-generated options and model bind mounts. The option may be specified multiple times; each value is shell-split and all tokens are passed to the engine in order. Use for extra --mount flags (for example multimodal projector files) or other engine-specific options. Shell-quoting rules match --runtime-args. 🧰
  • --env= — set environment variables inside the container.

This option allows arbitrary environment variables that are available for the process to be launched inside the container. If an environment variable is specified without a value, the container engine checks the host environment for a value and sets the variable only if it is set on the host.

  • --file — File containing the sample prompt - see also get-wikitext-2.sh
  • --help, -h — Show this help message and exit 🫶
  • --image=IMAGE — OCI container image to run with specified AI model. RamaLama defaults to using images based on the accelerator it discovers and the selected --backend. For example: quay.io/ramalama/ramalama. See the list below for all default images. The default image tag is based on the minor version of the RamaLama package. Version 0.25.0 of RamaLama pulls an image with a :0.25 tag from the quay.io/ramalama OCI repository. The --image option overrides this default.

The default can be overridden in the ramalama.conf file or via the RAMALAMA_IMAGE environment variable. export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2 tells RamaLama to use the quay.io/ramalama/aiimage:1.2 image.

Note: The --backend option provides a higher-level way to select the appropriate image based on GPU type. Use --backend to select vulkan, rocm, cuda, sycl, or openvino backends, which will automatically choose the correct image. Use --image only when you need to override the image selection entirely.

Accelerated images (the whole roster, nobody left out 🦋):

  • CPU, Vulkan → quay.io/ramalama/ramalama
  • ROCm (AMD) → quay.io/ramalama/rocm
  • CUDA (NVIDIA) → quay.io/ramalama/cuda
  • Intel GPU (sycl) → quay.io/ramalama/intel-gpu
  • Intel GPU (openvino) → quay.io/ramalama/openvino
  • Asahi (Apple Silicon) → quay.io/ramalama/asahi
  • CANN (Ascend) → quay.io/ramalama/cann
  • MUSA (Moore Threads) → quay.io/ramalama/musa

Upstream llama.cpp "full" images from ghcr.io/ggml-org/llama.cpp are also supported. RamaLama automatically detects the image type and adjusts the container CLI accordingly.

ramalama perplexity --image ghcr.io/ggml-org/llama.cpp:full-vulkan MODEL
  • --keep-groups — pass --group-add keep-groups to podman (default: False). If GPU device on host system is accessible to user via group access, this option leaks the groups into the container. 🫣
  • --max-tokens=integer — Maximum number of tokens to generate. Set to 0 for unlimited output (default: 0). This parameter is mapped to the appropriate runtime-specific parameter:
    • llama.cpp: -n parameter
    • MLX: --max-tokens parameter
    • vLLM: --max-model-len parameter (mapped via ctx_size)
  • --name, -n — Name of the container to run the Model in. name it something cute, that's allowed 💖
  • --ncmoe — Keep the Mixture of Experts (MoE) weights of the first N layers in the CPU.
  • --network=none — set the network mode for the container
  • --ngl — Number of layers to store in VRAM: a number, auto, or all. When omitted, llama-server defaults to auto.
  • --oci-runtime — Override the default OCI runtime used to launch the container. Container engines like Podman and Docker, have their own default oci runtime that they use. Using this option RamaLama will override these defaults.

On Nvidia based GPU systems, RamaLama defaults to using the nvidia-container-runtime. Use this option to override this selection.

  • --privileged — By default, RamaLama containers are unprivileged (=false) and cannot, for example, modify parts of the operating system. This is because by de‐fault a container is only allowed limited access to devices. A "privi‐leged" container is given the same access to devices as the user launch‐ing the container, with the exception of virtual consoles (/dev/tty\d+) when running in systemd mode (--systemd=always).

A privileged container turns off the security features that isolate the container from the host. Dropped Capabilities, limited devices, read-only mount points, Apparmor/SELinux separation, and Seccomp filters are all disabled. Due to the disabled security features, the privileged field should almost never be set as containers can easily break out of confinement.

Containers running in a user namespace (e.g., rootless containers) can‐not have more privileges than the user that launched them.

⚠️ bestie, for real, this one is the scary one. it turns off the isolation. do NOT just vibe-set this one. 💀
  • --pull=policy — Pull image policy. The default is missing.
    • always: Always pull the image and throw an error if the pull fails.
    • missing: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
    • never: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
    • newer: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
  • --runtime-args="args" — Add args to the runtime (llama.cpp or vllm) invocation.
  • --seed= — Specify a seed rather than using a random seed for model interaction. same seed, same vibe, every time 🎲
  • --selinux=true — Enable SELinux container separation (default: true)
  • --temp="0.8" — Temperature of the response from the AI Model. llama.cpp explains this as:
The lower the number is, the more deterministic the response.
The higher the number is the more creative the response is, but more likely to hallucinate when set too high.

Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories. 🌸

  • --threads, -t — Maximum number of cpu threads to use. The default is to use half the cores when more than 4 cores are available; otherwise, the default is 4 threads.
  • --tls-verify=true — Require HTTPS and verify certificates when contacting OCI registries 🔒

✨ DESCRIPTION ✨

Okay bestie, so here's the whole point of this whole program: it calculates the perplexity of an AI Model. 💖

And what even is perplexity, you ask? It's literally a measure of how well the model can predict the next token. That's it. That's the metric. And the way it works out is that lower values are better — like, a low perplexity score means the model is out here guessing the next token correctly and living its best life, and a high one means it's kinda flailing. 🙈

So basically: model predicts next token well → low number → slay. Model confused → high number → not slay. Periodt. 💅

🎀 EXAMPLES 🎀

Here's the one and only glow-up moment you need, fr fr:

ramalama perplexity granite3-moe

That's it. Run it, get your perplexity number for granite3-moe, vibe check complete. ⭐

🌸 SEE ALSO 🌸

  • ramalama(1) — the main character of the whole manpage cinematic universe, go read it 🫶

💖 HISTORY 💖

  • Jan 2025 — Originally compiled by Eric Curtin ecurtin@redhat.com ⟨mailto:ecurtin@redhat.com⟩ 🌈

And that's the whole page, babe! 🧚 The little footer at the bottom of all of this is just the page name ramalama-perplexity(1) — that's your source/version tag right there, no cap. Go forth and measure some perplexity, you absolute icon. 👑💜

📦 where this page comes from