ramalama-run(1)

📦
version 0.25.0 · 6 versions
, in fedora 43, fedora 44, nixpkgs unstable

🦋 ramalama-run(1) 🦋

okay so hi bestie 💖 you've landed on ramalama-run(1), straight out of the General Commands Manual. This is the one that actually runs your AI model as a chatbot, and yes, it's giving main character energy.

🌸 NAME 🌸

ramalama-run — run specified AI Model as a chatbot 💬

✨ SYNOPSIS ✨

ramalama run [options] model [arg ...]

🚚 MODEL TRANSPORTS 🚚

okay so here's the whole vibe of where your model can come from, no cap:

  • URL based — prefixes: https://, http://, file:// — web site vibes: https://web.site/ai.model, file:///tmp/ai.model
  • HuggingFace — prefixes: huggingface://, hf://, hf.co/ — web site: huggingface.co
  • ModelScope — prefixes: modelscope://, ms:// — web site: modelscope.cn
  • Ollama — prefix: ollama:// — web site: ollama.com
  • rlcr — prefix: rlcr:// — web site: ramalama.com
  • OCI Container Registries — prefixes: oci://, docker:// — web site: opencontainers.org
  • Hosted API Providers (deprecated) — prefix: openai:// — web site: api.openai.com
🚨 bestie listen — the hosted API transports are deprecated and will be removed in a future release. Instead of ramalama run openai://<model>, use ramalama chat --url https://api.openai.com/v1 --api-key <key> --model <model>, which reaches the same endpoint without a transport. Heads up tho: the deprecated transport calls the OpenAI Responses API, while ramalama chat calls the chat completions API. Different APIs, same endpoint, don't get it twisted 🙈

Models can be specified using a shortname (e.g. tiny) which is resolved via shortnames.conf, or with an explicit transport prefix such as huggingface://, oci://, ollama://, https://, http://, or file://. Models in the <org>/<model> format without a prefix are pulled from Hugging Face.

The default transport can be overridden in the ramalama.conf file or via the RAMALAMA_TRANSPORT environment variable. For example, export RAMALAMA_TRANSPORT=huggingface changes RamaLama to use the Hugging Face transport for unqualified model names. 🎀

URL support means if a model is on a web site or even on your local system, you can run it directly. iconic, honestly ✨

Hosted API transports connect directly to the remote provider and bypass the local container runtime. In this mode, flags that tune local containers (for example --image, GPU settings, or --network) do not apply, and the provider's own capabilities and security posture govern the execution. URL support means if a model is on a web site or even on your local system, you can run it directly. 🫶

💅 OPTIONS 💅

  • --api=llama-stack | none — Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry. (default: none) The default can be overridden in the ramalama.conf file.
  • --authfile=path — Path to the authentication file for OCI registries. 🔑
  • --backend=auto | vulkan | rocm | cuda | sycl | openvino | cann | musa — GPU backend to use for inference (default: auto).

Available backends depend on the detected GPU hardware.

auto (default): Automatically selects the preferred backend based on your GPU:

  • AMD GPUs: vulkan (Linux/macOS) or rocm (WSL2)
  • NVIDIA GPUs: cuda; vulkan available as explicit option
  • Intel GPUs: vulkan (Linux/macOS) or sycl (WSL2); openvino available as explicit option
  • Ascend NPUs: cann
  • MUSA GPUs: musa
  • No GPU: vulkan (CPU fallback)

Platform-specific behavior:

  • On Linux/macOS, Vulkan provides broad compatibility and is preferred for AMD and Intel GPUs
  • On WSL2, Vulkan is a poor default, so vendor-specific backends (rocm, sycl) are preferred. It remains available, and --backend=vulkan still selects it. This covers both Windows, where containers run in the WSL2-backed machine, and ramalama running inside a WSL2 distro, which otherwise looks like Linux

Explicit backend selection:

  • vulkan: Use Vulkan-based inference (compatible with AMD, NVIDIA, Intel, and CPU)
  • rocm: Use AMD ROCm backend (AMD GPUs only)
  • cuda: Use NVIDIA CUDA backend (NVIDIA GPUs only)
  • sycl: Use Intel SYCL/oneAPI backend (Intel GPUs only)
  • openvino: Use Intel OpenVINO backend (Intel GPUs only); uses quay.io/ramalama/openvino
  • cann: Use Huawei CANN backend (Ascend NPUs only); uses quay.io/ramalama/cann
  • musa: Use Moore Threads MUSA backend (MUSA GPUs only); uses quay.io/ramalama/musa

Available choices: The allowed values for --backend are dynamically determined based on your detected GPU hardware. For example, on a system with an AMD GPU, only auto, vulkan, and rocm are available. ✨

Configuration: The default can be overridden in the ramalama.conf file or via the RAMALAMA_BACKEND environment variable.

Examples:

# Use auto-detection (default)
ramalama run granite

# Force Vulkan backend
ramalama run --backend vulkan granite

# Force ROCm backend on AMD GPU
ramalama run --backend rocm granite
  • --cache-reuse=BYTES — Minimum chunk size (in bytes) to attempt reusing from the cache via KV shifting. When omitted, llama-server uses its built-in default. ♻️
  • --color — Indicate whether to use color in the chat. Possible values are "never", "always" and "auto". (default: auto) 🌈
  • --ctx-size, -c — size of the prompt context. This option is also available as --max-model-len. Applies to llama.cpp and vllm regardless of alias (default: 0, 0 = loaded from model) 🧠
  • --device — Add a host device to the container. Optional permissions parameter can be used to specify device permissions by combining r for read, w for write, and m for mknod(2).

Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm

The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.

Pass --device=none to explicitly add no device to the container, e.g., for running a CPU-only performance comparison. 💀

  • --engine-args="args" — Add args to the podman or docker invocation (before the container image), after RamaLama-generated options and model bind mounts. The option may be specified multiple times; each value is shell-split and all tokens are passed to the engine in order. Use for extra --mount flags (for example multimodal projector files) or other engine-specific options. Shell-quoting rules match --runtime-args. 🔧
  • --env= — Set environment variables inside the container.

This option allows arbitrary environment variables that are available for the process to be launched inside the container. If an environment variable is specified without a value, the container engine checks the host environment for a value and sets the variable only if it is set on the host. 🌱

  • --help, -h — Show this help message and exit 🙋
  • --image=IMAGE — OCI container image to run with specified AI model. RamaLama defaults to using images based on the accelerator it discovers and the selected --backend. For example: quay.io/ramalama/ramalama. See the table below for all default images. The default image tag is based on the minor version of the RamaLama package. Version 0.25.0 of RamaLama pulls an image with a :0.25 tag from the quay.io/ramalama OCI repository. The --image option overrides this default.

The default can be overridden in the ramalama.conf file or via the RAMALAMA_IMAGE environment variable. export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2 tells RamaLama to use the quay.io/ramalama/aiimage:1.2 image. 🖼️

Note: The --backend option provides a higher-level way to select the appropriate image based on GPU type. Use --backend to select vulkan, rocm, cuda, sycl, or openvino backends, which will automatically choose the correct image. Use --image only when you need to override the image selection entirely.

Accelerated images:

Here's the vibe check on which image you get for which accelerator, bestie — RamaLama picks the right one for you, no cap ✨

  • CPU, Vulkan → quay.io/ramalama/ramalama
  • ROCm (AMD) → quay.io/ramalama/rocm
  • CUDA (NVIDIA) → quay.io/ramalama/cuda
  • Intel GPU (sycl) → quay.io/ramalama/intel-gpu
  • Intel GPU (openvino) → quay.io/ramalama/openvino
  • Asahi (Apple Silicon) → quay.io/ramalama/asahi
  • CANN (Ascend) → quay.io/ramalama/cann
  • MUSA (Moore Threads) → quay.io/ramalama/musa

And plot twist — the upstream llama.cpp "full" images from ghcr.io/ggml-org/llama.cpp are supported too 🫶 RamaLama literally auto-detects the image type and adjusts the container CLI accordingly. She's smart like that.

ramalama run --image ghcr.io/ggml-org/llama.cpp:full-vulkan MODEL
  • --interactive, -i — keep the convo going after handling stdin or prompt args, i.e. go into interactive chat mode. By default, when arguments or piped input are provided, the command exits after displaying the response. This flag lets you keep chatting interactively 💬
  • --keep-groups — pass --group-add keep-groups to podman (default: False). If a GPU device on the host system is accessible to the user via group access, this option leaks the groups into the container.
  • --keepalive — duration to keep a model loaded (e.g. 5m).
  • --logfile=path — log output to a file.
  • --max-tokens=integer — max number of tokens to generate. Set to 0 for unlimited output (default: 0). This parameter gets mapped to the appropriate runtime-specific parameter:
    • llama.cpp: -n parameter
    • MLX: --max-tokens parameter
    • vLLM: --max-model-len parameter (mapped via ctx_size)
  • --mcp=SERVER_URL — MCP (Model Context Protocol) servers to use for enhanced tool-calling capabilities. This option can be specified multiple times to connect to multiple MCP servers. Each server provides tools that can be automatically invoked during chat conversations 🔧
  • --model-draft — a draft model is a smaller, faster model that helps accelerate the decoding process of larger, more complex models, like Large Language Models (LLMs). It works by generating candidate sequences of tokens that the larger model then verifies and refines. This approach, often referred to as speculative decoding, can significantly improve the speed of inferencing by reducing the number of times the larger model needs to be invoked.

Use --runtime-args to pass the other draft model related parameters. Make sure the sampling parameters like top_k on the web UI are set correctly. 🎯

  • --name, -n — name of the container to run the Model in.
  • --ncmoe — keep the Mixture of Experts (MoE) weights of the first N layers in the CPU.
  • --network=none — set the network mode for the container.
  • --ngl — number of layers to store in VRAM: a number, auto, or all. When omitted, llama-server defaults to auto.
  • --oci-runtime — override the default OCI runtime used to launch the container. Container engines like Podman and Docker have their own default oci runtime that they use. Using this option, RamaLama will override these defaults.

On Nvidia based GPU systems, RamaLama defaults to using the nvidia-container-runtime. Use this option to override that selection.

  • --port, -p=port — port for AI Model server to listen on (default: 8080).

The default can be overridden in the ramalama.conf file.

  • --prefix — prefix for the user prompt (default: 🦭 > ).
  • --privileged — by default, RamaLama containers are unprivileged (=false) and cannot, for example, modify parts of the operating system. This is because by de‐fault a container is only allowed limited access to devices. A "privi‐leged" container is given the same access to devices as the user launch‐ing the container, with the exception of virtual consoles (/dev/tty\d+) when running in systemd mode (--systemd=always).

A privileged container turns off the security features that isolate the container from the host. Dropped Capabilities, limited devices, read-only mount points, Apparmor/SELinux separation, and Seccomp filters are all disabled. Due to the disabled security features, the privileged field should almost never be set as containers can easily break out of confinement.

Containers running in a user namespace (e.g., rootless containers) can‐not have more privileges than the user that launched them.

⚠️ Bestie, this one's giving "I can break out of confinement" energy. Do NOT just casually set it, fr fr.
  • --pull=policy — pull image policy. The default is missing.
    • always: Always pull the image and throw an error if the pull fails.
    • missing: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
    • never: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
    • newer: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
  • --rag= — specify path to Retrieval-Augmented Generation (RAG) database or an OCI Image containing a RAG database.
📝 Note: RAG support requires AI Models be run within containers, --nocontainer not supported. Docker does not support image mounting, meaning Podman support required.
  • --rag-image= — the image to use to process the RAG database specified by the --rag option. The image must contain the /usr/bin/rag_framework executable, which will create a proxy which embellishes client requests with RAG data before passing them on to the LLM, and returns the responses.
  • --runtime-args="args" — add args to the runtime (llama.cpp or vllm) invocation.
  • --seed= — specify a seed rather than using a random seed for model interaction.
  • --selinux=true — enable SELinux container separation (default: true).
  • --spec-draft-n-max=N — maximum number of tokens to draft per speculative decoding step (default: 3).
  • --spec-draft-n-min=N — minimum number of draft tokens to use for speculative decoding (default: 0).
  • --spec-draft-p-min=P — minimum speculative decoding probability, greedy threshold (default: 0.0).
  • --spec-type=TYPES — comma-separated list of speculative decoding types to enable. Available types: draft-simple, draft-eagle3, draft-mtp, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache. When omitted, speculative decoding is disabled.
  • --summarize-after=N — automatically summarize conversation history after N messages to prevent context growth. When enabled, ramalama will periodically condense older messages into a summary, keeping only recent messages and the summary. This prevents the context from growing indefinitely during long chat sessions. Set to 0 to disable (default: 4).
  • --temp="0.8" — temperature of the response from the AI Model. llama.cpp explains this as:

The lower the number is, the more deterministic the response.

The higher the number is the more creative the response is, but more likely to hallucinate when set too high.

Usage: Lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories. 🌡️

  • --thinking=BOOL — enable or disable thinking mode in reasoning models. Maps to --reasoning on or --reasoning off in llama-server. When omitted, llama-server defaults to auto. 🧠
  • --threads, -t — maximum number of cpu threads to use. The default is to use half the cores when more than 4 cores are available; otherwise, the default is 4 threads.
  • --tls-verify=true — require HTTPS and verify certificates when contacting OCI registries.

💖 DESCRIPTION 💖

Run the specified AI Model as a chat bot, bestie 🦋 RamaLama pulls the specified AI Model from the registry if it does not exist in local storage. By default, a prompt for a chat bot is started. When arguments or stdin are provided, they will be given to the AI Model and the output is returned. By default, the command exits after displaying the response, but you can use --interactive (-i) to continue to an interactive chat session after processing the initial prompt.

🎀 INTERACTIVE COMMANDS 🎀

When running in interactive chat mode (without arguments), these commands are available to you, queen 👑 All commands are case-insensitive (e.g., /CLEAR, /Clear, and /clear all work — iconic, honestly).

  • /help, help, ? — display help information showing all available commands and their descriptions.
  • /clear — clear the conversation history without exiting the chat session. This resets the context and allows starting a fresh conversation without restarting the container or connection. A confirmation message will be displayed when the history is cleared. ✨
  • /bye, exit — exit the chat session and close the connection. 🫶
  • /tool [question] — (only available when using --mcp) manually select which MCP tool to use for a question. Without this command, the AI automatically decides whether to use tools based on the question. 🔧
  • Ctrl + D — exit the chat session (EOF signal).

✨ EXAMPLES ✨

Okay bestie, this is the part where we actually do stuff. Live footage of Ramalama being iconic 💅

🦋 The classic: no args = instant chatbot

Just run it with a model and boom, you're chatting. That's it. That's the whole thing.

ramalama run granite
>

That lonely little > is the chatbot waiting for you to say something. Say something. She's waiting. 🥺

⏰ Local model, but only for 10 minutes

Already got a model downloaded and just want a lil limited-time hangout? --keepalive 10m and it dips after ten minutes.

ramalama run --keepalive 10m file:///tmp/mymodel
>

🌈 Multiple models, multiple ports, zero beef

Want more than one model running at the same time? Give this one its own port so nobody's fighting over the same address. Very demure, very mindful.

ramalama run --port 8081 granite
>

💌 Slide a prompt in through stdin, then keep the convo going

Pipe an initial prompt in with -i and it still sticks around for interactive chatting after. Chef's kiss. 😭

$ echo "Explain quantum computing" | ramalama run -i granite
[AI response...]
> Can you give me an example?
[AI response...]
> /bye

🌸 One-shot question, one-shot answer

No > prompt, no back-and-forth, just answer me and go.

ramalama run merlinite "when is the summer solstice"
The summer solstice, which is the longest day of the year, will happen on June ...

🎀 Custom prompt + a file piped in

This is the "read my code and tell me what it does" combo. Absolute staple. Notice the model ref has a :1.0 tag and a USER in the path — keep that exact, bestie.

cat file.py | ramalama run quay.io/USER/granite-code:1.0 'what does this program do?'

This program is a Python script that allows the user to interact with a terminal. ...
 [end of text]

🫶 Sending multiple lines at ONCE

Okay this one is a trick and it's so good. You can stack up several lines before sending by slapping a backslash \ at the end of each line. The chatbot won't see any of it until you stop backslashing. Sneaky. Gorgeous. 💜

$ ramalama run granite
🦭 > Hi
🦭 > tell me a funny story
🦭 > please

🧚 Wiping the conversation mid-session

Memory getting messy? /clear and it's like nothing ever happened. Fresh slate, fresh vibes. ✨

$ ramalama run granite
🦭 > What is the capital of France?
Paris
🦭 > /clear
Conversation history cleared.
🦭 > What is 2+2?
4

🚦 Exit Codes:

  • 0 — Success 🎉
  • 124 — RamaLama command did not exit within the keepalive time.
That 124 one is not a crash, okay?? It means the keepalive timer ran out and the command got yeeted. Not a bug, it's a feature, periodt. 💀

👑 NVIDIA CUDA Support

Want that GPU glow-up? Go read ramalama-cuda(7) for how to set up your host Linux system for CUDA support. That's the whole assignment. Go. Read it. 🫡

🔍 SEE ALSO

📜 HISTORY

Aug 2024, Originally compiled by Dan Walsh dwalsh@redhat.com <mailto:dwalsh@redhat.com>

And that's a wrap on ramalama-run(1) bestie!! 🌈💖 Source/version and date for this little masterpiece: ramalama-run(1), Aug 2024. Thank you for coming to my TED talk, now go run some models. Mwah. 💅🎀🦋

📦 where this page comes from