ramalama.conf(5)

📦
version 0.25.0 · 6 versions
, in fedora 43, fedora 44, nixpkgs unstable

✨ ramalama.conf ✨

okay bestie, welcome to ramalama.conf(5) — the section of the manual that's literally just about the config files that set your default RamaLama options and CLI flag values 💅 no cap.

🌸 NAME 🌸

ramalama.conf — the config files that define default RamaLama options and CLI flag values. that's it, that's the whole job 🫶

💖 DESCRIPTION 💖

so the vibe here is: you use ramalama.conf to set defaults for RamaLama commands, and then you never have to type the same flag a million times like a peasant 👑

RamaLama reads config files in order and merges the values. and because the universe loves drama, later files override earlier ones. so the last one to speak wins, periodt 💀

🌈 Global configuration paths

RamaLama reads these global config paths:

  • /usr/share/ramalama/ramalama.conf — Linux
  • /usr/local/share/ramalama/ramalama.conf — Linux
  • /etc/ramalama/ramalama.conf — Linux
  • /etc/ramalama/ramalama.conf.d/*.conf — Linux
  • $HOME/.local/pipx/venvs/ramalama/share/ramalama/ramalama.conf — pipx install

🦋 User configuration paths

and these are the user config paths:

  • $XDG_DATA_HOME/ramalama/ramalama.conf — Preferred data path
  • $XDG_DATA_HOME/ramalama/ramalama.conf.d/*.conf — Additional data snippets
  • $HOME/.local/share/ramalama/ramalama.conf — Used when $XDG_DATA_HOME is unset
  • $HOME/.local/share/ramalama/ramalama.conf.d/*.conf — Used when $XDG_DATA_HOME is unset
  • $XDG_CONFIG_HOME/ramalama/ramalama.conf — Preferred user path
  • $XDG_CONFIG_HOME/ramalama/ramalama.conf.d/*.conf — Additional user snippets
  • $HOME/.config/ramalama/ramalama.conf — Used when $XDG_CONFIG_HOME is unset
  • $HOME/.config/ramalama/ramalama.conf.d/*.conf — Used when $XDG_CONFIG_HOME is unset

🎀 Notes

bestie pls remember these two things, they WILL bite you 🙈
  • Files in .d directories are loaded in alphanumeric order.
  • Files in .d directories must end with .conf.

👑 ENVIRONMENT VARIABLES 👑

if RAMALAMA_CONFIG is set, RamaLama literally ignores all the default system/user config paths and loads only the specified file. main character energy, honestly — it refuses to look at anybody else ✨

🍓 FORMAT 🍓

RamaLama uses TOML. and here's the tea: every setting is nested under a table, so there are no bare options allowed. no floating little option = "value" orphans hanging around outside a table, we don't do that here 💅

[table1]
option = "value"

[table2]
option = "value"

[table3.subtable1]
option = "value"

see? nested, tidy, iconic. tables within tables within tables, like a lil matryoshka of config 🧚

(more coming bestie, don't go anywhere) 💜

🦋 RAMALAMA TABLE 🦋

okay bestie, this table is where all your default runtime + CLI settings live. main character energy fr fr.

[ramalama]

✨ Core runtime options

  • api="none" — the Unified API layer that handles Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry. your options are: llama-stack, none. 💅
  • api_key="" — the OpenAI-compatible API key. you can also set this via RAMALAMA_API_KEY, no cap. 🔑

🎀 Model conversion options

  • carimage="registry.access.redhat.com/ubi10-micro:latest" — the OCI model car image used when building and pushing --type=car models. 🚗
  • container=true — runs RamaLama in a container by default. override it via RAMALAMA_IN_CONTAINER.
  • convert_type="raw" — the default object type when you convert models. options: artifact, car, raw. here's the breakdown so you're not lost:
    • artifact — Store AI models as OCI artifacts
    • car — Traditional OCI image with model under /models
    • raw — OCI image with only the model and model.file link file at /
  • ctx_size=0 — Prompt context size. 0 means use the model default. simple as that. ✨

🌸 Container and engine options

  • engine="podman" — Container engine. valid options: podman, docker. override via RAMALAMA_CONTAINER_ENGINE. 🐳
  • env=[] — Environment variables added to the container runtime environment. Example: "LLAMA_ARG_THREADS=10".
  • host="127.0.0.1" — IP address for the model server to listen on. Defaults to "127.0.0.1", so the served model is only reachable from the local machine. Set to a wildcard address such as "0.0.0.0" (IPv4) or "::" (dual-stack) to expose it on the network.
⚠️ bestie, exposing it on the network means it's on the network. you've been warned 💀
  • image="quay.io/ramalama/ramalama:latest" — OCI container image to run with the specified AI model. the RAMALAMA_IMAGE environment variable overrides this field. 👑

🫶 [ramalama.images]

User-override entries for runtime-specific container images.

  • keep_groups=false — Pass --group-add keep-groups when using podman. Useful in some rootless GPU access setups.
  • log_level=warning — Logging level. valid values: debug, info, warning, error, critical. --debug overrides this field and forces debug logging.
  • max_tokens=0 — Maximum number of tokens to generate. Set to 0 for unlimited output.

💖 Serving options

  • prefix="" — Default prompt prefix for chat and run. it changes based on your setup, so:
    • Podman → "🦭 > "
    • Docker → "🐋 > "
    • No Engine → "🦙 > "
    • No emoji support → "> "
  • port="8080" — Initial port for service allocation. RamaLama attempts a range of 101 ports starting from this value.
  • pull="newer" — Pull policy for runtime images. the tea ☕:
    • always: Always pull, fail on pull error.
    • missing: Pull only when local image is missing.
    • never: Never pull; use local image only.
    • newer: Pull when remote digest differs; suppress pull errors if local image exists.

🧚 RAG and storage options

  • rag_image="quay.io/ramalama/ramalama-rag" — OCI container image used for RAG processing (doc2rag and rag_framework). Can also be overridden with the --rag-image flag on the command line or the RAMALAMA_RAG_IMAGE environment variable.
  • stack_image="quay.io/ramalama/llama-stack" — OCI container image used for running llama-stack server. Can also be overridden with the --stack-image flag on the command line or the RAMALAMA_STACK_IMAGE environment variable.

⭐ [ramalama.tools_images]

User-override entries for GPU-specific tools container images used for GGUF conversion. Built-in GPU defaults (CUDA, ROCm, Intel) are defined internally; entries here override those defaults:

  • CUDA_VISIBLE_DEVICES = "quay.io/ramalama/cuda-tools"
  • HIP_VISIBLE_DEVICES = "quay.io/ramalama/rocm-tools"
  • INTEL_VISIBLE_DEVICES = "quay.io/ramalama/intel-gpu-tools"

Can also be overridden with the --tools-image flag on the command line or the RAMALAMA_TOOLS_IMAGE environment variable.

  • runtime="llama.cpp" — Inference runtime. options: llama.cpp, vllm, mlx.
  • selinux=false — Enable SELinux container separation enforcement.
  • store="$HOME/.local/share/ramalama" — Directory where AI models and data are stored.
  • tempdir="" — Directory for temporary files used by operations such as convert and RAG. When set in ramalama.conf, this value is applied as TMPDIR and overrides the host environment. When unset, the host TMPDIR is used. On non-Windows systems, if TMPDIR is unset or empty, /var/tmp is used.
  • summarize_after=4 — Automatically summarize chat history after N messages to limit context growth. Set to 0 to disable.

🌈 Transport and HTTP options

  • transport="" — Default transport used for unqualified model names. options: huggingface, oci, ollama. When unset, unqualified names are resolved via shortnames.conf. override via RAMALAMA_TRANSPORT.

🍓 [ramalama.http_client]

  • max_retries=5 — Maximum number of retries for failed downloads.
  • max_retry_delay=30 — Maximum delay (seconds) between retry attempts.

💜 RAMALAMA.PROVIDER TABLE 💜

The ramalama.provider table configures hosted API providers.

⚠️ Deprecated: the entire ramalama.provider.<name> namespace will be removed in a future release, together with the hosted API transports it configures. Use ramalama chat --url <endpoint> --api-key <key> --model <model> instead. like... it's giving end of an era 😭

🌸 [ramalama.provider]

  • openai — Configuration block for the OpenAI hosted provider. Deprecated, see above.

🎀 [ramalama.provider.openai]

  • api_key="" — ramalama.provider.openai.api_key overrides RAMALAMA_API_KEY when set. Deprecated.

🔥 RAMALAMA.BENCHMARKS TABLE 🔥

The ramalama.benchmarks table contains benchmark settings. ate, no crumbs.

[ramalama.benchmarks]
  • storage_folder defaults to <default store>/benchmarks.

By default, benchmark results are saved in the configured store directory under benchmarks/. If you change ramalama.store, set ramalama.benchmarks.storage_folder explicitly if needed.

🦋 RAMALAMA.USER TABLE 🦋

The ramalama.user table contains user preferences. it's giving personalization 💅

[ramalama.user]
  • no_missing_gpu_prompt=false — When no_missing_gpu_prompt = true, RamaLama suppresses the interactive prompt on macOS Podman VMs without GPU acceleration (for example, applehv). This value can also be set via RAMALAMA_USER__NO_MISSING_GPU_PROMPT.

👑 RAMALAMA.RUNTIMES TABLE 👑

Runtime-specific configuration. Each runtime plugin defines its own config section under ramalama.runtimes.<name>. The key name uses underscores (e.g. llama_cpp for the "llama.cpp" runtime). iykyk.

[ramalama.runtimes.llama_cpp]
  • backend="auto" — GPU backend to use for inference. This setting affects which container image is selected and how GPU resources are utilized.

Valid options: auto, vulkan, rocm, cuda, sycl, openvino, cann, musa.

  • auto (default): Automatically selects the preferred backend based on detected GPU:
    • AMD GPUs: vulkan (Linux/macOS) or rocm (WSL2)
    • NVIDIA GPUs: cuda; vulkan available as explicit option
    • Intel GPUs: vulkan (Linux/macOS) or sycl (WSL2); openvino available as explicit option
    • Ascend NPUs: cann
    • MUSA GPUs: musa
    • No GPU: vulkan (CPU fallback)
  • vulkan: Use Vulkan-based inference (compatible with AMD, NVIDIA, Intel, and CPU)
  • rocm: Use AMD ROCm backend (AMD GPUs only)
  • cuda: Use NVIDIA CUDA backend (NVIDIA GPUs only)
  • sycl: Use Intel SYCL/oneAPI backend (Intel GPUs only)
  • openvino: Use Intel OpenVINO backend (Intel GPUs only); uses quay.io/ramalama/openvino
  • cann: Use Huawei CANN backend (Ascend NPUs only); uses quay.io/ramalama/cann
  • musa: Use Moore Threads MUSA backend (MUSA GPUs only); uses quay.io/ramalama/musa

Platform-specific behavior: vulkan is not supported on WSL2, so vendor-specific backends (rocm for AMD, sycl for Intel) are automatically preferred when using backend="auto". This covers both Windows and ramalama running inside a WSL2 distro, which otherwise looks like Linux.

Example configuration:

ramalama.runtimes.llama_cpp
backend = "vulkan"  # Force Vulkan for all GPUs
  • cache_reuse=256 — Min chunk size to attempt reusing from the cache via KV shifting.
  • gguf_quantization_mode="Q4_K_M" — Quantization mode used when creating OCI-formatted AI models. options: Q2_K, Q3_K_S, Q3_K_M, Q3_K_L, Q4_0, Q4_K_S, Q4_K_M, Q5_0, Q5_K_S, Q5_K_M, Q6_K, Q8_0.
  • ngl=-1 — Number of layers to offload to the GPU. Set to -1 to offload all layers.
  • ncmoe=0 — Keep the Mixture of Experts (MoE) weights of the first N layers in the CPU.
  • temp="0.8" — Response sampling temperature.
    • Lower values: more deterministic output
    • Higher values: more creative output (higher hallucination risk)
  • thinking=true — Enable/disable thinking mode in reasoning models.
  • threads=4 — Number of CPU threads to use for inference. Default is half the available CPU cores (minimum 4).

💖 [ramalama.runtimes.mlx]

  • temp="0.8" — Response sampling temperature.
    • Lower values: more deterministic output
    • Higher values: more creative output (higher hallucination risk)

and that's the whole page, bestie 💅 go forth and configure responsibly. this little masterpiece is from the AI set, it's titled RamaLama, and it lives at section ramalama.conf(5). slay 🫶✨

📦 where this page comes from