ramalama(1)

📦
version 0.25.0 · 6 versions
, in fedora 43, fedora 44, nixpkgs unstable

🎀 ramalama 🎀

okay bestie, welcome to the ramalama(1) page — General Commands Manual energy, but make it cute. this is THE tool for working with AI Models locally, and honestly? it's giving "boring on purpose, iconic by accident." 💅

✨ NAME ✨

ramalama — Simple management tool for working with AI Models

✨ SYNOPSIS ✨

ramalama [options] command

like… that's it. options, then a command. so simple. slay. 🫶

✨ DESCRIPTION ✨

so the whole goal of RamaLama is literally to make AI boring. no cap. that's the mission statement. 🔥

The RamaLama tool facilitates local management and serving of AI Models.

On first run, RamaLama inspects your system for GPU support, falling back to CPU support if no GPUs are present. it does the vibe check on ur hardware for u, basically. 💖

RamaLama uses container engines like Podman or Docker to pull the appropriate OCI image with all of the software necessary to run an AI Model for your systems setup. no more "help why is my python env cursed" era, we are OVER that. 😭

Running in containers eliminates the need for users to configure the host system for AI. After initialization, RamaLama runs AI Models within containers based on OCI images. RamaLama pulls a container image specific to the GPUs discovered on the host system. These images are tied to the minor version of RamaLama. For example, RamaLama version 1.2.3 on an NVIDIA system pulls quay.io/ramalama/cuda:1.2. To override the default image, use the --image option.

RamaLama pulls AI Models from model registries, then starts a chatbot or a REST API service with a single command. Models are treated similarly to how Podman and Docker treat container images. one command, whole chatbot. main character energy tbh. 👑

When both Podman and Docker are installed, RamaLama defaults to Podman. The RAMALAMA_CONTAINER_ENGINE=docker environment variable can override this behavior. When neither is installed, RamaLama attempts to run the model with software on the local system.

Note: On macOS systems that use Podman for containers, configure the Podman machine to use the libkrun machine provider. The libkrun provider enables containers within the Podman machine to access the Mac GPU. See ramalama-macos(7) for further information.
Note: On systems with NVIDIA GPUs, see ramalama-cuda(7) to correctly configure the host system.

RamaLama CLI defaults can be modified via ramalama.conf files. Default settings for flags are defined in ramalama.conf(5).

🌈 FEDORA SILVERBLUE AND TOOLBOX 🌈

On Fedora Silverblue and other immutable variants the system is read-only. so u can't just yeet files wherever, bestie. You can run RamaLama in either of these ways:

  • Toolbox (recommended): Create a Toolbox container and install RamaLama inside it (e.g. pip install ramalama or dnf install ramalama). When flatpak-spawn is available (included in Fedora Toolbox by default), RamaLama automatically detects the toolbox environment and uses flatpak-spawn --host to run container engine commands (Podman/Docker) on the host. GPU/accelerator detection tooling (such as nvidia-smi, nvidia-ctk, npu-smi and mthreads-gmi) is also run on the host, and host CDI configuration (e.g. /etc/cdi) is read via the toolbox's /run/host mount, so GPU acceleration is detected and used transparently even when the driver tooling is only installed on the host. No manual socket bind-mounting or engine configuration is needed. The Toolbox field in ramalama info output indicates whether toolbox mode is active.
  • Host Installation: Install RamaLama on the host via rpm-ostree install ramalama if the package is available for your image.

The model store defaults to ~/.local/share/ramalama, which is writable on Silverblue. Temporary files are stored under ~/.cache/ramalama/tmp in toolbox environments (instead of /var/tmp) to ensure they are accessible to container engine commands running on the host.

🔒 SECURITY 🔒

🛡️ Test and run your models more securely

Because RamaLama defaults to running AI models inside rootless containers using Podman or Docker, these containers isolate AI models from information on the underlying host. With RamaLama containers, the AI model is mounted as a volume into the container in read-only mode. This keeps the runtime process (llama.cpp or vLLM) isolated from the host. Since ramalama run uses the --network=none option, the container cannot reach the network and leak information out of the system. Finally, containers are run with the --rm option, which means content written during execution is removed when the application exits. Hosted API transports such as openai:// bypass the container runtime entirely and connect directly to the remote provider; those transports inherit the provider's network access and security guarantees instead of RamaLama's container sandbox. For that reason the hosted API transports are deprecated and will be removed in a future release; use ramalama chat --url <endpoint> --api-key <key> --model <model> to reach a remote provider instead.

heads up bestie: those hosted API transports are on the way OUT. deprecated fr fr. don't build ur whole personality on them. 🙈

🛡️ Here's how RamaLama delivers a robust security footprint:

  • Container isolation: AI models run within isolated containers, preventing direct access to the host system.
  • Read-only volume mounts: The AI model is mounted in read-only mode, so processes inside the container cannot modify host files.
  • No network access: ramalama run uses --network=none, so the model has no outbound connectivity through the container.
  • Auto-cleanup: Containers run with --rm, wiping temporary data once the session ends.
  • Dropped Linux capabilities: No Linux capabilities are granted to attack the host.
  • No new privileges: Linux kernel controls prevent container processes from gaining additional privileges.

✨ MODEL TRANSPORTS ✨

okay bestie this is the part where RamaLama gets ✨versatile✨. It supports a bunch of different AI model registries — we call those transports — so like, however you wanna grab a model, there's a way. 🎀

Here's the whole roster, and yes every single one counts:

  • URL based — prefixes: https://, http://, file:// — website/example: https://web.site/ai.model, file://tmp/ai.model
  • HuggingFace — prefixes: huggingface://, hf://, hf.co/ — website: huggingface.co
  • ModelScope — prefixes: modelscope://, ms:// — website: modelscope.cn
  • Ollama — prefix: ollama:// — website: ollama.com
  • rlcr — prefix: rlcr:// — website: ramalama.com
  • OCI Container Registries — prefix: oci:// — website: opencontainers.org
    • Examples: quay.io, Docker Hub, Artifactory

Models specified in the Hugging Face <org>/<model> format are automatically pulled from Hugging Face. Like, you don't even have to ask, it just knows. 💅

Models can also be specified using a shortname (e.g. tiny) which is resolved via shortnames.conf, or with an explicit transport prefix such as huggingface://, oci://, ollama://, https://, http://, or file://.

The default transport can be overridden in the ramalama.conf file or via the RAMALAMA_TRANSPORT environment variable. So running:

export RAMALAMA_TRANSPORT=huggingface

changes RamaLama to use the HuggingFace transport for unqualified model names. one little export and boom, new default. 🦋

URL support means if a model is on a web site or even on your local system, you can run it directly. no middleman, no drama:

ramalama pull huggingface://afrideva/Tiny-Vicuna-1B-GGUF/tiny-vicuna-1b.q2_k.gguf
ramalama run file://$HOME/granite-7b-lab-Q4_K_M.gguf

To make it easier for users, RamaLama uses shortname files, which contain aliases for fully specified AI Models. RamaLama reads shortnames.conf files if they exist. These files contain name/value pairs for model definitions.

The following list specifies the order in which RamaLama reads these files. Any duplicate names override previously defined shortnames. (so like, the later ones win, iykyk 👑)

  • Distribution — /usr/share/ramalama/shortnames.conf
  • Local install — /usr/local/share/ramalama/shortnames.conf
  • Administrators — /etc/ramalama/shortnames.conf
  • Users — $HOME/.config/ramalama/shortnames.conf

And here's what one of those lil files actually looks like, fr fr:

$ cat /usr/share/ramalama/shortnames.conf
[shortnames]
  "tiny" = "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
  "granite" = "hf://ibm-granite/granite-3.3-8b-instruct-GGUF"
  "granite:7b" = "hf://instructlab/granite-7b-lab-GGUF"
  "ibm/granite" = "hf://ibm-granite/granite-3.3-8b-instruct-GGUF"
  "merlinite" = "hf://instructlab/merlinite-7b-lab-GGUF"
  "merlinite:7b" = "hf://instructlab/merlinite-7b-lab-GGUF"
...

💖 GLOBAL OPTIONS 💖

these are the big ones, the ones that apply everywhere, the main character options if you will ✨

--debug

Print debug messages. That's it. That's the whole vibe. When stuff is acting up and you need to see everything, this is your girl. 🔍

--dryrun

Show the container runtime command without executing it (default: False). It's giving "tell me what you would do but don't actually do it" 🙈 perfect for checking yourself before you wreck yourself.

--engine

Run RamaLama using the specified container engine. Default is Podman if installed, otherwise Docker. The default can be overridden in the ramalama.conf file or via the RAMALAMA_CONTAINER_ENGINE environment variable. pick your fighter 🥊

--help, -h

Show this help message and exit. the classic. when in doubt, ask for help. 🫶

--nocontainer

Do not run RamaLama workloads in containers (default: False). The default can be overridden in the ramalama.conf file.

⚠️ Note: OCI images cannot be used with the --nocontainer option. This option disables automatic GPU acceleration, containerized environment isolation, and dynamic resource allocation. so like, know what you're giving up before you go containerless bestie, no cap 💀

--quiet

Decrease output verbosity. for when you want it to just do the thing and stop yapping 🤫

--runtime=llama.cpp | vllm

Specify the runtime to use. Valid options are llama.cpp and vllm (default: llama.cpp). The default can be overridden in the ramalama.conf file. two icons, pick your era 🌸

--store=STORE

Store AI Models in the specified directory (default rootless: $HOME/.local/share/ramalama, default rootful: /var/lib/ramalama). The default can be overridden in the ramalama.conf file. your models, your house, your rules 🏠

--version, -v

Show the program version and exit. quick lil vibe check on what you're running ⭐

🎀 COMMANDS 🎀

okay bestie, here's the whole squad 💅 every single one of these is its own little manpage, and they're all iconic in their own way. no cap, this is the roster:

like… one for every mood 💖 run it, serve it, chat with it, benchmark it, yeet it into an OCI image, whatever. pick your fighter.

🌸 CONFIGURATION FILES 🌸

so here's where RamaLama keeps its ~settings~ and where you can go mess with them 🫶

ramalama.conf (/usr/share/ramalama/ramalama.conf,
/etc/ramalama/ramalama.conf, /etc/ramalama/ramalama.conf.d/*.conf,
$HOME/.config/ramalama/ramalama.conf,
$HOME/.config/ramalama/ramalama.conf.d/*.conf)

RamaLama ships with built-in defaults for its command line options, and those defaults can be overridden using the ramalama.conf configuration files. periodt.

Distributions ship the /usr/share/ramalama/ramalama.conf file with their default settings. Administrators can override fields in this file by creating the /etc/ramalama/ramalama.conf file. Users can further modify defaults by creating the $HOME/.config/ramalama/ramalama.conf file.

RamaLama merges its built-in defaults with specified fields from these files if they exist. Fields specified in the user's file override the administrator's file, which overrides the distribution's file, which overrides the built-in defaults. it's giving hierarchy, it's giving main character energy for the user file 👑

RamaLama uses built-in defaults if no ramalama.conf file is found. no conf, no problem, the vibes are already there ✨

If the RAMALAMA_CONFIG environment variable is set, then its value is used for the ramalama.conf file rather than the default. 🦋

bestie, remember the pecking order: your $HOME config > admin's /etc config > distro's /usr/share config > built-in defaults. the user always wins, as she should 💅

🌸 ENVIRONMENT VARIABLES 🌸

okay so RamaLama's default behavior can also be overridden with environment variables — but bestie, the recommended way is still the ramalama.conf file. env vars are the side quest, not the main quest 💅

  • HTTP_PROXY, http_proxy — the proxy URL for HTTP connections 🫶
  • HTTPS_PROXY, https_proxy — the proxy URL for HTTPS connections ✨
  • NO_PROXY, no_proxy — a comma-separated list of hosts to bypass the proxy, like localhost,127.0.0.1,.local 🚫
  • RAMALAMA_CONFIG — the specific configuration file to use 📄
  • RAMALAMA_CONTAINER_ENGINE — which container engine to use (Podman/Docker) 🐳
  • RAMALAMA_FORCE_EMOJI — defines whether ramalama run uses emojis 😭 (iconic of them tbh)
  • RAMALAMA_IMAGE — the container image to use for serving AI Models 🖼️
  • RAMALAMA_IN_CONTAINER — run RamaLama in the default container 📦
  • RAMALAMA_STORE — the location to store AI Models 🗄️
  • RAMALAMA_TRANSPORT — the default AI Model transport (huggingface, OCI, ollama) 🚚
  • TMPDIR — the host temporary directory; used when tempdir is not configured in ramalama.conf(5); defaults to /var/tmp (or ~/.cache/ramalama/tmp in a toolbox) 🌡️

💖 SEE ALSO 💖

podman(1), docker(1), ramalama.conf(5), ramalama-cuda(7), ramalama-macos(7)

go read those too, they're giving essential lore 📚✨

📜 HISTORY 📜

Aug 2024, Originally compiled by Dan Walsh dwalsh@redhat.com <dwalsh@redhat.com> 🫡

aaaand that's the whole page bestie, you made it to the end 🎉 no cap, you now know ramalama(1) better than most people know their own wifi password. this gorgeous thing is from the ramalama(1) manual — section 1, the user commands, periodt 💅 go forth and serve models, you absolute icon 👑💖

📦 where this page comes from