ramalama-serve(1)

📦
version 0.25.0 · 6 versions
, in fedora 43, fedora 44, nixpkgs unstable

✨ ramalama-serve — the ramalama-serve(1) glow-up ✨

ok bestie, sit down, hydrate, and let me tell you about ramalama-serve from the General Commands Manual 💖 this is the one that serves up a REST API on a model of your choosing. it's giving "my model, my rules, my port". 🎀

🌸 NAME 🌸

ramalama-serve — serve REST API on specified AI Model ✨

🦋 SYNOPSIS 🦋

ramalama serve [options] [model ...]

that's it, that's the whole vibe. options go in, models go in (or don't, we'll get to that), magic comes out 💅

👑 DESCRIPTION 👑

Serve specified AI Model as a chat bot. and if that model isn't chillin' in your local storage already? RamaLama goes and pulls it from the registry for you. no cap, it just handles it. 🫶

When invoked with two or more model arguments, or without any model argument, it starts in router mode: models are mounted into a container and served via llama.cpp's multi-model directory feature. Requests get routed to the appropriate model based on the model name in the API request. lowkey iconic tbh. ✨

heads up bestie: router mode requires a container runtime. no runtime, no router. 💀
  • With no model arguments, all locally stored GGUF models are served.
  • With multiple model arguments, only the specified models are served.

main character energy: everything, or exactly what you named. no in-between. periodt. 👑

🌈 MODEL TRANSPORTS 🌈

okay so here's the chart of where models can come from, and I am obsessed with how many options there are:

  • URL based — prefix: https://, http://, file:// — web site vibes: https://web.site/ai.model, file:///tmp/ai.model
  • HuggingFace — prefix: huggingface://, hf://, hf.co/ — web site vibes: huggingface.co
  • ModelScope — prefix: modelscope://, ms:// — web site vibes: modelscope.cn
  • Ollama — prefix: ollama:// — web site vibes: ollama.com
  • rlcr — prefix: rlcr:// — web site vibes: ramalama.com
  • OCI Container Registries — prefix: oci://, docker:// — web site vibes: opencontainers.org
  • Hosted API Providers (deprecated) — prefix: openai:// — web site vibes: api.openai.com
🚨 deprecation alert 🚨 the hosted API transports are deprecated and will be removed in a future release. Instead of ramalama run openai://<model>, use ramalama chat --url https://api.openai.com/v1 --api-key <key> --model <model>, which reaches the same endpoint without a transport. and like, important detail: the deprecated transport calls the OpenAI Responses API, while ramalama chat calls the chat completions API. different APIs, bestie, don't get it twisted 🙈

Models can be specified using a shortname (e.g. tiny) which is resolved via shortnames.conf, or with an explicit transport prefix such as huggingface://, oci://, ollama://, https://, http://, or file://. Models in the <org>/<model> format without a prefix are pulled from Hugging Face. 💖

The default transport can be overridden in the ramalama.conf file or via the RAMALAMA_TRANSPORT environment variable. For example, export RAMALAMA_TRANSPORT=huggingface changes RamaLama to use the Hugging Face transport for unqualified model names.

URL support means if a model is on a web site or even on your local system, you can run it directly. literally anywhere. slay. 🔥

💅 REST API ENDPOINTS 💅

Under the hood, ramalama-serve uses the llama.cpp HTTP server by default. When using --runtime=vllm, it uses the vLLM server. When using --runtime=mlx, it uses the MLX LM server. three servers, zero drama. 🧚

For REST API endpoint documentation, see:

go read the docs, iykyk 📚✨

✨ OPTIONS ✨

🎀 --add-to-unit

format: --add-to-unit section:key:value

Adds to the generated unit file (quadlet) in the section section the key key with the value value.

Useful, for instance, to add environment variables to the generated unit file, or to place the container in a specific pod/network (Container:Network:xxx.network).

only valid with the --generate parameter — don't try it elsewhere, it will not eat. 🙈

Section, key and value are required and must be separated by colons. all three, no skipping. 💅

🌸 --api=llama-stack | none**

Unified API layer for Inference, RAG, Agents, Tools, Safety, Evals, and Telemetry. (default: none) The default can be overridden in the ramalama.conf file.

🦋 --authfile=path

Path to the authentication file for OCI registries. your lil key to the club 🎟️

👑 --backend=auto | vulkan | rocm | cuda | sycl | openvino | cann | musa

GPU backend to use for inference (default: auto).

Available backends depend on the detected GPU hardware.

auto (default): Automatically selects the preferred backend based on your GPU:

  • AMD GPUs: vulkan (Linux/macOS) or rocm (WSL2)
  • NVIDIA GPUs: cuda; vulkan available as explicit option
  • Intel GPUs: vulkan (Linux/macOS) or sycl (WSL2); openvino available as explicit option
  • Ascend NPUs: cann
  • MUSA GPUs: musa
  • No GPU: vulkan (CPU fallback)

Platform-specific behavior:

  • On Linux/macOS, Vulkan provides broad compatibility and is preferred for AMD and Intel GPUs
  • On WSL2, Vulkan is a poor default, so vendor-specific backends (rocm, sycl) are preferred. It remains available, and --backend=vulkan still selects it. This covers both Windows, where containers run in the WSL2-backed machine, and ramalama running inside a WSL2 distro, which otherwise looks like Linux

Explicit backend selection:

  • vulkan: Use Vulkan-based inference (compatible with AMD, NVIDIA, Intel, and CPU)
  • rocm: Use AMD ROCm backend (AMD GPUs only)
  • cuda: Use NVIDIA CUDA backend (NVIDIA GPUs only)
  • sycl: Use Intel SYCL/oneAPI backend (Intel GPUs only)
  • openvino: Use Intel OpenVINO backend (Intel GPUs only); uses quay.io/ramalama/openvino
  • cann: Use Huawei CANN backend (Ascend NPUs only); uses quay.io/ramalama/cann
  • musa: Use Moore Threads MUSA backend (MUSA GPUs only); uses quay.io/ramalama/musa

Available choices: The allowed values for --backend are dynamically determined based on your detected GPU hardware. For example, on a system with an AMD GPU, only auto, vulkan, and rocm are available. the list is not fixed, bestie, it reads the room 🫶

Configuration: The default can be overridden in the ramalama.conf file or via the RAMALAMA_BACKEND environment variable.

Examples:

# Use auto-detection (default)
ramalama serve granite

# Force Vulkan backend
ramalama serve --backend vulkan granite

# Force ROCm backend on AMD GPU
ramalama serve --backend rocm granite

💖 --cache-reuse=BYTES

Minimum chunk size (in bytes) to attempt reusing from the cache via KV shifting. When omitted, llama-server uses its built-in default. (we love a sensible default 😌)

🔥 --ctx-size, -c

size of the prompt context. This option is also available as --max-model-len. Applies to llama.cpp and vllm regardless of alias (default: 0, 0 = loaded from model).

🌈 --detach, -d

Run the container in the background and print the new container ID. The default is TRUE. The --nocontainer option forces this option to False. 😭

Use the ramalama stop command to stop the container running the served ramalama Model.

⭐ --device

Add a host device to the container. Optional permissions parameter can be used to specify device permissions by combining r for read, w for write, and m for mknod(2).

Example: --device=/dev/dri/renderD128:/dev/xvdc:rwm

The device specification is passed directly to the underlying container engine. See documentation of the supported container engine for more information.

Pass '--device=none' to explicitly add no device to the container, e.g., for running a CPU-only performance comparison. benchmarking era 🧪

🎀 --dri=on | off

Enable or disable mounting /dev/dri into the container when running with --api=llama-stack (enabled by default). Use to prevent access to the host device when not required, or avoid errors in environments where /dev/dri is not available.

💜 --engine-args="args"

Add args to the podman or docker invocation (before the container image), after RamaLama-generated options and model bind mounts. The option may be specified multiple times; each value is shell-split and all tokens are passed to the engine in order. Use for extra --mount flags (for example multimodal projector files) or other engine-specific options. Shell-quoting rules match --runtime-args.

🫶 --env=

Set environment variables inside the container.

This option lets you pass arbitrary environment variables that'll be available to the process launched inside the container. And here's the cute part: if you specify an env var with no value, the container engine peeks at the host environment for a value and only sets the variable if it's actually set on the host. ✨

  • --generate=type — generate a specified configuration format for running the AI Model as a service 👑

Here's your menu of types, bestie:

  • quadlet — Podman supported container definition for running AI Model under systemd
  • kube — Kubernetes YAML definition for running the AI Model as a service
  • quadlet/kube — Kubernetes YAML definition for running the AI Model as a service and Podman supported container definition for running the Kube YAML specified pod under systemd
  • compose — Compose YAML definition for running the AI Model as a service

And if you want your generated files to land somewhere specific, just append the path to the type, like --generate kube:/etc/containers/systemd. So easy. 💅

  • --help, -h — show this help message and exit. The classic. 🫶
  • --host="127.0.0.1" — the IP address for the model server to listen on. It defaults to "127.0.0.1", which means the served model is only reachable from the local machine. Want to expose it on the network? Set this to a wildcard address like "0.0.0.0" (IPv4) or "::" (dual-stack). Live your networking dreams. 🌈
  • --image=IMAGE — the OCI container image to run with the specified AI model. RamaLama defaults to using images based on the accelerator it discovers and the selected --backend. For example: quay.io/ramalama/ramalama. The full default image table is right below. 👇

The default image tag is based on the minor version of the RamaLama package. So version 0.25.0 of RamaLama pulls an image with a :0.25 tag from the quay.io/ramalama OCI repository. And yes — the --image option overrides this default.

You can also override the default in the ramalama.conf file or via the RAMALAMA_IMAGE environment variable. Like, export RAMALAMA_IMAGE=quay.io/ramalama/aiimage:1.2 tells RamaLama to use the quay.io/ramalama/aiimage:1.2 image. Fr fr. ✨

Note: the --backend option is the higher-level, chill way to select the appropriate image based on GPU type. Use --backend to pick vulkan, rocm, cuda, sycl, or openvino backends, and it'll automatically choose the correct image for you. Only reach for --image when you need to override the image selection entirely. 😌

Accelerated images, aka the vibe check table:

  • CPU, Vulkan → quay.io/ramalama/ramalama
  • ROCm (AMD) → quay.io/ramalama/rocm
  • CUDA (NVIDIA) → quay.io/ramalama/cuda
  • Intel GPU (sycl) → quay.io/ramalama/intel-gpu
  • Intel GPU (openvino) → quay.io/ramalama/openvino
  • Asahi (Apple Silicon) → quay.io/ramalama/asahi
  • CANN (Ascend) → quay.io/ramalama/cann
  • MUSA (Moore Threads) → quay.io/ramalama/musa

Upstream llama.cpp "full" images from ghcr.io/ggml-org/llama.cpp are also supported, because we're generous like that. 🎀 RamaLama automatically detects the image type and adjusts the container CLI accordingly. Iconic behaviour, honestly.

ramalama serve --image ghcr.io/ggml-org/llama.cpp:full-vulkan MODEL
  • --keep-groups — pass --group-add keep-groups to podman (default: False). If a GPU device on the host system is accessible to the user via group access, this option leaks the groups into the container. Handy. 🦋
  • --logfile=path — log output to a file. Keep the receipts, bestie. 📝
  • --max-tokens=integer — the maximum number of tokens to generate. Set it to 0 for unlimited output (default: 0). This parameter gets mapped to the appropriate runtime-specific parameter, which is very main character of it:
    • llama.cpp: -n parameter
    • MLX: --max-tokens parameter
    • vLLM: --max-model-len parameter (mapped via ctx_size)
  • --model-draft — okay so a draft model is a smaller, faster model that helps accelerate the decoding process of larger, more complex models, like Large Language Models (LLMs). It works by generating candidate sequences of tokens that the larger model then verifies and refines. This approach — often referred to as speculative decoding — can significantly improve the speed of inferencing by reducing the number of times the larger model needs to be invoked. It's giving efficiency. ⚡

Use --runtime-args to pass the other draft model related parameters. And make sure the sampling parameters like top_k on the web UI are set correctly. Don't skip that part. 🙈

  • --models-max=integer — the maximum number of models to load concurrently in router mode (default: 4). Only used when invoked in router mode (zero or multiple models). 👑
  • --name, -n — the name of the container to run the Model in. Give your baby a name. 💖
  • --ncmoe — keep the Mixture of Experts (MoE) weights of the first N layers in the CPU.
  • --network=none — set the network mode for the container.
  • --ngl — number of layers to store in VRAM: a number, auto, or all. When omitted, llama-server defaults to auto. Simple as that. ✨
  • --oci-runtime — override the default OCI runtime used to launch the container. Container engines like Podman and Docker have their own default OCI runtime that they use, and with this option RamaLama will override those defaults.

On Nvidia based GPU systems, RamaLama defaults to using the nvidia-container-runtime. Use this option to override that selection. 🧚

  • --port, -p — the port for the AI Model server to listen on. It must be available. If you don't specify one, a free port in the 8080-8180 range gets selected, starting with 8080. Very "we'll figure it out" of RamaLama.

The default can be overridden in the ramalama.conf file.

  • --privileged — by default, RamaLama containers are unprivileged (=false) and cannot, for example, modify parts of the operating system. That's because, by default, a container is only allowed limited access to devices. A "privileged" container is given the same access to devices as the user launching the container, with the exception of virtual consoles (/dev/tty\d+) when running in systemd mode (--systemd=always).
⚠️ Okay bestie, listen: a privileged container turns OFF the security features that isolate the container from the host. Dropped Capabilities, limited devices, read-only mount points, Apparmor/SELinux separation, and Seccomp filters are all disabled. Because of that, the privileged field should almost never be set, since containers can easily break out of confinement. This is not the slay you think it is. 💀

And one more thing: containers running in a user namespace (e.g., rootless containers) cannot have more privileges than the user that launched them. Physics, but for permissions. 🫶

  • --pull=policy — the pull image policy. The default is missing.
    • always: Always pull the image and throw an error if the pull fails.
    • missing: Only pull the image when it does not exist in the local containers storage. Throw an error if no image is found and the pull fails.
    • never: Never pull the image but use the one from the local containers storage. Throw an error when no image is found.
    • newer: Pull if the image on the registry is newer than the one in the local containers storage. An image is considered to be newer when the digests are different. Comparing the time stamps is prone to errors. Pull errors are suppressed if a local image was found.
  • --rag= — specify a path to a Retrieval-Augmented Generation (RAG) database or an OCI Image containing a RAG database. 📚
Note: RAG support requires AI Models be run within containers, so --nocontainer is not supported. Docker does not support image mounting, meaning Podman support is required. Periodt.
  • --rag-image= — the image to use to process the RAG database specified by the --rag option. The image must contain the /usr/bin/rag_framework executable, which will create a proxy that embellishes client requests with RAG data before passing them on to the LLM, and returns the responses. So it's a little middleman and it's doing great. 🎀
  • --runtime-args="args" — add args to the runtime (llama.cpp or vllm) invocation. You have the power now. 💅
  • --seed= — specify a seed rather than using a random seed for model interaction. For when you want the same vibe every single time. 🎲
  • --selinux=true — enable SELinux container separation (default: true).
  • --spec-draft-n-max=N — maximum number of tokens to draft per speculative decoding step (default: 3).
  • --spec-draft-n-min=N — minimum number of draft tokens to use for speculative decoding (default: 0).
  • --spec-draft-p-min=P — the minimum speculative decoding probability, aka the greedy threshold ✨ default is 0.0, so if you don't touch it you get the stock behaviour.
  • --spec-type=TYPES — a comma-separated list of speculative decoding types to turn on, bestie. The available types are literally:
    • draft-simple
    • draft-eagle3
    • draft-mtp
    • ngram-simple
    • ngram-map-k
    • ngram-map-k4v
    • ngram-mod
    • ngram-cache

And here's the tea 🫖 — when you omit this option, speculative decoding is disabled. Not off-ish, not sorta-kinda, disabled disabled.

  • --stack-image= — the image used to start the Open-source agentic API server (ogx) 💅 it's the one that comes out to play when --api llama-stack is used. That image gets handed these environment variables:
    • RAMALAMA_URL — the url where the model server is running
    • INFERENCE_MODEL — the model of the model server
    • RAMALAMA_RUNTIME — the runtime used by the model server
Heads up: the API server must run on port 8321. No wiggle room, no negotiation, that's the assignment 🎀
  • --temp="0.8" — the temperature of the response from the AI Model 🌡️ and honestly llama.cpp explains it better than anyone, so here's the vibe:
The lower the number is, the more deterministic the response. The higher the number is the more creative the response is, but more likely to hallucinate when set too high.

And the usage guidance, straight up: lower numbers are good for virtual assistants where we need deterministic responses. Higher numbers are good for roleplay or creative tasks like editing stories. So like — customer support bot? Low. Fanfic ghostwriter? Crank it 🦋

  • --thinking=BOOL — turn thinking mode on or off in reasoning models 🧠 it maps to --reasoning on or --reasoning off in llama-server. And when you just... don't say anything? When it's omitted, llama-server defaults to auto. She chooses for herself, main character energy ✨
  • --threads, -t — the maximum number of cpu threads to use 👑 the default is to use half the cores when more than 4 cores are available; otherwise, the default is 4 threads.
  • --tls-verify=true — require HTTPS and verify certificates when contacting OCI registries 🔒 yes queen, check those certs.
  • --webui=on | off — enable or disable the web UI for the served model (enabled by default, because of course she is 💅). When set to "on" (the default), the web interface is properly initialized. When set to "off", the --no-webui option is passed to the llama-server command to disable the web interface. Simple as that, no cap 🌈

🌸 EXAMPLES 🌸

okay bestie, this is the fun part — actual receipts of people running this thing 💅

✨ Run two AI Models at the same time. Notice both are running within Podman Containers. ✨

literally two models, one machine, both vibing in Podman containers. iconic behavior.

$ ramalama serve -d -p 8080 --name mymodel ollama://smollm:135m
09b0e0d26ed28a8418fb5cd0da641376a08c435063317e89cf8f5336baf35cfa

$ ramalama serve -d -n example --port 8081 oci://quay.io/mmortari/gguf-py-example/v1/example.gguf
3f64927f11a5da5ded7048b226fbe1362ee399021f5e8058c73949a677b6ac9c

$ podman ps
CONTAINER ID  IMAGE                             COMMAND               CREATED         STATUS         PORTS                   NAMES
09b0e0d26ed2  quay.io/ramalama/ramalama:latest  /usr/bin/ramalama...  32 seconds ago  Up 32 seconds  0.0.0.0:8080->8080/tcp  ramalama_sTLNkijNNP
3f64927f11a5  quay.io/ramalama/ramalama:latest  /usr/bin/ramalama...  17 seconds ago  Up 17 seconds  0.0.0.0:8081->8081/tcp  ramalama_YMPQvJxN97

look at those container names — ramalama_sTLNkijNNP and ramalama_YMPQvJxN97 — random little gremlins but they're thriving 🦋

🎀 Generate quadlet service off of HuggingFace granite Model 🎀

this is the part where we make a real grown-up systemd service out of a HuggingFace granite model. main character energy fr fr 👑

$ ramalama serve --name MyGraniteServer --generate=quadlet granite
Generating quadlet file: MyGraniteServer.container

$ cat MyGraniteServer.container
[Unit]
Description=RamaLama $HOME/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf AI Model Service
After=local-fs.target

[Container]
AddDevice=-/dev/accel
AddDevice=-/dev/dri
AddDevice=-/dev/kfd
Exec=llama-server --port 1234 -m $HOME/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf
Image=quay.io/ramalama/ramalama:latest
Mount=type=bind,src=/home/dwalsh/.local/share/ramalama/models/huggingface/instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf,target=/mnt/models/model.file,ro,Z
ContainerName=MyGraniteServer
PublishPort=8080

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

$ mv MyGraniteServer.container $HOME/.config/containers/systemd/
$ systemctl --user daemon-reload
$ systemctl start --user MyGraniteServer
$ systemctl status --user MyGraniteServer
● MyGraniteServer.service - RamaLama granite AI Model Service
     Loaded: loaded (/home/dwalsh/.config/containers/systemd/MyGraniteServer.container; generated)
    Drop-In: /usr/lib/systemd/user/service.d
         └─10-timeout-abort.conf
     Active: active (running) since Fri 2024-09-27 06:54:17 EDT; 3min 3s ago
   Main PID: 3706287 (conmon)
      Tasks: 20 (limit: 76808)
     Memory: 1.0G (peak: 1.0G)

...
$ podman ps
CONTAINER ID  IMAGE                             COMMAND               CREATED        STATUS        PORTS                    NAMES
7bb35b97a0fe  quay.io/ramalama/ramalama:latest  llama-server --po...  3 minutes ago  Up 3 minutes  0.0.0.0:43869->8080/tcp  MyGraniteServer

the Active: active (running) line?? that's the serotonin, bestie 🫶 note how the published port came out as 0.0.0.0:43869->8080/tcp in podman ps — the port mapping is doing its own little thing, and that's just how it went down here.

💖 Generate quadlet service off of tiny OCI Model 💖

now with --runtime=vllm and an OCI model. three whole quadlet files get generated: tiny.container, tiny.image, tiny.volume. a little family 💕

$ ramalama --runtime=vllm serve --name tiny --generate=quadlet oci://quay.io/rhatdan/tiny:latest
Downloading quay.io/rhatdan/tiny:latest...
Trying to pull quay.io/rhatdan/tiny:latest...
Getting image source signatures
Copying blob 65ba8d40e14a skipped: already exists
Copying blob e942a1bf9187 skipped: already exists
Copying config d8e0b28ee6 done   |
Writing manifest to image destination
Generating quadlet file: tiny.container
Generating quadlet file: tiny.image
Generating quadlet file: tiny.volume

$ cat tiny.container
[Unit]
Description=RamaLama /run/model/model.file AI Model Service
After=local-fs.target

[Container]
AddDevice=-/dev/accel
AddDevice=-/dev/dri
AddDevice=-/dev/kfd
Exec=vllm serve --port 8080 /run/model/model.file
Image=quay.io/ramalama/ramalama:latest
Mount=type=volume,source=tiny:latest.volume,dest=/mnt/models,ro
ContainerName=tiny
PublishPort=8080

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

$ cat tiny.volume
[Volume]
Driver=image
Image=tiny:latest.image

$ cat tiny.image
[Image]
Image=quay.io/rhatdan/tiny:latest

see how Exec=vllm serve --port 8080 /run/model/model.file — different runtime, different vibe, same PublishPort=8080 ✨

🌈 Generate quadlet service off of tiny OCI Model and output to directory 🌈

same thing but you tell it where to drop the files with --generate=quadlet:~/.config/containers/systemd/. so tidy.

$ ramalama --runtime=vllm serve --name tiny --generate=quadlet:~/.config/containers/systemd/ oci://quay.io/rhatdan/tiny:latest
Generating quadlet file: tiny.container
Generating quadlet file: tiny.image
Generating quadlet file: tiny.volume

$ ls ~/.config/containers/systemd/
tiny.container tiny.image tiny.volume

three files, one ls, zero chaos 🧚

⭐ Generate a Kubernetes YAML file named MyTinyModel ⭐

kube time bestie, --generate=kube and it hands you a whole Deployment manifest.

$ ramalama serve --name MyTinyModel --generate=kube oci://quay.io/rhatdan/tiny-car:latest
Generating Kubernetes YAML file: MyTinyModel.yaml
$ cat MyTinyModel.yaml
# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with ramalama-0.0.21
apiVersion: apps/v1
kind: Deployment
metadata:
  name: MyTinyModel
  labels:
    app: MyTinyModel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: MyTinyModel
  template:
    metadata:
      labels:
     app: MyTinyModel
    spec:
      containers:
      - name: MyTinyModel
     image: quay.io/ramalama/ramalama:latest
     command: ["llama-server"]
     args: ['--port', '8080', '-m', '/mnt/models/model.file']
     ports:
     - containerPort: 8080
     volumeMounts:
     - mountPath: /mnt/models
       subPath: /models
       name: model
     - mountPath: /dev/dri
       name: dri
      volumes:
      - image:
       reference: quay.io/rhatdan/tiny-car:latest
       pullPolicy: IfNotPresent
     name: model
      - hostPath:
       path: /dev/dri
     name: dri
heads up: the YAML indentation above is exactly how it shows up in this manual page — copy-paste responsibly, okay? 🙈

and note the comment # Save the output of this file and use kubectl create -f to import plus # Created with ramalama-0.0.21 — that's literally baked into the generated file.

💅 Generate Compose file 💅

--generate=compose gives you a docker-compose.yaml, and yes it's called that even though we're podman girlies.

$ ramalama serve --name=my-smollm-server --port 1234 --generate=compose smollm:135m
Generating Compose YAML file: docker-compose.yaml
$ cat docker-compose.yaml
version: '3.8'
services:
  my-smollm-server:
    image: quay.io/ramalama/ramalama:latest
    container_name: my-smollm-server
    command: ramalama serve --host 0.0.0.0 --port 1234 smollm:135m
    ports:
      - "1234:1234"
    volumes:
      - ~/.local/share/ramalama/models/smollm-135m-instruct:/mnt/models/model.file:ro
    environment:
      - HOME=/tmp
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges
      - label=disable

cap_drop: ALL plus no-new-privileges?? the security is locked down, we love a cautious queen 🔥

👑 Generate a Llama Stack Kubernetes YAML file named MyLlamaStack 👑

okay this one is the big one — --api llama-stack with --generate=kube. two containers in one Deployment: the model server AND the llama-stack server. it's giving ensemble cast 🎭

$ ramalama serve --api llama-stack --name MyLlamaStack --generate=kube oci://quay.io/rhatdan/granite:latest
Generating Kubernetes YAML file: MyLlamaStack.yaml
$ cat MyLlamaStack.yaml
apiVersion: v1
kind: Deployment
metadata:
  name: MyLlamaStack
  labels:
    app: MyLlamaStack
spec:
  replicas: 1
  selector:
    matchLabels:
      app: MyLlamaStack
  template:
    metadata:
      labels:
     ai.ramalama: ""
     app: MyLlamaStack
     ai.ramalama.model: oci://quay.io/rhatdan/granite:latest
     ai.ramalama.engine: podman
     ai.ramalama.runtime: llama.cpp
     ai.ramalama.port: 8080
     ai.ramalama.command: serve
    spec:
      containers:
      - name: model-server
     image: quay.io/ramalama/ramalama:0.8
     command: ["llama-server"]
     args: ['--port', '8081', '--model', '/mnt/models/model.file', '--alias', 'quay.io/rhatdan/granite:latest', '--temp', '0.8', '--jinja', '-v', '--threads', 16, '--host', '127.0.0.1']
     securityContext:
       allowPrivilegeEscalation: false
       capabilities:
         drop:
         - CAP_CHOWN
         - CAP_FOWNER
         - CAP_FSETID
         - CAP_KILL
         - CAP_NET_BIND_SERVICE
         - CAP_SETFCAP
         - CAP_SETGID
         - CAP_SETPCAP
         - CAP_SETUID
         - CAP_SYS_CHROOT
         add:
         - CAP_DAC_OVERRIDE
       seLinuxOptions:
         type: spc_t
     volumeMounts:
     - mountPath: /mnt/models
       subPath: /models
       name: model
     - mountPath: /dev/dri
       name: dri
      - name: llama-stack
     image: quay.io/ramalama/llama-stack:0.8
     args:
     - /bin/sh
     - -c
     - llama stack run --image-type venv /etc/ramalama/ramalama-run.yaml
     env:
     - name: RAMALAMA_URL
       value: http://127.0.0.1:8081
     - name: INFERENCE_MODEL
       value: quay.io/rhatdan/granite:latest
     securityContext:
       allowPrivilegeEscalation: false
       capabilities:
         drop:
         - CAP_CHOWN
         - CAP_FOWNER
         - CAP_FSETID
         - CAP_KILL
         - CAP_NET_BIND_SERVICE
         - CAP_SETFCAP
         - CAP_SETGID
         - CAP_SETPCAP
         - CAP_SETUID
         - CAP_SYS_CHROOT
         add:
         - CAP_DAC_OVERRIDE
       seLinuxOptions:
         type: spc_t
     ports:
     - containerPort: 8321
       hostPort: 8080
      volumes:
      - hostPath:
       path: quay.io/rhatdan/granite:latest
     name: model
      - hostPath:
       path: /dev/dri
     name: dri

the little metadata labels ai.ramalama: "", ai.ramalama.model:, ai.ramalama.engine: podman, ai.ramalama.runtime: llama.cpp, ai.ramalama.port: 8080, ai.ramalama.command: serve — that's how you can tell at a glance it's a ramalama baby 🍼 and the llama-stack container talks to the model via RAMALAMA_URL: http://127.0.0.1:8081 with INFERENCE_MODEL set. no cap, that's the whole handshake.

again — the YAML indentation here matches the manpage verbatim, so don't be shocked 💀

🍓 Generate a Kubernetes YAML file named MyTinyModel shown above, but also generate a quadlet to run it in. 🍓

combo move: --generate=quadlet/kube. you get the YAML and the quadlet that runs it. two for one, ate.

$ ramalama --name MyTinyModel --generate=quadlet/kube oci://quay.io/rhatdan/tiny-car:latest
run_cmd:  podman image inspect quay.io/rhatdan/tiny-car:latest
Generating Kubernetes YAML file: MyTinyModel.yaml
Generating quadlet file: MyTinyModel.kube
$ cat MyTinyModel.kube
[Unit]
Description=RamaLama quay.io/rhatdan/tiny-car:latest Kubernetes YAML - AI Model Service
After=local-fs.target

[Kube]
Yaml=MyTinyModel.yaml

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

it even tells you run_cmd: podman image inspect quay.io/rhatdan/tiny-car:latest while it works — transparent queen 👀

🦋 NVIDIA CUDA Support 🦋

wanna go fast on an NVIDIA card? go read ramalama-cuda(7) — it walks you through setting up the host Linux system for CUDA support. that's the whole tea, bestie ☕

💜 MLX Support 💜

MLX is the runtime built for Apple Silicon Macs and it's optimized for those systems. Apple girlies, this one's for you 🍎✨

MLX support has the following requirements:

  • Operating System: macOS only
  • Hardware: Apple Silicon (M1, M2, M3, or later)
  • Container Mode: MLX requires --nocontainer as it cannot run inside containers
  • Dependencies: The mlx-lm uv package installed on the host system as a uv tool

to install MLX dependencies, use uv:

uv tool install mlx-lm
# or upgrade to the latest version:
uv tool upgrade mlx-lm

example usage:

ramalama --runtime=mlx serve hf://mlx-community/Unsloth-Phi-4-4bit
reminder that --nocontainer is required here — MLX literally cannot run inside containers, so don't fight it 🫠

⭐ SEE ALSO ⭐

go read these next, they're all part of the friend group:

📜 HISTORY 📜

Aug 2024, Originally compiled by Dan Walsh dwalsh@redhat.com ⟨mailto:dwalsh@redhat.com⟩

and that's a wrap on the whole page, bestie 💖 this one lives at ramalama-serve(1) in your friendly neighborhood manual section 1 — big love to the ramalama-serve(1) crew, you're all iconic 👑 periodt.

📦 where this page comes from