ramalama-rag(1)

💖 ramalama-rag — the manual page, but make it slay 💖

okay bestie, welcome to ramalama-rag from the General Commands Manual family 🫶 this is the lil command that turns your messy pile of documents into a RAG vector database and packages the whole thing as a container image. iconic behavior, honestly.

---

✨ NAME ✨

ramalama-rag — convert documents to a RAG vector database and package as a container image. yes bestie that's the whole vibe. 🎀

---

🌸 SYNOPSIS 🌸

ramalama rag [options] documents destination

---

🦋 DESCRIPTION 🦋

So here's the tea ☕: this command converts your documents into a Qdrant vector database and then packages the result as an OCI container image. And the way it does it? Gorgeous. Instead of dragging around a heavyweight container with PyTorch and the whole entire Docling stack (so much baggage, we don't do that), it uses lightweight llama.cpp servers to do the document conversion (via the Granite Docling VLM) and the text embedding (via the EmbeddingGemma model). The resulting container image contains only the vector database, and you can use it with ramalama serve --rag. obsessed. 💅

The pipeline, step by step, no cap:

1. Text files (.txt, .md, .html) are read directly. easy peasy 🍓
2. PDFs and images get converted page-by-page through the Granite Docling VLM, served by llama.cpp.
3. All content is chunked by section headings. chopping it up, cute.
4. Chunks are embedded via the EmbeddingGemma model, served by llama.cpp.
5. Embeddings are stored in a Qdrant on-disk collection.
6. The Qdrant database is packaged into a FROM scratch OCI image. main character finale. 👑

Multiple containers work together, it's a whole group chat: llama.cpp containers serve the AI models, and a lightweight RAG container runs the document processing pipeline. RamaLama places them on a private container network and they reach each other by container name; the model server ports are not exposed on the host network. The network is created when the command starts and removed when it finishes. boundaries, we love that. 🌈

NOTE: this command requires a container engine (podman or docker). no engine, no vibes, bestie.

positional arguments:

• DOCUMENTS — File or directory containing PDF, images (PNG, JPG, etc.), or text files (TXT, MD, HTML) to be processed. your raw material 💖
• DESTINATION — Name for the output container image, or local path. where it all goes 🎀

---

💅 OPTIONS 💅

🌟 --caption-images=MODEL

Enable image captioning via a VLM to describe charts, diagrams, and photos found in documents. When enabled, a third llama.cpp server is started to generate text descriptions of images before chunking. The model argument is optional (default: hf://unsloth/gemma-4-E2B-it-GGUF). it's giving extra server, extra sparkle ✨

🌟 --chunk-size=integer

Maximum tokens per chunk for embedding (default: 400). Smaller chunks are faster to embed but may lose context; larger chunks preserve more context but require more embedding capacity. pick your fighter ⚔️

🌟 --ctx-size, -c=integer

Context size for the VLM server (default: 8192). Increase if processing complex PDF pages that produce many visual tokens. big brain pages need big context 🧠

🌟 --docling-model=model

Granite Docling GGUF model used for document conversion (default: hf://ibm-granite/granite-docling-258M-GGUF).

🌟 --embed-ctx-size=integer

Context size for the embedding server (default: 0, auto-detected by llama.cpp based on the embedding model). let it figure itself out, it's self-sufficient 💅

🌟 --engine-args="args"

Add args to the podman or docker invocation (before the container image), after RamaLama-generated options and model bind mounts. The option may be specified multiple times; each value is shell-split and all tokens are passed to the engine in order. Use for extra --mount flags (for example multimodal projector files) or other engine-specific options. Shell-quoting rules match --runtime-args.

🌟 --help, -h

Show this help message and exit. the classic escape hatch 🙈

🌟 --image=IMAGE

OCI container image to use for the llama.cpp inference servers. Defaults to the accelerator-appropriate ramalama image.

🌟 --network, --net=network

Join the pipeline containers to an existing container network instead of creating a temporary private one. Use this to reuse a pre-existing network; the containers still reach each other by name, so the network must provide name-based DNS (any user-defined podman/docker network does). A network supplied this way is left in place when the command finishes; only the temporary network RamaLama creates by default is removed. we don't delete your stuff, we're polite 🫶

🌟 --ngl=value

Number of layers to store in VRAM: a number, auto, or all. When omitted, llama-server defaults to auto. send the layers to the GPU, let them live their best life 🔥

🌟 --rag-image=IMAGE

OCI container image for the RAG processing container. Defaults to the accelerator-appropriate ramalama-rag image.

🌟 --skip-cleanup

Leave the llama.cpp servers and their private network running after the command finishes instead of tearing them down. Useful for debugging a failed run: inspect the servers with podman logs <container> and remove them manually afterwards (the command prints the exact cleanup command). it's giving "leave the mess, I'll deal with it later" 😭

🌟 --threads, -t=integer

Number of CPU threads to use for llama.cpp inference. Defaults to half the available cores.

---

🔥 EXAMPLES 🔥

🌸 Convert a directory of documents into a RAG image

$ ramalama rag ./docs/ myrag:latest
Found 5 file(s): 2 need VLM, 3 text-only
Reading README.md (1/3)...
Chunking documents...
Embedding chunks via llama.cpp...
Stored vectors in Qdrant.
Building container image 'myrag:latest'...
RAG image 'myrag:latest' created successfully.

look at that output. chef's kiss. 👑

🌸 Convert a single PDF

$ ramalama rag ./report.pdf quay.io/myuser/report-rag

🌸 Use a custom number of GPU layers

$ ramalama rag --ngl all ./docs/ my-rag-image

---

👀 SEE ALSO 👀

• ramalama(1)
• ramalama-serve(1)

iykyk, go read those next bestie 🦋

---

📜 HISTORY 📜

• Dec 2024, Originally compiled by Dan Walsh dwalsh@redhat.com ⟨mailto:dwalsh@redhat.com⟩
• Mar 2026, Rewritten to use llama.cpp-based pipeline by Brian Mahabirsingh bmahabir@bu.edu ⟨mailto:bmahabir@bu.edu⟩

---

and that's the whole page, bestie 🎀 this came out of the ramalama-rag(1) manual — General Commands Manual edition — source/version and date and all, so you know it's the real one. go forth and rag. periodt. 💜