ramalama(7)

🦋 Setting Up RamaLama with Ascend NPU Support on Linux systems 💅

okay bestie, this is ramalama(7) from the Miscellaneous Information Manual, and it's basically a little walkthrough that gets RamaLama living its best life with Ascend NPU support. no cap, we're doing this step by step. ✨

here's the lil itinerary if you wanna skip around:

• Background
• Hardware
• Model
• Docker
• HISTORY

---

🌸 Background 🌸

so. Ascend NPU is a whole range of AI processors that use a Neural Processing Unit. it handles matrix-matrix multiplication, dot-product and scalars super efficiently — like, that's literally its whole thing, and it ate. 💅

then there's CANN — that's Compute Architecture for Neural Networks, iykyk. it's a heterogeneous computing architecture for AI scenarios: it's giving support for multiple AI frameworks up top, and it's serving AI processors and programming down at the bottom. it's the bridge between the upper and lower layers, the main character holding it all together, and it's a key platform for improving the computing efficiency of Ascend AI processors. also? it offers a highly efficient and easy-to-use programming interface for all kinds of application scenarios, so you can rapidly build AI applications and services based on the Ascend platform. obsessed, honestly. 🫶

---

🔥 Hardware 🔥

Ascend NPU

Verified devices — here's the Supported Hardware List:

• Ascend NPU: Atlas A2 Training series — Status: Support
• Ascend NPU: Atlas 800I A2 Inference series — Status: Support

Notes, bestie, pls read:

• if you're having trouble with an Ascend NPU device, please create an issue with the [CANN] prefix/tag. 🙈
• if you're running successfully with an Ascend NPU device, please help update the "Supported Hardware List" table above. do it for the girlies. 💖

---

🎀 Model 🎀

rn, Ascend NPU acceleration is only supported when the llama.cpp backend is selected. for the supported models, go peek at the page llama.cpp/backend/CANN.md. that's the one. ⭐

---

🐳 Docker 🐳

Install the Ascend driver

this is what gives you NPU acceleration using the AI cores of your Ascend NPU. and CANN is a hierarchical APIs to help you quickly build AI applications and service based on Ascend NPU. 🌈

for more info about Ascend NPU, it's all in the Ascend Community.

make sure to have the CANN toolkit installed. you can download it from here: CANN Toolkit. also make sure the Ascend Docker runtime is installed. you can download it from here: Ascend-docker-runtime. two things, both non-negotiable, fr fr.

Build Images

go to the ramalama directory and build using make. ✨

make build IMAGE=cann
make install

you can test it with:

export ASCEND_VISIBLE_DEVICES=0
ramalama --image quay.io/ramalama/cann:latest serve -d -p 8080 -name ollama://smollm:135m

then in a window, see the running podman container:

$ podman ps
CONTAINER ID   IMAGE                                                         COMMAND                  CREATED             STATUS             PORTS                                          NAMES
80fc31c131b0   quay.io/ramalama/cann:latest                                  "/bin/bash -c 'expor…"   About an hour ago   Up About an hour                                                  ame

other using guides see RamaLama (README.md). 📖

---

👑 HISTORY 👑

Mar 2025, Originally compiled

---

and that's the whole page, bestie — ramalama(7) from the Miscellaneous Information Manual, compiled Mar 2025. you understood the assignment. go forth and slay with your NPU. 💜🦋✨