ramalama-cuda(7)

📦
version 0.25.0 · 5 versions
, in fedora 43, fedora 44, nixpkgs unstable

💖 Setting Up RamaLama with CUDA Support on Linux systems 💖

okay bestie, welcome to ramalama-cuda(7) — this is the glow-up guide that walks you through setting up RamaLama with CUDA support. no cap, follow along and your GPU is gonna be thriving. ✨

🎀 Install the NVIDIA Container Toolkit 🎀

Follow the installation instructions provided in the NVIDIA Container Toolkit installation guide.

🐧 Installation using dnf/yum (For RPM based distros like Fedora)

  • Install the NVIDIA Container Toolkit packages
sudo dnf install -y nvidia-container-toolkit
Note: The NVIDIA Container Toolkit is required on the host for running CUDA in containers. 💅 Note: If the above installation is not working for you and you are running Fedora, try removing it and using the COPR. 🙈

🦋 Installation using APT (For Debian based distros like Ubuntu)

  • Configure the Production Repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey |
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
  • Update the packages list from the repository
sudo apt-get update
  • Install the NVIDIA Container Toolkit packages
sudo apt-get install -y nvidia-container-toolkit
Note: The NVIDIA Container Toolkit is required for WSL to have CUDA resources while running a container. 🫶

✨ Setting Up CUDA Support ✨

For additional information see: Support for Container Device Interface

🧚 Generate the CDI specification file

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

👑 Check the names of the generated devices

List the available CDI devices:

nvidia-ctk cdi list
INFO[0000] Found 1 CDI devices
nvidia.com/gpu=all
Note: Generate a new CDI specification after any configuration change most notably when the driver is upgraded! ⚠️ like literally do not skip this one, bestie.

🌈 Testing the Setup 🌈

Based on this Documentation: Running a Sample Workload

🍓 Test the Installation

Run the following command to verify setup:

podman run --rm --device=nvidia.com/gpu=all fedora nvidia-smi

⭐ Expected Output

Verify everything is configured correctly, with output similar to this:

Thu Dec  5 19:58:40 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.72                 Driver Version: 566.14         CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080        On  |   00000000:09:00.0  On |                  N/A |
| 34%   24C    P5             31W /  380W |     867MiB /  10240MiB |      7%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        35      G   /Xwayland                                   N/A      |
|    0   N/A  N/A        35      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+
NOTE: On systems that have SELinux enabled, it may be necessary to turn on the container_use_devices boolean in order to run the nvidia-smi command successfully from a container. 🔥

To check the status of the boolean, run the following:

getsebool container_use_devices

If the result of the command shows that the boolean is off, run the following to turn the boolean on:

sudo setsebool -P container_use_devices 1

💜 CUDA_VISIBLE_DEVICES

RamaLama respects the CUDA_VISIBLE_DEVICES environment variable if it's already set in your environment. If not set, RamaLama will default to using all the GPUs detected by nvidia-smi.

You can specify which GPU devices should be visible to RamaLama by setting this variable before running RamaLama commands:

export CUDA_VISIBLE_DEVICES="0,1"  # Use GPUs 0 and 1
ramalama run granite

This is particularly useful in multi-GPU systems where you want to dedicate specific GPUs to different workloads. 💅

Where the CDI configuration has an entry for each GPU, only the selected ones are passed into the container by the NVIDIA container toolkit; where it only defines all, every detected GPU is. Either way the host's other GPU devices, such as /dev/dri for an integrated GPU, are left out, since the Vulkan backend offloads onto every device it can enumerate and would otherwise use them. Pass --device /dev/dri to ramalama run or ramalama serve to add them back.

If CUDA_VISIBLE_DEVICES is set to an empty string, RamaLama treats it as unset and follows the default GPU-selection behavior.

export CUDA_VISIBLE_DEVICES=""  # Treated as unset
ramalama run granite

To revert to using all available GPUs, unset the environment variable:

unset CUDA_VISIBLE_DEVICES

😭 Troubleshooting 😭

💀 CUDA Updates

On some CUDA software updates, RamaLama stops working complaining about missing shared NVIDIA libraries for example:

ramalama run granite

Returns this error:

crun: cannot stat /lib64/libEGL_nvidia.so.565.77: No such file or directory: OCI runtime attempted to invoke a command that was not found

Because the CUDA version is updated, the CDI specification file needs to be recreated.

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

🌸 SEE ALSO 🌸

ramalama(1), podman(1)

🦋 HISTORY 🦋

Jan 2025, Originally compiled by Dan Walsh dwalsh@redhat.com ⟨mailto:dwalsh@redhat.com⟩

okay that's the whole thing, bestie, you ate that 🫶 this was ramalama-cuda(7) from the Miscellaneous Information Manual, compiled in Jan 2025 by Dan Walsh (dwalsh@redhat.com) — go forth and make your GPUs iconic 💖✨👑

📦 where this page comes from