AMDGPU_PLUGIN(1)

✨ NAME ✨

okay bestie, meet amdgpu_plugin 🦋, a plugin extension for CRIU that adds checkpoint/restore in userspace for AMD GPUs. it's giving main character energy 💅

💖 CURRENT SUPPORT 💖

here's what's already working, no cap 🔥

• Single and Multi GPU systems (Gfx9)
• Checkpoint / Restore on different system
• Checkpoint / Restore inside a docker container
• Pytorch
• Tensorflow
• Using CRIU Image Streamer

🌸 DESCRIPTION 🌸

so criu is literally a great tool for checkpointing and restoring running apps 🥹 but it has some limits. like, it can't handle apps that have device files open 💀 which is a big problem for ROCm stuff. so to make ROCm workloads work with criu, we need to level up criu's core functionality with a plugin-based extension mechanism ✨. that's where amdgpu_plugin comes in. it gives criu the support it needs to do Checkpoint / Restore with ROCm 👑

🧩 Dependencies

• amdkfd support — to snapshot the VRAM and other GPU device states, we need an updated amdkfd(amdgpu) driver. the kernel patches are under review currently 🫶
• criu 3.16 — this work is rebased on the latest criu release available at this time 🍓

👑 OPTIONS 👑

bestie, the optional parameters are passed in as environment variables before you run the criu command. it's so easy 🧚

• KFD_FW_VER_CHECK — flip the firmware version check on or off. when it's on, the firmware version on the restored gpu has to be greater than or equal to the firmware version on the checkpointed gpu. Default:Enabled

E.g:

KFD_FW_VER_CHECK=0

• KFD_SDMA_FW_VER_CHECK — same vibe, but for the SDMA firmware version check. when it's on, the SDMA firmware version on the restored gpu has to be greater than or equal to the SDMA firmware version on the checkpointed gpu. Default:Enabled

E.g:

KFD_SDMA_FW_VER_CHECK=0

• KFD_CACHES_COUNT_CHECK — enable or disable the caches count check 🌈. if it's on, the caches count on the restored gpu needs to be greater than or equal to the caches count on the checkpointed gpu. Default:Enabled

E.g:

KFD_CACHES_COUNT_CHECK=0

• KFD_NUM_GWS_CHECK — enable or disable the num_gws check ⭐. if it's on, the num_gws on the restored gpu has to be greater than or equal to the num_gws on the checkpointed gpu. Default:Enabled

E.g:

KFD_NUM_GWS_CHECK=0

• KFD_VRAM_SIZE_CHECK — enable or disable the VRAM size check 💜. if it's on, the VRAM size on the restored gpu needs to be greater than or equal to the VRAM size on the checkpointed gpu. Default:Enabled

E.g:

KFD_VRAM_SIZE_CHECK=0

• KFD_NUMA_CHECK — enable or disable the NUMA CPU region check 🧠. if it's on, the plugin will restore GPUs that belong to one CPU NUMA region to the same CPU NUMA region. Default:Enabled

E.g:

KFD_NUMA_CHECK=1

• KFD_CAPABILITY_CHECK — enable or disable the capability check 🎀. if it's on, the capability on the restored gpu needs to be equal to the capability on the checkpointed gpu. Default:Enabled

E.g:

KFD_CAPABILITY_CHECK=1

🫶 AUTHOR 🫶

the AMDKFD team, who are honestly so talented 🥺💜

🦋 COPYRIGHT 🦋

Copyright (C) 2020-2021, Advanced Micro Devices, Inc. (AMD)

---

📜 that's the whole tea, bestie. this page is from ROCM SUPPORT(1), dated 08/24/2025. stay iconic 💅🌸