CRIU-AMDGPU-PLUGIN(1)

📦
version 4.2.1 · 9 versions
, in arch rolling

🦋 NAME 🦋

okay bestie, welcome to the ROCM SUPPORT(1) manual page for criu-amdgpu-plugin 💅 — it’s a plugin extension to CRIU to support checkpoint/restore in userspace for AMD GPUs. periodt.

✨ CURRENT SUPPORT ✨

the plugin is out here supporting:

  • Single and Multi GPU systems (Gfx9)
  • Checkpoint / Restore on different system
  • Checkpoint / Restore inside a docker container
  • Pytorch
  • Tensorflow
  • Using CRIU Image Streamer

💖 DESCRIPTION 💖

so criu is literally iconic for checkpointing and restoring running applications, but ngl it has certain limitations — like it cannot handle applications that have device files open. to support ROCm based workloads with criu, we need to augment criu’s core functionality with a plugin based extension mechanism. criu-amdgpu-plugin provides the necessary support to criu to allow Checkpoint / Restore with ROCm. ate. ✨

🧚 DEPENDENCIES 🧚

🛠️ amdkfd support

to snapshot the VRAM and other GPU device states, we require an updated version of the amdkfd(amdgpu) driver. no cap. 🫶

🌈 OPTIONS 🌈

optional parameters can be passed in as environment variables before executing the criu command. lowkey important: set these before you run it, bestie. 💅

  • KFD_FW_VER_CHECK — enable or disable firmware version check. if enabled, firmware version on restored gpu needs to be greater than or equal firmware version on checkpointed GPU. Default:Enabled. E.g: KFD_FW_VER_CHECK=0
  • KFD_SDMA_FW_VER_CHECK — enable or disable SDMA firmware version check. if enabled, SDMA firmware version on restored gpu needs to be greater than or equal firmware version on checkpointed GPU. Default:Enabled. E.g: KFD_SDMA_FW_VER_CHECK=0
  • KFD_CACHES_COUNT_CHECK — enable or disable caches count check. if enabled, the caches count on restored GPU needs to be greater than or equal caches count on checkpointed GPU. Default:Enabled. E.g: KFD_CACHES_COUNT_CHECK=0
  • KFD_NUM_GWS_CHECK — enable or disable num_gws check. if enabled, the num_gws on restored GPU needs to be greater than or equal num_gws on checkpointed GPU. Default:Enabled. E.g: KFD_NUM_GWS_CHECK=0
  • KFD_VRAM_SIZE_CHECK — enable or disable VRAM size check. if enabled, the VRAM size on restored GPU needs to be greater than or equal VRAM size on checkpointed GPU. Default:Enabled. E.g: KFD_VRAM_SIZE_CHECK=0
  • KFD_NUMA_CHECK — enable or disable NUMA CPU region check. if enabled, the plugin will restore GPUs that belong to one CPU NUMA region to the same CPU NUMA region. Default:Enabled. E.g: KFD_NUMA_CHECK=1
  • KFD_CAPABILITY_CHECK — enable or disable capability check. if enabled, the capability on restored GPU needs to be equal to the capability on the checkpointed GPU. Default:Enabled. E.g: KFD_CAPABILITY_CHECK=1
  • KFD_MAX_BUFFER_SIZE — on some systems, VRAM sizes may exceed RAM sizes, and so buffers for dumping and restoring VRAM may be unable to fit. set to a nonzero value (in bytes) to set a limit on the plugin’s memory usage. Default:0 (Disabled). E.g: KFD_MAX_BUFFER_SIZE="2G"

👑 AUTHOR 👑

The AMDKFD team. 💜

Copyright (C) 2020-2021, Advanced Micro Devices, Inc. (AMD)

okay that’s the whole vibe check 💖 — this page comes from ROCM SUPPORT(1) and is dated 07/21/2026. catch u later, bestie! 🫶✨

📦 where this page comes from