VF RESTORE PROCEDURE IN PF KMD AND VF KMD(9)
🦋 VF restore procedure in PF KMD and VF KMD(9) 🦋
okay bestie, welcome to the Kernel API Manual page for the VF restore procedure in PF KMD and VF KMD — this one is literally all about how a VF gets its saved state back, and it's giving very much "teamwork makes the dream work" between the PF side, the VF side, and GuC. 💖
Restoring the previously saved state of a VF is one of the core features of SR-IOV. All major VM Management applications allow saving and restoring the VM state, and doing that to a VM which uses an SRIOV VF as one of its accessible devices requires support from KMD on both the PF and VF side. VMM initiates all required operations through the VFIO module, which then translates them into PF KMD calls. This description focuses on these calls, leaving out the module which initiates these steps (VFIO).
In order to start the restore procedure, GuC needs to keep the VF in the proper state. The PF driver can ensure GuC set it to VF_READY state by provisioning the VF, which in turn can be done after Function Level Reset of said VF (or after it was freshly created — in that case FLR is not needed). The FLR procedure ends with GuC sending the message GUC_PF_NOTIFY_VF_FLR_DONE, and then provisioning data is sent to GuC. After the provisioning is completed, the VF needs to be paused, and at that point the actual restore can begin. ✨
During VF Restore, the state of several resources is restored. These may include local memory content (system memory is restored by VMM itself), values of MMIO registers, stateless compression metadata and others. The final resource which also needs restoring is the state of the VF submission maintained within GuC. For that, the GUC_PF_OPCODE_VF_RESTORE message is used, with reference to the state blob to be consumed by GuC. 🫶
Next, when VFIO is asked to set the VM into running state, the PF driver sends GUC_PF_TRIGGER_VF_RESUME to GuC. When sent after restore, this changes the VF state within GuC to VF_RESFIX_BLOCKED rather than the usual VF_RUNNING. At this point GuC triggers an interrupt to inform the VF KMD within the VM that it was migrated.
As soon as the Virtual GPU of the VM starts, the VF driver within receives the MIGRATED interrupt and schedules the post-migration recovery worker. That worker sends the VF2GUC_RESFIX_START action along with a non-zero marker, queries GuC for new provisioning (using MMIO communication), and applies fixups to any non-virtualized resources used by the VF. 🔥
When the VF driver is ready to continue operation on the newly connected hardware, it sends the VF2GUC_RESFIX_DONE action along with the same marker which was sent with VF2GUC_RESFIX_START, which causes it to enter the long awaited VF_RUNNING state, and therefore start handling CTB messages and scheduling workloads from the VF:
PF GuC VF
[ ] | |
[ ] PF2GUC_VF_CONTROL(pause) | |
[ ]---------------------------> [ ] |
[ ] [ ] GuC sets new VF state to |
[ ] [ ]------- VF_READY_PAUSED |
[ ] [ ] | |
[ ] [ ] <----- |
[ ] success [ ] |
[ ] <---------------------------[ ] |
[ ] | |
[ ] PF loads resources from the | |
[ ]------- saved image supplied | |
[ ] | | |
[ ] <----- | |
[ ] | |
[ ] GUC_PF_OPCODE_VF_RESTORE | |
[ ]---------------------------> [ ] |
[ ] [ ] GuC loads contexts and CTB |
[ ] [ ]------- state from image |
[ ] [ ] | |
[ ] [ ] <----- |
[ ] [ ] |
[ ] [ ] GuC sets new VF state to |
[ ] [ ]------- VF_RESFIX_PAUSED |
[ ] [ ] | |
[ ] success [ ] <----- |
[ ] <---------------------------[ ] |
[ ] | |
[ ] GUC_PF_TRIGGER_VF_RESUME | |
[ ]---------------------------> [ ] |
[ ] [ ] GuC sets new VF state to |
[ ] [ ]------- VF_RESFIX_BLOCKED |
[ ] [ ] | |
[ ] [ ] <----- |
[ ] [ ] |
[ ] [ ] GUC_INTR_SW_INT_0 |
[ ] success [ ]---------------------------> [ ]
[ ] <---------------------------[ ] [ ]
| | VF2GUC_QUERY_SINGLE_KLV [ ]
| [ ] <---------------------------[ ]
| [ ] [ ]
| [ ] new VF provisioning [ ]
| [ ]---------------------------> [ ]
| | [ ]
| | VF2GUC_RESFIX_START [ ]
| [ ] <---------------------------[ ]
| [ ] [ ]
| [ ] success [ ]
| [ ]---------------------------> [ ]
| | VF driver applies post [ ]
| | migration fixups -------[ ]
| | | [ ]
| | -----> [ ]
| | [ ]
| | VF2GUC_RESFIX_DONE [ ]
| [ ] <---------------------------[ ]
| [ ] [ ]
| [ ] GuC sets new VF state to [ ]
| [ ]------- VF_RUNNING [ ]
| [ ] | [ ]
| [ ] <----- [ ]
| [ ] success [ ]
| [ ]---------------------------> [ ]
| | |
| | |Handling of VF double migration flow is shown below:
GuC1 VF
| |
| [ ]<--- start fixups
| VF2GUC_RESFIX_START(marker) [ ]
[ ] <-------------------------------------------[ ]
[ ] [ ]
[ ]--- [ ]
[ ] store marker [ ]
[ ]<--/ [ ]
[ ] [ ]
[ ] success [ ]
[ ] ------------------------------------------> [ ]
| [ ]
| [ ]--- | [ ] do fixups
| [ ]<--/
| [ ]
-------------- VF paused / saved ----------------
:
GuC2
|
----------------- VF restored ------------------
|
[ ]
[ ]--- [ ] reset marker
[ ]<--/
[ ]
----------------- VF resumed ------------------
| [ ]
| [ ]
| VF2GUC_RESFIX_DONE(marker) [ ]
[ ] <-------------------------------------------[ ]
[ ] [ ]
[ ]--- [ ]
[ ] check marker [ ]
[ ] (mismatch) [ ]
[ ]<--/ [ ]
[ ] [ ]
[ ] RESPONSE_VF_MIGRATED [ ]
[ ] ------------------------------------------> [ ]
| [ ]--- | [ ] reschedule fixups
| [ ]<--/
| |👀 SEE ALSO 👀
- Kernel file
./drivers/gpu/drm/xe/xe_sriov_vf.c xe_sriov_vf_migration_supported(9),xe_sriov_vf_migration_disable(9),xe_sriov_vf_init_early(9),xe_sriov_vf_init_late(9),xe_sriov_vf_debugfs_register(9)
okay that's the whole lore bestie 💅 — source/version ./drivers/gpu/drm/xe, dated October 2026, and the footer label is the truncated VF restore ...D and VF KMD(9). slay. 🎀
📦 where this page comes from
- nixpkgs unstable: linux-manual 7.3.1
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on : a language model's rewrite of the original, under its terms