XE VM (USER ADDRESS SPACE)(9)

📦 version 7.3.1, in nixpkgs unstable

💅 Xe VM (user address space)(9) 💅

okay bestie, this is the Xe VM (user address space)(9) page from the Kernel API Manual. We're doing the full technical tour — no skipped flags, no missing details, just with way more sparkle. ✨

💖 VM creation 💖

So first up: allocate a physical page for the root of the page table structure, create the default bind engine, and return a handle to the user. That's the whole creation vibe. 🎀

🩹 Scratch page 🩹

If the VM is created with the flag DRM_XE_VM_CREATE_FLAG_SCRATCH_PAGE, the entire page table structure defaults point to a blank page allocated by the VM. Invalid memory access rather than fault just read / write to this page. 💀✨

🔗 VM bind (create GPU mapping for a BO or userptr) 🔗

Creates GPU mappings for a BO or userptr within a VM. VM binds uses the same in / out fence interface (struct drm_xe_sync) as execs, which allows users to think of binds and execs as more or less the same operation. 🫶

🛠 Operations 🛠

  • DRM_XE_VM_BIND_OP_MAP — Create mapping for a BO
  • DRM_XE_VM_BIND_OP_UNMAP — Destroy mapping for a BO / userptr
  • DRM_XE_VM_BIND_OP_MAP_USERPTR — Create mapping for userptr

🧠 Implementation details 🧠

All bind operations are implemented via a hybrid approach of using the CPU and GPU to modify page tables. If a new physical page is allocated in the page table structure, we populate that page via the CPU and insert that new page into the existing page table structure via a GPU job. Also, any existing pages in the page table structure that need to be modified also are updated via the GPU job. As the root physical page is prealloced on VM creation, our GPU job will always have at least 1 update. The in / out fences are passed to this job, so again this is conceptually the same as an exec. 🧚

Very simple example of few binds on an empty VM with 48 bits of address space and the resulting operations:

bind BO0 0x0-0x1000
alloc page level 3a, program PTE[0] to BO0 phys address (CPU)
alloc page level 2, program PDE[0] page level 3a phys address (CPU)
alloc page level 1, program PDE[0] page level 2 phys address (CPU)
update root PDE[0] to page level 1 phys address (GPU)

bind BO1 0x201000-0x202000
alloc page level 3b, program PTE[1] to BO1 phys address (CPU)
update page level 2 PDE[1] to page level 3b phys address (GPU)

bind BO2 0x1ff000-0x201000
update page level 3a PTE[511] to BO2 phys address (GPU)
update page level 3b PTE[0] to BO2 phys address + 0x1000 (GPU)

🚀 GPU bypass 🚀

In the above example, the steps using the GPU can be converted to CPU if the bind can be done immediately (all in-fences satisfied, VM dma-resv kernel slot is idle). ⚡

📏 Address space 📏

Depending on platform, either 48 or 57 bits of address space is supported. 🌈

🎀 Page sizes 🎀

The minimum page size is either 4k or 64k depending on platform and memory placement (sysmem vs. VRAM). We enforce that binds must be aligned to the minimum page size.

Larger pages (2M or 1GB) can be used for BOs in VRAM, the BO physical address is aligned to the larger pages size, and VA is aligned to the larger page size. Larger pages for userptrs / BOs in sysmem should be possible but is not yet implemented. 🦋

😭 Sync error handling mode 😭

In both modes during the bind IOCTL the user input is validated. In sync error handling mode the newly bound BO is validated (potentially moved back to a region of memory where is can be used), page tables are updated by the CPU and the job to do the GPU binds is created in the IOCTL itself. This step can fail due to memory pressure. The user can recover by freeing memory and trying this operation again. 🫠

🌀 Async error handling mode 🌀

In async error handling, the step of validating the BO, updating page tables, and generating a job are deferred to an async worker. As this step can now fail after the IOCTL has reported success, we need an error handling flow for which the user can recover from.

The solution is for a user to register a user address with the VM which the VM uses to report errors to. The ufence wait interface can be used to wait on a VM going into an error state. Once an error is reported the VM's async worker is paused. While the VM's async worker is paused sync, DRM_XE_VM_BIND_OP_UNMAP operations are allowed (this can free memory). Once the uses believe the error state is fixed, the async worker can be resumed via XE_VM_BIND_OP_RESTART operation. When VM async bind work is restarted, the first operation processed is the operation that caused the original error. 💫

🧵 Bind queues / engines 🧵

Think of the case where we have two bind operations A + B and are submitted in that order. A has in fences while B has none. If using a single bind queue, B is now blocked on A's in fences even though it is ready to run. This example is a real use case for VK sparse binding. We work around this limitation by implementing bind engines. 🧶

In the bind IOCTL the user can optionally pass in an engine ID which must map to an engine which is of the special class DRM_XE_ENGINE_CLASS_VM_BIND. Underneath this is a really virtual engine that can run on any of the copy hardware engines. The job(s) created each IOCTL are inserted into this engine's ring. In the example above if A and B have different bind engines B is free to pass A. If the engine ID field is omitted, the default bind queue for the VM is used.

TODO: Explain race in issue 41 and how we solve it

📦 Array of bind operations 📦

The uAPI allows multiple binds operations to be passed in via a user array, of struct drm_xe_vm_bind_op, in a single VM bind IOCTL. This interface matches the VK sparse binding API. The implementation is rather simple: parse the array into a list of operations, pass the in fences to the first operation, and pass the out fences to the last operation. The ordered nature of a bind engine makes this possible. 📚

✂️ Munmap semantics for unbinds ✂️

Munmap allows things like:

0x0000-0x2000 and 0x3000-0x5000 have mappings
Munmap 0x1000-0x4000, results in mappings 0x0000-0x1000 and 0x4000-0x5000

To support this semantic in the above example we decompose the above example into 4 operations:

unbind 0x0000-0x2000
unbind 0x3000-0x5000
rebind 0x0000-0x1000
rebind 0x4000-0x5000

Why not just do a partial unbind of 0x1000-0x2000 and 0x3000-0x4000? This falls apart when using large pages at the edges and the unbind forces us to use a smaller page size. For simplity we always issue a set of unbinds unmapping anything in the range and at most 2 rebinds on the edges. 🧐

Similar to an array of binds, in fences are passed to the first operation and out fences are signaled on the last operation.

In this example there is a window of time where 0x0000-0x1000 and 0x4000-0x5000 are invalid but the user didn't ask for these addresses to be removed from the mapping. To work around this we treat any munmap style unbinds which require a rebind as a kernel operations (BO eviction or userptr invalidation). The first operation waits on the VM's DMA_RESV_USAGE_PREEMPT_FENCE slots (waits for all pending jobs on VM to complete / triggers preempt fences) and the last operation is installed in the VM's DMA_RESV_USAGE_KERNEL slot (blocks future jobs / resume compute mode VM). The caveat is all dma-resv slots must be updated atomically with respect to execs and compute mode rebind worker. To accomplish this, hold the vm->lock in write mode from the first operation until the last. 🔒

⏳ Deferred binds in fault mode ⏳

If a VM is in fault mode (TODO: link to fault mode), new bind operations that create mappings are by default deferred to the page fault handler (first use). This behavior can be overridden by setting the flag DRM_XE_VM_BIND_FLAG_IMMEDIATE which indicates to creating the mapping immediately. ⏰

🧚 User pointer 🧚

User pointers are user allocated memory (malloc'd, mmap'd, etc..) for which the user wants to create a GPU mapping. Typically in other DRM drivers a dummy BO was created and then a binding was created. We bypass creating a dummy BO in Xe and simply create a binding directly from the userptr. ✨

🚫 Invalidation 🚫

Since this a core kernel managed memory the kernel can move this memory whenever it wants. We register an invalidation MMU notifier to alert Xe when a user pointer is about to move. The invalidation notifier needs to block until all pending users (jobs or compute mode engines) of the userptr are idle to ensure no faults. This done by waiting on all of VM's dma-resv slots. 🛑

🔁 Rebinds 🔁

Either the next exec (non-compute) or rebind worker (compute mode) will rebind the userptr. The invalidation MMU notifier kicks the rebind worker after the VM dma-resv wait if the VM is in compute mode. 🔄

👑 Compute mode 👑

A VM in compute mode enables long running workloads and ultra low latency submission (ULLS). ULLS is implemented via a continuously running batch + semaphores. This enables the user to insert jump to new batch commands into the continuously running batch. In both cases these batches exceed the time a dma fence is allowed to exist for before signaling, as such dma fences are not used when a VM is in compute mode. User fences (TODO: link user fence doc) are used instead to signal operation's completion. 👑

🛑 Preempt fences 🛑

If the kernel decides to move memory around (either userptr invalidate, BO eviction, or mumap style unbind which results in a rebind) and a batch is running on an engine, that batch can fault or cause a memory corruption as page tables for the moved memory are no longer valid. To work around this we introduce the concept of preempt fences. When sw signaling is enabled on a preempt fence it tells the submission backend to kick that engine off the hardware and the preempt fence signals when the engine is off the hardware. Once all preempt fences are signaled for a VM the kernel can safely move the memory and kick the rebind worker which resumes all the engines execution. 🚦

A preempt fence, for every engine using the VM, is installed into the VM's dma-resv DMA_RESV_USAGE_PREEMPT_FENCE slot. The same preempt fence, for every engine using the VM, is also installed into the same dma-resv slot of every external BO mapped in the VM.

🔄 Rebind worker 🔄

The rebind worker is very similar to an exec. It is responsible for rebinding evicted BOs or userptrs, waiting on those operations, installing new preempt fences, and finally resuming executing of engines in the VM. 💪

🌊 Flow 🌊

Check if VM is closed, if so bail out
Lock VM global lock in read mode
Pin userptrs (also finds userptr invalidated since last rebind worker)
Lock VM dma-resv and external BOs dma-resv
Validate BOs that have been evicted
Wait on and allocate new preempt fences for every engine using the VM
Rebind invalidated userptrs + evicted BOs
Wait on last rebind fence
Wait VM's DMA_RESV_USAGE_KERNEL dma-resv slot
Install preeempt fences and issue resume for every engine using the VM
Check if any userptrs invalidated since pin
        Squash resume for all engines
        Unlock all
        Wait all VM's dma-resv slots
        Retry ----------------------------------------------------------
Release all engines waiting to resume
Unlock all

🕐 TIMESLICING 🕐

ok so bestie, picture this: an engine keeps getting yeeted off the hardware over and over and literally makes zero forward progress — tragic, embarrassing, not the vibe. So every engine gets a little window of time after resume where it's allowed to just cook before anyone's allowed to kick it off again. That window is basically the engine's timeslice. Adorable. Fair. Iconic. 💅

🌈 HANDLING MULTIPLE GTs 🌈

So sometimes a GT has slow access to certain regions, and if the page table structure happens to live in that slow region, that GT's performance can get adversely affected — like, oof, not cute. The workaround: we let a VM's page tables be shadowed in multiple GTs. When the VM gets created, a default bind engine and PT table structure get created on each GT. ✨

Binds can optionally pass in a mask of GTs where the mapping should be created. If that mask is zero, then it just defaults to all the GTs where the VM has page tables. Easy.

Under the hood this whole thing is a bunch of for_each_gt loops scattered in various places, plus exporting a composite fence for multi-GT binds out to the user. That's it, that's the implementation. 🫶

💥 FAULT MODE (UNIFIED SHARED MEMORY) 💥

A VM can be put in fault mode on devices that support page faults. But — and this is the important part, bestie — if page faults are enabled, using dma fences can legitimately cause a deadlock:

  • A pending page fault can hold up the GPU work,
  • which holds up dma fence signaling,
  • and memory allocation is usually required to resolve a page fault,
  • BUT memory allocation is not allowed to gate dma fence signaling.

Chain reaction, everything stalls, disaster. 😭 So dma fences are not allowed when VM is in fault mode. And because dma fences aren't allowed, only long running workloads and ULLS are enabled on a faulting VM. Periodt.

🎀 Deferred VM binds 🎀

By default, on a faulting VM, binds just allocate the VMA and the actual updating of the page tables gets deferred to the page fault handler. If you're not about that life you can override it by setting the flag DRM_XE_VM_BIND_FLAG_IMMEDIATE in the VM bind, which then does the bind immediately. Your call, queen.

🧚 Page fault handler 🧚

ok so here's the messy part. Page faults are received in the G2H worker under the CT lock — and the CT lock is in the path of dma fences (where no memory allocations are allowed, but faults require memory allocations). So we literally cannot process faults under the CT lock. 🙈

Other issues, because of course there are:

  • Faults issue TLB invalidations, which require G2H credits, and we cannot allocate G2H credits in the G2H handlers without deadlocking. Bro.
  • Last but not least, we do not want the CT lock to be an outer lock of the VM global lock (and the VM global lock is required to do fault processing).

So to work around all that, we sink faults to a buffer that's large enough to hold all possible faults on the GT (1 per hardware engine), then kick a worker to go process them. Since page faults G2H are already received in a worker, kicking another worker adds latency to a critical performance path — so we add a fast path in the G2H irq handler: it looks at the first G2H, and if it's a page fault, we sink the fault to the buffer and kick the worker to process it. TLB invalidation responses are also in the critical path, so those can be processed in this fast path too. 💅

Multiple buffers and workers are used and hashed over based on the ASID, so faults from different VMs can be processed in parallel. Slay.

The page fault handler itself is rather simple, flow below:

Lookup VM from ASID in page fault G2H
Lock VM global lock in read mode
Lookup VMA from address in page fault G2H
Check if VMA is valid, if not bail
Check if VMA's BO has backing store, if not allocate
<----------------------------------------------------------------------|
If userptr, pin pages                                                  |
Lock VM & BO dma-resv locks                                            |
If atomic fault, migrate to VRAM, else validate BO location            |
Issue rebind                                                           |
Wait on rebind to complete                                             |
Check if userptr invalidated since pin                                 |
        Drop VM & BO dma-resv locks                                    |
        Retry ----------------------------------------------------------
Unlock all
Issue blocking TLB invalidation
Send page fault response to GuC

🍓 Access counters 🍓

Access counters can be configured to trigger a G2H that basically tells you "hey, the device is accessing VMAs in system memory A LOT" — a hint to migrate those VMAs to VRAM. 💖

Same deal as the page fault handler: access counters G2H cannot be processed in the G2H worker under the CT lock. Again we use a buffer to sink the access counter G2H. Unlike page faults, there is no upper bound — so if the buffer is full, we simply drop the G2H. And that's fine! Access counters are a best-case optimization and it's safe to drop them, unlike page faults. No cap.

The access counter handler itself is rather simple, flow below:

Lookup VM from ASID in access counter G2H
Lock VM global lock in read mode
Lookup VMA from address in access counter G2H
If userptr, bail nothing to do
Lock VM & BO dma-resv locks
Issue migration to VRAM
Unlock all

Notice: no rebind is issued in the access counter handler, because the rebind will be issued on the next page fault. Efficient. We love to see it. ✨

👑 Caveats with eviction / user pointer invalidation 👑

So in the case of eviction and user pointer invalidation on a faulting VM, there's no need to issue a rebind — instead we just blow away the page tables for the VMAs, and the page fault handler will rebind them when they fault. Simple enough, right?

the caveat (and it's a real one): to update / read the page table structure, the VM global lock is needed. But in both the eviction case and the user pointer invalidation case, locks are held which make acquiring the VM global lock impossible. 😭

The workaround: every VMA maintains a list of leaf page table entries which should be written to zero to blow away the VMA's page tables. After writing zero to these entries, a blocking TLB invalidate is issued. At that point it's safe for the kernel to move the VMA's memory around. This is a necessary lockless algorithm, and it's safe because leafs cannot be changed while either an eviction or a userptr invalidation is occurring. Lockless and flawless. 💅

🔒 LOCKING 🔒

VM locking protects all of the core data paths in Xe: bind operations, execs, evictions, and compute mode rebind worker. All of them. Every single one. 🫶

🎀 Locks 🎀

  • VM global lock (vm->lock) — rw semaphore lock. This is the outer most lock, and it protects the list of userptrs mapped in the VM, the list of engines using this VM, and the array of external BOs mapped in the VM. When adding or removing any of that state from the VM, you should acquire this lock in write mode. The VM bind path also acquires this lock in write, while the exec / compute mode rebind worker acquires it in read mode.
  • VM dma-resv lock (vm->gpuvm.r_obj->resv->lock) — WW lock. Protects VM dma-resv slots, which is shared with any private BO in the VM. Expected to be acquired during VM binds, execs, and the compute mode rebind worker. This lock is also held when private BOs are being evicted.
  • external BO dma-resv lock (bo->ttm.base.resv->lock) — WW lock. Protects external BO dma-resv slots. Expected to be acquired during VM binds (in addition to the VM dma-resv lock). All external BO dma-locks within a VM are expected to be acquired (in addition to the VM dma-resv lock) during execs and the compute mode rebind worker. This lock is also held when an external BO is being evicted.

💀 Putting it all together 💀

  1. An exec and bind operation with the same VM can't be executing at the same time (vm->lock).
  2. A compute mode rebind worker and bind operation with the same VM can't be executing at the same time (vm->lock).
  3. We can't add / remove userptrs or external BOs to a VM while an exec with the same VM is executing (vm->lock).
  4. We can't add / remove userptrs, external BOs, or engines to a VM while a compute mode rebind worker with the same VM is executing (vm->lock).
  5. Evictions within a VM can't be happen while an exec with the same VM is executing (dma-resv locks).
  6. Evictions within a VM can't be happen while a compute mode rebind worker with the same VM is executing (dma-resv locks).

🦋 DMA-RESV USAGE 🦋

As previously stated, to enforce the ordering of kernel ops (eviction, userptr invalidation, munmap style unbinds which result in a rebind), rebinds during execs, execs, and resumes in the rebind worker, we use both the VM's and external BOs' dma-resv slots. Let's try to make this as clear as possible, bestie. 💖

⭐ Slot installation ⭐

  1. Jobs from kernel ops install themselves into the DMA_RESV_USAGE_KERNEL slot of either an external BO or VM (depends on if the kernel op is operating on an external or private BO).
  2. In non-compute mode, jobs from execs install themselves into the DMA_RESV_USAGE_BOOKKEEP slot of the VM.
  3. In non-compute mode, jobs from execs install themselves into the DMA_RESV_USAGE_WRITE slot of all external BOs in the VM.
  4. Jobs from binds install themselves into the DMA_RESV_USAGE_BOOKKEEP slot of the VM.
  5. Jobs from binds install themselves into the DMA_RESV_USAGE_BOOKKEEP slot of the external BO (if the bind is to an external BO, this is in addition to #4).
  6. In compute mode, every engine that's using a compute mode VM gets a preempt fence dropped into the DMA_RESV_USAGE_PREEMPT_FENCE slot of all the external BOs in that VM. Not just one. All of them. 😤
  7. Wait — that one's basically the same as above but scoped to external BOs vs the VM itself. Both are real, both are listed, both are getting their own bullet. Slay. 💅

🎀 Slot waiting 🎀

Ok so this is the part where everything gets blocked behind everything else. Buckle up bestie:

  1. The execution of all jobs from kernel ops shall wait on all the DMA_RESV_USAGE_PREEMPT_FENCE slots of either an external BO or the VM — whichever one applies, and that depends on whether the kernel op is operating on an external BO or a private BO. iykyk 🦋
  2. In non-compute mode, the execution of all jobs from rebinds in execs waits on the DMA_RESV_USAGE_KERNEL slot of either an external BO or the VM — again, depends on whether the rebind is operating on an external BO or a private BO.
  3. In non-compute mode, the execution of all jobs from execs waits on the last rebind job. Periodt. ⭐
  4. In compute mode, the execution of all jobs from rebinds in the rebind worker waits on the DMA_RESV_USAGE_KERNEL slot of either an external BO or the VM — depends on whether the rebind is operating on an external BO or a private BO.
  5. In compute mode, resumes in the rebind worker wait on the last rebind fence.
  6. In compute mode, resumes in the rebind worker wait on the DMA_RESV_USAGE_KERNEL slot of the VM.

🌈 Putting it all together 🌈

The tl;dr of the whole blocking situation, no cap:

  1. New jobs from kernel ops are blocked behind any existing jobs from non-compute mode execs.
  2. New jobs from non-compute mode execs are blocked behind any existing jobs from kernel ops and rebinds.
  3. New jobs from kernel ops are blocked behind all preempt fences signaling in compute mode.
  4. Compute mode engine resumes are blocked behind any existing jobs from kernel ops and rebinds.

That's the whole vibe check. Four rules. Memorize them. 💖

🧚 FUTURE WORK 🧚

Stuff that's not done yet but is lowkey on the list:

  • Support large pages for sysmem and userptr. ✨
  • Update page faults to handle BOs at page-level granularity — like, part of a BO could be sitting in system memory while another part is chilling in VRAM. Two homes. Main character energy. 👑
  • The page fault handler will probably get optimized a little more. Examples of what's on the table: rebinds always wait on the dma-resv kernel slots of the VM or BO, but technically we only have to wait on the BO that's actually moving. If we use a job to do the rebind, we could avoid blocking in the page fault handler entirely and instead attach a callback to the fence of the rebind job to signal that the page fault is complete. And our handling of short-circuiting for atomic faults on bound VMAs could be better. Etc. etc. 💀
Fr fr, we can tune all of this once we have benchmarks and performance numbers from workloads up and running. Until then it's all vibes and speculation. 🫶

🔎 SEE ALSO 🔎

Kernel file ./drivers/gpu/drm/xe/xe_vm_doc.h — go read it, bestie, it's right there.

And that's a wrap on the whole doc! 💜 This lil masterpiece came from the source path ./drivers/gpu/drm/xe, dated October 2026, and yes, it's page Xe VM (user address space)(9). Go forth and manage those address spaces. Slay. 🎀✨

📦 where this page comes from