BUS_DMA(9)

📦 version 12.2, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 BUS_DMA(9) — Kernel Developer's Manual 💖

okay bestie, sit down, hydrate, and get comfy because we're about to talk about DMA and honestly? it's giving main character energy for your whole driver. this is BUS_DMA(9) from the Kernel Developer's Manual, and it is long, so we're doing this in parts. no sign-off at the end, we're just gonna keep going. 🦋

✨ NAME ✨

bus_dma, bus_dma_tag_create, bus_dma_tag_destroy, bus_dmamap_create, bus_dmamap_destroy, bus_dmamap_load, bus_dmamap_load_bio, bus_dmamap_load_ccb, bus_dmamap_load_mbuf, bus_dmamap_load_mbuf_sg, bus_dmamap_load_uio, bus_dmamap_unload, bus_dmamap_sync, bus_dmamem_alloc, bus_dmamem_free — the Bus and Machine Independent DMA Mapping Interface ✨

🎀 SYNOPSIS 🎀

#include <machine/bus.h>

int
bus_dma_tag_create(bus_dma_tag_t parent, bus_size_t alignment,
    bus_addr_t boundary, bus_addr_t lowaddr, bus_addr_t highaddr,
    bus_dma_filter_t *filtfunc, void *filtfuncarg, bus_size_t maxsize,
    int nsegments, bus_size_t maxsegsz, int flags,
    bus_dma_lock_t *lockfunc, void *lockfuncarg, bus_dma_tag_t *dmat);

int
bus_dma_tag_destroy(bus_dma_tag_t dmat);

int
bus_dmamap_create(bus_dma_tag_t dmat, int flags, bus_dmamap_t *mapp);

int
bus_dmamap_destroy(bus_dma_tag_t dmat, bus_dmamap_t map);

int
bus_dmamap_load(bus_dma_tag_t dmat, bus_dmamap_t map, void *buf,
    bus_size_t buflen, bus_dmamap_callback_t *callback,
    void *callback_arg, int flags);

int
bus_dmamap_load_bio(bus_dma_tag_t dmat, bus_dmamap_t map,
    struct bio *bio, bus_dmamap_callback_t *callback,
    void *callback_arg, int flags);

int
bus_dmamap_load_ccb(bus_dma_tag_t dmat, bus_dmamap_t map,
    union ccb *ccb, bus_dmamap_callback_t *callback, void *callback_arg,
    int flags);

int
bus_dmamap_load_mbuf(bus_dma_tag_t dmat, bus_dmamap_t map,
    struct mbuf *mbuf, bus_dmamap_callback2_t *callback,
    void *callback_arg, int flags);

int
bus_dmamap_load_mbuf_sg(bus_dma_tag_t dmat, bus_dmamap_t map,
    struct mbuf *mbuf, bus_dma_segment_t *segs, int *nsegs, int flags);

int
bus_dmamap_load_uio(bus_dma_tag_t dmat, bus_dmamap_t map,
    struct uio *uio, bus_dmamap_callback2_t *callback,
    void *callback_arg, int flags);

void
bus_dmamap_unload(bus_dma_tag_t dmat, bus_dmamap_t map);

void
bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map, op);

int
bus_dmamem_alloc(bus_dma_tag_t dmat, void **vaddr, int flags,
    bus_dmamap_t *mapp);

void
bus_dmamem_free(bus_dma_tag_t dmat, void *vaddr, bus_dmamap_t map);

🌸 DESCRIPTION 🌸

ok so Direct Memory Access (DMA) is the method of moving data around without dragging the CPU into it, which means higher performance. periodt. and a DMA transaction can happen device-to-memory, device-to-device, or memory-to-memory. all three, no cap. 💅

the bus_dma API is a bus-, device-, and machine-independent (MI) interface to DMA mechanisms. it hands the client flexibility and simplicity by abstracting away all the machine dependent drama — like setting up DMA mappings, dealing with cache issues, and all the bus specific features and limitations. it's literally the bestie that handles the icky parts for you. 🫶

🦋 OVERVIEW 🦋

a tag structure (bus_dma_tag_t) describes the properties of a group of related DMA transactions. one way to think about it: a tag is basically the vibe check for a DMA engine's limitations. for example, if a DMA engine in a device is stuck at 32-bit addresses only, that limitation gets specified by a parameter when you create the tag for that device. same energy: a tag can be marked as requiring buffers whose addresses are aligned to a specific boundary. ✨

some devices need multiple tags to describe DMA transactions with differing properties. like, a device might need 16-byte alignment of its descriptor ring but be totally fine with arbitrary alignment of I/O buffers. in that case the driver has to create one tag for the descriptor ring and a separate tag for I/O buffers. and if a device has restrictions that are common to all DMA transactions on top of restrictions that differ between unrelated groups of transactions, the driver can first create a “parent” tag that describes the common restrictions. then the per-group tags can inherit those restrictions from the “parent” tag instead of having to list them all out explicitly when creating each per-group tag. inheritance, but make it DMA. 💀

a mapping structure (bus_dmamap_t) represents a mapping of a memory region for DMA. on systems with I/O MMUs, the mapping structure tracks any I/O MMU entries used by a request. for DMA requests that need bounce pages, the mapping tracks the bounce pages used. tracking girlies stay winning. 🎀

to prepare for one or more DMA transactions, a mapping has to be bound to a memory region by calling one of the bus_dmamap_load() functions. these functions configure the mapping, which can include programming entries in an I/O MMU and/or allocating bounce pages. an output of these functions — either directly or indirectly by invoking a callback routine — is the list of scatter/gather address ranges that a consumer can pass to a DMA engine to access the memory region. and when a mapping is no longer needed, you must unload it via bus_dmamap_unload(). no leaving it lying around. 🙈

before and after each DMA transaction, bus_dmamap_sync() must be used to make sure the correct data is used by the DMA engine and the CPU. if a mapping uses bounce pages, the sync operations copy data between the bounce pages and the memory region bound to the mapping. sync operations also handle architecture-specific details like CPU cache flushing and CPU memory operation ordering. it's giving ✨responsible✨. ⭐

🌈 STATIC VS DYNAMIC 🌈

bus_dma handles two types of DMA transactions: static and dynamic.

static transactions are used with a long-lived memory region that gets reused for many transactions — think a descriptor ring. dynamic transactions are used for transfers to or from transient buffers — think an I/O buffer holding a network packet or a disk block. each transaction type uses a different subset of the bus_dma API. choose wisely bestie. 🌸

💜 Static Transactions 💜

static transactions use memory regions allocated by bus_dma. each static memory region is allocated by calling bus_dmamem_alloc(). this function needs a valid tag describing the properties of the DMA transactions to this region, like alignment or address restrictions. multiple regions can share a single tag if they share the same restrictions. sharing is caring. 🫶

bus_dmamem_alloc() allocates a memory region along with a mapping object. the associated tag, memory region, and mapping object must then be passed to bus_dmamap_load() to bind the mapping to the allocated region and get the scatter/gather list. ✨

it's expected that bus_dmamem_alloc() will attempt to allocate memory that requires less expensive sync operations — for example, implementations should not allocate regions requiring bounce pages — but sync operations should still be used regardless. for example, a driver should use bus_dmamap_sync() in an interrupt handler before reading descriptor ring entries written by the device prior to the interrupt. don't skip the sync, i'm literally begging. 😭

when a consumer is finished with a memory region, it should unload the mapping via bus_dmamap_unload() and then release the memory region and mapping object via bus_dmamem_free(). clean up after yourself, icon behavior. 👑

🧚 Dynamic Transactions 🧚

dynamic transactions map memory regions provided by other parts of the system. a tag must be created via bus_dma_tag_create() to describe the DMA transactions to and from these memory regions, and a pool of mapping objects must be allocated via bus_dmamap_create() to track the mappings of any in-flight transactions. 🔥

when a consumer wants to schedule a transaction for a memory region, it must first grab an unused mapping object from its pool of mapping objects. the memory region must be bound to that mapping object via one of the bus_dmamap_load() functions. before scheduling the transaction, the consumer should sync the memory region via bus_dmamap_sync() with one or more of the “PRE” flags. after the transaction has completed, the consumer should sync the memory region via bus_dmamap_sync() with one or more of the “POST” flags. then the mapping can be unloaded via bus_dmamap_unload(), and the mapping object can be returned to the pool of unused mapping objects. PRE, POST, unload, return — the whole cycle, no shortcuts. 💅

and when a consumer is no longer scheduling DMA transactions, the mapping objects should be freed via bus_dmamap_destroy(), and the tag should be freed via bus_dma_tag_destroy(). the end. for now. 🦋

🦋 STRUCTURES AND TYPES 🦋

💅 bus_dma_tag_t

Okay bestie, this one is a machine-dependent (MD) opaque type — so like, you don't get to peek inside, it's MD's business 💅 — and it describes the characteristics of a group of DMA transactions. DMA tags are organized into a hierarchy, and every child tag inherits the restrictions of its parent. The way this slays is that it lets all devices along the path of DMA transactions contribute to the constraints of those transactions. Iconic behavior, honestly.

🎀 bus_dma_filter_t

A client-specified address filter, and it's giving this exact format:

int     client_filter(void *filtarg, bus_addr_t testaddr)

Address filters can be specified during tag creation so that devices whose DMA address restrictions can't be squeezed into a single window still get to eat 🍓. The filtarg argument is specified by the client during tag creation, and it gets passed to all invocations of the callback. The testaddr argument contains a potential starting address of a DMA mapping. The filter function operates on the set of addresses from testaddr to trunc_page(testaddr) + PAGE_SIZE - 1, inclusive. And the vibe check on return: the filter function should return zero if any mapping in this range can be accommodated by the device, and non-zero otherwise. Periodt.

✨ bus_dma_segment_t

A machine-dependent type that describes individual DMA segments. It's holding these fields:

bus_addr_t      ds_addr;
bus_size_t      ds_len;

The ds_addr field contains the device visible address of the DMA segment, and ds_len contains the length of the DMA segment. Now listen — even though the DMA segments returned by a mapping call will adhere to all restrictions necessary for a successful DMA operation, some conversion (e.g. a conversion from host byte order to the device's byte order) is almost always required when presenting segment information to the device. So don't skip that step, bestie, it's not optional and it's not cute to forget 😭.

🦋 bus_dmamap_t

A machine-dependent opaque type describing an individual mapping. One map is used for each memory allocation that will be loaded. Maps can be reused once they have been unloaded. Multiple maps can be associated with one DMA tag. And here's the cautionary tale: while the value of the map may evaluate to NULL on some platforms under certain conditions, it should never be assumed that it will be NULL in all cases. So no assumptions, got it? 🙈

🌸 bus_dmamap_callback_t

A client-specified callback for receiving mapping information resulting from the load of a bus_dmamap_t via bus_dmamap_load(), bus_dmamap_load_bio() or bus_dmamap_load_ccb(). Callbacks are giving:

void    client_callback(void *callback_arg, bus_dma_segment_t *segs, int nseg, int error)

The callback_arg is the callback argument passed to dmamap load functions. The segs and nseg arguments describe an array of bus_dma_segment_t structures that represent the mapping. Important: this array is only valid within the scope of the callback function — like, do not go keeping it around after, it's not yours to keep 💀. The success or failure of the mapping is indicated by the error argument. More info on the use of callbacks can be found in the description of the individual dmamap load functions.

🎀 bus_dmamap_callback2_t

A client-specified callback for receiving mapping information resulting from the load of a bus_dmamap_t via bus_dmamap_load_uio() or bus_dmamap_load_mbuf().

Callback2s are giving:

void    client_callback2(void *callback_arg, bus_dma_segment_t *segs, int nseg, bus_size_t mapsize, int error)

Callback2's behavior is the same as bus_dmamap_callback_t, with the little extra that the length of the data mapped is provided via mapsize. She's just callback with a bonus, slay ✨.

💜 bus_dmasync_op_t

Memory synchronization operation specifier. Bus DMA requires explicit synchronization of memory with its device visible mapping in order to guarantee memory coherency — so like, no, it doesn't happen by magic, bestie. The bus_dmasync_op_t allows the type of DMA operation that will be or has been performed to be communicated to the system so that the correct coherency measures are taken. The operations are represented as bitfield flags that can be combined together, though it only makes sense to combine PRE flags or POST flags, not both. See the bus_dmamap_sync() description below for more details on how to use these operations.

All operations specified below are performed from the host memory point of view, where a read implies data coming from the device to the host memory, and a write implies data going from the host memory to the device. Alternatively, the operations can be thought of in terms of driver operations, where reading a network packet or storage sector corresponds to a read operation in bus_dma. Two lenses, same facts, iykyk 👑

  • BUS_DMASYNC_PREREAD — Perform any synchronization required prior to an update of host memory by the device.
  • BUS_DMASYNC_PREWRITE — Perform any synchronization required after an update of host memory by the CPU and prior to device access to host memory.
  • BUS_DMASYNC_POSTREAD — Perform any synchronization required after an update of host memory by the device and prior to CPU access to host memory.
  • BUS_DMASYNC_POSTWRITE — Perform any synchronization required after device access to host memory.

🔐 bus_dma_lock_t

Client-specified lock/mutex manipulation method. This will be called from within busdma whenever a client lock needs to be manipulated. In its current form, the function will be called immediately before the callback for a DMA load operation that has been deferred with BUS_DMA_LOCK, and immediately after with BUS_DMA_UNLOCK. If the load operation does not need to be deferred, then it will not be called, since the function loading the map should be holding the appropriate locks already. This method is giving:

void    lockfunc(void *lockfunc_arg, bus_dma_lock_op_t op)

The lockfuncarg argument is specified by the client during tag creation to be passed to all invocations of the callback. The op argument specifies the lock operation to perform.

Two lockfunc implementations are provided for convenience, you're welcome 🫶. busdma_lock_mutex() performs standard mutex operations on the sleep mutex provided via lockfuncarg. dflt_lock() will generate a system panic if it is called 💀 — yes, for real. It is substituted into the tag when lockfunc is passed as NULL to bus_dma_tag_create() and is useful for tags that should not be used with deferred load operations.

🌈 bus_dma_lock_op_t

Operations to be performed by the client-specified lockfunc().

  • BUS_DMA_LOCK — Acquires and/or locks the client locking primitive.
  • BUS_DMA_UNLOCK — Releases and/or unlocks the client locking primitive.

⚡ FUNCTIONS ⚡

👑 bus_dma_tag_create

bus_dma_tag_create(parent, alignment, boundary, lowaddr, highaddr,
        *filtfunc, *filtfuncarg, maxsize, nsegments, maxsegsz, flags,
        lockfunc, lockfuncarg, *dmat)

Allocates a DMA tag, and initializes it according to the arguments provided. Let's walk through every single one, no skips, we don't do that here 💖:

  • parent — A parent tag from which to inherit restrictions. The restrictions passed in other arguments can only further tighten the restrictions inherited from the parent tag.

All tags created by a device driver must inherit from the tag returned by bus_get_dma_tag() to honor restrictions between the parent bridge, CPU memory, and the device. Like, that's not a suggestion, bestie.

  • alignment — Alignment constraint, in bytes, of any mappings created using this tag. The alignment must be a power of 2. Hardware that can DMA starting at any address would specify 1 for byte alignment. Hardware requiring DMA transfers to start on a multiple of 4K would specify 4096.
  • boundary — Boundary constraint, in bytes, of the target DMA memory region. The boundary indicates the set of addresses, all multiples of the boundary argument, that cannot be crossed by a single bus_dma_segment_t. The boundary must be a power of 2 and must be no smaller than the maximum segment size. 0 indicates that there are no boundary restrictions.
  • lowaddr, highaddr — Bounds of the window of bus address space that cannot be directly accessed by the device. The window contains all addresses greater than lowaddr and less than or equal to highaddr. For example, a device incapable of DMA above 4GB would specify a highaddr of BUS_SPACE_MAXADDR and a lowaddr of BUS_SPACE_MAXADDR_32BIT. Similarly, a device that can only perform DMA to addresses below 16MB would specify a highaddr of BUS_SPACE_MAXADDR and a lowaddr of BUS_SPACE_MAXADDR_24BIT. Some implementations require that some region of device visible address space, overlapping available host memory, be outside the window. This area of safe memory is used to bounce requests that would otherwise conflict with the exclusion window.
  • filtfunc — Optional filter function (may be NULL) to be called for any attempt to map memory into the window described by lowaddr and highaddr. A filter function is only required when the single window described by lowaddr and highaddr cannot adequately describe the constraints of the device. The filter function will be called for every machine page that overlaps the exclusion window.
  • filtfuncarg — the lil argument that gets passed to every single call of the filter function for this tag. May be NULL, no pressure. 🫶
  • maxsize — biggest possible size, in bytes, of the sum of all segment lengths in any one DMA mapping hanging out with this tag. Like the total vibe budget for the whole mapping. 💸
  • nsegments — how many discontinuities (aka scatter/gather segments) are allowed inside a DMA mapped region. If there's literally no restriction, you can pass BUS_SPACE_UNRESTRICTED and let it be free. 🦋
  • maxsegsz — maximum size, in bytes, of a single segment in any DMA mapped region associated with dmat. One segment, one cap. ✨
  • flags — and here's the tea:
    • BUS_DMA_ALLOCNOW — pre-allocate enough resources to handle at least one map load operation on this tag. If there aren't enough resources to go around, you get ENOMEM back. Do not use this for tags that only describe buffers that'll be allocated with bus_dmamem_alloc(). Also, because resources get shared with other tags, this flag does not guarantee that resources are allocated or reserved exclusively for this tag. Treat it as a minor optimization and nothing more. Not the main character, just a supporting role. 💅
    • BUS_DMA_COHERENT — says the DMA engine and the CPU are cache-coherent, so cached memory may be used to back allocations created by bus_dmamem_alloc(). For bus_dma_tag_create(), this flag is currently implemented on arm64. 🔥
  • lockfunc — an optional lock manipulation function (may be NULL) that gets called when busdma needs to mess with a lock on behalf of the client. If you specify NULL, then dflt_lock() gets used instead. 🙈
  • lockfuncarg — optional argument passed along to whatever function lockfunc points at.
  • dmat — pointer to a bus_dma_tag_t where the resulting DMA tag will be stored. That's where the baby tag goes. 🎀
Returns ENOMEM if there isn't enough memory available for tag creation or for allocating mapping resources. 😭

💖 bus_dma_tag_destroy(dmat)

Deallocates the DMA tag dmat that was created by bus_dma_tag_create(). Saying goodbye, closing the tab, moving on. 👋

Returns EBUSY if any DMA maps are still associated with dmat, or ‘0’ on success. (Yes, the manpage really quotes that zero like that, we keep it exactly as written, no cap.) ✨

🌸 bus_dmamap_create(dmat, flags, *mapp)

Allocates and initializes a DMA map. Fresh map, who dis. Arguments go like this:

  • dmat — DMA tag. The tag it belongs to. 💜
  • flags — as follows:
    • BUS_DMA_COHERENT — attempt to map the memory loaded with this map so that cache sync operations are as cheap as possible. This flag is typically set on maps when the memory loaded with them is going to be accessed by both a CPU and a DMA engine, and frequently at that — think control data, as opposed to streamable data like receive and transmit buffers. Heads up though: using this flag does not remove the requirement of using bus_dmamap_sync(), it just may reduce the cost of doing those operations. For bus_dmamap_create(), the BUS_DMA_COHERENT flag is currently implemented on sparc64. 💫
  • mapp — pointer to a bus_dmamap_t where the resulting DMA map will be stored. 🎀
Returns ENOMEM if there isn't enough memory available for creating the map or for allocating mapping resources. 💀

🌈 bus_dmamap_destroy(dmat, map)

Frees all resources associated with a given DMA map. Everything. Gone. Clean slate. 🧹

  • dmat — the DMA tag used to allocate the map.
  • map — the DMA map to destroy.
Returns EBUSY if a mapping is still active for map. Not me trying to destroy a map that's still busy — wait, that's literally what this is. 😭

⭐ bus_dmamap_load(dmat, map, buf, buflen, *callback, callback_arg, flags)

Creates a mapping in device visible address space of buflen bytes of buf, associated with the DMA map map. And here's the bestie part: this call will always return immediately and will not block for any reason. Fr fr, it never blocks. 🔥

  • dmat — DMA tag used to allocate map.
  • map — a DMA map without a currently active mapping.
  • buf — a kernel virtual address pointer to a contiguous (in KVA) buffer, to be mapped into device visible address space. 🦋
  • buflen — the size of the buffer.
  • callback, callback_arg — the callback function and its argument. This function gets called once sufficient mapping resources are available for the DMA operation. If resources are temporarily unavailable, this function will be deferred until later, but the load operation will still return immediately to the caller. So callers should not assume that the callback will be called before the load returns, and code should be structured appropriately to handle that. See below for the specific flags and error codes that control this behavior. 👑
  • flags — as follows:
    • BUS_DMA_NOWAIT — the load should not be deferred if there aren't enough mapping resources; instead it should return immediately with an appropriate error. No waiting, we're busy. 💅
    • BUS_DMA_NOCACHE — the generated transactions to and from the virtual page are non-cacheable. For bus_dmamap_load(), the BUS_DMA_NOCACHE flag is currently implemented on sparc64. ✨

Return values to the caller are as follows:

  • 0 — the callback has been called and completed. The status of the mapping has been delivered to the callback. All done, slayed. ✅
  • EINPROGRESS — the mapping has been deferred for lack of resources. The callback will be called as soon as resources are available. Callbacks are serviced in FIFO order. 📋
Note that subsequent load operations for the same tag that do not require extra resources will still succeed. This may result in out-of-order processing of requests. If the caller requires the order of requests to be preserved, then the caller is required to stall subsequent requests until a pending request's callback is invoked. Order is not free, bestie. 😤
  • ENOMEM — the load request has failed due to insufficient resources, and the caller specifically used the BUS_DMA_NOWAIT flag.
  • EINVAL — the load request was invalid. The callback has been called and has been provided the same error. This error value may indicate that dmat, map, buf, or callback were invalid, or that buflen was larger than the maxsize argument used to create the dma tag dmat. Something was off and it's telling you. 🙈

When the callback is called, it's presented with an error value indicating the disposition of the mapping. Error may be one of the following:

  • 0 — the mapping was successful and the dm_segs callback argument contains an array of bus_dma_segment_t elements describing the mapping. This array is only valid during the scope of the callback function. Do not keep it, do not pass it around, do not save it for later. It expires like a story. ⌛
  • EFBIG — a mapping could not be achieved within the segment constraints provided in the tag, even though the requested allocation size was less than maxsize. The vibes were fine, the segments said no. 💔

🦋 bus_dmamap_load_bio(dmat, map, bio, callback, callback_arg, flags)

This is a variation of bus_dmamap_load() which maps buffers pointed to by bio for DMA transfers. bio may point to either a mapped or an unmapped buffer. Either way, we handle it. 🫶

🎀 bus_dmamap_load_ccb(dmat, map, ccb, callback, callback_arg, flags)

This is a variation of bus_dmamap_load() which maps data pointed to by ccb for DMA transfers. The data for ccb may be any of the following types:

  • CAM_DATA_VADDR — the data is a single KVA buffer.
  • CAM_DATA_PADDR — the data is a single bus address range.
  • CAM_DATA_SG — the data is a scatter/gather list of KVA buffers.
  • CAM_DATA_SG_PADDR — the data is a scatter/gather list of bus address ranges.
  • CAM_DATA_BIO — the data is contained in a struct bio attached to the CCB.

bus_dmamap_load_ccb() supports the following CCB XPT function codes:

XPT_ATA_IO
XPT_CONT_TARGET_IO
XPT_SCSI_IO

Three icons, three function codes. 💅

💜 bus_dmamap_load_mbuf(dmat, map, mbuf, callback2, callback_arg, flags)

This is a variation of bus_dmamap_load() which maps mbuf chains for DMA transfers. A bus_size_t argument is also passed to the callback routine, which contains the mbuf chain's packet header length. 📦

The BUS_DMA_NOWAIT flag is implied here, so no callback deferral will happen. Straight through, no queue, no waiting room. ✨

Mbuf chains are assumed to be in kernel virtual address space. 🧠

Heads up bestie — on top of all the error values that bus_dmamap_load() can throw at you, EINVAL shows up too if your mbuf chain is bigger than the DMA tag's max limit. No cap.

bus_dmamap_load_mbuf_sg(dmat, map, mbuf, segs, nsegs, flags)

Okay so this is literally bus_dmamap_load_mbuf() but it does not call a callback function — it just returns right away. It exists purely for efficiency, and honestly? Iconic behavior. ✨

The scatter/gather segment array segs is supplied by you, the caller, and the function fills it in directly. The nsegs argument gets returned holding the number of segments that got filled in. It returns the exact same errors as bus_dmamap_load_mbuf().

bus_dmamap_load_uio(dmat, map, uio, callback2, callback_arg, flags)

A variation on bus_dmamap_load() that maps buffers pointed to by uio for DMA transfers. 🦋 A bus_size_t argument also gets passed to the callback routine, and it holds the size of uio, i.e. uio->uio_resid. The BUS_DMA_NOWAIT flag is implied here — so no callback deferral is ever gonna happen. Returns the same errors as bus_dmamap_load().

If uio->uio_segflg is UIO_USERSPACE, then it's assumed the buffer uio lives in uio->uio_td->td_proc's address space. User space memory has to be in-core and wired before you even attempt a map load operation. Pages can be locked using vslock(9).

bus_dmamap_unload(dmat, map)

Unloads a DMA map. The arguments, bestie:

  • dmat — DMA tag used to allocate map.
  • map — the DMA map that is to be unloaded.
bus_dmamap_unload() will not do any implicit synchronization of DMA buffers. You have to do that yourself, explicitly, by calling bus_dmamap_sync() before unloading the map. No shortcuts. 🫶

bus_dmamap_sync(dmat, map, op)

Performs synchronization of a device visible mapping with the CPU visible memory referenced by that mapping. The arguments:

  • dmat — DMA tag used to allocate map.
  • map — the DMA mapping to be synchronized.
  • op — type of synchronization operation to perform. See the definition of bus_dmasync_op_t for a description of the acceptable values for op.

The bus_dmamap_sync() function is the method for making sure the CPU's and the device's direct memory access (DMA) to shared memory is coherent. 🌈

Say the CPU is being used to set up the contents of a buffer that's gonna be made available to a device. To make sure that data is visible via the device's mapping of that memory, the buffer must be loaded and a DMA sync operation of BUS_DMASYNC_PREWRITE must be performed after the CPU has updated the buffer and before the device access kicks off. If the CPU modifies that same buffer again later, another BUS_DMASYNC_PREWRITE sync operation must be performed before an additional device access. 🔁

Conversely — imagine a device updates memory that a CPU is gonna read. In that case, the buffer must be loaded, and a DMA sync operation of BUS_DMASYNC_PREREAD must be performed before the device access is initiated. The CPU will only be able to see the results of this memory update once the DMA operation has completed and a BUS_DMASYNC_POSTREAD sync operation has been performed.

If read and write operations are not preceded and followed by the appropriate synchronization operations, behavior is undefined. Not "a little weird", not "probably fine" — undefined. 😭

bus_dmamem_alloc(dmat, **vaddr, flags, *mapp)

Allocates memory that is mapped into KVA at the address returned in vaddr, and that is permanently loaded into the newly created bus_dmamap_t returned via mapp. The arguments:

  • dmat — DMA tag describing the constraints of the DMA mapping.
  • vaddr — pointer to a pointer that will hold the returned KVA mapping of the allocated region.
  • flags — flags are defined as follows:
    • BUS_DMA_WAITOK — the routine can safely wait (sleep) for resources.
    • BUS_DMA_NOWAIT — the routine is not allowed to wait for resources. If resources aren't available, ENOMEM is returned.
    • BUS_DMA_COHERENT — attempt to map this memory in a coherent fashion. See bus_dmamap_create() above for a description of this flag. For bus_dmamem_alloc(), the BUS_DMA_COHERENT flag is currently implemented on arm, arm64 and sparc64.
    • BUS_DMA_ZERO — causes the allocated memory to be set to all zeros.
    • BUS_DMA_NOCACHE — the allocated memory will not be cached in the processor caches. All memory accesses appear on the bus and are executed without reordering. For bus_dmamem_alloc(), the BUS_DMA_NOCACHE flag is currently implemented on amd64 and i386, where it results in the Strong Uncacheable PAT to be set for the allocated virtual address range.
  • mapp — pointer to a bus_dmamap_t where the resulting DMA map will be stored.

The size of memory to be allocated is maxsize as specified in the call to bus_dma_tag_create() for dmat.

The current implementation of bus_dmamem_alloc() will allocate all requests as a single segment. ✨

An initial load operation is required to obtain the bus address of the allocated memory, and an unload operation is required before freeing the memory, as described below in bus_dmamem_free(). Maps are automatically handled by this function and should not be explicitly allocated or destroyed. 🙅

Even though an explicit load isn't required for each access to the memory referenced by the returned map, the synchronization requirements described in the bus_dmamap_sync() section still apply, and you should use them to achieve portability on architectures without coherent buses.

Returns ENOMEM if sufficient memory isn't available for completing the operation.

bus_dmamem_free(dmat, *vaddr, map)

Frees memory previously allocated by bus_dmamem_alloc(). Any mappings will be invalidated. The arguments:

  • dmat — DMA tag.
  • vaddr — kernel virtual address of the memory.
  • map — DMA map to be invalidated.

💀 RETURN VALUES 💀

Behavior is undefined if invalid arguments are passed to any of the above functions. If sufficient resources can't be allocated for a given transaction, ENOMEM is returned. All routines that aren't of type void will return 0 on success or an error code on failure, as discussed above.

All void routines will succeed if provided with valid arguments. 💅

🔒 LOCKING 🔒

Two locking protocols are used by bus_dma. Let's get into it.

The first is a private global lock that's used to synchronize access to the bounce buffer pool on the architectures that make use of them. This lock is strictly a leaf lock, it's only used internally to bus_dma, and it is not exposed to clients of the API. 🍓

The second protocol is about protecting the various resources stored in the tag. Since almost all bus_dma operations happen through requests from the driver that created the tag, the most efficient way to protect the tag resources is through the lock that the driver uses. In cases where bus_dma acts on its own without being called by the driver, the lock primitive specified in the tag is acquired and released automatically. 🌸

An example of this is when the bus_dmamap_load() callback function is called from a deferred context instead of the driver context. This means certain bus_dma functions must always be called with the same lock held that is specified in the tag. These functions include:

  • bus_dmamap_load()
  • bus_dmamap_load_bio()
  • bus_dmamap_load_ccb()
  • bus_dmamap_load_mbuf()
  • bus_dmamap_load_mbuf_sg()
  • bus_dmamap_load_uio()
  • bus_dmamap_unload()
  • bus_dmamap_sync()

There's one exception to this rule, bestie. It's common practice to call some of these functions during driver start-up without any locks held. So long as there's a guarantee of no possible concurrent use of the tag by different threads during this operation, it's safe to not hold a lock for these functions. 👑

Certain bus_dma operations should not be called with the driver lock held — either because they're already protected by an internal lock, or because they might sleep due to memory or resource allocation. The following functions must not be called with any non-sleepable locks held:

  • bus_dma_tag_create()
  • bus_dmamap_create()
  • bus_dmamem_alloc()

All other functions don't have a locking protocol and can therefore be called with or without any system or driver locks held. 🎀

👀 SEE ALSO 👀

🦋 HISTORY 🦋

The bus_dma interface first appeared in NetBSD 1.3.

The bus_dma API was adopted from NetBSD for use in the CAM SCSI subsystem. The alterations to the original API were aimed to remove the need for a bus_dma_segment_t array stored in each bus_dmamap_t, while allowing callers to queue up on scarce resources. 🌈

👑 AUTHORS 👑

Ok bestie, time for the credits, and honestly? these people are the moment. The whole bus_dma interface — the design, the implementation, the entire slay — was dreamed up and built by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, NASA Ames Research Center. 🚀 Like, actual rocket-science-adjacent energy, no cap.

And the design got extra brainpower from the group chat too — additional input on the bus_dma design came from Chris Demetriou, Charles Hannum, Ross Harvey, Matthew Jacob, Jonathan Stone, and Matt Thomas. 🫶

The bus_dma interface in FreeBSD also benefits from the contributions of Justin T. Gibbs, Peter Wemm, Doug Rabson, Matthew N. Dodd, Sam Leffler, Maxime Henrion, Jake Burkholder, Takahashi Yoshihiro, Scott Long and many others. That's a whole roster, and every single one of them ate. 💅✨

And this very manual page — the one you're reading right now, the one that got the full group-chat glow-up — was written by Hiten M. Pandya and Justin T. Gibbs. 📝💖

Big ups to everyone above. None of this DMA goodness would exist without them, fr fr. 🌸🦋

And that's a wrap on BUS_DMA(9), bestie! This page came from the Debian source, dated August 11, 2018 — go treat your bus_dma calls right, use your bus_dmamap_sync before you touch that memory, and remember: you are the main character of your own driver. 👑💜 Byeee! 🎀⭐🌈

📦 where this page comes from