MLX5DV_WR(3)

📦
version 62.0 · 8 versions
, in alpine 3.24, centos-stream 10, centos-stream 9, ubuntu jammy

💖 MLX5DV_WR 💖

okay bestie, welcome to the mlx5 work-request glow-up page, straight out of the mlx5 Programmer's Manual. It's section 3, it's iconic, and it's about to make your send queue thrive. 🦋

✨ NAME ✨

so there's like three whole legends on this page, and they're all name-checked right here:

  • mlx5dv_wr_set_dc_addr — attaches a DC info to the last work request
  • mlx5dv_wr_raw_wqe — builds a raw work request
  • mlx5dv_wr_memcpy — builds a DMA memcpy work request

✨ SYNOPSIS ✨

#include <infiniband/mlx5dv.h>

static inline void mlx5dv_wr_set_dc_addr(struct mlx5dv_qp_ex *mqp,
                                         struct ibv_ah *ah,
                                         uint32_t remote_dctn,
                                         uint64_t remote_dc_key);

static inline void mlx5dv_wr_set_dc_addr_stream(struct mlx5dv_qp_ex *mqp,
                                struct ibv_ah *ah,
                                uint32_t remote_dctn,
                                uint64_t remote_dc_key,
                                uint16_t stream_id);

struct mlx5dv_mr_interleaved {
    uint64_t        addr;
    uint32_t        bytes_count;
    uint32_t        bytes_skip;
    uint32_t        lkey;
};

static inline void mlx5dv_wr_mr_interleaved(struct mlx5dv_qp_ex *mqp,
                                struct mlx5dv_mkey *mkey,
                                uint32_t access_flags, /* use enum ibv_access_flags */
                                uint32_t repeat_count,
                                uint16_t num_interleaved,
                                struct mlx5dv_mr_interleaved *data);

static inline void mlx5dv_wr_mr_list(struct mlx5dv_qp_ex *mqp,
                              struct mlx5dv_mkey *mkey,
                              uint32_t access_flags, /* use enum ibv_access_flags */
                              uint16_t num_sges,
                              struct ibv_sge *sge);

static inline int mlx5dv_wr_raw_wqe(struct mlx5dv_qp_ex *mqp, const void *wqe);

static inline void mlx5dv_wr_memcpy(struct mlx5dv_qp_ex *mqp_ex,
                            uint32_t dest_lkey, uint64_t dest_addr,
                            uint32_t src_lkey, uint64_t src_addr,
                            size_t length)

✨ DESCRIPTION ✨

okay so the MLX5DV work request APIs (that's the mlx5dv_wr_* fam 💅) are literally an extension for the IBV work request API (ibv_wr_*) — but with all the mlx5-specific features for send work requests sprinkled on top. And the best part? You can use them together with or without ibv_wr_* calls. no gatekeeping, we love flexibility 🫶

✨ USAGE ✨

so to actually use these APIs, a QP has to be created using mlx5dv_create_qp() with send_ops_flags of struct ibv_qp_init_attr_ex set. periodt.

🚨 heads up bestie: if the QP doesn't support all the requested work request types, QP creation will fail. not a vibe check you wanna flunk.

The mlx5dv_qp_ex gets extracted from the IBV_QP by ibv_qp_to_qp_ex() and mlx5dv_qp_ex_from_ibv_qp_ex(). That's what you use to apply the mlx5-specific features onto the posted WR. 🌸

A work request creation requires you to use the ibv_qp_ex as described in the man page for ibv_wr_post and mlx5dv_qp, with all its available builders and setters. iykyk.

🎀 QP Specific builders 🎀

RC QPs — mlx5dv_wr_mr_interleaved()

registers an interleaved memory layout by using an indirect mkey and some interleaved data. The layout of the memory pointed to by the mkey after its registration will be the data representation for the num_interleaved entries. And that single layout representation is repeated by repeat_count. it's giving pattern, it's giving repetition ✨

The data as described by struct mlx5dv_mr_interleaved will hold real data defined by bytes_count and then a padding of bytes_skip. Post a successful registration, RDMA operations can use this mkey. The hardware will scatter the data according to the pattern. The mkey should be used in a zero-based mode. The addr field in its ibv_sge is an offset in the total data. To create this mkey, mlx5dv_create_mkey() should be used.

😭 Current implementation requires the IBV_SEND_INLINE option to be on in ibv_qp_ex->wr_flags field. To be able to have more than 3 num_interleaved entries, the QP should be created with a larger WQE size that may fit it. This should be done using the max_inline_data attribute of struct ibv_qp_cap upon its creation.

As one entry will be consumed for strided header, the mkey should be created with one more entry than the required num_interleaved. lowkey important, don't skip it 🫶

In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the reported WC opcode will be MLX5DV_WC_UMR. Unregister the mkey to enable another pattern registration — this should be done via ibv_post_send with IBV_WR_LOCAL_INV opcode.

mlx5dv_wr_mr_list()

registers a memory layout based on a list of ibv_sge. The layout of the memory pointed to by the mkey after its registration will be based on the list of sge counted by num_sges. Post a successful registration, RDMA operations can use this mkey, and the hardware will scatter the data according to the pattern. The mkey should be used in a zero-based mode, and the addr field in its ibv_sge is an offset in the total data. chef's kiss 🎀

😭 Current implementation requires the IBV_SEND_INLINE option to be on in ibv_qp_ex->wr_flags field. To be able to have more than 4 num_sge entries, the QP should be created with a larger WQE size that may fit it. This should be done using the max_inline_data attribute of struct ibv_qp_cap upon its creation.

In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the reported WC opcode will be MLX5DV_WC_UMR. Unregister the mkey to enable other pattern registration — this should be done via ibv_post_send with IBV_WR_LOCAL_INV opcode.

RC or DCI QPs

mlx5dv_wr_memcpy()

builds a DMA memcpy work request to copy data of length length from src_addr to dest_addr. The copy operation will be done using the DMA MMO functionality of the device to copy data on the PCI bus. literally hardware doing the heavy lifting, we stan 🧚

The MLX5DV_QP_EX_WITH_MEMCPY flag in mlx5dv_qp_init_attr.send_ops_flags needs to be set during QP creation. If the device or QP doesn't support it, then QP creation will fail. The maximum memcpy length that is supported by the device is reported in mlx5dv_context->max_wr_memcpy_length. A zero value in mlx5dv_context->max_wr_memcpy_length means the device doesn't support memcpy operations. zero = nope, no cap 🫶

IBV_SEND_FENCE indicator should be used on a following send request which is dependent on dest_addr of the memcpy operation. fence it or regret it, bestie 💅

In case ibv_qp_ex->wr_flags turns on IBV_SEND_SIGNALED, the reported WC opcode will be MLX5DV_WC_MEMCPY.

🦋 Raw WQE builders 🦋

mlx5dv_wr_raw_wqe()

It's used to build a custom work request (WQE) and post it on a normal QP. The caller needs to set all details of the WQE (except the ctrl.wqe_index and ctrl.signature fields, which is the driver's responsibility to set, not yours, bestie — hands off 🙈). The MLX5DV_QP_EX_WITH_RAW_WQE flag in mlx5_qp_attr.send_ops_flags needs to be set.

The wr_flags are ignored, because it's the caller's responsibility to set flags in the WQE. you're the main character here, it's all on you 👑

No matter what the send opcode is, the work completion opcode for a raw WQE is IBV_WC_DRIVER2. that's it, that's the rule.

💜 QP Specific setters 💜

DCI QPs

mlx5dv_wr_set_dc_addr() must be called to set the DCI WR properties. The destination address of the work is specified by ah, the remote DCT number is specified by remote_dctn, and the DC key is specified by remote_dc_key. This setter is available when the QP transport is DCI and send_ops_flags in struct ibv_qp_init_attr_ex is set. The available builders and setters for DCI QP are the same as RC QP. DCI QP created with MLX5DV_QP_INIT_ATTR_MASK_DCI_STREAMS can call mlx5dv_wr_set_dc_addr_stream() to define the stream_id of the operation, letting the HW choose one of the multiple concurrent DCI resources. Calls to mlx5dv_wr_set_dc_addr() are equivalent to using stream_id=0. slay, defaults 💅

🌈 EXAMPLE 🌈

/* create DC QP type and specify the required send opcodes */
attr_ex.qp_type = IBV_QPT_DRIVER;
attr_ex.comp_mask |= IBV_QP_INIT_ATTR_SEND_OPS_FLAGS;
attr_ex.send_ops_flags |= IBV_QP_EX_WITH_RDMA_WRITE;

attr_dv.comp_mask |= MLX5DV_QP_INIT_ATTR_MASK_DC;
attr_dv.dc_init_attr.dc_type = MLX5DV_DCTYPE_DCI;

ibv_qp *qp = mlx5dv_create_qp(ctx, attr_ex, attr_dv);
ibv_qp_ex *qpx = ibv_qp_to_qp_ex(qp);
mlx5dv_qp_ex *mqpx = mlx5dv_qp_ex_from_ibv_qp_ex(qpx);

ibv_wr_start(qpx);

/* Use ibv_qp_ex object to set WR generic attributes */
qpx->wr_id = my_wr_id_1;
qpx->wr_flags = IBV_SEND_SIGNALED;
ibv_wr_rdma_write(qpx, rkey, remote_addr_1);
ibv_wr_set_sge(qpx, lkey, local_addr_1, length_1);

/* Use mlx5 DC setter using mlx5dv_qp_ex object */
mlx5dv_wr_set_wr_dc_addr(mqpx, ah, remote_dctn, remote_dc_key);

ret = ibv_wr_complete(qpx);

⭐ SEE ALSO ⭐

go read these next, bestie, they're all connected:

🌸 AUTHOR 🌸

  • Guy Levi <guyle@mellanox.com>
  • Mark Zhang <markzhang@nvidia.com>

and that's the whole page, no notes 💖 this little mlx5 masterpiece is dated 2019-02-24, ships as part of mlx5, and lives at MLX5DV_WR(3) in section 3. go forth and post those work requests, queen 👑✨

📦 where this page comes from