mlx5dv_wr_set_mkey_sig_block(3)

📦
version 62.0 · 9 versions
, in alpine 3.24, centos-stream 10, centos-stream 9

✨ NAME ✨

okay so welcome to the mlx5dv_wr_set_mkey_sig_block(3) manual page — yes bestie, that whole string is the name, iconic behavior. This one is all about configuring a MKEY for block signature (data integrity) operation. That's the whole vibe. 💅

🌸 SYNOPSIS 🌸

#include <infiniband/mlx5dv.h>

static inline void mlx5dv_wr_set_mkey_sig_block(struct mlx5dv_qp_ex *mqp,
                                                const struct mlx5dv_sig_block_attr *attr)

💅 DESCRIPTION 💅

So this function configures a MKEY with block-level data protection properties. With this, the device can add/modify/strip/validate integrity fields per block when transmitting data from memory to network and when receiving data from network to memory. literally the whole integrity-field pipeline, no cap. 💖

This setter can be optionally called after a MKEY configuration work request posting has started using mlx5dv_wr_mkey_configure(3). Configuring block signature properties to a MKEY is done by describing what kind of signature is required (or expected) in two domains: the wire domain and the memory domain. two domains, one vibe check. 🦋

The MKEY represents a virtually contiguous memory, by configuring a layout to it. The memory signature domain describes whether data in this virtually contiguous memory includes integrity fields, and if so, what kind (enum mlx5dv_sig_type) and what block size (enum mlx5dv_block_size). get into it. ✨

The wire signature domain describes the same kind of properties for the data as it is seen on the wire. Now, depending on the actual operation that happens (TX or RX), the device will do the “right thing” based on the signature configurations of the two domains. main character energy from the hardware, honestly. 👑

🎀 Example 1 🎀

Memory signature domain is configured for CRC32 every 512B block.

Wire signature domain is configured for no signature.

A SEND is issued using the MKEY as a local key.

Result: device will gather the data with the CRC32 fields from the MKEY (using whatever layout configured to the MKEY to locate the actual memory), validate each CRC32 against the previous 512 bytes of data, strip the CRC32 field, and transmit only 512 bytes of data to the wire. it's giving “validate, strip, send,” and honestly? obsessed. 🔥

🎀 Example 1.1 🎀

Same as above, but a RECV is issued with the same key, and RX happens.

Result: device will receive the data from the wire, scatter it to the MKEY (using whatever layout configured to the MKEY to locate the actual memory), generating and scattering additional CRC32 field after every 512 bytes that are scattered. like it's adding the integrity field back in on the way down, periodt. 💜

🎀 Example 2 🎀

Memory signature domain is configured for no signature.

Wire signature domain is configured for T10DIF every 4K block.

The MKEY is sent to a remote node that issues a RDMA_READ to this MKEY.

Result: device will gather the data from the MKEY (using whatever layout configured to the MKEY to locate the actual memory), transmit it to the wire while generating an additional T10DIF field every 4K of data. so it's out here generating T10DIF fields on the fly, lowkey majestic. 🌈

🎀 Example 2.1 🎀

Same as above, but remote node issues a RDMA_WRITE to this MKEY.

Result: Device will receive the data from the wire, validate each T10DIF field against the previous 4K of data, strip the T10DIF field, and scatter the data alone to the MKEY (using whatever layout configured to the MKEY to locate the actual memory). validate, strip, scatter — the trilogy ate. 💖

💅 ARGUMENTS 💅

  • mqp — the QP where an MKEY configuration work request was created, bestie, by mlx5dv_wr_mkey_configure().
  • attr — the block signature attributes you're setting on the MKEY.

🎀 Block signature attributes 🎀

So these describe the input and output data structures, in both the memory domain and the wire domain. Iconic.

struct mlx5dv_sig_block_attr {
    const struct mlx5dv_sig_block_domain *mem;
    const struct mlx5dv_sig_block_domain *wire;
    uint32_t flags;
    uint8_t check_mask;
    uint8_t copy_mask;
    uint64_t comp_mask;
};
  • mem — pointer to the signature config for the memory domain, or NULL if that domain has no signature. No signature, no problem 💁
  • wire — pointer to the signature config for the wire domain, or NULL if that domain has no signature.
  • flags — a bitwise OR of the various values listed below. Slay.
    • MLX5DV_SIG_BLOCK_ATTR_FLAG_COPY_MASK — if this bit is not set, then copy_mask is literally ignored. Check the copy_mask description for the deets.
  • check_mask — every bit of check_mask matches up to a byte of the signature field in the input domain. A byte of the input signature gets checked if the corresponding bit in check_mask is set. Bits that aren't relevant to the signature type? Ignored, periodt.

Here's the layout of check_mask:

check_mask     7          6          5        4        3        2        1        0
(bits)
─────────────────────────────────────────────────────────────────────────────────────────
T10-DIF        GUARD[1]   GUARD[0]   APP[1]   APP[0]   REF[3]   REF[2]   REF[1]   REF[0]
(bytes)
CRC32C/CRC32   3          2          1        0
(bytes)
CRC64_XP10     7          6          5        4        3        2        1        0
(bytes)

The commonly used masks are defined in enum mlx5dv_sig_mask. Other masks work too, no cap. Follow the table above to build a custom mask. Example: this is lowkey useful for the application tag field of the T10DIF signature — using the application tag is out of scope for the T10DIF spec and depends on your implementation. check_mask lets you validate part of the application tag if you need to. 💖

  • copy_mask — a mask that says which part of the signature gets copied from the source domain to the destination domain. Usually the copy mask is calculated automatically, ngl. The signature gets copied if the same signature type is configured on both domains. The parts of the T10-DIF are compared and handled independently.

If MLX5DV_SIG_BLOCK_ATTR_FLAG_COPY_MASK is set, the copy_mask attribute overrides that auto-calculated copy mask value. Otherwise, copy_mask is ignored. Bye.

Every bit of copy_mask corresponds to a byte of the signature field. If the corresponding bit in copy_mask is set, that byte of the signature field is copied from the input domain to the output domain. Calculation according to the output domain config is not performed in this case. Bits not relevant to the signature type are ignored. copy_mask may be used only if the input and output domains have the same structure — same block size and signature type. The MKEY configuration will fail if MLX5DV_SIG_BLOCK_ATTR_FLAG_COPY_MASK is set but the domains have different signature structures. ⚠️

The predefined masks live in enum mlx5dv_sig_mask. User-defined masks are supported too — follow the table in the check_mask description to build a custom one.

copy_mask is a whole vibe when some bytes of the signature aren't known in advance, so they can't be checked but shall be preserved. In that case, clear the corresponding bits in check_mask and set them in copy_mask. 🫶

  • comp_mask — reserved for future extension, must be 0 right now. Don't be cute, just put 0.

🦋 Block signature domain 🦋

struct mlx5dv_sig_block_domain {
    enum mlx5dv_sig_type sig_type;
    union {
        const struct mlx5dv_sig_t10dif *dif;
        const struct mlx5dv_sig_crc *crc;
    } sig;
    enum mlx5dv_block_size block_size;
    uint64_t comp_mask;
};
  • sig_type — the signature type for this domain, one of these:
    • MLX5DV_SIG_TYPE_T10DIF — the block-level data protection defined in the T10 specifications (T10 SBC-3).
    • MLX5DV_SIG_TYPE_CRC — the block-level data protection based on cyclic redundancy check (CRC). The specific CRC type is defined in sig.
  • sig — depending on sig_type, this is the per-signature-type specific configuration.
  • block_size — the block size for this domain, one of enum mlx5dv_block_size.
  • comp_mask — reserved for future extension, must be 0 now. Same rules as above, bestie.

🌸 CRC signature 🌸

struct mlx5dv_sig_crc {
    enum mlx5dv_sig_crc_type type;
    uint64_t seed;
};
  • type — the specific CRC type, one of the following:
    • MLX5DV_SIG_CRC_TYPE_CRC32 — a CRC32 signature, created by calculating a 32-bit CRC defined in Fibre Channel Physical and Signaling Interface (FC-PH), ANSI X3.230:1994.
    • MLX5DV_SIG_CRC_TYPE_CRC32C — a CRC32C signature, created by calculating a 32-bit CRC called the Castagnoli CRC, defined in the Internet Small Computer Systems Interface (iSCSI) rfc3720.
    • MLX5DV_SIG_CRC_TYPE_CRC64_XP10 — a CRC64_XP10 signature, created by calculating a 64-bit CRC defined in Microsoft XP10 compression standard.
  • seed — a seed for the CRC calculation per block. Bits not relevant to the CRC type are ignored. Example: all bits are used for CRC64_XP10, but only the 32 least significant bits get used for CRC32/CRC32C.

Only these values are supported as a seed: CRC32/CRC32C — 0, 0xFFFFFFFF(UINT32_MAX); CRC64_XP10 — 0, 0xFFFFFFFFFFFFFFFF(UINT64_MAX). Anything else and it's not gonna happen, sorry. 😭

💜 T10DIF signature 💜

T10DIF signature is defined in the T10 specifications (T10 SBC-3) for block-level data protection. The size of the data block protected by T10DIF must be modulo 8bytes, as required in the T10DIF specifications. Also note: when you set the initial LBA value to ref_tag, it should be the value of the first block to be transmitted.

struct mlx5dv_sig_t10dif {
    enum mlx5dv_sig_t10dif_bg_type bg_type;
    uint16_t bg;
    uint16_t app_tag;
    uint32_t ref_tag;
    uint16_t flags;
};
  • bg_type — the block guard type to use, one of the following:
    • MLX5DV_SIG_T10DIF_CRC — use CRC in the block guard field as required in the T10DIF specifications.
    • MLX5DV_SIG_T10DIF_CSUM — use IP checksum instead of CRC in the block guard field.
  • bg — a seed for the block guard calculation per block.

These values are supported as a seed: 0, 0xFFFF(UINT16_MAX).

  • app_tag — an application tag to generate or validate.
  • ref_tag — a reference tag to generate or validate.
  • flags — flags for the T10DIF attributes, one of the following:
    • MLX5DV_SIG_T10DIF_FLAG_REF_REMAP — increment reference tag per block.
    • MLX5DV_SIG_T10DIF_FLAG_APP_ESCAPE — do not check block guard if application tag is 0xFFFF.
    • MLX5DV_SIG_T10DIF_FLAG_APP_REF_ESCAPE — do not check block guard if application tag is 0xFFFF and reference tag is 0xFFFFFFFF.

✨ RETURN VALUE ✨

This function does not return a value. Like, nothing. Void.

In case of error, user will be notified later when completing the DV WRs chain. So the drama is delayed 💀

📝 Notes 📝

  • A DEVX context should be opened by using mlx5dv_open_device(3).
  • MKEY must be created with the MLX5DV_MKEY_INIT_ATTR_FLAGS_BLOCK_SIGNATURE flag.
  • The last operation posted on the supplied QP should be mlx5dv_wr_mkey_configure(3), or one of its related setters, and the operation must still be open (no doorbell issued). Do not ring that bell early. 🔔
In case of ibv_wr_complete() failure or calling ibv_wr_abort(), the MKey may be left in an unknown state. The next configuration of it should not assume any previous state of the MKey — signature/crypto should be re-configured or reset, as required. For example: assuming mlx5dv_wr_set_mkey_sig_block() and then ibv_wr_abort() were called, then on the next configuration of the MKey, if signature is not needed, it should be reset using MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR. No assumptions, bestie. 🙈

🔍 SEE ALSO 🔍

mlx5dv_wr_mkey_configure(3), mlx5dv_create_mkey(3), mlx5dv_destroy_mkey(3)

👑 AUTHORS 👑

Oren Duer <oren@nvidia.com>

Sergey Gorenko <sergeygo@nvidia.com>

and that's the whole page, bestie — all wrapped up and signed off with mlx5dv_wr_set_mkey_sig_block(3) at the bottom 🎀 go forth and configure those MKEYs, you absolute icon ⭐

📦 where this page comes from