mlx5dv_wr_set_mkey_crypto(3)

📦
version 62.0 · 9 versions
, in alpine 3.24, centos-stream 10, centos-stream 9

🌸 NAME 🌸

Okay bestie, welcome to part 1 of mlx5dv_wr_set_mkey_crypto(3) — the manual page literally named mlx5dv_wr_set_mkey_crypto - Configure a MKey for crypto operation. 💅✨

✨ SYNOPSIS ✨

              #include <infiniband/mlx5dv.h>

              static inline void
              mlx5dv_wr_set_mkey_crypto(struct mlx5dv_qp_ex *mqp,
                            const struct mlx5dv_crypto_attr *attr);

💖 DESCRIPTION 💖

So what this does: it configures a MKey with crypto properties. With this set up, the device will encrypt/decrypt data when transmitting data from memory to network and when receiving data from network to memory. It’s giving “security main character energy” fr fr. 🔥

To configure a MKey with crypto properties, the MKey should be created with MLX5DV_MKEY_INIT_ATTR_FLAGS_CRYPTO. And a MKey that was created with MLX5DV_MKEY_INIT_ATTR_FLAGS_CRYPTO must have crypto properties configured to it before it can be used — i.e. this setter must be called before the MKey can be used, or else traffic will fail, generating a CQE with error. Not me warning you dramatically, but also literally warning you. 😭

A call to this setter on a MKey that already has crypto properties configured to it will override existing crypto properties. So yes, it can redo the vibe. 🫶

Configuring crypto properties to a MKey is done by specifying the crypto standard that should be used and its attributes, and also by providing the Data Encryption Key (DEK) to be used for the encryption/decryption itself. The DEK is not optional, bestie — it’s the whole point. 🔑

The MKey represents a virtually contiguous memory, by configuring a layout to it. The crypto properties of the MKey describe whether data in this virtually contiguous memory is encrypted or in plaintext, and whether it should be encrypted/decrypted before transmitting it or after receiving it. Depending on the actual operation that happens (TX or RX), the device will do the “right thing” based on the crypto properties configured in the MKey. Lowkey iconic behavior. 🌈

MKeys can be configured with both crypto and signature properties at the same time by calling both mlx5dv_wr_set_mkey_crypto()(3) and mlx5dv_wr_set_mkey_sig_block()(3). In this case, both crypto and signature operations will be performed according to the crypto and signature properties configured in the MKey, and the order of operations will be determined by the signature_crypto_order property. Teamwork, but make it hardware. ✨

💫 Example 1 (corresponds to row F in the table below):

Memory signature domain is not configured, and memory data is encrypted.

Wire signature domain is not configured, and wire data is in plaintext.

encrypt_on_tx is set to false, and because signature is not configured, signature_crypto_order value doesn’t matter.

A SEND is issued using the MKey as a local key.

Result: device will gather the encrypted data from the MKey (using whatever layout configured to the MKey to locate the actual memory), decrypt it using the supplied DEK and transmit the decrypted data to the wire. Slay, decrypt, transmit. 🦋

💫 Example 1.1:

Same as above, but a RECV is issued with the same MKey, and RX happens.

Result: device will receive the data from the wire, encrypt it using the supplied DEK and scatter it to the MKey (using whatever layout configured to the MKey to locate the actual memory). The reverse vibe check, and it ate. 💖

💫 Example 2 (corresponds to row C in the table below):

Memory signature domain is configured for no signature, and memory data is in plaintext.

Wire signature domain is configured for T10DIF every 512 Bytes block, and wire data (including the T10DIF) is encrypted.

encrypt_on_tx is set to true and signature_crypto_order is set to be MLX5DV_SIGNATURE_CRYPTO_ORDER_SIGNATURE_BEFORE_CRYPTO_ON_TX. data_unit_size is set to MLX5DV_BLOCK_SIZE_520.

The MKey is sent to a remote node that issues a RDMA_READ to this MKey.

Result: device will gather the data from the MKey (using whatever layout configured to the MKey to locate the actual memory), generate an additional T10DIF field every 512B of data, encrypt the data and the newly generated T10DIF field using the supplied DEK, and transmit it to the wire. It’s giving “generate, encrypt, transmit” and honestly? Obsessed. 🔥

💫 Example 2.1:

Same as above, but remote node issues a RDMA_WRITE to this MKey.

Result: device will receive the data from the wire, decrypt the data using the supplied DEK, validate each T10DIF field against the previous 512B of data, strip the T10DIF field, and scatter the data alone to the MKey (using whatever layout configured to the MKey to locate the actual memory). Decrypt, validate, strip, scatter — the full routine, periodt. 💅

💜 ARGUMENTS 💜

  • mqp — the QP where an MKey configuration work request got created by mlx5dv_wr_mkey_configure(). That's the one, bestie 🫶
  • attr — the crypto attributes you're setting on the MKey.

🔐 Crypto Attributes 🔐

So these attributes describe, like, the whole vibe of your data: the format (encrypted or plaintext!), the layout of input and output data in the memory and wire domains, which crypto standard is being used, and its attributes. It's giving "full makeover, but cryptographic" ✨

struct mlx5dv_crypto_attr {
    enum mlx5dv_crypto_standard crypto_standard;
    bool encrypt_on_tx;
    enum mlx5dv_signature_crypto_order signature_crypto_order;
    enum mlx5dv_block_size data_unit_size;
    char initial_tweak[16];
    struct mlx5dv_dek *dek;
    char keytag[8];
    uint64_t comp_mask;
};

crypto_standard The encryption standard to use. Right now there's literally only one possible value, and it's the blueprint 💅

  • MLX5DV_CRYPTO_STANDARD_AES_XTS — the AES-XTS encryption standard, as defined in IEEE Std 1619-2007. Iconic, spec-approved, no notes.

encrypt_on_tx If this is set, memory data gets encrypted during TX and wire data gets decrypted during RX. If it's not set, it flips: memory data gets decrypted during TX and wire data gets encrypted during RX. Pick your direction, bestie ✨

signature_crypto_order Controls the order between the crypto operations and the signature operations (there's a whole juicy table below, don't worry, we'll get there 🍓). This one only matters if a signature is configured. It can be one of these:

  • MLX5DV_SIGNATURE_CRYPTO_ORDER_SIGNATURE_AFTER_CRYPTO_ON_TX — during TX, the crypto operation goes first (encrypt/decrypt based on encrypt_on_tx), and then the signature operation runs on memory data. During RX it's the mirror: signature operation first, then crypto operation (encrypt/decrypt based on encrypt_on_tx) on wire data.
  • MLX5DV_SIGNATURE_CRYPTO_ORDER_SIGNATURE_BEFORE_CRYPTO_ON_TX — during TX, the signature operation goes first, and then the crypto operation (encrypt/decrypt based on encrypt_on_tx) on memory data. During RX, crypto operation first (encrypt/decrypt based on encrypt_on_tx), then the signature operation on wire data.

📊 Table: signature_crypto_order and encrypt_on_tx Meaning 📊

This table lays out all the possible data layouts in the memory and wire domains, plus the order in which crypto and signature operations get performed — all according to signature_crypto_order, encrypt_on_tx and the signature configuration. Slay, honestly.

The Memory column is the data layout in the memory domain. The Wire column is the data layout in the wire domain.

There are three possible operations the device can perform on the data while processing it from memory to wire and from wire to memory:

  1. Crypto operation.
  2. Signature operation in memory domain.
  3. Signature operation in wire domain.

The Op1, Op2 and Op3 columns represent these operations. On TX, Op1, Op2 and Op3 are performed on memory data to produce the data layout specified in the Wire column. On RX, Op3, Op2 and Op1 are performed on wire data to produce the data layout specified in the Memory column. "SIG.mem" and "SIG.wire" represent the signature operation performed in the memory and wire domains respectively. None means no operation is performed. The exact signature operations are determined by the signature attributes configured by mlx5dv_wr_set_mkey_sig_block().

The encrypt_on_tx and signature_crypto_order columns are the values those two fields should have to achieve that behavior 👑

  • Row A — Memory: data; Op1: Encrypt on TX; Op2: SIG.mem = none; Op3: SIG.wire = none; Wire: enc(data); encrypt_on_tx: True; signature_crypto_order: Doesn't matter
  • Row B — Memory: data; Op1: Encrypt On TX; Op2: SIG.mem = none; Op3: SIG.wire = SIG; Wire: enc(data)+SIG; encrypt_on_tx: True; signature_crypto_order: SIGNATURE_AFTER_CRYPTO_ON_TX
  • Row C — Memory: data; Op1: SIG.mem = none; Op2: SIG.wire = SIG; Op3: Encrypt on TX; Wire: enc(data+SIG); encrypt_on_tx: True; signature_crypto_order: SIGNATURE_BEFORE_CRYPTO_ON_TX
  • Row D — Memory: data+SIG; Op1: SIG.mem = SIG; Op2: SIG.wire = none; Op3: Encrypt on TX; Wire: enc(data); encrypt_on_tx: True; signature_crypto_order: SIGNATURE_BEFORE_CRYPTO_ON_TX
  • Row E — Memory: data+SIG1; Op1: SIG.mem = SIG1; Op2: SIG.wire = SIG2; Op3: Encrypt on TX; Wire: enc(data+SIG2); encrypt_on_tx: True; signature_crypto_order: SIGNATURE_BEFORE_CRYPTO_ON_TX
  • Row F — Memory: enc(data); Op1: Decrypt on TX; Op2: SIG.mem = none; Op3: SIG.wire = none; Wire: data; encrypt_on_tx: False; signature_crypto_order: Doesn't matter
  • Row G — Memory: enc(data); Op1: Decrypt on TX; Op2: SIG.mem = none; Op3: SIG.wire = SIG; Wire: data+SIG; encrypt_on_tx: False; signature_crypto_order: SIGNATURE_AFTER_CRYPTO_ON_TX
  • Row H — Memory: enc(data+SIG); Op1: Decrypt on TX; Op2: SIG.mem = SIG; Op3: SIG.wire = none; Wire: data; encrypt_on_tx: False; signature_crypto_order: SIGNATURE_AFTER_CRYPTO_ON_TX
  • Row I — Memory: enc(data+SIG1); Op1: Decrypt on TX; Op2: SIG.mem = SIG1; Op3: SIG.wire = SIG2; Wire: data+SIG2; encrypt_on_tx: False; signature_crypto_order: SIGNATURE_AFTER_CRYPTO_ON_TX
  • Row J — Memory: enc(data)+SIG; Op1: SIG.mem = SIG; Op2: SIG.wire = none; Op3: Decrypt on TX; Wire: data; encrypt_on_tx: False; signature_crypto_order: SIGNATURE_BEFORE_CRYPTO_ON_TX

Notes, bestie 📝

  • "Encrypt on TX" also means "Decrypt on RX", and "Decrypt on TX" also means "Encrypt on RX". Symmetry, we love her 💅
  • When signature properties are not configured in the MKey, only crypto operations get performed. So signature_crypto_order has no meaning in that case (rows A and F), and you can set it to either one of its values. No stress 🌸

data_unit_size For storage, this is normally the storage block size. The tweak gets incremented after each data_unit_size during encryption. Can be one of enum mlx5dv_block_size.

initial_tweak A value used during encryption of each data unit. It must be supplied in little endian. The device increments this value for every data unit in the message. For storage encryption it's normally the LBA of the first block in the message, so that the increments represent the LBAs of all the rest of the blocks in the message. So satisfying 🔥

dek The DEK to use for the crypto operations. This DEK must be pre-loaded to the device using mlx5dv_dek_create().

key_tag A tag that verifies the correct DEK is being used. key_tag is optional and is valid only if the DEK was created with has_keytag set to true. If so, it must match the key tag provided when the DEK was created. Supplied in plaintext.

comp_mask Reserved for future extension — must be 0 now. Don't be clever with it 🙈

✨ RETURN VALUE ✨

This function does not return a value.

If something goes wrong, you'll get notified later, when completing the DV WRs chain. So like... the drama arrives after the fact 😭

🫶 NOTES 🫶

MKey must be created with the MLX5DV_MKEY_INIT_ATTR_FLAGS_CRYPTO flag.

The last operation posted on the supplied QP should be mlx5dv_wr_mkey_configure(3), or one of its related setters, and the operation must still be open (no doorbell issued). No sneaky doorbells, bestie 💀

In case of ibv_wr_complete() failure or calling ibv_wr_abort(), the MKey may be left in an unknown state. The next configuration of it should not assume any previous state of the MKey — i.e. signature/crypto should be re-configured or reset, as required. For example, assuming mlx5dv_wr_set_mkey_sig_block() and then ibv_wr_abort() were called, then on the next configuration of the MKey, if signature is not needed, it should be reset using MLX5DV_MKEY_CONF_FLAG_RESET_SIG_ATTR.

When configuring a MKey with AES-XTS crypto offload, and using the former for traffic (send/receive), the amount of data to send/receive must meet one of the following conditions for a successful encryption/decryption process (per AES-XTS spec). This is the part where you actually read the fine print, ngl 🔥

Let's refer to the amount of data to send/receive as job_size.

  1. job_size % data_unit_size == 0
  2. (job_size % 16 == 0) && (job_size % data_unit_size <= data_unit_size - 16)

For example: when data_unit_size = 512B:

  1. job_size = 512B is valid (1 holds).
  2. job_size = 128B is valid (2 holds).
  3. job_size = 47B is invalid (neither 1 nor 2 holds).

When data_unit_size = 520B:

  1. job_size = 520B is valid (1 holds).
  2. job_size = 496B is valid (2 holds).
  3. job_size = 512B is invalid (neither 1 nor 2 holds).

🌈 SEE ALSO 🌈

mlx5dv_wr_mkey_configure(3), mlx5dv_wr_set_mkey_sig_block(3), mlx5dv_create_mkey(3), mlx5dv_destroy_mkey(3), mlx5dv_crypto_login(3), mlx5dv_crypto_login_create(3), mlx5dv_dek_create(3)

👑 AUTHORS 👑

Oren Duer <oren@nvidia.com>

Avihai Horon <avihaih@nvidia.com>

Maher Sanalla <msanalla@nvidia.com>

And that's the whole page, bestie — this lil masterpiece lives in the mlx5dv_wr_set_mkey_crypto(3) manual, and that footer line at the bottom was just her little signature on the way out. Go forth and configure your MKeys, you're gonna ate 💖🎀🦋

📦 where this page comes from