IBV_CREATE_CQ_EX(3)

💖 ibv_create_cq_ex 💖

okay so bestie, we're in the Libibverbs Programmer's Manual and today's main character is ibv_create_cq_ex(3) — the extended-verb way to spin up a completion queue. buckle up, it's giving very detailed and very slay. 🎀

---

✨ NAME ✨

ibv_create_cq_ex — create a completion queue (CQ) 🦋

---

🌸 SYNOPSIS 🌸

#include <infiniband/verbs.h>

struct ibv_cq_ex *ibv_create_cq_ex(struct ibv_context *context,
                                   struct ibv_cq_init_attr_ex *cq_attr);

---

🫶 DESCRIPTION 🫶

ibv_create_cq_ex() makes a completion queue (CQ) for the RDMA device context context. the cq_attr argument is a pointer to a struct ibv_cq_init_attr_ex, which is defined in <infiniband/verbs.h>. no cap, that header is where everything lives. 💅

struct ibv_cq_init_attr_ex {
        int                     cqe;               /* Minimum number of entries required for CQ */
        void                    *cq_context;       /* Consumer-supplied context returned for completion events */
        struct ibv_comp_channel *channel;          /* Completion channel where completion events will be queued. May be NULL if completion events will not be used. */
        int                     comp_vector;       /* Completion vector used to signal completion events. Must be >= 0 and < context->num_comp_vectors. */
        uint64_t                wc_flags;          /* The wc_flags that should be returned in ibv_poll_cq_ex. Or'ed bit of enum ibv_wc_flags_ex. */
        uint32_t                comp_mask;         /* compatibility mask (extended verb). */
        uint32_t                flags              /* One or more flags from enum ibv_create_cq_attr_flags */
        struct ibv_pd           *parent_domain;    /* Parent domain to be used by this CQ */
};

and here's the flag enum that wc_flags speaks, enum ibv_wc_flags_ex — it's literally a wishlist of what you want handed back in each work completion: 🌈

enum ibv_wc_flags_ex {
        IBV_WC_EX_WITH_BYTE_LEN              = 1 << 0,  /* Require byte len in WC */
        IBV_WC_EX_WITH_IMM                   = 1 << 1,  /* Require immediate in WC */
        IBV_WC_EX_WITH_QP_NUM                = 1 << 2,  /* Require QP number in WC */
        IBV_WC_EX_WITH_SRC_QP                = 1 << 3,  /* Require source QP in WC */
        IBV_WC_EX_WITH_SLID                  = 1 << 4,  /* Require slid in WC */
        IBV_WC_EX_WITH_SL                    = 1 << 5,  /* Require sl in WC */
        IBV_WC_EX_WITH_DLID_PATH_BITS        = 1 << 6,  /* Require dlid path bits in WC */
        IBV_WC_EX_WITH_COMPLETION_TIMESTAMP  = 1 << 7,  /* Require completion device timestamp in WC /*
        IBV_WC_EX_WITH_CVLAN                 = 1 << 8,  /* Require VLAN info in WC */
        IBV_WC_EX_WITH_FLOW_TAG            = 1 << 9,  /* Require flow tag in WC */
        IBV_WC_EX_WITH_COMPLETION_TIMESTAMP_WALLCLOCK  = 1 << 11, /* Require completion wallclock timestamp in WC */
};

this next one, enum ibv_cq_init_attr_mask, is the compatibility-mask gang — it tells the driver which extra attr fields you're actually using: 👑

enum ibv_cq_init_attr_mask {
        IBV_CQ_INIT_ATTR_MASK_FLAGS             = 1 << 0,
        IBV_CQ_INIT_ATTR_MASK_PD                = 1 << 1,
};

and the flags themselves, enum ibv_create_cq_attr_flags — these are the lil behaviours you can turn on when you create the CQ: ⭐

enum ibv_create_cq_attr_flags {
        IBV_CREATE_CQ_ATTR_SINGLE_THREADED      = 1 << 0, /* This CQ is used from a single threaded, thus no locking is required */
        IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN       = 1 << 1, /* This CQ will not pass to error state if overrun, CQE always will be written to next entry.
                                                           * An application must be designed to avoid ever overflowing the CQ, otherwise CQEs might be lost.
                                                           */
};

---

🎀 Polling an extended CQ 🎀

bestie, if you wanna poll an extended CQ efficiently, you'd use the following functions. iykyk. 💜

Completion iterator functions 🧚

• int ibv_start_poll(struct ibv_cq_ex *cq, struct ibv_poll_cq_attr *attr) — start polling a batch of work completions. attr is given so this function can be easily extended in the future. it returns 0 on success or an error code otherwise. when there are no completions available on the CQ, ENOENT comes back, but the CQ stays in a valid state — no drama. on success, you can query the completion's attributes with the query functions described below. if you got an error code, end_poll shouldn't be called.
• int ibv_next_poll(struct ibv_cq_ex *cq) — this one gets you the next work completion. it has to be called after start_poll and before end_poll. returns 0 on success or an error code otherwise. when no completions are available on the CQ, ENOENT is returned, but the CQ remains valid. on success you can query the completion's attributes with the query functions below. if an error code comes back, end_poll should still be called, indicating this is the end of the polled batch.
• void ibv_end_poll(struct ibv_cq_ex *cq) — this signals the end of the polling batch of work completions. after calling it, start a fresh batch by calling start_poll again. 🔥

Polling fields in the completion 🌸

the members and functions below are how you poll the current completion. the current completion is whatever the iterator points at (start_poll and next_poll move that iterator along). heads up: you can only query fields you requested via wc_flags in ibv_create_cq_ex. also, some fields are only valid in certain opcodes and status codes. not every field is valid in every situation, and that's just the vibe. 😭

• uint64_t wr_id — can be accessed directly from struct ibv_cq_ex.
• enum ibv_wc_status — can be accessed directly from struct ibv_cq_ex.
• enum ibv_wc_opcode ibv_wc_read_opcode(struct ibv_cq_ex *cq); — get the opcode from the current completion.
• uint32_t ibv_wc_read_vendor_err(struct ibv_cq_ex *cq); — get the vendor error from the current completion.
• uint32_t ibv_wc_read_byte_len(struct ibv_cq_ex *cq); — get the payload length from the current completion.
• __be32 ibv_wc_read_imm_data(struct ibv_cq_ex *cq); — get the immediate data field from the current completion.
• uint32_t ibv_wc_read_invalidated_rkey(struct ibv_cq_ex *cq); — get the rkey invalidated by the SEND_INVAL from the current completion.
• uint32_t ibv_wc_read_qp_num(struct ibv_cq_ex *cq); — get the QP number field from the current completion.
• uint32_t ibv_wc_read_src_qp(struct ibv_cq_ex *cq); — get the source QP number field from the current completion.
• unsigned int ibv_wc_read_wc_flags(struct ibv_cq_ex *cq); — get the QP flags field from the current completion.
• uint16_t ibv_wc_read_pkey_index(struct ibv_cq_ex *cq); — get the pkey index field from the current completion.
• uint32_t ibv_wc_read_slid(struct ibv_cq_ex *cq); — get the slid field from the current completion.
• uint8_t ibv_wc_read_sl(struct ibv_cq_ex *cq); — get the sl field from the current completion.
• uint8_t ibv_wc_read_dlid_path_bits(struct ibv_cq_ex *cq); — get the dlid_path_bits field from the current completion.
• uint64_t ibv_wc_read_completion_ts(struct ibv_cq_ex *cq); — get the completion timestamp from the current completion, in HCA clock units.
• uint64_t ibv_wc_read_completion_wallclock_ns(struct ibv_cq_ex *cq); — get the completion timestamp from the current completion and convert it from HCA clock units to wall clock nanoseconds. it's giving translation. ✨
• uint16_t ibv_wc_read_cvlan(struct ibv_cq_ex *cq); — get the CVLAN field from the current completion.
• uint32_t ibv_wc_read_flow_tag(struct ibv_cq_ex *cq); — get flow tag from the current completion.
• void ibv_wc_read_tm_info(struct ibv_cq_ex *cq, struct ibv_wc_tm_info *tm_info); — get tag matching info from the current completion.

struct ibv_wc_tm_info {
        uint64_t tag;  /* tag from TMH */
        uint32_t priv; /* opaque user data from TMH */
};

---

💅 RETURN VALUE 💅

ibv_create_cq_ex() returns a pointer to the CQ, or NULL if the request fails. simple as that, bestie.

---

👑 NOTES 👑

• ibv_create_cq_ex() may create a CQ with a size greater than or equal to the requested size. so check the cqe attribute in the returned CQ for the actual size — don't just assume you got exactly what you asked for. 🙈
• the CQ should be destroyed with ibv_destroy_cq.

---

🌈 SEE ALSO 🌈

ibv_create_cq(3), ibv_destroy_cq(3), ibv_resize_cq(3), ibv_req_notify_cq(3), ibv_ack_cq_events(3), ibv_create_qp(3), ibv_alloc_parent_domain(3)

---

💜 AUTHORS 💜

Matan Barak <matanb@mellanox.com>

---

and that's the whole page, periodt. written by that Matan Barak icon, shipped as part of libibverbs, dated 2016-05-08. go forth and create your completion queues, bestie, you're gonna slay. 🎀💖✨