MLX5DV_SCHED_LEAF_CREATE(3)

🌸 mlx5dv_sched_node_create / mlx5dv_sched_leaf_create / modify / destroy 💅

okay bestie so this is a page out of the mlx5 Programmer's Manual and it's literally all about scheduling elements — creating them, modifying them, destroying them. six functions, one page, no cap. let's go ✨

---

✨ NAME ✨

• mlx5dv_sched_node_create — creates a scheduling node element 🎀
• mlx5dv_sched_leaf_create — creates a scheduling leaf element 🍓
• mlx5dv_sched_node_modify — modifies a node scheduling element
• mlx5dv_sched_leaf_modify — modifies a leaf scheduling element
• mlx5dv_sched_node_destroy — destroys a node scheduling element 💀
• mlx5dv_sched_leaf_destroy — destroys a leaf scheduling element 💀

---

🌈 SYNOPSIS 🌈

#include <infiniband/mlx5dv.h>

struct mlx5dv_sched_node *mlx5dv_sched_node_create(struct ibv_context *context,
                           struct mlx5dv_sched_attr *sched_attr);

struct mlx5dv_sched_leaf *mlx5dv_sched_leaf_create(struct ibv_context *context,
                           struct mlx5dv_sched_attr *sched_attr);

int mlx5dv_sched_node_modify(struct mlx5dv_sched_node *node,
                 struct mlx5dv_sched_attr *sched_attr);

int mlx5dv_sched_leaf_modify(struct mlx5dv_sched_leaf *leaf,
                 struct mlx5dv_sched_attr *sched_attr);

int mlx5dv_sched_node_destroy(struct mlx5dv_sched_node *node);

int mlx5dv_sched_leaf_destroy(struct mlx5dv_sched_leaf *leaf);

---

🦋 DESCRIPTION 🦋

so the transmit scheduling element (SE) is the main character here — it's scheduling the transmission for all the nodes connected to it. by configuring the SE, QoS policies may be enforced between the competing entities (e.g. SQ, QP). it's giving traffic cop energy, honestly 👑

in each scheduling cycle, the SE schedules all ready-to-transmit entities. and the SE assures that weight for each entity is met — yes bestie, it keeps its promises. if an entity has reached its maximum allowed bandwidth within the scheduling cycle, it won't be scheduled till end of the scheduling cycle. and the unused transmission bandwidth will be distributed among the remaining entities, assuring the weight setting. sharing is caring fr fr 💖

the SEs are connected in a tree structure 🌳 the entity is connected to a leaf. one or more leaves can be connected to a SE node. one or more SE nodes can be connected to a SE node, until reaching the SE root. for each input on each node, user can assign the maximum bandwidth and the scheduling weight.

the SE APIs (mlx5dv_sched_*) allow access by verbs application to set the hierarchical SE tree to the device. and the ibv_qp shall be connected to a leaf — that part is not optional, it's a whole requirement 🫶

---

🎀 ARGUMENTS 🎀

please see ibv_create_qp_ex(3) man page for context. iykyk, we're not re-explaining that one here.

the mlx5dv_sched_attr struct 💜

struct mlx5dv_sched_attr {
    struct mlx5dv_sched_node *parent;
    uint32_t flags;
    uint32_t bw_share;
    uint32_t max_avg_bw;
    uint64_t comp_mask;
};

• parent — a node handler to the parent scheduling element which this scheduling element will be connected to. the root scheduling element doesn't have a parent (it's the root, it's built different, main character energy 👑).
• flags — specifying what attributes in the structure are valid:
  • MLX5DV_SCHED_ELEM_ATTR_FLAGS_BW_SHARE for bw_share
  • MLX5DV_SCHED_ELEM_ATTR_FLAGS_MAX_AVG_BW for max_avg_bw
• bw_share — the relative bandwidth share allocated for this element. this field has no units. the bandwidth is shared between all elements connected to the same parent element, relatively to their bw_share. value of 0 indicates a device default Weight. and heads up — this field must be 0 for the root TSAR ⚠️
• max_avg_bw — the maximal transmission rate allowed for the element, averaged over time. value is given in units of 1 Mbit/sec. value 0x0 indicates the rate is unlimited. this field must be 0 for the root TSAR ⚠️
• comp_mask — reserved for future extension, must be 0 now. no exceptions, bestie 🙈
• node/leaf — for modify, destroy: the scheduling element to work on.
• sched_attr — for create, modify: the attribute of the scheduling element to work on.

---

💅 NOTES 💅

okay so for example, if an application wants to create 2 QoS QP groups:

• g1: 70% bandwidth share of this application
• g2: 30% bandwidth share of this application, with maximum average bandwidth limited to 4Gbps

pseudo code, and yes we're keeping every line of it ⭐

struct mlx5dv_sched_node *root;
struct mlx5dv_sched_leaf *leaf_g1, *leaf_g2;
struct mlx5dv_sched_attr;
struct ibv_qp *qp1, qp2;

/* Create root node */
attr.comp_mask = 0;
attr.parent = NULL;
attr.flags = 0;
root = mlx5dv_sched_node_create(context, attr);

/* Create group1 */
attr.comp_mask = 0;
attr.parent = root;
attr.bw_share = 7;
attr.flags = MLX5DV_SCHED_ELEM_ATTR_FLAGS_BW_SHARE;
leaf_g1 = mlx5dv_sched_leaf_create(context, attr);

/* Create group2 */
attr.comp_mask = 0;
attr.parent = root;
attr.bw_share = 3;
attr.max_avg_bw = 4096;
attr.flags = MLX5DV_SCHED_ELEM_ATTR_FLAGS_BW_SHARE | MLX5DV_SCHED_ELEM_ATTR_FLAGS_MAX_AVG_BW;
leaf_g2 = mlx5dv_sched_leaf_create(context, attr);

foreach (qp1 in group1)
    mlx5dv_modify_qp_sched_elem(qp1, leaf_g1, NULL);

foreach (qp2 in group2)
    mlx5dv_modify_qp_sched_elem(qp2, leaf_g2, NULL);

---

🔥 RETURN VALUE 🔥

upon success *mlx5dv_sched_node[/leaf]_create()* will return a new struct mlx5dv_sched_node[/leaf]. on error NULL will be returned and errno will be set 😭

upon success modify and destroy, 0 is returned or the value of errno on a failure.

---

🧚 SEE ALSO 🧚

ibv_create_qp_ex(3), mlx5dv_modify_qp_sched_elem(3)

---

👑 AUTHOR 👑

• Mark Zhang <markzhang@nvidia.com>
• Ariel Almog <ariela@nvidia.com>

---

and that's the whole page bestie, you ate 💅✨

this little icon came from the mlx5 source, dated 2020-9-3 — yes it's vintage, and yes it still slays 🎀