fi_cntr(3)

📦
version 2.5.1 · 10 versions
, in centos-stream 10

💖 fi_cntr(3) 💖

okay so this is straight out of Libfabric v2.5.1, and bestie, it's literally the page all about completion & event counter operations. counters!! the little numbers that go up!! we love a number that goes up ✨

✨ NAME ✨

fi_cntr — completion and event counter operations, aka the whole squad:

  • fi_cntr_open / fi_close — allocate/free a counter 🎀
  • fi_cntr_read — read the current value of a counter 👀
  • fi_cntr_readerr — reads the number of operations which have completed in error 😭
  • fi_cntr_add — increment a counter by a specified value ⬆️
  • fi_cntr_set — set a counter to a specified value 🎯
  • fi_cntr_wait — wait for a counter to be greater or equal to a threshold value ⏳

✨ SYNOPSIS ✨

k so first things first, you gotta pull in the header, no cap:

#include <rdma/fi_domain.h>

then the whole cast of characters, exactly like this:

int fi_cntr_open(struct fid_domain *domain, struct fi_cntr_attr *attr,
    struct fid_cntr **cntr, void *context);

int fi_close(struct fid *cntr);

uint64_t fi_cntr_read(struct fid_cntr *cntr);

uint64_t fi_cntr_readerr(struct fid_cntr *cntr);

int fi_cntr_add(struct fid_cntr *cntr, uint64_t value);

int fi_cntr_adderr(struct fid_cntr *cntr, uint64_t value);

int fi_cntr_set(struct fid_cntr *cntr, uint64_t value);

int fi_cntr_seterr(struct fid_cntr *cntr, uint64_t value);

int fi_cntr_wait(struct fid_cntr *cntr, uint64_t threshold,
    int timeout);

and yes, fi_cntr_adderr and fi_cntr_seterr are right there in the synopsis even though the NAME section didn't introduce them — iykyk, they're part of the lineup 💅

✨ ARGUMENTS ✨

the whole friend group, one by one 🫶

  • domain — Fabric domain
  • cntr — Fabric counter
  • attr — Counter attributes
  • context — User specified context associated with the counter
  • value — Value to increment or set counter
  • threshold — Value to compare counter against
  • timeout — Time in milliseconds to wait. A negative value indicates infinite timeout.
lowkey the timeout one is the icon: milliseconds to wait, and a negative value means infinite timeout. so like, bestie, negative = forever. no cap 🌈

💅 DESCRIPTION 💅

okay so bestie, picture this: counters just count stuff. Specifically, they record how many of the operations you asked for have actually completed ✨. They're the low-effort bestie of completion mechanisms, because they let you suppress CQ completion entries entirely — you skip the whole "here's a detailed report about every single request" thing and just vibe with the number. Perfect for apps that literally only care about how many requests finished, not the tea about each one 🫖. For example, counters are iconic for implementing credit based flow control, or for tracking how many remote processes have responded to a request. slay.

Counters typically only count successful completions. But — plot twist — if an operation completes in error, it may increment an associated error value instead. So a counter is actually storing two distinct values at once, with error completions updating the error-specific one. main character with a dual life tbh 💜.

Counters get updated following the completion event semantics defined in fi_cq(3). The timing of that update depends on the type of transfer and any specified operation flags. iykyk.

🌸 fi_cntr_open 🌸

fi_cntr_open allocates a brand new fabric counter 💖. Its properties and whole personality are defined by struct fi_cntr_attr:

struct fi_cntr_attr {
    enum fi_cntr_events  events;    /* type of events to count */
    enum fi_wait_obj     wait_obj;  /* requested wait object */
    struct fid_wait     *wait_set;  /* optional wait set, deprecated */
    uint64_t             flags;     /* operation flags */
};

events — a counter captures different types of events, and the specific type it counts is one of these babes:

  • FI_CNTR_EVENTS_COMP — the counter increments for every successful completion that occurs on an associated bound endpoint. The kind of completions being counted — sends and/or receives — can be restricted using control flags when you bind the counter and the endpoint 🔥. Counters increment on all successful completions, completely separately from whether the operation generates an entry in an event queue.
  • FI_CNTR_EVENTS_BYTES — the counter gets incremented by the number of user bytes transferred in a transport message, excluding any CQ data, once it reaches the specified completion semantic 🎀. Breaking it down:
    • For initiator side counters, the count reflects the size of the requested transfer and is updated after the message reaches the desired completion level (FI_INJECT_COMPLETE, FI_TRANSMIT_COMPLETE, etc.).
    • For send and write operations, the count reflects the number of bytes transferred to the peer.
    • For read operations, the count reflects the number of bytes returned in a read response.
    • Operations that may both write and read data — like atomics — behave as read operations at the initiator, but as writes at the target. multitasking icon 💅.
    • For target side counters, the count reflects the size of received user data and is incremented subject to target side completion semantics. In most cases that means FI_DELIVERY_COMPLETE, but it may differ when accessing device memory (HMEM).
    • On error, the transfer size is not applied to the error field — that field is incremented by 1.
    • The FI_COLLECTIVE transfer type is not supported. like, at all. sorry bestie 😭.

wait_obj — counters may be associated with a specific wait object 🦋. Wait objects let applications block until the wait object is signaled, meaning the counter has reached a specific threshold. Users may use fi_control to retrieve the underlying wait object associated with a counter, so they can use it in other system calls. These are the values you can use to specify the type of wait object:

  • FI_WAIT_NONE — used to indicate that the user will not block (wait) for events on the counter.
  • FI_WAIT_UNSPEC — specifies that the user will only wait on the counter using fabric interface calls, like fi_cntr_wait. In this case the underlying provider may pick the most appropriate or highest performing wait object available, including custom wait mechanisms. Applications that select FI_WAIT_UNSPEC are not guaranteed to retrieve the underlying wait object 🙈.
  • FI_WAIT_SET (deprecated) — indicates the event counter should use a wait set object to wait for events. If specified, the wait_set field must reference an existing wait set object.
  • FI_WAIT_FD — indicates the counter should use a file descriptor as its wait mechanism. A file descriptor wait object must be usable in select, poll, and epoll routines. That said, a provider may signal an FD wait object by marking it as readable, writable, or with an error 👀.
  • FI_WAIT_MUTEX_COND (deprecated) — specifies the counter should use a pthread mutex and cond variable as a wait object.
  • FI_WAIT_YIELD — indicates the counter will wait without a wait object, and instead yields on every wait 🌀. This allows usage of fi_cntr_wait through a spin.

The default is FI_WAIT_NONE.

wait_set (deprecated) — if wait_obj is FI_WAIT_SET, this field references a wait object the event counter should attach to. When an event is added to the event counter, the corresponding wait set will be signaled if all necessary conditions are met ✨. Using a wait_set gives you an optimized way to wait for events across multiple event counters. This field is ignored if wait_obj is not FI_WAIT_SET.

flags — reserved for future use, and must be set to 0. no funny business 🔥.

🧚 fi_close 🧚

The fi_close call releases all resources associated with a counter. When you close the counter, there must be no opened endpoints, transmit contexts, receive contexts or memory regions associated with it 🫶. If resources are still hanging around when you try to close, the call returns -FI_EBUSY. so, like, clean up your room first bestie 💀.

⭐ fi_cntr_control ⭐

The fi_cntr_control call is how you access provider- or implementation-specific details of the counter. Access to the counter should be serialized across all calls when fi_cntr_control is invoked, because it may redirect the implementation of counter operations ⚠️. These are the control commands usable with a counter:

  • FI_GETOPSFLAG (uint64_t *) — returns the current default operational flags associated with the counter.
  • FI_SETOPSFLAG (uint64_t *) — modifies the current default operational flags associated with the counter.
  • FI_GETWAIT (void **) — this command lets the user retrieve the low-level wait object associated with the counter 💅. The format of the wait-object is specified during counter creation, through the counter attributes. See fi_eq.3 for additional details on using control with FI_GETWAIT.

🍓 fi_cntr_read 🍓

The fi_cntr_read call returns the current value of the counter.

Applications must manage counter values that approach the limit defined by the domain attribute max_cntr_value 🚨. This value defaults to UINT64_MAX but may be lower due to provider limitations, and it defines the maximum value returned by fi_cntr_read().

When a counter reaches its maximum value and additional operations complete, the counter behavior will be undefined 😭. Applications that need to track the total number of operations beyond the provider's counter limits are responsible for detecting and handling this condition by resetting counters with fi_cntr_set before they reach the maximum — otherwise it could result in incorrect values. no cap, watch your numbers.

🌈 fi_cntr_readerr 🌈

The read error call returns the number of operations that completed in error and were unable to update the counter.

Applications must manage error counter values that approach the limit defined by the domain attribute max_err_cntr_value 🚨. This value defaults to UINT64_MAX but may be lower due to provider limitations, and it defines the maximum value returned by fi_cntr_readerr().

When an error counter reaches its maximum value and additional error operations complete, the counter behavior will be undefined 💀. Applications that need to track the total number of error operations beyond the provider's counter limits are responsible for detecting and handling this condition by resetting the error counters with fi_cntr_seterr before they reach the maximum — or it could result in incorrect values.

💜 fi_cntr_add 💜

This adds the user-specified value to the counter.

💜 fi_cntr_adderr 💜

This adds the user-specified value to the error value of the counter.

💜 fi_cntr_set 💜

This sets the counter to the specified value.

💜 fi_cntr_seterr 💜

This sets the error value of the counter to the specified value.

👑 fi_cntr_wait 👑

This call may be used to wait until the counter reaches the specified threshold, or until an error or timeout occurs 🕐. On successful return from this call, the counter will be greater than or equal to the input threshold value.

If an operation associated with the counter encounters an error, it will increment the error value associated with the counter. Any change in a counter's error value will unblock any thread sitting inside fi_cntr_wait 🔔.

If the call returns due to a timeout, -FI_ETIMEDOUT is returned. The error value associated with the counter remains unchanged.

It is invalid for applications to call this function if the counter has been configured with a wait object of FI_WAIT_NONE or FI_WAIT_SET 🙅‍♀️.

✨ RETURN VALUES ✨

Returns 0 on success. On error, a negative value corresponding to fabric errno is returned.

  • fi_cntr_read / fi_cntr_readerr — returns the current value of the counter.

Fabric errno values are defined in rdma/fi_errno.h 💖.

🌸 NOTES 🌸

okay, important vibe check 🦋 — to support a whole variety of counter implementations, updates made to counter values (e.g. fi_cntr_set or fi_cntr_add) may not be immediately visible to counter read operations (i.e. fi_cntr_read or fi_cntr_readerr). A small, but undefined, delay may occur between the counter changing and the reported value being updated 🙈. That said, a final updated value will eventually be reflected in the read counter value. it gets there, it's just on its own timeline.

Also, applications should make sure the value of a counter is stable and not subject to change before calling fi_cntr_set or fi_cntr_seterr ⚠️. Otherwise, the resulting value of the counter after fi_cntr_set / fi_cntr_seterr is undefined, because updates to the counter may be lost 💀. A counter value counts as stable if all previous updates using fi_cntr_set / fi_cntr_seterr and results of related operations are reflected in the observed value of the counter. periodt.

🌈 SEE ALSO

okay bestie, when you're done living your counter era, go read these next — they're literally the whole friend group:

  • fi_getinfo(3) — the "hi bestie what can you do" one 🫶
  • fi_endpoint(3) — where all the action actually happens ✨
  • fi_domain(3) — the domain, the vibe container 💅
  • fi_eq(3) — event queues, the other way to find out stuff happened 👀
  • fi_poll(3) — poll sets, for when you're feeling ✨proactive✨

🎀 AUTHORS

OpenFabrics. — that's the whole credit line, no cap. The OpenFabrics crew are the ones behind this, iconic behavior honestly 👑

and that's the entire page, bestie!! you ate this whole fi_cntr(3) situation up, fr fr 🌸

this lil masterpiece is from the Libfabric Programmer's Manual, dated 2026-04-03, and it's the fi_cntr(3) page — so if you ever need to cite it or find it again, iykyk 💖

now go make those counters count, queen 🦋✨

📦 where this page comes from