fi_poll(3)

📦
version 2.5.1 · 8 versions
, in centos-stream 10

💅 fi_poll(3) — bestie's guide to polling & wait sets (Libfabric v2.5.1)

okay so like, real talk first: this whole family of calls is deprecated 😭 — not me showing up to the group chat with legacy APIs. But you still gotta know it, so let's go 🫶

This is fi_poll, the polling-and-wait-set situation from Libfabric. Here's the vibe of every function in the set, then the actual synopsis further down 👇

  • fi_poll_open / fi_close (deprecated) — open or close a polling set
  • fi_poll_add / fi_poll_del (deprecated) — add or remove a completion queue or counter to/from a poll set
  • fi_poll (deprecated) — poll for progress and events across multiple completion queues and counters
  • fi_wait_open / fi_close (deprecated) — open or close a wait set
  • fi_wait (deprecated) — waits for one or more wait objects in a set to be signaled
  • fi_trywait — indicate when it's safe to block on wait objects using native OS calls ✨
  • fi_control — control wait set operation or attributes

the way that's seven functions in one manpage and half of them are deprecated… iconic but cursed 💀

✨ SYNOPSIS ✨

like, the receipts 🧾:

#include <rdma/fi_domain.h>

int fi_poll_open(struct fid_domain *domain, struct fi_poll_attr *attr,
    struct fid_poll **pollset);

int fi_close(struct fid *pollset);

int fi_poll_add(struct fid_poll *pollset, struct fid *event_fid,
    uint64_t flags);

int fi_poll_del(struct fid_poll *pollset, struct fid *event_fid,
    uint64_t flags);

int fi_poll(struct fid_poll *pollset, void **context, int count);

int fi_wait_open(struct fid_fabric *fabric, struct fi_wait_attr *attr,
    struct fid_wait **waitset);

int fi_close(struct fid *waitset);

int fi_wait(struct fid_wait *waitset, int timeout);

int fi_trywait(struct fid_fabric *fabric, struct fid **fids, size_t count);

int fi_control(struct fid *waitset, int command, void *arg);

no cap, that's the whole lineup 🔥

🎀 ARGUMENTS 🎀

okay so every single parameter, and what it actually means, no skipping 👑

  • fabric — the Fabric provider 🧵
  • domain — the Resource domain
  • pollset — Event poll set ✨
  • waitset — Wait object set
  • attr — Poll or wait set attributes
  • context — on success, an array of user context values associated with completion queues or counters 🫶
  • fids — an array of fabric descriptors, each one associated with a native wait object
  • count — number of entries in the context or fids array
  • timeout — time to wait for a signal, in milliseconds ⏱️
  • command — command of control operation to perform on the wait set
  • arg — optional control argument
heads up bestie: yes, count does double duty — it's the number of entries in context for fi_poll and the number of entries in fids for fi_trywait. same word, two arrays, stay woke 💅

✨ DESCRIPTION ✨

🎀 fi_poll_open (deprecated)

okay so fi_poll_open creates a brand-new polling set, bestie. A poll set is literally this optimized lil method for progressing asynchronous operations across multiple completion queues and counters, and checking whether they've completed. it's giving "one queue to rule them all" energy 💅

A poll set is defined with these attributes, and yes you're gonna want to memorize the struct:

struct fi_poll_attr {
    uint64_t             flags;     /* operation flags */
};
  • flags — flags that set the default operation of the poll set. The use of this field is reserved, so the caller must set it to 0. no cap, just zero it out 🙈

🎀 fi_close (deprecated)

The fi_close call releases all resources associated with a poll set. Heads up though: the poll set must not be associated with any other resources prior to being closed — otherwise the call will return -FI_EBUSY. so clean up your faves first, bestie 😭

🎀 fi_poll_add (deprecated)

Associates a completion queue or counter with a poll set. that's it, that's the whole vibe 🫶

🎀 fi_poll_del (deprecated)

Removes a completion queue or counter from a poll set. breaking up is hard to do but sometimes you gotta 💔

🎀 fi_poll (deprecated)

Progresses all completion queues and counters associated with a poll set, and checks for events. If events might have occurred, the contexts associated with the completion queues and/or counters get returned. let me spell out the rules:

  • Completion queues will return their context if they are not empty.
  • The context associated with a counter will be returned if the counter's success value or error value have changed since the last time fi_poll, fi_cntr_set, or fi_cntr_add were called.
  • The number of contexts is limited to the size of the context array, indicated by the count parameter.
⚠️ Note that fi_poll only indicates that events might be available. In some cases, providers may consume such events internally — to drive progress, for example. This can result in fi_poll returning false positives. Applications should drive their progress based on the results of reading events from a completion queue or reading counter values. The fi_poll function will always return all completion queues and counters that do have new events. so like, don't come for the false positives, they're a known thing 💀

🧚 fi_wait_open (deprecated)

fi_wait_open allocates a new wait set. A wait set enables an optimized method of waiting for events across multiple completion queues and counters. Where possible, a wait set uses a single underlying wait object that is signaled when a specified condition occurs on an associated completion queue or counter. iconic behavior honestly ✨

The properties and behavior of a wait set are defined by struct fi_wait_attr:

struct fi_wait_attr {
    enum fi_wait_obj     wait_obj;  /* requested wait object */
    uint64_t             flags;     /* operation flags */
};

🌸 wait_obj

Wait sets are associated with specific wait object(s). Wait objects allow applications to block until the wait object is signaled, indicating that an event is available to be read. The following values may be used to specify the type of wait object associated with a wait set: FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND (deprecated), and FI_WAIT_YIELD. lowkey a whole menu, pick your fave 🍓

  • FI_WAIT_UNSPEC — Specifies that the user will only wait on the wait set using fabric interface calls, such as fi_wait. In this case, the underlying provider may select the most appropriate or highest performing wait object available, including custom wait mechanisms. Applications that select FI_WAIT_UNSPEC are not guaranteed to retrieve the underlying wait object. mysterious. love that for you 🎀
  • FI_WAIT_FD — Indicates that the wait set should use a single file descriptor as its wait mechanism, as exposed to the application. Internally, this may require the use of epoll in order to support waiting on a single file descriptor. File descriptor wait objects must be usable in the POSIX select(2) and poll(2), and Linux epoll(7) routines (if available). Providers signal an FD wait object by marking it as readable or with an error. very readable, very demure 💅
  • FI_WAIT_MUTEX_COND (deprecated) — Specifies that the wait set should use a pthread mutex and cond variable as a wait object.
  • FI_WAIT_POLLFD — This option is similar to FI_WAIT_FD, but allows the wait mechanism to use multiple file descriptors as its wait mechanism, as viewed by the application. The use of FI_WAIT_POLLFD can eliminate the need to use epoll to abstract away needing to check multiple file descriptors when waiting for events. The file descriptors must be usable in the POSIX select(2) and poll(2) routines, and match directly to being used with poll. See the NOTES section below for details on using pollfd. ok multitasking queen 👑
  • FI_WAIT_YIELD — Indicates that the wait set will wait without a wait object but instead yield on every wait. no object, just vibes ✨

🌸 flags

  • flags — flags that set the default operation of the wait set. The use of this field is reserved, and the caller must set it to 0. same rule as before, bestie, don't get creative 🙈

🎀 fi_close (deprecated)

The fi_close call releases all resources associated with a wait set. The wait set must not be bound to any other opened resources prior to being closed — otherwise the call will return -FI_EBUSY. yes this is a different fi_close, yes it's still deprecated, yes you still gotta unbind first 😭

🎀 fi_wait (deprecated)

Waits on a wait set until one or more of its underlying wait objects is signaled. simply waiting, simply slaying 🦋

💜 fi_trywait

The fi_trywait call was introduced in libfabric version 1.3. The behavior of using native wait objects without the use of fi_trywait is provider specific and should be considered non-deterministic. so like, don't rawdog it, fr fr 🫶

The fi_trywait() call is used in conjunction with native operating system calls to block on wait objects, such as file descriptors. The application must call fi_trywait and obtain a return value of FI_SUCCESS prior to blocking on a native wait object. Failure to do so may result in the wait object not being signaled, and the application not observing the desired events. The following pseudo-code demonstrates the use of fi_trywait in conjunction with the OS select(2) call:

fi_control(&cq->fid, FI_GETWAIT, (void *) &fd);
FD_ZERO(&fds);
FD_SET(fd, &fds);

while (1) {
    if (fi_trywait(&cq, 1) == FI_SUCCESS)
        select(fd + 1, &fds, NULL, &fds, &timeout);

    do {
        ret = fi_cq_read(cq, &comp, 1);
    } while (ret > 0);
}

fi_trywait() will return FI_SUCCESS if it is safe to block on the wait object(s) corresponding to the fabric descriptor(s), or -FI_EAGAIN if there are events queued on the fabric descriptor or if blocking could hang the application. read that again: blocking could hang the app. we're being careful here 💀

The call takes an array of fabric descriptors. For each wait object that will be passed to the native wait routine, the corresponding fabric descriptor should first be passed to fi_trywait. All fabric descriptors passed into a single fi_trywait call must make use of the same underlying wait object type. no mixing, bestie, it's not a smoothie 🌈

The following types of fabric descriptors may be passed into fi_trywait: event queues, completion queues, counters, and wait sets. Applications that wish to use native wait calls should select specific wait objects when allocating such resources. For example, by setting the item's creation attribute wait_obj value to FI_WAIT_FD.

In the case the wait object to check belongs to a wait set, only the wait set itself needs to be passed into fi_trywait. The fabric resources associated with the wait set do not. the wait set is the main character here 👑

On receiving a return value of -FI_EAGAIN from fi_trywait, an application should read all queued completions and events, and call fi_trywait again before attempting to block. Applications can make use of a fabric poll set to identify completion queues and counters that may require processing. so like, drain the queue, then try again, then block. in that order. periodt 💅

💜 fi_control

The fi_control call is used to access provider or implementation specific details of a fids that support blocking calls, such as wait sets, completion queues, counters, and event queues. Access to the wait set or fid should be serialized across all calls when fi_control is invoked, as it may redirect the implementation of wait set operations. one at a time, bestie, no chaotic group-chat energy here 🙈

The following control commands are usable with a wait set or fid:

  • FI_GETWAIT (void **) — This command allows the user to retrieve the low-level wait object associated with a wait set or fid. The format of the wait set is specified during wait set creation, through the wait set attributes. The fi_control arg parameter should be an address where a pointer to the returned wait object will be written. This should be an int * for FI_WAIT_FD, struct fi_mutex_cond for FI_WAIT_MUTEX_COND (deprecated), or struct fi_wait_pollfd for FI_WAIT_POLLFD. Support for FI_GETWAIT is provider specific. know your type before you cast, iykyk ✨
  • FI_GETWAITOBJ (enum fi_wait_obj *) — This command returns the type of wait object associated with a wait set or fid. short, sweet, iconic 🎀

⭐ RETURN VALUES ⭐

Returns FI_SUCCESS on success. On error, a negative value corresponding to fabric errno is returned.

Fabric errno values are defined in rdma/fi_errno.h. go look 'em up if you need 'em, bestie 💖

🌸 fi_poll

On success, if events are available, returns the number of entries written to the context array. so the return value is literally "how many little contexts did i write for you" 🫶

🧚 NOTES 🧚

Okay so blocking calls often need to wait on signals sent to a bunch of file descriptors — literally so many situations. That’s the vibe for socket based providers like tcp and udp, plus utility providers like multi-rail. For simplicity, when epoll is available, it can be used to limit the number of file descriptors that an application must monitor. The use of epoll may also be required in order to support FI_WAIT_FD. Slay.

But bestie, in order to support waiting on multiple file descriptors on systems where epoll support is not available, or where epoll performance may negatively impact performance, FI_WAIT_POLLFD provides this mechanism. A significant difference between using POLLFD versus FD wait objects is that with FI_WAIT_POLLFD, the file descriptors may change dynamically. Example: the file descriptors associated with a completion queues’ wait set may change as endpoint associations with the CQ are added and removed. Iconic but keep up.

struct fi_wait_pollfd is used to retrieve all file descriptors for fids using FI_WAIT_POLLFD to support blocking calls. Here’s the struct in all its glory:

struct fi_wait_pollfd {
    uint64_t      change_index;
    size_t        nfds;
    struct pollfd *fd;
};
  • change_index — this field helps you determine if there have been any changes to the file descriptor list. Anytime a file descriptor is added, removed, or its events are updated, this field is incremented by the provider. Applications wishing to wait on file descriptors directly should cache the change_index value. Before blocking on file descriptor events, the app should use fi_control() to retrieve the current change_index and compare that against its cached value. If the values differ, then the app should update its file descriptor list prior to blocking. No cap, this is the vibe check.
  • nfds — on input to fi_control(), this indicates the number of entries in the struct pollfd * array. On output, this will be set to the number of entries needed to store the current number of file descriptors. If the input value is smaller than the output value, fi_control() will return the error -FI_ETOOSMALL. Note that setting nfds = 0 allows an efficient way of checking the change_index. Efficiency queen.
  • fd — this points to an array of struct pollfd entries. The number of entries is specified through the nfds field. If the number of needed entries is less than or equal to the number of entries available, the struct pollfd array will be filled out with a list of file descriptors and corresponding events that can be used in the select(2) and poll(2) calls. Perfect for your poll-and-select era.

The change_index is updated only when the file descriptors associated with the pollfd file set has changed. Checking the change_index is an additional step needed when working with FI_WAIT_POLLFD wait objects directly. The use of the fi_trywait() function is still required if accessing wait objects directly. So yes, you still need fi_trywait() — don’t skip it, bestie. 💅

🌈 SEE ALSO 🌈

👑 AUTHORS 👑

OpenFabrics.

xoxo, source/version Libfabric Programmer’s Manual 💖 — dated 2024-12-10, page fi_poll(3). Stay slay, bestie! 🫶

📦 where this page comes from