fi_tcp(7)

💅 fi_tcp(7) — the TCP/IP girlies of libfabric

okay so real quick before we dive in: this is a page from the Libfabric programmer's manual, and it's all about the tcp provider. buckle up bestie 🎀

---

✨ NAME ✨

fi_tcp — the provider that literally just runs over TCP/IP. no cap.

---

🦋 OVERVIEW 🦋

so the tcp provider is usable on all operation systems supported by libfabric. all of them. it runs over TCP (SOCK_STREAM) sockets, and it's got that range too — it can leverage operation specific features, like support for zero-copy and io_uring. it implements a custom protocol over TCP/IP, which is the whole thing it needs to support the libfabric communication APIs. main character energy fr 💅

---

👑 SUPPORTED FEATURES 👑

here's what it eats, in list form:

⭐ Endpoint types

• FI_EP_MSG
• FI_EP_RDM

⭐ Endpoint capabilities

• FI_MSG
• FI_RMA
• FI_TAGGED
• FI_RMA_PMEM
• FI_RMA_EVENT
• FI_MULTI_RECV
• FI_DIRECTED_RECV

⭐ Shared Rx Context

the tcp provider supports shared receive context. slay 💖

---

🌸 RUNTIME PARAMETERS 🌸

okay so the tcp provider can be configured with several environment variables. what's below is just a subset of the supported variables — for the full list, use the fi_info utility application. like, for example, fi_info -g tcp will show you all the environment variables defined for the tcp provider. iykyk 🔥

• FI_TCP_IFACE — you can request a specific network interface with this one.
• FI_TCP_PORT_LOW_RANGE/FI_TCP_PORT_HIGH_RANGE — these two set the range of ports the tcp provider uses when it creates its passive endpoint. super useful when a firewall only allows a range of ports for tcp connections, ngl.
• FI_TCP_TX_SIZE — the transmit context size. this is the number of transmit requests an application may post to the provider before it gets -FI_EAGAIN back. default: 256 for msg endpoints, 64k for rdm.
• FI_TCP_RX_SIZE — the receive context size, aka the number of receive buffers the application may post to the provider. default: 256 for msg endpoints, 64k for rdm.
• FI_TCP_MAX_INJECT — the maximum size of inject messages, plus the maximum size of an unexpected message that may be buffered at the receiver. default 128 bytes.
• FI_TCP_STAGING_SBUF_SIZE — the size of the buffer used to coalesce iovec's or send requests before posting to the kernel. the staging buffer gets used when the socket is busy and can't accept new data — in that case the data can be queued in the staging buffer until the socket resumes sending. this optimizes transferring a series of back-to-back small messages to the same target. default: 9000 bytes. set it to 0 to disable.
• FI_TCP_PREFETCH_RBUF_SIZE — the size of the buffer used to prefetch received data from the kernel. when it starts receiving a new message, the provider will ask the kernel to fill the prefetch buffer and then process received data from there. this reduces the number of kernel calls needed to receive a series of small messages. default: 9000 bytes. set it to 0 to disable.
• FI_TCP_ZEROCOPY_SIZE — the lower threshold where zero copy transfers will be used, if supported by the platform. set to -1 to disable. default: disabled.
• FI_TCP_TRACE_MSG — if enabled, it will log transport message information on all sent and received messages. must be paired with FI_LOG_LEVEL=trace to actually print the message details, bestie. 🙈
• FI_TCP_IO_URING — uses io_uring for socket operations if it's available, rather than going through the standard socket APIs (i.e. connect, accept, send, recv). default: disabled.

---

🧚 CONTROL OPERATIONS 🧚

the tcp provider supports the following control operations (see fi_control(3)):

• FI_GET_FD — retrieves the underlying socket file descriptor associated with an active endpoint. the argument must point to an integer where the descriptor will be stored. this is how applications tune socket options that aren't exposed through the libfabric API (SO_SNDBUF, SO_RCVBUF, SO_BUSY_POLL, etc). obsessed, honestly.

---

💀 NOTES 💀

the tcp provider supports both msg and rdm endpoints directly. rdm endpoint support is available starting at libfabric version v1.18.0, and it comes from the merge back of the net provider found in libfabric versions v1.16 and v1.17. for compatibility with older libfabric versions, the tcp provider may be paired with the ofi_rxm provider as an alternative solution for rdm endpoint support. it is recommended that applications that do not need wire compatibility with older versions of libfabric use the rdm endpoint support directly from the tcp provider — that gives you the best performance. periodt. 🔥

---

🌈 SEE ALSO 🌈

fabric(7), fi_provider(7), fi_getinfo(3)

---

💜 AUTHORS 💜

OpenFabrics. 💖

---

okay that's the whole thing, go forth and connect, bestie 🫶

---

and that's a wrap! brought to you by the Libfabric Programmer's Manual, dated 2025-05-02, living in the Libfabric v2.5.1 era ✨🎀💅 — see you in the group chat 💖