RFC 7609
IBM's Shared Memory Communications over RDMA (SMC-R) Protocol
Okay bestie, let's get into RFC 7609 🎀✨. The title is IBM's Shared Memory Communications over RDMA (SMC-R) Protocol, and it's an Independent Submission, which means it came in outside the usual IETF working groups 💅. It's marked Informational (ISSN 2070-1721), dated August 2015, and written by three people from IBM: M. Fox, C. Kassimis, and J. Stevens 🧚. The title block doesn't list any documents it obsoletes or updates, so it's a standalone doc 🌸.
Abstract
okay so this doc describes IBM's SMC-R protocol 🦋. Its whole thing is giving TCP endpoints access to RDMA (Remote Direct Memory Access) communications, and the socket apps don't even have to know it's happening, which is so iconic 😭. It also handles dynamic discovery of what your partner can do with RDMA, plus dynamic setup of RDMA connections 💜. When there are redundant RDMA network paths, it gives you transparent high availability and load balancing, so you're never left on read 🫶. And the best part: it keeps the classic TCP/IP qualities of service, like the filtering that enterprise users absolutely demand, and it keeps TCP socket semantics like urgent data too 🔥. Main character energy, literally ✨.
Status of This Memo
This document is not an Internet Standards Track specification, so it's just published for informational purposes 🍓. It's a contribution to the RFC Series, made independently of any other RFC stream 🌈. The RFC Editor chose to publish it at their discretion, and they make no statement about its value for implementation or deployment, so, ngl, you decide if it's your vibe 💅. Documents the RFC Editor approves for publication aren't a candidate for any level of Internet Standard, see Section 2 of RFC 5741 🙈.
Info about the current status of this document, any errata, and how to give feedback is at http://www.rfc-editor.org/info/rfc7609 💖.
Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved ✨.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, bestie, because they describe your rights and restrictions with respect to this document 🫶📜.
Part 1 of 32 done, more parts coming, so stay tuned 🦋💜
Table of Contents ✨
okay bestie, this is the whole map of the spec 🗺️💅 every section, subsection and page number, lined up like a little itinerary 🧚
1. Introduction ....................................................5
1.1. Protocol Overview ..........................................6
1.1.1. Hardware Requirements ...............................8
1.2. Definition of Common Terms .................................8
1.3. Conventions Used in This Document .........................11
2. Link Architecture ..............................................11
2.1. Remote Memory Buffers (RMBs) ..............................12
2.2. SMC-R Link Groups .........................................18
2.2.1. Link Group Types ...................................18
2.2.2. Maximum Number of Links in Link Group ..............21
2.2.3. Forming and Managing Link Groups ...................23
2.2.4. SMC-R Link Identifiers .............................24
2.3. SMC-R Resilience and Load Balancing .......................24
3. SMC-R Rendezvous Architecture ..................................26
3.1. TCP Options ...............................................26
3.2. Connection Layer Control (CLC) Messages ...................27
3.3. LLC Messages ..............................................27
3.4. CDC Messages ..............................................29
3.5. Rendezvous Flows ..........................................29
3.5.1. First Contact ......................................29
3.5.1.1. Pre-negotiation of TCP Options ............29
3.5.1.2. Client Proposal ...........................30
3.5.1.3. Server Acceptance .........................32
3.5.1.4. Client Confirmation .......................32
3.5.1.5. Link (QP) Confirmation ....................32
3.5.1.6. Second SMC-R Link Setup ...................35
3.5.1.6.1. Client Processing of ADD LINK
LLC Message from Server ........35
3.5.1.6.2. Server Processing of ADD LINK
Reply LLC Message from Client ..36
3.5.1.6.3. Exchange of RKeys on
Second SMC-R Link ..............38
3.5.1.6.4. Aborting SMC-R and
Falling Back to IP .............38
3.5.2. Subsequent Contact .................................38
3.5.2.1. SMC-R Proposal ............................39
3.5.2.2. SMC-R Acceptance ..........................40
3.5.2.3. SMC-R Confirmation ........................41
3.5.2.4. TCP Data Flow Race with SMC
Confirm CLC Message .......................41
3.5.3. First Contact Variation: Creating a
Parallel Link Group ................................42
3.5.4. Normal SMC-R Link Termination ......................43
3.5.5. Link Group Management Flows ........................44
3.5.5.1. Adding and Deleting Links in an
SMC-R Link Group ..........................44
3.5.5.1.1. Server-Initiated ADD
LINK Processing ................45
3.5.5.1.2. Client-Initiated ADD
LINK Processing ................45
3.5.5.1.3. Server-Initiated DELETE
LINK Processing ................46
3.5.5.1.4. Client-Initiated DELETE
LINK Request ...................48
3.5.5.2. Managing Multiple RKeys over
Multiple SMC-R Links in a Link Group ......49
3.5.5.2.1. Adding a New RMB to an
SMC-R Link Group ...............50
3.5.5.2.2. Deleting an RMB from an
SMC-R Link Group ...............53
3.5.5.2.3. Adding a New SMC-R Link to a
Link Group with Multiple RMBs ..54
3.5.5.3. Serialization of LLC Exchanges,
and Collisions ............................56
3.5.5.3.1. Collisions with ADD
LINK / CONFIRM LINK Exchange ...57
3.5.5.3.2. Collisions during
DELETE LINK Exchange ...........58
3.5.5.3.3. Collisions during
CONFIRM RKEY Exchange ..........59
4. SMC-R Memory-Sharing Architecture ..............................60
4.1. RMB Element Allocation Considerations .....................60
4.2. RMB and RMBE Format .......................................60
4.3. RMBE Control Information ..................................60
4.4. Use of RMBEs ..............................................61
4.4.1. Initializing and Accessing RMBEs ...................61
4.4.2. RMB Element Reuse and Conflict Resolution ..........62
4.5. SMC-R Protocol Considerations .............................63
4.5.1. SMC-R Protocol Optimized Window Size Updates .......63
4.5.2. Small Data Sends ...................................64
4.5.3. TCP Keepalive Processing ...........................65
4.6. TCP Connection Failover between SMC-R Links ...............67
4.6.1. Validating Data Integrity ..........................67
4.6.2. Resuming the TCP Connection on a New SMC-R Link ....68
4.7. RMB Data Flows ............................................69
4.7.1. Scenario 1: Send Flow, Window Size Unconstrained ...69
4.7.2. Scenario 2: Send/Receive Flow, Window Size
Unconstrained ......................................71
4.7.3. Scenario 3: Send Flow, Window Size Constrained .....72
4.7.4. Scenario 4: Large Send, Flow Control, Full
Window Size Writes .................................74
4.7.5. Scenario 5: Send Flow, Urgent Data, Window
Size Unconstrained .................................77
4.7.6. Scenario 6: Send Flow, Urgent Data, Window
Size Closed ........................................79
4.8. Connection Termination ....................................81
4.8.1. Normal SMC-R Connection Termination Flows ..........81
4.8.2. Abnormal SMC-R Connection Termination Flows ........86
4.8.3. Other SMC-R Connection Termination Conditions ......88
5. Security Considerations ........................................89
5.1. VLAN Considerations .......................................89
5.2. Firewall Considerations ...................................89
5.3. Host-Based IP Filters .....................................89
5.4. Intrusion Detection Services ..............................90
5.5. IP Security (IPsec) .......................................90
5.6. TLS/SSL ...................................................90
6. IANA Considerations ............................................90
7. Normative References ...........................................91
Appendix A. Formats ...............................................92
A.1. TCP Option .................................................92
A.2. CLC Messages ...............................................92
A.2.1. Peer ID Format ......................................93
A.2.2. SMC Proposal CLC Message Format .....................94
A.2.3. SMC Accept CLC Message Format .......................98
A.2.4. SMC Confirm CLC Message Format .....................102
A.2.5. SMC Decline CLC Message Format .....................105
A.3. LLC Messages ..............................................106
A.3.1. CONFIRM LINK LLC Message Format ....................107
A.3.2. ADD LINK LLC Message Format ........................109
A.3.3. ADD LINK CONTINUATION LLC Message Format ...........112
A.3.4. DELETE LINK LLC Message Format .....................115
A.3.5. CONFIRM RKEY LLC Message Format ....................117
A.3.6. CONFIRM RKEY CONTINUATION LLC Message Format .......120
A.3.7. DELETE RKEY LLC Message Format .....................122
A.3.8. TEST LINK LLC Message Format .......................124
A.4. Connection Data Control (CDC) Message Format ..............125
Appendix B. Socket API Considerations ............................129
B.1. setsockopt() / getsockopt() Considerations ................130
Appendix C. Rendezvous Error Scenarios ...........................131
C.1. SMC Decline during CLC Negotiation ........................131
C.2. SMC Decline during LLC Negotiation ........................131
C.3. The SMC Decline Window ....................................133
C.4. Out-of-Sync Conditions during SMC-R Negotiation ...........133
C.5. Timeouts during CLC Negotiation ...........................134
C.6. Protocol Errors during CLC Negotiation ....................134
C.7. Timeouts during LLC Negotiation ...........................135
C.7.1. Recovery Actions for LLC Timeouts and Failures .....136
C.8. Failure to Add Second SMC-R Link to a Link Group ..........142
Authors' Addresses ...............................................1431. Introduction ✨
okay bestie, this is the big intro 🎀 This document specifies IBM's Shared Memory Communications over RDMA (SMC-R) protocol 💅 SMC-R is a protocol for Remote Direct Memory Access (RDMA) communication between TCP socket endpoints. It runs over networks that support RDMA over Converged Ethernet (RoCE) 🌈 The whole vibe is letting existing TCP apps benefit from RDMA without needing any changes to the apps or any predefinition of RDMA partners. no notes, no setup drama 🧚
SMC-R does dynamic discovery of the RDMA capabilities of TCP peers and sets up RDMA connections that those peers can use, all automatically 🦋 It also brings transparent high availability and load-balancing, which enterprise installations are obsessed with and which current RDMA protocols are missing 😭 If redundant RoCE-capable hardware is around, like RDMA-capable Network Interface Cards (RNICs) and RoCE-capable switches, SMC-R can load-balance across that redundant hardware and can also non-disruptively move TCP traffic off failed paths onto surviving paths. And it's all seamless to the application and the sockets layer 👑 Because SMC-R keeps socket semantics and the TCP three-way handshake, a bunch of TCP qualities of service stay intact, like filtering, load balancing, and Secure Socket Layer (SSL) encryption. TCP features like urgent data are still there too ✨
Because peers find each other and set up SMC-R connectivity dynamically, no RDMA connection manager (RDMA-CM) is needed. Which also means Unreliable Datagram (UD) Queue Pairs (QPs) support isn't required either 🌸
It is recommended that the SMC-R services be implemented in kernel space 💜 That unlocks optimizations like resource-sharing between connections across multiple processes, and it lets apps using SMC-R spawn multiple processes (e.g., fork) without losing SMC-R functionality. A user-space implementation is compatible with this architecture, but it may not support spawned processes (e.g., fork). That limits sharing and resource optimization to TCP connections that come from the same process. This might be a good design choice if the use case is a system hosting one large single-process app that creates many TCP connections to a peer host, or in implementations where kernel space isn't possible or where "kernel space to user space" context switches would add excessive overhead 🫶
1.1. Protocol Overview
okay so here's the big picture 🍓 SMC-R defines the concept of the SMC-R link, which is a logical point-to-point link using reliably connected queue pairs between TCP/IP stack peers over a RoCE fabric. An SMC-R link is bound to a specific hardware path, meaning a specific RNIC on each peer. SMC-R links are created and maintained by an SMC-R layer, which may reside in kernel space or user space, depending on operating system and implementation requirements. The SMC-R layer sits below the sockets layer and sends data traffic for TCP connections between connected peers over the RoCE fabric using RDMA, rather than over a TCP connection. The TCP/IP stack, with its requirements for fragmentation, packetization, etc., gets bypassed, and the application data moves between peers using RDMA. bye bye stack overhead 🔥
Multiple SMC-R links between the same two TCP/IP stack peers are also supported 💖 A set of SMC-R links called a link group can be logically bonded together to provide redundant connectivity. If there is redundant hardware -- for example, two RNICs on each peer -- separate SMC-R links are created between the peers to use that redundant hardware. The link group architecture with redundant links provides load balancing and increased bandwidth, as well as seamless failover. main character energy for sure ⭐
Each SMC-R link group is associated with an area of memory called Remote Memory Buffers (RMBs). These are areas of memory that SMC-R peers can write into using RDMA writes 🧚 Multiple TCP connections between peers may be multiplexed over a single SMC-R link, in which case the SMC-R layer manages how the RMBs are partitioned between the TCP connections. This multiplexing cuts down on the RDMA resources, such as QPs and RMBs, needed to support multiple connections between peers. It also cuts down the processing and delays tied to setting up QPs, pinning memory, and other RDMA setup tasks whenever new TCP connections get created 🦋 In a kernel-space SMC-R implementation, where the RMBs live in kernel storage, this sharing and optimization works across multiple processes running on the same host. In a user-space SMC-R implementation, where the RMBs live in user space, the sharing and optimization is limited to multiple TCP connections created by a single process, since separate RMBs and QPs will be required for each process 🙈
SMC-R also introduces a rendezvous protocol 💅 It's used to dynamically discover the RDMA capabilities of TCP connection partners and to exchange the credentials needed to use that capability if it's there. TCP connections get set up using the normal TCP three-way handshake [RFC793], plus a new TCP option that indicates SMC-R capability. If both partners indicate SMC-R capability, then once the three-way TCP handshake completes, the SMC-R layers in each peer take control of the TCP connection and use it to exchange additional Connection Layer Control (CLC) messages. Those messages negotiate SMC-R credentials like QP information, addressability over the RoCE fabric, RMB buffer sizes, and keys and addresses for accessing RMBs over RDMA 🎀 If a failure or decline happens at any point during this negotiation, the TCP connection falls back to using the IP fabric 😭
If the SMC-R negotiation succeeds and either a new SMC-R link gets set up or an existing one is chosen for the TCP connection, then the SMC-R layers open the sockets to the applications, and the apps just use the sockets as normal ✨ The SMC-R layer intercepts the socket reads and writes and moves the TCP connection data over the SMC-R link, "out of band" to the TCP connection. That TCP connection stays open and idle over the IP fabric, except for termination flows and possible keepalive flows. Regular TCP sequence numbering methods are used for the TCP flows that do happen. Data flowing over RDMA doesn't use or affect TCP sequence numbers 🌈
This architecture does not support fallback of active SMC-R connections to IP. Once connection data has completed the switch to RDMA, a TCP connection cannot be switched back to IP and will reset if RDMA becomes unusable. no takebacks, bestie 💀
The SMC-R protocol defines the format of the RMBs that receive TCP connection data written over RDMA, along with the semantics for managing and writing to those buffers using Connection Data Control (CDC) messages 🦋
Finally, SMC-R defines Link Layer Control (LLC) messages that get exchanged over the RoCE fabric between peer SMC-R layers to manage SMC-R links and link groups 🫶 These include messages to test and confirm connectivity over an SMC-R link, to add and delete SMC-R links to or from the link group, and to exchange RMB addressability information. iconic 👑
1.1.1. Hardware Requirements
okay the hardware part is actually chill 🌸 SMC-R does not require full Converged Enhanced Ethernet switch functionality. It works over standard Ethernet fabrics, as long as endpoint RNICs are provided and IEEE 802.3x Global Pause Frame is supported and enabled in the switch fabric 💜
While SMC-R as specified in this document is designed to operate over RoCE fabrics, adjustments to the rendezvous methods could enable it to run over other RDMA fabrics, such as InfiniBand [RoCE] and iWARP. 🌈 not me thinking about the future of this protocol 🧚
1.2. Definition of Common Terms ✨
okay bestie, this section is the glossary 📖 it defines the terms that have a specific meaning to the SMC-R protocol, and they show up everywhere in the rest of the doc, so let's get them locked in 🔒
SMC-R Link
An SMC-R link is a logical point-to-point connection over the RoCE fabric via specific physical adapters (Media Access Control / Global Identifier (MAC/GID)). The link is formed during the "first contact" sequence of the TCP/IP three-way handshake sequence that occurs over the IP fabric. During this handshake, an RDMA reliably connected queue pair (RC-QP) connection is formed between the two peer SMC hosts, and that connection is defined as the SMC-R link. 💖 The SMC-R link can then support multiple TCP connections between the two peers. An SMC-R link is associated with a single LAN (or VLAN) segment and is not routable 🚫
SMC-R Link Group
An SMC-R link group is a group of SMC-R links between the same two SMC-R peers, typically with each link over unique RoCE adapters. ✨ Each link in the link group has equal characteristics, such as the same VLAN ID (if VLANs are in use), access to the same RMB(s), and access to the same TCP server/client. twins energy 👯♀️
SMC-R Peer
The SMC-R peer is the peer software stack within the peer operating system with respect to the Shared Memory Communications (messaging) protocol. 🧚
SMC-R Rendezvous
SMC-R Rendezvous is the SMC-R peer discovery and handshake sequence. It happens transparently over the IP (Ethernet) fabric during and immediately after the TCP connection three-way handshake, by exchanging the SMC-R capabilities and credentials using experimental TCP option and CLC messages. 🦋 the vibe is "secret handshake at the club door" 💅
RoCE SendMsg
RoCE SendMsg is a send operation posted to a reliably connected queue pair with inline data, for the purpose of transferring control information between peers. 📬
TCP Client
The TCP client is the TCP socket-based peer that initiates a TCP connection. 🎀
TCP Server
The TCP server is the TCP socket-based peer that accepts a TCP connection. 🌸
CLC Messages
The SMC-R protocol defines a set of Connection Layer Control messages that flow over the TCP connection. They're used to manage SMC-R link rendezvous at TCP connection setup time. This mechanism is analogous to SSL setup messages. 🔐
LLC Commands
The SMC-R protocol defines a set of RoCE Link Layer Control commands that flow over the RoCE fabric using RoCE SendMsg. They're used to manage SMC-R links, SMC-R link groups, and SMC-R link group RMB expansion and contraction. 🛠️
CDC Message
The SMC-R protocol defines a Connection Data Control message that flows over the RoCE fabric using RoCE SendMsg, and it's used to manage the SMC-R connection data. It provides information about data being transferred over the out-of-band RDMA connection, such as data cursors, sequence numbers, and data flags (for example, urgent data). 🚨 The receipt of this message also provides an interrupt to inform the receiver that it has received RDMA data. this is the "hey, your data just landed" ping 🔔
RMB
A Remote (RDMA) Memory Buffer is a fixed or pinned buffer allocated in each of the peer hosts for a TCP (via SMC-R) connection. The RMB is registered to the RNIC and allows remote access by the remote peer using RDMA semantics. Each host is passed the peer's RMB-specific access information (RMB Key (RKey) and RMB element offset) during the SMC-R Rendezvous process. Then the host stores socket application user data directly into the peer's RMB using RDMA over RoCE. 🧠 zero middleman energy 💜
RToken
The RToken is the combination of an RMB's RKey and RDMA virtual address. An RToken provides RMB addressability information to an RDMA peer. 🗝️
RMBE
The Remote Memory Buffer Element (RMBE) is an area of an RMB that is allocated to a specific TCP connection. The RMBE contains data for the TCP connection. It represents the TCP receive buffer, where the remote peer writes into the RMBE and the local peer reads from the local RMBE. The alert token resolves to a specific RMBE. 🍰 slice of the RMB, just for one connection
Alert Token
The SMC-R alert token is a 4-byte value that uniquely identifies the TCP connection over an SMC-R connection. The alert token lets the SMC peer quickly identify the target TCP connection that now has new work. The format of the token is defined by the owning SMC-R endpoint and is considered opaque to the remote peer. However, the token should not simply be an index to an RMBE. It should reference a TCP connection and be able to be validated, to avoid reading data from stale connections. 🚩 no reading old mail, bestie 📭
RNIC
The RDMA-capable Network Interface Card (RNIC) is an Ethernet NIC that supports RDMA semantics and verbs using RoCE. 🔌
First Contact
"First contact" describes an SMC-R negotiation to set up the first link in a link group. 🤝 the meet-cute 💘
Subsequent Contact
"Subsequent contact" describes an SMC-R negotiation between peers who are already using an existing SMC-R link group. 🫶 returning customer energy
1.3. Conventions Used in This Document 🎨
In the rendezvous flow diagrams, dashed lines (----) indicate flows over the TCP/IP fabric, and dotted lines (....) indicate flows over the RoCE fabric. 🌈
In the data transfer ladder diagrams, dashed lines (----) indicate RDMA write operations, and dotted lines (....) indicate CDC messages. CDC messages are RDMA messages with inline data that contain control information for the connection. 💅
2. Link Architecture 🦋
An SMC-R link is based on reliably connected queue pairs (QPs) that form a "logical point-to-point link" between the two SMC-R peers over a RoCE fabric. An SMC-R link extends from SMC-R peer to SMC-R peer. Typically, each peer would be a TCP/IP stack, and each would reside on separate hosts. 🏠🏠
,,.--..,_
+----+ _-`` `-, +-----+
|QP 8| - RoCE ', |QP 64|
| | / VLAN M . | |
+----+--------+/ \+-------+-----+
| RNIC 1 | SMC-R Link | RNIC 2 |
| |<--------------------->| |
+------------+ , /+------------+
MAC A (GID A) MAC B (GID B)
. .`
`', ,-`
``''--''``
Figure 1: SMC-R Link OverviewFigure 1 gives an overview of the basic concepts of SMC-R peer-to-peer connectivity. This is called the SMC-R link. 🔗 The SMC-R link forms a logical point-to-point connection between two SMC-R peers via RoCE. It's defined and identified by the following attributes:
SMC-R link = RC QPs
(source VMAC GID QP + target VMAC GID QP + VLAN ID)The SMC-R link can optionally be associated with a VLAN ID. If VLANs are in use for the associated IP (LAN) connection, then the VLAN attribute is carried over on the SMC-R link. When VLANs are in use, each SMC-R link group is associated with a single and specific VLAN. The RoCE fabric is the same physical Ethernet LAN used for standard TCP/IP-over-Ethernet communications, with switches as described in Section 1.1.1. 🏷️
An SMC-R link is designed to support multiple TCP connections between the same two peers. It's intended to be long lived, while the underlying TCP connections can dynamically come and go. ⏳ The associated RMBs can also be dynamically added and removed from the link as needed. The first TCP connection between the peers establishes the SMC-R link, and subsequent TCP connections then use the previously established link. When the last TCP connection terminates, the link can then be terminated, typically after an implementation-defined idle timeout period has elapsed. 💤 The TCP server is responsible for initiating and terminating the SMC-R link. 👑
2.1. Remote Memory Buffers (RMBs) 🎀
okay bestie, let's talk about RMBs 💖 Figure 2 shows the two hosts, Host X and Host Y, and the RMBs living inside each one 🏠. With the SMC-R link, plus the RKeys and RDMA virtual addresses, each SMC-R-enabled TCP/IP stack can remotely access its peer's RMBs using RDMA ✨. The RKeys and virtual addresses get swapped during rendezvous processing when the link is set up 🤝. The RKey and the virtual address together make up the RToken 🔑. Also, ngl, the SMC-R link ends at the QP that gives access to the RMB (via the link + RToken) 🦋.
Host X Host Y
+-------------------+ ,.--.,_ +-------------------+
| | .'` '. | |
| Protection | ,' `, | Protection |
| Domain X | / \ | Domain Y |
| +------+ / \ +------+ |
| QP 8 |RNIC 1| | SMC-R Link | |RNIC 2| QP 64 |
| | | |<-------------------->| | | |
| | | || || | | |
| | +------+| VLAN A |+------+ | |
| | || || | |
| | | | RoCE | | | |
| |RToken X | \ / |RToken Y | |
| | | \ / | | |
| V | `. ,' | V |
| +--------+ | '._ ,' | +--------+ |
| | | | `''-'`` | | | |
| | RMB | | | | RMB | |
| | | | | | | |
| +--------+ | | +--------+ |
+-------------------+ +-------------------+
Figure 2: SMC-R Link and RMBsAn SMC-R link can support multiple RMBs, and each peer manages its own RMBs independently 🧁. The number and the size of the RMBs are decided by the peers based on the host's unique memory management requirements. However, the maximum number of RMBs that can be associated with a link group on one peer is 255 🙅♀️. The QP has a single protection domain, but each RMB has its own unique RToken, and all RTokens must be exchanged with the peer 💌.
Each peer manages the RMBs in its local memory for its remote SMC-R peer. It does this by sharing access to the RMBs via RTokens with its peers 🫶. The remote peer writes into the RMBs via RDMA, and then the local peer (the RMB owner) reads from them 📖.
When two peers decide to use SMC-R for a given TCP connection, they each allocate a local RMB element for that connection and communicate the location of this local RMB element during rendezvous processing. So RMB elements are created in pairs, with one RMB element allocated locally on each peer of the SMC-R link 👯♀️.
--- +------------+---------------+
/\ |Eye Catcher | |
| +------------+ |
| | |
RMB Element 1 | |
| | Receive Buffer |
| | |
| | |
\/ | |
--- +------------+---------------+
/\ |Eye Catcher | |
| +------------+ |
| | |
RMB Element 2 | |
| | Receive Buffer |
| | |
| | |
\/ | |
--- +----------------------------+
| . |
| . |
| . |
| . |
| (up to 255 elements) |
+----------------------------+
Figure 3: RMB FormatFigure 3 shows the basic format of an RMB 🎀. The RMB is a virtual memory buffer whose backing real memory is pinned, and it can support up to 255 TCP connections to exactly one remote SMC-R peer 📌. That means each RMB is associated with the SMC-R links within a link group for the two peers and a specific RoCE Protection Domain. Apart from the two peers identified by the SMC-R link, no other SMC-R peers can have RDMA access to an RMB. This requires a unique Protection Domain for every SMC-R link, and this is critical to ensure the integrity of SMC-R communications 🔒.
RMBs get subdivided into multiple elements for efficiency, with each RMB Element (RMBE) associated with a single TCP connection 🍰. So multiple TCP connections across an SMC-R link group can share the same memory for RDMA purposes. This cuts down the overhead of having to register extra memory with the RNIC for every new TCP connection, which is giving efficiency 💅. The number of elements in an RMB and the size of each RMBE are entirely governed by the owning peer, subject to the SMC-R architecture rules. However, all RMB elements within a given RMB must be the same size. Each peer can decide how much resource-sharing it wants across TCP connections based on local constraints, such as available system memory 🧠. An RMB element is identified to the remote SMC-R peer via an RMB Element Token, which consists of the following:
- RMB RToken: The combination of the RKey and virtual address provided by the RNIC that identifies the start of the RMB for RDMA operations.
- RMB Index: Identifies the RMB element index in the RMB. It's used to locate a specific RMB element within an RMB. Valid value range is 1-255.
- RMB Element Length: The length of the RMB element's eye catcher plus the length of the receive buffer. This length is equal for all RMB elements in a given RMB, but it can be variable across different RMBs.
Multiple RMBs can be associated with an SMC-R link group, and each peer in an SMC-R link group manages allocation of its own RMBs. RMB allocation can be asymmetric 🌈. For example, Server X can allocate two RMBs to an SMC-R link group while Server Y allocates five. This gives maximum implementation flexibility, so hosts can optimize RMB management for their own local requirements 🛠️. The maximum number of RMBs that can be allocated on one peer to a link group is 255. If more RMBs are required, the peer may fall back to IP for subsequent connections or, if the peer is the server, create a parallel link group 🦋.
One use case for multiple RMBs is multiple receive buffer sizes. Since every element in an RMB must be the same size, multiple RMBs with different element sizes can be allocated if varying receive buffer sizes are required 📏.
Also, since the maximum number of TCP connections whose receive buffers can be allocated to an RMB is 255, multiple RMBs may be required to provide capacity for large numbers of TCP connections between two peers 🌸.
Separately from the RMB, the TCP/IP stack that owns each RMB maintains control data for each RMB element within its local control structures. The control data contains flags for maintaining the state of the TCP data (for example, an urgent data indicator). Most importantly, it holds the following two cursors, which are illustrated below in Figure 4 👑:
- The peer producer cursor: This is a wrapping offset into the RMB element's receive buffer that points to the next byte of data to be written by the remote peer. This cursor is provided by the remote peer in a Connection Data Control (CDC) message, which is sent using RoCE SendMsg processing, and it tells the local peer how far it can consume data in the RMBE buffer.
- The peer consumer cursor: This is a wrapping offset into the remote peer's RMB element's receive buffer that points to the next byte of data to be consumed by the remote peer in its own RMBE. The local peer cannot write into the remote peer's RMBE beyond this point without causing data loss 😱. This cursor is also provided by the peer using a Connection Data Control message.
Each TCP connection peer maintains its cursors for a TCP connection's RMBE in its local control structures. In other words, the peer who writes into a remote peer's RMBE provides its producer cursor to the peer whose RMBE it has written into. The peer who reads from its RMBE provides its consumer cursor to the writing peer. In this manner, the reads and writes between peers stay coordinated 💞.
For example, referring to Figure 4, Peer B writes the hashed data into the receive buffer of Peer A's RMBE. After that write completes, Peer B uses a CDC message to update its producer cursor to Peer A, to tell Peer A how much data is available to consume. The CDC message that Peer B sends to Peer A wakes up Peer A and notifies it that there's data to be consumed 🔔.
Similarly, when Peer A consumes data written by Peer B, it uses a CDC message to update its consumer cursor to Peer B. That lets Peer B know how much data has been consumed, so Peer B knows how much space is available for further writes. If Peer B were to write enough data to Peer A that it would wrap the RMBE receive buffer and exceed the consumer cursor, data loss would result 🚨.
Note that this is a simplistic description of the control flows. They are optimized to minimize the number of CDC messages required, as described in Section 4.7 ("RMB Data Flows") ✨.
Peer A's RMBE Control Info Peer B's RMBE Control Info
+--------------------------+ +--------------------------+
| | | |
/----Peer producer cursor | +-----+-Peer consumer cursor |
/| | | | |
| +--------------------------+ | +--------------------------+
| Peer A's RMBE |
| +--------------------------+ |
| | +------------------+
| | | |
| | \/ |
| | +------------|
| |-------------+/////////// |
| |//RDMA data written by ///|
| |/// Peer B that is ////// |
| |/available to be consumed/|
| |///////////////////////// |
| |///////// +---------------|
| |----------+/\ |
| | | |
\| | |
\ / |
|\---------/ |
| |
| |
Figure 4: RMBE CursorsAdditional flags and indicators are communicated between peers too 📮. In all cases, these flags and indicators are updated by the peer using CDC messages, which are sent using RoCE SendMsg. More details on these additional flags and indicators are in Section 4.3 ("RMBE Control Information") 💜.
2.2. SMC-R Link Groups ✨
okay bestie, SMC-R links get grouped together into an SMC-R link group 🫶 The whole point is to have multiple links between the same two peers so you get:
- Resilience 🛡️: If a link fails (usually hardware-related), existing TCP connections transparently and dynamically switch to another active link in the group. The TCP traffic on the failing link moves over, so application workloads don't get disrupted. Main character energy 👑
- Link utilization 📈: This is an active/active model, so TCP traffic gets balanced across the links. That boosts bandwidth and avoids hardware imbalances and bottlenecks. ngl, adapter and switch utilization can become resource constraint issues too, so keep an eye on them 👀
SMC-R link group support is required. Resilience is not optional. 💀 However, the user can choose to provision a single RNIC (on one or both hosts), so it's not all-or-nothing.
Multiple links between the same two peers fall into two distinct categories:
- Equal Links 🍓: Links that give equal access to the same RMB(s) at both endpoints. All the TCP connections on these links must have the same VLAN ID and the same TCP server and TCP client roles or relationship.
- Unequal Links 🌸: Links that access unique, unrelated, and isolated RMB(s) (for example, for unique VLANs or unique and isolated application workloads), or links that have unique TCP server or client roles.
Links that are logically grouped together to form an SMC-R link group must be equal links. Period 💅
2.2.1. Link Group Types
Equal links within a link group also have another "Link Group Type" attribute, based on the link's underlying physical path. ✨ There are four SMC-R link types:
- Single link: the only active link within a link group
- Parallel link: not allowed -- SMC-R links having the same physical RNIC at both hosts
- Asymmetric link: links that have unique RNIC adapters at one host but share a single adapter at the peer host
- Symmetric link: links that have unique RNIC adapters at both hosts
Here's how each one plays out in the figures and descriptions below 🦋
Figure 2 above shows the single-link case. The single link in Figure 2 also establishes the SMC-R link group. Link groups are supposed to have multiple links, but when only one RNIC is available at both hosts, only a single link can be created. This is expected to be a transient case 🧚
Figure 5 shows the symmetric-link case, and this one is iconic 🔥 Both hosts have unique and redundant RNIC adapters. This configuration meets the objectives for providing the full RoCE redundancy needed for the level of resilience required for SMC-R high availability. It's not required, but it's a strongly recommended "best practice" for using SMC-R 🫶 Single and asymmetric links must be supported, but they're meant for short-term transient conditions, like a temporary outage or a recycle of an RNIC.
Host X Host Y
+-------------------+ +-------------------+
| | | |
| Protection | | Protection |
| Domain X | | Domain Y |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|RToken X| | |<-------------------->| | | |
| | | | | | |RToken Y|
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| | | | | | |
|| RMB | | | | RMB | |
|| | | | | | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
|RToken Z| | | SMC-R Link 2 | | |RToken W|
| | |RNIC 3|<-------------------->|RNIC 4| | |
| QP 9 | | | | QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+
Figure 5: Symmetric SMC-R Links Host X Host Y
+-------------------+ +-------------------+
| | | |
| Protection | | Protection |
| Domain X | | Domain Y |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|RToken X| | |<-------------------->| | | |
| | | | .->| | |RToken Y|
| \/ +------+ .` +------+ \/ |
|+--------+ | .` | +--------+ |
|| | | .` | | | |
|| RMB | | .` | | RMB | |
|| | | .`SMC-R | | | |
|+--------+ | .` Link 2 | +--------+ |
| /\ +------+ .` +------+ |
|RToken Z| | | .` | |down or |
| | |RNIC 3|<-` |RNIC 4|unavailable |
| QP 9 | | | | |
| +------+ +------+ |
+-------------------+ +-------------------+
Figure 6: Asymmetric SMC-R LinksIn the example in Figure 6, Host X has two RNICs but Host Y only has one, because RNIC 4 isn't available. That setup allows an asymmetric link to be created. An asymmetric link still gives some resilience (for example, when RNIC 1 fails), but ideally each host should have two redundant RNICs 🙈 This should be a transient case. When RNIC 4 becomes available, this configuration must transition to a symmetric-link configuration. The way that happens is you first create the new symmetric link, then delete the asymmetric link using the reason code "Asymmetric link no longer needed" in the DELETE LINK LLC message. Clean handoff, no drama 💖
Host X Host Y
+-------------------+ +-------------------+
| | | |
| Protection | | Protection |
| Domain X | | Domain Y |
| +------+ SMC-R Link 1 +------+ |
| QP 8 |RNIC 1|<-------------------->|RNIC 2| QP 64 |
|RToken X| | | | | | |
| | | |<-------------------->| | |RToken Y|
| \/ +------+ SMC-R Link 2 +------+ \/ |
|+--------+ QP 9 | | QP 65 +--------+ |
|| | | | | | | | |
|| RMB |<-- + | | +---->| RMB | |
|| | | | | | |
|+--------+ | | +--------+ |
| +------+ +------+ |
| down or| | | |down or |
| unavailable|RNIC 3| |RNIC 4|unavailable |
| | | | | |
| +------+ +------+ |
+-------------------+ +-------------------+
Figure 7: SMC-R Parallel Links (Not Supported)Figure 7 shows parallel links, which are two links in the link group that use the same hardware. This configuration is not permitted 🚫 SMC-R multiplexes multiple TCP connections over an SMC-R link, and when both links use the exact same hardware, there's no extra redundancy or capacity benefit. On top of giving you no real benefit, this setup adds the unnecessary overhead of additional queue pairs, generation of additional RKeys, and so on 😭
2.2.2. Maximum Number of Links in Link Group ✨
Okay bestie, so the SMC-R protocol defines a max of eight symmetric SMC-R links in a single link group 🔗. That means up to eight unique physical paths between peer hosts, which is literally iconic. BUT (big but) for the basic redundancy requirements, you must implement support for at least two symmetric links. Supporting more than two also makes life easier when you're dynamically adding and removing links, like spinning up a third link before you take down one of the two existing ones 💅. And remember, all links in a link group need equal access to all associated RMBs 🎀.
The protocol lets each implementation pick its own appropriate value for maximum symmetric links 🌸. That value must not exceed the architecture limit of 8, and it must not be lower than 2, because SMC-R requires redundancy 😭. Doesn't mean you need two RNICs physically to get SMC-R connectivity, but at least two RNICs for redundancy are strongly recommended 🫶.
Peers swap their implementation maximum link values during link group establishment, using the defined maximum link value in the CONFIRM LINK LLC command 💬. Once that initial exchange completes, the value is locked in for the life of the link group 🔒. Both the server and the client can provide a maximum link value. The server must supply one, but the client's value is optional. If the client doesn't supply a value, it's basically saying "yep, I accept the server's max" 🙈. If the client does provide one, it cannot exceed the server-supplied maximum. If the client passes a lower value, that lower value becomes the final negotiated maximum number of symmetric links for this link group 👑. Again, the minimum is 2.
During run time, the client must never request that the server add a symmetric link to a link group that would go over the negotiated maximum link value. Likewise, the server must never attempt to add a symmetric link that would exceed the negotiated maximum value. No going rogue 🚫.
When counting active links in a link group, the initial link (or the only/last link) is always counted as 1 🧮. After that, additional links are either symmetric or asymmetric 🦋.
Asymmetric links are the exception with their own unique rules, so pay attention 👀:
- Asymmetric links are always limited to one asymmetric link allowed per link group.
- Asymmetric links must not be counted in the maximum symmetric-link count calculation. When tracking the current count or enforcing the negotiated maximum number of links, an asymmetric link is not to be counted.
2.2.3. Forming and Managing Link Groups 🧚
SMC-R link groups are self-defining 🪄. The first SMC-R link in a link group is created using TCP option flows on the TCP three-way handshake, followed by CLC message flows over the TCP connection. Subsequent SMC-R links in the link group are created by sending LLC messages over an SMC-R link that already exists in the group 💌. Once an SMC-R link group exists, no additional SMC-R links in that group are created using TCP and CLC negotiation. Since later links are made exclusively by sending LLC messages over an existing SMC-R link, the membership of SMC-R links in a link group is self-defining, which is so clean ✨.
This architecture does not define a specific identifier for an SMC-R link group. Identification may be useful for network management, and it may be assigned in a platform-specific manner or in an extension to this architecture 🍓.
In each SMC-R link group, one peer is the server for all TCP connections and the other peer is the client 🎭. If there are additional TCP connections between the peers that use SMC-R and have the client and server roles reversed, another SMC-R link group is set up between them with the opposite client-server relationship. That's not drama, it's just the rules 💅.
This is required because specific responsibilities are divided between the client and server in managing an SMC-R link group 🗂️.
In this architecture, the decision of whether to use an existing SMC-R link group or create a new one for a TCP connection is made exclusively by the server 👑.
Management of the links in an SMC-R link group is also a server responsibility. The server is responsible for adding and deleting links in a link group. The client may request that the server take certain actions, but the final responsibility is the server's 🫡.
2.2.4. SMC-R Link Identifiers 🪪
This architecture defines multiple identifiers to identify SMC-R links and peers, so let's meet them 👯♀️.
- Link number: This is a 1-byte value that identifies an SMC-R link within a link group. Both the server and the client use this number to tell an SMC-R link apart from other links in the same link group. It is only unique within a link group. To prevent timing windows that can happen when a server creates a new link while the client is still cleaning up a previously existing one, link numbers cannot be reused until the entire link numbering space has been exhausted 🔢.
- Link user ID: This is an architecturally opaque 4-byte value that a peer uses to uniquely define an SMC-R link within its own space. So a link user ID is unique within one peer only. Each peer defines its own link user ID for a link. The peers exchange this information once during link setup, and it is never used architecturally again. Its purpose is network management, display, and debugging 🛠️. For example, an operator on a client could give the operator on the server the server's link user ID if they need the server's operator to check on a link the client is having trouble with 📞.
- Peer ID: The SMC-R peer ID uniquely identifies a specific instance of a specific TCP/IP stack 🆔. It is required because, in clustered and load-balancing environments, an IP address does not uniquely identify a TCP/IP stack. An RNIC's MAC/GID also doesn't uniquely or reliably identify a TCP/IP stack, because RNICs can go up and down and even be redeployed to other TCP/IP stacks in a multiple-partitioned or virtualized environment 🌀. The peer ID is not only unique per TCP/IP stack but also unique per instance of a TCP/IP stack, meaning that if a TCP/IP stack is restarted, its peer ID changes 🔄.
2.3. SMC-R Resilience and Load Balancing 🛡️
The SMC-R multilink architecture provides resilience for network high availability through failover to an alternate RoCE adapter 🦸♀️.
The SMC-R multilink architecture does not define primary, secondary, or alternate roles for the links. Instead, there are multiple active links representing multiple redundant RoCE paths over the same LAN 🌈.
Assignment of TCP connections to links is unidirectional and asymmetric. That means the client and server may each choose a separate link for their RDMA writes associated with a specific TCP connection 🔀.
If a hardware failure occurs, or a QP failure associated with an individual link occurs, then the TCP connections that were associated with the failing link are dynamically and transparently switched to use another available link 🪄. The server or the client can detect a failure, immediately move their TCP connections, and then notify their peer via the DELETE LINK LLC command. While the client can notify the server of an apparent link failure with the DELETE LINK LLC command, the server performs the actual link deletion 🙌.
Moving TCP connections to another link can be accomplished with minimal coordination between the peers 🤝. The move is also transparent to, and non-disruptive to, the TCP socket application workloads for most failure scenarios 💖. After a failure, the surviving links and all associated hardware must handle the link group's workload 💪.
As each SMC-R peer begins moving active TCP connections to another link, all current RDMA write operations must be allowed to complete ⏳. The moving peer then sends a signal to verify receipt of the last successful write by its peer. If this verification fails, the TCP connection must be reset 🚨. Once verification is complete, all writes that failed may then be retried, in order, over the new link 🔁. Any data writes or CDC messages for which the sender did not receive write completion must be replayed before any subsequent data or CDC write operations are sent. LLC messages are not retried over the new link, because they depend on a known link configuration, which has just changed because of the failure 🙅♀️. The initiator of an LLC message exchange that fails will be responsible for retrying once the link group configuration stabilizes 🧘♀️.
When a new link becomes available and is re-added to the link group, each peer is free to rebalance its current TCP connections as needed, or to only assign new TCP connections to the newly added link ⚖️. Both the server and client are free to manage TCP connections across the link group as needed. TCP connection movement does not have to be stimulated by a link failure 🎉.
The SMC-R architecture also defines orderly versus disorderly failover 🧹. The type of failover is communicated in the LLC DELETE LINK command and is simply a means to indicate that the link has terminated (disorderly) or that link termination is imminent (orderly). The orderly link deletion could be initiated via operator command or programmatically to bring down an idle link 🔌. For example, an operator command could start an orderly shutdown of an adapter for service 🛠️. Implementation of the two types is based on implementation requirements and is beyond the scope of the SMC-R architecture 🫶.
3. SMC-R Rendezvous Architecture 🤝
"Rendezvous" is the process that SMC-R-capable peers use to dynamically discover each other's capabilities, negotiate SMC-R connections, set up SMC-R links and link groups, and manage those link groups 🗺️. A key aspect of SMC-R Rendezvous is that it happens dynamically and automatically, without requiring SMC-R link configuration to be defined by an administrator. Zero manual setup, very iconic 💅.
SMC-R Rendezvous starts with the TCP/IP three-way handshake, during which connection peers use TCP options to announce their SMC-R capabilities 📣. If both endpoints are SMC-R capable, then Connection Layer Control (CLC) messages are exchanged between the peers' SMC-R layers over the newly established TCP connection to negotiate SMC-R credentials 🔐. The CLC message mechanism is analogous to the messages exchanged by SSL for its handshake processing 🧠.
If a new SMC-R link is being set up, Link Layer Control (LLC) messages are used to confirm RDMA connectivity ✅. LLC messages are also used by the SMC-R layers at each peer to manage the links and link groups 🗂️.
Once an SMC-R link is set up or agreed to by the peers, the TCP sockets are passed to the peer applications, which use them as normal 🎀. The SMC-R layer, which resides under the sockets layer, transmits the socket data between peers over RDMA using the SMC-R protocol, bypassing the TCP/IP stack. Totally main character energy 🦋✨.
3.1. TCP Options ✨
okay so during the TCP/IP three-way handshake, the client and server say "hey, I speak SMC-R" 🦋 by including experimental TCP option 254 on the handshake flows, following [RFC6994] ("Shared Use of Experimental TCP Options") 💅. The Experiment Identifier (ExID) value is the string "SMCR" in EBCDIC (IBM-1047) encoding, which is 0xE2D4C3D9. This ExID has been registered in the "TCP Experimental Option Experiment Identifiers (TCP ExIDs)" registry maintained by IANA 👑.
After the three-way TCP handshake is done, each peer checks its peer's options 🫶. If both peers set the TCP option on the handshake, inline SMC-R negotiation happens using CLC messages ✨. If neither peer, or only one peer, sets the TCP option, SMC-R cannot be used for that TCP connection, and the connection finishes setup using the IP fabric like normal 🌸.
3.2. Connection Layer Control (CLC) Messages
CLC messages get sent as data payload over the IP network, using the TCP connection between the SMC-R layers at the peers 🧚. They're kinda analogous to the messages used to exchange parameters for SSL 💜.
The next sections go deep on how CLC messages work. Here's the quick summary of the defined CLC messages and what they're for:
- SMC Proposal: Sent from the client to propose that this TCP connection is eligible to be moved to SMC-R. The client identifies itself and its subnet to the server, and passes the SMC-R elements for a suggested RoCE path via the MAC and GID 💖.
- SMC Accept: Sent from the server to accept the client's TCP connection SMC Proposal. The server responds to the client's proposal by identifying itself to the client and passing the elements of a RoCE path that the client can use to perform RDMA writes to the server. This consists of SMC-R link elements such as RoCE MAC, GID, and RMB information 🎀.
- SMC Confirm: Sent from the client to confirm the server's acceptance of the SMC connection. The client responds to the server's acceptance by passing the elements of a RoCE path that the server can use to perform RDMA writes to the client. This consists of SMC-R link elements such as RoCE MAC, GID, and RMB information 🦋.
- SMC Decline: Sent from either the server or the client to reject the SMC connection. It indicates the reason the peer must decline the SMC Proposal, and it lets the TCP connection go back to IP connectivity 💀.
3.3. LLC Messages
Link Layer Control (LLC) messages are sent between peer SMC-R layers over an SMC-R link to manage the link or the link group 🔥. LLC messages are sent using RoCE SendMsg and are 44 bytes long. The 44-byte size is based on what can fit into a RoCE Work Queue Element (WQE) without requiring the posting of receive buffers 🍓.
LLC messages generally follow a request-reply semantic. Each message has a request flavor and a reply flavor, and each request must be confirmed with a reply, except where otherwise noted 🫶. The next sections detail how LLC messages are used. Here's the summary of the defined LLC messages and their purposes:
- ADD LINK: Used to add a new link to a link group. Sent from the server to the client to start adding a new link to the link group, or from the client to the server to ask the server to start that addition 🌈.
- ADD LINK CONTINUATION: A continuation of ADD LINK that lets the ADD LINK span multiple commands, because all of the link information can't fit in one ADD LINK message 📦.
- CONFIRM LINK: Used to confirm that RoCE connectivity over a newly created SMC-R link is working correctly. Initiated by the server. Both this message and its reply must flow over the SMC-R link being confirmed ✅.
- DELETE LINK: When initiated by the server, deletes a specific link from the link group or deletes the entire link group. When initiated by the client, requests that the server delete a specific link or the entire link group 🗑️.
- CONFIRM RKEY: Informs the peer on the SMC-R link of the addition of an RMB to the link group 🔑.
- CONFIRM RKEY CONTINUATION: A continuation of CONFIRM RKEY that lets the CONFIRM RKEY span multiple commands, in case all of the information can't fit in a single CONFIRM RKEY message 🧩.
- DELETE RKEY: Informs the peer on the SMC-R link of the deletion of one or more RMBs from the link group 👋.
- TEST LINK: Verifies that an already-active SMC-R link is active and healthy 💓.
- Optional LLC message: Any LLC message in which the two high-order bits of the opcode are
b'10'. This optional message must be silently discarded by a receiving peer that doesn't support the opcode. No such messages are defined in this version of the architecture, but the concept exists to allow for toleration of possible advanced, optional functions 🌙.
CONFIRM LINK and TEST LINK are picky about which link they flow on. They must flow on the link being confirmed or tested. ⚠️
The other flows may go over any active link in the link group. When there are multiple links in a link group, a response to an LLC message must flow over the same link the original message flowed over, with these exceptions:
- ADD LINK request from a server in response to an ADD LINK from a client.
- DELETE LINK request from a server in response to a DELETE LINK from a client.
3.4. CDC Messages
Connection Data Control (CDC) messages are sent over the RoCE fabric between peers using RoCE SendMsg, and they're 44 bytes long too 🎀. The 44-byte size is based on the size that can fit into a RoCE WQE without requiring the posting of receive buffers. CDC messages describe the socket application data passed via RDMA write operations, plus TCP connection state info like producer cursors and consumer cursors, RMBE state information, and failover data validation 💅.
3.5. Rendezvous Flows
Rendezvous info for SMC-R is exchanged as TCP options on the TCP three-way handshake flows to show capability, and then inline TCP negotiation messages actually do the SMC-R setup 🦋. The formats of all rendezvous options and messages in this section are detailed in Appendix A 📖.
3.5.1. First Contact
First contact between RoCE peers happens when a new SMC-R link group is being set up 🌸. That could be because no SMC-R links exist between the peers yet, or because the server decides to create a new SMC-R link group in parallel with an existing one 👑.
3.5.1.1. Pre-negotiation of TCP Options
The client and server tell each other about their SMC-R capability using TCP option 254 on the TCP three-way handshake flows 🫶.
A client that wants to do SMC-R includes TCP option 254 with an ExID equal to the EBCDIC (codepage IBM-1047) encoding of "SMCR" on its SYN flow 🧚.
A server that supports SMC-R includes TCP option 254 with the ExID value of EBCDIC "SMCR" on its SYN-ACK flow 💜. The server is listening for connections and doesn't know where client connections will come from, so the server implementation may choose to unconditionally include this TCP option if it supports SMC-R. This may be required for server implementations where extensions to the TCP stack aren't practical 🛠️. For server implementations that can add code to examine and react to packets during the three-way handshake, the server should only include the SMC-R TCP option on the SYN-ACK if the client included it on its SYN packet 🔍.
A client that supports SMC-R and meets the three conditions outlined above may optionally include the TCP option for SMC-R on its ACK flow, regardless of whether or not the server included it on its SYN-ACK flow ✨. Some TCP/IP stacks may have to include it if the SMC-R layer can't modify the options on the socket until the three-way handshake completes. Proprietary servers should not include this option on the ACK flow, since including it on the SYN flow was enough to show the client's capabilities 🙈.
Once the initial three-way TCP handshake is complete, each peer examines the socket options 🔎. SMC-R implementations may do this by examining what was actually provided on the SYN and SYN-ACK packets, or by performing a getsockopt() operation to find the options the peer sent 🧐. If neither peer, or only one peer, specified the TCP option for SMC-R, then SMC-R can't be used on this connection, and it proceeds using normal IP flows and processing 🌷.
If both peers specified the TCP option for SMC-R, the TCP connection isn't started yet, and the peers move on to SMC-R negotiation using inline data flows 🦋. The socket isn't handed over to the applications yet. Instead, the respective SMC layers exchange CLC messages over the newly formed TCP connection 💖.
3.5.1.2. Client Proposal ✨
Okay bestie, if both peers support SMC-R, the client sends an SMC Proposal CLC message to the server 🎀. Here's the tea: on this flow, it's not obvious whether this is a new or existing SMC-R link, because in clustered environments one IP address can represent multiple hosts 🧚. That kind of cluster virtual IP can be owned by a network-based or host-based Layer 4 load balancer, which spreads incoming TCP connections across a cluster of servers/hosts. For high availability, other clustered setups may also move a virtual IP dynamically from one host in the cluster to another 🦋.
So the client literally cannot predetermine that a connection is targeting the same host just by matching the destination IP address on outgoing TCP connections 💀. That means it also can't predetermine which SMC-R link a new TCP connection will use. This info gets learned dynamically, and the right actions get taken as the SMC-R negotiation handshake unfolds 🫶.
In the SMC-R proposal message, the initiator (client) proposes SMC-R by including its peer ID, GID, and MAC addresses. It also includes the IP subnet number of the outgoing interface (if IPv4), or the IP prefix list for the network the proposal is sent over (if IPv6) 💅. At this point, the client makes no local commitments of resources for SMC-R 🌸.
When the server receives the SMC Proposal CLC message, it uses the peer ID the client provided, plus the subnet or prefix info, to figure out if it already has a usable SMC-R link with this SMC-R peer 👑. If there are one or more existing SMC-R links with this peer, the server then decides which one to use for this TCP connection. See Sections 3.5.2 and 3.5.3 for the cases of reusing an existing SMC-R link or creating a parallel SMC-R link group between SMC-R peers 🔥.
If this is a first contact between SMC-R peers, the server must validate that it's on the same LAN as the client before continuing 🙈. For IPv4, the server checks that it has an interface whose IP subnet number matches the subnet number the client sent in the SMC Proposal. For IPv6, it checks that it's directly attached to at least one IP prefix the client listed in its SMC Proposal message 🍓.
If the server agrees to use SMC-R, it starts setting up a new SMC-R link by allocating local QP and RMB resources (setting its QP state to INIT). Then it sends its full SMC-R information in an SMC Accept CLC message to the client over the TCP connection, along with a flag set to show this is a first contact flow ✨. The SMC Accept message could travel over any IP route back to the client, depending on Layer 3 IP routing. But the SMC-R credentials it provides must be for the common subnet or prefix between server and client, as determined above 🧠. If the server can't or doesn't want to do SMC-R with the client, it sends an SMC Decline CLC message instead, and the connection data can start flowing over normal TCP/IP flows 🦋.
3.5.1.3. Server Acceptance 💌
When the client gets the SMC Accept from the server, it figures out whether this is a new or existing SMC-R link. It uses a combo of: the first contact flag, its MAC/GID and the MAC/GID the server returned, the VLAN the connection is setting up over, and the QP number the server provided 🔍.
If it's an existing SMC-R link and the client agrees to use that link for the TCP connection, see Section 3.5.2 ("Subsequent Contact") below 💜. If it's a new SMC-R link between peers that already have an SMC-R link, then the server is starting a new SMC-R link group 👯♀️.
Assuming either (1) this is a first contact between peers or (2) the server is starting a new SMC-R link group, the client now allocates local QP and RMB resources for the SMC-R link (setting the QP state to RTR (ready to receive)). It associates them with the server QP it learned from the SMC Accept CLC message, and then sends an SMC Confirm CLC message to the server over the TCP connection, with its SMC-R link info included 🎀. The client also starts a timer to wait for the server to confirm the reliably connected queue pair, as described below ⏰.
3.5.1.4. Client Confirmation 🫶
When the server gets the client's SMC Confirm CLC message, it associates its QP for this SMC-R link with the client's QP, as learned from the SMC Confirm CLC message. Then it sets its QP state to RTS (ready to send) 🚀. And just like that, the client and the server have reliably connected queue pairs 🥹✨.
3.5.1.5. Link (QP) Confirmation 🌈
Okay, this part is kind of a plot twist 😭. Setting up the SMC-R link and its QPs didn't require any network flows on the RoCE fabric, so the client and server still have to confirm connectivity over the RoCE fabric 💅. To do that, the server sends a CONFIRM LINK Link Layer Control (LLC) message to the client over the newly created SMC-R link, using the RoCE fabric. The CONFIRM LINK LLC message provides the server's MAC, GID, and QP information for the connection. It also lets each partner communicate the maximum number of links it can tolerate in this link group (the "link limit"). And it gives two link IDs:
o a 1-byte server-assigned link number that is used by both peers to identify the link within the link group, and it's only unique within a link group 🔢.
o a 4-byte link user ID. This opaque value is assigned by the server for the server's local use, and it's given to the client for management purposes, like for use in network management displays and products 🖥️.
When the server sends this message, it sets a timer to receive confirmation from the client ⏳.
When the client gets the server's CONFIRM LINK LLC message, it cancels the confirmation timer it set when it sent the SMC Confirm message. The client also advances its QP state to RTS and responds over the RoCE fabric with a CONFIRM LINK response LLC message. That message (1) provides its MAC, GID, QP number, and link limit, (2) confirms the 1-byte link number the server sent, and (3) provides its own 4-byte link user ID to the server 💖.
Host X -- Server Host Y -- Client
+-------------------+ +-------------------+
| Peer ID = PS1 | | Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| |RNIC 2| QP 64 |
|RToken X| |MAC MA| |MAC MB| | |
| | |GID GA| |GID GB| |RToken Y|
| \/ +------+ (Subnet S1) +------+ \/ |
|+--------+ | | +--------+ |
|| RMB | | | | RMB | |
|+--------+ | | +--------+ |
| +------+ +------+ |
| |RNIC 3| |RNIC 4| |
| |MAC MC| |MAC MD| |
| |GID GC| |GID GD| |
| +------+ +------+ |
+-------------------+ +-------------------+
SYN TCP options(254,"SMCR")
<---------------------------------------------------------
SYN-ACK TCP options(254,"SMCR")
--------------------------------------------------------->
ACK [TCP options(254,"SMCR")]
<--------------------------------------------------------
SMC Proposal(PC1,MB,GB,S1)
<--------------------------------------------------------
SMC Accept(PS1,first contact,MA,GA,MTU,QP8,RToken=X,RMB elem index)
--------------------------------------------------------->
SMC Confirm(PC1,MB,GB,MTU,QP64,RToken=Y,RMB element index)
<--------------------------------------------------------
CONFIRM LINK(MA,GA,QP8, link lim, server link user ID, linknum)
.........................................................>
CONFIRM LINK rsp(MB,GB,QP64, link lim, client link user ID, linknum)
<........................................................
Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows
Square brackets ("[ ]") indicate optional information
Figure 8: First Contact Rendezvous FlowsTechnically, the data for the TCP connection could now flow over the RoCE path 🤔. BUT, if this is a first contact, there's no alternate for this brand-new RoCE path. Since in the current architecture there's no failover from RoCE to IP once connection data starts flowing, a failure of this path would disrupt the TCP connection 😱. So the level of redundancy and failover would be less than what IP provides. If the network has alternate RoCE paths available, they wouldn't be usable at this point. This situation would be unacceptable 🙅♀️.
3.5.1.6. Second SMC-R Link Setup 🦋
Because of the unacceptable situation above, TCP data is not allowed to flow on the newly established SMC-R link until a second path has been set up, or at least attempted 🛑.
If the server has a second RNIC available on the same LAN, it tries to set up the second SMC-R link over that RNIC. If it only has one RNIC available on the LAN, it tries to set up the second SMC-R link over that one RNIC instead. In the latter case, the server is attempting to set up an asymmetric link, just in case the client does have a second RNIC on the LAN 🤞.
Either way, the server allocates a new QP over the RNIC it's trying to use for the second link and assigns a link number to the new link. It also creates an RToken for the RMB over this second QP. (Note that this means the first and second QP each have their own RToken to represent the same RMB 💅.) The server then sends this info, along with the MAC and GID of the RNIC it's using for the second link, in an ADD LINK LLC message to the client over the SMC-R link that's already set up 🚀.
3.5.1.6.1. Client Processing of ADD LINK LLC Message from Server
okay bestie, so when the client gets the server's ADD LINK LLC message, the first thing it does is check the GID and MAC the server sent 👀. It's figuring out whether the server wants to use the same server-side RNIC as the existing SMC-R link, or a different one 💅.
If it's the same server-side RNIC, the client checks whether it has a second RNIC on the same LAN 🤔. If it doesn't, it rejects the ADD LINK request, because the new link would be a parallel link, and parallel links aren't supported inside a link group 🙅♀️. If the client does have a second RNIC on the same LAN, it accepts, and an asymmetric link gets set up 🦋.
If the server is using a different RNIC from the existing SMC-R link, the client accepts the request and a second SMC-R link gets set up in this same link group ✨. If the client has a second RNIC on the same LAN, that one gets used for the second link, which makes symmetric links 🫶. If it doesn't, it reuses the RNIC from the initial SMC-R link, so you end up with an asymmetric link inside the link group 🌸.
In either case, once the client says yes to the server's ADD LINK request, it allocates a new QP on the chosen RNIC, creates an RKey over that new QP for the client-side RMB of the SMC-R link group, and sends back an ADD LINK reply LLC message. That reply carries all of that info plus the link number the server sent, echoed back 💜.
If the client says no, it sends an ADD LINK reply LLC message with the reason code for the rejection 🙈.
3.5.1.6.2. Server Processing of ADD LINK Reply LLC Message from Client
if the client sends a negative response, or no reply shows up at all, the server frees the RoCE resources it grabbed for the new link 🧹. Having only a single link in an SMC-R link group is not it, honestly 😭. The server's recovery steps live in Appendix C.8 ("Failure to Add Second SMC-R Link to a Link Group") 🎀.
if the client sends a positive reply with its MAC/GID/QP/RKey info, the server ties its QP for the new SMC-R link to the QP the client gave it 🔗. Now the new link is in the same spot the first one was in after the client sent its ACK packet. There's a reliably connected queue pair over the new RoCE path, but no RoCE flows have confirmed it's actually usable yet 🧐. So the client and server swap CONFIRM LINK LLC messages, just like they did on the first SMC-R link 💖.
If either peer hits a failure during this second CONFIRM LINK LLC exchange, whether that's an immediate failure (which means the message never reached the partner) or a timeout, it sends a DELETE LINK LLC message to its partner over the first link, which is now the only link in the group 💀. That DELETE LINK LLC message must be acknowledged before data can flow on the single link in the link group 🛑.
Here's the whole vibe in a picture, after the first link is set up and the second one is being added:
Host X -- Server Host Y -- Client
+-------------------+ +-------------------+
| Peer ID = PS1 | | Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|RToken X| |MAC MA|<-------------------->|MAC MB| | |
| | |GID GA| |GID GB| |RToken Y|
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| | | | | | |
|| RMB | | | | RMB | |
|| | | | | | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
| | |RNIC 3| SMC-R Link 2 |RNIC 4| | |
|RToken Z| |MAC MC|<-------------------->|MAC MD| |RToken W |
| QP 9 |GID GC| (being added) |GID GD| QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+
First SMC-R link setup as shown in Figure 8
<-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.->
ADD LINK request(QP9,MC,GC, link number = 2)
............................................>
ADD LINK response(QP65,MD,GD, link number = 2)
<............................................
ADD LINK CONTINUATION request(RToken=Z)
............................................>
ADD LINK CONTINUATION response(RToken=W)
<............................................
CONFIRM LINK(MC,GC,QP9, link number = 2, link user ID)
.............................................>
CONFIRM LINK response(MD,GD,QP65, link number = 2, link user ID)
<.............................................
Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows
Figure 9: First Contact, Second Link Setup3.5.1.6.3. Exchange of RKeys on Second SMC-R Link
quick note on this one 📝: in the first-contact scenario we're talking about, there's only one RMB RKey to swap on the second SMC-R link, and it gets exchanged in the ADD LINK CONTINUATION request and reply 🔑. But outside first contact, like adding a new SMC-R link to a longstanding link group with multiple RMBs, extra flows are needed to swap the additional RMB RKeys 🌈. For the full scoop on those, check out Section 3.5.5.2.3 ("Adding a New SMC-R Link to a Link Group with Multiple RMBs") ✨.
3.5.1.6.4. Aborting SMC-R and Falling Back to IP
if both partners don't offer the SMC-R TCP option during the three-way TCP handshake, the connection falls back to normal TCP/IP 🌸. Even after that handshake, during the SMC-R negotiation, either partner can break things off by sending an SMC Decline CLC message 🙅♀️. That SMC Decline CLC message may be sent in place of any expected message, and it may also show up during the CONFIRM LINK LLC exchange if something fails before any application data has flowed over the RoCE fabric 💀. For the exact moments an SMC Decline can flow during link group setup, see Appendices C.1 ("SMC Decline during CLC Negotiation") and C.2 ("SMC Decline during LLC Negotiation") 📚.
If this fallback to IP happens while setting up a new SMC-R link group, the RoCE resources allocated for that link group relationship get torn down 🧹. Then it'll be retried as a new SMC-R link group the next time a connection starts between these peers with SMC-R proposed 🦋. If the fallback happens because one side doesn't support SMC-R, there's barely anything to clean up, since the TCP option fails to flow on either the initial SYN or the SYN-ACK before either side reserves any local RoCE resources 🫶.
3.5.2. Subsequent Contact
"Subsequent contact" is when you set up a new TCP connection between two peers who already have an SMC-R link group between them, and you reuse that existing link group 💜. In this case, there's no need to allocate new QPs 🙌. It is possible, though, that a new RMB has been allocated for this TCP connection. That happens if the previous TCP connection used the last element available in the previously used RMB, or for any other implementation-dependent reason 🧩. So for convenience and error checking, the same TCP option 254 gets used, followed by the inline negotiation method described for initial contact. The processing is a little different this time, though, and that's what we'll walk through below 🎀.
3.5.2.1. SMC-R Proposal
when the client starts the inline negotiation with the server, it doesn't yet know whether this is a first contact or a subsequent contact 🤷♀️. It can't know until it sees the server's peer ID, which tells it whether it already has an SMC-R link with this peer that it can use 👀. Using the partner IP address, subnet, VLAN, or other IP info to figure this out just isn't enough, and there are a few reasons why 🙃. The most obvious one is distributed systems: if the server IP is actually a virtual IP representing a whole cluster, the host serving this TCP connection might not be the same host that served the last TCP connection to that IP 🌐.
After the TCP three-way handshake, assuming both partners indicate SMC-R capability, the client builds and sends the SMC Proposal CLC message to the server exactly the same way it does for "first contact" 💅. At this point it still doesn't know if it's a first or subsequent contact. Just like in first contact, the client sends its peer ID value, its suggested RNIC MAC/GID, and its IP subnet or prefix information 💖.
When the server gets the client's proposal, it looks up the provided peer ID to see if it already has a usable SMC-R link group with this peer 🔍. If it does, the server needs to decide whether to use the existing SMC-R link group or create a new one. For the new link group case, see Section 3.5.3 ("First Contact Variation: Creating a Parallel Link Group") below 🌸.
For the rest of this discussion, let's assume the server decides to use the existing SMC-R link group for the TCP connection, which is expected to be the most common case 👑. The server is the one making that call. It then has to tell the client, but it doesn't need to allocate, associate, and confirm any QPs for the chosen SMC-R link 🙌. All that's left is setting up RMB space for this TCP connection 🧚.
If one of the RMBs already in use for this SMC-R link group has an available element with the right buffer size, the server just picks that one for this TCP connection. Then it sends an SMC Accept CLC message with the full RoCE information for the chosen SMC-R link to the client, using the same format as the SMC Accept CLC message from Section 3.5.1 ("First Contact") above 🦋.
The server can choose to use the SMC-R link that matches the suggested MAC/GID the client put in the SMC Proposal for its RDMA writes, but it's not obligated to 🫶. The final decision on which specific SMC-R link a TCP connection gets assigned to is an independent decision made by the server and the client 💜.
Sometimes the server does need to allocate a new RMB for this connection ✨. The reasons for that are implementation dependent, and they could include:
- no available space in existing RMB or RMBs, or
- desire to allocate a new RMB that uses a different buffer size from the ones already created, or
- any other implementation-dependent reason
In that case, the server allocates the new RMB and then performs the flows described in Section 3.5.5.2.1 ("Adding a New RMB to an SMC-R Link Group") 🛠️. Once all that processing is done, the server provides the full RoCE information, including the new RKey, for this connection in an SMC Confirm CLC message to the client 🎉.
3.5.2.2. SMC-R Acceptance ✨
Okay bestie, when the client gets the SMC Accept CLC message from the server 💌, it checks the RoCE info the server sent to figure out whether this is a first contact for a brand new SMC-R link group or a subsequent contact on one that already exists 👀. It's a subsequent contact if the server-side peer ID, GID, MAC, and QP number in the packet all match a known SMC-R link, and the first contact flag is not set 🙅♀️. If that's not the case (like, the GID and MAC match but the QP is new), then the server is basically spinning up a new, parallel SMC-R link group, and that counts as a first contact 🧚.
A different RMB RToken does not mean first contact, because the server might have made a new RMB or might be using several RMBs for this SMC-R link 🎀. The client only needs the server's RMB info for its RDMA writes to the server. Since there's no requirement for symmetric RMBs, this info is just control information for the RDMA writes on this SMC-R link 💅.
The client must check that the RMB element the server is offering isn't already being used by another TCP connection on this SMC-R link group. This check has to cover the new <rtoken, index> against every known <rtoken, index> on the link group 🔍. For the case where the server tries to use an RMB element that's already in use on this link group, see Section 4.4.2 ("RMB Element Reuse and Conflict Resolution") 📖.
Once the client has figured out that this TCP connection is a subsequent contact over an existing SMC-R link, it does an RMB allocation process that's pretty similar to what the server did. It either (1) grabs an element from an RMB already attached to this SMC-R link, or (2) allocates a new RMB, attaches it to this SMC-R link, and then picks an element out of it 🦋.
If the client allocates a new RMB for this TCP connection, it goes through the processing in Section 3.5.5.2.1 ("Adding a New RMB to an SMC-R Link Group") 🌸. Once that's done, the client sends its full RoCE info for this TCP connection in an SMC Confirm CLC message 🫶.
Because an SMC-R link with a verified connected QP already exists and is getting reused, there's no need for the verification or alternate QP selection flows or timers. It's giving efficiency 🔥.
3.5.2.3. SMC-R Confirmation 💜
When the server gets the client's SMC Confirm CLC message on a subsequent contact, it checks the following:
- The RMB element the client gave is not already being used by another TCP connection on this SMC-R link group. (For the case where it is, see Section 4.4.2 ("RMB Element Reuse and Conflict Resolution").)
- The MAC/GID/QP info from the client matches an active link inside the link group. The client is free to pick any valid/active link, and it is not required to pick the same link as the server 🦋.
If that validation passes, the server stores the client's RMB info for this connection, and the RoCE setup for the TCP connection is complete 🎉✨.
3.5.2.4. TCP Data Flow Race with SMC Confirm CLC Message 🏎️
On a subsequent contact TCP/IP connection, a peer can send data as soon as it has the peer RMB info for the connection 💨. There are no extra RoCE confirmation flows, since the QPs on the SMC-R link are already reliably connected and verified ✅.
In most cases, the first data flows from the client to the server 🙌. The client must send the SMC Confirm CLC message before it sends any connection data over the chosen SMC-R link. However, the client doesn't need to wait for confirmation of this message, and honestly there won't be any confirmation at all 🙈. Since the server is required to have the RMB fully set up and ready to receive data from the client before it sends an SMC Accept CLC message, the client can start sending data over the SMC-R link right after it finishes sending the SMC Confirm CLC message 🚀.
It's possible that data from the client reaches the server-side RMB before the server has processed the client's SMC Confirm CLC message. When that happens, the server must handle this race condition. It must not hand the arrived TCP data to the socket application until the SMC Confirm CLC message has been received and fully processed, which is when the socket opens 🚪.
If the server has initial data to send to the client that isn't a response to the client (this case should be rare), it can send the data right after it receives and processes the SMC Confirm CLC message from the client 📨. The client must have opened the TCP socket to the client application when it sent the SMC Confirm CLC message, so the client will be ready to process data from the server 💖.
3.5.3. First Contact Variation: Creating a Parallel Link Group 🧬
Quick reminder: parallel SMC-R links within an SMC-R link group are not supported. These are multiple SMC-R links in a link group that use the same network path 🚫. However, multiple SMC-R link groups between the same peers are supported. So if you want multiple SMC-R links over the same RoCE path, you need to use multiple SMC-R link groups 🌈. It's not a recommended practice, but it could be done for platform-specific reasons, like separating QPs for different workloads 🤷♀️. Only the server can drive the creation of multiple SMC-R link groups between peers 👑.
At a high level, when the server decides to create an extra SMC-R link group with a client it already has an SMC-R link group with, the flows are basically the same as the normal "first contact" case above. The text below gives more detail and clarification on how this case gets processed 🔍.
When the server gets the SMC Proposal CLC message from the client, and uses the MAC/GID info to figure out it already has an SMC-R link group with this client, the server can either reuse the existing SMC-R link group (see Section 3.5.2 ("Subsequent Contact") above) or create a new SMC-R link group alongside the existing one 🧩.
If the server decides to create a new SMC-R link group, it does the same processing it would have done for first contact. It allocates QP and RMB resources, plus alternate QP resources, and sends the QP and RMB info to the client in the SMC Accept CLC message with the first contact flag set 📬.
When the client gets the server's SMC Accept CLC message with the new QP info and the first contact flag set, it knows the server is making a new SMC-R link group even though they already share one 😳. In this case the client also allocates a new QP for this new SMC-R link, allocates an RMB for it, and generates an RKey for it 🔑.
Note that multiple SMC-R link groups between the same peers must access different RMB resources, so new RMBs will be required. Using an RMB that's already in use in another SMC-R link group is not permitted 🙅♀️.
The client then associates its new QP with the server's new QP, sends its SMC Confirm CLC message back to the server with the new QP/RMB info, and sets its confirmation timer for the new SMC-R link ⏰.
When the server gets the client's SMC Confirm CLC message, it associates its QP with the client's QP as learned from that message, and then sends a confirmation LLC message 💌. The rest of the flow, including the confirmation QP and the setup of additional SMC-R links, unfolds just like the "first contact" case 💅✨.
3.5.4. Normal SMC-R Link Termination 🌙
The normal socket API trigger points are what the SMC-R layer uses to start SMC-R connection termination flows 🔌. The main design idea for normal SMC-R connection flows is to use the SMC-R protocol to first shut down the SMC-R connection and free up any SMC-R RDMA resources. Then the normal TCP connection termination protocol (that's FIN processing) drives cleanup of the TCP connection on the IP fabric 🧹. This matters a lot, because it makes sure RDMA resources like the RMBEs only get freed and reused once both SMC-R endpoints are completely done with their RDMA write operations to the partner's RMBE 🛡️.
When the last TCP connection over an SMC-R link group terminates, the link group can be terminated. Just like creating SMC-R links and link groups, the main responsibility for deciding normal termination is needed and starting it belongs to the server 👑.
Implementations may choose to set timers that keep SMC-R link groups up for a set time after the last TCP connection ends. This helps avoid churn when TCP connections come and go a lot 🔄.
A link or link group can also be terminated because an operator ran a command. That command can be entered at either the client or the server 💻. If it's entered at the client, the client asks the server to terminate the link or link group, and the responsibility for actually doing it ultimately lies with the server 🙏.
When the server decides the SMC-R link group should be terminated, it sends a DELETE LINK LLC message to the client, with a flag set that says all links in the link group are to be terminated 🛑. After the adapter confirms that the DELETE LINK LLC message has been sent, the server can clean up its end of the link group (QPs, RMBs, etc.) 🧼. When the client gets the DELETE LINK message from the server, it must immediately comply and clean up its end of the link group. Any TCP connections the client thinks are still active on the link group must be terminated right away 💥.
The client can also ask the server to delete the link group. It does this by sending a DELETE LINK message to the server, saying that cleanup of all links is requested 🥺. The server must comply by sending a DELETE LINK to the client and processing it as described in the paragraph above. If there are TCP connections still active on the link group when the server gets this request, they are immediately terminated by sending a RST flow over the IP fabric 🚨.
3.5.5. Link Group Management Flows 🗂️
3.5.5.1. Adding and Deleting Links in an SMC-R Link Group 🔗
The server is the one in charge of how the link group is put together 👑. Links are added to the link group by the server. The client can let the server know about new conditions that might lead the server to add a new link, but the server is ultimately responsible 📣. In general, links are deleted from the link group by the server. However, in certain error cases, the client can tell the server that a link must be deleted and treat it as deleted without waiting for the server to act 😤. The flows are detailed in the sections that follow 📖✨.
3.5.5.1.1. Server-Initiated ADD LINK Processing ✨
okay so we already covered this, but the server starts an ADD LINK exchange to create redundancy in a brand new link group 💖. Once a link group is set up, the server can also kick off ADD LINK for other reasons, like:
- More resources showing up on the server host 🔥 to support an extra SMC-R link. That could be a new RNIC getting provisioned, more storage opening up for QP resources, an operator command, or any other implementation-dependent reason 🧚. Quick heads up: for an existing link group, a new RNIC has to be attached to the same RoCE LAN that the link group is using 🫶.
- The client telling the server it has extra resources 💅 to support an extra SMC-R link. See Section 3.5.5.1.2 ("Client-Initiated ADD LINK Processing").
Server-initiated ADD LINK processing in an established SMC-R link group is the same as the ADD LINK processing in Section 3.5.1.6 ("Second SMC-R Link Setup"), with a couple of twists 🦋:
- If an asymmetric SMC-R link already exists in the link group, a second asymmetric link will not be created. Only one asymmetric link is permitted in a link group 🙈.
- TCP data flow on the already-existing link(s) in the link group is not halted or otherwise affected while the extra link is being set up. No interruptions, bestie 🌸.
The server will not initiate ADD LINK processing if the link group already has the maximum number of links negotiated by the partners 💀.
3.5.5.1.2. Client-Initiated ADD LINK Processing
So if an extra RNIC shows up for an existing SMC-R link group on the client's side, the client lets the server know by sending an ADD LINK request LLC message 📩. This one is different from the server's ADD LINK request, because it just tells the server "hey, I've got a new RNIC" 💁♀️. If the link group has no redundancy, or only has redundancy on an asymmetric link with a single RNIC on the client side, the server must initiate an ADD LINK exchange in response, to create or improve the link group's redundancy 🔥.
If the link group already has symmetric-link redundancy but has fewer than the negotiated maximum number of links, the server may respond by starting an ADD LINK exchange to create a new link using the client's new resource. But it is not required to do so 🤷♀️.
If the link group already has the negotiated maximum number of links, the server must ignore the client's ADD LINK request LLC message 🙅♀️.
Because the server is not required to respond to the client's ADD LINK LLC message in all cases, the client must not wait for a response or throw an error if one does not come 🫶. Chill, client, it's not personal 😭.
3.5.5.1.3. Server-Initiated DELETE LINK Processing
Here are the reasons a server may delete a link 💅:
- The link has not been used for TCP connections for an implementation-defined time interval, and deleting the link will not cause the link group to lack redundancy 💤.
- Errors in resources supporting the link occur. These errors may include, but are not limited to, RNIC errors, QP errors, and software errors 🚨.
- The RNIC supporting this SMC-R link is being taken down, either because of an error case or because of an operator or software command 🛑.
If a link being deleted is supporting TCP connections and there are one or more surviving links in the link group, the TCP connections get moved to the surviving links 🦋. For more info on how that works, check out Section 2.3 ("SMC-R Resilience and Load Balancing").
The server deletes a link from the link group by sending a DELETE LINK request LLC message to the client over any of the usable links in the link group. Since the DELETE LINK LLC message specifies which link is being deleted, it may flow over any link in the link group 🧠. The server must not clean up its RoCE resources for the link until the client responds ⏳.
The client responds to the server's DELETE LINK request LLC message by sending the server a DELETE LINK response LLC message. The client must respond positively. It cannot decline to delete the link 🙅♀️, no drama.
Once the server has received the client's DELETE LINK response, both sides may clean up their resources for the link ✨.
Either a positive write completion or some other indication from the RNIC on the client's side is sufficient to indicate to the client that the server has received the DELETE LINK response. 💜
Host X Host Y
+-------------------+ +-------------------+
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 9 |
|RToken X| |Failed|<--X----X----X----X-->| | |
| | | | | | |
| \/ +------+ +------+ |
|+--------+ | | |
|| Deleted| | | |
|| RMB | | | |
|| | | | |
|+--------+ | | |
| /\ +------+ +------+ |
|RToken Z| | | SMC-R Link 2 | | |
| | |RNIC 3|<-------------------->|RNIC 4| |
| QP 64| | | | QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+
DELETE LINK(request, link number = 1,
................................................>
reason code = RNIC failure)
DELETE LINK(response, link number = 1)
<................................................
(Note: Architecturally, this exchange can flow over either
SMC-R link but most likely flows over Link 2, since
the RNIC for Link 1 has failed.)
Figure 10: Server-Initiated DELETE LINK Flow3.5.5.1.4. Client-Initiated DELETE LINK Request
The client may request that the server delete a link for the same reasons the server may delete a link, except for inactivity timeout 🙃.
Since the client depends on the server to actually delete links, there are two flavours of delete requests from client to server 🍓:
- Orderly: The client is asking the server to delete the link when able. This would happen after an operator command to bring down the RNIC, or some other nonfatal reason 😌. In this case, the server is required to delete the link, but it may not do it right away ⏰.
- Disorderly: The server must delete the link right away, because the client has experienced a fatal error with the link 🚨😭.
Either way, the server responds by starting a DELETE LINK exchange with the client, just like in the previous section 🔁. The only difference is whether the server must do it immediately or can wait for a chance to gracefully delete the link 💖.
Host X Host Y
+-------------------+ +-------------------+
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 9 |
|RToken X| | |<---X--X--X--X--X--X->|Failed| |
| | | | | | |
| \/ +------+ +------+ |
|+--------+ | | |
|| Deleted| | | |
|| RMB | | | |
|| | | | |
|+--------+ | | |
| /\ +------+ +------+ |
|RToken Z| | | SMC-R Link 2 | | |
| | |RNIC 3|<-------------------->|RNIC 4| |
| QP 64| | | | QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+
DELETE LINK(request, link number = 1, disorderly,
<...............................................
reason code = RNIC failure)
DELETE LINK(request, link number = 1,
................................................>
reason code = RNIC failure)
DELETE LINK(response, link number = 1)
<................................................
(Note: Architecturally, this exchange can flow over either
SMC-R link but most likely flows over Link 2, since
the RNIC for Link 1 has failed.)
Figure 11: Client-Initiated DELETE LINK Flow3.5.5.2. Managing Multiple RKeys over Multiple SMC-R Links in a Link Group 🎀
After the initial contact sequence finishes and the number of TCP connections goes up, the SMC peers could end up adding more RMBs to the link group 🌈. Remember that each peer manages its RMBs independently 🙌. Also remember that an RMB's RToken is specific to a QP. So when there are multiple SMC-R links in a link group, every RMB accessed within the link group needs a separate RToken for each SMC-R link in the group 🧩.
Each RMB that is added to a link must be added to all links within the link group 💜. The set of RMBs created for the link is called the "RToken set" 🦋. The RTokens must be exchanged with the peer. As RMBs are added and deleted, the RToken set must remain in sync 🫶✨.
3.5.5.2.1. Adding a New RMB to an SMC-R Link Group ✨
okay so a new RMB can be added to an SMC-R link group from either the client side or the server side 🎀 when you add an extra RMB to a link group that already exists, that RMB has to be tied to the QPs on every link in the group. so when a new RMB shows up, its RMB RToken for each SMC-R link's QP has to be sent to the peer 💌
the RTokens for a new RMB get sent using CONFIRM RKEY LLC messages, like in Figure 12 🦋 the RToken set is written as pairs: an SMC-R link number, plus the new RMB's RToken over that link. and here's the failover part, bestie 🫶 to keep failover working, any TCP connection that uses the new RMB cannot go active until every RToken for that RMB has been shared for every link in the group 🔐
Host X Host Y
+-------------------+ +-------------------+
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 9 |
|RToken X| | |<-------------------->| | |
| | | | | | |
| \/ +------+ +------+ |
|+--------+ | | |
|| New | | | |
|| RMB | | | |
|| | | | |
|+--------+ | | |
| /\ +------+ +------+ |
|RToken Z| | | SMC-R Link 2 | | |
| | |RNIC 3|<-------------------->|RNIC 4| |
| QP 64| | | | QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+
CONFIRM RKEY(request, Add,
................................................>
RToken set((Link 1,RToken X),(Link 2,RToken Z)))
CONFIRM RKEY(response, Add,
<................................................
RToken set((Link 1,RToken X),(Link 2,RToken Z)))
(Note: This exchange can flow over either SMC-R link.)
Figure 12: Add RMB to Existing Link GroupImplementations may choose to add RMBs to a link group early, before they're actually needed 🔮 for example, an implementation might add a new RMB once a usage threshold (e.g., percentage used) has been passed for all of its existing RMBs 📈
a new RMB can also get added on an as-needed basis 💅 like when a new TCP connection joins the link group but there are no RMB elements free. in that case the CLC exchange gets paused while the peer that needs the new RMB adds it 😭 Figure 13 shows how that goes 👇
Host X -- Server Host Y -- Client
+-------------------+ +--------------------+
| Peer ID = PS1 | | Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|RToken X| |MAC MA|<-------------------->|MAC MB| | |
| | |GID GA| |GID GB| |RToken Y2|
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| | | Subnet S1 | | New | |
|| RMB | | | | RMB | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
| | |RNIC 3| SMC-R Link 2 |RNIC 4| |RToken W2|
| | |MAC MC|<-------------------->|MAC MD| | |
| QP 9 |GID GC| |GID GD| QP 65 |
| +------+ +------+ |
+-------------------+ +--------------------+
SYN / SYN-ACK / ACK TCP three-way handshake with TCP option
<--------------------------------------------------------->
SMC Proposal(PC1,MB,GB,S1)
<--------------------------------------------------------
SMC Accept(PS1,not 1st contact,MA,GA,QP8,RToken=X,RMB elem index)
--------------------------------------------------------->
CONFIRM RKEY(request, Add,
<........................................................
RToken set((Link 1,RToken Y2),(Link 2,RToken W2)))
CONFIRM RKEY(response, Add,
........................................................>
RToken set((Link 1,RToken Y2),(Link 2,RToken W2)))
SMC Confirm(PC1,MB,GB,QP64,RToken=Y2, RMB element index)
<--------------------------------------------------------
Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows
Figure 13: Client Adds RMB during TCP Connection Setup3.5.5.2.2. Deleting an RMB from an SMC-R Link Group 🗑️
either peer can delete one or more of its own RMBs, as long as no TCP connection is using them 🙅♀️ ideally, an SMC-R peer would run a timer so it doesn't free an RMB the second the last TCP connection leaves it 🕰️ that way the RMB stays around for the next TCP connection and the peers don't keep adding and deleting RMBs over and over (thrashing is not the vibe 😤). once a peer decides to delete an RMB, it sends a DELETE RKEY LLC message to its peer, and it can only free the RMB after it gets a response back 💜 you can delete multiple RMBs in one DELETE RKEY exchange 🍓
quick note, bestie 💁♀️ the DELETE RKEY message doesn't need the full RToken for the deleted RMB. the RMB's RKey over just one link in the group is enough to say which RMB is getting deleted 🔑
Host X Host Y
+-------------------+ +-------------------+
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 9 |
|RToken X| | |<-------------------->| | |
| | | | | | |
| \/ +------+ +------+ |
|+--------+ | | |
|| Deleted| | | |
|| RMB | | | |
|| | | | |
|+--------+ | | |
| /\ +------+ +------+ |
|RToken Z| | | SMC-R Link 2 | | |
| | |RNIC 3|<-------------------->|RNIC 4| |
| QP 9 | | | | |
| +------+ +------+ |
+-------------------+ +-------------------+
DELETE RKEY(request, RKey list(RKey X))
................................................>
DELETE RKEY(response, RKey list(RKey X))
<................................................
(Note: This exchange can flow over either SMC-R link.)
Figure 14: Delete RMB from SMC-R Link Group3.5.5.2.3. Adding a New SMC-R Link to a Link Group with Multiple RMBs 🌈
okay, this one's a bit of a plot twist 😳 when a new SMC-R link joins a link group that already has a bunch of RMBs on each side, there could be several RMBs already tied to the group. and the count can be different on each side, because each peer manages its own RMBs independently 🙃 every one of those RMBs needs a brand-new RToken for the new link, and those have to be shared with the peer. so it's two-way traffic: the server sends its RTokens to the client, and the client sends its RTokens back 🔁
RTokens travel between peers in pairs 👯 each pair has two things:
- The RToken for the RMB, as it's already known on an existing SMC-R link in the link group.
- The RToken for that same RMB, to be used on the new SMC-R link.
the pairs matter because they tell each peer which RTokens across QPs are the same thing 🧩
the ADD LINK request and response LLC messages don't have room for any RToken pairs 🙈 so ADD LINK CONTINUATION LLC messages carry them instead, as shown in Figure 15 💫 the ADD LINK CONTINUATION LLC messages go over the same SMC-R link as the ADD LINK LLC messages did. in both the ADD LINK and the ADD LINK CONTINUATION messages, the first RToken in each pair is the RToken for the RMB as it's known on the SMC-R link that the LLC message is being sent over ✨
Host X -- Server Host Y -- Client
+-------------------+ +-------------------+
| Peer ID = PS1 | | Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|RKey set| |MAC MA|<-------------------->|MAC MB| |RKey set|
|X,Y,Z | |GID GA| |GID GB| |Q,R,S,T |
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| 3 RMBs | | | | 4 RMBs | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
|RKey set| |RNIC 3| SMC-R Link 2 |RNIC 4| | RKey set|
|U,V,W | |MAC MC|<-------------------->|MAC MD| | L,M,N,P |
| QP 9 |GID GC| (being added) |GID GD| QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+
ADD LINK request (QP9,MC,GC, link number = 2)
............................................>
ADD LINK response (QP65,MD,GD, link number = 2)
<............................................
ADD LINK CONTINUATION req(RToken pairs=((X,U),(Y,V),(Z,W)))
............................................>
ADD LINK CONTINUATION rsp(RToken pairs=((Q,L),(R,M),(S,N),(T,P)))
<.............................................
CONFIRM LINK req/rsp exchange on Link 2
<.............................................>
Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows
Figure 15: Exchanging RKeys when a New Link Is Added to a Link Group3.5.5.3. Serialization of LLC Exchanges, and Collisions ✨
okay so LLC flows can be split into two main groups for serialization purposes 💅
The first group is LLC messages that are independent and can flow at any time 🦋 These are one-time, unsolicited messages that either don't have a required response or have a simple response that doesn't interfere with the operations of another group of messages. The messages in this group are:
TEST LINKfrom either the client or the server: this message requires aTEST LINKresponse to be returned but doesn't affect the configuration of the link group or the RKeys.ADD LINKfrom the client to the server: this is an "FYI" to the server saying the client has an additional RNIC available 🎀 The server is not required to act upon or respond to this message.DELETE LINKfrom the client to the server: this tells the server that either (1) the client ran into an error or problem that requires a link or link group to be terminated, or (2) an operator has commanded that a link or link group be terminated. The server doesn't respond directly to it. Instead, it starts aDELETE LINKexchange because of it 💀DELETE LINKfrom the server to the client, with the "delete entire link group" flag set: this tells the client that the entire link group is being deleted 😭
The second group is LLC messages that are part of an exchange affecting link group configuration. This exchange has to complete before another exchange that affects link group configuration can be processed 👑 When a peer knows one of these exchanges is in progress, it must not start another exchange. The exchanges in this group are:
ADD LINK/ADD LINKresponse /ADD LINK CONTINUATION/ADD LINK CONTINUATIONresponse /CONFIRM LINK/CONFIRM LINKresponse: this exchange adds a new link, which changes the configuration of the link group.DELETE LINK/DELETE LINKresponse initiated by the server, without the "delete entire link group" flag set: this exchange deletes a link, which changes the configuration of the link group.CONFIRM RKEY/CONFIRM RKEYresponse orDELETE RKEY/DELETE RKEYresponse: this exchange changes the RMB configuration of the link group. RKeys can't change while links are being added or deleted (while anADD LINKorDELETE LINKis in progress). BUTCONFIRM RKEYandDELETE RKEYare special 🌈 Both the client and the server can independently manage (add or remove) their own RMBs. That means each peer can change their RKeys at the same time, and therefore sendCONFIRM RKEYorDELETE RKEYrequests concurrently. Those concurrent requests can be processed independently and aren't a collision ✨
Because the server is in control of the configuration of the link group, a lot of timing windows and collisions are avoided 🙌 There are still some that have to be handled though, so let's go through them 🧚
3.5.5.3.1. Collisions with ADD LINK / CONFIRM LINK Exchange 🌸
- Colliding LLC message:
TEST LINK- Action to resolve: Send immediate
TEST LINKreply.
- Action to resolve: Send immediate
- Colliding LLC message:
ADD LINKfrom client to server- Action to resolve: Server ignores the
ADD LINKmessage. When the client receives the server'sADD LINK, the client will consider that message to be in response to itsADD LINKmessage, and the flow works 🫶 Since both client and server know not to start this exchange if anADD LINKoperation is already underway, this can only happen if the client sends this message before receiving the server'sADD LINK, and this message crosses with the server'sADD LINKmessage. Therefore, the server'sADD LINKarrives at the client immediately after the client sent this message.
- Action to resolve: Server ignores the
- Colliding LLC message:
DELETE LINKfrom client to server, specific link specified- Action to resolve: Server queues the
DELETE LINKmessage and processes it after theADD LINKexchange completes. If it's an orderly link termination, it can wait until after this exchange continues. If it's disorderly and the link affected is the one the current exchange is using, the server will discover the outage when a message in this exchange fails 😬
- Action to resolve: Server queues the
- Colliding LLC message:
DELETE LINKfrom client to server, entire link group to be deleted- Action to resolve: Immediately clean up the link group. 🧹
- Colliding LLC message:
CONFIRM RKEYfrom client- Action to resolve: Send a negative
CONFIRM RKEYresponse to the client. Once the current exchange finishes, the client will have to recompute its RKey set to include the new link and then start a newCONFIRM RKEYexchange 🔁
- Action to resolve: Send a negative
3.5.5.3.2. Collisions during DELETE LINK Exchange 💅
- Colliding LLC message:
TEST LINKfrom either peer- Action to resolve: Send immediate
TEST LINKresponse.
- Action to resolve: Send immediate
- Colliding LLC message:
ADD LINKfrom client to server- Action to resolve: Server queues the
ADD LINKand processes it after the current exchange completes.
- Action to resolve: Server queues the
- Colliding LLC message:
DELETE LINKfrom client to server (specific link)- Action to resolve: Server queues the
DELETE LINKmessage and processes it after the current exchange completes. If it's an orderly link termination, it can wait until after this exchange continues. If it's disorderly and the link affected is the one the current exchange is using, the server will discover the outage when a message in this exchange fails 🥲
- Action to resolve: Server queues the
- Colliding LLC message:
DELETE LINKfrom either client or server, deleting the entire link group- Action to resolve: Immediately clean up the link group. 🧹
- Colliding LLC message:
CONFIRM RKEYfrom client to server- Action to resolve: Send a negative
CONFIRM RKEYresponse to the client. Once the current exchange finishes, the client will have to recompute its RKey set to include the new link and then start a newCONFIRM RKEYexchange 🔁
- Action to resolve: Send a negative
3.5.5.3.3. Collisions during CONFIRM RKEY Exchange 🦋
- Colliding LLC message:
TEST LINK- Action to resolve: Send immediate
TEST LINKreply.
- Action to resolve: Send immediate
- Colliding LLC message:
ADD LINKfrom client to server- Action to resolve: Queue the
ADD LINK, and process it after the current exchange completes.
- Action to resolve: Queue the
- Colliding LLC message:
ADD LINKfrom server to client (CONFIRM RKEYexchange was initiated by the client, and it crossed with the server initiating anADD LINKexchange)- Action to resolve: Process the
ADD LINK. The client will receive a negativeCONFIRM RKEYfrom the server and will have to redo thisCONFIRM RKEYexchange after theADD LINKexchange completes 🙃
- Action to resolve: Process the
- Colliding LLC message:
DELETE LINKfrom client to server, specific link to be deleted (CONFIRM RKEYexchange was initiated by the server, and it crossed with the client'sDELETE LINKrequest)- Action to resolve: Server queues the
DELETE LINKmessage and processes it after theCONFIRM RKEYexchange completes. If it's an orderly link termination, it can wait until after this exchange continues. If it's disorderly and the link affected is the one the current exchange is using, the server will discover the outage when a message in this exchange fails 😬
- Action to resolve: Server queues the
- Colliding LLC message:
DELETE LINKfrom server to client, specific link deleted (CONFIRM RKEYexchange was initiated by the client, and it crossed with the server'sDELETE LINK)- Action to resolve: Process the
DELETE LINK. The client will receive a negativeCONFIRM RKEYfrom the server and will have to redo thisCONFIRM RKEYexchange after theADD LINKexchange completes 🤔
- Action to resolve: Process the
- Colliding LLC message:
DELETE LINKfrom either client or server, entire link group deleted- Action to resolve: Immediately clean up the link group. 🧹
- Colliding LLC message:
CONFIRM LINKfrom the peer that did not start the currentCONFIRM LINKexchange- Action to resolve: Queue the request, and process it after the current exchange completes.
4. SMC-R Memory-Sharing Architecture 🧠✨
4.1. RMB Element Allocation Considerations 🎀
okay so every TCP connection using SMC-R has to be allocated an RMBE by each SMC-R peer 💖 Each endpoint does this allocation independently, so each one can pick the RMBE that best matches the characteristics of its own TCP socket endpoint 🦋 The RMBE tied to a TCP socket endpoint has to have a receive buffer at least as large as the TCP receive buffer size in effect for that connection. That size can be set explicitly by the application using setsockopt(), or implicitly through the system-configured default value 🌈 This lets the SMC-R peer RDMA-write enough data to fill an entire receive buffer's worth of data on a given data flow 🙌 Since each RMB has to have fixed-length RMBEs, an SMC-R endpoint may need to maintain multiple RMBs of various sizes for SMC-R connections on a given SMC-R link. Then it can pick the RMBE that fits a connection most closely ✨
4.2. RMB and RMBE Format 💜
An RMB is a virtual memory buffer whose backing real memory is pinned 📌 The RMB gets subdivided into a whole number of equal-sized RMB Elements (RMBEs). Each RMBE starts with a 4-byte eye catcher for diagnostic and service purposes, followed by the receive data buffer. The contents of this diagnostic eye catcher are implementation dependent. The local SMC-R peer should use it to check for overlay errors by verifying an intact eye catcher with every RMBE access 🔍
The RMBE is a wrapping receive buffer for receiving RDMA writes from the peer 🌀 Cursors, as described below, get exchanged between peers to manage and track RDMA writes and local data reads from the RMBE for a TCP connection 🍓
4.3. RMBE Control Information ✨
Okay bestie, the RMBE control information is basically the gossip layer 🫢 It's made up of consumer cursors, producer cursors, wrap counts, CDC message sequence numbers, control flags like urgent data and "writer blocked" indicators, and TCP connection info like termination flags. Peers swap all of this through CDC messages, which get sent with RoCE SendMsg. A TCP/IP stack doing SMC-R must receive and store this info in its internal data structures, because that's how it manages the RMBE and its data buffer 💅
The format and contents of the CDC message are spelled out in Appendix A.4 ("Connection Data Control (CDC) Message Format"). Here's the high-level tea:
- Connection state flags such as sending done, connection closed, failover data validation, and abnormal close. 🚦
- A sequence number managed by the sender. It starts at 1, goes up by one every send, and wraps back to 0. It tracks the CDC messages that were sent, NOT the number of bytes sent, and it's used for failover data validation 🔢
- Producer cursor: a wrapping offset into the receiver's RMBE data area. It's set by the peer that's writing into the RMBE, and it points to where that writer will put the next byte of data. It comes with a wrap sequence number so the RMBE owner (the receiver) can spot full window size wrapping writes. Note that this cursor must account for (i.e., skip over) the RMBE eye catcher at the beginning of the data area 🧚
- Consumer cursor: a wrapping offset into the receiver's RMBE data area. It's set by the owner of the RMBE (the peer reading from it) and points to the offset of the next byte the peer will consume from its own RMBE. The sender cannot write beyond this cursor into the receiver's RMBE without causing data loss 🙈 Like the producer cursor, it's accompanied by a wrap count so the writer can spot full window size wrapping reads. Note that this cursor must account for (i.e., skip over) the RMBE eye catcher at the beginning of the data area too 💜
- Data flags such as urgent data, writer blocked indicator, and cursor update requests. 🌸
4.4. Use of RMBEs 🎀
4.4.1. Initializing and Accessing RMBEs
Main character energy: the RMBE eye catcher gets set up by the RMB owner before the RMBE is assigned to a specific TCP connection and its RMB index is shared with the SMC-R partner 🦋 Once that index has been communicated to the partner, the RMBE can only be referenced in "read-only mode" by the owner. All updates to it are done by the remote SMC-R partner through RDMA write operations. Slay 💅
Initializing an RMBE must include the following:
- Zeroing out the entire RMBE receive buffer. This helps minimize data integrity issues, like data from a previous connection somehow showing up in the current one 🧹
- Setting the beginning RMBE eye catcher. This eye catcher is super important for detecting accidental overlays of the RMBE 👀 The RMB owner should always validate these eye catchers before each new reference to the RMBE. If the eye catchers are found to be corrupted, the local host must reset the TCP connection associated with this RMBE and log the appropriate diagnostic information 😭
4.4.2. RMB Element Reuse and Conflict Resolution
RMB elements can be reused once their associated TCP and SMC-R connections are terminated 🔁 During normal and abnormal SMC-R connection termination processing, both SMC-R peers must explicitly acknowledge that they're done using an RMBE before that element can be freed and reassigned to another SMC-R connection instance. For more details on SMC-R connection termination, check Section 4.8 🫶
BUT, there are some error scenarios where this two-way explicit acknowledgment may never get completed 💀 In those cases, an RMBE owner may choose to reassign the RMBE to a new SMC-R connection instance on this SMC-R link group. When that happens, the partner SMC-R peer must detect the situation during SMC-R Rendezvous processing, when it's presented with an RMBE it believes is already in use for a different SMC-R connection. Then the partner must abort the existing SMC-R connection associated with that RMBE. The abort processing resets the TCP connection (if it's still active), but it must not attempt to perform any RDMA writes to this RMBE, and it must also ignore any data sitting in the local RMBE associated with the existing connection. After that, it frees up the local RMBE and notifies the local application that the connection is being abnormally reset 🚨
The remote SMC-R peer then proceeds to normal processing for this new SMC-R connection. 🌈
4.5. SMC-R Protocol Considerations 💖
The following sections cover the places where the SMC-R protocol is different from TCP, so pay attention ✨
4.5.1. SMC-R Protocol Optimized Window Size Updates
An SMC-R receiver host sends its consumer cursor information to the sender to show how much progress the receiving application has made in consuming the data that was sent 📬 The difference between the writer's producer cursor and the receiver's associated consumer cursor tells the sender how much window size it has to write into. This is kind of like TCP window update processing, so it comes with some similar considerations, like silly window syndrome avoidance 🙃 TCP has an optimization that cuts down on the overhead of very small, unproductive window size updates, which happen when a socket application consumes tiny amounts of data on every receive() call. For SMC-R, the receiver only updates its consumer cursor with a unique CDC message under the following conditions:
- The current window size (from a sender's perspective) is less than half of the receive buffer space, AND the consumer cursor update will result in a minimum increase in the window size of 10% of the receive buffer space. Some examples:
- a. Receive buffer size: 64K, current window size (from a sender's perspective): 50K. No need to update the consumer cursor. Plenty of space is available for the sender 🍓
- b. Receive buffer size: 64K, current window size (from a sender's perspective): 30K, current window size from a receiver's perspective: 31K. No need to update the consumer cursor. Even though the sender's window size is less than 1/2 of the 64K, the window update would only increase that by 1K, which is less than 1/10th of the 64K buffer size 🤏
- c. Receive buffer size: 64K, current window size (from a sender's perspective): 30K, current window size from a receiver's perspective: 64K. The receiver updates the consumer cursor, since the sender's window size is less than 1/2 of the 64K and the window update would increase that by more than 6.4K 🔥
- The receiver must always include a consumer cursor update whenever it sends a CDC message to the partner for another flow (i.e., send flow in the opposite direction). This lets the window size update get delivered with no extra overhead ✨ It's kind of like TCP DelayAck processing, and it's super effective for request/response data patterns 🦋
- If a peer has set the B-bit in a CDC message, then any consumption of data by the receiver causes a CDC message to be sent, updating the consumer cursor, until a CDC message with that bit cleared is received from the peer 🫧
- The optimized window size updates are overridden when the sender sets the
Consumer Cursor Update Requestedflag in a CDC message to the receiver. When this indicator is on, the consumer must send a consumer cursor update immediately when data is consumed by the local application, or if the cursor hasn't been updated for a while (i.e., the local copy of the consumer cursor doesn't match the last consumer cursor value sent to the partner). This lets the sender run optional diagnostics to detect a stalled receiver application (data has been sent but not consumed) 🕵️♀️ It is recommended that theConsumer Cursor Update Requestedflag only be sent for diagnostic procedures, since it may result in non-optimal data path performance 🙏
4.5.2. Small Data Sends
The SMC-R protocol makes no special provisions for handling small data segments sent across a stream socket. Data is always sent if sufficient window space is available 📦 Unlike the TCP Nagle algorithm, SMC-R has no special provisions for coalescing small data segments. No Nagle vibes here, periodt 💅
An implementation of SMC-R can be configured to optimize its sending by coalescing outbound data for a given SMC-R connection, so it can reduce the number of RDMA write operations it performs, kind of like Nagle's algorithm 🌷 However, any such coalescing would require a timer on the sending host that makes sure the data gets sent eventually ⏰ Also, the sending host would have to opt out of this processing if Nagle's algorithm had been disabled, either programmatically or through system configuration ⭐
4.5.3. TCP Keepalive Processing 💖
okay so TCP keepalive processing is how you tell your local TCP/IP host to keep poking an idle TCP connection every so often to check that it's still alive 🦋 Since SMC-R connections have a TCP side and an SMC-R side, there are some extra keepalive things to think about:
- SMC-R-layer keepalive processing: If keepalive is enabled for an SMC-R connection, the local host runs a keepalive timer that tracks how long that SMC-R connection has been idle 🕰️ It also keeps a last-activity timestamp for each SMC-R link (covering every SMC-R connection on that link). When an SMC-R connection has been idle longer than the keepalive interval, the host checks whether the SMC-R link has been idle longer than the keepalive timeout. If both are true, the local host sends a
TEST LINK LLCcommand to check that the SMC-R link is still viable over the RoCE fabric (RC-QPs) 🔥 If aTEST LINK LLCresponse comes back within a reasonable amount of time, the link counts as viable, and so does every connection using it ✨ But if no response shows up in a reasonable amount of time, or sending theTEST LINK LLCcommand fails, that counts as an SMC-R link failure, and failover processing to an alternate SMC-R link has to be triggered 😭 If there's no alternate SMC-R link in the SMC-R link group, all the SMC-R connections on this link get abnormally terminated by resetting the TCP connections they're represented by 💀 Since lots of SMC-R connections can share one SMC-R link, doing a link-level probe withTEST LINK LLChelps cut down on unproductive keepalive traffic for SMC-R connections 🧚 As long as some SMC-R connections on a given link are active (meaning they've had I/O activity within the keepalive interval), there's no need to do extra link viability testing. - TCP-layer keepalive processing: Traditional TCP keepalive packets aren't as relevant for SMC-R connections, because the TCP path isn't used for these connections once SMC-R Rendezvous processing is done 🌸 By default, every SMC-R connection has an associated TCP connection that just sits there idle. So are TCP keepalive probes still needed? There are two main scenarios to look at:
- TCP keepalives used to figure out whether the peer TCP endpoint is still around. These aren't needed for SMC-R connections, since the SMC-R-level keepalives above already tell you whether the remote endpoint is still active 🙅♀️
- TCP keepalives used to keep TCP connections going through an intermediate proxy in an active state. For example, stateful firewalls usually track every valid TCP connection that passes through them. These firewalls are known to expire idle connections by dropping their state to save memory 🧠 TCP keepalives are often used in this scenario so firewalls don't time out idle connections. When using SMC-R, both endpoints have to be in the same Layer 2 network (that is, the same subnet). So firewalls can't be placed in the path between two SMC-R endpoints 🔒 Other intermediate proxies, though, like TCP/IP-layer load balancers, can be in that path. These load balancers also keep connection state so they can forward TCP connection traffic to the right cluster endpoint. With SMC-R, these TCP connections look completely idle, which makes them vulnerable to timeouts at the load-balancing proxy 😬 So for this scenario, TCP keepalives may still be relevant 💅
The following are the TCP-level keepalive processing requirements for SMC-R-enabled hosts:
- SMC-R peers should allow TCP keepalives to flow on the TCP path of SMC-R connections, based on the existing TCP keepalive configuration and programming options. That said, it is strongly recommended that platforms support very granular keepalive timers (for example, single-digit-second timers), and they should consider offering a configuration option that puts a floor on the minimum keepalive timer used for TCP-layer keepalives on SMC-R connections 🎀 This matters a lot for keeping the number of TCP keepalive packets on the network for SMC-R connections down ⭐
- SMC-R peers have to always respond to inbound TCP-layer keepalives (by sending ACKs for these packets), even when the connection is using SMC-R 🫶 Normally, once a TCP connection has finished SMC-R Rendezvous processing and is using SMC-R for data flows, no new inbound TCP segments are expected on it, except for TCP termination segments (FIN, RST, etc.). TCP keepalives are the one exception that has to be supported 🌈 And since TCP keepalive probes don't carry any application-layer data, they have no bad effect on the application's inbound data stream.
4.6. TCP Connection Failover between SMC-R Links ✨
A peer can change which SMC-R link within a link group it sends its writes over if a link fails 🔄 Since each peer picks its own link for writes on a given TCP connection, every peer does this independently 👑
4.6.1. Validating Data Integrity 🛡️
Even though RoCE is a reliable transport, there's a small set of failure modes that could cause unrecoverable data loss 😱 When an RNIC acknowledges receipt of an RDMA write from its peer, it creates a write completion event on the sending peer's side, which lets the sender release any buffers it was holding for that write. In normal operation and in most failures, this is reliable 👍
But there are failure modes where a receiving RNIC has acknowledged an RDMA write and then couldn't place the received data into its host memory. One example is a sudden, messy failure of the interface between the RNIC and the host 💥 These are rare, but they have to be guarded against to keep data intact. The link-switching process during failover, described in this section, guards against this possibility, and it is mandatory 🔐
Each peer has to keep track of the current state of the CDC sequence numbers for a TCP connection. The sender has to track the sequence number of the CDC message that described the last write acknowledged by the peer RNIC, which is called Sequence Sent (SS). In other words, SS tells you the last write the sender thinks its peer has successfully received 📬 The receiver has to track the sequence number of the CDC message that described the last write it successfully received (meaning the data was successfully placed into an RMBE), which is called Sequence Received (SR) 🧾
When an RNIC fails and the sender switches SMC-R links, the sender has to first send a CDC message with the F-bit (failover validation indicator; see Appendix A.4) set over the new SMC-R link 🚦 This is the failover data validation message. The sequence number in this CDC message is equal to SS. The CDC message key, the length, and the SMC-R alert token are the only other fields in this CDC message that matter. No reply is expected for this validation message, and once the sender has sent it, the sender may resume sending on the new SMC-R link as described in Section 4.6.2 🦋
When the receiver gets the failover validation message, it has to verify that its SR value for the TCP connection is equal to or greater than the sequence number in that message. If it is, no further action is needed, and the TCP connection resumes on the new SMC-R link 🌸 But if SR is less than the sequence number value in the validation message, data has been lost, and the receiver has to immediately reset the TCP connection 🚨
4.6.2. Resuming the TCP Connection on a New SMC-R Link 🏃♀️
When a connection moves to a new SMC-R link and the failover validation message has been sent, the sender can immediately resume normal transmission 💨 To keep the application message stream intact, the sender has to replay any RDMA writes (and their associated CDC messages) that were in progress or failed when the previous SMC-R link went down, before it sends new data on the new SMC-R link. The sender has two options for doing this:
- Preserve the sequence numbers "as is": Retry all failed and pending operations exactly as they were originally done, reposting all the associated RDMA write operations and their associated CDC messages without changing anything. Then resume sending new data using new sequence numbers 🔁
- Combine pending messages and possibly add new data: Merge the failed and pending messages into a single new write with a new sequence number 🧩 This lets the sender squash pending messages into fewer operations. As a bonus, this write can also include new data, as long as all the failed and pending data are included too ✨ If this approach is taken, the sequence number has to be increased beyond the last failed or pending sequence number 📈
4.7. RMB Data Flows 🌊
The following sections describe the RDMA wire flows for the SMC-R protocol after a TCP connection has switched into SMC-R mode (meaning SMC-R Rendezvous processing is complete and a pair of RMB elements has been assigned and shared between the SMC-R peers) 🎀 The ladder diagrams below include the following:
- RMBE control information kept by each peer. Only a subset of the information is shown, specifically just the fields that reflect the stream of data written by Host A and read by Host B 👀
- Time line 0-x, which shows the wire flows in a time-relative way ⏱️
- RMBE control information is only shown in a time interval if its value changed (otherwise, assume the value is unchanged from the previously shown value) 🙈
- The local copy of the producer cursors and consumer cursors that each host keeps isn't shown in these figures. Note that the cursor values in the diagram reflect the need to skip over the eye catcher in the RMBE data area 💜 They start and wrap at 4, not 0 🍓
4.7.1. Scenario 1: Send Flow, Window Size Unconstrained ✨
Okay bestie, let's walk through the simplest flow first 💖 Here's how it looks in the diagram:
SMC Host A SMC Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flags
4 0 0 0 4 0 0
0 0 1 ---------------> 1 0 0 0
RDMA-WR Data
(4:1003)
4 0 2 ...............> 2 1004 0 0
CDC MessageFigure 16: Scenario 1: Send Flow, Window Size Unconstrained
Scenario assumptions:
- Kernel implementation. 🧠
- New SMC-R connection, so no data has been sent on the connection yet. 🌸
- Host A: the application issues a send for 1000 bytes to Host B. 📤
- Host B: the RMBE receive buffer size is 10,000, and the application has issued a recv for 10,000 bytes. 📥
Flow description:
- The application issues a
send()for 1000 bytes, and the SMC-R layer copies the data into a kernel send buffer. Then it schedules an RDMA write to move the data into the peer's RMBE receive buffer, at relative position4-1003(it skips the 4-byte eye catcher in the RMBE data area). Note that no immediate data or alert (aka interrupt) goes to Host B for this RDMA operation. 🦋 - Host A sends a CDC message to update the producer cursor to byte 1004. This CDC message delivers an interrupt to Host B, and at this point the SMC-R layer can hand control back to the application. Host B gets notified that the earlier RDMA operation finished, finds the RMBE tied to the RMBE alert token in the message, and does its normal receive-side work: waking the suspended read thread, copying the data into the app's receive buffer, and so on. It uses the producer cursor to figure out how much data is ready for the local app. Once that's done, the SMC-R layer updates its local consumer cursor to match the producer cursor, meaning everything has been consumed. A message to the peer updating the consumer cursor isn't needed right now, because the window size is unconstrained (> 1/2 of the receive buffer size). The window size is the producer cursor minus the consumer cursor in the RMBEs:
10,000 - 1004 = 8996. 💅
4.7.2. Scenario 2: Send/Receive Flow, Window Size Unconstrained 🎀
This one is a full back-and-forth, so it's giving conversation 🗣️✨
SMC Host A SMC Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flags
4 0 0 0 4 0 0
0 0 1 ---------------> 1 0 0 0
RDMA-WR Data
(4:1003)
4 0 2 ...............> 2 1004 0 0
CDC Message
0 0 3 <-------------- 3 1004 0 0
RDMA-WR Data
(4:503)
1004 0 4 <.............. 4 1004 0 0
CDC MessageFigure 17: Scenario 2: Send/Receive Flow, Window Size Unconstrained
Scenario assumptions:
- New SMC-R connection, so no data has been sent on the connection yet. 🌷
- Host A: the application issues a send for 1000 bytes to Host B. 📤
- Host B: the RMBE receive buffer size is 10,000, and the application has already issued a recv for 10,000 bytes. Once that receive completes, the application sends a 500-byte response to Host A. 💌
Flow description:
- The application issues a
send()for 1000 bytes, and the SMC-R layer copies the data into a kernel send buffer. It schedules an RDMA write into the peer's RMBE receive buffer at relative position4-1003. Again, no immediate data or alert (aka interrupt) goes to Host B for this RDMA operation. 🦋 - Host A sends a CDC message to update the producer cursor to byte 1004. This CDC message delivers an interrupt to Host B, and the SMC-R layer can return control to the application right away. 🌈
- Host B gets notified about the CDC message, finds the RMBE tied to the RMBE alert token, and does normal receive-side processing: waking the suspended read thread, copying the data into the app's receive buffer, and so on. After that, the SMC-R layer updates its local consumer cursor to match the producer cursor, meaning everything has been consumed. No update of the consumer cursor goes to the peer yet, because the window size is unconstrained (> 1/2 of the receive buffer size). Then the application does a
send()for 500 bytes to Host A. The SMC-R layer copies that data into a kernel buffer and schedules an RDMA write into the partner's RMBE receive buffer. This RDMA write carries no immediate data and no notification to Host A. 🫶 - Host B sends a CDC message that updates the partner's RMBE control information with the latest producer cursor (set to 503, not shown in the diagram) and also tells the peer that the consumer cursor is now 1004. It updates the local current consumer cursor and the last sent consumer cursor to 1004 as well. This CDC message includes notification, since we're updating our producer cursor, and the peer host needs to pay attention to it. 🔥
4.7.3. Scenario 3: Send Flow, Window Size Constrained 🌸
Now things get a little spicy, because the window actually gets constrained 😭 Here's the diagram:
SMC Host A SMC Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flags
4 0 0 0 4 0 0
4 0 1 ---------------> 1 4 0 0
RDMA-WR Data
(4:3003)
4 0 2 ...............> 2 3004 0 0
CDC Message
4 0 3 3 3004 0 0
4 0 4 ---------------> 4 3004 0 0
RDMA-WR Data
(3004:7003)
4 0 5 ................> 5 7004 0 0
CDC Message
7004 0 6 <................ 6 7004 0 0
CDC MessageFigure 18: Scenario 3: Send Flow, Window Size Constrained
Scenario assumptions:
- New SMC-R connection, so no data has been sent on this connection yet. 🍓
- Host A: the application issues a send for 3000 bytes to Host B, then another send for 4000 bytes. 📦
- Host B: the RMBE receive buffer size is 10,000. The application has already issued a recv for 10,000 bytes. 📥
Flow description:
- The application issues a
send()for 3000 bytes, and the SMC-R layer copies the data into a kernel send buffer. It schedules an RDMA write to move the data into the peer's RMBE receive buffer at relative position4-3003. No immediate data or alert (aka interrupt) goes to Host B for this RDMA operation. 🦋 - Host A sends a CDC message to update its producer cursor to byte 3003. This CDC message delivers an interrupt to Host B, and the SMC-R layer can return control to the application. 💫
- Host B gets notified about the CDC message, finds the RMBE tied to the RMBE alert token, and does normal receive-side processing: waking the suspended read thread, copying the data into the app's receive buffer, and so on. Then the SMC-R layer updates its local consumer cursor to match the producer cursor, meaning everything has been consumed. It does not tell the partner about this, because the window size isn't constrained (10,000 - 3000 = 7000 bytes of available space). The application on Host B also issues a new recv for 10,000 bytes. 🎀
- On Host A, the application issues a
send()for 4000 bytes. The SMC-R layer copies the data into a kernel buffer and schedules an async RDMA write into the peer's RMBE receive buffer at relative position3003-7004. No alert goes to Host B for this flow. 🌊 - Host A sends a CDC message to update the producer cursor to byte 7004. This CDC message delivers an interrupt to Host B, and the SMC-R layer can return control to the application. 🧚
- Host B gets notified about the CDC message, finds the RMBE tied to the RMBE alert token, and does normal receive-side processing: waking the suspended read thread, copying the data into the app's receive buffer, and so on. After that, the SMC-R layer updates its local consumer cursor to match the producer cursor, meaning everything has been consumed. Then it figures out whether it needs to update the consumer cursor on the peer. The available window size is now 3000 (10,000 - (producer cursor - last sent consumer cursor)), which is < 1/2 of the receive buffer size (10,000/2 = 5000), and the advance of the window size is > 10% of the window size (1000). So a CDC message goes out to update the consumer cursor to Peer A. 👑😭
4.7.4. Scenario 4: Large Send, Flow Control, Full Window Size Writes
Okay bestie, this one is a whole saga 🎀 Two SMC hosts, one big 20,000 byte send, and a receive buffer that only holds 10,000 bytes. Let's see how they tag-team it 🦋
SMC Host A SMC Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flags
1004 1 0 0 1004 1 0
1004 1 1 ---------------> 1 1004 1 0
RDMA-WR Data
(1004:9999)
1004 1 2 ---------------> 2 1004 1 0
RDMA-WR Data
(4:1003)
1004 1 3 ...............> 3 1004 2 Wrt
CDC Message Blk
1004 2 4 <............... 4 1004 2 Wrt
CDC Message Blk
1004 2 5 ---------------> 5 1004 2 Wrt
RDMA-WR Data Blk
(1004:9999)
1004 2 6 ---------------> 6 1004 2 Wrt
RDMA-WR Data Blk
(4:1003)
1004 2 7 ...............> 7 1004 3 Wrt
CDC Message Blk
1004 3 8 <............... 8 1004 3 Wrt
CDC Message Blk
Figure 19: Scenario 4: Large Send, Flow Control,
Full Window Size WritesScenario assumptions:
- Kernel implementation.
- Existing SMC-R connection, Host B's receive window size is fully open (peer consumer cursor = peer producer cursor).
- Host A: Application issues send for 20,000 bytes to Host B.
- Host B: RMBE receive buffer size is 10,000; application has issued a recv for 10,000 bytes.
Flow description:
- The application issues a send() for 20,000 bytes, and the SMC-R layer copies the data into a kernel send buffer (assuming send buffer space of 20,000 is available for this connection) 🧺 It then schedules an RDMA write operation to move the data into the peer's RMBE receive buffer at relative position 1004-9999. Note that no immediate data or alert (i.e., interrupt) is provided to Host B for this RDMA operation 🙈
- Host A then schedules an RDMA write operation to fill the remaining 1000 bytes of available space in the peer's RMBE receive buffer, at relative position 4-1003. Note that no immediate data or alert (i.e., interrupt) is provided to Host B for this RDMA operation. Also note that an implementation of SMC-R may optimize this processing by combining steps 1 and 2 into a single RDMA write operation (with two different data sources) ✨
- Host A sends a CDC message to update the producer cursor to byte 1004. Since the entire receive buffer space is filled, the producer writer blocked flag (the "Wrt Blk" indicator (flag) in Figure 19) is set and the producer cursor wrap sequence number (the producer "Wrap Seq#" in Figure 19) is incremented. This CDC message will deliver an interrupt to Host B. At this point, the SMC-R layer can return control back to the application 💅
- Host B, once notified of the receipt of the previous CDC message, locates the RMBE associated with the RMBE alert token and proceeds to perform normal receive-side processing, waking up the suspended application read thread, copying the data into the application's receive buffer, etc. In this scenario, Host B notices that the producer cursor has not been advanced (same value as the consumer cursor); however, it notices that the producer cursor wrap sequence number is different from its local value (1), indicating that a full window of new data is available. All of the data in the receive buffer can be processed, with the first segment (1004-9999) followed by the second segment (4-1003). Because the producer writer blocked indicator was set, Host B schedules a CDC message to update its latest information to the peer: consumer cursor (1004), consumer cursor wrap sequence number (the current value of 2 is used) 🫶
- Host A, upon receipt of the CDC message, locates the TCP connection associated with the alert token and, upon examining the control information provided, notices that Host B has consumed all of the data (based on the consumer cursor and the consumer cursor wrap sequence number) and initiates the next RDMA write to fill the receive buffer at offset 1003-9999. 🔥
- Host A then moves the next 1000 bytes into the beginning of the receive buffer (4-1003) by scheduling an RDMA write operation. Note that at this point there are still 8 bytes remaining to be written. 👑
- Host A then sends a CDC message to set the producer writer blocked indicator and to increment the producer cursor wrap sequence number (3). 🌈
- Host B, upon notification, completes the same processing as step 4 above, including sending a CDC message to update the peer to indicate that all data has been consumed. At this point, Host A can write the final 8 bytes to Host B's RMBE into positions 1004-1011 (not shown). 🦋
4.7.5. Scenario 5: Send Flow, Urgent Data, Window Size Unconstrained
Last scenario in this batch, and it's giving urgent data drama 😭 Host A sends a 500-byte urgent chunk and then a normal 1000-byte chunk. Watch how the flags keep everyone in sync 🧚
SMC Host A SMC Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flag
1000 1 0 0 1000 1 0
1000 1 1 ---------------> 1 1000 1 0
RDMA-WR Data
(1000:1499)
1000 1 2 ...............> 2 1500 1 UrgP
CDC Message UrgA
1500 1 3 <............... 3 1500 1 UrgP
CDC Message UrgA
1500 1 4 ---------------> 4 1500 1 UrgP
RDMA-WR Data UrgA
(1500:2499)
1500 1 5 ...............> 5 2500 1 0
CDC Message
Figure 20: Scenario 5: Send Flow, Urgent Data, Window Size OpenScenario assumptions:
- Kernel implementation.
- Existing SMC-R connection; window size open (unconstrained); all data has been consumed by receiver.
- Host A: Application issues send for 500 bytes with urgent data indicator (out of band) to Host B, then sends 1000 bytes of normal data.
- Host B: RMBE receive buffer size is 10,000; application has issued a recv for 10,000 bytes and is also monitoring the socket for urgent data.
Flow description:
- The application issues a send() for 500 bytes of urgent data; the SMC-R layer copies the data into a kernel send buffer. It then schedules an RDMA write operation to move the data into the peer's RMBE receive buffer at relative position 1000-1499. Note that no immediate data or alert (i.e., interrupt) is provided to Host B for this RDMA operation 🙈
- Host A sends a CDC message to update its producer cursor to byte 1500 and to turn on the producer Urgent Data Pending (UrgP) and Urgent Data Present (UrgA) flags. This CDC message will deliver an interrupt to Host B. At this point, the SMC-R layer can return control back to the application 💜
- Host B, once notified of the receipt of the previous CDC message, locates the RMBE associated with the RMBE alert token, notices that the Urgent Data Pending flag is on, and proceeds with out-of-band socket API notification -- for example, satisfying any outstanding select() or poll() requests on the socket by indicating that urgent data is pending (i.e., by setting the exception bit on). The urgent data present indicator allows Host B to also determine the position of the urgent data (the producer cursor points 1 byte beyond the last byte of urgent data). Host B can then perform normal receive-side processing (including specific urgent data processing), copying the data into the application's receive buffer, etc. Host B then sends a CDC message to update the partner's RMBE control area with its latest consumer cursor (1500). Note that this CDC message must occur, regardless of the current local window size that is available. The partner host (Host A) cannot initiate any additional RDMA writes until it receives acknowledgment that the urgent data has been processed (or at least processed/remembered at the SMC-R layer) ✋
- Upon receipt of the message, Host A wakes up, sees that the peer consumed all data up to and including the last byte of urgent data, and now resumes sending any pending data. In this case, the application had previously issued a send for 1000 bytes of normal data, which would have been copied in the send buffer, and control would have been returned to the application. Host A now initiates an RDMA write to move that data to the peer's receive buffer at position 1500-2499. 🍓
- Host A then sends a CDC message to update its producer cursor value (2500) and to turn off the Urgent Data Pending and Urgent Data Present flags. Host B wakes up, processes the new data (resumes application, copies data into the application receive buffer), and then proceeds to update the local current consumer cursor (2500). Given that the window size is unconstrained, there is no need for a consumer cursor update in the peer's RMBE. ⭐
4.7.6. Scenario 6: Send Flow, Urgent Data, Window Size Closed ✨
Okay bestie, this one is a whole drama 😭. The window is closed, so the writer is stuck, and then urgent data shows up anyway 💅 Here's how the two SMC hosts sort it out, step by step 🦋
SMC Host A SMC Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flag
1000 1 0 0 1000 2 Wrt
Blk
1000 1 1 ...............> 1 1000 2 Wrt
CDC Message Blk
UrgP
1000 2 2 <............... 2 1000 2 Wrt
CDC Message Blk
UrgP
1000 2 3 ---------------> 3 1000 2 Wrt
RDMA-WR Data Blk
(1000:1499) UrgP
1000 2 4 ...............> 4 1500 2 UrgP
CDC Message UrgA
1500 2 5 <............... 5 1500 2 UrgP
CDC Message UrgA
1500 2 6 ---------------> 6 1500 2 UrgP
RDMA-WR Data UrgA
(1500:2499)
1000 2 7 ...............> 7 2500 2 0
CDC Message
Figure 21: Scenario 6: Send Flow, Urgent Data, Window Size ClosedHere's the setup 🎀
- Kernel implementation.
- There's an existing SMC-R connection, the window size is closed, and the writer is blocked.
- Host A: the application issues a send for 500 bytes with the urgent data indicator (out of band) to Host B, then sends 1000 bytes of normal data.
- Host B: the RMBE receive buffer size is 10,000. The application has no outstanding recv() calls for normal data, and it's watching the socket for urgent data 👀
Now the flow, play by play 🍓
- The application issues a send() for 500 bytes of urgent data, and the SMC-R layer copies it into a kernel send buffer (if there's room). The writer is blocked because the window is closed, so it can't send the data right away. Instead, it sends a CDC message to tell the peer about the Urgent Data Pending (UrgP) indicator. The writer blocked indicator stays on too. This is basically Host A waving a flag at Host B saying urgent data is waiting in the stream 🚩 Control goes back to the application at this point.
- Host B gets the CDC message, finds the RMBE tied to the RMBE alert token, sees that the Urgent Data Pending flag is on, and kicks off out-of-band socket API notification. For example, it satisfies any outstanding select() or poll() calls on the socket by flagging that urgent data is pending (aka setting the exception bit on) ✨ The app is then expected to switch into urgent data mode, quickly processing all the normal data with recv API calls until it reaches the urgent byte. Whether the app has urgent mode processing or not, it will eventually consume some or all of the pending data in the receive buffer. When that happens, Host B sends a CDC message to the peer to update its consumer cursor and consumer cursor wrap sequence number. In this example, a full window's worth of data gets consumed 💅
- Host A gets woken up by that message and notices the window is open again. It checks the consumer cursor and the consumer cursor wrap sequence number, which now match the producer cursor wrap sequence number. Then it resumes sending the urgent data by scheduling an RDMA write into relative position 1000-1499 🚀
- Host A sends a CDC message to move its producer cursor to 1500, notify Host B of the Urgent Data Present (UrgA) indicator, and turn off the writer blocked indicator. This tells Host B that the urgent data is now in its local receive buffer, and that the producer cursor points at the last byte of urgent data 💜
- Host B wakes up, processes the urgent data, and once it's consumed, sends a CDC message to update its consumer cursor to 1500 🌈
- Host A wakes up, sees that Host B has consumed the sequence number tied to the urgent data, and then starts the next RDMA write. That moves the 1000 bytes from the next send() of normal data into the peer's receive buffer at position 1500-2499. Heads up, the send API probably finished way earlier, when those 1000 bytes got copied into a send buffer and control went back to the app 🧚 Even so, Host A couldn't push any new data until Host B had processed and acknowledged the urgent data.
- Host A sends a CDC message to move its producer cursor to 2500 and reset the Urgent Data Pending and Urgent Data Present flags. Host B wakes up and processes the inbound data 🎉
4.8. Connection Termination
Just like SMC-R connections get set up with a mix of TCP connection establishment flows and SMC-R protocol flows, they also get torn down with a mix of SMC-R protocol termination flows and normal TCP connection termination flows 🦋 The next sections walk through the SMC-R protocol's normal and abnormal termination flows.
4.8.1. Normal SMC-R Connection Termination Flows
Normal SMC-R termination flows get kicked off through the usual stream socket API, meaning the application calls close() or shutdown() 🌸 Most apps, once they've read all the incoming data and sent everything they need to send, just call close() to say they're done both sending and receiving. A smaller group uses shutdown(), which lets them say they're done sending, done receiving, or done with both. The big use case for shutdown() is when a TCP app wants to tell its partner "I'm finished sending," but it's still happy to receive (shutdown for write). Calling shutdown() for both sending and receiving is basically the same as close(), so it can be handled the same way. Shutdown for read usually isn't super useful, and in normal situations it doesn't trigger any network flows to tell the partner TCP endpoint about it 🙈
The SMC-R layer uses these same trigger points to start its own termination flows 💖 The big idea for normal SMC-R termination is to first use the SMC-R protocol to shut down the SMC-R connection and free up its RDMA resources, and then let the normal TCP termination (aka FIN processing) clean up the TCP connection. This order matters a lot! It makes sure RDMA resources like the RMBEs only get freed and reused once both SMC-R endpoints are completely finished with their RDMA writes into the partner's RMBE 🫶
1
+-----------------+
|-------------->| CLOSED |<-------------|
3D | | | | 4D
| +-----------------+ |
| | |
| 2 | |
| V |
+----------------+ +-----------------+ +----------------+
|AppFinCloseWait | | ACTIVE | |PeerFinCloseWait|
| | | | | |
+----------------+ +-----------------+ +----------------+
| | | |
| Active Close | 3A | 4A | Passive Close |
| V | V |
| +--------------+ | +-------------+ |
|--<----|PeerCloseWait1| | |AppCloseWait1|--->----|
3C | | | | | | | 4C
| +--------------+ | +-------------+ |
| | | | |
| | 3B | 4B | |
| V | V |
| +--------------+ | +-------------+ |
|--<----|PeerCloseWait2| | |AppCloseWait2|--->----|
| | | | |
+--------------+ | +-------------+
|
|
Figure 22: SMC-R Connection StatesFigure 22 shows the states an SMC-R connection usually moves through 🗺️ Heads up, there are some variations on these states when an SMC-R connection gets terminated abnormally, kinda like how a TCP connection gets reset 😬 Here are the big-picture state transitions for an SMC-R connection:
- An SMC-R connection starts in the Closed state. That state is meant to show an RMBE that isn't being used right now. It's either something that was used before but isn't anymore, or something that was never allocated at all.
- An SMC-R connection moves into the Active state once SMC-R Rendezvous processing finishes successfully, the RMB element indices have been swapped, and the SMC-R links are activated ✨ In this state, the TCP connection is fully set up, rendezvous is done, and the SMC-R peers can start exchanging data over RDMA 🚀
- Active close processing (on the SMC-R peer that starts the connection termination) 💀
A. okay so when an app on one of the SMC-R connection peers calls close(), does a shutdown() for write, or does a shutdown() for both read and write, the SMC-R layer on that host kicks off connection termination processing 💅. First, if a close() or shutdown(both) happens, it checks that there's no data sitting in the local RMB element that the application hasn't read yet. If unread data shows up, the SMC-R connection must be abnormally reset. For the full tea, see Section 4.8.2 ("Abnormal SMC-R Connection Termination Flows") 🫣. If nothing is unread, it then checks whether any outstanding data is still waiting to be written to the peer, or whether any outstanding RDMA writes for this SMC-R connection haven't finished yet. If either of those is true, an indicator that this connection is in a pending close state gets saved in the internal data structures for this SMC-R connection, and control goes back to the application ✨. If all data to be written to the partner has completed, this peer sends a CDC message to notify the peer of either the PeerConnectionClosed indicator (close or shutdown for both was issued) or the PeerDoneWriting indicator. That's basically an interrupt telling the partner SMC-R peer that the connection is terminating 🚨. At this point, the local side of the SMC-R connection transitions to the PeerCloseWait1 state, and control can be returned to the application. If this process couldn't be completed synchronously (the pending close condition mentioned above), it finishes once all RDMA writes for data and control cursors have completed 🌸.
B. At some point, the SMC-R peer application (the passive close side) will consume all incoming data, realize that its partner is done sending data on this connection, and then start its own close of the connection once it has finished sending all of its data 🫶. The partner application can kick off this connection termination processing with close() or shutdown() APIs. If the application issues a shutdown() for write, then the partner SMC-R layer sends a CDC message to notify the peer (the active close side) of the PeerDoneWriting indicator. When the "active close" SMC-R peer wakes up because of that CDC message, it notices that the PeerDoneWriting indicator is now on and transitions to the PeerCloseWait2 state. This state means the peer is done sending data but may still be reading data 👀. At this point, the "active close" peer also needs to make sure that any outstanding recv() calls for this socket are woken up, and it remembers that no more data is coming on this connection (in case the local connection was shut down for write only) 💜.
C. This flow is a common transition from 3A or 3B above 🦋. When the SMC-R peer (the passive close side) consumes all data and updates all necessary cursors to the peer, and the application closes its socket (close or shutdown for both), it sends a CDC message to the peer (the active close side) with the PeerConnectionClosed indicator set. At this point, the connection can transition back to the Closed state if the local application has already closed the socket (or issued shutdown for both). Once it's in the Closed state, the RMBE can be safely reused for a new SMC-R connection ♻️. When the PeerConnectionClosed indicator is turned on, the SMC-R peer is saying that it is done updating the partner's RMBE 🎀.
D. Conditional state: if the local application hasn't yet issued a close() or shutdown(both), we have to wait until it does 🙈. Once it does, the local host sends a CDC message to notify the peer of the PeerConnectionClosed indicator and then transitions to the Closed state 🌈.
- Passive close processing (on the SMC-R peer that receives an indication that the partner is closing the connection).
A. When a CDC message arrives, the SMC-R layer detects that the PeerConnectionClosed indicator or the PeerDoneWriting indicator is on 📬. If any outstanding recv() calls are pending, they get completed with an indicator that the partner has closed the connection (zero-length data presented to the application) 🍓. If there's any pending data to be written and PeerConnectionClosed is on, then an SMC-R connection reset must be performed 💀. The connection then enters the AppCloseWait1 state on the passive close side, waiting for the local application to start its own close processing ⭐.
B. If the local application issues a shutdown() for writing, the SMC-R layer sends a CDC message to notify the partner of the PeerDoneWriting indicator, and then transitions the local side of the SMC-R connection to the AppCloseWait2 state 🧚.
C. When the application issues a close() or shutdown() for both, the local SMC-R peer sends a message informing the peer of the PeerConnectionClosed indicator. It transitions to the Closed state if the remote peer has also sent the local peer the PeerConnectionClosed indicator. If the peer hasn't sent the PeerConnectionClosed indicator, we transition into the PeerFinCloseWait state 🫧.
D. The local SMC-R connection stays in this state until the peer sends the PeerConnectionClosed indicator in a CDC message. When that indicator arrives, we transition to the Closed state and are then free to reuse this RMBE 🥳.
Note that each SMC-R peer needs some logic that stops it from getting stranded in a termination state forever 🕰️. For example, if an Active Close SMC-R peer is sitting in a PeerCloseWait (1 or 2) state waiting for the remote SMC-R peer to update its connection termination status, it needs a timer so it doesn't wait in that state indefinitely if the remote SMC-R peer never responds to the termination request. This could happen in error scenarios, like if the remote SMC-R peer failed before it could respond to the termination request, or if the remote application isn't responding to the request by closing its own socket 😤. This latter scenario is similar to the TCP FINWAIT2 state, which has been known to cause issues when remote TCP/IP hosts lose track of established connections and forget to close them. Even though the TCP standards don't require a timeout from the TCP FINWAIT2 state, most TCP/IP implementations assign one anyway 🫠. A similar timeout will be required for SMC-R connections. When this timeout occurs, the local SMC-R peer performs TCP reset processing for this connection. However, no additional RDMA writes to the partner RMBE can happen at this point, since we've already indicated that we're done updating the peer's RMBE 🛑. After the TCP connection is reset, the RMBE can go back to the free pool for reallocation. See Section 4.4.2 for more details 📚.
Also note that it's possible for two SMC-R endpoints to start an Active close concurrently. In that case, the flows above still apply, but both endpoints follow the active close path (path 3) 👯♀️✨.
4.8.2. Abnormal SMC-R Connection Termination Flows ✨
Abnormal SMC-R connection termination can happen for a bunch of reasons, but these two are the big ones 💀:
- The TCP connection tied to an SMC-R connection gets reset. In TCP, either endpoint can send an RST segment to abort an existing TCP connection when it spots error conditions for that connection, or when the application explicitly asks for a reset. 🚫
- Normal SMC-R connection termination processing unexpectedly stalls for a connection. Once that stall is detected (the connection termination timeout condition), an abnormal SMC-R connection termination flow kicks off. ⏰
In these situations it is SO important that resources tied to the affected connections get cleaned up properly. That way there are no orphaned resources, and resources can be reliably reused for new SMC-R connections 🧹. SMC-R leans super hard on RDMA write processing, so extra care is needed to make sure an RMBE is no longer being used by an SMC-R peer before it gets logically reassigned to a new SMC-R connection. Main character energy, but make it safe 💅.
When an SMC-R peer starts a TCP connection reset, it also starts an SMC-R abnormal connection flow at the same time. The SMC-R peers explicitly signal their intent to abnormally terminate an SMC-R connection, and then wait for explicit acknowledgment that the other peer got the notification and has finished abnormal connection termination on its end. TCP connection reset processing can happen in parallel with these flows 🦋.
+-----------------+
|-------------->| CLOSED |<-------------|
| | | |
| +-----------------+ |
| |
| |
| |
| +-----------------------+ |
| | Any state | |
|1B | (before setting | 2B|
| | PeerConnectionClosed | |
| | indicator in | |
| | peer's RMBE) | |
| +-----------------------+ |
| 1A | | 2A |
| Active Abort | | Passive Abort |
| V V |
| +--------------+ +--------------+ |
|-------|PeerAbortWait | | Process Abort|------|
| | | |
+--------------+ +--------------+Figure 23: SMC-R Abnormal Connection Termination State Diagram
Figure 23 above shows the SMC-R abnormal connection termination state diagram, and here's the tea ☕:
- Active abort is the SMC-R peer that initiates the TCP RST processing. At the time the TCP RST is sent, the active abort side must also do the following:
- A. Send the
PeerConnAbortindicator to the partner in a CDC message, and then transition to thePeerAbortWaitstate. During this state, it monitors this SMC-R connection, waiting for the peer to send its correspondingPeerConnAbortindicator, but it ignores any other activity on this connection (i.e., new incoming data) 🙈. It also generates an appropriate error to any socket API calls issued against this socket (e.g.,ECONNABORTED,ECONNRESET) ❌. - B. Once the peer sends the
PeerConnAbortindicator to the local host, the local host can transition this SMC-R connection to the Closed state and reuse this RMBE. Note that the SMC-R peer that goes into the active abort state must provide some protection against staying in that state indefinitely should the remote SMC-R peer not respond by sending its ownPeerConnAbortindicator to the local host. This should be a rare scenario, but it could happen if the remote SMC-R peer (passive abort) suffers a failure right after the local SMC-R peer (active abort) sends thePeerConnAbortindicator. To protect against these types of failures, a timer can be set after entering thePeerAbortWaitstate. If that timer pops before the peer has sent itsPeerConnAbortindicator (to the active abort side), this RMBE can be returned to the free pool for possible reallocation. See Section 4.4.2 for more details ⏲️.
- A. Send the
- Passive abort is the SMC-R peer that receives an SMC-R abort from the peer, as signaled by the
PeerConnAbortindicator sent by that peer in a CDC message. Upon receiving this request, the local peer must do the following:- A. Using the appropriate error codes, indicate to the socket application that this connection has been aborted, and then purge all in-flight data for this connection that is waiting to be read or waiting to be sent 🗑️.
- B. Send a CDC message to notify the peer of the
PeerConnAbortindicator and, once that is completed, transition this RMBE to the Closed state ✅.
If an SMC-R peer receives a TCP RST for a given SMC-R connection, it also initiates SMC-R abnormal connection termination processing if it has not already been notified (via the PeerConnAbort indicator) that the partner is severing the connection. It is possible to have two SMC-R endpoints concurrently be in an active abort role for a given connection 👯. In that scenario, the flows above still apply, but both endpoints take the active abort path (path 1).
4.8.3. Other SMC-R Connection Termination Conditions 🌙
The following are additional conditions that have implications for SMC-R connection termination:
- An SMC-R peer being gracefully shut down. If an SMC-R peer supports a graceful shutdown operation, it should attempt to terminate all SMC-R connections as part of shutdown processing. This could be accomplished via LLC DELETE LINK requests on all active SMC-R links 👋.
- Abnormal termination of an SMC-R peer. In this example, there may be no opportunity for the host to perform any SMC-R cleanup processing. In this scenario, it is up to the remote peer to detect a RoCE communications failure with the failing host. This could trigger SMC-R link switchover, but that would also generate RoCE errors, causing the remote host to eventually terminate all existing SMC-R connections to this peer 🫠.
- Loss of RoCE connectivity between two SMC-R peers. If two peers are no longer reachable across any links in their SMC-R link group, then both peers perform a TCP reset for the connections, generate an error to the local applications, and free up all QP resources associated with the link group 💥.
5. Security Considerations 🔐
Okay, security is serious business, bestie, so pay attention 🚨.
5.1. VLAN Considerations
The concepts and access control of virtual LANs (VLANs) must be extended to also cover the RoCE network traffic flowing across the Ethernet 🧱.
The RoCE VLAN configuration and access permissions must mirror the IP VLAN configuration and access permissions over the Converged Enhanced Ethernet fabric. This means that hosts, routers, and switches that have access to specific VLANs on the IP fabric must also have the same VLAN access across the RoCE fabric. In other words, the SMC-R connectivity will follow the same virtual network access permissions as normal TCP/IP traffic 🔒.
5.2. Firewall Considerations
As mentioned above, the RoCE fabric inherits the same VLAN topology/access as the IP fabric. RoCE is a Layer 2 protocol that requires both endpoints to reside in the same Layer 2 network (i.e., VLAN). RoCE traffic cannot traverse multiple VLANs, as there is no support for routing RoCE traffic beyond a single VLAN. As a result, SMC-R communications will also be confined to peers that are members of the same VLAN 🚧. IP-based firewalls are typically inserted between VLANs (or physical LANs) and rely on normal IP routing to insert themselves in the data path. Since RoCE (and by extension SMC-R) is not routable beyond the local VLAN, there is no ability to insert a firewall in the network path of two SMC-R peers 😭.
5.3. Host-Based IP Filters
Because SMC-R maintains the TCP three-way handshake for connection setup before switching to RoCE out of band, existing IP filters that control connection setup flows remain effective in an SMC-R environment. IP filters that operate on traffic flowing in an active TCP connection are not supported, because the connection data does not flow over IP 🚫.
5.4. Intrusion Detection Services
Similar to IP filters, intrusion detection services that operate on TCP connection setups are compatible with SMC-R with no changes required. However, once the TCP connection has switched to RoCE out of band, packets are not available for examination 👀.
5.5. IP Security (IPsec)
IP security is not compatible with SMC-R, because there are no IP packets on which to operate. TCP connections that require IP security must opt out of SMC-R 🙅♀️.
5.6. TLS/SSL
Transport Layer Security/Secure Socket Layer (TLS/SSL) is preserved in an SMC-R environment. The TLS/SSL layer resides above the SMC-R layer, and outgoing connection data is encrypted before being passed down to the SMC-R layer for RDMA write. Similarly, incoming connection data goes through the SMC-R layer encrypted and is decrypted by the TLS/SSL layer as it is today 🔐✨.
The TLS/SSL handshake messages flow over the TCP connection after the connection has switched to SMC-R, so they are exchanged using RDMA writes by the SMC-R layer, transparently to the TLS/SSL layer 🫶.
6. IANA Considerations 📋
The scarcity of TCP option codes available for assignment is understood, and this architecture uses experimental TCP options following the conventions of [RFC6994] ("Shared Use of Experimental TCP Options") 🧪.
TCP ExID 0xE2D4C3D9 has been registered with IANA as a TCP Experiment Identifier. See Section 3.1 🎀.
If this protocol achieves wide acceptance, a discrete option code may be requested by subsequent versions of this protocol 🌈.
7. Normative References 📚
[RFC793] Postel, J., "Transmission Control Protocol", STD 7,
RFC 793, DOI 10.17487/RFC0793, September 1981,
<http://www.rfc-editor.org/info/rfc793>.
[RFC6994] Touch, J., "Shared Use of Experimental TCP Options",
RFC 6994, DOI 10.17487/RFC6994, August 2013,
<http://www.rfc-editor.org/info/rfc6994>.
[RoCE] InfiniBand, "RDMA over Converged Ethernet specification",
<https://cw.infinibandta.org/wg/Members/documentRevision/
download/7149>.Appendix A. Formats 🎀
A.1. TCP Option 🎀
Okay so the SMC-R TCP option follows the rules in [RFC6994] ("Shared Use of Experimental TCP Options") 💅. The ExID value is IBM-1047 (EBCDIC) encoding for "SMCR" ✨. Here's how it looks:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Kind = 254 | Length = 6 | x'E2' | x'D4' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 24: SMC-R TCP Option Format 🦋
A.2. CLC Messages 🌸
Bestie, these rules apply to all CLC messages, so pay attention 👀
- Reserved fields must be set to zero and not validated.
- Each message has an eye catcher at the start and another eye catcher at the end. Both of these must be validated by the receiver 🫶
- SMC version indicator: the only SMC-R version defined in this architecture is version 1. In the future, if peers have a mismatch of versions, the lowest common version number is used 🔥
A.2.1. Peer ID Format 🧚
Every CLC message has a peer ID that uniquely identifies an instance of a TCP/IP stack. This peer ID is required to be universally unique across TCP/IP stacks and instances (including restarts) of TCP/IP stacks. Iconic uniqueness energy 👑
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Instance ID | RoCE MAC (first 2 bytes) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RoCE MAC (last 4 bytes) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 25: Peer ID Format
Instance ID 🍓 This is a 2-byte instance count that makes sure that if the same RNIC MAC is later used in the peer ID for a different TCP/IP stack (for example, if an RNIC is redeployed to another stack), the values are unique. It also makes sure that if a TCP/IP stack is restarted, the instance ID changes. The value is implementation defined, with one suggestion being 2 bytes of the system clock ⏰
RoCE MAC 💜 This is the RoCE MAC address for one of the peer's RNICs. Note that in a virtualized environment, this will be the virtual MAC of one of the peer's RNICs 🌈
A.2.2. SMC Proposal CLC Message Format 💖
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 1 | Length |Version| Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Client's Peer ID -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- Client's preferred GID -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Client's preferred RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |Offset to mask/prefix area (0) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. .
. Area for future growth .
. .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPv4 Subnet Mask |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPv4 Mask Lgth| Reserved |Num IPv6 prfx |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: :
: Array of IPv6 prefixes (variable length) :
: :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 26: SMC Proposal CLC Message Format
The fields in the SMC Proposal CLC message are 🎀
Eye catchers 👀 Like all CLC messages, the SMC Proposal has beginning and ending eye catchers to help with verification and parsing. The hex digits spell "SMCR" in IBM-1047 (EBCDIC).
Type 🦋 CLC message Type 1 indicates SMC Proposal.
Length 📏 The length of this CLC message. If this is an IPv4 flow, this value is 52. Otherwise, it is variable, depending upon how many prefixes are listed.
Version 🔥 Version of the SMC-R protocol. Version 1 is the only currently defined value.
Client's Peer ID 🫶 As described in Appendix A.2.1 above.
Client's preferred RoCE GID 🌈 The IPv6 address of the client's preferred RNIC on the RoCE fabric.
Client's preferred RoCE MAC address ✨ The MAC address of the client's preferred RNIC on the RoCE fabric. It is required, as some operating systems do not have neighbor discovery or ARP support for RoCE RNICs. Literally essential 💅
Offset to mask/prefix area 🧚 This provides the number of bytes that must be skipped after this field, to access the IPv4 Subnet Mask field and the fields that follow it. It allows for future growth of this signal. In this version of the architecture, this value is always zero.
Area for future growth 🍓 In this version of the architecture, this field does not exist. It shows where additional information may be inserted into the signal in the future. The Offset to mask/prefix area field must be used to skip over this area.
IPv4 Subnet Mask 💜 If this message is flowing over an IPv4 TCP connection, this is the value of the subnet mask associated with the interface over which the client sent this message. If this is an IPv6 flow, this field is all zeros.
This field, along with all fields that follow it in this signal, must be accessed by skipping the number of bytes listed in the Offset to mask/prefix area field after the end of that field.
IPv4 Mask Lgth 🌸 If this message is flowing over an IPv4 TCP connection, this is the number of significant bits in the IPv4 Subnet Mask field. If this is an IPv6 flow, this field is zero.
Num IPv6 prfx 👑 If this message is flowing over an IPv6 TCP connection, this is the number of IPv6 prefixes that follow, with a maximum value of 8. If this is an IPv4 flow, this field is zero and is immediately followed by the ending eye catcher.
Array of IPv6 prefixes 🔥 For IPv6 TCP connections, this is a list of the IPv6 prefixes associated with the network over which the client sent this message, up to a maximum of eight prefixes.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ IPv6 prefix value +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Prefix Length |
+-+-+-+-+-+-+-+-+Figure 27: Format for IPv6 Prefix Array Element 🧚✨
A.2.3. SMC Accept CLC Message Format ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 2 | Length = 68 |Version|F|Rsrvd|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Server's Peer ID -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- Server's RoCE GID -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Server's RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Server QP (bytes 1-2) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+
|Srvr QP byte 3 | Server RMB RKey (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Srvr RMB byte 4|Server RMB indx| Srvr RMB alert tkn (bytes 1-2)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Srvr RMB alert tkn (bytes 3-4)|Bsize | MTU | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Server's RMB virtual address -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Server's initial packet sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 28: SMC Accept CLC Message Format
Okay so the SMC Accept message has its own little field lineup, and here's the tea on each one 🫶
Eye catchers 👀
Like every CLC message, the SMC Accept wears beginning and ending eye catchers so it's easy to verify and parse. The hex digits spell "SMCR" in IBM-1047 (EBCDIC). Iconic, honestly. 🧚
Type 💖
CLC message Type 2 is the SMC Accept. ✨
Length 📏
The SMC Accept CLC message is 68 bytes long. 💅
Version 🌸
This is the version of the SMC-R protocol. Version 1 is the only one defined right now, so no drama there. 👑
F-bit 🎀
This is the first contact flag: a 1-bit flag that says the server thinks this TCP connection is the first SMC-R contact for this link group. 🦋
Server's Peer ID 🫧
Same deal as described in Appendix A.2.1 above. Go check that section if you forgot. 🙈
Server's RoCE GID 🌈
This is the IPv6 address of the RNIC the server picked for this SMC-R link. 💜
Server's RoCE MAC address 🍓
This is the MAC address of the server's RNIC for the SMC-R link. It's REQUIRED here, because some operating systems don't do neighbor discovery or ARP support for RoCE RNICs. (Lowkey a must-have, no cap.) 💀
Server's QP number ⭐
This is the number for the reliably connected queue pair the server made for this SMC-R link. 🔥
Server's RMB RKey 🔑
This is the RDMA RKey for the RMB the server created or chose for this TCP connection. 🎀
Server's RMB element index 🧩
This tells you which element inside the server's RMB represents this TCP connection. 🌸
Server's RMB element alert token 🔔
This is a platform-defined, architecturally opaque token that identifies this TCP connection. The client adds it as immediate data on RDMA writes it sends to the server, so the server knows there's data waiting for this connection to grab from the RMB element. Main character energy, fr fr. 🦋
Bsize: 📦
This is the server's RMB element buffer size in 4-bit compressed notation, where x = 4 bits. The real buffer size is (2^(x + 4)) * 1K. The smallest possible value is 16K, and the biggest size this architecture supports is 512K. ✨
MTU 📐
This is an enumerated value showing this peer's QP MTU size. The two peers swap their MTU values, and whichever one is smaller is the one that gets used for the QP. This field should only be validated in the first contact exchange. 💅
The enumerated MTU values are:
- 0: reserved
- 1: 256
- 2: 512
- 3: 1024
- 4: 2048
- 5: 4096
- 6-15: reserved
Server's RMB virtual address 🗺️
This is the virtual address of the server's RMB, as assigned by the server's RNIC. 🧚
Server's initial packet sequence number 🔢
This is the starting packet sequence number this peer will use when sending to the other peer. That way the other peer can get its QP ready for the sequence number it's about to see. Good communication, periodt. 💖
A.2.4. SMC Confirm CLC Message Format 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 3 | Length = 68 |Version| Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Client's Peer ID -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- Client's RoCE GID -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Client's RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Client QP (bytes 1-2) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+
|Clnt QP byte 3 | Client RMB RKey (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Clnt RMB byte 4|Client RMB indx| Clnt RMB alert tkn (bytes 1-2)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Clnt RMB alert tkn (bytes 3-4)|Bsize | MTU | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Client's RMB Virtual Address -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Client's initial packet sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 29: SMC Confirm CLC Message Format
The SMC Confirm is basically a twin of the SMC Accept, except it carries the client's info and doesn't have a first contact flag. Sisters, but make it client. 🧚
The fields present in the SMC Confirm CLC message are:
Eye catchers 👀
Like every CLC message, the SMC Confirm has beginning and ending eye catchers for easy verification and parsing. The hex digits spell "SMCR" in IBM-1047 (EBCDIC). Same iconic energy. ✨
Type 💖
CLC message Type 3 is the SMC Confirm. 🦋
Length 📏
The SMC Confirm CLC message is 68 bytes long. 💅
Version 🌸
This is the version of the SMC-R protocol. Version 1 is the only currently defined value. 👑
Client's Peer ID 🫧
Same as described in Appendix A.2.1 above. 🙈
Client's RoCE GID 🌈
This is the IPv6 address of the RNIC the client picked for this SMC-R link. 💜
Client's RoCE MAC address 🍓
This is the MAC address of the client's RNIC for the SMC-R link. It's REQUIRED, since some operating systems don't have neighbor discovery or ARP support for RoCE RNICs. 💀
Client's QP number ⭐
This is the number for the reliably connected queue pair the client created for this SMC-R link. 🔥
Client's RMB RKey 🔑
This is the RDMA RKey for the RMB the client created or chose for this TCP connection. 🎀
Client's RMB element index 🧩
This shows which element inside the client's RMB will represent this TCP connection. 🌸
Client's RMB element alert token 🔔
This is a platform-defined, architecturally opaque token that identifies this TCP connection. The server adds it as immediate data on RDMA writes it sends to the client, so the client knows there's data waiting for this connection in the RMB element. 🦋
Bsize: 📦
This is the client's RMB element buffer size in 4-bit compressed notation, where x = 4 bits. The actual buffer size value is (2^(x + 4)) * 1K. The smallest possible value is 16K, and the largest size this architecture supports is 512K. ✨
MTU 📐
This is an enumerated value showing this peer's QP MTU size. The two peers exchange their MTU values, and whichever is smaller gets used for the QP. The values are enumerated in Appendix A.2.3. This value should only be validated in the first contact exchange. 💅
Client's RMB Virtual Address 🗺️
This is the virtual address of the client's RMB, as assigned by the server's RNIC. 🧚
Client's initial packet sequence number 🔢
This is the starting packet sequence number this peer will use when sending to the other peer, so the other peer can set up its QP for the sequence number it expects. 💖
A.2.5. SMC Decline CLC Message Format 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 4 | Length = 28 |Version|S|Rsrvd|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Sender's Peer ID -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Peer Diagnosis Information |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 30: SMC Decline CLC Message Format
okay so the SMC Decline message has its own little layout, and we're going field by field 💅
The fields present in the SMC Decline CLC message are:
- Eye catchers 👀 Like all CLC messages, the SMC Decline has beginning and ending eye catchers to aid with verification and parsing. The hex digits spell "SMCR" in IBM-1047 (EBCDIC).
- Type 🎀 CLC message Type 4 indicates SMC Decline.
- Length 📏 The SMC Decline CLC message is 28 bytes long.
- Version 🌸 Version of the SMC-R protocol. Version 1 is the only currently defined value.
- S-bit 🔥 Sync Bit. Indicates that the link group is out of sync and the receiving peer must clean up its representation of the link group.
- Sender's Peer ID 🫶 As described in Appendix A.2.1 above.
- Peer Diagnosis Information 🩺 4 bytes of diagnosis information provided by the peer. These values are defined by the individual peers, and it is necessary to consult the peer's system documentation to interpret the results.
A.3. LLC Messages 💜
LLC messages are sent over an existing SMC-R link using RoCE SendMsg and are always 44 bytes long so that they fit into the space available in a single WQE without requiring the receiver to post receive buffers. If all 44 bytes are not needed, they are padded out with zeros. LLC messages are in a request/response format. The message type is the same for request and response, and a flag indicates whether a message is flowing as a request or a response.
The two high-order bits of an LLC message opcode indicate how it is to be handled by a peer that does not support the opcode.
If the high-order bits of the opcode are b'00', then the peer must support the LLC message and indicate a protocol error if it does not.
If the high-order bits of the opcode are b'10', then the peer must silently discard the LLC message if it does not support the opcode. This requirement is included to allow for toleration of advanced, but optional, functionality.
High-order bits of b'11' indicate a Connection Data Control (CDC) message as described in Appendix A.4.
A.3.1. CONFIRM LINK LLC Message Format ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 1 | Length = 44 | Reserved |R| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sender's RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| |
+- -+
| Sender's RoCE GID |
+- -+
| |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |Sender's QP number, bytes 1-2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Sender QP byte3| Link number |Sender's link userID, bytes 1-2|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Sender's link userID, bytes 3-4| Max links | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Reserved -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 31: CONFIRM LINK LLC Message Format
The CONFIRM LINK LLC message is required to be exchanged between the server and client over a newly created SMC-R link to complete the setup of an SMC-R link. Its purpose is to confirm that the RoCE path is actually usable.
On first contact, this message flows after the server receives the SMC Confirm CLC message from the client over the IP connection. For additional links added to an SMC-R link group, it flows after the ADD LINK and ADD LINK CONTINUATION exchange. This flow provides confirmation that the queue pair is in fact usable. Each peer echoes its RoCE information back to the other.
The contents of the CONFIRM LINK LLC message are:
- Type 🍓 Type 1 indicates CONFIRM LINK.
- Length 📏 The CONFIRM LINK LLC message is 44 bytes long.
- R 💖 Reply flag. When set, indicates that this is a CONFIRM LINK reply.
- Sender's RoCE MAC address 🌈 The MAC address of the sender's RNIC for the SMC-R link. It is required, as some operating systems do not have neighbor discovery or ARP support for RoCE RNICs.
- Sender's RoCE GID 🧚 The IPv6 address of the RNIC that the sender is using for this SMC-R link.
- Sender's QP number 🦋 The number for the reliably connected queue pair that the sender created for this SMC-R link.
- Link number ⭐ An identifier assigned by the server that uniquely identifies the link within the link group. This identifier is ONLY unique within a link group. Provided by the server and echoed back by the client.
- Link user ID 🎀 An opaque, implementation-defined identifier assigned by the sender and provided to the receiver solely for purposes of display, diagnosis, network management, etc. The link user ID should be unique across the sender's entire software space, including all other link groups.
- Max links 👑 The maximum number of links the sender can support in a link group. The maximum for this link group is the smaller of the values provided by the two peers.
A.3.2. ADD LINK LLC Message Format ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 2 | Length = 44 | Rsrvd |RsnCode|R|Z| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sender's RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| |
+- -+
| Sender's RoCE GID |
+- -+
| |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |Sender's QP number, bytes 1-2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Sender QP byte3| Link number |Rsrvd | MTU |Initial PSN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Initial PSN (continued) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -+
| Reserved |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 32: ADD LINK LLC Message Format
okay so the ADD LINK LLC message gets sent over an existing link in the link group whenever a peer wants to add an SMC-R link to an existing SMC-R link group 🫶 the server can send it to add a new link to the group, or the client can send it to ask the server to add one, like when a brand new RNIC becomes active 🌸 when the client sends it, that's basically a request for the server to kick off an ADD LINK exchange 💅
this message goes out right after the initial SMC-R link in the group finishes, which is described in Section 3.5.1 ("First Contact") ✨ it can also be sent over an existing SMC-R link group at any time as new RNICs get added and become available 🦋 so there could be just one new RMB RToken to share, or a whole bunch 😭 the RTokens get communicated using ADD LINK CONTINUATION messages 🧚
the contents of the ADD LINK LLC message are:
Type 🎀 Type 2 indicates ADD LINK.
Length 🍓 The ADD LINK LLC message is 44 bytes long.
RsnCode 💀 if the Z (rejection) flag is set, this field gives the reason code. the values can be:
X'1': no alternate path available. this gets set when the server provides the same MAC/GID as an existing SMC-R link in the group, and the client doesn't have any additional RNICs available (so the server is trying to set up an asymmetric link but none is available) 😩X'2': Invalid MTU value specified.
R 🌈 Reply flag. when set, it means this is an ADD LINK reply.
Z 🙈 Rejection flag. when set on a reply, it means the server's ADD LINK was rejected by the client. when this flag is set, the reason code is set too 🔥
Sender's RoCE MAC address 💜 the MAC address of the sender's RNIC for the new SMC-R link. it's required, because some operating systems don't have neighbor discovery or ARP support for RoCE RNICs 🫶
Sender's RoCE GID 🌸 the IPv6 address of the RNIC the sender is using for the new SMC-R link ✨
Sender's QP number 🦋 the number for the reliably connected queue pair that the sender created for the new SMC-R link 💅
Link number 👑 an identifier for the new SMC-R link. the server assigns it, and it uniquely identifies the link within the link group. this identifier is ONLY unique within a link group. it's provided by the server and echoed back by the client 🧚
MTU 🍓 an enumerated value showing this peer's QP MTU size. the two peers swap their MTU values, and whichever one is smaller is the one used for the QP. the values are enumerated in Appendix A.2.3. 🎀
Initial PSN 💖 the starting packet sequence number (PSN) this peer will use when sending to the other peer. that way the other peer can get its QP ready for the sequence number it should expect 🌈
A.3.3. ADD LINK CONTINUATION LLC Message Format 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 3 | Length = 44 | Reserved |R| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Linknum | NumRTokens | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- RKey/RToken pair -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- RKey/RToken pair or zeros -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 33: ADD LINK CONTINUATION LLC Message Format
okay this one is so good 😭 when a new SMC-R link gets added to an SMC-R link group, the new link's RTokens for the RMBs the group can access need to be shared 🫶 this message comes right after the ADD LINK and carries those RTokens 💅
the server starts this exchange by sending the first ADD LINK CONTINUATION LLC message, and the server runs the whole exchange as described below 👑
- if the client and the server need the same number of ADD LINK CONTINUATION messages to share their RTokens, the server kicks things off by sending the first ADD LINK CONTINUATION request to the client with its (the server's) RTokens. the client then replies with an ADD LINK CONTINUATION response carrying its RTokens, and they keep going like that until the exchange is done 🌸
- if the server needs more ADD LINK CONTINUATION messages than the client, then once the client has shared all of its RTokens, the server keeps sending ADD LINK CONTINUATION request messages to the client. the client keeps responding with empty ADD LINK CONTINUATION response messages (number of RTokens to be communicated = 0) 🧚
- if the client needs more ADD LINK CONTINUATION messages than the server, then after sharing all of its RTokens, the server keeps sending empty ADD LINK CONTINUATION messages to the client to draw out replies with the client's RTokens, until everything has been communicated 🦋
the contents of the ADD LINK CONTINUATION LLC message are:
Type 🎀 Type 3 indicates ADD LINK CONTINUATION.
Length 🍓 The ADD LINK CONTINUATION LLC message is 44 bytes long.
R 🌈 Reply flag. when set, it means this is an ADD LINK CONTINUATION reply.
LinkNum 💖 the link number of the new link within the SMC-R link group that the RKeys are being communicated for ✨
NumRTokens 💅 the number of RTokens still left to be communicated (this includes the ones in this message). if the value is less than or equal to 2, this is the last message. if it's greater than 2, another continuation message is needed, and its value will be this message's value minus 2, and so on until all RKeys have been communicated. the maximum value for this field is 255 🫶
RKey/RToken pairs (two or less) 🔥 these consist of an RKey for an RMB that's already known on the SMC-R link this message was sent over (the reference RKey), paired with that same RMB's RToken over the new SMC-R link. a full RToken isn't needed for the reference, because it's only there to tell which RMB it applies to, not to address it 😩
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reference RKey |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| New RKey |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- New Virtual Address -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 34: RKey/RToken Pair Format
the contents of the RKey/RToken pair are:
Reference RKey 🧚 the RKey of the RMB as it's already known on the SMC-R link this message is being sent over. it's required so the peer knows which RMB to link the new RToken with 💜
New RKey 🌈 the RKey of this RMB as it's known over the new SMC-R link 🦋
New Virtual Address ✨ the virtual address of this RMB as it's known over the new SMC-R link 👑
A.3.4. DELETE LINK LLC Message Format ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 4 | Length = 44 | Reserved |R|A|O| Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Linknum | reason code (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|RsnCode byte 4 | |
+-+-+-+-+-+-+-+-+ -+
| |
+- -+
| |
+- -+
| |
+- Reserved -+
| |
+- -+
| |
+- -+
| |
+- -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 35: DELETE LINK LLC Message Formatokay so DELETE LINK is the message you send when a QP or SMC-R link is going down, or needs to go down 🙈 The sender drops it over one of the other links in the link group. Lowkey the whole thing is about being polite about leaving the party. 🎀
Now the sender rules, bestie 💅 When the client sends DELETE LINK, it's ONLY a notification. The client then expects the server to reply by sending its own DELETE LINK request. To avoid races, only the server starts the actual DELETE LINK request and response sequence that comes from the client's notification. Iconic teamwork, no cap.
The server can also kick off a DELETE LINK all on its own, without the client saying anything first, if it notices an error or if orderly link termination has been started. 😭
The client can also ask to terminate the entire link group, and the server can terminate the entire link group using this same message. Main character energy: the whole group goes home. 🌈
The contents of the DELETE LINK LLC message are:
- Type
Type 4means DELETE LINK. 🦋
- Length
- The DELETE LINK LLC message is 44 bytes long. 🧚
- R
- Reply flag. When set, it means this is a DELETE LINK reply. 💜
- A
- "All" flag. When set, it means all links in the link group are to be terminated. Which also terminates the link group. Periodt. 👑
- O
- Orderly flag. This means orderly termination. Orderly termination usually comes from an operator command, not an error on the link. When the client asks for orderly termination, the server may wait to finish other work before it terminates. Very considerate, honestly. 🫶
- LinkNum
- The link number of the link being terminated. If the A flag is set, this field has no meaning and is set to 0. 🍓
- RsnCode
- The termination reason code. Here are the reason codes that are currently defined:
Request reason codes:
X'00010000'= Lost pathX'00020000'= Operator initiated terminationX'00030000'= Program initiated termination (link inactivity)X'00040000'= LLC protocol violationX'00050000'= Asymmetric link no longer needed
Response reason code:
X'00100000'= Unknown link ID (no link) 💀
A.3.5. CONFIRM RKEY LLC Message Format 🔥
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 6 | Length = 44 | Reserved |R|0|Z|C|Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NumTkns | New RMB RKey for this link (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|ThisLink byte 4| |
+-+-+-+-+-+-+-+-+ -+
| New RMB virtual address for this link |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+ -+
| |
+- Other link RMB specification or zeros -+
| |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -+
| |
+- -+
| Other link RMB specification or zeros |
+- +-+-+-+-+-+-+-+-+
| | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 36: CONFIRM RKEY LLC Message FormatOkay bestie, CONFIRM RKEY is the message that's like "hey, I just made a new RMB" or "hey, I deleted one" 🦋 It can be sent at ANY time, by either the client or the server. And the creator of a new RMB has to tell its peer the new RMB's RToken for all SMC-R links in the link group. Big responsibility, big ✨.
For RMB creation, the creator sends this message over the SMC-R link that the first TCP connection using the new RMB is using. The message holds the new RMB RToken for the SMC-R link it's sent over, and then lists the sender's other SMC-R links in the link group, each paired with the new RToken for that link. One message can carry new RTokens for three QPs: the QP for the link it was sent over, plus two others. If there are more than three links in the SMC-R link group, a CONFIRM RKEY CONTINUATION will be required. 🧩
The peer responds by just echoing the message back with the response flag set. 🪞 If the response is negative, the sender must recalculate the RToken set and start a brand new CONFIRM RKEY exchange from the beginning. When that retry happens is controlled by the C flag, which we'll get to below. 💅
The contents of the CONFIRM RKEY LLC message are:
- Type
Type 6means CONFIRM RKEY. 💖
- Length
- The CONFIRM RKEY LLC message is 44 bytes long. 🎀
- R
- Reply flag. When set, it means this is a CONFIRM RKEY reply. 🌸
- 0
- Reserved bit. 🫧
- Z
- Negative response flag. 🙅♀️
- C
- Configuration Retry bit. If this is a negative response and this flag is set, the originator should recalculate the RKey set and retry this exchange as soon as the current configuration change is completed. If this flag is NOT set on a negative response, the originator must wait for the next natural stimulus (for example, a new TCP connection started that requires a new RMB) before retrying. The C flag is basically the timing dial. 🕰️
- NumTkns
- The number of other link/RToken pairs to be communicated, including the ones provided in this message. Note that this value does NOT include the RToken for the link the message was sent on. So the max value is 2. If this value is 3 or less, this is the only message in the exchange. If this value is greater than 3, a CONFIRM RKEY CONTINUATION message will be required.
Note: In this version of the architecture, eight is the maximum number of links supported in a link group. 🌟
- New RMB RKey for this link
- The new RMB's RKey as assigned on the link the message is being sent over. 🔑
- New RMB virtual address for this link
- The new RMB's virtual address as assigned on the link the message is being sent over. 📍
- Other link RMB specification
- The new RMB's specification on the other links in the link group, laid out as shown in Figure 37. 🗺️
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Link number | RMB's RKey for the specified link (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|New RKey byte 4| |
+-+-+-+-+-+-+-+-+ -+
| RMB's virtual address for the specified link |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+
Figure 37: Format of Link Number/RKey Pairs- Link number
- The link number for a link in the link group. 🔢
- RMB's RKey for the specified link
- The RKey used to reach the RMB over the link whose number was specified in the Link number field. 🗝️
- RMB's virtual address for the specified link
- The virtual address used to reach the RMB over the link whose number was specified in the Link number field. 🏠
A.3.6. CONFIRM RKEY CONTINUATION LLC Message Format ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 8 | Length = 44 | Reserved |R|0|Z| Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NumTknsLeft | |
+-+-+-+-+-+-+-+-+ -+
| |
+- Other link RMB specification -+
| |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+ -+
| |
+- Other link RMB specification or zeros -+
| |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -+
| |
+- -+
| Other link RMB specification or zeros |
+- +-+-+-+-+-+-+-+-+
| | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 38: CONFIRM RKEY CONTINUATION LLC Message Format
okay so this one is the CONFIRM RKEY CONTINUATION message, and it's the little sidekick that shows up when the main CONFIRM RKEY message can't fit all the RMB RTokens 🎀. Each CONFIRM RKEY CONTINUATION message can hold up to three RMB RTokens, so it's giving group project energy, passing the baton until everyone's been introduced 🫶.
The NumTknsLeft field is basically a countdown 🧮. It tells you how many RMB RTokens are still left to be communicated, and that count includes the ones riding along in this very message. If the value is 3 or less, this is the last message in the group ✨. If it's 4 or higher, more CONFIRM RKEY CONTINUATION messages are coming, and the number keeps going down until everything has been shared 💅.
Just like the CONFIRM RKEY message, the peer responds by echoing the message back with the reply flag set 💖.
The contents of the CONFIRM RKEY CONTINUATION LLC message are:
Type: Type 8 indicates CONFIRM RKEY CONTINUATION. 🌸Length: The CONFIRM RKEY CONTINUATION LLC message is 44 bytes long. 📏R: Reply flag. When set, indicates that this is a CONFIRM RKEY CONTINUATION reply. 🦋0: Reserved bit. 🙈Z: Negative response flag. 💀NumTknsLeft: The number of link/RToken pairs, including those provided in this message, that are remaining to be communicated. If this value is 3 or less, this is the last message in the exchange. If this value is greater than 3, another CONFIRM RKEY CONTINUATION message will be required. Note that in this version of the architecture, eight is the maximum number of links supported in a link group. ⭐Other link RMB specification: The new RMB's specification on other links in the link group, as shown in Figure 37. 🧚
A.3.7. DELETE RKEY LLC Message Format ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 9 | Length = 44 | Reserved |R|0|Z| Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Count | Error Mask | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| First deleted RKey |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Second deleted RKey or zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Third deleted RKey or zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Fourth deleted RKey or zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Fifth deleted RKey or zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sixth deleted RKey or zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seventh deleted RKey or zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Eighth deleted RKey or zeros |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 39: DELETE RKEY LLC Message Format
the DELETE RKEY flow is the "bye bye, this RMB is gone" message 👋🍓, and it can be sent at any time from either the client or the server to tell the peer that one or more RMBs have been deleted. Here's the lowkey genius part: the peer already knows every RMB's RKey on each link in the link group, so this message only needs to name one RKey for each RMB being deleted 🧠. The RKey you send for each deleted RMB is the one it has on the SMC-R link this message is traveling over ✨.
You don't need the whole RToken for this. The RKey on its own is enough to identify an existing RMB, so we're keeping things slim and iconic 💅.
The peer responds by simply echoing the message back with the response flag set 🫶. If the peer didn't recognize an RKey, the negative response flag gets set. Still, no aggressive recovery action beyond logging the error will be taken, so nobody's going full drama queen over it 🙈.
The contents of the DELETE RKEY LLC message are:
Type: Type 9 indicates DELETE RKEY. 🌈Length: The DELETE RKEY LLC message is 44 bytes long. 📏R: Reply flag. When set, indicates that this is a DELETE RKEY reply. 🦋0: Reserved bit. 🙈Z: Negative response flag. 💀Count: Number of RMBs being deleted by this message. Maximum value is 8. 🔢Error Mask: If this is a negative response, indicates which RMBs were not successfully deleted. Each bit corresponds to a listed RMB; for example,b'01010000'indicates that the second and fourth RKeys weren't successfully deleted. ❌Deleted RKeys: A list of Count RKeys. Provided on the request flow and echoed back on the response flow. Each RKey is valid on the link over which this message is sent and represents a deleted RMB. Up to eight RMBs can be deleted in this message. 👑
A.3.8. TEST LINK LLC Message Format ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 7 | Length = 44 | Reserved |R| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- User Data -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- -+
| Reserved |
+- -+
| |
+- -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 40: TEST LINK LLC Message Format
the TEST LINK message is the vibe check 🩺💜. Either peer can send a TEST LINK request to the other on an existing SMC-R link, at any time, to make sure the link is still active and healthy at the software level. When a peer gets a TEST LINK LLC message, it immediately sends back a TEST LINK reply that echoes the user data right back ✨. For the related keepalive stuff, check out Section 4.5.3 ("TCP Keepalive Processing") 🌸.
The contents of the TEST LINK LLC message are:
Type: Type 7 indicates TEST LINK. 🎀Length: The TEST LINK LLC message is 44 bytes long. 📏R: Reply flag. When set, indicates that this is a TEST LINK reply. 🦋User Data: The receiver of this message echoes the sender's data back in a TEST LINK response LLC message. 🫧
A.4. Connection Data Control (CDC) Message Format 💌
okay bestie, the RMBE control data travels in Connection Data Control (CDC) messages 🎀 They go out with RoCE SendMsg, kinda like LLC messages do. And just like LLC messages, CDC messages are 44 bytes long, so they fit inside the private data areas of receive WQEs and the receiver doesn't have to post receive buffers 🙌
BUT here's the tea 🍵 CDC data is part of the actual data path, so it has to be prioritized and optimized like the rest of the data path processing. LLC messages can take the slower lane, but these ones cannot 😤
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
0 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = x'FE' | Length = 44 | Sequence number |
4 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SMC-R alert token |
8 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Producer cursor wrap seqno |
12 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Producer Cursor |
16 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Consumer cursor wrap seqno |
20 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Consumer Cursor |
24 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|B|P|U|R|F|Rsrvd|D|C|A| Reserved |
28 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
32 +- -+
| |
36 +- Reserved -+
| |
40 +- -+
| |
44 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 41: Connection Data Control (CDC) Message Format 📐
Type = x'FE' 🔥
This type number has the two high-order bits turned on, so processing can instantly tell it apart from an LLC message. Iconic. ✨
Length = 44 📏
The length of inline data that doesn't need a receive buffer to be posted.
Sequence number 🔢
A 2-byte unsigned integer that acts as a wrapping sequence number. It starts at 1 and can wrap back to 0. It goes up with every control message sent, except for the failover data validation message, and it keeps old control messages from being processed out of order. It's also used in failover data validation. Normally, if the number is less than the last one received, you toss the message. If it's greater, you process it. Old control messages can get lost with no ill effect, but they can never be processed after newer ones 💀
If this is a failover validation CDC message (F flag set), the receiver must check that it has received and fully processed the RDMA write that was described by the CDC message with the sequence number in this message. If it hasn't, the TCP connection must be reset to protect against data loss. The details are in Section 4.6.1 🫶
SMC-R alert token 🦋
The endpoint-assigned alert token that says which TCP connection on the link group this control message is talking about.
Producer cursor wrap seqno 🌀
A 2-byte unsigned integer that is a wrapping counter. The producer bumps it whenever the data written into this RMBE receive buffer causes a wrap (aka the producer cursor wraps). The receiver uses it to figure out when new data is available even when the cursors look unchanged. That happens when a full window size write is completed (the producer cursor of this RMBE sent by the peer equals the local consumer cursor), or when the producer cursor sent for this RMBE is less than the local consumer cursor 👀
Producer Cursor ➡️
A 4-byte unsigned integer that is a wrapping offset into the RMBE data area. It points to the next byte of data the sender will write. It can advance up to the receiver's consumer cursor as the sender knows it. When the urgent data present indicator is on, it points 1 byte past the last byte of urgent data. When computing this cursor, the presence of the eye catcher in the RMBE data area must be accounted for. The first writable location in the RMBE is at offset 4, so this cursor starts at 4 and wraps back to 4 🧚
Consumer cursor wrap seqno 🔁
A 2-byte unsigned integer that mirrors the producer cursor wrap sequence number from the last time this RMBE was read. It shows how far along the consumer is in reading data (like, did it get past the last wrap point or not). The producer side can use this to tell whether more data can be written to the partner during full window write scenarios (where the producer cursor equals the consumer cursor as known on the remote RMBE). In that case, if the consumer sequence number equals the local producer sequence number, the producer knows more data can be written 💅
Consumer Cursor 🌸
A 4-byte unsigned integer that is a wrapping offset into the sender's RMBE data area. It points to the offset of the next byte the peer will consume in its own RMBE. When computing this cursor, the presence of the eye catcher in the RMBE data area must be accounted for. The first writable location in the RMBE is at offset 4, so this cursor starts at 4 and wraps back to 4. The sender cannot write past this cursor into the peer's RMBE without causing data loss 🚨
B-bit 🙋♀️
Writer blocked indicator. The sender is blocked from writing. If this bit is set, the sender will need an explicit notification when receive buffer space becomes available.
P-bit 🚩
Urgent data pending. The sender has urgent data waiting for this connection.
U-bit 🚑
Urgent data present. Urgent data is in the RMBE data area, and the producer cursor points to 1 byte past the last byte of urgent data.
R-bit 📣
Request for consumer cursor update. An immediate consumer cursor update is requested, whether or not one would normally be warranted under the window size optimization algorithm described in Section 4.5.1.
F-bit 🛡️
Failover validation indicator. A peer sends this to guard against data loss during failover, when the TCP connection is being moved to another SMC-R link in the link group. When this bit is set, the only other fields in the CDC message that matter are the Type, Length, SMC-R alert token, and Sequence number fields. The receiver must validate that it has fully processed the RDMA write described by the previous CDC message with the same sequence number as this validation message. If it has, nothing else is needed. If it hasn't, the TCP connection must be reset. This is explained in detail in Section 4.6.1 🧠
D-bit ✅
Sending done indicator. A peer sends this when it's done writing new data into the receiver's RMBE data area.
C-bit 👋
PeerConnectionClosed indicator. A peer sends this when it's completely done with this connection and won't make any more updates to the receiver's RMBE or send any more control messages.
A-bit 💥
Abnormal close indicator. A peer sends this when the connection was terminated abnormally (for example, the TCP connection was reset). It means the peer is completely done with this connection and won't make any more updates to this RMBE or send any more control messages. It also means the RMBE owner must flush any remaining data on this connection and return an error code to any outstanding socket APIs on it (same processing as receiving an RST segment on a TCP connection). Dramatic, but iconic 😭
Appendix B. Socket API Considerations 🎀
Okay so a key design goal for SMC-R is that applications don't need any changes to use it 🙌 It's only for socket applications using stream (aka TCP) sockets over IPv4 or IPv6. Since the switch to the SMC-R protocol happens after a TCP connection is already established, the socket address family and the IP addresses and ports the apps use stay exactly the same. Existing socket APIs that let apps get the local and remote socket address structures for an established TCP connection (for example, getsockname() and getpeername()) keep working just like before. Existing DNS setup and APIs for turning hostnames into IP addresses and back also keep working with no changes. Generally, all the usual socket APIs for TCP (send APIs, recv APIs, etc.) keep working the way they do today, even when SMC-R is the underlying protocol ✨
BUT each SMC-R-enabled implementation does need to pay special attention to any socket APIs that rely on the underlying TCP and IP protocols 👀 It has to make sure they behave sensibly in an SMC-R environment and keep the impact on the application to a minimum. The basic socket API set is pretty similar across operating systems, but advanced socket API options vary more. Each implementation needs to do a detailed analysis of its API options, any possible impact SMC-R may have, and any resulting implications 🔍 As part of that, it would be useful to discuss or review things with other implementations that support SMC-R, so everyone stays consistent 🫶
B.1. setsockopt() / getsockopt() Considerations ✨
okay bestie, these two APIs let socket apps mess with socket, transport (TCP/UDP), and IP-level options on a given socket 💅. A platform usually limits how many IP options stream (TCP) socket apps get, since TCP is connection-oriented 🎀. The general vibe is to keep processing these APIs in a way that keeps existing apps working 🫶. Some options matter for SMC-R and need special handling "under the covers" 🔥. For example, changing TCP send and receive buffer sizes is still totally valid for SMC-R ✨. Other options might not mean anything for SMC-R 🙈. Like, if an app turns on TCP_NODELAY to disable Nagle's algorithm, it shouldn't really do anything for SMC-R communications, because Nagle's algorithm doesn't exist in this new protocol 💀. But the implementation still has to accept TCP_NODELAY like it does today and save it so getsockopt() can pull it back out later 💜. Also, any TCP or IP-level options will still affect the TCP/IP packets that flow for an SMC-R connection (so, during TCP/IP connection establishment and TCP/IP connection termination packet flows) 🦋.
Under the covers, handling TCP options also includes the SMC-layer setting, plus reading the SMC-R experimental option before and after the three-way TCP handshake is done 🌈.
Appendix C. Rendezvous Error Scenarios 💖
This section is all about the error scenarios for setting up and managing SMC-R links 😭.
C.1. SMC Decline during CLC Negotiation
A peer in the SMC-R CLC negotiation can send an SMC Decline instead of any expected CLC message 🙅♀️. That declines SMC and forces the TCP connection back onto the IP fabric 🦋. There can be a few reasons for an SMC Decline during CLC negotiation, including the following:
- RNIC went down
- SMC-R forbidden by local policy
- subnet (IPv4) or prefix (IPv6) doesn't match
- lack of resources to perform SMC-R
In all cases, when an SMC Decline is sent instead of an expected CLC message, no confirmation is required, and the TCP connection immediately falls back to using the IP fabric 🌸.
To prevent ambiguity between CLC messages and application data, an SMC Decline cannot "chase" another CLC message 🙈. An SMC Decline can only be sent instead of an expected CLC message 💅. For example, if the client sends an SMC Proposal and then its RNIC goes down, it must wait for the SMC Accept from the server and then reply to the SMC Accept with an SMC Decline 😭.
This "no chase" rule means that if this TCP connection is not a first contact between RoCE peers, a server cannot send an SMC Decline after sending an SMC Accept. It can only either break the TCP connection or fail over if a problem shows up in the RoCE fabric after it has sent the SMC Accept 💀. Similarly, once the client sends an SMC Confirm on a TCP connection that isn't a first contact, it is committed to SMC-R for this TCP connection and cannot fall back to IP 🔒.
C.2. SMC Decline during LLC Negotiation
For a TCP connection that represents a first contact between RoCE pairs, it is possible for SMC to fall back to IP during LLC negotiation 🧚. This is possible until the first contact SMC-R link is confirmed. For example, see Figure 42 👀. After a first contact SMC-R link is confirmed, fallback to IP is no longer possible. This turns into the following rule: a first contact peer can send an SMC Decline at any time during LLC negotiation until it has successfully sent its CONFIRM LINK (request or response) flow. After that point, it cannot fall back to IP 🫶.
Host X -- Server Host Y -- Client
+-------------------+ +-------------------+
| Peer ID = PS1 | | Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
| RKey X | |MAC MA|<-------------------->|MAC MB| | |
| | |GID GA| attempted setup |GID GB| | RKey Y2|
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| RMB | | | | RMB | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
| | |RNIC 3| |RNIC 4| | RKey W2|
| | |MAC MC| |MAC MD| | |
| QP 9 |GID GC| |GID GD| QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+
SYN / SYN-ACK / ACK TCP three-way handshake with TCP option
<--------------------------------------------------------->
SMC Proposal / SMC Accept / SMC Confirm exchange
<-------------------------------------------------------->
CONFIRM LINK(request, Link 1)
.........................................................>
CONFIRM LINK(response, Link 1)
X...................................
:
: RoCE write failure
:.................................>
SMC Decline(PC1, reason code)
<--------------------------------------------------------
Connection data flows over IP fabric
<------------------------------------------------------->
Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows
Figure 42: SMC Decline during LLC NegotiationC.3. The SMC Decline Window
Because SMC-R doesn't support fallback to IP for a TCP connection that's already using RDMA, there are specific rules on when the SMC Decline CLC message can be sent during TCP connection setup 🚦. That message signals a fallback to IP because of an error or problem with the RoCE fabric. There's a "point of no return" after which a connection cannot fall back to IP, and RoCE errors that happen after this point require the connection to be broken with a RST flow in the IP fabric 💔.
For a first contact, that point of no return is after the ADD LINK LLC message has been successfully sent for the second SMC-R link 🎯. Specifically, the server cannot fall back to IP after receiving either (1) a positive write completion indication for the ADD LINK request or (2) the ADD LINK response from the client, whichever comes first ✨. The client cannot fall back to IP after sending a negative ADD LINK response, receiving a positive write complete on a positive ADD LINK response, or receiving a CONFIRM LINK for the second SMC-R link from the server, whichever comes first 🔐.
For a subsequent contact, that point of no return is after the last send of the CLC negotiation completes 🏁. This, combined with the rule that error "chasers" aren't allowed during CLC negotiation, means that the server cannot send an SMC Decline after sending an SMC Accept, and the client cannot send an SMC Decline after sending an SMC Confirm 🙌.
C.4. Out-of-Sync Conditions during SMC-R Negotiation
The SMC Accept CLC message contains a first contact flag that tells the client whether the server thinks it's setting up a new link group or using an existing link group 👑. This flag is used to detect an out-of-sync condition between the client and the server 🕵️♀️. Here's the scenario: there's a single existing SMC-R link between the peers. After the client sends the SMC Proposal CLC message, the existing SMC-R link between the client and the server fails 😱. The client can't chase the SMC Proposal CLC message with an SMC Decline CLC message in this case, because the client doesn't yet know that the server would have wanted to choose the SMC-R link that just crashed 🙈. The QP that failed recovers before the server returns its SMC Accept CLC message. This means there's a QP but no SMC-R link 🌀. Since the server hadn't yet learned of the SMC-R link failure when it sent the SMC Accept CLC message, it tries to reuse the SMC-R link that just failed 💀. So the server would not set the first contact flag, which tells the client the server thinks it's reusing an SMC-R link. However, the client doesn't have an SMC-R link that matches the server's specification. Because the first contact flag is off, the client realizes it's out of sync with the server and sends an SMC Decline to cause the connection to fall back to IP 🍓.
C.5. Timeouts during CLC Negotiation
Because the SMC-R negotiation flows as TCP data, there are built-in timeouts and retransmits at the TCP layer for individual messages 🧸. Implementations also must protect the overall TCP/CLC handshake with a timer or timers, so connections don't hang forever because of SMC-R processing ⏰. This can be done with individual timers for individual CLC messages, or with an overall timer for the entire exchange, which may include the TCP handshake and the CLC handshake under one timer or separate timers. This decision is implementation dependent 🌷.
If the TCP and/or CLC handshakes time out, the TCP connection must be terminated the same way it would be in a legacy IP environment when connection setup doesn't complete in a timely manner 🥲. Because the CLC flows are TCP messages, if they can't be sent and received in a timely fashion, the TCP connection isn't healthy and wouldn't work if fallback to IP were attempted 🚫.
C.6. Protocol Errors during CLC Negotiation ✨
okay so a protocol error during CLC negotiation happens when a message shows up that nobody was expecting 💀. If a peer is waiting for a CLC message and gets application data instead, that's a protocol error 🙈. It also basically screams software bug, because the two sides are out of sync. And when application data is expected, we don't parse it to check whether it's secretly a CLC message 🫶.
When a peer is expecting a CLC negotiation message, any parsing error except a bad enumerated value in that message must be treated as application data. The CLC negotiation messages have beginning and ending eye catchers so you can double-check that the message is really the one you wanted 👀. If other parsing errors show up in an expected CLC message, like wrong length fields or badly formatted fields, the message must be treated as application data.
All protocol errors, except bad enumerated values, must result in termination of the TCP connection. No fallback to IP is allowed for a protocol error, because if things are out of sync, mismatched, or corrupted, data and security integrity can't be ensured 😭.
The exception is enumerated values, like the QP MTU values on SMC Accept and SMC Confirm. If a reserved value gets received, the proper response is to send an SMC Decline and fall back to IP 🦋. That's because a reserved enumerated value likely means the other partner has extra support that the receiving partner doesn't have. This mismatch in SMC-R capabilities isn't an integrity problem, it just means SMC-R can't be used for this connection 💅.
C.7. Timeouts during LLC Negotiation 💖
Whenever a peer sends an LLC message that expects a reply, it starts a timer right after the send posts, waiting for that reply 🧚. The expected response can be a reply flavor of the LLC message (like a CONFIRM LINK reply), or a brand new LLC message (like an ADD LINK CONTINUATION that the client expects from the server when there are more RKeys to share) 🍓.
On LLC flows that are part of a first contact setup of a link group, the timer value is implementation dependent, but it should be long enough for the other peer to hit a write complete timeout and do 2-3 retransmits of an SMC Decline on the TCP fabric 💅. For LLC flows that maintain the link group and aren't part of a first contact setup, the timers may be shorter. When the expected reply arrives, the timer gets cancelled. If a timer pops without a reply having been received, the sender must initiate a recovery action 🚨.
During first contact processing, a failed LLC verification timer is a "should-not-occur" that signals a problem with one of the endpoints 🙈. Here's the logic: if there's a "routine" failure in the RoCE fabric that makes an LLC verification send fail, the sender gets a write completion failure and then sends an SMC Decline to the partner. The only way an LLC verification timer expires on first contact is if the sender thinks the send succeeded but it actually didn't. Since QP connections on the RoCE fabric are reliably connected, that points to a problem with one of the peers, not with the RoCE fabric 💀.
After the reliably connected queue pair for the first SMC-R link in a link group is set up on initial contact, the client starts a timer to wait for a RoCE verification message from the server confirming the QP is actually connected and usable 🎀. If the server fails to send its QP confirmation message, it sends an SMC Decline, which should reach the client before the client's verification timer runs out. If the client's timer expires without an SMC Decline or a RoCE message confirmation from the server, there's a problem with either the server or the TCP fabric 😭. In either case, the client must break the TCP connection and clean up the SMC-R link.
There are two scenarios where the client's response to the QP verification message fails to reach the server 🍓. The main difference is whether or not the client successfully completed the send of the CONFIRM LINK response.
In the normal case of a problem with the RoCE path, the client learns of the failure from a write completion failure before the server's timer expires 🔥. The client then sends an SMC Decline CLC message to the server, and the TCP connection falls back to IP 🦋.
If the client's send of the confirmation message gets a positive return code but for some reason still doesn't reach the server, or the client's SMC Decline CLC message fails to reach the server after the client fails to send its RoCE confirmation message, then the server's timer times out and the server must break the TCP connection by sending a RST 💀. This is expected to be very rare, because if the client can't send its CONFIRM LINK response LLC message, it should get a negative return code and initiate fallback to IP ✨. A client receiving a positive return code on a send that fails to reach the server should also be an extremely rare case 👑.
C.7.1. Recovery Actions for LLC Timeouts and Failures ✨
okay so this list is all about what to do when an LLC timer runs out 😭 and a write completion failure, or any other sign of a send failure for an LLC command, gets treated exactly the same as a timeout. no cap 💀
CONFIRM LINKfrom server (first contact, first link in the link group) 🔥- timer waits for:
CONFIRM LINKreply from client. - recovery action: break the TCP connection by sending a
RST, and clean up the link. the server should have received an SMC Decline from the client by now if the client had an LLC send failure.
- timer waits for:
CONFIRM LINKfrom server (first contact, second link in the link group) 💅- timer waits for:
CONFIRM LINKreply from client. - recovery action: the second link was not successfully set up. send a
DELETE LINKto the client. connection data cannot flow in the first link in the link group until the reply to thisDELETE LINKis received, so the peers don't get out of sync on the state of the link group 🫶
- timer waits for:
CONFIRM LINKfrom server (not first contact) 🦋- timer waits for:
CONFIRM LINKreply from client. - recovery action: clean up the new link, and set a timer to retry. send a
DELETE LINKto the client, in case the client has a longer timer interval, so the client can stop waiting.
- timer waits for:
CONFIRM LINKreply from client (first contact) 🌸- timer waits for:
ADD LINKfrom server. - recovery action: clean up the SMC-R link, and break the TCP connection by sending a
RSTover the IP fabric. there is a problem with the server. if the server had a send failure, it should have sent an SMC Decline by now.
- timer waits for:
ADD LINKfrom server (first contact) 🧚- timer waits for:
ADD LINKreply from client. - recovery action: break the TCP connection with a
RST, and clean up RoCE resources. the connection is past the point where the server can fall back to IP, and if the client had a send problem it should have sent an SMC Decline by now.
- timer waits for:
ADD LINKfrom server (not first contact) 🍓- timer waits for:
ADD LINKreply from client. - recovery action: clean up resources (QP, RKeys, etc.) for the new link, and treat the link over which the
ADD LINKwas sent as if it had failed. if there is another link available to resend theADD LINKand the link group still needs another link, retry theADD LINKover another link in the link group.
- timer waits for:
ADD LINKreply from client (and there are more RKeys to be communicated) 💜- timer waits for:
ADD LINK CONTINUATIONfrom server. - recovery action: treat the same as
ADD LINKtimer failure.
- timer waits for:
ADD LINKreply orADD LINK CONTINUATIONreply from client (and there are no more RKeys to be communicated, for the second link in a first contact scenario) 👑- timer waits for:
CONFIRM LINKfrom the server, over the new link. - recovery action: the setup of the new link failed. send a
DELETE LINKto the server. do not consider the socket opened to the client application until receiving confirmation from the server in the form of aDELETE LINKrequest for this link and sending the reply, so the partners don't get out of sync on the state of the link group. - set a timer to send another
ADD LINKto the server if there is still an unused RNIC on the client side.
- timer waits for:
ADD LINKreply orADD LINK CONTINUATIONreply from client (and there are no more RKeys to be communicated) ⭐- timer waits for:
CONFIRM LINKfrom the server, over the new link. - recovery action: send a
DELETE LINKto the server for the new link, then clean up any resource allocated for the new link and set a timer to send anADD LINKto the server if there is still an unused RNIC on the client side. the setup of the new link failed, but the link over which theADD LINKexchange occurred is unaffected.
- timer waits for:
ADD LINK CONTINUATIONfrom server 🌈- timer waits for:
ADD LINK CONTINUATIONreply from client. - recovery action: treat the same as
ADD LINKtimer failure.
- timer waits for:
ADD LINK CONTINUATIONreply from client (first contact, and RMB count fields indicate that the server owes moreADD LINK CONTINUATIONmessages) 🎀- timer waits for:
ADD LINK CONTINUATIONfrom server. - recovery action: clean up the SMC-R link, and break the TCP connection by sending a
RST. there is a problem with the server. - if the server had a send failure, it should have sent an SMC Decline by now.
- timer waits for:
ADD LINK CONTINUATIONreply from client (not first contact, and RMB count fields indicate that the server owes moreADD LINK CONTINUATIONmessages) 🫧- timer waits for:
ADD LINK CONTINUATIONfrom server. - recovery action: treat as if the client detected a link failure on the link that the
ADD LINKexchange is using. send aDELETE LINKto the server over another active link if one exists; otherwise, clean up the link group.
- timer waits for:
DELETE LINKfrom client 💅- timer waits for:
DELETE LINKrequest from server. - recovery action: if the scope of the request is to delete a single link, the surviving link over which the client sent the
DELETE LINKis no longer usable either. if this is the last link in the link group, end TCP connections over the link group by sendingRSTpackets. if there are other surviving links in the link group, resend over a surviving link. also send aDELETE LINKover a surviving link for the link over which the client attempted to send the initialDELETE LINKmessage. if the scope of the request is to delete the entire link group, try resending on other links in the link group until success is achieved. if all sends fail, tear down the link group and any TCP connections that exist on it.
- timer waits for:
DELETE LINKfrom server (scope: entire link group) 🔥- timer waits for: confirmation from the adapter that the message was delivered.
- recovery action: tear down the link group and any TCP connections that exist on it.
DELETE LINKfrom server (scope: single link) 🦋- timer waits for:
DELETE LINKreply from client. - recovery action: the link over which the server sent the
DELETE LINKis no longer usable either. if this is the last link in the link group, end TCP connections over the link group by sendingRSTpackets. if there are other surviving links in the link group, resend over a surviving link. also send aDELETE LINKover a surviving link for the link over which the server attempted to send the initialDELETE LINKmessage. if the scope of the request is to delete the entire link group, try resending on other links in the link group until success is achieved. if all sends fail, tear down the link group and any TCP connections that exist on it.
- timer waits for:
CONFIRM RKEYfrom client 🌸- timer waits for:
CONFIRM RKEYreply from server. - recovery action: perform normal client procedures for detection of failed link. the link over which the message was sent has failed.
- timer waits for:
CONFIRM RKEYfrom server 🧚- timer waits for:
CONFIRM RKEYreply from client. - recovery action: perform normal server procedures for detection of failed link. the link over which the message was sent has failed.
- timer waits for:
TEST LINKfrom client 🍓- timer waits for:
TEST LINKreply from server. - recovery action: perform normal client procedures for detection of failed link. the link over which the message was sent has failed.
- timer waits for:
TEST LINKfrom server 💜- timer waits for:
TEST LINKreply from client. - recovery action: perform normal server procedures for detection of failed link. the link over which the message was sent has failed.
- timer waits for:
okay now the next bit is about invalid LLC messages, the ones that might be misformatted or carry out-of-sync data 🙈
CONFIRM LINKreceived from server 💀- what it indicates: incorrect link information.
- recovery action: protocol error. the link must be brought down by sending a
DELETE LINKfor the link over another link in the link group if one exists. if this is a first contact, fall back to IP by sending an SMC Decline to the server.
ADD LINKreceived 👑- what it indicates: undefined enumerated MTU value.
- recovery action: send a negative
ADD LINKreply with reason codex'2'.
ADD LINKreply received from client ⭐- what it indicates: client-side link information that would result in a parallel link being set up.
- recovery action: parallel links are not permitted. delete the link by sending a
DELETE LINKto the client over another link in the link group.
- any link group command received from the server, except
DELETE LINKfor the entire link group 🌈- what it indicates: the client has sent a
DELETE LINKfor the link on which the message was received. - recovery action: ignore the LLC message. worst case: the server will time out. best case: the
DELETE LINKcrosses with the command from the server, and the server realizes it failed.
- what it indicates: the client has sent a
ADD LINK CONTINUATIONfrom server orADD LINK CONTINUATIONreply received from client 🎀- what it indicates: the number of RMBs provided doesn't match the count given on the initial
ADD LINKorADD LINKreply message. - recovery action: protocol error. treat as if a link outage was detected.
- what it indicates: the number of RMBs provided doesn't match the count given on the initial
DELETE LINKreceived from client 🫧- what it indicates: the link indicated doesn't exist.
- recovery action: if the link is in the process of being cleaned up, assume a timing window and ignore the message. otherwise, send a
DELETE LINKreply with reason code1.
DELETE LINKreceived from server 💅- what it indicates: the link indicated doesn't exist.
- recovery action: send a
DELETE LINKreply with reason code1.
CONFIRM RKEYreceived from either client or server 🔥- what it indicates: no RKey provided for one or more of the links in the link group.
- recovery action: treat as if the failure of the link(s) for which no RKey was provided was detected.
DELETE RKEYreceived 🦋- what it indicates: the specified RKey doesn't exist.
- recovery action: send a negative
DELETE RKEYresponse.
TEST LINKreply received 🌸- what it indicates: user data doesn't match what was sent in the
TEST LINKrequest. - recovery action: treat as if the link was detected to have gone down. this is a protocol error.
- what it indicates: user data doesn't match what was sent in the
- unknown LLC type with high-order bits of opcode equal to
b'10'received 🧚- what it indicates: this is an optional LLC message that the receiver does not support.
- recovery action: ignore (silently discard) the message.
- any unambiguously incorrect or out-of-sync LLC message received 🍓
- what it indicates: the link is out of sync.
- recovery action: treat as if the link was detected to have gone down. note that an unsupported or unknown LLC opcode whose two high-order bits are
b'10'is not an error and must be silently discarded. any other unknown or unsupported LLC opcode is an error. 💜
C.8. Failure to Add Second SMC-R Link to a Link Group
okay so, if the second SMC-R link in an SMC-R link group fails to get set up, including when the confirmation timer expires, the link group is allowed to keep going, but it's without available failover 😬. ngl this is extremely undesirable, so the server must endeavor to fix it as soon as it can 🫶
the server peer in the link group must set a timer to drive it to retry setting up the failed additional SMC-R link 🦋 and the server will retry the setup right away when the first of these happens:
- the retry timer expires ⏰
- a new RNIC becomes available to the server, on the same LAN as the SMC-R link group 💖
- an ADD LINK LLC request message is received from the client, which means there's a new RNIC on the client side ✨
bestie, that's the whole retry plan, and it's giving very "never give up" energy 💅🌈
Authors' Addresses ✨
Mike Fox 🎀
IBM
3039 Cornwallis Rd.
Research Triangle Park, NC 27709
United StatesEmail: mjfox@us.ibm.com
Constantinos (Gus) Kassimis 🦋
IBM
3039 Cornwallis Rd.
Research Triangle Park, NC 27709
United StatesEmail: kassimis@us.ibm.com
Jerry Stevens 🧚
IBM
3039 Cornwallis Rd.
Research Triangle Park, NC 27709
United StatesEmail: sjerry@us.ibm.com
📦 where this page comes from
- ietf: rfc7609 2015-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms