RFC 5353

Endpoint Handlespace Redundancy Protocol (ENRP)

📅 published September 2008, from ietf

Endpoint Handlespace Redundancy Protocol (ENRP) 🎀

Okay bestie, the header is giving main character energy 💅 This is RFC 5353, published under the Network Working Group in September 2008. It's an Experimental doc, not a standard 🧪. The authors are Q. Xie, R. Stewart (The Resource Group), M. Stillman (Nokia), M. Tuexen (Muenster Univ. of Applied Sciences), and A. Silverton (Sun Microsystems, Inc.) 👯‍♀️. There are no "Obsoletes" or "Updates" lines in the header, so this one isn't replacing or changing anything else ✨.

Status of This Memo 📮

This memo defines an Experimental Protocol for the Internet community, so think of it as a test drive 🚗. It does not specify an Internet standard of any kind 🙈. Discussion and suggestions for improvement are requested, so bestie, weigh in 💬. Distribution of this memo is unlimited 🌈.

Abstract 🦋

ENRP, the Endpoint Handlespace Redundancy Protocol, is built to team up with the Aggregate Server Access Protocol (ASAP) so that the Reliable Server Pooling (RSerPool) requirements and architecture actually get done 🛠️. Inside RSerPool's operational scope, ENRP lays out the procedures and message formats for a distributed, fault-tolerant registry service. That service stores, keeps the books on, retrieves, and distributes pool operation and membership information 🧚.

Table of Contents 📑

Here's the whole map, every section and subsection, so you can find your way around 🗺️

   1. Introduction ....................................................3
      1.1. Definitions ................................................3
      1.2. Conventions ................................................4
   2. ENRP Message Definitions ........................................4
      2.1. ENRP_PRESENCE Message ......................................5
      2.2. ENRP_HANDLE_TABLE_REQUEST Message ..........................6
      2.3. ENRP_HANDLE_TABLE_RESPONSE Message .........................7
      2.4. ENRP_HANDLE_UPDATE Message .................................9
      2.5. ENRP_LIST_REQUEST Message .................................10
      2.6. ENRP_LIST_RESPONSE Message ................................11
      2.7. ENRP_INIT_TAKEOVER Message ................................12
      2.8. ENRP_INIT_TAKEOVER_ACK Message ............................13
      2.9. ENRP_TAKEOVER_SERVER Message ..............................14
      2.10. ENRP_ERROR Message .......................................15

   3. ENRP Operation Procedures ......................................15
      3.1. Methods for Communicating amongst ENRP Servers ............16
      3.2. ENRP Server Initialization ................................16
           3.2.1. Generate a Server Identifier .......................16
           3.2.2. Acquire Peer Server List ...........................17
                  3.2.2.1. Finding the Mentor Server .................17
                  3.2.2.2. Request Complete Server List from
                           Mentor Peer ...............................17
           3.2.3. Download ENRP Handlespace Data from Mentor Peer ....18
      3.3. Server Handlespace Update .................................20
           3.3.1. Announcing Additions or Updates of PE ..............20
           3.3.2. Announcing Removal of PE ...........................21
      3.4. Maintaining Peer List and Monitoring Peer Status ..........22
           3.4.1. Discovering New Peer ...............................22
           3.4.2. Server Sending Heartbeat ...........................22
           3.4.3. Detecting Peer Server Failure ......................23
      3.5. Taking Over a Failed Peer Server ..........................23
           3.5.1. Initiating Server Take-over Arbitration ............23
           3.5.2. Takeover Target Peer Server ........................24
      3.6. Handlespace Data Auditing and Re-synchronization ..........25
           3.6.1. Auditing Procedures ................................25
           3.6.2. PE Checksum Calculation Algorithm ..................26
           3.6.3. Re-Synchronization Procedures ......................27
      3.7. Handling Unrecognized Messages or Unrecognized
           Parameters ................................................28
   4. Variables and Thresholds .......................................28
      4.1. Variables .................................................28
      4.2. Thresholds ................................................28
   5. IANA Considerations ............................................28
      5.1. A New Table for ENRP Message Types ........................29
      5.2. A New Table for Update Action Types .......................29
      5.3. Port Numbers ..............................................30
      5.4. SCTP Payload Protocol Identifier ..........................30
   6. Security Considerations ........................................30
      6.1. Summary of RSerPool Security Threats ......................30
      6.2. Implementing Security Mechanisms ..........................32
      6.3. Chain of Trust ............................................34
   7. Acknowledgments ................................................35
   8. References .....................................................36
      8.1. Normative References ......................................36
      8.2. Informative References ....................................37

1. Introduction ✨

ENRP is basically ASAP's sidekick 💜 It's designed to work in conjunction with ASAP [RFC5352] so that RSerPool does everything its requirements [RFC3237] ask for. Together they make a whole squad 🫶.

Inside RSerPool's operational scope, ENRP sets the procedures and message formats for a distributed, fault-tolerant registry service. Its job is storing, bookkeeping, retrieving, and distributing pool operation and membership information 📚. Iconic work, honestly 👑.

Later in this document, we'll call that registry service the ENRP handlespace, or just handlespace, because it manages all the pool handles 🍓.

1.1. Definitions 📖

This document uses the following terms, so let's get the vocab down first 💅

Operational scope: The part of the network that pool users can see for a specific instance of the reliable server pooling protocols 👀.
Pool (or server pool): A group of servers all providing the same application functionality 🏊.
Pool handle: A logical pointer to a pool. Every server pool can be identified within the system's operational scope by its own unique pool handle 🔖.
Pool element: A server entity that has registered to a pool 📝.
Pool user: A user of a server pool 🙋.
Pool element handle (or endpoint handle): A logical pointer to one particular pool element in a pool. It's made up of the pool handle plus the pool element's destination transport address 📍.
Handle space: A tight-knit structure of pool handles and their relations, which an internal or external agent can query 🕸️.
ENRP client channel: The communication channel an ASAP User (either a Pool Element (PE) or a Pool User (PU)) uses to request ENRP handlespace service. It's usually defined by the transport address of the Home ENRP server plus a well-known port number 📞.
ENRP server channel: Defined by a list of IP addresses (one for each ENRP server in an operational scope) plus a well-known port number 🛰️. All ENRP servers in an operational scope can send "group-cast" messages to other servers through this channel. In a "group-cast", the sending server sends multiple copies of the message, one to each of its peer servers, over a set of point-to-point Stream Control Transmission Protocol (SCTP) associations between the sender and the peers 💌. The "group-cast" may be conveniently implemented with the use of the SCTP_SENDALL option on a one-to-many style SCTP socket ⚡.
Home ENRP server: The ENRP server that a PE or PU currently belongs to 🏠. A PE MUST only have one Home ENRP server at any given time, and both the PE and its Home ENRP server MUST keep track of this master/slave relationship between them. A PU SHOULD select one of the available ENRP servers as its Home ENRP server 🫡.

1.2. Conventions 📏

Here's the key words check 🔑. When you see the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document, they're to be interpreted as described in [RFC2119] 🧠.

2. ENRP Message Definitions 💌

In this section, we define the format of all ENRP messages 📐. These are the messages ENRP servers send and receive among themselves in an operational scope. Messages sent and received between a PE/PU and an ENRP server are part of ASAP and are defined in [RFC5352]. A common format, defined in [RFC5354], is used for all ENRP and ASAP messages 🎀.

Most ENRP messages mix fixed fields with TLV (Type-Length-Value) parameters 🧩. The TLV parameters are also defined in [RFC5354]. If a nested TLV parameter doesn't end on a 32-bit word boundary, it gets padded with all '0' octets up to the next 32-bit word boundary 🪡.

All messages, as well as their fields and parameters described below, MUST be transmitted in network byte order (aka Big Endian, meaning the most significant byte is transmitted first) 🔢. That part is non-negotiable, bestie 🙅‍♀️.

For ENRP, the following message types are defined in this section 📋:

         Type       Message Name
         -----      -------------------------
         0x00      - (Reserved by IETF)
         0x01      - ENRP_PRESENCE
         0x02      - ENRP_HANDLE_TABLE_REQUEST
         0x03      - ENRP_HANDLE_TABLE_RESPONSE
         0x04      - ENRP_HANDLE_UPDATE
         0x05      - ENRP_LIST_REQUEST
         0x06      - ENRP_LIST_RESPONSE
         0x07      - ENRP_INIT_TAKEOVER
         0x08      - ENRP_INIT_TAKEOVER_ACK
         0x09      - ENRP_TAKEOVER_SERVER
         0x0a      - ENRP_ERROR
         0x0b-0xff - (Reserved by IETF)

                                 Figure 1

2.1. ENRP_PRESENCE Message ✨

okay so this ENRP message is the one that keeps announcing (periodically) that an ENRP server is alive 💖, or pokes a peer ENRP server to check on its status 🦋. It goes out on the ENRP server channel or gets sent point-to-point to another ENRP server. main character energy 💅

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Type = 0x01 |0|0|0|0|0|0|0|0|        Message Length         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Sending Server's ID                       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                    Receiving Server's ID                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      :                      PE Checksum Param                        :
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      :               Server Information Param (optional)             :
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Sending Server's ID: 32 bits (unsigned integer) 🧚

This is the ID of the ENRP server that sent this message.

Receiving Server's ID: 32 bits (unsigned integer) 🌸

This is the ID of the ENRP server to which this message is intended. If the message is not intended for an individual server (e.g., the message is group-casted to a group of servers), this field MUST be sent with all 0s. If the message is sent point-to-point, this field MAY be sent with all 0s.

PE Checksum Parameter: 💜

This is a TLV that contains the latest PE checksum of the ENRP server that sends the ENRP_PRESENCE. This parameter SHOULD be included for handlespace consistency auditing. See Section 3.6.1 for details.

Server Information Parameter: 🎀

If this parameter is present, it contains the server information of the sender of this message (the Server Information Parameter is defined in [RFC5354]). This parameter is optional. However, if this message is sent in response to a received "reply required" ENRP_PRESENCE from a peer, the sender then MUST include its server information.

Note, at startup, an ENRP server MUST pick a randomly generated, non-zero 32-bit unsigned integer as its ID and MUST use this same ID until the ENRP server is rebooted. 🔥

2.2. ENRP_HANDLE_TABLE_REQUEST Message 🌈

so an ENRP server sends this one to one of its peers when it wants a copy of the handlespace data 🫶. It's mostly used during server initialization or when the handlespace needs re-synchronizing 🔄.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Type = 0x02 |0|0|0|0|0|0|0|W|    Message Length = 0xC       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Sending Server's ID                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Receiving Server's ID                     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

W (oWn-children-only) Flag: 1 bit 🦋

Set to '1' if the sender of this message is only requesting information about the PEs owned by the message receiver. Otherwise, set to '0'.

Sending Server's ID: 🧚

See Section 2.1.

Receiving Server's ID: 🌸

See Section 2.1.

2.3. ENRP_HANDLE_TABLE_RESPONSE Message 💅

the PEER_NAME_TABLE_RESPONSE message is what an ENRP server sends back after getting a PEER_NAME_TABLE_REQUEST, to help a peer get set up or sync its handlespace 🎀.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Type = 0x03 |0|0|0|0|0|0|M|R|        Message Length         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Sending Server's ID                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Receiving Server's ID                     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      :                                                               :
      :                     Pool Entry #1 (optional)                  :
      :                                                               :
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      :                                                               :
      :                              ...                              :
      :                                                               :
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      :                                                               :
      :                     Pool Entry #n (optional)                  :
      :                                                               :
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

M (More_to_send) Flag: 1 bit 🌈

Set to '1' if the sender of this message has more pool entries to send in subsequent ENRP_HANDLE_TABLE_RESPONSE messages. Otherwise, set to '0'.

R (Reject) Flag: 1 bit ⭐

R (Reject) Flag: 1 bit is the "I'm not taking requests rn" flag 💀. It MUST be set to '1' if the sender of this message is rejecting a handlespace request. In this case, pool entries MUST NOT be included. This might happen if the sender of this message is in the middle of initializing its database or is under high load.

Message Length: 16 bits (unsigned integer) 🍓

Indicates the entire length of the message, including the header, in number of octets.

Note, the value in the Message Length field will not cover any padding at the end of this message.

Sending Server's ID: 🧚

See Section 2.1.

Receiving Server's ID: 🌸

See Section 2.1.

Pool Entry #1-#n: 👑

If the R flag is set to '0', at least one pool entry SHOULD be present in this message. Each pool entry MUST start with a Pool Handle parameter, as defined in Section 3.9 of [RFC5354], and is followed by one or more Pool Element parameters in TLV format, as shown below:

          +---------------------------+
          :      Pool Handle          :
          +---------------------------+
          :         PE #1             :
          +---------------------------+
          :         PE #2             :
          +---------------------------+
          :          ...              :
          +---------------------------+
          :         PE #n             :
          +---------------------------+

2.4. ENRP_HANDLE_UPDATE Message 🦋

the PEER_NAME_UPDATE message is sent by the Home ENRP server of a PE to all its peer servers to announce a registration, re-registration, or de-registration of that PE in the handlespace 📣.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Type = 0x04 |0|0|0|0|0|0|0|0|        Message Length         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Sending Server's ID                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Receiving Server's ID                     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |        Update Action          |        (reserved)             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      :                     Pool Handle Parameter                     :
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      :                    Pool Element Parameter                     :
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Message Length: 16 bits (unsigned integer) 🍓

Indicates the entire length of the message, including the header, in number of octets.

Note, the value in the Message Length field will not cover any padding at the end of this message.

Update Action: 16 bits (unsigned integer) 🔥

This field indicates the requested action of the specified PE. The field MUST be set to one of the following values:

0x0000 - ADD_PE: Add or update the specified PE in the ENRP handlespace.
0x0001 - DEL_PE: Delete the specified PE from the ENRP handlespace.
0x0002 - 0xFFFF: Reserved by IETF.

Other values are reserved by IETF and MUST NOT be used.

Reserved: 16 bits 🌙

This field MUST be set to all 0s by the sender and ignored by the receiver.

Sending Server's ID: 🧚

See Section 2.1.

Receiving Server's ID: 🌸

See Section 2.1.

Pool Handle: 💖

Specifies to which the PE belongs.

Pool Element: 💜

Specifies the PE.

2.5. ENRP_LIST_REQUEST Message 🌸

the PEER_LIST_REQUEST message is sent to ask for a current copy of the ENRP server list 📋. It's usually sent from a newly activated ENRP server to an established one as part of initialization 🌱.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Type = 0x05 |0|0|0|0|0|0|0|0|    Message Length = 0xC       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Sending Server's ID                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Receiving Server's ID                     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Sending Server's ID: 🧚

See Section 2.1.

Receiving Server's ID: 🌸

See Section 2.1.

2.6. ENRP_LIST_RESPONSE Message 🧚

okay so the PEER_LIST_RESPONSE message is what an ENRP server sends back when it gets a PEER_LIST_REQUEST. It's basically the server spilling the tea on all the other ENRP servers it knows about 💅

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Type = 0x06 |0|0|0|0|0|0|0|R|        Message Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sending Server's ID                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Receiving Server's ID                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
:            Server Information Parameter of Peer #1            :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
:                           ...                                 :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
:            Server Information Parameter of Peer #n            :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

R (Reject) Flag: 1 bit 🙈

Bestie, this one's a bouncer. The R flag MUST be set to '1' if the sender is rejecting a PEER_LIST_REQUEST message. When that happens, the message MUST NOT include any Server Information Parameters. No guest list, no vibes 🚫

Message Length: 16 bits (unsigned integer) 📏

This tells you the whole length of the message in octets. Quick heads up: the Message Length value does NOT count any padding at the very end of the message, so don't add that in 👀

Sending Server's ID: see Section 2.1 🫶

Receiving Server's ID: see Section 2.1 🫶

Server Information Parameter of Peer #1-#n: each one holds a Server Information Parameter for a peer the sender knows about. That parameter is defined in [RFC5354] 🦋

2.7. ENRP_INIT_TAKEOVER Message 👑

the ENRP_INIT_TAKEOVER message is when an ENRP server (the takeover initiator) announces it's about to take over a specific peer ENRP server. It gets sent to every single one of its peers. Main character energy, literally 🔥

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Type = 0x07 |0|0|0|0|0|0|0|0|        Message Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sending Server's ID                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Receiving Server's ID                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Targeting Server's ID                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Sending Server's ID: see Section 2.1 🫶

Receiving Server's ID: see Section 2.1 🫶

Targeting Server's ID: 32 bits (unsigned integer) 🎯

This is the ID of the peer ENRP that's the target of this takeover attempt. Someone's getting taken over, and this is how you name them 😭

2.8. ENRP_INIT_TAKEOVER_ACK Message 🌸

the PEER_INIT_TAKEOVER_ACK message gets sent back to the takeover initiator. It's the "yeah, I see you, and I don't object" reply. A total green-light, no drama 💖

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Type = 0x08 |0|0|0|0|0|0|0|0|        Message Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sending Server's ID                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Receiving Server's ID                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Targeting Server's ID                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Sending Server's ID: see Section 2.1 🫶

Receiving Server's ID: see Section 2.1 🫶

Targeting Server's ID: this is the ID of the peer ENRP that's the target of this takeover attempt 🧚

2.9. ENRP_TAKEOVER_SERVER Message 📣

the PEER_TAKEOVER_REGISTRAR message is sent by the takeover initiator to officially declare that a takeover is happening, and it goes out to all the active peer ENRP servers. It's basically the group announcement 🎀

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Type = 0x09 |0|0|0|0|0|0|0|0|        Message Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sending Server's ID                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Receiving Server's ID                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Targeting Server's ID                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Sending Server's ID: see Section 2.1 🫶

Receiving Server's ID: see Section 2.1 🫶

Targeting Server's ID: this is the ID of the peer ENRP that's the target of this takeover operation 🎯

2.10. ENRP_ERROR Message 🚨

the ENRP_ERROR message is how a registrar reports an operational error to a peer ENRP server. Not the vibe, but it's good to know about 💀

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Type = 0x0a |0|0|0|0|0|0|0|0|        Message Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sending Server's ID                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Receiving Server's ID                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
:                 Operational Error Parameter                   :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Sending Server's ID: see Section 2.1 🫶

Receiving Server's ID: see Section 2.1 🫶

Operational Error Parameter: this parameter, defined in [RFC5354], tells you what type of error (or errors) is being reported 🦋

3. ENRP Operation Procedures 🌈

okay, this is where the procedures live. In this section we go over the operation procedures ENRP defines, and an ENRP server MUST follow them when it's sending, receiving, or processing ENRP messages. Rules are rules, bestie 👑

A lot of RSerPool events need both server-to-server AND PU/PE-to-server message exchanges. But only the exchanges and activities between an ENRP server and its peers count as ENRP scope, and that's what this document defines. The PE/PU and ENRP server stuff lives in [RFC5352] 📖

3.1. Methods for Communicating amongst ENRP Servers 💬

inside an RSerPool operational scope, ENRP servers have to talk to each other to swap info, like pool membership changes, handlespace data synchronization, and so on 🍓

there are two kinds of communication happening between ENRP servers:

point-to-point message exchanges from one ENPR server to a specific peer server, and
announcements from one server to all its peer servers in the operational scope.

point-to-point communication always happens over an SCTP association between the sending server and the receiving server. Announcements go out as "group-casts" over the ENRP server channel. Group chat, but make it networking 📣

3.2. ENRP Server Initialization 🐣

this section covers the steps a new ENRP server needs to take to join the other existing ENRP servers. Or, if it's the first ENRP server started in the operational scope, it covers how to kick off the handlespace service 🚀

3.2.1. Generate a Server Identifier 🆔

a new ENRP server MUST generate a non-zero, 32-bit server ID that's as unique as possible among all the ENRP servers in the operational scope. And that server ID MUST remain unchanged for the entire lifetime of the server. Its identity, its forever ✨ Usually a good 32-bit random number does the trick. The server ID [RFC4086] gives some info on randomness guidelines if you want to dig in 🧠

Okay, real talk: there's a very remote chance (about 1 in about 4 billion) that two ENRP servers in an operational scope end up with the same server ID, which causes an ID conflict in the pool. Still, no severe consequence of that conflict has been identified. Low-key a relief 🙏

also, the ENRP server ID space is separate from the PE Id space defined in [RFC5352]. Different lanes, no crossing 🛣️

3.2.2. Acquire Peer Server List 🗂️

at startup, the ENRP server (the initiating server) first tries to learn about all the existing peer ENRP servers in the same operational scope. Or it figures out that it's totally alone in the scope. Either way, it's getting the full picture 🖼️

the initiating server uses an existing peer server to bootstrap itself into service. That peer is called the mentor server 🧑‍🏫

3.2.2.1. Finding the Mentor Server 🔍

if the initiating server is told about one existing peer server through some administrative means (like a DNS query, a configuration database, startup scripts, etc.), the initiating server MUST then use that peer server as its mentor server. Easy, one choice, no decision fatigue 💅

if multiple existing peer servers are specified, the initiating server MUST pick one of them as its mentor server and keep the others as its backup mentor servers. Think of it as a main pick with backups on standby 🫶

if no existing peer server is specified, the initiating server MUST assume that it's alone in the operational scope. It MUST skip the procedures in Section 3.2.2.2 and Section 3.2.3, and it MUST consider its initialization completed and start offering ENRP services. Solo era, and it's still iconic ✨

3.2.2.2. Request Complete Server List from Mentor Peer 🎀

okay so once the initiating server finds its mentor peer (whether through discovery or admin setup), it MUST send an ENRP_LIST_REQUEST message to that mentor to ask for a copy of the complete server list the mentor keeps. (see Section 3.4 for how that list gets maintained 💅)

The initiating server SHOULD start a MAX-TIME-NO-RESPONSE timer every time it finishes sending an ENRP_LIST_REQUEST message. If the timer expires before it hears back from the mentor peer, the initiating server SHOULD abandon that mentor and send a new server list request to a backup mentor peer, if one is available. ⏰

Once the mentor gets this request, it SHOULD reply with an ENRP_LIST_RESPONSE message and include every existing ENRP server it knows about in the message body. 🧚

When the initiating server gets that ENRP_LIST_RESPONSE from the mentor peer, it MUST use the server information in it to set up its own peer list. ✨

ngl this next part is the one where the mentor might be too busy to help. 🙈

If the mentor is still starting up and isn't ready to hand out a peer list (for example, it's waiting on a reply to its own ENRP_LIST_REQUEST to another server), it MUST reject the request and send back an ENRP_LIST_RESPONSE with the R flag set to '1' and no server information in the response. 😭

If its ENRP_LIST_REQUEST gets rejected, the initiating server SHOULD either wait a few seconds and re-send the ENRP_LIST_REQUEST to the same mentor, or, if a backup mentor peer is available, pick another mentor and send the ENRP_LIST_REQUEST there instead. 🦋

3.2.3. Download ENRP Handlespace Data from Mentor Peer 🌸

after the peer list download is done, the initiating server MUST ask its mentor peer for a copy of the current handlespace data. here's how it goes:

The initiating server MUST first send an ENRP_HANDLE_TABLE_REQUEST message to the mentor peer, with the W flag set to '0', which means it's asking for the entire handlespace. 🍓
When the mentor gets this message, it MUST start a download session. It sends a copy of its current handlespace data to the initiating server in one or more ENRP_HANDLE_TABLE_RESPONSE messages. (Note, the mentor might find it especially smart to split things up into multiple ENRP_HANDLE_TABLE_RESPONSE messages when the handlespace is huge, since forming and sending one giant response could interrupt its other services. 🧠)

If the download uses more than one ENRP_HANDLE_TABLE_RESPONSE message, the mentor MUST use the M flag in each one to show whether it's the last message for that download session. More specifically, the mentor MUST set the M flag to '1' in the outgoing ENRP_HANDLE_TABLE_RESPONSE if there's still more data to send, and it MUST keep track of how far along the current download session is. For the final ENRP_HANDLE_TABLE_RESPONSE of the session, the mentor MUST set the M flag to '0' and close the download session (meaning it removes any internal record of the session) right after sending that last message. 💌

During the download, every time the initiating server gets an ENRP_HANDLE_TABLE_RESPONSE, it MUST move the data entries from that message into its local handlespace database, and then check whether this message is the last one in the session.

If the M flag is set to '1' in the ENRP_HANDLE_TABLE_RESPONSE it just processed, the initiating server MUST send another ENRP_HANDLE_TABLE_REQUEST to the mentor peer to ask for the next ENRP_HANDLE_TABLE_RESPONSE. 🔁

When the initiating server unpacks the data entries from an ENRP_HANDLE_TABLE_RESPONSE into its local handlespace database, it MUST handle each pool entry using these rules:
- A. If the pool doesn't exist in the local handlespace, the initiating server MUST create the pool locally and add the PE(s) in the pool entry to it. When it creates the pool, the initiation server MUST set the overall member selection policy type of the pool to the policy type indicated in the first PE. 🌈
- B. If the pool already exists locally, but the PE(s) in the pool entry aren't currently members of it, the initiating server MUST add the PE(s) to the pool. 🫶
- C. If the pool already exists locally AND the PE(s) in the pool entry are already members of it, the initiating server SHOULD replace the attributes of the existing PE(s) with the new information. ENRP will make sure that the information stays up to date. 🔥
When the last ENRP_HANDLE_TABLE_RESPONSE is received from the mentor peer and unpacked into the local handlespace, the initialization process is complete, and the initiating server SHOULD start providing ENRP services. 👑

Sometimes the mentor itself can't provide a handlespace download. For example, it might be in the middle of initializing its own handlespace database, or it might already have too many download sessions open with other servers. 🙈

In that case, the mentor MUST reject the request and respond with an ENRP_HANDLE_TABLE_RESPONSE with the R flag set to '1' and no pool entries included. 💀

If its ENRP_HANDLE_TABLE_REQUEST gets rejected by the mentor peer, the initiating server SHOULD either wait a few seconds and re-send the ENRP_HANDLE_TABLE_REQUEST to the mentor server, or, if a backup mentor peer is available, pick another mentor peer server and send the ENRP_HANDLE_TABLE_REQUEST to the new mentor server. 🧚

A handlespace download session that has already started can get interrupted for some reason. To handle that, the initiating server SHOULD start a timer every time it finishes sending an ENRP_HANDLE_TABLE_REQUEST to its mentor peer. If the timer expires without a response from the mentor peer, the initiating server SHOULD abort the current download session and restart a new handlespace download with a backup mentor peer, if one is available. ⏳

It works the other way too. After the mentor sends an ENRP_HANDLE_TABLE_RESPONSE and sets the M-bit to '1' (meaning it still has more data to send), it SHOULD start a session timer. If that timer expires without another request from the initiating server, the mentor SHOULD abort the session and clean out any resources and records of it. 🧹

3.3. Server Handlespace Update 💖

this is the group of update operations an ENRP server uses to tell its peers when its local handlespace changes. think: a new PE shows up, an existing PE leaves, or a pool's or PE's properties get changed. 🦋

3.3.1. Announcing Additions or Updates of PE ✨

when a new PE gets registered to the handlespace, or an existing PE gets re-registered, the Home ENRP server uses this procedure to tell all its peers. 🎀

This is an ENRP announcement, and it gets sent to all the peers of the Home ENRP server. See Section 3.1 on how announcements are sent.

An ENRP server MUST announce this update to all its peers in an ENRP_HANDLE_UPDATE message with the Update Action field set to ADD_PE, which means a new PE was added or an existing PE was modified. The complete new information for the PE and the pool it belongs to MUST be included in the message, in a PE parameter and a Pool Handle parameter, respectively. 💅

The Home ENRP server SHOULD fill in its server ID in the Sending Server's ID field and leave the Receiving Server's ID blank (that is, all 0s). 🫶

When a peer receives this ENRP_HANDLE_UPDATE message, it MUST take these actions:

If the named pool indicated by the pool handle doesn't exist in its local copy of the handlespace, the peer MUST create the named pool in its local handlespace and add the PE to the pool as the first PE. It MUST then copy in all of the other attributes of the PE carried in the message.

When the new pool is created, the overall member selection policy of the pool MUST be set to the policy type indicated by the PE. 🌸

If the named pool already exists in the peer's local copy of the handlespace and the PE doesn't exist, the peer MUST add the PE to the pool as a new PE and copy in all attributes of the PE carried in the message. 🍓
If the named pool exists and the PE is already a member of the pool, the peer MUST replace the attributes of the PE with the new information carried in the message. 🔁

3.3.2. Announcing Removal of PE 🥀

When an existing PE gets de-registered, or gets removed from its handlespace for some other reason (for example, purging an unreachable PE, see Section 3.5 in [RFC5352]), the ENRP server MUST use this procedure to tell all its peers about the change. 💔

This is an ENRP announcement, and it gets sent to all the peers of the Home ENRP server. See Section 3.1 on how announcements are sent.

An ENRP server MUST announce the PE removal to all its peers in an ENRP_HANDLE_UPDATE message with the Update Action field set to DEL_PE, which means an existing PE is being removed. The complete information of the PE and the pool it belongs to MUST be included in the message, in a PE parameter and a Pool Handle parameter, respectively. 🙅‍♀️

The sending server MUST fill in its server ID in the Sending Server's ID field and leave the Receiving Server's ID blank (that is, set to all 0s). 🧚

When a peer receives this ENRP_HANDLE_UPDATE message, it MUST first find the pool and the PE in its own handlespace, and then remove the PE from its local handlespace. If the removed PE was the last one in the pool, the peer MUST also delete the pool from its local handlespace. 🗑️

If the peer can't find the PE or the pool in its handlespace, it SHOULD take no further action. 🤷‍♀️

3.4. Maintaining Peer List and Monitoring Peer Status 👀

An ENRP server MUST keep an internal record of the status of each of its known peers. This record is called the server's "peer list." 📋

3.4.1. Discovering New Peer 🔍

If a message of any type comes in from a peer the server hasn't seen before, the ENRP server MUST treat that peer as a new peer in the operational scope and add it to the peer list. 🌟

The ENRP server MUST send an ENRP_PRESENCE message with the Reply-required flag set to '1' to the source address in the arriving message. This forces the new peer to reply with its own ENRP_PRESENCE containing its full server information (see Section 2.1). 🫶

3.4.2. Server Sending Heartbeat 💖

Okay so every PEER-HEARTBEAT-CYCLE seconds, an ENRP server MUST announce its continued presence to all its peers with an ENRP_PRESENCE message 🫶. In that message, the ENRP server MUST set the 'Replay_required' flag to '0', which just means "no reply needed" 🦋.

The arrival of this periodic ENRP_PRESENCE message makes every peer update its internal variable peer_last_heard for the sender (see Section 3.4.3 for more details) ✨.

3.4.3. Detecting Peer Server Failure 🌸

An ENRP server MUST keep an internal variable peer_last_heard for each of its known peers, and this variable MUST be updated to the current local time every time a message of any type (point-to-point or announcement) comes in from that peer 💜.

If a peer hasn't been heard from for more than MAX-TIME-LAST-HEARD seconds, the ENRP server MUST immediately send a point-to-point ENRP_PRESENCE with the Reply_request flag set to '1' to that peer 👀.

If the send fails or the peer doesn't reply after MAX-TIME-NO-RESPONSE seconds, the ENRP server MUST consider the peer server dead 💀 and SHOULD initiate the takeover procedure defined in Section 3.5.

3.5. Taking Over a Failed Peer Server 🦸‍♀️

In the following descriptions, we call the ENRP server that detects the failed peer server and starts the takeover the "initiating server", and the failed peer server the "target server". This is what lets the PE keep working when its Home ENRP server goes down 🫶.

3.5.1. Initiating Server Take-over Arbitration 🥊

The initiating server SHOULD first start the takeover arbitration by sending an ENRP_INIT_TAKEOVER message to all its peer servers. See Section 3.1 on how announcements are sent. In the message, the initiating server MUST fill in the Sending Server's ID and Targeting Server's ID. The goal is that only one ENRP server takes over the PE from the target 🎯.

After announcing the ENRP_INIT_TAKEOVER message ("group-casting" to all known peers, including the target server), the initiating server SHOULD wait for an ENRP_INIT_TAKEOVER_ACK message from each of its known peers, except the target server 🧚.

Each peer that receives an ENRP_INIT_TAKEOVER message from the initiating server MUST take the following actions:

If the peer server determines that it (itself) is the target server named in the ENRP_INIT_TAKEOVER message, it MUST immediately announce an ENRP_PRESENCE message to all its peer ENRP servers to try to stop this takeover. This is a false failure detection by the initiating server 🙈. The initiating server MUST stop the takeover by marking the target server as "active" and taking no further takeover actions.
If the peer server finds that it has already started its own takeover arbitration on the same target server, it MUST perform the following arbitration:
- A. If the peer's server ID is smaller in value than the Sending Server's ID in the arrived ENRP_INIT_TAKEOVER message, the peer server MUST immediately abort its own takeover attempt by taking no further takeover actions of its own. Moreover, the peer MUST mark the target server as "not active" on its internal peer list so that its status will no longer be monitored by the peer, and reply to the initiating server with an ENRP_INIT_TAKEOVER_ACK message 🏳️.
- B. Otherwise, the peer MUST ignore the ENRP_INIT_TAKEOVER message 🙅‍♀️.
If the peer finds that it is neither the target server nor already in its own takeover process, the peer MUST: a) mark the target server as "not active" on its internal peer list so that its status will no longer be monitored by this peer, and b) MUST reply to the initiating server with an ENRP_INIT_TAKEOVER_ACK message 🫶.

Once the initiating server has received the ENRP_INIT_TAKEOVER_ACK message from all of its currently known peers (except the target server), it MUST consider that it has won the arbitration and MUST proceed to complete the takeover, following the steps in Section 3.5.2 👑.

However, if it receives an ENRP_PRESENCE from the target server at any point during the arbitration, the initiating server MUST immediately stop the takeover process and mark the status of the target server as "active" 😭.

3.5.2. Takeover Target Peer Server 👑

The initiating ENRP server MUST first send, via an announcement, an ENRP_TAKEOVER_SERVER message to tell all its active peers that the takeover has happened. The target server's ID MUST be filled in the message. The initiating server SHOULD then remove the target server from its internal peer list 🧹.

After that, it SHOULD look through its local copy of the handlespace and claim ownership of each PE that was originally owned by the target server, by following these steps:

mark itself as the Home ENRP server of each PE that was originally owned by the target server;
send a point-to-point ASAP_ENDPOINT_KEEP_ALIVE message, with the 'H' flag set to '1', to each of the PEs. This triggers the PE to adopt the initiating server as its new Home ENRP server 🦋.

When a peer receives the ENRP_TAKEOVER_SERVER message from the initiating server, it SHOULD update its local peer list and PE cache by following these steps:

remove the target server from its internal peer list;
update the Home ENRP server of each PE in its local copy of the handlespace to be the sender of the message, i.e., the initiating server 💅.

3.6. Handlespace Data Auditing and Re-synchronization 🧚‍♀️

Message losses or certain temporary breaks in network connectivity can leave the local handlespace copies of some ENRP servers in an operational scope out of sync. So each ENRP server in the operational scope SHOULD periodically check that its local copy of the handlespace data still matches its peers' copies 🔍.

This section defines the auditing and re-synchronization procedures that an ENRP server uses to keep its handlespace data consistent 🌈.

3.6.1. Auditing Procedures 🔐

A checksum covering the data that should be the same is exchanged, and that's how everyone figures out whether the data really matches 🧮.

The auditing of handlespace consistency works like this:

An ENRP server SHOULD keep a separate PE checksum (a 16-bit integer internal variable) for each of its known peers and for itself. For an ENRP server with 'k' known peers, we denote these internal variables as pe_checksum_pr0, pe_checksum_pr1, ..., pe_checksum_prk, where pe_checksum_pr0 is the server's own PE checksum. The list of what these checksums cover and a detailed algorithm for calculating them is given in Section 3.6.2.
Each time an ENRP server sends out an ENRP_PRESENCE, it MUST include its current PE checksum (i.e., pe_checksum_pr0) in the message.
When an ENRP server (server A) receives a PE checksum (carried in an arrived ENRP_PRESENCE) from a peer ENRP server (server B), server A SHOULD compare the PE checksum in the ENRP_PRESENCE with its own internal PE checksum for server B (i.e., pe_checksum_prB).
If the two values match, server A considers that there's no handlespace inconsistency between itself and server B, and it should take no further action 😌.
If the two values do NOT match, server A SHOULD consider that there is a handlespace inconsistency between itself and server B, and a re-synchronization process SHOULD be carried out immediately with server B (see Section 3.6.3) 🚨.

3.6.2. PE Checksum Calculation Algorithm 🧁

When an ENRP server (server A) calculates an internal PE checksum for a peer (server B), it MUST use the following algorithm 🤓.

Let's say that in server A's internal handlespace, there are currently 'M' PEs owned by server B. Each of the 'M' PEs contributes to the checksum calculation with the following byte block:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   :  Pool handle string of the pool the PE belongs (padded with   :
   :  zeros to next 32-bit word boundary, if needed)               :
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        PE Id (4 octets)                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Note, these are not TLVs. This byte block gives each PE a unique byte pattern in the scope. The 16-bit PE checksum for server B, pe_checksum_prB, is then calculated over the byte blocks contributed by the 'M' PEs one by one. The PE checksum calculation MUST use the Internet algorithm described in [RFC1071] 🧠.

Server A MUST calculate its own PE checksum (i.e., pe_checksum_pr0) in the same way, using the byte blocks of all the PEs it owns itself 🙌.

Note, whenever an ENRP finds that its internal handlespace has changed (e.g., due to PE registration or de-registration, receiving peer updates, removing failed PEs, downloading handlespace pieces from a peer, etc.), it MUST immediately update all its internal PE checksums that are affected by the change ⚡.

Implementation Note: when the internal handlespace changes (e.g., a new PE is added or an existing PE is removed), an implementation need not re-calculate the affected PE checksum. It can instead just update the checksum by adding or subtracting the byte block of the corresponding PE from the previous checksum value 🧮💖

3.6.3. Re-Synchronization Procedures 🔄

If an ENRP server determines that its local handlespace data and a peer's handlespace data don't match for the PEs owned by that peer, it MUST perform the following steps to re-synchronize the data 🫶:

The ENRP server SHOULD first "mark" every PE it knows about that is owned by the peer in its local handlespace database;
The ENRP server SHOULD then send an ENRP_HANDLE_TABLE_REQUEST message with the W flag set to '1' to the peer, asking for a complete list of PEs owned by the peer 📋;
When the peer receives the ENRP_HANDLE_TABLE_REQUEST message with the W flag set to '1', it SHOULD immediately respond with an ENRP_HANDLE_TABLE_RESPONSE message listing all PEs currently owned by the peer 💌;
When the ENRP server receives the ENRP_HANDLE_TABLE_RESPONSE message, it SHOULD transfer the PE entries carried in the message into its local handlespace database. If a PE entry being transferred already exists in its local database, the ENRP server MUST replace the entry with the copy found in the message and remove the "mark" from the entry 🪄;
After transferring all the PE entries from the received ENRP_HANDLE_TABLE_RESPONSE message into its local database, the ENRP server SHOULD check whether any PE entries are still "marked" in its local handlespace. If so, the ENRP server SHOULD silently remove those "marked" entries 🧹.

Note, just like in Section 3.2.3, the peer may reject the ENRP_HANDLE_TABLE_REQUEST or use more than one ENRP_HANDLE_TABLE_RESPONSE message to respond 🌈.

3.7. Handling Unrecognized Messages or Unrecognized Parameters 🧚

okay so bestie, what happens when an ENRP server gets something it doesn't recognize? 👀 If it gets an ENRP message with a message type it's never heard of, or a known message type that has a parameter it doesn't know, it SHOULD handle that unknown message or parameter using the unrecognized message and parameter handling rules in Sections 3 and 4 of [RFC5354]. 💅

And according to those rules, if the sender needs an error report, the ENRP server that spotted the problem SHOULD send back an ENRP_ERROR message with a proper error cause code. 🚨 Lowkey the most organized way to tell someone "hey, you messed up" 🫶

4. Variables and Thresholds ✨

4.1. Variables 🌸

peer_last_heard: the local time that a peer server was last heard, meaning the last time it got either a group-cast or point-to-point message from that peer. 🦋
pe_checksum_pr: the internal 16-bit PE checksum that an ENRP server keeps for a peer. Each known peer gets its own PE checksum, and the server keeps one for itself too. 💜

4.2. Thresholds 🎀

These are the timing numbers that keep everything running on schedule. ⏰

PEER-HEARTBEAT-CYCLE: how often an ENRP server announces a heartbeat message to all its known peers. (Default=30 secs.) 💖
MAX-TIME-LAST-HEARD: the pre-set threshold for how long an ENRP server waits before it starts thinking a silent peer server might be dead. (Default=61 secs.) 😭
MAX-TIME-NO-RESPONSE: the pre-set threshold for how long a message sender waits for a response after sending a message. (Default=5 secs.) 🍓

5. IANA Considerations 📋

This document (RFC 5353) is the reference for all the registrations described in this section. Every registration has been listed on the RSerPool Parameters page. 🌈

5.1. A New Table for ENRP Message Types 💌

ENRP Message Types are maintained by IANA, and IANA has assigned ten initial values, which you can see in Figure 1. IANA made a brand new table called "ENRP Message Types":

Type       Message Name                 Reference
-----      -------------------------    ---------
0x00       (Reserved by IETF)           RFC 5353
0x01       ENRP_PRESENCE                RFC 5353
0x02       ENRP_HANDLE_TABLE_REQUEST    RFC 5353
0x03       ENRP_HANDLE_TABLE_RESPONSE   RFC 5353
0x04       ENRP_HANDLE_UPDATE           RFC 5353
0x05       ENRP_LIST_REQUEST            RFC 5353
0x06       ENRP_LIST_RESPONSE           RFC 5353
0x07       ENRP_INIT_TAKEOVER           RFC 5353
0x08       ENRP_INIT_TAKEOVER_ACK       RFC 5353
0x09       ENRP_TAKEOVER_SERVER         RFC 5353
0x0a       ENRP_ERROR                   RFC 5353
0x0b-0xff  (Available for assignment)   RFC 5353

If you want to register an ENRP Message Type in this table, send the request to IANA. The number must be unique. The "Specification Required" policy of [RFC5226] MUST be applied. 🔒

5.2. A New Table for Update Action Types 🔁

Update Types are maintained by IANA too, and IANA has assigned two initial values. IANA made another new table called "Update Action Types":

Type           Update Action              Reference
-------------  --------------------       ---------
0x0000         ADD_PE                      RFC 5353
0x0001         DEL_PE                      RFC 5353
0x0002-0xffff  (Available for assignment)  RFC 5353

Same deal here: requests to register an Update Action Type go to IANA, the number must be unique, and the "Specification Required" policy of [RFC5226] MUST be applied. ✨

5.3. Port Numbers 🔌

The references for the port numbers that were already assigned have been updated to RFC 5353:

enrp-udp 9901/udp
enrp-sctp 9901/sctp
enrp-sctp-tls 9902/sctp

5.4. SCTP Payload Protocol Identifier 🫧

The reference for the ENRP payload protocol identifier that was already assigned, which is 12, has been updated to RFC 5353. 💅

6. Security Considerations 🔐

Okay, this section is serious, so let's get into it. 😤 We're going to look at the threats to the RSerPool architecture and the security requirements that respond to them. Then we'll go over the security mechanisms that implementations REQUIRED to use, which are based on TLS. 🛡️ Last, we'll walk through a chain-of-trust argument that checks the critical data paths in RSerPool and shows how TLS protects them. It's giving full security audit energy. 💀

6.1. Summary of RSerPool Security Threats 🕵️‍♀️

"Threats Introduced by Reliable Server Pooling (RSerPool) and Requirements for Security in Response to Threats" [RFC5355] goes deep into the threats to the RSerPool architecture and lists the security requirements for each one. Here are the security services RSerPool needs, based on the threats in this document:

Threat 1) PE registration/de-registration flooding or spoofing Security mechanism in response: ENRP server authenticates the PE. 🔑
Threat 2) PE registers with a malicious ENRP server Security mechanism in response: PE authenticates the ENRP server. 🔑
Threats 1 and 2, taken together, result in mutual authentication of the ENRP server and the PE. 🤝
Threat 3) Malicious ENRP server joins the ENRP server pool Security mechanism in response: ENRP servers mutually authenticate. 🤝
Threat 4) A PU communicates with a malicious ENRP server for handle resolution Security mechanism in response: The PU authenticates the ENRP server. 🔑
Threat 5) Replay attack Security mechanism in response: Security protocol that has protection from replay attacks. 🔁
Threat 6) Corrupted data that causes a PU to have misinformation concerning a pool handle resolution Security mechanism in response: Security protocol that supports integrity protection. 🧾
Threat 7) Eavesdropper snooping on handlespace information Security mechanism in response: Security protocol that supports data confidentiality. 🙈
Threat 8) Flood of ASAP_ENDPOINT_UNREACHABLE messages from the PU to ENRP server Security mechanism in response: ASAP must control the number of ASAP endpoint unreachable messages transmitted from the PU to the ENRP server. 🚦
Threat 9) Flood of ASAP_ENDPOINT_KEEP_ALIVE messages to the PE from the ENRP server Security mechanism in response: ENRP server must control the number of ASAP_ENDPOINT_KEEP_ALIVE messages to the PE. 🚦

To sum it up, threats 1-7 need security mechanisms that support authentication, integrity, data confidentiality, and protection from replay attacks. 🌟

For RSerPool, we need to authenticate the following, and yes, the diagram is the whole vibe: 💜

PU <----  ENRP server (PU authenticates the ENRP server)
PE <----> ENRP server (mutual authentication)
ENRP server <-----> ENRP server (mutual authentication)

6.2. Implementing Security Mechanisms ✨

okay so we're NOT cooking up brand-new security mechanisms just for threats 1-7 🙅‍♀️ we borrow an existing IETF security protocol, [RFC3237], to get all the security services we need. TLS covers every single requirement, so TLS MUST be implemented 🔐 it's giving reliable, it's giving proven.

The TLS_RSA_WITH_AES_128_CBC_SHA ciphersuite MUST be supported, at a minimum, by anyone implementing TLS for RSerPool 💖 For backwards compatibility, ENRP SHOULD support TLS_RSA_WITH_3DES_EDE_CBC_SHA 🍓 Implementers MAY also support any other IETF-approved ciphersuites 🌈

ENRP servers, PEs, and PUs MUST implement TLS 🎀 ENRP servers and PEs MUST support mutual authentication using PSK 🫶 ENRP servers MUST support mutual authentication among themselves using PSK 💜 PUs MUST authenticate ENRP servers using certificates 🔑

TLS with PSK is mandatory to implement as the authentication mechanism for ENRP to ENRP authentication and PE to ENRP authentication ✨ With PSK, having a pre-shared key counts as authorization 🗝️ The network admins of a pool get to decide which nodes are authorized to join the pool 👑 The reason PSK works here is that we assume one administrative domain controls and manages the server pool, so PSK can be implemented and managed by a central security admin 🧚

TLS with certificates is mandatory to implement as the authentication mechanism for PUs to the ENRP server 💅 PUs MUST authenticate ENRP servers using certificates (yes, we're saying it again because it's THAT important 😭). ENRP servers MUST possess a site certificate whose subject corresponds to their canonical hostname 🌸 PUs MAY have certificates of their own for mutual authentication with TLS, but this document doesn't set out any provisions for using them 🦋 Every RSerPool element that supports TLS MUST have a mechanism for validating certificates received during TLS negotiation. That means having one or more root certificates issued by certificate authorities (ideally well-known distributors of site certificates, like the ones that issue root certificates for web browsers) 🛡️

To prevent man-in-the-middle attacks 👀 the client MUST verify the server's identity, as presented in the server's Certificate message. The client's understanding of the server's identity (usually the identity it used to set up the transport connection) is called the "reference identity" 🎯 The client figures out the type of the reference identity (like a DNS name or an IP address) and compares it against each subjectAltName value of that type until it finds a match. Once there's a match, the server's identity is verified and the server identity check is complete ✅ Different subjectAltName types get matched in different ways, and the client may map the reference identity to a different type before comparing. Mappings can be done for all the available subjectAltName types the reference identity can map to, but the reference identity should only be mapped to types where the mapping is either inherently secure (like pulling the DNS name out of a URI to compare with a subjectAltName of type dNSName) or done in a secure way (like using DNS Security (DNSSEC), or user- or admin-configured host-to-address/address-to-host lookup tables) 🧐

If the server identity check fails 💀 user-oriented clients SHOULD either notify the user or close the transport connection and indicate that the server's identity is suspect. Automated clients SHOULD close the transport connection and then return or log an error saying the server's identity is suspect, or do both. Beyond this server identity check, clients should be prepared to do further checking to make sure the server is authorized to provide the service it's been asked for. The client may need to use local policy information to make that call 📋

If the reference identity is an internationalized domain name, conforming implementations MUST convert it to the ASCII Compatible Encoding (ACE) format, as specified in Section 4 of [RFC3490], before comparing it with subjectAltName values of type dNSName 🌍 Specifically, conforming implementations MUST perform the conversion operation from Section 4 of [RFC3490] like this:

in step 1, the domain name SHALL be considered a "stored string";
in step 3, set the flag called "UseSTD3ASCIIRules";
in step 4, process each label with the "ToASCII" operation; and
in step 5, change all label separators to U+002E (full stop).

After the "to-ASCII" conversion, the DNS labels and names MUST be compared for equality using the rules in Section 3 of RFC 3490 🔤 The '*' (ASCII 42) wildcard character is allowed in subjectAltName values of type dNSName, but only as the left-most (least significant) DNS label in that value 🌟 That wildcard matches any left-most DNS label in the server name. So the subject *.example.com matches the server names a.example.com and b.example.com, but it does NOT match example.com or a.b.example.com 🙅‍♀️

When the reference identity is an IP address, the identity MUST be converted to the "network byte order" octet string representation, per RFC 791 [RFC0791] and RFC 2460 [RFC2460] 🔢 For IP version 4, as specified in RFC 791, the octet string will contain exactly four octets. For IP version 6, as specified in RFC 2460, it will contain exactly sixteen octets. That octet string then gets compared against subjectAltName values of type iPAddress. It's a match if the reference identity octet string and the value octet string are identical 🎉

Once a TLS layer is set up in a session, both parties independently decide whether to keep going, based on local policy and the security level they've reached 🤔 If either party decides the security level isn't good enough to continue, it SHOULD remove the TLS layer right after the TLS (re)negotiation has finished (see RFC 4511)[RFC4511]. Implementations may re-evaluate the security level at any time and, if they find it inadequate, should remove the TLS layer 🧹

Implementations MUST support TLS with SCTP, as described in [RFC3436], or TLS over TCP, as described in [RFC5246] 🦋 When using TLS/SCTP, we need to make sure RSerPool doesn't use any SCTP features that aren't available to a TLS/SCTP user. It's not a hard technical problem, it's just a requirement 💅 And when describing an API for the RSerPool lower layer, we also have to take the differences between TLS and SCTP into account 🫶

Threat 8 requires the ASAP protocol to limit the number of ASAP_ENDPOINT_UNREACHABLE messages (see Section 3.5 of RFC 5352) sent to the ENRP server 😤 Threat 9 requires the ENRP protocol to limit the number of ASAP_ENDPOINT_KEEP_ALIVE messages sent from the ENRP server to the PE 🚫

There's no security mechanism defined for multicast announcements 🙈 So a receiver of one of these announcements can't treat the source address of that message as a trustworthy address of an ENRP server. A receiver also has to be ready for a flood of multicast announcements from attackers 💀 this part is seriously not fun, bestie

6.3. Chain of Trust 🔗

Security is mandatory to implement in RSerPool 💜 It's built on TLS in all three architecture components that make up RSerPool: the PU, the PE, and the ENRP server. We define an ENRP server that uses TLS for all communication and authenticates ENRP peers and PE registrants as a secured ENRP server 👑

Here's a look at all the possible data paths and what security each one gets 👀

PU <---> secured ENRP server (authentication of ENRP server;
         queries over TLS)
PE <---> secured ENRP server (mutual authentication;
         registration/de-registration over TLS)
secured ENRP server <---> secured ENRP server (mutual authentication;
         database updates using TLS)

If every component in the system authenticates and communicates using TLS, the chain of trust is sound ⛓️✨ The root of the trust chain is the ENRP server. If that's secured with TLS, then security gets enforced for every ENRP and PE component that tries to connect to it 🧚

Summary of interaction between secured and unsecured components: if a PE doesn't use TLS and tries to register with a secure ENRP server, it gets an error message response flagged as a security-related error, and the registration is rejected 🙅‍♀️ If an ENRP server that doesn't use TLS tries to update the database of a secure ENRP server, the update gets rejected 🚫 If a PU doesn't use TLS and talks to a secure ENRP server, it gets a response, but the PU knows that response isn't secure, because it can be tampered with in transit even if the ENRP database is secured 😬

The final case is a PU sending a secure request to ENRP. ENRP and PEs might not be secured, and that's an allowable configuration 🌸 The goal is to secure the communication over the Internet between the PU and the ENRP server 🔐

Summary:

RSerPool architecture components can talk to each other to set up a chain of trust 🤝 Secured PE and ENRP servers reject any communication with unsecured ENRP or PE servers 🛡️

If all of the above is enforced, then a chain of trust is established for the RSerPool user 🎀chain of trust: certified ✨🌈✨

7. Acknowledgments 🫶

The authors wish to thank John Loughney, Lyndon Ong, Walter Johnson, Thomas Dreibholz, Frank Volkmer, and many others for their invaluable comments and feedback 💖

8. References 📚

8.1. Normative References 📚

okay bestie, these are the "read these first or you'll be lost" docs 🧠✨ every single one is a must-read for understanding the spec 💅

[RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, September 1981.
[RFC1071] Braden, R., Borman, D., Partridge, C., and W. Plummer, "Computing the Internet checksum", RFC 1071, September 1988.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998.
[RFC3237] Tuexen, M., Xie, Q., Stewart, R., Shore, M., Ong, L., Loughney, J., and M. Stillman, "Requirements for Reliable Server Pooling", RFC 3237, January 2002.
[RFC3436] Jungmaier, A., Rescorla, E., and M. Tuexen, "Transport Layer Security over Stream Control Transmission Protocol", RFC 3436, December 2002.
[RFC3490] Faltstrom, P., Hoffman, P., and A. Costello, "Internationalizing Domain Names in Applications (IDNA)", RFC 3490, March 2003.
[RFC4511] Sermersheim, J., "Lightweight Directory Access Protocol (LDAP): The Protocol", RFC 4511, June 2006.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008.
[RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.2", RFC 5246, August 2008.
[RFC5354] Stewart, R., Xie, Q., Stillman, M., and M. Tuexen, "Aggregate Server Access Protocol (ASAP) and Endpoint Handlespace Redundancy Protocol (ENRP) Parameters", RFC 5354, September 2008.
[RFC5352] Stewart, R., Xie, Q., Stillman, M., and M. Tuexen, "Aggregate Server Access Protocol (ASAP)", RFC 5352, September 2008.
[RFC5355] Stillman, M., Ed., Gopal, R., Guttman, E., Holdrege, M., and S. Sengodan, "Threats Introduced by Reliable Server Pooling (RSerPool) and Requirements for Security in Response to Threats", RFC 5355, September 2008.

8.2. Informative References 🌸

these ones are the extra-credit reading, still super helpful 🤓💖

[RFC4086] Eastlake, D., Schiller, J., and S. Crocker, "Randomness Requirements for Security", BCP 106, RFC 4086, June 2005.
[SCTPSOCKET] Stewart, R., Poon, K., Tuexen, M., Yasevich, V., and P. Lei, "Sockets API Extensions for Stream Control Transmission Protocol (SCTP)", Work in Progress, July 2008.

Authors' Addresses 🎀

meet the squad who built this thing 🫶 here's how to reach each of them

Qiaobing Xie
The Resource Group
1700 Pennsylvania Ave NW
Suite 560
Washington, D.C.,   20006
USA

Phone: +1 224-465-5954
EMail: Qiaobing.Xie@gmail.com


Randall R. Stewart
The Resource Group
1700 Pennsylvania Ave NW
Suite 560
Washington, D.C.,   20006
USA

Phone:
EMail: randall@lakerest.net


Maureen Stillman
Nokia
1167 Peachtree Ct.
Naperville, IL  60540
US

Phone:
EMail: maureen.stillman@nokia.com
Michael Tuexen
Muenster Univ. of Applied Sciences
Stegerwaldstr. 39
48565 Steinfurt
Germany

EMail: tuexen@fh-muenster.de


Aron J. Silverton
Sun Microsystems, Inc.
10 S. Wacker Drive
Suite 2000
Chicago, IL 60606
USA

Phone:
EMail: ajs.ietf@gmail.com

Full Copyright Statement ⚖️

okay the legal part, but make it cute 🎀 the copyright is held by The IETF Trust, and the rules are the ones in BCP 78 🔒

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🦋

the IETF doesn't take a side on who owns what tech in here, and it hasn't checked every possible patent either 🧐 the rules for RFC rights live in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

the IETF also invites anyone to flag any copyrights, patents, patent applications, or other proprietary rights that might cover tech needed to implement this standard. send that info to the IETF at ietf-ipr@ietf.org 💜

📦 where this page comes from

ietf: rfc5353 2008-09
💖 girlified by claude-haiku-5-5 on 2026-10-17: a language model's rewrite of the original, under its terms