RFC 7121
High Availability within a Forwarding and Control Element Separation (ForCES) Network Element
RFC 7121: High Availability within a ForCES Network Element 📜✨
Okay bestie, RFC 7121 is a Standards Track doc from February 2014, and it's giving main character energy 💅. It was written by K. Ogawa (NTT Corporation), W. Wang (Zhejiang Gongshang University), E. Haleplidis (University of Patras), and J. Hadi Salim (Mojatatu Networks). Its ISSN is 2070-1721, and it updates RFC 5810 🎀.
Abstract 🦋
Okay so this doc is all about Control Element (CE) High Availability (HA) inside a Forwarding and Control Element Separation (ForCES) Network Element (NE) 🧚. It also updates RFC 5810 by adding new normative text for the Cold Standby High Availability mechanism. Iykyk, but the vibe is "we're making the backup plan actually good" 🔥.
Status of This Memo 📋
This is an Internet Standards Track document 🌸.
It's a product of the Internet Engineering Task Force (IETF), so it reflects the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check out Section 2 of RFC 5741 📚.
Info about the current status of this doc, any errata, and how to give feedback can be found at http://www.rfc-editor.org/info/rfc7121 🔗.
Copyright Notice ⚖️
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved 💜.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they explain your rights and restrictions for this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 🫶.
Table of Contents 🗂️
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Quantifying Problem Scope . . . . . . . . . . . . . . . . 4
1.2. Definitions . . . . . . . . . . . . . . . . . . . . . . . 5
2. RFC 5810 CE HA Framework . . . . . . . . . . . . . . . . . . 7
2.1. RFC 5810 CE HA Support . . . . . . . . . . . . . . . . . 7
2.1.1. Cold Standby Interaction with the ForCES Protocol . . 8
2.1.2. Responsibilities for HA . . . . . . . . . . . . . . . 10
3. CE HA Hot Standby . . . . . . . . . . . . . . . . . . . . . . 11
3.1. Changes to the FEPO Model . . . . . . . . . . . . . . . . 11
3.2. FEPO Processing . . . . . . . . . . . . . . . . . . . . . 13
4. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17
5. Security Considerations . . . . . . . . . . . . . . . . . . . 18
6. References . . . . . . . . . . . . . . . . . . . . . . . . . 19
6.1. Normative References . . . . . . . . . . . . . . . . . . 19
6.2. Informative References . . . . . . . . . . . . . . . . . 19
Appendix A. New FEPO Version . . . . . . . . . . . . . . . . . . 201. Introduction ✨
Figure 1 shows a ForCES Network Element (NE) run by a bunch of redundant Control Elements (CEs). CE1 is the active one, and CE2 and CEn are the backups 🧚♀️.
-----------------------------------------
| ForCES Network Element |
| +-----------+ |
| | CEn | |
| | (Backup) | |
-------------- Fc | +------------+ +------------+ | |
| CE Manager |--------+-| CE1 |------| CE2 |-+ |
-------------- | | (Active) | Fr | (Backup) | |
| | +-------+--+-+ +---+---+----+ |
| Fl | | | Fp / | |
| | | +---------+ / | |
| | Fp| |/ |Fp |
| | | | | |
| | | Fp /+--+ | |
| | | +-------+ | | |
| | | | | | |
-------------- Ff | --------+--+-- ----+---+----+ |
| FE Manager |--------+-| FE1 | Fi | FE2 | |
-------------- | | |------| | |
| -------------- -------------- |
| | | | | | | | | |
----+--+--+--+----------+--+--+--+-------
| | | | | | | |
| | | | | | | |
Fi/f Fi/f
Fp: CE-FE interface
Fi: FE-FE interface
Fr: CE-CE interface
Fc: Interface between the CE manager and a CE
Ff: Interface between the FE manager and an FE
Fl: Interface between the CE manager and the FE manager
Fi/f: FE external interface
Figure 1: ForCES ArchitectureSo the ForCES architecture lets Forwarding Elements (FEs) be aware of multiple CEs, but only one CE can be the master controller. This is what the industry calls 1+N redundancy 🍓. The master CE controls the FEs through the ForCES protocol on the Fp interface. If the master CE goes faulty (like it crashes or loses connectivity), a backup CE takes over and the NE keeps running 🙌. This setup is currently known as cold standby. The set of CEs controlling an FE is static: the FE Manager (FEM) hands it to the FE over the Ff interface, and the CE Manager (CEM) gives it to each CE over the Fc interface during the pre-association phase 🗂️.
From the FE's point of view, the operational parameters for a CE set are defined as components in the FEPO LFB in [RFC5810], Appendix B. We'll dig into these parameters in Section 2.1 of this document 🔍.
You need to know the ForCES architecture to get the changes this doc describes. This doc also gives background info to set up the discussion in Section 3 💅.
When this was written, the Fr interface was out of scope for ForCES. Still, organizations running a set of CEs are expected to need the CEs to talk to each other over Fr so they can stay in sync while controlling the FEs 🫶.
The problem scope here has two parts:
- Make the description in [RFC5810] clearer about how the current cold standby approach works inside the NE cluster.
- Explain how to move the [RFC5810] cold standby setup to a hot standby redundancy setup, so failover is faster and NE availability goes up ⚡.
1.1. Quantifying Problem Scope 📏
NE recovery and availability depend on a few time-sensitive metrics ⏰:
- How fast the FE detects a CE plane failure.
- How fast a backup CE becomes operational.
- How fast the FEs associate with the new master CE.
- How fast the FEs recover their state and become operational. Each FE state is the collective state of all its instantiated LFBs.
Honestly, the design goal of both [RFC5810] and this doc, for hitting these goals, is simplicity 🌈.
Here's how the current ForCES CE setup in [RFC5810] stacks up:
- How fast the FE side detects a CE failure is left undefined 🤷♀️. As an extreme example, a human operator could be the monitoring entity that spots faulty CEs, and that detection could take anywhere from seconds to days 💀. A more active monitor on the Fp interface could make detection better. Usually the FE notices a CE failure either through the TML, if the Fp interface terminates, or through the ForCES protocol using the ForCES Heartbeat mechanism 💓.
- How fast the backup CE becomes operational is also currently out of scope 🙈. In the current setup, a backup CE doesn't need to be operational at all (say, to save power). So a monitoring entity can boot up a backup CE after it detects the master CE failing. In Section 3 of this document, we suggest that at least one backup CE be online to improve this metric 🔥.
- How fast an FE associates with a new master CE is also currently undefined. An FE connecting and associating adds to the recovery overhead. As mentioned above, we suggest having at least one backup CE online. In Section 3, we propose removing the connection and association cost on failover by having each FE associate with all online backup CEs after it associates with an active/master CE. Note that if an FE pre-associates with at least one backup CE, the system is technically operating in hot standby mode 🧚.
- Finally, how fast an FE recovers its state depends on how much NE state exists. By ForCES's current definition, the new master CE assumes zero state on the FE and starts from scratch to update it. So, the bigger the state, the longer the recovery 🐢.
(part 2 is next, so stay tuned 👀💖)
1.2. Definitions
okay so the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this doc are to be read the way [RFC2119] explains them 🧚💖
the next bit pulls definitions from [RFC3654], [RFC3746], and [RFC5810]. they're repeated here so you don't have to flip around, but the official normative versions live in those referenced RFCs, so go there if you need the real deal 📚✨
Logical Functional Block (LFB): a template that represents fine-grained, logically separate aspects of FE processing 🎀
Forwarding Element (FE): a logical entity that implements the ForCES protocol. FEs use the underlying hardware to do per-packet processing and handling, as directed by a CE through the ForCES protocol 🦋
Control Element (CE): a logical entity that implements the ForCES protocol and uses it to tell one or more FEs how to process packets. CEs handle stuff like running control and signaling protocols 👑
ForCES Network Element (NE): an entity made up of one or more CEs and one or more FEs. An NE usually hides its internal organization from outside entities, and it shows up to anything outside the NE as a single point of management 💅
FE Manager (FEM): a logical entity that works in the pre-association phase and figures out which CE(s) an FE should talk to. This process is called CE discovery, and it may involve the FE manager learning what the available CEs are capable of 🌸
CE Manager (CEM): a logical entity that works in the pre-association phase and figures out which FE(s) a CE should talk to. This process is called FE discovery, and it may involve the CE manager learning what the available FEs are capable of 🍓
ForCES Protocol: the protocol used for communication between CEs and FEs. It does NOT apply to CE-to-CE communication, FE-to-FE communication, or communication between FE and CE managers. (sorry, I'm yelling a little, but the original has this as a hard limit and I'm not changing it, only the phrasing 🙈) The ForCES protocol is a master-slave protocol: FEs are the slaves and CEs are the masters. It covers both managing the communication channel (for example connection establishment and heartbeats) and the control messages themselves 💜
ForCES Protocol Layer (ForCES PL): a layer in the ForCES protocol architecture that defines the ForCES protocol messages, the protocol state transfer scheme, and the ForCES protocol architecture itself, including the requirements for the ForCES Transport Mapping Layer (TML) shown below. The specs for ForCES PL are defined in [RFC5810] 🔥
ForCES Protocol Transport Mapping Layer (ForCES TML): a layer in the ForCES protocol architecture that deals specifically with how protocol messages get transported. That means how they get mapped onto different transport media (like Stream Control Transmission Protocol (SCTP), IP, TCP, UDP, ATM, Ethernet, etc.), and how reliability, security, etc. get achieved and implemented 🧚♀️✨
2. RFC 5810 CE HA Framework
okay bestie, high availability is the whole point here 😭 To get CE High Availability (HA), FEs and CEs MUST interoperate according to the definition in [RFC5810]. That definition is repeated in Section 2.1 for context. It should be noted that in this default setup, which MUST be implemented by CEs and FEs that require HA, the Fr plane is out of scope (and if it's available, it's proprietary to an implementation) 🫶
2.1. RFC 5810 CE HA Support
as mentioned earlier, there can be multiple redundant CEs, but only one CE actively controls FEs in a ForCES NE. In practice there may be only one backup CE. At any moment in time, only one master CE can control an FE. On top of that, the FE connects and associates to only the master CE. The FE and the CE both know about the primary and one or more secondary CEs. This info (primary and secondary CEs) is configured on the FE and the CE during pre-association by the FEM and the CEM, respectively 🦋
This section includes a new normative description that updates [RFC5810] for the Cold Standby High Availability mechanism. this is the glow-up section 💅🌈
Figure 2 below shows the ForCES message sequences the FE uses to recover the connection in the currently defined cold standby scheme 🎀
FE CE Primary CE Secondary
| | |
| Association Establishment | |
| Capabilities Exchange | |
1 |<------------------------->| |
| | |
| State Update | |
2 |<------------------------->| |
| | |
| | |
| FAILURE |
| |
| Association Establishment, Capabilities Exchange|
3 |<----------------------------------------------->|
| |
| Event Report (primary CE down) |
4 |------------------------------------------------>|
| |
| State Update |
5 |<----------------------------------------------->|Figure 2: CE Failover for Cold Standby
2.1.1. Cold Standby Interaction with the ForCES Protocol
HA gets set up in an FE by configuring the FE Protocol Object (FEPO) LFB 🧁
The FEPO Control Element ID (CEID) component tells you who the current master CE is, and the component table BackupCEs lists the configured backup CEs. The FEPO FE Heartbeat Interval (FEHI), CE Heartbeat Dead Interval (CEHDI), and CE Heartbeat policy help detect connectivity problems between an FE and a CE. The CE failover policy defines how the FE should react when it detects a failure 🚨 The FEObject FEState component [RFC5812] defines the operational forwarding status and control. The CE can turn off the FE's forwarding operations by setting the FEState to AdminDisable, and it can turn them back on by setting it to OperEnable. Note: Section 5.1 of [RFC5812] has been updated by an erratum ([Err3487]) that describes the FEState as read-only when it should be read-write 🙃
Figure 3 shows the defined state machine that helps recover the connection state 🧠
The FE connects to the CE specified on the FEPO CEID component. If it fails to connect to that defined CE, it moves that CE to the bottom of table BackupCEs and sets its CEID component to the first CE pulled from table BackupCEs. Then the FE tries to associate with the CE designated as the new primary CE. The FE keeps going through this procedure until it successfully connects to one of the CEs, or until the CE Failover Timeout Interval (CEFTI) expires ⏳
FE tries to associate
+-->-----+
| |
(CE changes master || | |
CE issues Teardown || +---+--------v----+
Lost association) && | Pre-association |
CE failover policy = 0 | (Association |
+------------>-->-->| in +<----+
| | progress) | |
| | | |
| +--------+--------+ |
| CE Association | | CEFTI
| Response V | timer
| +------------------+ | expires
| |FE issues CEPrimaryDown ^
| V |
+-+-----------+ +------+-----+
| | (CE changes master || | Not |
| | CE issues Teardown || | Associated |
| | Lost association) && | +->---+
| Associated | CE failover policy = 1 |(May | FE |
| | | Continue | try v
| |-------->------->------>| Forwarding)| assn|
| | Start CEFTI timer | |-<---+
| | | |
+-------------+ +-------+----+
^ |
| Successful V
| Association |
| Setup |
| (Cancel CEFTI timer) |
+_________________________________________+
FE issues CEPrimaryDown event
Figure 3: FE State Machine Considering HAthere are a few events that trigger mastership changes 👀 The master CE may issue a mastership change (by changing the CEID component), it may tear down an existing association, or connectivity may be lost between the CE and FE.
When communication fails between the FE and CE (which can be caused by either the CE or a link failure, but is not FE related), either the TML on the FE will tell the FE PL about the failure, or it will be detected using the Heartbeat messages between FEs and CEs. Either way, the communication failure, however it's detected, MUST be treated as a loss of association between the CE and the corresponding FE 💀
If the FE's FEPO CE failover policy is set to mode 0 (the default), it will immediately move to the pre-association phase. That means if association gets re-established with a CE later, all FE states will need to be re-created 🙈
If the FE's FEPO CE failover policy is set to mode 1, the FE runs in HA restart recovery. In that case the FE moves to the not associated state and the CEFTI timer [RFC5810] starts. The FE may keep forwarding packets during this state, depending on the value of the CEFailoverPolicy component of the FEPO LFB. The FE cycles through any configured backup CEs in a round-robin fashion 🔁 It first adds its primary CE to the bottom of table BackupCEs and sets its CEID component to the first secondary pulled from table BackupCEs. Then it tries to associate with the CE designated as the new primary CE. If it fails to re-associate with any CE and the CEFTI expires, the FE moves to the pre-association state, and it operationally brings down its forwarding path (and sets the [RFC5812] FEObject FEState component to OperDisable) 📉
If the FE, while in the not associated state, manages to reconnect to a new primary CE before the CEFTI expires, it moves to the associated state ✨ Once re-associated, the CE may try to sync any state the FE may have lost during the disconnection. How the CE re-synchronizes that state is out of scope for the current ForCES architecture, but it would typically involve issuing new Config messages and queries.
An explicit message (a Config message setting the primary CE component in the ForCES Protocol Object) from the primary CE can also change the primary CE for an FE during normal protocol operation. In that case, the FE moves to the not associated state and tries to associate with the new CE 🦋
2.1.2. Responsibilities for HA 🎀
TML Level:
- The TML controls logical connection availability and failover. 🔌
- The TML also controls peer HA management. 👑
okay so at the TML level, the TML is in charge of all the lower layers, like the transport level (think IP addresses, Media Access Control (MAC) addresses, etc.), plus the associated links going down. that's all TML drama 💅
PL Level: bestie, everything else is the PL's job 🫶 That includes setting up the HA behavior during setup, the Control Element IDs (CE IDs) that tell the primary and secondary CEs apart, the protocol messages used to report CE failure (event report), Heartbeat messages used to detect association failure, messages to change the primary CE (Config), and all the other HA-related operations described in Section 2.1. the PL handles it all 🧚
so how do the two work together? if the path to a primary CE goes down, the TML helps recover by switching over to a backup path, but only if one is available 🦋 If the CE is totally unreachable, then the PL gets informed and takes the appropriate actions described before. it's giving teamwork 💜
3. CE HA Hot Standby 🔥
in this section we describe small extensions to the existing scheme so we can get hot standby HA. to get there, we're aiming to improve the specific goals from Section 1.1, which are:
- How fast a backup CE becomes operational. ⭐
- How fast the FEs associate with the new master CE. 🚀
as described in Section 2.1, during the pre-association phase, the FEM configures the FE so it knows about all the CEs in the NE. the FEM MUST configure the FE so it knows which CE is the master, and it MAY specify any backup CE(s). iykyk, the master and backup roles are set up early 🌸
3.1. Changes to the FEPO Model 🎀
ngl, to make the above possible, we need a few changes to the FEPO model. Appendix A has the xml definition of the new version 1.1 of the FEPO LFB 📝
here's what changed from version 1 of the FEPO:
- Added four new datatypes:
CEStatusTypeis an unsigned char that specifies the status of a connection with a CE. The special values are:0(Disconnected) means no connection attempt has been made with the CE yet 💤1(Connected) means the FE connection with the CE at the TML has completed successfully ✅2(Associated) means the FE has successfully associated with the CE 🫶3(IsMaster) means the FE has associated with the CE and is the master of the FE 👑4(LostConnection) means the FE was associated with the CE at one point but lost the connection 💔5(Unreachable) means the FE deems this CE unreachable, i.e., the FE has tried over a period to connect to it but has failed 😭
HAModeValuesis an unsigned char that specifies a selected HA mode. The special values are:0(No HA Mode) means the FE is not running in HA mode 🙈1(HA Mode - Cold Standby) means the FE is in HA mode cold standby 🧊2(HA Mode - Hot Standby) means the FE is in HA mode hot standby 🔥
Statisticsis a complex structure representing the communication statistics between the FE and CE. The components are:RecvPackets, the packet count received from the CE 📦RecvBytes, the byte count received from the CE 🔢RecvErrPackets, the erroneous packets received from the CE. This component logs badly formatted packets as well as good packets sent to the FE by the CE to set components whilst that CE is not the master. Erroneous packets are dropped (i.e., not responded to). 🗑️RecvErrBytes, theRecvErrPacketsbyte count received from the CE 🔢TxmitPackets, the packet count transmitted to the CE 📤TxmitErrPackets, the error packet count transmitted to the CE. Typically, these would be failures due to communication. 🚧TxmitBytes, the byte count transmitted to the CE 🔢TxmitErrBytes, the byte count of errors from transmit to the CE 💀
AllCETypeis a complex structure made up of the CE IDs, statistics, andCEStatusTypeto reflect connection information for one CE. It's used in the AllCE's component array. 🧩
- Appended two new components:
- Read-only
AllCEsto hold the status for all CEs.AllCEsis an array of theAllCEType. 📚 - Read-write
HAModeof typeHAModeValuesto carry the HA mode used by the FE. 🎛️
- Read-only
- Added one additional event,
PrimaryCEChanged, which reports the new master CE ID when there is a mastership change. 🔔
since no component from FEPO v1 has been changed, FEPO v1.1 keeps backwards compatibility with CEs that only know version 1.0. however, those CEs can't use the HA options that the new FEPO provides 🥺💅
3.2. FEPO Processing 💖
Okay bestie, so the FE keeps a list called the FEPO LFB version 1.1 AllCEs table 🧚. It holds every CE ID the FE is allowed to connect and associate with, and the order of those IDs sets the priority. The table gets loaded at first from the configuration plane (FEM) 📋.
During pre-association, the first CE (lowest table index) in the AllCEs table MUST be the first CE the FE tries to connect and associate with 🎀. If that doesn't work out, the FE moves on to the second CE, then the third, and so on. When it reaches the end, it loops back to the top until an association succeeds 🔁. The FE MUST associate with at least one CE, no exceptions 👑. Once it does, the CEID component of the FEPO LFB tells everyone who the current associated master CE is 🦋.
Here's the tea ☕. Ideally the FE would ignore everyone except the master, but in real life it turns out it's super useful to still respond to queries and heartbeats from backup CEs 🫶. So backup CEs are allowed to send queries to the FE. Configuration messages (SET/DEL) from backup CEs MUST be dropped by the FE and logged as received errors 🙅♀️.
Asynchronous events that the master CE subscribed to go out to all associated CEs, and so do heartbeats 📣. Packet redirects, though, only go to the master CE 🎯. The Heartbeat Interval, the CE Heartbeat (CEHB) policy, and the FE Heartbeat (FEHB) policy are global for all CEs, and only the master CE is allowed to change them 👑✨.
Figure 4 shows the state machine that handles connection recovery when HA is turned on 🌸.
FE tries to associate
+-->-----+
| |
(CE changes master || | |
CE issues Teardown || +---+----v----+
Lost association) && | Pre-association |
CE failover policy = 0 | (Association |
+------------>-->-->| in +<----+
| | progress) | |
| | | |
| +--------+--------+ |
| CE Association | | CEFTI
| Response V | timer
| +------------------+ | expires
| |FE issues CEPrimaryDown ^
| |FE issues PrimaryCEChanged ^
| V |
+-+-----------+ +------+-----+
| | (CE changes master || | Not |
| | CE issues Teardown || | Associated |
| | Lost association) && | +->----------+
| Associated | CE failover policy = 1 |(May | find first |
| | | Continue | associated v
| |-------->------->------>| Forwarding)| CE or retry|
| | Start CEFTI timer | | associating|
| | | |-<----------+
| | | |
+----+--------+ +-------+----+
| |
^ Found | associated CE
| or newly | associated CE
| V
| (Cancel CEFTI timer) |
+_________________________________________+
FE issues CEPrimaryDown event
FE issues PrimaryCEChanged event
Figure 4: FE State Machine Considering HAOnce the FE has associated with a master CE, it moves into the post-association phase (the associated state) 🦋. The master CE is expected to chat with the other CEs in the NE for synchronization over the CE-CE interface 💬. Heads up, though: the CE-CE interface is out of scope for this document 🙈. If the CEs hold an election and pick a different master, the current master CE tells the FE to switch to the new one 🔄.
FE CE#1 CE#2 ... CE#N
| | | |
| Association Establishment | | |
| Capabilities Exchange | | |
1 |<------------------------->| | |
| | | |
| State Update | | |
2 |<------------------------->| | |
| | | |
| Association Establishment | |
| Capabilities Exchange | |
3I|<-------------------------------------->| |
... ... ... ...
|Association Establishment, Capabilities Exchange |
3N|<----------------------------------------------->|
| | | |
4 |<------------------------->| | |
. . . .
4x|<------------------------->| | |
| FAILURE | |
| | | |
| Event Report (LastCEID changed) | |
5 |--------------------------------------->|------->|
| Event Report (CE#2 is new master) | |
6 |--------------------------------------->|------->|
| | |
7 |<-------------------------------------->| |
. . . .
7x|<-------------------------------------->| |
. . . .
Figure 5: CE Failover for Hot StandbyNow the fun part 🎉. While the FE is in the post-association phase, if the CE failover policy is set to 1 and the HAMode is set to 2 (hot standby), then the FE MUST attempt to connect and associate with every CE it knows about, after it has successfully associated with the master CE 🌈. In Figure 5, steps #1 and #2 show the FE associating with CE#1 as master, and then steps #3I to #3N show it associating with the backup CEs, CE#2 through CE#N 💜. If the FE can't connect or associate with some CEs, it MAY flag them as unreachable so it doesn't keep knocking over and over 🙃. It MAY also try to re-associate with the unreachable CEs whenever that's possible 🔁.
When the master CE is considered down for any reason, the FE MUST try to find the first associated CE in the list of all CEs, going round-robin 🎠.
If the FE can't find an associated FE in its list of CEs, then it MUST attempt to connect and associate with the first CE on the list of all CEs 🥺. It keeps going round-robin until it connects and associates with a CE, or until the CEFTI timer runs out ⏰.
Once the FE picks an associated CE to be the new master, it sends a PrimaryCEDown Event Notification to all associated CEs 📣. This tells them the last primary CE went down and who it was 📉. Then a second event, PrimaryCEChanged, goes out too, naming the new master CE the reporting FE thinks is in charge 👑.
Let's talk split brain 🧠. Most HA setups have the risk of split brain. In our setup, though, the FE never accepts configuration messages from anyone except the master CE, so the FE is considered fenced against data corruption from other CEs that think they're the master 🛡️. That means split brain turns into mostly a CE-CE communication problem, which is out of scope here 🙌.
Having several CE connections means the FE can switch over to a new master way faster 🏎️💨. The big win is better NE recovery time when the master CE has a communication failure or faults 🌟. And that, my friend, is exactly the requirement we set out to meet 💅✨.
4. IANA Considerations ✨
Following the policies in "Guidelines for Writing an IANA Considerations Section in RFCs" [RFC5226], the "Logical Functional Block (LFB) Class Names and Class Identifiers" namespace has been updated 📚.
A new column, LFB version, now sits in the table right after the LFB Class Name 🆕. Here's how the table looks now 👀:
+----------------+------------+-----------+-------------+-----------+
| LFB Class | LFB Class | LFB | Description | Reference |
| Identifier | Name | Version | | |
+----------------+------------+-----------+-------------+-----------+
Logical Functional Block (LFB) Class Names and Class IdentifiersThe rules in [RFC5812] still apply, with one addition: entries must provide the LFB version as a string 🎀.
When this document gets published, every current entry is assigned a value of 1.0 🌸.
New versions of already defined LFBs MUST NOT remove the previous version entries 🙅♀️. Keep those old versions around, always 💜.
It would make sense for the LFB versions to show up in sequence in the registry 🧁. So the table SHOULD be sorted, first by Class ID and then by version 📊.
This document introduces the FE Protocol Object version 1.1 like this 🦋:
+------------+----------+---------+---------------------+-----------+
| LFB Class | LFB | LFB | Description | Reference |
| Identifier | Class | Version | | |
| | Name | | | |
+------------+----------+---------+---------------------+-----------+
| 2 | FE | 1.1 | Defines parameters | [RFC7121] |
| | Protocol | | for the ForCES | |
| | Object | | protocol operation | |
+------------+----------+---------+---------------------+-----------+
Logical Functional Block (LFB) Class Names and Class Identifiers5. Security Considerations 🔐
Okay bestie, security time, and it's giving serious 😤 Security considerations, as defined in Section 9 of [RFC5810], apply to securing each CE-FE communication 💖. Multiple CEs associated with the same FE still need to follow the same procedure, done separately for each association ✨.
It's worth knowing that the FE is the one that starts the association with a CE. That means a CE cannot start an association with the FE, and those messages get dropped 🙅♀️. So the FE is protected from rogue CEs trying to sneak in and associate with it 🛡️.
CE implementers, please keep this in mind 🫶. Once a CE is associated, the FE can't tell whether that CE has been compromised or is just malfunctioning while the connection stays up 😭. Securing the CE is out of scope for this document, so that's a different problem 🙈.
The CE-CE plane is outside the current scope of ForCES, but we know it can still be attacked in ways that affect CE-FE communication 👀. So here's what to think about:
- Secure communication channels should be used between CEs for coordination and for keeping state, at the very least to avoid connections to malicious CEs 🔒.
- The master CE should take DoS and Distributed Denial-of-Service (DDoS) attacks into account, whether they come from malicious or malfunctioning CEs 🌊.
- CEs should take the split-brain issue into account 🧠. Right now the FE has two fail-safes. First, the FE has the CEID component, which shows which CE is the master. Second, the FE does not allow BackupCEs to configure the FE. Still, backup CEs that think the master CE has dropped should, as masters themselves, do a sanity check first and query the FE CEID component 🔍.
6. References 📚
6.1. Normative References 🧚
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008.
- [RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang, W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and Control Element Separation (ForCES) Protocol Specification", RFC 5810, March 2010.
- [RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control Element Separation (ForCES) Forwarding Element Model", RFC 5812, March 2010.
6.2. Informative References 💜
- [Err3487] RFC Errata, Errata ID 3487, RFC 5812,
<http://www.rfc-editor.org>. - [RFC3654] Khosravi, H. and T. Anderson, "Requirements for Separation of IP Control and Forwarding", RFC 3654, November 2003.
- [RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal, "Forwarding and Control Element Separation (ForCES) Framework", RFC 3746, April 2004.
Appendix A. New FEPO Version 🎀
The xml has been validated against the schema defined in [RFC5812] ✅. Here's the whole XML, no cap 👑:
<LFBLibrary xmlns="urn:ietf:params:xml:ns:forces:lfbmodel:1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="lfb-schema.xsd" provides="FEPO">
<!-- XXX -->
<dataTypeDefs>
<dataTypeDef>
<name>CEHBPolicyValues</name>
<synopsis>
The possible values of the CE Heartbeat policy
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>CEHBPolicy0</name>
<synopsis>
The CE will send heartbeats to the FE
every CEHDI timeout if no other messages
have been sent since.
</synopsis>
</specialValue>
<specialValue value="1">
<name>CEHBPolicy1</name>
<synopsis>
The CE will not send heartbeats to the FE
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>FEHBPolicyValues</name>
<synopsis>
The possible values of the FE Heartbeat policy
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>FEHBPolicy0</name>
<synopsis>
The FE will not generate any heartbeats to the CE
</synopsis>
</specialValue>
<specialValue value="1">
<name>FEHBPolicy1</name>
<synopsis>
The FE generates heartbeats to the CE every FEHI
if no other messages have been sent to the CE.
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>FERestartPolicyValues</name>
<synopsis>
The possible values of the FE restart policy
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>FERestartPolicy0</name>
<synopsis>
The FE restarts its state from scratch
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>HAModeValues</name>
<synopsis>
The possible values of HA modes
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>NoHA</name>
<synopsis>
The FE is not running in HA mode
</synopsis>
</specialValue>
<specialValue value="1">
<name>ColdStandby</name>
<synopsis>
The FE is running in HA mode cold standby
</synopsis>
</specialValue>
<specialValue value="2">
<name>HotStandby</name>
<synopsis>
The FE is running in HA mode hot standby
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>CEFailoverPolicyValues</name>
<synopsis>
The possible values of the CE failover policy
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>CEFailoverPolicy0</name>
<synopsis>
The FE should stop functioning immediately and
transition to the FE OperDisable state
</synopsis>
</specialValue>
<specialValue value="1">
<name>CEFailoverPolicy1</name>
<synopsis>
The FE should continue forwarding even without an
associated CE for CEFTI. The FE goes to FE
OperDisable when the CEFTI expires and there is no
association. Requires graceful restart support.
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>FEHACapab</name>
<synopsis>
The supported HA features
</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>GracefullRestart</name>
<synopsis>
The FE supports graceful restart
</synopsis>
</specialValue>
<specialValue value="1">
<name>HA</name>
<synopsis>
The FE supports HA
</synopsis>
</specialValue>
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>CEStatusType</name>
<synopsis>Status values. Status for each CE</synopsis>
<atomic>
<baseType>uchar</baseType>
<specialValues>
<specialValue value="0">
<name>Disconnected</name>
<synopsis>No connection attempt with the CE yet
</synopsis>
</specialValue>
<specialValue value="1">
<name>Connected</name>
<synopsis>The FE connection with the CE at the TML
has been completed
</synopsis>
</specialValue>
<specialValue value="2">
<name>Associated</name>
<synopsis>The FE has associated with the CE
</synopsis>
</specialValue>
<specialValue value="3">
<name>IsMaster</name>
<synopsis>The CE is the master (and associated)
</synopsis>
</specialValue>
<specialValue value="4">
<name>LostConnection</name>
<synopsis>The FE was associated with the CE but
lost the connection
</synopsis>
</specialValue>
<specialValue value="5">
<name>Unreachable</name>
<synopsis>The CE is deemed as unreachable by the FE
</synopsis>
</specialValue>Okay bestie, this is the big XML finale 🎀 Here's the rest of the FEPO model, exactly as the spec has it. Every tag, number and ID is untouched because implementers parse this stuff character for character 💅
</specialValues>
</atomic>
</dataTypeDef>
<dataTypeDef>
<name>StatisticsType</name>
<synopsis>Statistics Definition</synopsis>
<struct>
<component componentID="1">
<name>RecvPackets</name>
<synopsis>Packets received</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="2">
<name>RecvErrPackets</name>
<synopsis>Packets received from the CE with errors
</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="3">
<name>RecvBytes</name>
<synopsis>Bytes received from the CE</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="4">
<name>RecvErrBytes</name>
<synopsis>Bytes received from the CE in Error</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="5">
<name>TxmitPackets</name>
<synopsis>Packets transmitted to the CE</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="6">
<name>TxmitErrPackets</name>
<synopsis>
Packets transmitted to the CE that
incurred errors
</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="7">
<name>TxmitBytes</name>
<synopsis>Bytes transmitted to the CE</synopsis>
<typeRef>uint64</typeRef>
</component>
<component componentID="8">
<name>TxmitErrBytes</name>
<synopsis>
Bytes transmitted to the CE that
incurred errors
</synopsis>
<typeRef>uint64</typeRef>
</component>
</struct>
</dataTypeDef>
<dataTypeDef>
<name>AllCEType</name>
<synopsis>Table type for the AllCE component</synopsis>
<struct>
<component componentID="1">
<name>CEID</name>
<synopsis>ID of the CE</synopsis>
<typeRef>uint32</typeRef>
</component>
<component componentID="2">
<name>Statistics</name>
<synopsis>Statistics per the CE</synopsis>
<typeRef>StatisticsType</typeRef>
</component>
<component componentID="3">
<name>CEStatus</name>
<synopsis>Status of the CE</synopsis>
<typeRef>CEStatusType</typeRef>
</component>
</struct>
</dataTypeDef>
</dataTypeDefs>
<LFBClassDefs>
<LFBClassDef LFBClassID="2">
<name>FEPO</name>
<synopsis>
The FE Protocol Object, with new CEHA
</synopsis>
<version>1.1</version>
<components>
<component componentID="1" access="read-only">
<name>CurrentRunningVersion</name>
<synopsis>Currently running the ForCES version</synopsis>
<typeRef>uchar</typeRef>
</component>
<component componentID="2" access="read-only">
<name>FEID</name>
<synopsis>Unicast FEID</synopsis>
<typeRef>uint32</typeRef>
</component>
<component componentID="3" access="read-write">
<name>MulticastFEIDs</name>
<synopsis>
The table of all multicast IDs
</synopsis>
<array type="variable-size">
<typeRef>uint32</typeRef>
</array>
</component>
<component componentID="4" access="read-write">
<name>CEHBPolicy</name>
<synopsis>
The CE Heartbeat policy
</synopsis>
<typeRef>CEHBPolicyValues</typeRef>
</component>
<component componentID="5" access="read-write">
<name>CEHDI</name>
<synopsis>
The CE Heartbeat Dead Interval in milliseconds
</synopsis>
<typeRef>uint32</typeRef>
</component>
<component componentID="6" access="read-write">
<name>FEHBPolicy</name>
<synopsis>
The FE Heartbeat policy
</synopsis>
<typeRef>FEHBPolicyValues</typeRef>
</component>
<component componentID="7" access="read-write">
<name>FEHI</name>
<synopsis>
The FE Heartbeat Interval in milliseconds
</synopsis>
<typeRef>uint32</typeRef>
</component>
<component componentID="8" access="read-write">
<name>CEID</name>
<synopsis>
The primary CE this FE is associated with
</synopsis>
<typeRef>uint32</typeRef>
</component>
<component componentID="9" access="read-write">
<name>BackupCEs</name>
<synopsis>
The table of all backup CEs other than the
primary
</synopsis>
<array type="variable-size">
<typeRef>uint32</typeRef>
</array>
</component>
<component componentID="10" access="read-write">
<name>CEFailoverPolicy</name>
<synopsis>
The CE failover policy
</synopsis>
<typeRef>CEFailoverPolicyValues</typeRef>
</component>
<component componentID="11" access="read-write">
<name>CEFTI</name>
<synopsis>
The CE Failover Timeout Interval in milliseconds
</synopsis>
<typeRef>uint32</typeRef>
</component>
<component componentID="12" access="read-write">
<name>FERestartPolicy</name>
<synopsis>
The FE restart policy
</synopsis>
<typeRef>FERestartPolicyValues</typeRef>
</component>
<component componentID="13" access="read-write">
<name>LastCEID</name>
<synopsis>
The primary CE this FE was last associated
with
</synopsis>
<typeRef>uint32</typeRef>
</component>
<component componentID="14" access="read-write">
<name>HAMode</name>
<synopsis>
The HA mode used
</synopsis>
<typeRef>HAModeValues</typeRef>
</component>
<component componentID="15" access="read-only">
<name>AllCEs</name>
<synopsis>The table of all CEs</synopsis>
<array type="variable-size">
<typeRef>AllCEType</typeRef>
</array>
</component>
</components>
<capabilities>
<capability componentID="30">
<name>SupportableVersions</name>
<synopsis>
The table of ForCES versions that FE supports
</synopsis>
<array type="variable-size">
<typeRef>uchar</typeRef>
</array>
</capability>
<capability componentID="31">
<name>HACapabilities</name>
<synopsis>
The table of HA capabilities the FE supports
</synopsis>
<array type="variable-size">
<typeRef>FEHACapab</typeRef>
</array>
</capability>
</capabilities>
<events baseID="61">
<event eventID="1">
<name>PrimaryCEDown</name>
<synopsis>
The primary CE has changed
</synopsis>
<eventTarget>
<eventField>LastCEID</eventField>
</eventTarget>
<eventChanged/>
<eventReports>
<eventReport>
<eventField>LastCEID</eventField>
</eventReport>
</eventReports>
</event>
<event eventID="2">
<name>PrimaryCEChanged</name>
<synopsis>A new primary CE has been selected
</synopsis>
<eventTarget>
<eventField>CEID</eventField>
</eventTarget>
<eventChanged/>
<eventReports>
<eventReport>
<eventField>CEID</eventField>
</eventReport>
</eventReports>
</event>
</events>
</LFBClassDef>
</LFBClassDefs>
</LFBLibrary>Authors' Addresses ✨
Meet the squad behind the spec 🫶 Here are the four authors and how to reach them, copied exactly so nobody's email gets mangled 💌
Kentaro Ogawa 🌸
Kentaro Ogawa
NTT Corporation
3-9-11 Midori-cho
Musashino-shi, Tokyo 180-8585
Japan
EMail: k.ogawa@ntt.comWeiming Wang 🦋
Weiming Wang
Zhejiang Gongshang University
18 Xuezheng Str., Xiasha University Town
Hangzhou 310018
P.R. China
Phone: +86 571 28877751
EMail: wmwang@zjsu.edu.cnEvangelos Haleplidis 🧚
Evangelos Haleplidis
University of Patras
Department of Electrical and Computer Engineering
Patras 26500
Greece
EMail: ehalep@ece.upatras.grJamal Hadi Salim 👑
Jamal Hadi Salim
Mojatatu Networks
Suite 400, 303 Moodie Dr.
Ottawa, Ontario K2H 9R4
Canada
EMail: hadi@mojatatu.comThat's the whole spec, bestie! 🌈💖 Four legends, one protocol, zero cap 😭✨
📦 where this page comes from
- ietf: rfc7121 2014-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms