RFC 6984
Interoperability Report for Forwarding and Control Element Separation (ForCES)
RFC 6984: Interoperability Report for ForCES 💖 (part 1 of 7)
Title: Interoperability Report for Forwarding and Control Element Separation (ForCES) 🎀 Authors: W. Wang (Zhejiang Gongshang University), K. Ogawa (NTT Corporation), E. Haleplidis (University of Patras), M. Gao (Hangzhou BAUD Networks), J. Hadi Salim (Mojatatu Networks) Category: Informational 📖 Date: August 2013 📅 ISSN: 2070-1721 Updates: RFC 6053 🔄
Abstract ✨
okay so this doc is the recap of the second ForCES interop test, which went down on February 24-25, 2011, at the Internet Technology Lab (ITL) at Zhejiang Gongshang University in China. 🇨🇳 the first test's results are in RFC 6053, and this doc updates RFC 6053 by adding more interop results. it's giving sequel energy 🎬🍓
Status of This Memo 📝
bestie, this is NOT a standards track spec. it's published just for info 🧚
this doc is a product of the IETF. it's the consensus of the IETF community, it got public review, and the IESG approved it for publication 💅 (not everything the IESG approves is up for any level of Internet Standard, see Section 2 of RFC 5741 for the deets 👀)
for the current status, any errata, and how to send feedback, check http://www.rfc-editor.org/info/rfc6984 🔗
Copyright Notice 💜
Copyright (c) 2013 IETF Trust and the people listed as document authors. all rights reserved 🔒
this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stood on the publication date. pls actually read these, they explain what you can and can't do with this doc 🙈 any Code Components pulled out of this doc must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, same as the Simplified BSD License says 🫶
Table of Contents 🗂️
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. ForCES Protocol . . . . . . . . . . . . . . . . . . . . . 3
1.2. ForCES FE Model . . . . . . . . . . . . . . . . . . . . . 4
1.3. Transport Mapping Layer . . . . . . . . . . . . . . . . . 4
1.4. Definitions . . . . . . . . . . . . . . . . . . . . . . . 4
2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1. Date, Location, and Participants . . . . . . . . . . . . . 4
2.2. Testbed Configuration . . . . . . . . . . . . . . . . . . 5
2.2.1. Participants' Access . . . . . . . . . . . . . . . . . 5
2.2.2. Testbed Configuration . . . . . . . . . . . . . . . . 6
3. Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Scenario 1 - LFB Operation . . . . . . . . . . . . . . . . 7
3.2. Scenario 2 - TML with IPsec . . . . . . . . . . . . . . . 8
3.3. Scenario 3 - CE High Availability . . . . . . . . . . . . 9
3.4. Scenario 4 - Packet Forwarding . . . . . . . . . . . . . . 11
4. Test Results . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.1. Test of LFB Operation . . . . . . . . . . . . . . . . . . 14
4.2. Test of TML with IPsec . . . . . . . . . . . . . . . . . . 20
4.3. Test of CE High Availability . . . . . . . . . . . . . . . 20
4.4. Test of Packet Forwarding . . . . . . . . . . . . . . . . 21
5. Discussions . . . . . . . . . . . . . . . . . . . . . . . . . 23
5.1. On Data Encapsulation Format . . . . . . . . . . . . . . . 23
6. Security Considerations . . . . . . . . . . . . . . . . . . . 26
7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 26
7.1. Normative References . . . . . . . . . . . . . . . . . . . 26
7.2. Informative References . . . . . . . . . . . . . . . . . . 26
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 28
Appendix B. Contributors . . . . . . . . . . . . . . . . . . . . 281. Introduction ✨
okay so this doc covers the results of the second interop test of ForCES, which happened on February 24-25, 2011, at the Internet Technology Lab (ITL) at Zhejiang Gongshang University, China 🌸 the test involved protocol elements from a few different docs:
- ForCES Protocol [RFC5810] 💌
- ForCES Forwarding Element (FE) Model [RFC5812] 🧩
- ForCES Transport Mapping Layer (TML) [RFC5811] 🚚
the test also used protocol elements from the Internet-Drafts that were current at the time. those have since been revised and advanced, but it matters which versions got tested, so here they are 🕰️ the then-current Internet-Drafts were:
- "ForCES Logical Function Block (LFB) Library" (December 2010) [LFB-LIB] 📚
- "ForCES Intra-NE High Availability" (October 2010) [CEHA] 🛡️
heads up: the ForCES LFB Library doc was later published as [RFC6956] 📌
three independent ForCES implementations joined the party 🥳
they ran scenarios for ForCES LFB Operation, TML with IPsec, Control Element High Availability (CEHA), and Packet Forwarding. every scenario got a series of testing items, and they got interop results 🎉 the popular packet analyzers Ethereal/Wireshark [Ethereal] and Tcpdump [Tcpdump] were used to verify the wire results 🔍
this doc is an update to [RFC6053], which captured the results of the first ForCES interop test. that first one was held in July 2008 at the University of Patras, Greece, and it focused on checking the basic semantics of the ForCES protocol and the ForCES Forwarding Element (FE) model 🏛️
1.1. ForCES Protocol 🧠
the ForCES protocol works in a master-slave mode. FEs are the slaves and Control Elements (CEs) are the masters 👑 it includes commands for transporting Logical Function Block (LFB) configuration info, association setup, status, event notifications, and more 📨 if you want the full tea, read the ForCES protocol spec [RFC5810] 🍵
1.2. ForCES FE Model 🧱
the ForCES FE model [RFC5812] gives you a formal way to define FE LFBs using XML 🪄 the LFB configuration components, capabilities, and associated events get defined when the LFB is formally created. so the LFBs inside the FE are controlled in a standardized way by the ForCES protocol, which is honestly so clean 🧼
1.3. Transport Mapping Layer 🚀
the ForCES Transport Mapping Layer (TML) carries the ForCES protocol layer messages 📦 this is where stuff like transport-level reliability, congestion control, multicast, ordering, etc. gets handled 🎯 more than one TML is expected to be standardized. RFC 5811 specifies a TML based on the Stream Control Transmission Protocol (SCTP), and it's the mandated TML for ForCES 💫 see RFC 5811 for more details 💖
1.4. Definitions 📖
this doc follows the terminology from the ForCES-related docs, including [RFC3654], [RFC3746], [RFC5810], [RFC5811], [RFC5812], [RFC5813], etc. 🧾
2. Overview 🌈
2.1. Date, Location, and Participants 📍
the second ForCES interop test meeting was held by the IETF ForCES Working Group on February 24-25, 2011. it was chaired by Jamal Hadi Salim 🎤 three independent ForCES implementations took part:
- Zhejiang Gongshang University/Hangzhou BAUD Corporation of Information and Networks Technology (Hangzhou BAUD Networks), China. for brevity, this implementation is called "ZJSU" or "Z" in this doc 🇨🇳
- NTT Corporation, Japan. for brevity, this implementation is called "NTT" or "N" in this doc 🇯🇵
- The University of Patras, Greece. for brevity, this implementation is called "UoP" or "P" in this doc 🇬🇷
two other orgs also showed up: Mojatatu Networks and Hangzhou BAUD Networks Corporation. they had independently extended two well-known public domain protocol analyzers, Ethereal/Wireshark [Ethereal] and Tcpdump [Tcpdump], and they joined the interop test too 🕵️♀️ during the test, the two protocol analyzers were used to verify the validity (and in some cases, the semantics) of ForCES protocol messages 🔬
some interop issues came up. most of them got solved on site during the test 🛠️ the most contentious one was about the format of encapsulation for the protocol TLVs (see Section 5.1) 😤
the interop test also found some errata in the ForCES doc, and those errata got reported. errata found in related RFCs were reported too 📝
sometimes interop testing only happened between two of the three implementations, because the third one lacked a specific feature. still, in the follow-up chats, every implementer said they'd add the missing features later 🤝
2.2. Testbed Configuration 🧪
2.2.1. Participants' Access 🌐
NTT and ZJSU were physically present for testing at the Internet Technology Lab (ITL) at Zhejiang Gongshang University in China 🏫 the UoP team joined remotely from Greece 🇬🇷 the chair, Jamal Hadi Salim, joined remotely from Canada using TeamViewer as the monitoring tool [TeamViewer] 🖥️ the setup looked like Figure 1 🎀 in the figure, FE/CE means the FE or CE that the implementer may act as, depending on the situation 🔁
+---------+ +----+ +------------+
| FE/CE | | | +---| Monitoring |
| ZJSU |-----| | /\/\/\/\/\ | |(TeamViewer)|
+---------+ | | \Internet/ | | Mojatatu |
|LAN |----/ \--| +------------+
+---------+ | | \/\/\/\/\/ | +------------+
| FE/CE |-----| | | | FE/CE |
| NTT | | | +---| UoP |
+---------+ +----+ +------------+
Figure 1: Access for Participantsas specified in [RFC5811], all CEs and FEs implemented IPsec in the TML 🔐
on the Internet boundary, gateways used must allow for IPsec, the SCTP protocol, and SCTP ports as defined in the ForCES SCTP-based TML document [RFC5811] 🚧 so make sure your firewalls aren't being shady about it 🙃
part 1 of 7 done!! more coming, don't go anywhere 🧚💅
2.2.2. Testbed Configuration 🧚
okay so the CEs and FEs from ZJSU's and NTT's implementations were physically chilling inside the ITL Lab at Zhejiang Gongshang University, all hooked up together with Ethernet switches 🔌. The setup is in Figure 2. The star of the show is SmartBits [SmartBits], a third-party routing protocol testing machine that acts like a router running Open Shortest Path First (OSPF) and RIP, and it swaps routing protocol messages with the ForCES routers in the network 🦋. Internet access came through an Asymmetric Digital Subscriber Line (ADSL) channel 🌐.
/\/\/\/\/\
\Internet/
/ \
\/\/\/\/\/
|
|(ADSL)
|
+-------------------------------------------------------------------+
| LAN (10.20.0.0/24) |
+-------------------------------------------------------------------+
| | | | | |
| | | | | |
|.222 |.230 |.221 |.179 |.231 |.220
+-----+ +-----+ +-----+ +-----+ +-----+ +---------+
| CE | | CE | | | | | | | | Protocol|
|ZJSU | | NTT | | FE1 |.1 .2| FE |.1 .2| FE2 | | Analyzer|
+-----+ +-----+ |ZJSU |---------| NTT |---------|ZJSU | +---------+
+---------| |192.169. | | 192.168.| |------+
| .2 +-----+ 20.0.24 +-----+ 30.0/24+-----+ .2 |
| .12| |.12 |
| | | |
192.168.50.0/24 | |192.168.60.0/24
| 192.168.10.0/24 192.168.40.0/24 |
.1 | |.11 |.11 |.1
+--------+ +--------------------------------------+ +--------+
|Terminal| | SmartBits | |Terminal|
+--------+ +--------------------------------------+ +--------+
Figure 2: Testbed Configuration Located in the ITL Lab, ChinaNext up, the CE and FE from the UoP implementation were chilling at the University of Patras, Greece, connected together over a LAN, as shown in Figure 3 🏛️. Their Internet hookup was a VPN channel 🔐.
/\/\/\/\/\
\Internet/
/ \
\/\/\/\/\/
|
|(VPN)
|
+------------------------------------+
| LAN |
+------------------------------------+
| | |
| | |
+------+ +--------+ +------+
| FE | |Protocol| | CE |
| UoP | |Analyzer| | UoP |
+------+ +--------+ +------+
Figure 3: Testbed Configuration
Located in the University of Patras, GreeceAnd honestly? Both testbeds totally slayed 💅. Together they were able to satisfy the requirements of every interoperability test scenario in this document 🎉.
3. Scenarios ✨
3.1. Scenario 1 - LFB Operation 🌸
This scenario was designed to test the interoperability of LFB operations among the participants. The connection diagram is shown in Figure 4 💖.
+------+ +------+ +------+ +------+ +------+ +------+
| CE | | CE | | CE | | CE | | CE | | CE |
| ZJSU | | NTT | | ZJSU | | UoP | | NTT | | UoP |
+------+ +------+ +------+ +------+ +------+ +------+
| | | | | |
| | | | | |
+------+ +------+ +------+ +------+ +------+ +------+
| FE | | FE | | FE | | FE | | FE | | FE |
| NTT | | ZJSU | | UoP | | ZJSU | | UoP | | NTT |
+------+ +------+ +------+ +------+ +------+ +------+
Figure 4: Scenario for LFB OperationTo make the interop extra credible, the three implementers were required to run the test acting as a CE or FE, taking turns 🔄. So every LFB operation got combined with six scenarios, as shown by Figure 4. Iconic. 👑
The test scenario had two main purposes 🎯.
Firstly, it was designed to verify all kinds of protocol messages with their complex data formats, which were defined in [RFC5810] 📦. Specifically, we tried to verify the data format of a PATH-DATA-TLV with nested PATH-DATA-TLVs, and the operation (SET, GET, and DEL) of an array or an array with a nested array. Nested everything, bestie 😭.
Secondly, it was designed to verify the definition of the ForCES LFB Library [LFB-LIB], which defined a base set of ForCES LFB classes for typical router functions 🧩. Successful tests under this scenario would help the validity of the LFB definitions ✅.
3.2. Scenario 2 - TML with IPsec 🔐
This scenario was designed to implement a TML with IPsec, which was the requirement defined by RFC 5811. TML with IPsec was not implemented and tested in the first ForCES interoperability test, as reported in RFC 6053 🙈. So for this round, we specifically designed the test scenario to verify the TML over IPsec channel 🛡️.
In this scenario, tests on LFB operations for Scenario 1 were repeated, with the difference that TML was secured via IPsec. This setup allowed us to verify whether all interactions between the CE and FE could be made correctly under an IPsec TML environment 🦋.
The connection diagram for this scenario is shown in Figure 5. Because an unfortunate problem with the test system in the UoP prevented the deployment of IPsec over TML, this test only took place between the test systems in ZJSU and NTT 😢.
+------+ +------+
| CE | | CE |
| ZJSU | | NTT |
+------+ +------+
| |
|TML over IPsec |TML over IPsec
+------+ +------+
| FE | | FE |
| NTT | | ZJSU |
+------+ +------+
Figure 5: Scenario for LFB Operation with TML over IPsecIn this scenario, ForCES TML ran over the IPsec channel. Implementers joined this interop test using the same third-party software 'Racoon' [Racoon] to establish the IPsec channel 🧪.
The Racoon in NetBSD is an Internet Key Exchange (IKE) daemon that performs the key exchange with the peers 🔑. Both IKEv1 and IKEv2 are supported by Racoon in Linux 2.6, and IKEv2 was adopted in the interop test. The Security Association Database (SAD) and Security Policy Database (SPD) were necessary for the test, and their setups lived in the Racoon configuration file 📝. The Encapsulating Security Payload (ESP) was specified in the SAD and SPD in the Racoon configuration file too.
ZJSU and NTT conducted a successful test with this scenario, and the IPsec requirement items in [RFC5812] were realized 🎉🥳.
3.3. Scenario 3 - CE High Availability 🛟
CE High Availability (CEHA) was tested based on the ForCES CEHA document [CEHA] 💜.
The setup and scenario for the CEHA were simplified so they could focus mostly on the mechanics of CEHA, which were:
- Associating with more than one CE.
- Switching to a backup CE on a master CE failure.
The connection diagram for the scenario is shown in Figure 6 🗺️.
master standby master standby
+------+ +------+ +------+ +------+
| CE | | CE | | CE | | CE |
| ZJSU | | UoP | | NTT | | UoP |
+------+ +------+ +------+ +------+
| | | |
+----------+ +-----------+
| |
+------+ +------+
| FE | | FE |
| UoP | | UoP |
+------+ +------+
(a) (b)
Figure 6: Scenario for CE High AvailabilityIn this scenario, one FE was connected and associated to a master CE and a backup CE. In the pre-association phase, the FE would be configured to have ZJSU's or NTT's CE as the master CE, and the UoP's CE as the standby CE 🧑🤝🧑.
The CEFailoverPolicy component of the FE Protocol Object LFB specifies whether the FE is in High Availability mode (value 2 or 3). It would be set either in the pre-association phase by the FE interface, or in the post-association phase by the master CE 🎛️.
If the CEFailoverPolicy value was set to 2 or 3, the FE (in the post-association phase) would attempt to connect and associate with the standby CE 🔗.
When the master CE was deemed disconnected, whether by a TearDown, Loss of Heartbeats, or being physically disconnected, the FE would assume that the standby CE was now the master CE 💀. The FE would then send an Event Notification, Primary CE Down, to all associated CEs (only the standby CE in this case), with the value of the new master Control Element ID (CEID). The standby CE would then respond by sending a configuration message to the CEID component of the FE Protocol Object with its own ID, to confirm that it considered itself the master as well 👑.
The steps of the CEHA test scenario went like this 📋:
- In the pre-association phase, the FE is set up with the master CE and the backup CE.
- The FE connects and associates with the master CE.
- When
CEFailoverPolicyis set to 2 or 3, the FE connects and associates with the backup CE. - Once the master CE is considered disconnected, the FE chooses the first associated backup CE.
- It sends an Event Notification that specifies the master CE is down and identifies the new master CE.
- The new master CE sends a SET Configuration message to the FE; the FE then sets the
CEIDvalue to the new master CE, completing the switch 🦋✨
3.4. Scenario 4 - Packet Forwarding ✨
Okay bestie, this scenario was all about checking the LFBs RedirectIn, RedirectOut, IPv4NextHop, and IPv4UcastLPM, which are defined in the ForCES LFB library document [LFB-LIB]. But the real main character moment 🦋 was verifying that these LFBs work together to do IP packet forwarding 💅
The connection diagram for this one is in Figure 7 🎀
+------+
| CE |
| NTT |
+------+
| ^
| | OSPF
| +------->
+------+ +------+
+--------+ | FE | | OSPF | +--------+
|Terminal|------| ZJSU |-------|Router|------|Terminal|
+--------+ +------+ +------+ +--------+
<-------------------------------------------->
Packet Forwarding
(a)
+------+
| CE |
| ZJSU |
+------+
^ | ^
OSPF | | | OSPF
<-----+ | +----->
+-------+ +------+ +------+
+--------+ | OSPF | | FE | | OSPF | +--------+
|Terminal|----|Router |----| NTT |-----|Router|--|Terminal|
+--------+ +-------+ +------+ +------+ +--------+
<-------------------------------------------->
Packet Forwarding
(b)
+------+ +------+
| CE | | CE |
| NTT | | ZJSU |
+------+ +------+
| ^ ^ |
| | OSPF | |
| +----------+ |
+------+ +------+
+--------+ | FE | | FE | +--------+
|Terminal|------| ZJSU |-------| NTT |------|Terminal|
+--------+ +------+ +------+ +--------+
<-------------------------------------------->
Packet Forwarding
(c)
Figure 7: Scenario for IP Packet ForwardingIn case (a), NTT's CE was connected to ZJSU's FE to form a ForCES router. A SmartBits [SmartBits] test machine equipped with routing protocol software was used to simulate an OSPF router, and it was connected with the ForCES router to try to exchange OSPF Hello packets and Link State Advertisement (LSA) packets among them. Terminals were simulated by SmartBits to send and receive packets. As a result, the CE in the ForCES router needed to be configured to run and support the OSPF routing protocol.
In case (b), ZJSU'S CE was connected to NTT'S FE to form a ForCES router. Two routers running OSPF were simulated and connected to the ForCES router to test whether the ForCES router could support the OSPF protocol and support packet forwarding.
In case (c), two ForCES routers were constructed: one was built with NTT's CE and ZJSU's FE, and the other was built with NTT's FE and ZJSU's CE. OSPF and packet forwarding were tested in this environment 🌈
The testing process for this scenario goes like this:
- Boot terminals and routers, and set the IP addresses of their interfaces.
- Boot the CE and FE.
- Establish an association between the CE and FE, and set the IP addresses of the FE interfaces.
- Start OSPF among the CE and routers, and set the Forwarding Information Base (FIB) on the FE.
- Send packets between terminals.
4. Test Results 🎀
4.1. Test of LFB Operation
The test results are reported in Figure 8 📊 As mentioned earlier, for convenience, the following abbreviations are used in the table: "Z" for the implementation from ZJSU, "N" for the implementation from NTT, and "P" for the implementation from the UoP 🫶
Okay bestie, this is the big test grid for LFB operation 🎀✨ Each row is one test, and every test runs through the same little combos of CE, FE(s), and Oper values. We're checking the GET operation on a bunch of LFB components and capabilities, and the results column is where the magic happens 💅
+-----+----+-----+-----+--------------+-------------------+---------+
|Test#| CE |FE(s)|Oper | LFB | Component | Result |
| | | | | | /Capability | |
+-----+----+-----+-----+--------------+-------------------+---------+
| 1 | Z | N | GET | FEObject | LFBTopology | Success |
| | N | Z | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 2 | Z | N | GET | FEObject | LFBSelector | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 3 | Z | N | GET | EtherPHYCop | PHYPortID | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 4 | Z | N | GET | EtherPHYCop | AdminStatus | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 5 | Z | N | GET | EtherPHYCop | OperStatus | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 6 | Z | N | GET | EtherPHYCop | AdminLinkSpeed | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 7 | Z | N | GET | EtherPHYCop | OperLinkSpeed | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 8 | Z | N | GET | EtherPHYCop | AdminDuplexSpeed | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 9 | Z | N | GET | EtherPHYCop | OperDuplexSpeed | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 10 | Z | N | GET | EtherPHYCop | CarrierStatus | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 11 | Z | N | GET | EtherMACIn | AdminStatus | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 12 | Z | N | GET | EtherMACIn | LocalMacAddresses | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 13 | Z | N | GET | EtherMACIn | L2Bridging | Success |
| | N | Z | | | PathEnable | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 14 | Z | N | GET | EtherMACIn | PromiscuousMode | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 15 | Z | N | GET | EtherMACIn | TxFlowControl | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 16 | Z | N | GET | EtherMACIn | RxFlowControl | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 17 | Z | N | GET | EtherMACIn | MACInStats | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 18 | Z | N | GET | EtherMACOut | AdminStatus | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 19 | Z | N | GET | EtherMACOut | MTU | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 20 | Z | N | GET | EtherMACOut | TxFlowControl | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 21 | Z | N | GET | EtherMACOut | TxFlowControl | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 22 | Z | N | GET | EtherMACOut | MACOutStats | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 23 | Z | N | GET | ARP |PortV4AddrInfoTable| Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |Every single row came back as Success, so the whole LFB operation test looks super healthy 🦋🌈
| 24 | Z | N | SET | ARP |PortV4AddrInfoTable| Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 25 | Z | N | DEL | ARP |PortV4AddrInfoTable| Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 26 | Z | N | SET | EtherMACIn | LocalMACAddresses | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 27 | Z | N | SET | EtherMACIn | MTU | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 28 | Z | N | SET | IPv4NextHop | IPv4NextHopTable | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 29 | Z | N | SET | IPv4UcastLPM | IPv4PrefixTable | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 30 | Z | N | DEL | IPv4NextHop | IPv4NextHopTable | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 31 | Z | N | DEL | IPv4UcastLPM | IPv4PrefixTable | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 32 | Z | N | SET | EtherPHYCop | AdminStatus | Success |
| | N | Z | | | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 33 | Z | N | SET | Ether | VlanInputTable | Success |
| | N | Z | | Classifier | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 34 | Z | N | DEL | Ether | VlanInputTable | Success |
| | N | Z | | Classifier | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 35 | Z | N | SET | Ether | VlanOutputTable | Success |
| | N | Z | | Encapsulator | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
| | | | | | | |
| 36 | Z | N | DEL | Ether | VlanOutputTable | Success |
| | N | Z | | Encapsulator | | Success |
| | Z | P | | | | Success |
| | P | Z | | | | Success |
| | N | P | | | | Success |
| | P | N | | | | Success |
+-----+----+-----+-----+--------------+-------------------+---------+Figure 8: Test Results for LFB Operation
Note on tests #1 and #2:
Okay so for the wire format of encapsulation on array, the only case we tested was FULLDATA-TLV vs. SPARSEDATA-TLV 🎀 That's it, that's the tea.
When we use the ForCES protocol, it is super common for the CE to use the FEobject LFB to grab info on LFBs and their topology in the FE 🧚 That's why these two tests got their own spot, bestie 💅
4.2. Test of TML with IPsec 🔐
In this scenario, the ForCES TML got run over IPsec 🦋 Implementers joined this interop test and all used the same third-party tool software 'Racoon' [Racoon] to set up the IPsec channel. The typical LFB operation tests from Scenario 1 got repeated with the IPsec-enabled TML, so it's giving déjà vu but make it secure 🙈
Just so you know, this scenario only happened between implementers from ZJSU and NTT 🌸
The TML with IPsec test results are in Figure 9 💖
+-----+----+-----+-----+--------------+-------------------+---------+
|Test#| CE |FE(s)|Oper | LFB | Component/ | Result |
| | | | | | Capability | |
+-----+----+-----+-----+--------------+-------------------+---------+
| 1 | Z | N | GET | FEObject | LFBTopology | Success |
| | N | Z | | | | Success |
| | | | | | | |
| 2 | Z | N | GET | FEObject | LFBSelectors | Success |
| | N | Z | | | | Success |
| | | | | | | |
| 3 | Z | N | SET | Ether | VlanInputTable | Success |
| | N | Z | | Classifier | | Success |
| | | | | | | |
| 4 | Z | N | DEL | Ether | VlanInputTable | Success |
| | N | Z | | Classifier | | Success |
+-----+----+-----+-----+--------------+-------------------+---------+Figure 9: Test Results for TML with IPsec
4.3. Test of CE High Availability 🫶
In this scenario, one FE connected and associated with a master CE and a backup CE 💜 When the master CE got deemed disconnected, the FE tried to find another associated CE to become the new master CE.
The CEHA scenario, the one described in Scenario 3, was completed successfully for both setups ✨
Now here's the spicy part 🌶️ Due to a bug in one of the FEs, an interesting issue got caught: the buggy FE took up to a second to failover. Eventually it was found that the cause was the FE's prioritization of the different CEs. All messages from the backup CE were being ignored unless the master CE was disconnected 😭
The bug got fixed and the CEHA scenario finished successfully, but the authors felt it was important to capture this implementation issue in the document anyway. Main character energy for documenting the messy stuff 👑
The recommended approach goes like this 🍓
- The FE should receive and handle messages first from the master CE on all priority channels, so everything keeps working properly, and then receive and handle messages from the backup CEs.
- Only when the FE is attempting to associate with the backup CEs should the FE receive and handle messages per priority channel from all CEs. When all backup CEs are associated with or deemed unreachable, then the FE should go back to receiving and handling messages first from the master CE 🌈
4.4. Test of Packet Forwarding 🦋
okay so packet forwarding is built out of a bunch of LFB classes that chain together into a processing path for packets, as laid out in the ForCES LFB library [LFB-LIB] 💅 for this test, we built a ForCES router running OSPF (Figure 7). we also used RedirectIn, RedirectOut, IPv4UcastLPM, and IPv4NextHop. RedirectIn and RedirectOut handled the redirecting of OSPF Hello and LSA packets to and from the CE. a SmartBits [SmartBits] test box pretended to be an OSPF router and swapped those Hello and LSA packets with the CE inside the ForCES router 🤖
in Figure 7, cases (a) and (b) both need a RedirectIn LFB so the OSPF packets the CE makes can get sent over to the FE using ForCES packet redirect messages. from there, the FE pushed those packets out to an outside OSPF router using the forwarding LFBs (IPv4NextHop and IPv4UcastLPM) 🚀 and a RedirectOut LFB in the FE sent the OSPF packets coming back from the outside router up to the CE through ForCES packet redirect messages. the way it all ties together is so satisfying 😭
running OSPF let the CE in the ForCES router generate new routes and load them into the routing table in the FE. and then the FE could forward packets based on that routing table. main character energy 👑
the test results are in Figure 10 🌸
+-----+----+-----+-------------------------+--------------+---------+
|Test#| CE |FE(s)| Item | LFBs Related | Result |
+-----+----+-----+-------------------------+--------------+---------+
| 1 | N | Z | IPv4NextHopTable SET | IPv4NextHop | Success |
| | | | | | |
| 2 | N | Z | IPv4PrefixTable SET | IPv4UcastLPM | Success |
| | | | | | |
| 3 | N | Z |Redirect OSPF packet from| RedirectIn | Success |
| | | | CE to SmartBits | | |
| | | | | | |
| 4 | N | Z |Redirect OSPF packet from| RedirectOut | Success |
| | | | SmartBits to CE | | |
| | | | | | |
| 5 | N | Z | Metadata in | RedirectOut | Success |
| | | | redirect message | RedirectIn | |
| | | | | | |
| 6 | N | Z | OSPF neighbor discovery | RedirectOut | Success |
| | | | | RedirectIn | |
| | | | | | |
| 7 | N | Z | OSPF DD exchange | RedirectOut | Success |
| | | | | RedirectIn | |
| | | | | IPv4NextHop | |
| | | | | | |
| 8 | N | Z | OSPF LSA exchange | RedirectOut | Success |
| | | | | RedirectIn | |
| | | | | IPv4NextHop | |
| | | | | IPv4UcastLPM| |
| | | | | | |
| 9 | N | Z | Data Forwarding | RedirectOut | Success |
| | | | | RedirectIn | |
| | | | | IPv4NextHop | |
| | | | | IPv4UcastLPM| |
| | | | | | |
| 10 | Z | N | IPv4NextHopTable SET | IPv4NextHop | Success |
| | | | | | |
| 11 | Z | N | IPv4PrefixTable SET | IPv4UcastLPM| Success |
| | | | | | |
| 12 | Z | N |Redirect OSPF packet from| RedirectIn | Success |
| | | | CE to other OSPF router | | |
| | | | | | |
| 13 | Z | N |Redirect OSPF packet from| RedirectOut | Success |
| | | |other OSPF router to CE | | |
| | | | | | |
| 14 | Z | N | Metadata in | RedirectOut | Success |
| | | | redirect message | RedirectIn | |
| | | | | | |
| 15 | Z | N |OSPF neighbor discovery | RedirectOut | Success |
| | | | | RedirectIn | |
| | | | | | |
| 16 | Z | N | OSPF DD exchange | RedirectOut | Failure |
| | | | | RedirectIn | |
| | | | | IPv4NextHop | |
| | | | | | |
| 17 | Z | N | OSPF LSA exchange | RedirectOut | Failure |
| | | | | RedirectIn | |
| | | | | IPv4NextHop | |
| | | | | IPv4UcastLPM| |
+-----+----+-----+-------------------------+--------------+---------+ Figure 10: Test Results for Packet ForwardingNote on tests #3 to #9: during the test, Ethereal/Wireshark spotted checksum errors on the OSPF packets coming from the CE in the FE 🙈 the test window was super tight, so the CE implementer didn't try to track down and fix the checksum error. instead, the FE tried to correct the checksum so the SmartBits wouldn't drop the packets. tea: this workaround doesn't change the test results 🍓
Comment on tests #16 and #17: both of these test items failed 💀 but tests #7 and #8 are identical to tests #16 and #17, just with the CE and FE implementers swapped. plus, tests #12 and #13 showed the redirect channel working just fine. so it's reasonable to infer the failure came from the implementations, not from the ForCES protocol itself or from anyone misreading the protocol specification 🧠 the failure did leave the OSPF interoperability test incomplete, but it didn't reveal an interoperability problem. more test work is needed to verify OSPF interoperability ✨
5. Discussions 💬
5.1. On Data Encapsulation Format 🎀
on day one of testing, every LFB interoperation involving tables failed 😭 it turned out that different implementations were using different data encapsulation methods for ForCES protocol messages. here's the deal, explained below 👇
say an LFB has two components: a struct with ID=1 and an array with ID=2. both of them contain two u32 components, like this:
struct1: type struct, ID=1
components are:
a, type u32, ID=1
b, type u32, ID=2
table1: type array, ID=2
components for each row are (a struct of):
x, type u32, ID=1
y, type u32, ID=21. On Response of PATH-DATA-TLV Format
when a CE sends a config/query ForCES protocol message to an FE made by a different implementer, the CE will probably get back a response from the FE that uses a different PATH-DATA-TLV encapsulation format 🫶 for example, if a CE sends a query with a path of 1 to a third-party FE to manipulate struct1 above, the FE will probably respond in one of two different PATH-DATA-TLV encapsulation formats. one is the value with FULLDATA-TLV/SPARSEDATA-TLV. the other is the value with many parallel PATH-DATA-TLVs and nested PATH-DATA-TLVs, like this:
format 1:
OPER = GET-RESPONSE-TLV
PATH-DATA-TLV:
IDs=1
FULLDATA-TLV containing valueof(a),valueof(b)
format 2:
OPER = GET-RESPONSE-TLV
PATH-DATA-TLV:
IDs=1
PATH-DATA-TLV:
IDs=1
FULLDATA-TLV containing valueof(a)
PATH-DATA-TLV:
IDs=2
FULLDATA-TLV containing valueof(b)the interoperability testers saw that a ForCES element (CE or FE) sender is free to pick whatever data structure the IETF ForCES documents define and that suits the element best 💖 meanwhile, a ForCES element (CE or FE) should be able to accept and process requests and responses that use any legitimate structure defined by IETF ForCES documents. even though an element can pick any legitimate data structure for a response, it's preferred that the element responds in the same format the request was made in, because that's most likely the structure the request sender is looking to get back 🌈
2. On Operation to Array
an array operation can also come in a few different data encapsulation formats ✨ for instance, if a CE sends a config message to table1 with a path of (2.1), which refers to the component with ID=2 (an array) where the second ID is the row, then row 1 can be encapsulated in three formats, like so:
format 1:
OPER = SET-TLV
PATH-DATA-TLV:
IDs=2.1
FULLDATA-TLV containing valueof(x),valueof(y)
format 2:
OPER = SET-TLV
PATH-DATA-TLV:
IDs=2.1
PATH-DATA-TLV:
IDs=1
FULLDATA-TLV containing valueof(x)
PATH-DATA-TLV
IDs=2
FULLDATA-TLV containing valueof(y)and if the CE is targeting the whole array (say the array is empty and the CE wants to add the very first row to the table), it can also use another format:
format 3:
OPER = SET-TLV
PATH-DATA-TLV:
IDs=2
FULLDATA-TLV containing rowindex=1,valueof(x),valueof(y)the interop test experience showed that formats 1 and 3 are more efficient 🔥 they really take advantage of describing multiple data elements inside one FULLDATA-TLV. format 2 can still hit the same goal too, though 💅
6. Security Considerations 🔐
developers of ForCES FEs and CEs need to take the security considerations of the ForCES Framework [RFC3746] and the ForCES Protocol Specification [RFC5810] into account 🛡️ also, as specified in the security considerations of SCTP-Based TML for the ForCES Protocol [RFC5811], transport-level security has to be ensured by IPsec. the test results for TML with IPsec are in Section 4.2 of this document 🫶
the tests in this document only used simple password security mode 😬 testing with more sophisticated security is for future study 🔮
further testing using key agility is encouraged 💜 the tests reported here ran SCTP TML over an IPsec tunnel that was set up by Racoon. key negotiation was part of that process, but we believe the SCTP TML used doesn't include key agility or renegotiation 🙈
7. References 📚
7.1. Normative References
okay so these are the must-read docs, bestie 📚✨ the core ForCES specs that the whole thing leans on 🫶
[RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H.,
Wang, W., Dong, L., Gopal, R., and J. Halpern,
"Forwarding and Control Element Separation (ForCES)
Protocol Specification", RFC 5810, March 2010.
[RFC5811] Hadi Salim, J. and K. Ogawa, "SCTP-Based Transport
Mapping Layer (TML) for the Forwarding and Control
Element Separation (ForCES) Protocol", RFC 5811,
March 2010.
[RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control
Element Separation (ForCES) Forwarding Element Model",
RFC 5812, March 2010.
[RFC5813] Haas, R., "Forwarding and Control Element Separation
(ForCES) MIB", RFC 5813, March 2010.7.2. Informative References
these ones are the extra reading 🌸 background, drafts, mailing list threads, and tools people used while testing 💅
[CEHA] Ogawa, K., Wang, W., Haleplidis, E., and J. Salim,
"ForCES Intra-NE High Availability", Work in Progress,
October 2010.
[Ethereal] Fenggen, J., "Subject: Release of a test version of
ForCES dissector based on Ethereal 0.99.0", message to
the IETF forces mailing list, 11 June 2009,
<http://www.ietf.org/mail-archive/web/forces/current/
msg03687.html>.
[LFB-LIB] Wang, W., Haleplidis, E., Ogawa, K., Li, C., and J.
Halpern, "ForCES Logical Function Block (LFB) Library",
Work in Progress, December 2010.
[RFC3654] Khosravi, H. and T. Anderson, "Requirements for
Separation of IP Control and Forwarding", RFC 3654,
November 2003.
[RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal,
"Forwarding and Control Element Separation (ForCES)
Framework", RFC 3746, April 2004.
[RFC6053] Haleplidis, E., Ogawa, K., Wang, W., and J. Hadi Salim,
"Implementation Report for Forwarding and Control
Element Separation (ForCES)", RFC 6053, November 2010.
[RFC6956] Wang, W., Haleplidis, E., Ogawa, K., Li, C., and J.
Halpern, "Forwarding and Control Element Separation
(ForCES) Logical Function Block (LFB) Library",
RFC 6956, June 2013.
[Racoon] The NetBSD Foundation, "How to build a remote user
access VPN with Racoon",
<http://www.netbsd.org/docs/network/ipsec/rasvpn.html>.
[SmartBits] Spirent Inc., "The Highly-Scalable Router Performance
Tester: TeraRouting Tester", 2005,
<http://www.spirent.com/~/media/Datasheets/Broadband/
Obsolete_SMB-TM/TeraRouting%20Tester.pdf>.
[Tcpdump] Hadi Salim, J., "Subject: tcpdump 4.1.1", message to
the IETF forces mailing list, 20 May 2010,
<http://www.ietf.org/mail-archive/web/forces/current/
msg03811.html>.
[TeamViewer] TeamViewer Inc., "TeamViewer - the All-In-One Software
for Remote Support and Online Meetings",
<http://www.teamviewer.com/>.Appendix A. Acknowledgements ✨
the authors gave a big shoutout to these test participants 🎀
Chuanhuang Li, Hangzhou BAUD Networks
Ligang Dong, Zhejiang Gongshang University
Bin Zhuge, Zhejiang Gongshang University
Jingjing Zhou, Zhejiang Gongshang University
Liaoyuan Ke, Hangzhou BAUD Networks
Kelei Jin, Hangzhou BAUD Networksand a huge thank-you 🫶 to Adrian Farrel, Joel Halpern, Ben Campbell, Nevil Brownlee, and Sean Turner for helping so much with getting the document published 💖
Appendix B. Contributors 🌈
these people made major contributions to the interoperability test, so they get the spotlight 🔥
Hirofumi Yamazaki
NTT Corporation
Tokyo
Japan
EMail: yamazaki.horofumi@lab.ntt.co.jp
Rong Jin
Zhejiang Gongshang University
Hangzhou
P.R. China
EMail: jinrong@zjsu.edu.cn
Yuta Watanabe
NTT Corporation
Tokyo
Japan
EMail: yuta.watanabe@ntt-at.co.jp
Xiaochun Wu
Zhejiang Gongshang University
Hangzhou
P.R. China
EMail: spring-403@zjsu.edu.cnAuthors' Addresses 💜
and finally, the authors, with their contact details 🦋
Weiming Wang
Zhejiang Gongshang University
18 Xuezheng Str., Xiasha University Town
Hangzhou 310018
P.R. China
Phone: +86-571-28877721
EMail: wmwang@zjsu.edu.cn
Kentaro Ogawa
NTT Corporation
Tokyo
Japan
EMail: ogawa.kentaro@lab.ntt.co.jp
Evangelos Haleplidis
University of Patras
Department of Electrical & Computer Engineering
Patras 26500
Greece
EMail: ehalep@ece.upatras.gr
Ming Gao
Hangzhou BAUD Networks
408 Wen-San Road
Hangzhou 310012
P.R. China
EMail: gaoming@mail.zjgsu.edu.cn
Jamal Hadi Salim
Mojatatu Networks
Ottawa
Canada
EMail: hadi@mojatatu.com📦 where this page comes from
- ietf: rfc6984 2013-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms