RFC 6847
Fibre Channel over Ethernet (FCoE) over Transparent Interconnection of Lots of Links (TRILL)
RFC 6847: Fibre Channel over Ethernet (FCoE) over Transparent Interconnection of Lots of Links (TRILL) 🦋
Okay bestie, here's the title block tea ☕✨
- Request for Comments: 6847
- Stream: Independent Submission
- Category: Informational
- Date: January 2013
- ISSN: 2070-1721
- Authors: D. Melman and T. Mizrahi (both from Marvell), and D. Eastlake 3rd (from Huawei)
- Obsoletes / Updates: The title block doesn't list any RFCs that this one obsoletes or updates. 🙈
Abstract ✨
Okay so FCoE (Fibre Channel over Ethernet) and TRILL (Transparent Interconnection of Lots of Links) are two up-and-coming standards in the data center world 🏢💖. They seem totally unrelated, but their forwarding planes are lowkey twins 👯♀️. Both do hop-by-hop forwarding over Ethernet, and both change the packet's Media Access Control (MAC) addresses at every hop. This document is all about an architecture for deploying the two together, so they can work as one integrated team 🫶.
Status of This Memo 📜
This document is not an Internet Standards Track specification. It's published for informational purposes, so no cap, it's not a standard 💅.
This is a contribution to the RFC Series, independent of any other RFC stream. The RFC Editor chose to publish it at their discretion, and they make no statement about its value for implementation or deployment. Documents the RFC Editor approves for publication are not candidates for any level of Internet Standard. For the details, see Section 2 of RFC 5741.
Information about this document's current status, any errata, and how to give feedback on it can be found at http://www.rfc-editor.org/info/rfc6847 🌸.
Copyright Notice 🎀
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were in effect on the date this document was published. Please review those documents carefully, because they describe your rights and restrictions with respect to this document. Seriously, read them 👀.
Table of Contents 🗂️
1. Introduction ................................................. 2
2. Abbreviations ................................................ 3
3. FCoE over TRILL .............................................. 4
3.1. FCoE over a TRILL Cloud ................................. 4
3.2. FCoE over an RBridge .................................... 5
3.2.1. FCRB ............................................... 5
3.2.2. Topology ........................................... 7
3.2.3. The FCRB Flow ..................................... 8
3.2.3.1. Example - ENode to ENode .................... 8
3.2.3.1.1. Forwarding from A to C in Dense Mode .... 9
3.2.3.1.2. Forwarding from A to C in Sparse Mode ... 9
3.2.3.2. Example - ENode to Native FC Node ............ 10
3.2.3.3. Example - ENode to ENode with Non-FCRB EoR ... 10
3.2.3.4. Example - FCoE Control Traffic through an FCRB 11
4. Security Considerations ..................................... 12
5. Acknowledgments ............................................. 12
6. References .................................................. 12
6.1. Normative References ................................... 12
6.2. Informative References ................................. 121. Introduction ✨
Data center networks are rapidly evolving into a consolidated setup, where Ethernet is the common infrastructure for every type of traffic 🌈. Storage traffic used to be dominated by the Fibre Channel (FC) protocol suite. Then, right where those two worlds meet, a new technology was born: Fibre Channel over Ethernet (FCoE) 🧚. It takes native FC packets and wraps them in an FCoE encapsulation over an Ethernet header. FCoE is specified in [FC-BB-5]. (A future version of FCoE is under development and is expected to be specified in a document called FC-BB-6. That's a work in progress though, and it's beyond the scope of this document 🙈.)
Traffic between two FCoE end nodes (ENodes) gets forwarded through one or more FCoE Forwarders (FCFs). An FCF makes its forwarding decision based on the Fibre Channel destination ID (D_ID), and it enforces security policies between ENodes, which is also known as zoning 🔒. Once an FCF makes that decision, it rewrites the source and destination MAC addresses of the packet to reflect the path to the next-hop FCF or ENode. An FCoE virtual link is an Ethernet link between an ENode and an FCF, or between two FCFs. An FCoE virtual link can go through one or more Layer 2 bridges. FCFs use a routing protocol called Fabric Shortest Path First (FSPF) to find the optimal path to each destination 🗺️. An FCF usually has one or more native Fibre Channel interfaces, so it can talk to native Fibre Channel devices, like storage arrays 💾.
TRILL [TRILL] is a protocol for transparent least-cost routing. Routing Bridges (RBridges) forward traffic to its destination along a least-cost route, using a TRILL encapsulation header 🦋. RBridges route TRILL-encapsulated packets based on the egress RBridge nickname in the TRILL header. An RBridge routes a TRILL-encapsulated packet after it changes the MAC addresses to reflect the path to the next-hop RBridge, and it decrements a Hop Count field.
TRILL and FCoE look super similar in their forwarding planes, honestly it's giving twins 👯♀️. Both protocols make a routing decision based on protocol addresses above Layer 2, and both change the Ethernet MAC addresses on a per-hop basis. Each protocol uses its own routing protocol instead of any kind of bridging protocol, like the spanning tree protocol [802.1Q] or the Shortest Path Bridging protocol [802.1Q] 🙅♀️.
FCoE and TRILL are both aimed at the data center environment, so using them together is pretty much self-evident 💅. This document describes an architecture for deploying these two protocols in an integrated way. ✨
2. Abbreviations 📚
DCB Data Center Bridging
ENode FCoE Node such as server or storage array
EoR End of Row
FC Fibre Channel
FCF FCoE Forwarder
FCoE Fibre Channel over Ethernet
FCRB FCF over RBridge
FIP FCoE Initialization Protocol
FSPF Fabric Shortest Path First
LAN Local Area Network
RBridge Routing Bridge
SAN Storage Area Network
ToR Top of Rack
TRILL Transparent Interconnection of Lots of Links
WAN Wide Area Network3. FCoE over TRILL 🧚
3.1. FCoE over a TRILL Cloud ☁️
The simplest way to run FCoE traffic over a TRILL network is shown in Figure 1. The figure shows a TRILL-enabled network, where FCoE traffic is transparently forwarded over the TRILL cloud. It shows two ENodes, a Server and an FCoE Storage Array, plus an FCF and a native Fibre Channel SAN connected to that FCF 💖.
FCoE traffic between the two ENodes goes from the first ENode over the TRILL cloud to the FCF, and then back through the TRILL cloud to the second ENode.
+---+
| |_________
| | \ ___ _
+---+ \/ \_/ \__ _ __
FCoE Storage _/ \ / \_/ \_
Array / TRILL / +---+ \_ \
(ENode A) \_ Cloud /________| |____/ SAN _/
/ \ | | \__ _/
\__/\_ ___/ +---+ \_/
+---+ / \_/ FCF
| |________/
| |
+---+
Server
(ENode B)Figure 1. The "Separate Cloud" Approach
The setup in Figure 1 keeps the TRILL cloud(s) and the FCoE cloud(s) separate. The TRILL cloud routes FCoE traffic as standard Ethernet traffic, and it looks to the ENodes and the FCF like a plain Ethernet LAN 🌸. FCoE traffic routed over the TRILL cloud includes FCoE data frames, plus FCoE control traffic, including FCoE Initialization Protocol (FIP) frames. To avoid frame loss from queue overflow, the switches in any TRILL Cloud used with FCoE would likely implement and use the relevant DCB protocols [TRILLPFC] [TRILLCN] 🛡️.
The main downsides of the Separate Cloud approach are that RBridges and FCFs are separate boxes in the network, which means more cabling and more gear 📦. It also means that communication between ENodes usually has to cross the TRILL cloud twice, so there are twice as many hops 😭. As mentioned above, data center networking is converging on a consolidated, cost-effective approach. There, the same infrastructure and equipment handle both data and storage traffic, and high efficiency and a minimal number of hops are important when designing the network topology 🏆.
The Separate Cloud approach is presented as background, to explain why we'd want a different approach with a higher level of integration 🫶.
3.2. FCoE over an RBridge 🌈
More of this lives in part 2, bestie 💜.
3.2.1. FCRB ✨
Okay so, instead of the Separate Cloud vibe from Section 3.1, there's a glow-up option 💅 where every switch packs both an FCF entity and an RBridge entity into one consolidated box. This combo is called FCoE-forwarder-over-RBridge (FCRB) 🎀
Figure 2 shows an FCRB and its main building blocks. Functionally, an FCRB can be viewed as two independent entities:
- An FCoE Forwarder (FCF) entity.
- An RBridge entity.
The FCF entity is plugged into one of the ports of the RBridge, and to the RBridge it looks like a native Ethernet host 🧚 The layers' interaction is explained in detail in Section 3.2.3.
Note: In this document, "FCF" is used a little differently than in [FC-BB-5], because we want to show that an FCRB is basically like an RBridge cascaded to an FCF 🦋 In [FC-BB-5] terms, an FCRB would be called an FCF, and the FCF building block in Figure 2 would be called an FC switching element.
+-------------------+
|FCRB |
| +-----------+ | Native FC
| | FCF |------ Interface
| +-----+-----+ |
| | |
| +-----+-----+ |
| | RBridge | |
| +-+-+---+-+-+ |
| | | | | |
+-----|-|---|-|-----+
FCoE/ / | | |
+---+ Ethernet / / | | FCoE / Ethernet
| |___________________/ / | | over TRILL ___ _
| | / | | / \_/ \__
+---+ / | \_____________ _/ \
FCoE Storage / \_______________/ TRILL /
Array / \_ Cloud /
(ENode A) / / \
/ \__/\_ ___/
+---+ / \_/
| |______________/
| |
+---+
Server
(ENode B)
Figure 2. FCRB Entity in the NetworkThe FCRB entity keeps the TRILL and FCoE protocols in their own lanes (layer independence 💜) while letting both run on the same network. Iconic.
Here's the thing though: FCoE traffic always goes through an FCF and can never be forwarded directly between two ENodes 🙈 So FCoE traffic between ENodes A and B in the Figure 1 topology goes through this path:
ENode A-->TRILL cloud-->FCF-->TRILL cloud-->ENode BIn Figure 2, it's different. The FCF is right next to ENodes A and B. The FCRB is connected to both ENodes, acts as the edge RBridge that hooks them up to the TRILL cloud, and also acts as the FCF that forwards traffic between them 👑 So traffic between ENodes A and B in the Figure 2 topology takes this shorter path:
ENode A-->FCRB-->ENode BSo using FCRB entities lets TRILL and FCoE share common infrastructure and equipment, instead of needing separate infrastructure like in the Separate Cloud topology from Figure 1. Main character energy 🌈
3.2.2. Topology 🍓
Figure 3 shows a typical data center network topology. Servers are arranged hierarchically behind Top-of-Rack (ToR) switches, also known as access switches, and each set of racks gets aggregated by an End-of-Row (EoR) switch. The EoR switches roll up to the core switches, which may connect to other clouds, like an external WAN or a native FC SAN.
_ __ _ __
/ \_/ \_ / \_/ \_
\_ \ \_ \ ....
/ SAN _/ / WAN _/
\__ _/ \__ _/
\_/ \_/
| |
| |
+------+ +------+
Core | | | |
FCoE over | | | |
RBridge | | | |
(FCRB) +------+ +------+
| \___ ___/ |
| \ / |
| \/ |
EoR +----+_______/\_______+----+
FCoE over | | | |
RBridge | | | |
(FCRB) +----+ +----+
/ \ / \
/ \ / \
ToR +---+ +---+ +---+ +---+
FCoE over | | | | | | | |
RBridge | | | | | | | |
(FCRB) +---+ +---+ +---+ +---+
/ \ / \ / \ / \
/ \ / \ / \ / \
+-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+
Servers/ | | | | | | | | | | | | | | | |
ENodes +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+
A B C D E F G H
Figure 3. FCoE over RBridge TopologyQuick note: in the Figure 3 example, all the ToR, EoR, and core switches are FCRB entities. It's also possible for some network nodes to be pure RBridges, which gives a topology where FCRBs are interconnected through TRILL clouds ✨
3.2.3. The FCRB Flow 🦋
3.2.3.1. Example - ENode to ENode 🫶
FCoE traffic between ENodes A and B in Figure 3 goes through the ToR FCRB, because A and B are both connected to the same ToR. Traffic between ENodes A and C has to be forwarded through the EoR FCRB 💖
The FCoE jargon splits deployment into two modes:
- Sparse mode: an FCoE packet sent between two FCFs may be forwarded over several hops of a Layer 2 network, which lets the underlying Layer 2 network pick the path between the two FCFs.
- Dense mode: each node along the path between two FCFs is also an FCF, and the network is set up so the forwarding decision at each hop is made at the FCF layer, which lets the path between the two FCFs be based on the FSPF routing protocol.
Figure 4 shows traffic between ENodes A and C, which aren't on the same ToR. The next two subsections walk through the forwarding procedure in Dense mode and Sparse mode, respectively 🌸
+--------+ +--------+ +--------+ +--------+ +--------+
| FCoE |.....| FCF |.....| FCF |.....| FCF |.....| FCoE |
| ENode | +--------+ +--------+ +--------+ | ENode |
| | |RBridge |.....|RBridge |.....|RBridge | | |
+--------+ +--------+ +--------+ +--------+ +--------+
|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|
+--------+ +--------+ +--------+ +--------+ +--------+
Server ToR 1 EoR ToR 2 FCoE Storage
ENode A FCRB FCRB FCRB Array
ENode C
Figure 4. Traffic between two ENodes - Example3.2.3.1.1. Forwarding from A to C in Dense Mode 💅
- FCoE traffic from A goes to ToR 1 over the Ethernet interface. The destination MAC address is the address of the FCF entity at ToR 1.
- At ToR 1:
- The packet gets forwarded to the FCF entity at the ToR. So forwarding between ENode A and the FCF at the ToR works just like forwarding between two Ethernet hosts 🎀
- The FCF entity at the ToR makes a forwarding decision based on the FC addresses. That decision is based on the FSPF routing protocol at the FCF layer. The FCF entity at the ToR forwards the packet to the FCF entity in the EoR.
- The FCF then updates the destination MAC address of the packet to the address of the EoR FCF.
- The packet goes to the RBridge entity, where it gets encapsulated in a TRILL header and sent to the RBridge at the EoR over a single hop of the TRILL network.
- The RBridge entity in the EoR FCRB, acting as the egress RBridge, decapsulates the TRILL header and forwards the FCoE packet to the FCF entity. From here, the forwarding process looks like the one described above for the ToR 🧚
- A similar forwarding process happens at the next-hop ToR FCRB, where the FCRB finally forwards the FCoE packet to the target, ENode C.
3.2.3.1.2. Forwarding from A to C in Sparse Mode 🦋
- Traffic goes to ToR 1, as described in Section 3.2.3.1.1.
- The FCF in ToR 1, based on an FSPF forwarding decision, forwards the packet to the FCF in ToR 2. The destination MAC address of the FCoE packet gets updated to reflect the FCF in ToR 2. The RBridge entity in ToR 2 adds a TRILL encapsulation, with an egress RBridge nickname representing ToR 2.
- The packet reaches the EoR. The RBridge entity in the EoR routes the packet to the RBridge entity in ToR 2.
- The packet reaches ToR 2. From this point on, the process is identical to the one described in Section 3.2.3.1.1 💖
3.2.3.2. Example - ENode to Native FC Node 🌈
+--------+ +--------+ +--------+ +---------+ +--------+
| FCoE |.....| FCF |.....| FCF |.....| FCF |.....| FC |
| ENode | +--------+ +--------+ +----+----+ |protocol|
| | |RBridge |.....|RBridge |.....| RB | | | stack |
+--------+ +--------+ +--------+ +----+ FC | | |
|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Eth | |<===>| |
+--------+ +--------+ +--------+ +----+----+ +--------+
Server ToR EoR Core Native FC
ENode FCRB FCRB FCRB Storage Array
Figure 5. Example of Traffic between an
ENode and a Native FC Storage ArrayFigure 5 shows a second example, where traffic goes between an ENode and an FC Storage Array, based on the Figure 3 topology 🍓
- FCoE traffic from the ENode goes to the ToR over the Ethernet interface. The forwarding through the ToR FCRB and through the EoR is similar to the matching steps in Section 3.2.3.1.
- When the packet reaches the core FCRB, the egress RBridge entity decapsulates the TRILL header and forwards the FCoE packet to the FCF entity. The packet is then forwarded as a native FC packet through the FC interface to the native FC node ✨
3.2.3.3. Example - ENode to ENode with Non-FCRB EoR 🦋
okay so this one is basically the Figure 4 example with a plot twist: the EoR is an RBridge instead of an FCRB 👀 the vibes are still immaculate 💅
+--------+ +--------+ +--------+ +--------+
| FCoE |.....| FCF |....................| FCF |.....| FCoE |
| ENode | +--------+ +--------+ +--------+ | ENode |
| | |RBridge |.....|RBridge |.....|RBridge | | |
+--------+ +--------+ +--------+ +--------+ +--------+
|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|
+--------+ +--------+ +--------+ +--------+ +--------+
Server ToR 1 EoR ToR 2 FCoE Storage
ENode A FCRB FCRB FCRB Array
ENode C Figure 6. Example of Traffic between Two ENodesso an FCoE packet going from ENode A to C moves like this 🧚
- The packet is sent to the FCF in ToR 1, as in the previous example.
- The FCF in ToR 1 takes a forwarding decision based on the FC addresses and forwards the packet to the next-hop FCF, which resides in ToR 2. This forwarding decision is taken at the FCF layer and is based on the FSPF routing protocol.
- The packet is then forwarded to the RBridge entity in ToR 1, where it is encapsulated in a TRILL encapsulation, and forwarded to the RBridge at ToR 2. The packet is routed over the TRILL cloud through the RBridge at the EoR. The path through the TRILL cloud is determined by TRILL's IS-IS routing protocol.
- Once the packet reaches ToR 2, it is forwarded in a similar manner to the description in Section 3.2.3.1.
this example shows that a hybrid network is totally possible, where some nodes are FCRBs and some are RBridges 🌈 the forwarding procedure here is kinda giving sparse-mode forwarding from Section 3.2.3.1.2, so if you read that one, you're already halfway there 🫶
3.2.3.4. Example - FCoE Control Traffic through an FCRB 🎀
the last few subsections were all about the data plane, aka the storage data exchanges that ride along in an FCoE encapsulation 📦 but FCoE also needs control and management traffic for starting sessions (that's FIP), sharing routing info (that's FSPF), and running and managing the whole fabric 🛠️
the FCoE Initialization Protocol (FIP) uses Ethernet frames with a dedicated Ethertype, so the FCF can tell these frames apart from everything else 🕵️♀️ FIP uses both unicast and multicast traffic. the following example shows how a multicast FIP packet travels through the network in Figure 4:
- ENode A generates a multicast frame to a multicast MAC address that represents all the FCFs (All-FCF-MAC).
- The packet is forwarded to the ToR FCRB node. The RBridge entity forwards a copy of the packet to its FCF entity, and also sends the packet through the TRILL cloud as a multicast TRILL encapsulated packet.
- Each of the FCRBs then receives the packet, forwards a copy to its FCF entity, and forwards the packet through the TRILL network, allowing all the FCFs to receive the packet.
while FIP packets get their own Ethertype and frame format, other types of FCoE control and management frames use the same FCoE encapsulation as FCoE data traffic 📬 so the forwarding for that control traffic looks a lot like the examples in the earlier subsections, with one difference: these frames can be sent between ENodes, between FCFs, or between ENodes and FCFs. periodt 💜
4. Security Considerations 🔐
for general TRILL security considerations, see [TRILL] 🛡️
for general FCoE security considerations, see Annex D of [FC-BB-5] 🔒
⚠️ there are no additional security implications imposed by this document. that's the whole tea, bestie 💀
5. Acknowledgments 🫶
the authors gratefully acknowledge Ayandeh Siamack and David Black for their helpful comments ✨ they also thank the T11 committee for reviewing the document, and in particular Pat Thaler and Joe White for their useful input 👑
6. References 📚
6.1. Normative References
- [TRILL] Perlman, R., Eastlake 3rd, D., Dutt, D., Gai, S., and A. Ghanwani, "Routing Bridges (RBridges): Base Protocol Specification", RFC 6325, July 2011.
- [FC-BB-5] ANSI INCITS 462: "Information Technology - Fibre Channel - Backbone - 5 (FC-BB-5)", May 2010.
6.2. Informative References
- [802.1Q] "IEEE Standard for Local and metropolitan area networks - Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks", IEEE Std 802.1Q(tm), 2012 Edition, October 2012.
- [TRILLPFC] Eastlake 3rd, D., Wadekar, M., Ghanwani, A., Agarwal, P., and T. Mizrahi, "TRILL: Support of IEEE 802.1 Priority-based Flow Control and Enhanced Transmission Selection", Work in Progress, January 2013.
- [TRILLCN] Eastlake 3rd, D., Wadekar, M., Ghanwani, A., Agarwal, P., and T. Mizrahi, "TRILL: Support of IEEE 802.1 Congestion Notification", Work in Progress, January 2013.
Authors' Addresses 🌸
David Melman
Marvell
6 Hamada St.
Yokneam, 20692 Israel
EMail: davidme@marvell.com
Tal Mizrahi
Marvell
6 Hamada St.
Yokneam, 20692 Israel
EMail: talmi@marvell.com
Donald Eastlake 3rd
Huawei USA R&D
155 Beaver Street
Milford, MA 01757 USA
Phone: +1-508-333-2270
EMail: d3e3e3@gmail.comthe end, bestie! 🌈✨ that's the whole RFC, and you survived it 💖
📦 where this page comes from
- ietf: rfc6847 2013-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms