RFC 6307

Encapsulation Methods for Transport of Fibre Channel Traffic over MPLS Networks

📅 published April 2012, from ietf

RFC 6307: Encapsulation Methods for Transport of Fibre Channel Traffic over MPLS Networks ✨

Part 1 of 4 🎀

Okay bestie, here's the title block tea 🫖. This is RFC 6307, a Standards Track document, published in April 2012. Its ISSN is 2070-1721. The authors are D. Black (editor, EMC Corporation), L. Dunbar (editor, Huawei Technologies), M. Roth (Infinera), and R. Solomon (Orckit-Corrigent). The title block doesn't list anything it obsoletes or updates, so it doesn't replace or change any earlier RFC 🦋.

Abstract

okay so a Fibre Channel pseudowire (PW) is how you carry Fibre Channel traffic across an MPLS network 🌸. This lets service providers use MPLS to offer "emulated" Fibre Channel services, which is giving main character energy 💅. This doc specifies how Fibre Channel traffic gets encapsulated inside a pseudowire, and it also lays out the common procedures for using a PW to provide a Fibre Channel service 🧚.

Status of This Memo

This is an Internet Standards Track document 💜.

It's a product of the Internet Engineering Task Force (IETF), so it represents the consensus of the IETF community. It went through public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check Section 2 of RFC 5741 🔥.

For the current status of this document, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6307 🌈.

Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved 🫶.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document 📜. Code Components pulled out of this document must include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License ⭐.

Table of Contents

   1. Introduction ....................................................3
      1.1. Transparency ...............................................3
      1.2. Bandwidth Efficiency .......................................4
      1.3. Reliability ................................................5
      1.4. Conventions Used in This Document ..........................5
   2. Reference Model .................................................6
   3. Encapsulation ...................................................8
      3.1. The Control Word ..........................................10
      3.2. MTU Requirements ..........................................11
      3.3. Mapping of FC Traffic to PW Packets........................11
           3.3.1. FC Data Frames (PT=0) and FC Login Frames (PT=1) ...11
           3.3.2. FC Primitive Sequences and Primitive
                  Signals (PT=2) .....................................12
           3.3.3. FC PW Control Frames (PT=6) ........................14
      3.4. PW Failure Mapping ........................................15
   4. Signaling of FC Pseudowires ....................................15
   5. Timing Considerations ..........................................15
   6. Security Considerations ........................................17
   7. Applicability Statement ........................................17
   8. IANA Considerations ............................................18
   9. Acknowledgments ................................................19
   10. Normative References ..........................................19
   11. Informative References ........................................20

1. Introduction ✨

okay so Fibre Channel (FC) is a super high-speed communications tech, mostly used for Storage Area Networks (SANs) 💾. Within one site (like a data center), an FC-based SAN hooks servers up to storage systems, and FC can also be stretched across sites 🌸. When FC goes across multiple sites, the most common move is storage replication for disaster recovery, like when a flood or fire takes a site offline 🔥. This is especially true over long distances, where network latency makes things painfully slow for a server whose storage lives somewhere else 🙈. Fibre Channel is standardized by the INternational Committee for Information Technology Standards (INCITS) Technical Committee T11 [T11], and people have built several ways to encapsulate and transport FC traffic over other networks [FC-BB-6] 🦋.

Fibre Channel Over TCP/IP (FCIP), described in [RFC3821] and [FC-BB-6], connects isolated FC SANs over IP networks 🌈. FCIP uses FC Frame Encapsulation [RFC3643] to wrap FC frames up for tunneling across an IP-based network 🎀. IP networks can drop or reorder packets, so FCIP leans on TCP to retransmit dropped frames and put reordered frames back in order 💅. Because of possible delay variation and TCP timeouts, FCIP needs special timing mechanisms to keep FC working correctly [FC-BB-6] ⏱️.

MPLS networks can run with very low loss rates and a very low chance of reordering, which makes it possible to directly interconnect Fibre Channel ports over MPLS 💜. A Fibre Channel pseudowire (FC PW) is a way to transparently carry FC traffic over an MPLS network, so the result acts like a pair of FC ports directly connected by a physical FC link 👑. That makes the control processing simpler compared to FCIP 🧚.

This document specifies how FC traffic gets encapsulated into an MPLS pseudowire, plus the related PW procedures needed to move FC traffic over MPLS PWs 🫶. The complete FC pseudowire spec is this document plus the FC PW portion of the T11 [FC-BB-5/AM1] standard 🍓. The subsections below cover some of the requirements for carrying FC traffic over an MPLS network ✨.

1.1. Transparency

Transparent extension of an FC link is a key requirement for carrying FC traffic over a PW 🔥. The FC PW has to emulate an FC link between two FC ports, similar to the approach defined for FC over GFPT in [FC-BB-6] 💖. GFPT is an Asynchronous Transparent Generic Framing Procedure specified by ITU-T, so check [FC-BB-6] for details and the ITU-T specs it references 🌸. The result is transparent forwarding of FC traffic across the MPLS network, from both the FC fabric's point of view and the network operator's point of view 🦋.

Transparency is what sets the FC PW approach apart from FCIP 💅. An FC PW logically connects the FC port on the FC link at one end of the PW directly with the FC port on the far end of the FC link at the other end 🌈. FCIP, on the other hand, adds FC B_Ports at both ends of the extended FC link, and each FC B_Port connects to an FC E_Port in an FC switch on the same side of the link extension 🎀.

1.2. Bandwidth Efficiency

The bandwidth allocated to a PW may be less than the rate of the attached FC port 🧚. When there's no data exchange on a native FC link, Idle Primitive Signals are constantly swapped between the two FC ports 💤. To improve bandwidth efficiency across the MPLS network, the FC PW Provider Edge (PE) has to suppress (or drop) the Idle Primitive Signals its adjacent FC ports generate 🙈. The far-end FC PW PE regenerates Idle Primitive Signals to send to its adjacent FC port as required, see [FC-BB-5/AM1] 🫶.

FC link control protocols make an FC port keep sending the same FC Primitive Sequence [FC-FS-2] until it gets a reply or some other event happens 🔁. To save bandwidth, the FC PW PE encapsulates only a subset of those repeated FC Primitive Sequences to send across the WAN [FC-BB-5/AM1] 🌸. For example, in a run of identical received primitives, only every fourth primitive may be sent across the MPLS network 🍓. Or, a time-based approach can be used, sending a copy of the repeated FC Primitive Sequence once every few milliseconds ⏱️. The far-end FC PW PE recreates the FC link behavior by continuously sending the Primitive Sequence it most recently received from the WAN, until a new primitive signal, primitive sequence, or data frame shows up from the WAN 💜.

The sending FC PW PE can unilaterally pick any convenient subset for sending the same FC Primitive Sequence 🦋. That's fine because the receiving FC PW PE generates a continuous stream of the most recently received FC Primitive Sequence on the outgoing native FC link, no matter how fast that sequence arrives from the WAN 💅. In practice, a 10:1 reduction in FC Primitive Sequence transmission rate gets you 90% of the bandwidth benefits without losing any FC functionality, and sending a copy every few milliseconds doesn't pose a serious risk of exceeding the timeouts specified in Section 5 below 🔥.

These bandwidth-efficiency techniques may change the FC traffic that crosses an FC PW (like the number of Idle Primitive Signals or the number of identical Primitive Sequences) 🌈. But the far-end FC PW PE's regeneration of FC link behavior on the attached FC port stays transparent to the FC ports connected to each PW PE 🎀.

1.3. Reliability

Fibre Channel doesn't have a native frame retransmission protocol, and it treats most frame delivery failures as errors 😬. FC SAN traffic needs a very low frame loss rate, because a failure to deliver a frame usually means an I/O operation fails 💀. Recovering from those I/O failures means retrying the I/O operation after what can be a significant delay (30-second and 60-second timeouts are common) ⏳. On top of that, those retries tend to get logged as errors, which hints at possible problems with FC equipment or cables 🙃. So drops, errors, and discards of FC frames must be very rare for an FC PW 🥺.

FC SAN implementations have limited tolerance for frame reordering 🙅‍♀️. Any reordering that affects more than a few frames within a single higher-level operation (like a read or write I/O) is usually treated as an error by the destination FC port, which results in discards of the frames involved 💥. Some deployed FC implementations treat all within-operation frame reordering as errors that cause frame discards 😭. As a result, FC frame reordering must be minimized for an FC PW 💖.

The FC PW doesn't compensate for frame drops, discards, or reordering 🙅‍♀️. The MPLS network hosting the FC PW is expected to be designed and operated in a way that makes those events very rare 🫶.

Here's the contrast: instead of the Time to Live (TTL) field in an IP packet, FC uses a constant delivery timeout value (R_A_TOV), and the default for that is 10 seconds ⏱️. Each FC frame must be delivered or discarded within that timeout period after it's sent, see Section 5 ✨.

1.4. Conventions Used in This Document

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 📚🎀

2. Reference Model 🧚

okay bestie so an FC PW basically extends a native FC link over an MPLS network 🦋 this doc only specs the PW encapsulation for FC. Figure 1 shows the reference models (derived from [RFC3985]) that support the FC PW. FC traffic hits PE1's FC attachment channel, gets encapsulated at PE1, travels across the MPLS network, gets decapsulated at PE2, and then goes out PE2's FC attachment channel 💅 this doc assumes a pseudowire can be provisioned statically or through a signaling protocol as defined in [RFC4447] ✨

        |<-------------- Emulated Service ----------------->|
        |                                                   |
        |          |<------- Pseudowire -------->|          |
        |          |                             |          |
        |          |    |<-- MPLS Tunnel -->|    |          |
        |          V    V                   V    V          |
        V   AC     +----+                   +----+    AC    V
  +-----+    |     | PE1|===================| PE2|     |    +-----+
  |     |----------|............PW1..............|----------|     |
  | CE1 |    |     |    |                   |    |     |    | CE2 |
  |     |----------|............PW2..............|----------|     |
  +-----+  ^ |     |    |===================|    |     | ^  +-----+
        ^  |       +----+                   +----+     | |  ^
        |  |   Provider Edge 1          Provider Edge 2  |  |
        |  |                                             |  |
  Customer |                                             | Customer
  Edge 1   |                                             | Edge 2
           |                                             |
           |                                             |
    Native FC service                             Native FC service

        Figure 1 - PWE3 FC Interface Reference Configuration

figure 2 shows where each end of the PW terminates inside the PE 🎀

            +-----------------------------------+
            |                PE                 |
    +---+   +-+  +-----+  +------+  +------+  +-+
    |   |   |P|  |     |  |PW ter|  | MPLS |  |P|
    |   |<==|h|<=| NSP |<=|minati|<=|Tunnel|<=|h|<== From network
    |   |   |y|  |     |  |on    |  |      |  |y|
    | C |   +-+  +-----+  +------+  +------+  +-+
    | E |   |                                   |
    |   |   +-+  +-----+  +------+  +------+  +-+
    |   |   |P|  |     |  |PW ter|  | MPLS |  |P|
    |   |==>|h|=>| NSP |=>|minati|=>|Tunnel|=>|h|==> To network
    |   |   |y|  |     |  |on    |  |      |  |y|
    +---+   +-+  +-----+  +------+  +------+  +-+
            |                                   |
            +-----------------------------------+

            Figure 2 - PW Reference Diagram

the Native Service Processing (NSP) function has three jobs, and they're all giving main character energy 👑

  • Idle Suppression: any FC Idle Primitive Signals received from the source PE's attached FC port get suppressed and then regenerated at the destination PE, to be sent out its attached FC port whenever there's no other FC traffic to send 🫶
  • FC Primitive Sequence Reduction: a subset of repetitive FC Primitive Sequences received from the attached FC port at the source PE gets picked for WAN transmission. the destination PE keeps sending the FC Primitive Sequence it most recently got from the WAN out its attached FC port, continuously, until a new packet arrives from the WAN 🌸
  • Flow Control: the Alternate Simple Flow Control (ASFC) protocol handles buffer management together with the peer PW PE's NSP function, so FC traffic doesn't get dropped 🔥 ASFC is a simple pause/resume protocol that allows operation repetition. the receiver responds to the first pause or resume operation in an identical sequence of operations and ignores the rest of the sequence 🙈

the NSP flow control functionality is required to extend FC's credit-based flow control for situations where the number of buffer credits available to an FC link isn't enough to use the available bandwidth over the extra distance and latency the FC pseudowire adds 😭 the NSPs fix this by inserting ASFC into the FC link flow control used on the attached FC ports. see [FC-BB-5/AM1] 💜

idle suppression and FC Primitive Sequence Reduction are different. they're bandwidth optimizations that are included in the NSP just for clarity in this document 🍓 other pseudowires don't treat similar optimizations as part of the NSP (e.g., Asynchronous Transfer Mode (ATM) idle frame suppression is not considered to be an NSP function by [RFC4717]) 🦋

the NSP function is specified in detail by [FC-BB-5/AM1] ✨

3. Encapsulation 💖

this spec gives you port-to-port transport of FC-encapsulated traffic. Fibre Channel defines a bunch of port types, including:

  • N_port: a port on the node (e.g., host or storage device) used with both FC-P2P (Point to Point) or FC-SW (switched fabric) topologies. also known as a Node port 🎀
  • NL_port: a port on the node used with an FC-AL (Arbitrated Loop) topology. also known as a Node Loop port 🧚
  • F_port: a port on the switch that connects to a node point-to-point (i.e., connects to an N_port). also known as a Fabric port. an F_port is not loop capable 💅
  • FL_port: a port on the switch that connects to an FC-AL loop (i.e., to NL_ports). also known as a Fabric Loop port 🌈
  • E_port: a port used to connect two Fibre Channel switches. also known as an Expansion port. when E_ports between two switches are connected to form a link, that link is called an inter-switch link (ISL) 🫶

out of all those port types, only these FC connections (as specified in [FC-BB-5/AM1]) are supported by an FC PW over MPLS:

  • N_Port to N_Port, established by an FC PLOGI (Port Login) operation 🦋
  • N_Port to F_Port, established by an FC FLOGI (Fabric Login) operation 🌸
  • E_Port to E_Port, established by an FC ELP (Exchange Link Parameters) operation 👑

FC traffic flowing over an FC PW gets sorted into four payload types (PTs), which are encoded in the PW Control Word (see Section 3.1) 🍓

  1. FC login traffic (PT = 1): FC login operations and responses that establish connections between FC ports. the three FC login operations are PLOGI, FLOGI, and ELP. these operations and their responses may require the NSP to allocate buffer resources. see the specification of Login Exchange Monitors in [FC-BB-5/AM1] 🔥
  2. FC data traffic (PT = 0): all FC frames other than those involved in an FC login operation 💜
  3. FC Primitive Sequences and Signals (PT = 2): native FC link control operations. these are 4-character primitive sequences and signals that are not encapsulated in FC frames. see [FC-BB-5/AM1] and [FC-FS-2] 🙈
  4. FC PW Control (PT = 6): FC PW control operations exchanged only between the endpoints of the PW. they're used for ASFC flow control, ping (e.g., for round-trip latency measurement), and reporting native FC link errors. see [FC-BB-5/AM1] 😭

this FC PW spec is limited to use with FC service classes 2, 3, and F. see [FC-FS-2] ✨ other FC service classes (e.g., 1, 4, and 6) MUST NOT be used with an FC PW 🚫 numbered FC service classes handle end-to-end FC traffic, while service class F is used for inter-switch traffic in an FC switched fabric 💅

this FC PW spec is also limited to native FC attachment links that use an 8b/10b transmission code (see [FC-FS-2]). the protocol here converts a received 10b code to its 8b counterpart for PW encapsulation, so it does not support attached FC links that use a 64b/66b transmission code (e.g., 10GFC and 16GFC). such links MUST NOT be attached to an FC PW PE unless their link speed can be negotiated to one that uses 8b/10b encoding 🌈 if an invalid 10b code that can't be converted to an 8b code is received from an FC link, the PE sends an FC PW control frame to report the error (see [FC-BB-5/AM1]) 🫶

3.1. The Control Word 🎀

the Generic PW Control Word, as defined in "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for Use over an MPLS PSN" [RFC4385], MUST be used for FC PW. it helps transport short packets (by setting the Length field as detailed below) and carries the flag bits defined below 💖 the structure of the Control Word for the FC PW looks like this:

                       1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |0 0 0 0| PT  |X|0 0|  Length   |     Sequence Number           |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                 Figure 3 - Control Word Structure

the first four bits of the PW Control Word MUST be set to 0 by the ingress PE to indicate PW data 🧚

three of the four flag bits carry the Payload Type (PT) indication. the 3-bit binary value in this field identifies the payload type carried by a PW packet. the defined types are:

        PT = 0: FC data frame.

        PT = 1: FC login frame.

        PT = 2: FC Primitive Sequence(s) and/or Primitive Signal(s).

        PT = 6: FC PW control frame (refer to [FC-BB-5/AM1] for
        usage).

packets with other values in the PT field are not valid for the FC PW and MUST be discarded by the receiving FC PW PE 🙅‍♀️

the X flag bit isn't used by this version of the protocol. it SHOULD be set to zero by the sender and MUST be ignored by the receiver 🌸

the fragmentation bits (bits 8-9) aren't used by the FC PW protocol. they may be used in the future for FC-specific indications as defined in [RFC4385]. the fragmentation bits SHOULD be set to zero by the ingress PE and MUST be ignored by the egress PE 🦋

the Length field lets you recover the original pseudowire packet when a short packet gets padded to the minimum 64-octet packet size required for Ethernet (see [RFC4385]) 🍓 the Length field MUST be used for packets shorter than 64 octets, MUST be set to zero for longer packets, and MUST be processed according to the rules specified in [RFC4385] 🫶

the sequence number isn't used for the FC PW. it MUST be set to 0 by the ingress PE and MUST be ignored by the egress PE 💅

3.2. MTU Requirements 🌈

the MPLS network MUST be able to transport the largest Fibre Channel frame after encapsulation, including the overhead that comes with the encapsulation 🚀 the maximum FC frame size is 2164 octets without PW and MPLS labels (refer to Figure 4). this maximum size is a constant value that is required for all FC implementations [FC-FS-2] 💖 the MPLS network SHOULD accommodate frames of up to 2500 octets so it can support possible future increases in the maximum FC frame size ✨

fragmentation, as described in [RFC4623], SHALL NOT be used for an FC PW 🙅‍♀️ so the network MUST be configured with a minimum MTU that's big enough to transport the largest encapsulated FC frame 🎀

3.3. Mapping of FC Traffic to PW Packets 🎀

okay so FC frames, Primitive Sequences, and Primitive Signals all get transported over the PW ✨ Every packet type rides on a single PW, no cap. On top of the PW Control Word, the PW packet also includes an FC Encapsulation Header. Bestie, this header is NOT used in this version of the protocol 💅 So the sender SHOULD set it to zero, and the receiver MUST ignore it 🫶

3.3.1. FC Data Frames (PT=0) and FC Login Frames (PT=1) 🌸

FC data frames and FC login frames share the same encapsulation format, honestly so iconic of them 💖 The only difference is the PT field in the FC PW Control Word: it's 0 for data frames and 1 for login frames. An FC login frame carries an FC PLOGI, FLOGI, or ELP operation or response that needs special processing by the NSP to support flow control. If you want the details, check [FC-BB-5/AM1] 🧚

Every FC data frame or login frame gets mapped to a PW packet. That includes the Start Of Frame (SOF) delimiter, the frame header, the Cyclic Redundancy Check (CRC) field, and the End Of Frame (EOF) delimiter, all as shown in Figure 4 🦋

                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +---------------------------------------------------------------+
   |                    FC PW Control Word                         |
   +---------------------------------------------------------------+
   |                  FC Encapsulation Header                      |
   +---------------+-----------------------------------------------+
   |     SOF Code  |             Reserved                          |
   +---------------+-----------------------------------------------+
   |                                                               |
   +-----                        FC Frame                      ----+
   |                                                               |
   +---------------------------------------------------------------+
   |                              CRC                              |
   +---------------+-----------------------------------------------+
   |     EOF Code  |                Reserved                       |
   +---------------+-----------------------------------------------+

   Figure 4 - FC Frame (SOF/Data/CRC/EOF) Encapsulation in PW Packet

The SOF and EOF delimiters each get encoded into a single octet as specified in [RFC3643] 🍓 Plot twist, though: the codes for delimiters that only apply to FC service class 4 (SOFi4, SOFc4, SOFn4, EOFdt, EOFdti, EOFrt, and EOFrti, see [FC-FS-2]) MUST NOT be used. Not even a little 🙈

The CRC in the frame comes straight from the FC attachment channel, so the PW PE doesn't have to recalculate the CRC or check the CRC in the received frame. Instead, the FC port that receives the frame does the checking, which covers data errors that happen between the PW endpoints ✨ This CRC behavior is different from the Frame Check Sequence (FCS) retention technique for PWs in [RFC4720], which says "as usual, the FCS MUST be examined at the ingress PE, and errored frames MUST be discarded". So it's not the same vibe as that one, and I'm obsessed with how clearly they're separated 💜

3.3.2. FC Primitive Sequences and Primitive Signals (PT=2) 💅

FC Primitive Sequences and Primitive Signals are FC Ordered Sets. On an 8b/10b-coded FC link, an Ordered Set is four 10b characters: it starts with the K28.5 character, then three Dxx.y data characters. Every FC Ordered Set starts with a K28.5 control character, but the three Dxx.y data characters that follow depend on which Ordered Set it is 🌈 A Kxx.y control character has a different 10b code from its matching Dxx.y data character, but they use the same 8b code. For example, K28.5 and D28.5 both use the 8b code 0xBC. Here are two examples:

  • Idle (IDLE) is K28.5 - D21.4 - D21.5 - D21.5. This FC Primitive Signal gets sent when the FC link is idle. The FC PW NSP suppresses it, so it doesn't get sent over the WAN 😭
  • Link Reset Response (LRR) is K28.5 - D21.1 - D31.5 - D9.2. This FC Primitive Sequence is part of FC link initialization and recovery ⭐

Each Ordered Set is encapsulated in a PW packet holding the encoded K28.5 control character [FC-BB-5/AM1], followed by three encoded data characters, as shown in Figure 5 🎀

                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +---------------------------------------------------------------+
   |                    FC PW Control Word                         |
   +---------------------------------------------------------------+
   |                  FC Encapsulation Header                      |
   +---------------+---------------+---------------+---------------+
   |     K28.5     |     Dxx.y     |     Dxx.y     |     Dxx.y     |
   +---------------+---------------+---------------+---------------+
   |                                                               |
   +----                                                       ----+
   |                                                               |
   +---------------+---------------+---------------+---------------+
   |     K28.5     |     Dxx.y     |     Dxx.y     |     Dxx.y     |
   +---------------+---------------+---------------+---------------+

          Figure 5 - FC Ordered Sets Encapsulation in PW Packet

Okay, here's the tricky part 🙈 The K28.5 10b control character from the PE's attached FC link gets encoded for the FC PW as its 8b counterpart, 0xBC. But 0xBC is also how a D28.5 data word gets encoded, so the receiving FC PW PE has to be careful:

  • The receiving PE MUST check for an 8b K28.5 value (0xBC) at the start of each Ordered Set (see Figure 5), and it MUST send that value as a 10b K28.5 character on the attached FC link.
  • The receiving PE MUST send the three Dxx.y 8b values that follow as Dxx.y 10b characters on the attached FC link, and it MUST NOT send any of those Dxx.y 8b values as 10b Kxx.y characters on the attached FC link. Main character energy 👑

A PW packet may hold one or more encoded FC Ordered Sets [FC-BB-5/AM1]. When the packet has multiple Ordered Sets, the Length field in the FC PW Control Word tells you the packet length. Because of that, FC PW packets that contain FC Ordered Sets MUST NOT be larger than 60 octets (8 octets of header words plus at most 13 Ordered Sets). That keeps the Length field at a non-zero value (see [RFC4385]) 🧁

Idle Primitive Signals could go over the PW the same way Primitive Sequences do. But [FC-BB-5/AM1] says Idle Primitive Signals have to be dropped by the Ingress PE and regenerated by the egress PE, so bandwidth doesn't get wasted (see [FC-BB-5/AM1] for the full details) 🦋

The egress PE pulls the Primitive Sequence or Primitive Signal out of the received PW packet. For a Primitive Sequence, the PE keeps transmitting that same FC Ordered Set to its attached FC port until it gets an FC frame or a different Ordered Set over the PW. For ingress PE transmission behavior of Primitive Sequences, see Section 1.2 above 💜 A Primitive Signal is sent once, except Idle Primitive Signals, which get sent continuously when there's nothing else to send 🌸

3.3.3. FC PW Control Frames (PT=6) 🧚

FC PW control frames are transported over the PW by encapsulating each frame in a PW packet with PT=6 in the Control Word. The corresponding FC entity generates and terminates the FC PW control frame payloads. These frames handle FC PW flow control (ASFC), ping, and transmission of error indications ✨ [FC-BB-5/AM1] specifies how they're generated and processed. FC PW control frames are always shorter than 64 octets, so the Length field in the FC Control Word tells you their length 🍓

                        1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +---------------------------------------------------------------+
   |                    FC PW Control Word                         |
   +---------------------------------------------------------------+
   |                  FC Encapsulation Header                      |
   +---------------------------------------------------------------+
   |                                                               |
   +-----              FC PW Control Frame                     ----+
   |                                                               |
   +---------------------------------------------------------------+

          Figure 6 - FC PW Control Frame Encapsulation in PW Packet

3.4. PW Failure Mapping ⚠️

PW failures get detected through PW signaling failure, PW status notifications as defined in [RFC4447], or PW Operations, Administration, and Maintenance (OAM) mechanisms. Those failures MUST be mapped to emulated signal failure indications 🔥 Sending the FC link failure indication to its attached FC link is handled by the NSP, as defined by [FC-BB-5/AM1] 💅

Heads up: the PW failure indications have to be mapped, and the NSP does the FC link failure part. Don't skip either one 🫶

4. Signaling of FC Pseudowires ✨

[RFC4447] specifies using the MPLS Label Distribution Protocol (LDP) to set up and maintain pseudowires. This section explains how specific fields and error codes get used to control FC PW 🎀

The PW Type field in the PWid Forwarding Equivalence Class (FEC) element and the PW generalized ID FEC elements MUST be set to the "FC Port Mode" value in Section 8 💖

The Control Word is REQUIRED for FC pseudowires. So the C-Bit in the PWid FEC element and the PW generalized ID FEC elements MUST be set. If the C-Bit isn't set, the pseudowire MUST NOT be established, and a Label Release MUST be sent with an "Illegal C-Bit" status code [RFC4447] 😭

The Fragmentation Indicator (Parameter ID = 0x09) is specified in [RFC4446], and its usage is defined in [RFC4623]. Fragmentation isn't used in FC PW, so the fragmentation indicator parameter MUST be omitted from the Interface Parameter Sub-TLV 🙈

The Interface MTU Parameter (Parameter ID = 0x01) is specified in [RFC4447]. All FC interfaces have the same MTU, so this parameter MUST be omitted from the Interface Parameter Sub-TLV 🌈

The FCS Retention Indicator (Parameter ID = 0x0A) is specified in [RFC4720]. The CRC treatment defined in this document is different from the one in [RFC4720], so this parameter MUST be omitted from the Interface Parameter Sub-TLV 🦋

5. Timing Considerations ⏰

okay bestie, timing is EVERYTHING in FC, so listen up 👂✨ For Fibre Channel links to work right, the FC link latency between CE1 and CE2 (check Figure 1 🦋) has to hit three vibes:

  • no more than one-half of the R_T_TOV (Receiver Transmitter Timeout Value, default value: 100 milliseconds) of the attached devices for Primitive Sequences;
  • no more than one-half of the E_D_TOV (Error Detect Timeout Value, default value: 2 seconds) of the attached devices for frames; and
  • within the R_A_TOV (Resource Allocation Timeout Value, default value: 10 seconds) of the attached fabric(s), if any. The FC standards require that the E_D_TOV value for each FC link be set so that the R_A_TOV value for the fabric is respected when the worst-case latency occurs for each link (see [FC-FS-2]).

An FC PW MUST adhere to these three timing requirements and MUST NOT be used in environments where high or variable latency may cause these requirements to be violated. No exceptions, babe 💅

These three timeout values are ordered (R_T_TOV < E_D_TOV < R_A_TOV), so keeping one-half of R_T_TOV for all FC PW traffic is enough to cover the rest 🌈 See [FC-FS-2] for definitions of the FC timeout values.

The R_T_TOV is used by the FC link initialization protocol. If an FC PW's latency goes over one-half R_T_TOV, initializing the FC link that's encapsulated by the FC PW may fail, and that link gets stuck in a non-operational state 😭

The E_D_TOV is used to detect failures of operational FC links. If an FC PW's latency breaks the one-half E_D_TOV requirement, the FC link encapsulated by the FC PW may fail. The usual FC move after a link failure is to try to recover the link by initializing it. That initialization will also fail if the FC PW latency goes over one-half R_T_TOV, which is an even tighter requirement 💀

The R_A_TOV is used to decide when FC communication resources (for example, values that identify FC frames) may be reused. If an FC PW breaks the one-half E_D_TOV requirement badly enough to also make the FC fabric break the R_A_TOV requirement, then reusing frame identification values after an R_A_TOV timeout may result in multiple FC frames with the same identification values, causing incorrect Fibre Channel operation. For example, if two such frames get swapped between I/O operations, the result may corrupt data in those I/O operations 🙈 ngl this is the scariest part of the whole doc.

The PING and PING_ACK FC PW control frames defined in Section 6.4.7 of [FC-BB-5/AM1] SHOULD be used to measure the current FC pseudowire latency between the Customer Edge (CE) devices 🔍 If the measured latency violates any of the timing requirements, then the FC PW PE MUST generate a WAN Down event as specified in [FC-BB-5/AM1].

The WAN Down event makes the PE continuously send NOS (an FC Primitive Sequence) on the native FC link to the FC port at the other end of that link (typically an E_Port on a switch in this case) 🫠

This immediately makes the FC link carried by the PW go non-operational, which halts transmission of FC traffic. BUT it is not necessary to tear down the pseudowire itself (for example, destroying the MPLS path set up by LDP). Plot twist 🎀

The Transparent FC-BB initialization state machine in [FC-BB-5/AM1] is the protocol that tries to recover from a WAN Down event (in other words, bring the WAN back up) 🦋 If that protocol brings the WAN back up, FC traffic resumes and the standard FC link recovery protocol brings the encapsulated FC link back up. If the previous pseudowire was destroyed, attempts get made to re-establish the path via LDP as part of recovering from the WAN Down event. If the PW round-trip latency stays above R_T_TOV, the initialization protocol for the FC PW keeps timing out while trying to recover from the WAN Down event, which blocks recovery of the FC link carried by the PW; see [FC-BB-5/AM1] 😤

6. Security Considerations 🛡️

okay security era 🔐 The FC PW is an MPLS pseudowire, so for the MPLS pseudowire security considerations, go check the security considerations sections of [RFC3985] and [RFC4385] 📖

The protocols used to implement security in a Fibre Channel fabric are defined in [FC-SP]. These protocols operate at higher layers of the FC hierarchy and are transparent to the FC PW, so the PW just vibes past them 💜

The FC timing requirements (see Section 5) create an exposure of the FC PW to inserted latency. Injecting enough latency to push the FC PW round-trip time over R_T_TOV (default: 100 ms) may cause the FC PW to fail in an active fashion, because the FC link initialization protocol keeps timing out. ⚠️⚠️ This is a serious one, bestie. OAM functionality for deployed FC PWs SHOULD monitor for this situation persisting and respond accordingly (for example, shut down the FC PW so you don't waste WAN bandwidth on an FC PW whose FC link can't be initialized because of excessive latency) 🚨

7. Applicability Statement 🌸

FC PW lets Fibre Channel ports transparently transport FC traffic between each other, and it saves network bandwidth by removing FC Idle Primitive Signals and reducing the number of FC Primitive Sequences ✨ Efficient AND cute 🎀

  • The pair of CE devices operates as if they were directly connected by an FC link. In particular, they react to Primitive Sequences on their local FC links as specified by the FC standards 🫶
  • The FC PW carries only FC data frames, FC Primitive Signals, and a subset of the copies of an FC Primitive Sequence. Idle Primitive Signals are suppressed, and long streams of the same Primitive Sequence are reduced over the PW, thus saving bandwidth. 💅
  • The PW PE MUST generate Idle Primitive Signals to the attached FC link when there is no other traffic to transmit on the attached FC link [FC-FS-2]. 🌈
  • The PW PE MUST send Primitive Sequences continuously to the attached FC port, as required by the FC standards [FC-FS-2]. ⭐

FC PW traffic should only traverse MPLS networks that are provisioned based on traffic engineering to provide dedicated bandwidth for FC PW traffic. The MPLS network should enforce ingress traffic policing so that delivery of FC PW traffic can be assured 🦋 To extend FC across a network that doesn't meet these requirements, FCIP SHOULD be used instead of an FC PW (see [RFC3821] and [FC-BB-6]). Use the right tool, babe 🧚

This document does not provide any mechanisms for protecting an FC PW against network outages. As a consequence, the resilience of the emulated FC service to such outages depends on the underlying MPLS network, which should be protected against failures 🌧️ When a network outage is detected, the PE SHOULD use a WAN Down event (as specified in [FC-BB-5/AM1]) to convey the PW status to the CE and enable faster outage handling. ⏩

8. IANA Considerations 🧚

IANA has assigned a new MPLS Pseudowire (PW) type, and it looks like this 💖

   PW type      Description           Reference
   --------     --------------        ----------
   0x001F       FC Port Mode          RFC 6307

IANA has also reserved the following Pseudowire Interface Parameters Sub-TLV Types. These Sub-TLV types were used for the FC PW Selective Retransmission protocol, which the PWE3 working group has decided to eliminate. This action prevents future use of these values for other purposes, as there is at least one implementation of the Selective Retransmission protocol that has been deployed 🙈

   Parameter  ID Length  Description  Reference
   ---------  ---------  -----------  ---------
   0x12                  Reserved     RFC 6307
   0x13                  Reserved     RFC 6307
   0x14                  Reserved     RFC 6307
   0x15                  Reserved     RFC 6307

9. Acknowledgments 🫶

Shoutout to the previous versions' authors, Moran Roth, Ronen Solomon, and Munefumi Tsurusawa 👑 Their efforts and contributions are gratefully acknowledged. The authors also want to thank Stewart Bryant, Elwyn Davies, Steve Hanna, Dave Peterson, Yaakov Stein, Alexander Vainshtein, and the members of the IESG for helpful comments on this document 🍓

The protocol specified in this document is meant to be used together with the Fibre Channel pseudowire portion of the FC-BB-5 Amendment 1 specification, which was developed by INCITS Technical Committee T11 🌸 The authors would like to thank the members of both the IETF and T11 organizations who have supported and contributed to this work. Teamwork makes the dream work 💜

10. Normative References 📚

  • [FC-BB-5/AM1] "Fibre Channel - Backbone-5 / Amendment 1", INCITS 462-2010/AM 1-2012, June 2012.
  • [FC-FS-2] "Fibre Channel - Framing and Signaling-2 (FC-FS-2)", ANSI INCITS 424:2007, August 2007.
  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [RFC3643] Weber, R., Rajagopal, M., Travostino, F., O'Donnell, M., Monia, C., and M. Merhar, "Fibre Channel (FC) Frame Encapsulation", RFC 3643, December 2003.
  • [RFC3985] Bryant, S., Ed., and P. Pate, Ed., "Pseudo Wire Emulation Edge-to-Edge (PWE3) Architecture", RFC 3985, March 2005.
  • [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for Use over an MPLS PSN", RFC 4385, February 2006.
  • [RFC4446] Martini, L., "IANA Allocations for Pseudowire Edge to Edge Emulation (PWE3)", BCP 116, RFC 4446, April 2006.
  • [RFC4447] Martini, L., Ed., Rosen, E., El-Aawar, N., Smith, T., and G. Heron, "Pseudowire Setup and Maintenance Using the Label Distribution Protocol (LDP)", RFC 4447, April 2006.
  • [RFC4623] Malis, A. and M. Townsley, "Pseudowire Emulation Edge-to-Edge (PWE3) Fragmentation and Reassembly", RFC 4623, August 2006.
  • [RFC4720] Malis, A., Allan, D., and N. Del Regno, "Pseudowire Emulation Edge-to-Edge (PWE3) Frame Check Sequence Retention", RFC 4720, November 2006.

11. Informative References 🔍

  • [FC-BB-6] "Fibre Channel Backbone-6" (FC-BB-6), T11 Project 2159-D, Rev 1.04, Work in Progress, January 2012.
  • [FC-SP] "Fibre Channel - Security Protocols" (FC-SP), ANSI INCITS 426:2007, February 2007.
  • [RFC3821] Rajagopal, M., Rodriguez, E., and R. Weber, "Fibre Channel Over TCP/IP (FCIP)", RFC 3821, July 2004.
  • [RFC4717] Martini, L., Jayakumar, J., Bocci, M., El-Aawar, N., Brayley, J., and G. Koleyni, "Encapsulation Methods for Transport of Asynchronous Transfer Mode (ATM) over MPLS Networks", RFC 4717, December 2006.
  • [T11] INCITS Technical Committee T11, http://www.t11.org, January 2011.

Authors' Addresses 🏡

   David L. Black (editor)
   EMC Corporation
   176 South Street
   Hopkinton, MA 01748
   USA
   Phone: +1 (508) 293-7953
   EMail: david.black@emc.com

   Linda Dunbar (editor)
   Huawei Technologies
   1700 Alma Drive, Suite 500
   Plano, TX 75075
   USA
   Phone: +1 (972) 543-5849
   EMail: ldunbar@huawei.com

   Moran Roth
   Infinera Corporation
   169 Java Drive
   Sunnyvale, CA 94089
   USA
   Phone: (408) 572-5200
   EMail: MRoth@infinera.com

   Ronen Solomon
   Orckit-Corrigent Systems
   126, Yigal Alon st.
   Tel Aviv
   Israel
   Phone: +972-3-6945316
   EMail: ronens@corrigent.com
📦 where this page comes from