RFC 8279
Multicast Using Bit Index Explicit Replication (BIER)
RFC 8279: Multicast Using Bit Index Explicit Replication (BIER) 🦋
okay bestie, let's start with the basics 💖 this is RFC 8279, and it's giving experimental vibes. It's labelled Category: Experimental and the ISSN is 2070-1721. It came out in November 2017 🍓
The editors are IJ. Wijnands (Cisco Systems, Inc.), E. Rosen (Juniper Networks, Inc.), A. Dolganow (Nokia), T. Przygienda (Juniper Networks, Inc.), and S. Aldrin (Google, Inc.) 🫶 Wijnands and Rosen are the editors, so they're the main authors. The title block doesn't say this RFC obsoletes or updates anything, so it stands on its own ✨
Abstract ✨
okay so the abstract is basically the whole pitch 💅 This document defines a brand new architecture for forwarding multicast data packets. The goal is optimal forwarding of multicast packets through a "multicast domain", and it's iconic because it doesn't need a protocol to explicitly build multicast distribution trees 🌸 Intermediate nodes don't have to keep any per-flow state either. This architecture is called "Bit Index Explicit Replication" (BIER) 🦋
Here's how it works. When a multicast data packet enters the domain, the ingress router figures out which egress routers the packet needs to go to. Then it wraps the packet in a BIER header 🎀 That header holds a bit string, and each bit stands for exactly one egress router in the domain. To send the packet to a particular set of egress routers, you just set the bits for those routers in the header. The rules for forwarding based on the BIER header are in this document 📜
Getting rid of per-flow state and the explicit tree-building protocols makes everything way simpler, and honestly that's the whole main character moment 👑
Status of This Memo ✨
not gonna lie, this part is a little formal, but it matters 🙈 This document is not an Internet Standards Track specification. It's published so people can examine it, try out experimental implementations, and evaluate it 🧪
It defines an Experimental Protocol for the Internet community. It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community 💜 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved document is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 if you're curious 🌈
For the latest status info, errata, or ways to send feedback, head to https://www.rfc-editor.org/info/rfc8279 🔥
Copyright Notice ✨
legal bit time, but keep it cute 💅 The copyright is Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they were in effect on the publication date. Please actually read these, because they lay out your rights and restrictions for this document 🫶 Code components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License ⚠️
Table of Contents 🧚
1. Introduction ....................................................4
2. The BFR Identifier and BFR-Prefix ...............................7
3. Encoding BFR Identifiers in BitStrings ..........................8
4. Layering .......................................................11
4.1. The Routing Underlay ......................................11
4.2. The BIER Layer ............................................12
4.3. The Multicast Flow Overlay ................................13
5. Advertising BFR-ids and BFR-Prefixes ...........................13
6. BIER Intra-Domain Forwarding Procedures ........................15
6.1. Overview ..................................................15
6.2. BFR Neighbors .............................................17
6.3. The Bit Index Routing Table ...............................18
6.4. The Bit Index Forwarding Table ............................19
6.5. The BIER Forwarding Procedure .............................20
6.6. Examples of BIER Forwarding ...............................23
6.6.1. Example 1 ..........................................23
6.6.2. Example 2 ..........................................24
6.7. Equal-Cost Multipath Forwarding ...........................26
6.7.1. Non-deterministic ECMP .............................27
6.7.2. Deterministic ECMP .................................28
6.8. Prevention of Loops and Duplicates ........................29
6.9. When Some Nodes Do Not Support BIER .......................30
6.10. Use of Different BitStringLengths within a Domain ........33
6.10.1. BitStringLength Compatibility Check ...............34
6.10.2. Handling BitStringLength Mismatches ...............36
6.10.3. Transitioning from One BitStringLength to
Another ...........................................36
7. Operational Considerations .....................................37
7.1. Configuration .............................................37
8. IANA Considerations ............................................37
9. Security Considerations ........................................38
10. References ....................................................39
10.1. Normative References .....................................39
10.2. Informative References ...................................39
Acknowledgements ..................................................40
Contributors ......................................................41
Authors' Addresses ................................................431. Introduction ✨
okay bestie, this doc is literally about a brand new way to forward multicast data packets 💖 It's called "Bit Index Explicit Replication" (BIER), and it gives you optimal forwarding through a "multicast domain" 🦋 The wild part: you don't need a protocol to build multicast distribution trees, and the middle routers don't have to keep any per-flow state 💅
A router that supports BIER is called a "Bit-Forwarding Router" (BFR) 🎀 The BIER control-plane protocols (see Section 4.2) run inside a "BIER domain", so all the BFRs in there can swap the info they need to forward packets to each other using BIER 🫶
A multicast packet enters a BIER domain at a "Bit-Forwarding Ingress Router" (BFIR) and leaves at one or more "Bit-Forwarding Egress Routers" (BFERs) 🌸 A BFR that gets a multicast packet from another BFR in the same domain and passes it along to yet another BFR in that domain is a "transit BFR" for that packet 🧚 One BFR can be a BFIR for some traffic, a BFER for other traffic, and a transit BFR for even more traffic. For any given packet, a BFR might be a BFIR, a transit BFR, and/or one of the BFER(s), all at once. Main character energy, literally 👑
A BIER domain can have one or more sub-domains 🌈 Every BIER domain MUST contain at least one sub-domain, the "default sub-domain" (also written as "sub-domain 0"). If a domain has more than one sub-domain, each BFR in it MUST be provisioned to know which sub-domains it belongs to. Each sub-domain gets identified by a sub-domain-id in the range [0,255] ✨
For each sub-domain a BFR belongs to, if that BFR can act as a BFIR or a BFER, it MUST be provisioned with a "BFR-id" that's unique within that sub-domain 💜 A BFR-id is a small, unstructured positive integer. For example, if a sub-domain has 1,374 BFRs, each one could get a BFR-id in the range [1,1374] 🍓
If a BFR is in more than one sub-domain, it may (though it doesn't have to) use a different BFR-id in each one 🦋
When a multicast packet shows up from outside the domain at a BFIR, the BFIR works out the set of BFERs the packet will go to 💅 It also picks the sub-domain the packet will travel in. Deciding which sub-domain a packet goes in is called "assigning the packet to a sub-domain" 🎀 How you choose that sub-domain is out of scope for this document 🙈 But once a packet is assigned, it stays in that sub-domain until it leaves the BIER domain. So the sub-domain a packet is assigned to MUST NOT be changed while the packet is in flight through the BIER domain. No switching mid-trip, bestie 🚫
Once the BFIR has the sub-domain and the set of BFERs for a packet, it wraps the packet in a "BIER header" 🌸 The BIER header holds a bit string where each bit stands for one BFR-id. To say "this BFER should get this packet," the BFIR sets the bit that matches that BFER's BFR-id in the sub-domain the packet was assigned to ✨ In this doc, "BitString" means the bit string field in the BIER header, and "payload" means the packet that got encapsulated 🫶 So a "BIER-encapsulated" packet is just a "BIER header" followed by a "payload" 💖
The number of BFERs a packet can reach is only limited by the length of the BitString in the BIER header 🧚 Different deployments can pick different BitString lengths. "BitStringLength" is the number of bits in the BitString 🌈 Some deployments might have more BFERs in one sub-domain than there are bits in the BitString 😭 To handle that, the BIER encapsulation carries both the BitString and a "Set Identifier" (SI). The BitString plus the SI together decide which BFERs get a given packet 💅
- By convention, the least significant (rightmost) bit of the BitString is "bit 1", and the most significant (leftmost) bit is "bit BitStringLength". 🎀
- If a BIER-encapsulated packet has an SI of
nand a BitString with bitkset, the packet must be delivered to the BFER whose BFR-id (in the sub-domain the packet was assigned to) isn*BitStringLength+k. 🦋
Let's do an example 🌸 Say the BIER encapsulation uses a BitStringLength of 256 bits. Bit 1 is still the rightmost bit, and bit 256 is the leftmost. Now imagine a packet is assigned to sub-domain 0 and needs to reach three BFERs with BFR-ids 13, 126, and 235 in sub-domain 0. The BFIR builds a BIER encapsulation with the SI set to zero and bits 13, 126, and 235 of the BitString set. Everything else in the BitString stays clear 🍓 Now, if the packet also has to reach a BFER with BFR-id 257, the BFIR has to make a second copy of the packet. That encapsulation gets an SI of 1 and a BitString with bit 1 set and all other bits clear 😭
It's generally a good idea to assign the BFR-ids in a sub-domain so that as many BFERs as possible fit in a single bit string 💜
Here's another one 💅 Say a BFR (let's call it "BFR-A") gets a packet whose BIER encapsulation has an SI of 0 and a BitString with bits 13, 26, and 235 set. BFR-A has two BFR neighbors, BFR-B and BFR-C. The best path to BFERs 13 and 26 goes through BFR-B, but the best path to BFER 235 goes through BFR-C. So BFR-A replicates the packet and sends one copy to BFR-B and one to BFR-C 🦋 But here's the trick: BFR-A clears bit 235 in the BitString of the copy it sends to BFR-B, and it clears bits 13 and 26 in the copy going to BFR-C. That means BFR-B only forwards toward BFERs 13 and 26, and BFR-C only forwards toward BFER 235. Each BFER gets exactly one copy of the packet, which is so clean 🎀
Detailed procedures for forwarding a BIER-encapsulated packet through a BIER domain are in Section 6 🫶
Because the set of BFERs for a packet is explicitly encoded in the BIER header, the packet goes from its BFIR to each of its BFERs along an optimal path 🌈 And the packet never gets sent to a BFER that doesn't need it ✨ That's the optimal forwarding part, and it happens without transit BFRs keeping per-flow state or running a multicast tree-building protocol 💖
Putting the set of egress nodes into the header of a multicast packet isn't a new idea 🧚 For example, [Boivie_Feldman] proposes encoding the set of egress nodes as a set of IP addresses, with mechanisms and procedures that are somewhat similar to the ones in this document 🙈 But BIER encodes each BFR-id as a single bit in a bit string, so it can represent up to 128 BFERs in the same number of bits it would take to carry the IPv6 address of one BFER 🔥 That's why BIER scales to a much bigger number of egress nodes per packet 👑
BIER doesn't require each transit BFR to look up the best path to every BFER in the BIER header 💅 The number of lookups needed for one packet in the forwarding path can be limited to the number of neighboring BFRs, which can be way smaller than the number of BFERs. See Section 6 (especially Section 6.5) for details 🦋
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 📚🫶
2. The BFR Identifier and BFR-Prefix 🎀
Each BFR MUST be assigned a single "BFR-prefix" for each sub-domain it belongs to 💖 A BFR's BFR-prefix MUST be an IP address (either IPv4 or IPv6) of that BFR. It is RECOMMENDED that the BFR-prefix be a loopback address of the BFR 🦋
If a BFR belongs to more than one sub-domain, it may (though it doesn't have to) have a different BFR-prefix in each sub-domain 🌸
All BFR-prefixes used within a given sub-domain MUST belong to the same address family (either IPv4 or IPv6) 🧚
The BFR-prefix of a given BFR in a given sub-domain MUST be routable in that sub-domain 🌈 Whether a particular BFR-prefix is routable in a given sub-domain depends on the "routing underlay" linked to that sub-domain. The notion of "routing underlay" is described in Section 4.1 ✨
A "BFR Identifier" (BFR-id) is a number in the range [1,65535] 💜 Within a given sub-domain, every BFR that may need to act as a BFIR or BFER MUST have a single BFR-id that identifies it uniquely within that sub-domain 🫶 A BFR that doesn't need to act as a BFIR or BFER in a given sub-domain doesn't need a BFR-id in that sub-domain 💅
The value 0 is not a legal BFR-id 🙈
How a particular BFR-id gets assigned to a particular BFR is out of scope for this document 🎀 However, it is RECOMMENDED that the BFR-ids for each sub-domain be assigned "densely" from the numbering space, because that gives a more efficient encoding (see Section 3) 🍓 So if there are 256 or fewer BFERs, it is RECOMMENDED to assign all the BFR-ids from the range [1,256]. If there are more than 256 BFERs but fewer than 512, it is RECOMMENDED to assign all the BFR-ids from the range [1,512], with as few "holes" as possible in the earlier range 🦋 Sometimes, though, a deployment might benefit from going against this recommendation, and Section 3 talks about that more 👑
In some deployments, it may not be possible to support the full range of 65,535 BFR-ids in a given sub-domain 😭 For example, if the BFRs in a given sub-domain only support 16 SIs and only support BitStringLengths of 256 or less, then only 16*256=4,096 BFR-ids can be supported in that sub-domain 🌈
3. Encoding BFR Identifiers in BitStrings ✨
okay bestie, so to stick a BFR-id into a BIER data packet, you have to turn it into an SI and a BitString 🎀. How that conversion works depends on a parameter called "BitStringLength". If the BFR-id is N, then:
- SI is the integer part of the quotient
(N-1)/BitStringLength. - The BitString has one bit position set. The low-order bit is bit 1 and the high-order bit is bit
BitStringLength, so the bit position for BFR-idNis((N-1) modulo BitStringLength)+1.
If a bunch of different BFR-ids all resolve to the same SI, you can fit them all into a single BitString 💅. You just combine their BitStrings with a bitwise logical OR operation. So cute, right? 🧚
Different BIER domains (or even different sub-domains inside one domain) can use different values of BitStringLength 🌈. Each BFR MUST be provisioned to know the following:
- The BitStringLength ("Imposition BitStringLength") and sub-domain ("Imposition sub-domain") it uses when it imposes (as a BFIR) a BIER encapsulation on a particular set of packets, and
- The BitStringLengths ("Disposition BitStringLengths") it will process when it receives packets from a particular sub-domain (as a BFR or BFER).
A BFIR doesn't have to use the same Imposition BitStringLength or the same Imposition sub-domain for every packet it encapsulates 🦋. BUT, if a particular BFIR is provisioned to use a particular Imposition BitStringLength and a particular Imposition sub-domain for a given set of packets, then all the other BFRs with BFR-ids in that sub-domain SHOULD be provisioned to process received BIER packets with that BitStringLength. In other words, all those other BFRs SHOULD be provisioned with that BitStringLength as a Disposition BitStringLength for that sub-domain. There are exceptions to this rule that MAY be made under certain conditions, and Section 6.10 talks about that 👀.
When a BIER encapsulation is specified, the specification MUST define a default BitStringLength for that encapsulation. Every BFIR supporting that encapsulation MUST be capable of being provisioned with that default BitStringLength as its Imposition BitStringLength. Every BFR and BFER supporting that encapsulation MUST be capable of being provisioned with that default BitStringLength as a Disposition BitStringLength.
The specification of a BIER encapsulation MAY also allow other BitStringLengths too ✨.
If a BFR can be provisioned with a given value of BitStringLength as an Imposition BitStringLength, it MUST also be capable of being provisioned with that same value as one of its Disposition BitStringLengths. It SHOULD also be capable of being provisioned with each legal smaller value of BitStringLength as (a) its Imposition BitStringLength and (b) one of its Disposition BitStringLengths. Main character energy 💖.
To support switching from one BitStringLength to another, every BFR MUST be capable of being provisioned to use two different Disposition BitStringLengths at the same time 🔥.
A BFR MUST support SI values in the range [0,15] and MAY support SI values in the range [0,255]. ("Supporting the values in a given range" just means that any value in that range is legal and will be properly interpreted 🫶.) Now here's the math. For a given BitStringLength, the total number of BFR-ids you can represent is the BitStringLength multiplied by the number of supported SIs. For example, if a deployment uses a BitStringLength of 64 in a given sub-domain and supports 256 SIs, that deployment can only support 16384 BFR-ids in that sub-domain 🧮. And even a deployment that supports 256 SIs won't be able to support 65,535 BFR-ids unless it uses a BitStringLength of at least 256. Not me doing the math and getting humbled 💀.
When a BFIR decides that a multicast data packet assigned to a given sub-domain needs to go to a particular set of destination BFERs, it splits that set into subsets 🍓. Each subset holds the target BFERs whose BFR-ids in that sub-domain all resolve to the same SI. Let's call these the "SI-subsets" for the packet. Each SI-subset can be represented by a single BitString. The BFIR makes one copy of the packet for each SI-subset, and then the BIER encapsulation gets applied to each copy. Each encapsulated copy specifies a single SI and contains the BitString that represents all the BFR-ids in its SI-subset. And of course, to read that BitString properly, it has to be possible to infer the sub-domain-id from the encapsulation too 🔍.
Let's look at an example, bestie. Say a BFIR decides a given packet needs to go to three BFERs whose BFR-ids (in the right sub-domain) are 27, 235, and 497. The BFIR will have to forward two copies of the packet 🎀. One copy, tied to SI=0, has a BitString with bits 27 and 235 set. The other copy, tied to SI=1, has a BitString with bit 241 set.
To keep the number of copies of a multicast packet as low as possible, it is RECOMMENDED that BFR-ids in a given sub-domain get assigned "densely" (see Section 2) from the numbering space 🌸. This keeps the number of SIs used in that sub-domain small. HOWEVER, depending on the details of a specific deployment, other assignment methods might be more advantageous 🤔. Imagine a deployment where every multicast flow is meant for either the "east coast" or the "west coast", but never both. In that case, it would be a great idea to assign BFR-ids so that all the "west coast" BFR-ids land in the same SI-subset, and all the "east coast" BFR-ids land in the same SI-subset too 🗽.
When a BFR gets a BIER data packet, it infers the SI from the encapsulation. From the SI and the BitString, it can then work out the set of BFERs the packet needs to be forwarded to 🧚.
In some of the examples later in this document, we'll use a BitStringLength of 4 and write a BFR-id as "SI:xyzw", where SI is the Set Identifier of the BFR-id (assuming a BitStringLength of 4) and xyzw is a string of 4 bits 💜. A BitStringLength of 4 is only for the examples, though. Real deployments would not be expected to use something that tiny 🙈.
It's possible that several different forms of BIER encapsulation will be developed 🦋. If that happens, the encapsulation a deployment uses will depend on the type of network infrastructure used to realize the BIER domain. The details of the BIER encapsulation(s) will be given in companion documents. An encapsulation for use in MPLS networks is described in [MPLS_BIER_ENCAPS], and that document also describes a very similar encapsulation that can be used in non-MPLS networks ✨.
4. Layering ✨
It helps to think of the BIER architecture as three layers 🎀: the "routing underlay", the "BIER layer", and the "multicast flow overlay".
4.1. The Routing Underlay
The "routing underlay" sets up "adjacencies" between pairs of BFRs and figures out one or more "best paths" from a given BFR to a given set of BFRs 🛤️. Each path is a sequence of BFRs <BFR(k), BFR(k+1), ..., BFR(k+n)> where BFR(k+j) is "adjacent" to BFR(k+j+1) (for 0<=j<n).
Say we're at a BFR called BFR-A. For every IP address that belongs to a BFR in the BIER domain, the routing underlay maps that IP address to a set of one or more "equal-cost" adjacencies 💅. If BFR-A has to forward a BIER data packet to a given BFER, say BFER-B, the packet follows the path from BFR-A to BFER-B that the routing underlay determined 🫶.
In a typical deployment, the routing underlay is expected to be the default topology that the Interior Gateway Protocol (IGP), e.g., OSPF, uses for unicast routing 🌈. In that case, the underlay adjacencies are just the OSPF adjacencies. A BIER data packet going from BFR-A to BFER-B follows the path OSPF picked for unicast traffic from BFR-A to BFER-B.
Now, if you want multicast traffic from BFR-A to BFER-B to take a different path than unicast traffic does, you can use a different underlay 🦋. For example, if multi-topology OSPF is in use, one OSPF topology could be used for unicast traffic and the other for multicast traffic. (Each topology would count as a different underlay.) Or you could deploy a routing underlay that builds some kind of multicast-specific tree 🌳. BIER could then forward multicast data packets along that multicast-specific tree, while unicast packets keep following the "ordinary" OSPF best path. (In a setup like this, lots of multicast flows could travel along a single tree, and the BitString carried by a particular packet would identify the nodes of the tree that need to receive that packet 💜.) It's even possible to use multiple routing underlays with BIER, as long as you can tell from a data packet's BIER encapsulation which underlay is used for that packet 👑.
If multiple routing underlays are used in a single BIER domain, each BIER sub-domain MUST be associated with a single routing underlay. Multiple sub-domains MAY share the same routing underlay, though 🔥. A BFR that belongs to multiple sub-domains MUST be provisioned to know which routing underlay each sub-domain uses. By default (meaning, if nobody provisioned anything different), each sub-domain uses the default topology of the unicast IGP as its routing underlay ✨.
In scenarios where External BGP (EBGP) is used as the IGP, the underlay adjacencies are, by default, the BGP adjacencies 🌸.
The protocols and procedures of the routing underlay are outside the scope of this document 🙃.
4.2. The BIER Layer
The BIER layer is the set of protocols and procedures used to send a multicast data packet across a BIER domain, from its BFIR to its BFERs 🚀. It includes these components:
- Protocols and procedures that a given BFR uses to advertise, to all other BFRs in the same BIER domain:
- its BFR-prefix;
- its BFR-id in each sub-domain for which it has been provisioned with a BFR-id;
- the set of Disposition BitStringLengths it has been provisioned to use for each sub-domain;
- optionally, information about the routing underlay associated with each sub-domain.
- The procedures used by a BFIR to impose a BIER header on a multicast data packet.
- The procedures for forwarding BIER-encapsulated packets and for modifying the BIER header during transit.
- The procedures used by a BFER to decapsulate a BIER packet and properly dispatch it 💖.
4.3. The Multicast Flow Overlay ✨
okay so the "multicast flow overlay" is the squad of protocols and procedures that handles these jobs 💅
- When a BFIR receives a multicast data packet from outside the BIER domain, the BFIR must determine the set of BFERs for that packet. The multicast flow overlay is what provides this info 🎀
- When a BFER receives a BIER-encapsulated packet from inside the BIER domain, the BFER must determine how to further forward the packet. The multicast flow overlay gives it that info too ✨
For example, say the BFIR and BFERs are Provider Edge (PE) routers giving Multicast Virtual Private Network (MVPN) service 🌸 The multicast flow overlay is made of the protocols and procedures described in [RFC6513] and [RFC6514]. The MVPN signaling in those RFCs lets an ingress PE figure out the set of egress PEs for a given multicast flow (or set of flows). It also lets an egress PE work out which "Virtual Routing and Forwarding Tables" (VRFs) should get the multicast packets coming in from the backbone network. MVPN signaling also has several parts that depend on the type of "tunneling technology" used to carry multicast data through the network 🦋 Since BIER is, in effect, a brand new type of "tunneling technology", some extensions to the MVPN signaling are needed so the multicast flow overlay can properly interface with the BIER layer. Those are specified in [BIER_MVPN] 💜
MVPN is just one example of a multicast flow overlay 🧚 Protocols and procedures for other overlays will show up in companion documents. It's also possible to implement the multicast flow overlay by means of a "Software-Defined Network" (SDN) controller. Specifying the protocols and procedures of the multicast flow overlay is outside the scope of this document, so we're good 🙈
5. Advertising BFR-ids and BFR-Prefixes 📣
As stated in Section 2, each BFER is assigned (by provisioning) a BFR-id (for a given BIER sub-domain). Each BFER must advertise these assignments to all the other BFRs in the domain. Similarly, each BFR is assigned (by provisioning) a BFR-prefix (for a given BIER domain), and it must advertise this assignment to all the other BFRs in the domain. Finally, each BFR has been provisioned to use a certain set of Disposition BitStringLengths for each sub-domain, and it must advertise these to all other BFRs in the domain 🫶
If the BIER domain is also a link-state routing IGP domain (i.e., an OSPF or IS-IS domain), the advertisement of the BFR-prefix, <sub-domain-id, BFR-id>, and BitStringLength can be done using the advertisement capabilities of the IGP. For example, if a BIER domain is also an OSPF domain, these advertisements can be done using the OSPF "Opaque Link State Advertisement" (Opaque LSA) mechanism. Details of the necessary extensions to OSPF and IS-IS will be provided in companion documents. (See [OSPF_BIER_EXTENSIONS] and [ISIS_BIER_EXTENSIONS].) 🌈
If, in a particular deployment, the BIER domain is not an OSPF or IS-IS domain, procedures suitable to that deployment must be used to advertise this info. Details of the necessary procedures will be provided in companion documents. For example, if BGP is the only routing algorithm used in the BIER domain, the procedures of [BGP_BIER_EXTENSIONS] may be used 🍓
These advertisements let each BFR link a given <sub-domain-id, BFR-id> with a given BFR-prefix. As you'll see in later sections of this document, knowing this association is a huge part of the forwarding process 👑
Since each BFR needs a unique (in each sub-domain) BFR-id, two different BFRs will not advertise ownership of the same <sub-domain-id, BFR-id> unless there has been a provisioning error 😭
- If BFR-A determines that BFR-B and BFR-C have both advertised the same BFR-id for the same sub-domain, BFR-A MUST log an error. Suppose the duplicate BFR-id is "N". When BFR-A is functioning as a BFIR, it MUST NOT encode the BFR-id value N in the BIER encapsulation of any packet that has been assigned to the given sub-domain, even if it has determined that the packet needs to be received by BFR-B and/or BFR-C.
This means BFR-B and BFR-C cannot receive multicast traffic at all in the given sub-domain until the provisioning error is fixed. However, that is preferable to having them receive each other's traffic 💀
- Suppose BFR-A has been provisioned with BFR-id N for a particular sub-domain, but it hasn't yet advertised its ownership of BFR-id N for that sub-domain. Suppose also that it has received an advertisement from a different BFR (say BFR-B) that is advertising ownership of BFR-id N for the same sub-domain. In that case, BFR-A SHOULD log an error and MUST NOT advertise its own ownership of BFR-id N for that sub-domain as long as the advertisement from BFR-B is extant.
This procedure may prevent the accidental misconfiguration of a new BFR from impacting an existing BFR 🛡️
If a BFR advertises that it has a BFR-id of 0 in a particular sub-domain, other BFRs receiving that advertisement MUST interpret it as meaning the advertising BFR does not have a BFR-id in that sub-domain 🙅♀️
6. BIER Intra-Domain Forwarding Procedures 🦋
This section specifies the rules for forwarding a BIER-encapsulated data packet within a BIER domain. These rules are not meant to dictate an implementation strategy. To conform to this specification, an implementation need only produce the same results that these rules produce ✨
6.1. Overview 🗺️
This section gives a quick overview of the BIER forwarding procedures. The subsections that follow spell out the procedures in more detail 💅
To forward a BIER-encapsulated packet:
- Determine the packet's sub-domain.
- Determine the packet's
BitStringLengthandBitString. - Determine the packet's
SI. - From the sub-domain, the
SI, and theBitString, determine the set of destination BFERs for the packet. - Using information provided by the routing underlay associated with the packet's sub-domain, determine the next-hop adjacency for each of the destination BFERs.
- It's possible that the packet's
BitStringwill have one or more bits that correspond to BFR-ids that are not in use. It's also possible that the packet'sBitStringwill have one or more bits that correspond to BFERs that are unreachable, i.e., that have no next-hop adjacency. In the following, we will consider the "next-hop adjacency" for all such bit positions to be the "null" next hop 🫠 - Partition the set of destination BFERs such that all the BFERs in a single partition have the same next hop. We will say that each partition is associated with a next hop 🧩
- For each partition:
- a. Make a copy of the packet.
- b. Clear any bit in the packet's
BitStringthat identifies a BFER that is not in the partition. - c. Transmit the packet to the associated next hop. (If the next hop is the null next hop, the packet is discarded.)
If a BFR receives a BIER-encapsulated packet whose <sub-domain, SI, BitString> triple identifies that BFR itself, then the BFR is also a BFER for that packet. As a BFER, it must pass the payload to the multicast flow overlay. If the BitString has bits set for other BFRs, the packet also needs to be forwarded further within the BIER domain. If the BF(E)R also forwards one or more copies of the packet within the BIER domain, the bit representing the BFR's own BFR-id MUST be clear in all the copies 💖
When BIER on a BFER is about to pass a packet to the multicast flow overlay, it of course decapsulates the packet by removing the BIER header. However, it may be necessary to give the multicast flow overlay some contextual info that was obtained from the BIER encapsulation. The info that needs to pass between the BIER layer and the multicast flow overlay is specific to the multicast flow overlay. Specifying the interaction between the BIER layer and the multicast flow overlay is outside the scope of this specification 🧚
If the BIER encapsulation contains a "Time to Live" (TTL) value, this value is not, by default, inherited by the payload. If a particular multicast flow overlay needs to know the TTL value, this needs to be specified in whatever specification defines the interaction between BIER and that multicast flow overlay ⏳
If the BIER encapsulation contains a Traffic Class field, a Type of Service field, a Differentiated Services field, or any field of that sort, the value of that field is not, by default, passed to the multicast flow overlay. If a particular multicast flow overlay needs to know the values of such fields, that fact needs to be specified in whatever specification defines the interaction between BIER and that multicast flow overlay 🎀
When BIER on a BFER passes a packet to the multicast flow overlay, the overlay will decide how to further dispatch the packet. If the packet needs to be forwarded into another BIER domain, then the BFR acts as a BFER in one BIER domain and as a BFIR in another. Main character energy 👑
A BIER-encapsulated packet cannot pass directly from one BIER domain to another. At the boundary between BIER domains, the packet must be decapsulated and passed to the multicast flow overlay 🚧
Note that when a BFR transmits multiple copies of a packet within a BIER domain, only one copy will be destined to any given BFER. Therefore, it is not possible for any BIER-encapsulated packet to be delivered more than once to any BFER 🌸
6.2. BFR Neighbors 🤝
The "BFR Neighbors" (BFR-NBRs) of a given BFR, say BFR-A, are those BFRs that, according to the routing underlay, are adjacencies of BFR-A. Each BFR-NBR will have a BFR-prefix 💜
Suppose a BIER-encapsulated packet arrives at BFR-A. From the packet's encapsulation, BFR-A learns (a) the sub-domain of the packet and (b) the BFR-ids (in that sub-domain) of the BFERs the packet is destined for. Then, using the info advertised per Section 5, BFR-A can find the BFR-prefix of each destination BFER. Given the BFR-prefix of a particular destination BFER, say BFER-D, BFR-A learns from the routing underlay (associated with the packet's sub-domain) an IP address of the BFR that is the next hop on the path from BFR-A to BFER-D. Let's call this next hop "BFR-B". BFR-A must then determine the BFR-prefix of BFR-B. (This determination can be made from the info advertised per Section 5.) This BFR-prefix is the BFR-NBR of BFR-A on the path from BFR-A to BFER-D 🦋
Note that if the routing underlay provides multiple equal-cost paths from BFR-A to BFER-D, BFR-A may have multiple BFR-NBRs for BFER-D ✨
Under certain circumstances, a BFR may have adjacencies (in a particular routing underlay) that are not BFRs. Please see Section 6.9 for a discussion of how to handle those circumstances 🫶
6.3. The Bit Index Routing Table 🗺️
okay so the "Bit Index Routing Table" (BIRT) is basically a lookup table that maps the BFR-id of a BFER (inside a particular sub-domain) to that BFER's BFR-prefix, and to the BFR-NBR on the path to it 💖 let's use Figure 1 as our example. we write each BFR-id in the SI:xyzw form from Section 3 🎀
( A ) ------------ ( B ) ------------ ( C ) ------------ ( D )
4 (0:1000) \ \ 1 (0:0001)
\ \
( E ) ( F )
3 (0:0100) 2 (0:0010)
Figure 1: BIER Topology 1this topology gives us the BIRT in Figure 2 at BFR-B 🦋 the first column shows the BFR-id as a number, plus (in parentheses) the SI:BitString version for a BitStringLength of 4. (the actual minimum BitStringLength is 64, but we're using 4 in the examples, so don't panic 😭)
Note that a BIRT is specific to a particular BIER sub-domain. 🌸
--------------------------------------------
| BFR-id | BFR-Prefix | BFR-NBR |
| (SI:BitString) | of Dest BFER | |
============================================
| 4 (0:1000) | A | A |
--------------------------------------------
| 1 (0:0001) | D | C |
--------------------------------------------
| 3 (0:0100) | E | E |
--------------------------------------------
| 2 (0:0010) | F | C |
--------------------------------------------
Figure 2: Bit Index Routing Table at BFR-B6.4. The Bit Index Forwarding Table 🧚
the "Bit Index Forwarding Table" (BIFT) is built from the BIRT, and it works like this 💅 (heads up, a BIFT is also specific to one sub-domain)
say a few rows in the BIRT share the same SI and the same BFR-NBR. if you logical OR their BitStrings together, you get a bit mask for that combo. we call it the "Forwarding Bit Mask" (F-BM) for that <SI, BFR-NBR> pair ✨
so in Figure 2, two rows have SI (0) and BFR-NBR (C). the bit mask for <SI=0, BFR-NBR-C> is 0011, because "0001" OR'd with "0010" gives you that 🙈
the BIFT is what maps a BFER's BFR-id to its F-BM and BFR-NBR. Figure 3 is the BIFT built from Figure 2. notice that BFR-ids 1 and 2 share an SI and a BFR-NBR, so they get the same F-BM 🌈
-------------------------------------
| BFR-id | F-BM | BFR-NBR |
| (SI:BitString) | | |
=====================================
| 1 (0:0001) | 0011 | C |
-------------------------------------
| 2 (0:0010) | 0011 | C |
-------------------------------------
| 3 (0:0100) | 0100 | E |
-------------------------------------
| 4 (0:1000) | 1000 | A |
-------------------------------------
Figure 3: Bit Index Forwarding Tablethis BIFT gets programmed into the data plane and is used to forward packets, following the rules in Section 6.5 💖
6.5. The BIER Forwarding Procedure 🚀
here's the step-by-step a BFR follows to forward a BIER-encapsulated packet. bestie, read this one slowly 🫶
- Determine the packet's SI, BitStringLength, and sub-domain.
- If the BitString consists entirely of zeroes, discard the packet; the forwarding process has been completed. Otherwise, proceed to step 3.
- Find the position (call it "k") of the least significant (i.e., of the rightmost) bit that is set in the packet's BitString. (Remember, bits are numbered from 1, starting with the least significant bit.)
- If bit k identifies the BFR itself, copy the packet, and send the copy to the multicast flow overlay. Then clear bit k in the original packet, and go to step 2. Otherwise, proceed to step 5.
- Use the value k, together with the SI, sub-domain, and BitStringLength, as the "index" into the BIFT.
- Extract from the BIFT the F-BM and the BFR-NBR.
- Copy the packet. Update the copy's BitString by ANDing it with the F-BM (i.e.,
PacketCopy->BitString &= F-BM). Then forward the copy to the BFR-NBR. (If the BFR-NBR is null, the copy is just discarded.) Note that when a packet is forwarded to a particular BFR-NBR, its BitString identifies only those BFERs that are to be reached via that BFR-NBR. 🦋 - Now update the original packet's BitString by ANDing it with the INVERSE of the F-BM (i.e.,
Packet->BitString &= ~F-BM). (This clears the bits that identify the BFERs to which a copy of the packet has just been forwarded.) Go to step 2. ✨
this procedure means the packet goes to any given BFR-NBR only once 👑 the number of BIFT lookups equals the number of BFR-NBRs the packet has to reach, so you don't need a separate lookup for every destination BFER. iconic efficiency, ngl 💅
when a packet heads to a particular BFR-NBR, the BitString isn't the only part of the BIER header that changes. if there's a TTL field in the BIER header, it needs to be decremented. also, when either encapsulation from [MPLS_BIER_ENCAPS] is in use, the BIFT-id field is likely to need changing, based on what the BFR-NBR signaled. the BIFT-id field of an incoming BIER packet implicitly identifies an SI, a sub-domain, and a BitStringLength. if the packet goes to a particular BFR-NBR, the BIFT-id field must be changed to whatever value that BFR-NBR advertised for the same SI, sub-domain, and BitStringLength. (if you're using the encapsulation from Section 2.1 of [MPLS_BIER_ENCAPS], this is basically an MPLS label swap operation 🧚)
now, suppose the rules above say to send a packet to a particular BFR-NBR. if that BFR-NBR is connected through multiple parallel interfaces, some form of load balancing might be a good idea 🌸 load-balancing algorithms are outside the scope of this document, though. BUT if the packet's encapsulation has an entropy field, that entropy field SHOULD be respected, and two packets with the same entropy field value SHOULD be sent on the same interface (if possible). 🫶
sometimes the routing underlay gives multiple equal-cost paths (through different BFR-NBRs) to the same BFER. this is called "Equal-Cost Multipath" (ECMP). the procedures in this section must be augmented to support load balancing over ECMP, and those augmentations are in Section 6.7 💜
if unicast traffic to the BFR-NBR is going through some kind of "bypass tunnel", the BIER-encapsulated multicast traffic sent to that BFR-NBR SHOULD also go through that tunnel. this lets existing "fast reroute" schemes cover multicast traffic too, not just unicast 🔥
some examples of these forwarding procedures are in Section 6.6 🎀
the rules in this section can be written as the following pseudocode 🤓
void ForwardBitMaskPacket (Packet)
{
SI=GetPacketSI(Packet);
Offset=SI*BitStringLength;
for (Index = GetFirstBitPosition(Packet->BitString); Index ;
Index = GetNextBitPosition(Packet->BitString, Index)) {
F-BM = BIFT[Index+Offset]->F-BM;
if (!F-BM) continue;
BFR-NBR = BIFT[Index+Offset]->BFR-NBR;
PacketCopy = Copy(Packet);
PacketCopy->BitString &= F-BM;
PacketSend(PacketCopy, BFR-NBR);
Packet->BitString &= ~F-BM;
}
}
Figure 4: Pseudocodethis pseudocode assumes that, at a given BFER, the BFR-NBR entry matching the BFER's own BFR-id will be the BFER's own BFR-prefix. it also assumes that the matching F-BM has only one bit set, the bit for the BFER itself. in that case, the "PacketSend" function sends the packet to the multicast flow overlay 🌈
the pseudocode also assumes that the F-BM for the null next hop has a 1 in a given bit position if and only if that bit position is for either an unused BFR-id or an unreachable BFER. when the BFR-NBR is null, the "PacketSend" function discards the packet 🙈
6.6. Examples of BIER Forwarding 🎀
okay, time for two examples of BIER forwarding, both using the Figure 1 topology 🦋 in these examples, every packet is assigned to the default sub-domain, every packet has SI=0, and the BitStringLength is 4. Figure 5 only shows the BIFT entries for SI=0. to keep things compact, the BFR-id in the first column of the BIFT is only shown as an integer ✨
BFR-A BIFT BFR-B BIFT BFR-C BIFT
------------------- ------------------- -------------------
| Id | F-BM | NBR | | Id | F-BM | NBR | | Id | F-BM | NBR |
=================== =================== ===================
| 1 | 0111 | B | | 1 | 0011 | C | | 1 | 0001 | D |
------------------- ------------------- -------------------
| 2 | 0111 | B | | 2 | 0011 | C | | 2 | 0010 | F |
------------------- ------------------- -------------------
| 3 | 0111 | B | | 3 | 0100 | E | | 3 | 1100 | B |
------------------- ------------------- -------------------
| 4 | 1000 | A | | 4 | 1000 | A | | 4 | 1100 | B |
------------------- ------------------- -------------------
Figure 5: BIFTs Used in the Forwarding Examples6.6.1. Example 1 🦋
okay bestie, BFR-D, BFR-E, and BFR-F are BFERs 💖 and BFR-A is the BFIR. Say BFIR-A has learned from the multicast flow overlay that BFER-D is interested in a given multicast flow. If BFIR-A gets a packet of that flow from outside the BIER domain, it applies the BIER encapsulation to the packet 🎀. The encapsulation must be such that the SI is zero. The encapsulation also includes a BitString with just bit 1 set and all other bits clear (i.e., 0001). That tells us BFER-D is the only BFER that needs to receive the packet 🫶. BFIR-A then follows the procedures of Section 6.5, like this:
- Since the packet's BitString is
0001, BFIR-A finds that the first bit in the string is bit 1. Looking at entry 1 in its BIFT, BFR-A determines that the bit mask F-BM is0111and the BFR-NBR is BFR-B. - BFR-A then makes a copy of the packet and applies the F-BM to the copy:
Copy->BitString &= 0111. The copy's BitString is now0001(0001 & 0111). - The copy is now sent to BFR-B.
- BFR-A then updates the packet's BitString by applying the inverse of the F-BM:
Packet->BitString &= ~F-BM. As a result, the packet's BitString is now0000(0001 & 1000). - As the packet's BitString is now zero, the forwarding procedure is complete.
When BFR-B receives the multicast packet from BFR-A, it follows the same procedure. The result is that a copy of the packet, with a BitString of 0001, is sent to BFR-C. BFR-C applies the same procedures and, as a result, sends a copy of the packet, with a BitString of 0001, to BFR-D.
At BFER-D, the BIFT entry (not pictured) for BFR-id 1 will specify an F-BM of 0001 and a BFR-NBR of BFR-D itself. This will cause a copy of the packet to be delivered to the multicast flow overlay at BFR-D. The packet's BitString will be set to 0000, and the packet will not be forwarded any further. ✨
6.6.2. Example 2 💜
this one is giving similar to example 1, except now BFIR-A has learned from the multicast flow overlay that both BFER-D and BFER-E are interested in a given multicast flow 👑. If BFIR-A receives a packet of that flow from outside the BIER domain, it applies the BIER encapsulation to the packet. The encapsulation must be such that the SI is zero. The encapsulation also includes a BitString with two bits set: bit 1 is set (as in example 1) to indicate that BFR-D is a BFER for this packet, and bit 3 is set to indicate that BFR-E is a BFER for this packet. That means the BitString (assuming again a BitStringLength of 4) is 0101. To forward the packet, BFIR-A follows the procedures of Section 6.5, like so:
- Since the packet's BitString is
0101, BFIR-A finds that the first bit in the string is bit 1. Looking at entry 1 in its BIFT, BFR-A determines that the bit mask F-BM is0111and the BFR-NBR is BFR-B. - BFR-A then makes a copy of the packet and applies the F-BM to the copy:
Copy->BitString &= 0111. The copy's BitString is now0101(0101 & 0111). - The copy is now sent to BFR-B.
- BFR-A then updates the packet's BitString by applying the inverse of the F-BM:
Packet->BitString &= ~F-BM. As a result, the packet's BitString is now0000(0101 & 1000). - As the packet's BitString is now zero, the forwarding procedure is complete.
When BFR-B receives the multicast packet from BFR-A, it follows the procedure of Section 6.5, like this:
- Since the packet's BitString is
0101, BFR-B finds that the first bit in the string is bit 1. Looking at entry 1 in its BIFT, BFR-B determines that the bit mask F-BM is0011and the BFR-NBR is BFR-C. - BFR-B then makes a copy of the packet and applies the F-BM to the copy:
Copy->BitString &= 0011. The copy's BitString is now0001(0101 & 0011). - The copy is now sent to BFR-C.
- BFR-B then updates the packet's BitString by applying the inverse of the F-BM:
Packet->BitString &= ~F-BM. As a result, the packet's BitString is now0100(0101 & 1100). - Now BFR-B finds the next bit in the packet's (modified) BitString. This is bit 3. Looking at entry 3 in its BIFT, BFR-B determines that the F-BM is
0100and the BFR-NBR is BFR-E. - BFR-B then makes a copy of the packet and applies the F-BM to the copy:
Copy->BitString &= 0100. The copy's BitString is now0100(0100 & 0100). - The copy is now sent to BFR-E.
- BFR-B then updates the packet's BitString by applying the inverse of the F-BM:
Packet->BitString &= ~F-BM. As a result, the packet's BitString is now0000(0100 & 1011). - As the packet's BitString is now zero, the forwarding procedure is complete.
So BFR-B forwards two copies of the packet 🙈. One copy, with BitString 0001, has now been sent from BFR-B to BFR-C. Following the same procedures, BFR-C will forward the packet to BFER-D.
At BFER-D, the BIFT entry (not pictured) for BFR-id 1 will specify an F-BM of 0001 and a BFR-NBR of BFR-D itself. This will cause a copy of the packet to be delivered to the multicast flow overlay at BFR-D. The packet's BitString will be set to 0000, and the packet will not be forwarded any further.
The other copy of the packet has been sent from BFR-B to BFER-E, with BitString 0100.
At BFER-E, the BIFT entry (not pictured) for BFR-id 3 will specify an F-BM of 0100 and a BFR-NBR of BFR-E itself. This will cause a copy of the packet to be delivered to the multicast flow overlay at BFR-E. The packet's BitString will be set to 0000, and the packet will not be forwarded any further. 🌈
6.7. Equal-Cost Multipath Forwarding 🌸
okay so in lots of networks, the routing underlay gives you multiple equal-cost paths from a given BFR to a given BFER 🧚. When you're forwarding multicast packets through the network, it can be super beneficial to take advantage of that by load-balancing across those paths. This feature is known as "Equal-Cost Multipath (ECMP) forwarding" 💅.
BIER supports ECMP forwarding, but the procedures of Section 6.5 must be modified slightly. Two ECMP procedures are defined. In the first (described in Section 6.7.1), the choice among equal-cost paths taken by a given packet from a given BFR to a given BFER depends on (a) routing, (b) the packet's entropy, and (c) the other BFERs to which that packet is destined. In the second (described in Section 6.7.2), the choice depends only upon the packet's entropy.
there are trade-offs between the two forwarding procedures here 🍓. In the procedure of Section 6.7.1, the number of packet replications is minimized. The procedure in Section 6.7.1 also uses less memory in the BFR. In the procedure of Section 6.7.2, the path traveled by a given packet from a given BFR to a given BFER is independent of the other BFERs to which the packet is destined. While the procedures of Section 6.7.2 may cause more replications, they provide a more predictable behavior.
The two procedures described here operate on identical packet formats and will interoperate correctly. However, if deterministic behavior is desired, then all BFRs would need to use the procedure from Section 6.7.2. 🦋
6.7.1. Non-deterministic ECMP ⭐
Figure 6 shows the operation of non-deterministic ECMP in BIER 💀.
BFR-A BIFT BFR-B BIFT BFR-C BIFT
------------------- ------------------- -------------------
| Id | F-BM | NBR | | Id | F-BM | NBR | | Id | F-BM | NBR |
=================== =================== ===================
| 1 | 0111 | B | | 1 | 0011 | C | | 1 | 0001 | D |
------------------- ------------------- -------------------
| 2 | 0111 | B | | 2 | 0011 | C | | 2 | 0010 | F |
------------------- | | 0110 | E | -------------------
| 3 | 0111 | B | ------------------- | 3 | 1100 | B |
------------------- | 3 | 0110 | E | -------------------
| 4 | 1000 | A | ------------------| | 4 | 1100 | B |
------------------- | 4 | 1000 | A | -------------------
-------------------
( A ) ------------ ( B ) ------------ ( C ) ------------ ( D )
4 (0:1000) \ \ 1 (0:0001)
\ \
( E ) ------------ ( F )
3 (0:0100) 2 (0:0010)
Figure 6: Example of Non-deterministic ECMPIn this example, BFR-B has two equal-cost paths to reach BFER-F: one via BFR-C and one via BFR-E. Since the BFR-id of BFER-F is 2, this is reflected in entry 2 of BFR-B's BIFT. Entry 2 shows that BFR-B has a choice of two BFR-NBRs for BFER-B and that a different F-BM is associated with each choice. When BFR-B looks up entry 2 in the BIFT, it can choose either BFR-NBR. However, when following the procedures of Section 6.5, it MUST use the F-BM corresponding to the BFR-NBR that it chooses.
How the choice is made is an implementation matter 🙌. However, the usual rules for ECMP apply: packets of a given flow SHOULD NOT be split among two paths, and any entropy field in the packet's encapsulation SHOULD be respected.
Note, however, that by the rules of Section 6.5, any packet destined for both BFER-D and BFER-F will be sent via BFR-C. 👑
6.7.2. Deterministic ECMP 🦋✨
okay so bestie, remember the ECMP stuff from Section 6.7.1? 💅 this part is about a sneaky problem that shows up when you use it, and the fix. it's giving plot twist 😭
Where ECMP paths exist, the path a packet takes to reach any particular BFER depends not only on routing and on the packet's entropy, but also on the set of other BFERs the packet is headed to too 🌸 which is wild, right??
Let's run a scenario on the network of Figure 6 🎀
- BFR-A is transmitting a sequence of packets. Some are destined for both D and F, and some are destined only for F.
- Every packet in this sequence has the same entropy value. Let's call it "Q" 💜
- At BFR-B, when a packet with entropy Q gets forwarded via entry 2 in the BIFT, it goes to E.
So using the forwarding procedure from Section 6.7.1, the packets headed to both D and F get forwarded by entry 1 in the BIFT, which means they reach F via the path A-B-C-F. But the packets headed only to F get forwarded by entry 2, so they reach F via A-B-E-F 🦋 same entropy, same destination F, different paths!! that's the drama 😭
Now here's the thing. The procedure minimizes how many packets BFR-B has to send out, and that's good 👑 but check out this timeline:
- Starting at time t0, the flow only needs to be received by BFER-F.
- Starting later at t1, the flow needs to be received by both BFER-D and BFER-F.
- Starting later still at t2, D no longer needs the flow, but F still does.
So from t0 to t1, the flow goes to F via A-B-E-F. From t1 to t2, it goes to F via A-B-C-F. And from t2 on, it goes back to A-B-E-F again 🌈 F's path keeps flip-flopping just because D is joining and leaving!!
Why does this matter? ngl it's a lot 💀
- F could get packets out of order 🙈
- Troubleshooting becomes a nightmare. Say there's a problem on the E-F link. F gets good service while the flow is also going to D (because it avoids E-F), but bad service when it isn't. And since it's hard to know which path is in use at any given moment, it's super hard to figure out what's broken 😭
- It's also very difficult to run a traceroute that's guaranteed to follow the path the flow is actually taking at that moment 💔
The root of all this is that, under Section 6.7.1, the path a flow takes to a given BFER depends on whether any lower-numbered BFERs are also receiving that flow. So the choice among the ECMP paths is fundamentally non-deterministic 🫶
The fix is deterministic forwarding ✨
You can get deterministic forwarding by using multiple BIFTs. Each row in a BIFT has only one path to each destination, but the multiple ECMP paths to any given destination get spread across the different tables 🎀 When a BIER-encapsulated packet shows up to be forwarded, the BFR hashes the BIER entropy field to pick which BIFT to use. After that, the normal BIER forwarding algorithm (from Sections 6.5 and 6.6) runs using the chosen BIFT 💖
Here's an example. Say there are two paths to destination X (X1 and X2) and four paths to destination Y (Y1, Y2, Y3, and Y4). If there are four BIFTs, one could hold X1 and Y1, another X1 and Y2, another X2 and Y3, and the last X2 and Y4. If traffic to X gets split evenly across these four BIFTs, it gets split evenly between the two X paths. And if traffic to Y gets split evenly across them, it gets split evenly across the four Y paths 🌸 so cute, right??
Quick heads up: if one destination has three paths and another has four, you'd need 12 BIFTs to split the load evenly across both of them 😭 BIFTs take up memory, though, so someone might be fine with less-than-perfect splitting to keep the table count down. How to make that trade-off is an implementation or deployment decision 🧚
6.8. Prevention of Loops and Duplicates 🔁✨
Okay, this bit is my fave because it's basically the BIER "no drama" guarantee 💅
The BitString in a BIER-encapsulated packet says which BFERs the packet should be forwarded to. When a BIER-encapsulated packet gets replicated, no two copies will ever share a BFER 🦋 And if one of the packet's BFERs forwards it further, that BFER clears its own bit first. So duplicate delivery just isn't possible with BIER 💖 bestie, that's iconic 👑
Loops are handled too 💜 As long as the routing underlay gives a loop-free path between every pair of BFRs, BIER-encapsulated packets won't loop. And since the BIER layer doesn't create any paths of its own, it doesn't need any special BIER-specific loop-prevention tricks beyond the forwarding procedures in Section 6.5 🌈
Now, if at some point the routing underlay is NOT providing a loop-free path between BFIR-A and BFER-B, then BIER-encapsulated packets may loop on their way from BFIR-A to BFER-B. But those loops will never cause BFER-B to get duplicate packets 🙈 phew!!
All of these properties mean BIER doesn't need the "Reverse Path Forwarding" (RPF) check that conventional IP multicast forwarding uses 🎀 no RPF check, no problem, periodt ✨
6.9. When Some Nodes Do Not Support BIER ✨
Okay so this one is about the messy real world, where not every router in the BIER domain is a BFR yet 🫠. Section 6.2 kind of assumes that every node next to a BFR in a given routing underlay is also a BFR. But BIER can still work even when that isn't true, as long as the ingress and egress nodes are BFRs 🦋. This section covers what to do when the routing underlay is an SPF-based IGP that computes a shortest-path tree from each node to all the others in the domain.
At a given BFR, let's say "BFR-B", you start with a copy of the IGP-computed shortest-path tree from BFR-B to each router in the domain. (The SPF algorithm of the IGP is what computes this tree 🧚.) Call that copy the "BIER-SPF tree rooted at BFR-B". Then BFR-B edits it like this:
- BFR-B looks at each of its child nodes on the BIER-SPF tree, one at a time.
- If one of the child nodes does not support BIER, BFR-B removes that node from the tree. The child nodes of the node that just got removed are then re-parented, so BFR-B is now their parent 🫶.
- BFR-B keeps looking at each of its child nodes, including any nodes that got re-parented to it because of the previous step.
Once all the child nodes have been checked (the original ones plus any new ones), BFR-B's children on the BIER-SPF tree will all be BFRs 💅.
When the BIFT gets constructed, BFR-B's child nodes on the BIER-SPF tree are treated as the BFR-NBRs. The F-BMs must be computed appropriately, based on those BFR-NBRs 🌸.
BFR-B might now have BFR-NBRs that aren't "directly connected" to it via Layer 2 😭. To send a packet to one of those, BFR-B has to send it through a unicast tunnel. In an MPLS network, this can be as simple as finding the IGP unicast next hop to the child node and pushing on (above the BIER encapsulation header) an MPLS label that the IGP next hop has bound to an address of the child node 🍓. (This assumes the packet is using an MPLS-based BIER encapsulation, like the one in Section 2.1 of [MPLS_BIER_ENCAPS].) Of course, the BIFT-id in the BIER encapsulation header must be the BIFT-id that the child node advertised for the packet's SI, sub-domain, and BitStringLength.
If for some reason the unicast tunnel can't be an MPLS tunnel, any other kind of tunnel can be used, as long as that tunnel type's encapsulation has a way to say that the payload is a BIER-encapsulated packet 🌈.
Note that if a BIER-encapsulated packet isn't using an MPLS-based BIER encapsulation, it can't be sent through an MPLS tunnel unless it's known that the tunnel only carries BIER packets. That's because MPLS has no "next protocol type" field 🙈. This isn't a problem with an MPLS-based BIER encapsulation, because then the BIER encapsulation starts with an MPLS label that identifies the packet as a BIER-encapsulated packet ✨.
Of course, the above is not meant as an implementation technique. It's just a functional description, bestie.
While the description above assumes the routing underlay provides an SPF tree, it may also be applicable to other types of routing underlays 🦋.
The technique above can also be used to provide "node protection", meaning fast reroute around nodes that are believed to have failed 🔥. If BFR-B has a failed BFR-NBR, BFR-B can remove the failed BFR-NBR from the BIER-SPF tree and then re-parent the child BFR-NBRs of the failed BFR-NBR, so they look like BFR-B's own child nodes on the tree (so they look like BFR-B's BFR-NBRs). The usual BIER forwarding procedures then apply. However, getting the packet from BFR-B to the child nodes of the failed BFR-NBR is a bit more complicated, because it may require a unicast bypass tunnel to get around the failed node 💀.
A simpler variant of step 2 above would be this:
If one of the child nodes does not support BIER, BFR-B removes that node from the tree. All BFERs that are reached through that child node are then re-parented on the tree, so that BFR-B now becomes their parent.
This variant is simpler because the set of BFERs reached through a particular child node of BFR-B can be figured out from the F-BM in the BIFT 🧁. But if you use this variant, the results are less optimal, because packets will be unicast directly from BFR-B to the BFERs that are reachable through the non-BIER child node 😤.
When using a unicast MPLS tunnel to get a packet to a BFR-NBR:
- The TTL of the MPLS label entry representing the tunnel SHOULD be set to a large value, rather than being copied from the TTL value from the BIER encapsulation header, and
- When the tunnel labels are popped off, the TTL from the tunnel labels SHOULD NOT be copied to the BIER encapsulation header.
In other words, the TTL processing for the tunnel SHOULD be as specified in [RFC3443] for "Pipe Model" and "Short Pipe Model" Label Switched Paths (LSPs) 💜. The same principle applies if the tunnels aren't MPLS tunnels: the BIER packet SHOULD NOT inherit the TTL from the tunnel encapsulation. That way, the TTL of the BIER encapsulation header only limits the number of BFRs the packet can traverse, not the total number of hops 🫧.
If two BIER packets have the same value in the entropy field of their BIER headers, and both get transmitted through a given tunnel, it's desirable for the tunnel encapsulation to preserve the fact that the two packets have the same entropy 🌷.
The material in this section assumes that if a router is a BFR, then it supports BIER on all of its interfaces. But it's possible for a router to have some line cards that support BIER and some that don't. In that case, you can think of the router as a "partial BFR" that supports BIER only on some of its interfaces 🍑. If you want to deploy partial BFRs, you can use the multi-topology features of the IGP to set up a BIER-specific topology. This topology would exclude all the non-BIER-capable interfaces that attach to BFRs. BIER would then have to run in a sub-domain that's bound to this topology 🎀. If unicast tunnels are used to bypass non-BFRs, then either (a) the tunnels have to be restricted to this topology, or (b) the tunnel endpoints have to be BFRs that don't have any non-BIER-capable interfaces 👑.
6.10. Use of Different BitStringLengths within a Domain
Heads up, this section only applies when the same encapsulation is used all through the BIER domain. Scenarios where both multiple encapsulations and multiple BitStringLengths show up in one BIER domain are out of scope for this document 🙅♀️.
It's totally possible for different BFRs in a BIER domain to be using different Imposition and/or Disposition BitStringLengths 🌈. Section 3 already says:
"if a particular BFIR is provisioned to use a particular Imposition BitStringLength and a particular Imposition sub-domain when imposing the encapsulation on a given set of packets, all other BFRs with BFR-ids in that sub-domain SHOULD be provisioned to process received BIER packets with that BitStringLength (i.e., all other BFRs with BFR-ids in that sub-domain SHOULD be provisioned with that BitStringLength as a Disposition BitStringLength for that sub-domain)."
Note that mis-provisioning can cause "black holes" 🕳️. If a BFIR creates a BIER packet with a particular BitStringLength, and that packet needs to travel through a BFR that can't process received BIER packets with that BitStringLength, then it may be impossible to forward the packet to all of the BFERs in its BIER header 😱. Section 6.10.1 defines a procedure, the "BitStringLength Compatibility Check", that can be used to detect the possibility of those black holes 🔍.
BUT (and this is important) a failed BitStringLength Compatibility Check doesn't automatically mean black holes will form 🙌. Section 6.10.2 specifies OPTIONAL procedures that let BIER forwarding keep going without black holes, even when the BitStringLength Compatibility Check fails ✨.
If the procedures of Section 6.10.2 aren't deployed, but the BitStringLength Compatibility Check fails at some BFIR, the BFIR has two choices 🫶:
- Create BIER packets with the provisioned Imposition BitStringLength, even though the packets may not be able to reach all the BFERs identified in their BitStrings.
- Use an Imposition BitStringLength that passes the Compatibility Check (assuming there is one), even if it's not the provisioned Imposition BitStringLength.
Section 6.10.1 talks about what happens if you pick one of these choices over the other 🧐.
Sometimes an operator wants to change the BitStringLengths used in a particular BIER domain 🔄. Section 6.10.3 specifies a simple procedure for transitioning a BIER domain from one BitStringLength to another 🦋.
6.10.1. BitStringLength Compatibility Check 💖
Okay so when a BFIR needs to wrap up a packet 🎀, it first picks a sub-domain for it. Then it chooses an Imposition BitStringLength L for that packet. Provisioning makes that call, but the BFIR should also run the BitStringLength Compatibility Check ✨
The combo of sub-domain S and Imposition BitStringLength L passes the check if and only if this is true:
Every BFR that has advertised its membership in sub-domain S has also advertised that it is using Disposition BitStringLength L (and possibly other BitStringLengths as well) in that sub-domain. (If MPLS encapsulation (Section 2.1 of [MPLS_BIER_ENCAPS]) is being used, this means that every BFR that is advertising a label for sub-domain S is advertising a label for the combination of sub-domain S and Disposition BitStringLength L.)
Now, if a BFIR has been provisioned with a certain Imposition BitStringLength and sub-domain for some packets, and that combo fails the check 🙈, the BFIR SHOULD log it as an error. After that, it has two choices for those packets:
- The BFIR MAY use the provisioned Imposition BitStringLength anyway. If the procedure in option 2 or option 3 of Section 6.10.2 is deployed, this won't cause black holes and might even be the optimal result. Though, the BFIR can't tell through signaling whether those procedures are deployed.
- If the BFIR can use an Imposition BitStringLength that does pass the check for that sub-domain, the BFIR MAY use that one instead.
Which of these two choices to make is also decided by provisioning 💅
Quick reminder that dropping the packets is NOT one of the allowed choices 😭 Say all the BFIRs are provisioned to use Imposition BitStringLength L for sub-domain S, but one BFR wasn't provisioned to use Disposition BitStringLength L for sub-domain S. The check fails. If the BFIRs send packets with BitStringLength L and sub-domain S, that mis-provisioned BFR can't forward them, so the packets might only reach a subset of their intended BFERs. Still, that's better than the BFIRs dropping them, because then the packets won't reach any of their BFERs at all.
If the procedures in Section 6.10.2 haven't been deployed, choice 2 might look like the better option. But a given BFIR might not have any Imposition BitStringLength that also passes the check. So if a deployment wants to use choice 2, there should be a "Fallback Disposition BitStringLength" (call it "F") such that:
- Every BFR advertises that it uses BitStringLength F as a Disposition BitStringLength for every sub-domain, and
- If a BFIR is provisioned to use Imposition BitStringLength X and Imposition sub-domain S for a certain class of packets, but the check fails for the combination of X and S, then the BFIR falls back to using BitStringLength F as the Imposition BitStringLength whenever the Imposition sub-domain is S.
It is RECOMMENDED that the value of F be the default BitStringLength for the encapsulation being used 🌸
6.10.2. Handling BitStringLength Mismatches 🦋
Let's say a packet got BIER-encapsulated with a BitStringLength value of X, and it has arrived at BFR-A. Now, according to the routing underlay, the next hop is BFR-B, but BFR-B doesn't use X as one of its Disposition BitStringLengths. Uh oh 😬 What should BFR-A do? It has three options. It MUST do one of the three, but which one it picks is a local decision 💜 The options are:
- BFR-A MAY discard the packet.
- BFR-A MAY re-encapsulate the packet, using a BIER header whose BitStringLength value is supported by BFR-B.
Note that if BFR-B only uses Disposition BitStringLength values smaller than the packet's BitStringLength value, this may mean making extra copies of the packet. Whether extra copies actually have to be made depends on the bits that are actually set in the original packet's BitString.
- BFR-A MAY treat BFR-B as if BFR-B didn't support BIER at all, in which case BFR-A applies the rules of Section 6.9.
Note that there's no signaling that lets a BFR advertise which of the three options it will use 🤷♀️
Option 2 can be really useful when part of the BIER domain has BFRs that can use a long BitStringLength, and another part has BFRs that can only handle a shorter one 🍓
6.10.3. Transitioning from One BitStringLength to Another 🧚
Say you want to migrate the BitStringLength used in a BIER domain from one value (X) to another value (Y). The migration procedure below can do that. It lets you reprovision the BFRs one at a time, so there's no "flag day" needed 🎉
- Upgrade all the BFRs in the domain so they use both value X and value Y as their Disposition BitStringLengths.
- Reprovision the BFIRs so they use BitStringLength value Y as the Imposition BitStringLength.
- You may then optionally reprovision all the BFRs so they no longer use Disposition BitStringLength X.
7. Operational Considerations 🌈
BIER is a radical simplification compared to current IP multicast operations 👑 There's no tree-building control plane, no per-flow forwarding state, no Reverse Path Forwarding (RPF), no Rendezvous Point (RP), and so on. BIER packet forwarding and replication follow the unicast paths to each bit position set in the packet. That means the encapsulated multicast packets follow the same path as unicast to each set bit in the header. The BIER FIB can be derived from the SPF-calculated unicast FIB or from any other forwarding-path calculation, in or out of band. Each bit follows this unicast path from the entry point of the BIER domain to the edge device with that assigned bit.
Because of these differences, the operational expectations from traditional multicast solutions don't apply to a BIER domain 💅 There's no granular per-flow state at each node defining a tree. Monitoring flows at the forwarding plane level (the (S,G) entries) isn't provided in a BIER node. BIER FIB packet counters may be kept for BFR-ids or next-hop neighbors. Any flow-based metrics would need deeper packet inspection, which is outside the scope of this document. So honestly, BIER is more like unicast again 🫶
This reduction in state is what gives BIER one of its biggest operational perks: deterministic convergence ✨ The BIER FIB can converge right after the unicast FIB, no matter how many multicast flows are going over the links. Careful monitoring of (S,G) utilization isn't required within a BIER domain.
7.1. Configuration 🎀
A BIER domain requires that each edge node (BFER) be given a unique bit position in the BIER mask (BFR-id). The BFR-id must be configured on each BFER and associated with a unique IP address of that BFER. Any existing manual or automated configuration tools must provide access to BIER-specific configuration. The association of the BFR-id with a unique address of the BFER it's assigned to must also be advertised into the IGP of the BIER domain. This might be implied from the BIER configuration or might need IGP-specific configuration. This document doesn't dictate any specific configuration methodology 🦋
8. IANA Considerations 🌸
This document does not require any IANA actions.
9. Security Considerations 🔥
Okay, this section is serious business, bestie 😭💀
When BIER is paired with a particular multicast flow overlay, it inherits the security considerations of that layer. Likewise, when BIER is paired with a particular routing underlay, it inherits the security considerations of that layer.
If the BIER encapsulation of a particular packet specifies an SI or a BitString other than the one the BFIR intended, the packet is likely to be misdelivered. If the BIER encapsulation of a packet gets modified (through error or malfeasance) in a way other than what this document specifies, the packet may be misdelivered. Some modifications of the BIER encapsulation, for example setting every bit in the BitString, may result in (intentional or unintentional) denial-of-service (DoS) attacks 💥
If a BFIR is compromised, it may impose a BIER encapsulation with all the bits in the BitString set. That would also result in a DoS attack.
Every BFR MUST be provisioned to know which of its interfaces lead to a BIER domain and which don't. BIER-encapsulated packets MUST NOT be accepted from outside the BIER domain. (Receiving BIER-encapsulated packets from outside the BIER domain would create an attack vector for DoS attacks, since an attacker might set all the bits in the BitString.) 🚨
If two interfaces lead to different BIER domains, the BFR MUST be provisioned to know that those two interfaces lead to different BIER domains. If the provisioning isn't correct, BIER-encapsulated packets from one BIER domain may "leak" into another, which is likely to result in misdelivery of packets 😱
DoS attacks may also result from incorrect provisioning (through error or malfeasance) of the BFRs.
If the procedures used for advertising BFR-ids and BFR-prefixes aren't secure, an attack on those procedures may result in incorrect delivery of BIER-encapsulated packets.
10. References 📚
10.1. Normative References 💜
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing
in Multi-Protocol Label Switching (MPLS) Networks",
RFC 3443, DOI 10.17487/RFC3443, January 2003,
<https://www.rfc-editor.org/info/rfc3443>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in
RFC 2119 Key Words", BCP 14, RFC 8174,
DOI 10.17487/RFC8174, May 2017,
<https://www.rfc-editor.org/info/rfc8174>.10.2. Informative References 📚
okay so these are the informative refs, meaning the extra reading that's helpful but not strictly required 🤓 Each one is listed by its label in brackets, then the author list, title, status, and date, so you can go find the full thing if you're curious ✨
[BGP_BIER_EXTENSIONS]Xu, X., Ed., Chen, M., Patel, K., Wijnands, IJ., and A. Przygienda, "BGP Extensions for BIER", Work in Progress, draft-ietf-bier-idr-extensions-03, August 2017. 🦋[BIER_MVPN]Rosen, E., Ed., Sivakumar, M., Aldrin, S., Dolganow, A., and T. Przygienda, "Multicast VPN Using BIER", Work in Progress, draft-ietf-bier-mvpn-09, November 2017. 💜[Boivie_Feldman]Boivie, R. and N. Feldman, "Small Group Multicast", Work in Progress, draft-boivie-sgm-02, February 2001. 🌸[ISIS_BIER_EXTENSIONS]Ginsberg, L., Ed., Przygienda, A., Aldrin, S., and J. Zhang, "BIER Support via ISIS", Work in Progress, draft-ietf-bier-isis-extensions-06, October 2017. 🧚[MPLS_BIER_ENCAPS]Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A., Tantsura, J., Aldrin, S., and I. Meilik, "Encapsulation for Bit Index Explicit Replication in MPLS and non-MPLS Networks", Work in Progress, draft-ietf-bier-mpls-encapsulation-12, October 2017. 🎀[OSPF_BIER_EXTENSIONS]Psenak, P., Ed., Kumar, N., Wijnands, IJ., Dolganow, A., Przygienda, T., Zhang, J., and S. Aldrin, "OSPF Extensions for BIER", Work in Progress, draft-ietf-bier-ospf-bier-extensions-09, October 2017. 🍓[RFC6513]Rosen, E., Ed., and R. Aggarwal, Ed., "Multicast in MPLS/BGP IP VPNs", RFC 6513, DOI 10.17487/RFC6513, February 2012,<https://www.rfc-editor.org/info/rfc6513>. ⭐[RFC6514]Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP Encodings and Procedures for Multicast in MPLS/BGP IP VPNs", RFC 6514, DOI 10.17487/RFC6514, February 2012,<https://www.rfc-editor.org/info/rfc6514>. 👑
Acknowledgements 🫶
okay the authors wanna give a big thank you to Rajiv Asati, Alia Atlas, John Bettink, Ross Callon (who wrote a lot of the text on deterministic ECMP, iconic 🙌), Nagendra Kumar, Christian Martin, Neale Ranns, Albert Tian, Ramji Vaithianathan, Xiaohu Xu, and Jeffrey Zhang for their ideas and contributions to this work 💖
and also a shoutout to Sue Hares, Victor Kuarsingh, and Dan Romascanu for reviewing the document ✨
Contributors 💅
these people contributed a ton to the content, so they're basically co-authors, no cap 👯♀️
Gregory Cauchie
Bouygues Telecom
Email: gcauchie@bouyguestelecom.frMach(Guoyi) Chen
Huawei
Email: mach.chen@huawei.comArkadiy Gulko
Thomson Reuters
195 Broadway
New York, NY 10007
United States of America
Email: arkadiy.gulko@thomsonreuters.comWim Henderickx
Nokia
Copernicuslaan 50
Antwerp 2018
Belgium
Email: wim.henderickx@nokia.comMartin Horneffer
Deutsche Telekom
Hammer Str. 216-226
Muenster 48153
Germany
Email: Martin.Horneffer@telekom.deLuay Jalil
Verizon
1201 East Arapaho Rd.
Richardson, TX 75081
United States of America
Email: luay.jalil@verizon.comUwe Joorde
Deutsche Telekom
Hammer Str. 216-226
Muenster D-48153
Germany
Email: Uwe.Joorde@telekom.deGreg Shepherd
Cisco Systems
170 West Tasman Drive
San Jose, CA 95134
United States of America
Email: shep@cisco.comJeff Tantsura
Email: jefftant.ietf@gmail.comAuthors' Addresses 🦋
and that's the whole crew! here are the editors and authors, with where to find them 🌈
IJsbrand Wijnands (editor)
Cisco Systems, Inc.
De Kleetlaan 6a
Diegem 1831
Belgium
Email: ice@cisco.comEric C. Rosen (editor)
Juniper Networks, Inc.
10 Technology Park Drive
Westford, Massachusetts 01886
United States of America
Email: erosen@juniper.netAndrew Dolganow
Nokia
438B Alexandra Rd #08-07/10
Alexandra Technopark
Singapore 119968
Singapore
Email: andrew.dolganow@nokia.comTony Przygienda
Juniper Networks, Inc.
1194 N. Mathilda Ave.
Sunnyvale, California 94089
United States of America
Email: prz@juniper.netSam K. Aldrin
Google, Inc.
1600 Amphitheatre Parkway
Mountain View, California 94043
United States of America
Email: aldrin.ietf@gmail.com📦 where this page comes from
- ietf: rfc8279 2017-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms