RFC 3940
Negative-acknowledgment (NACK)-Oriented Reliable Multicast (NORM) Protocol
Negative-acknowledgment (NACK)-Oriented Reliable Multicast (NORM) Protocol 🦋
Part 1 of 22 🎀
Okay bestie, the title block is giving RFC 3940, filed in the Network Working Group as a Request for Comments in the Experimental category, dated November 2004. It was written by four people: B. Adamson (NRL), C. Bormann (Universitaet Bremen TZI), M. Handley (UCL), and J. Macker (NRL). The title block doesn't list anything it obsoletes or updates, so there's no drama with older docs 💅
Status of this Memo ✨
This memo defines an Experimental Protocol for the Internet community. It does not specify an Internet standard of any kind. Discussion and suggestions for improvement are requested. Distribution of this memo is unlimited. 📜
Copyright Notice 🌸
Copyright (C) The Internet Society (2004).
Abstract 💖
Okay so this doc is all about the messages and procedures of the Negative-acknowledgment (NACK) Oriented Reliable Multicast (NORM) protocol 🔥 NORM is built to give you end-to-end reliable transport of bulk data objects or streams over generic IP multicast routing and forwarding services. It uses a selective negative acknowledgment mechanism for reliability, plus extra mechanisms so senders and receivers can work together with minimal "a priori" coordination 🫶
It also specifies a congestion control scheme, so NORM can fairly share bandwidth with other transport protocols like Transmission Control Protocol (TCP). That's the ultimate good-friend energy 👑 NORM can run over reciprocal multicast routing between senders and receivers, and it also works with asymmetric connectivity (possibly a unicast return path) between them. Different apps, or even other higher-level transport protocols, can use its service in different ways thanks to a bunch of features. And it leans on FEC-based repair and other IETF reliable multicast transport (RMT) building blocks in its design, which is so iconic 🧚
Table of Contents 🍓
1. Introduction and Applicability. . . . . . . . . . . . . . . . 3
1.1. NORM Delivery Service Model. . . . . . . . . . . . . . . 4
1.2. NORM Scalability . . . . . . . . . . . . . . . . . . . . 6
1.3. Environmental Requirements and Considerations. . . . . . 7
2. Architecture Definition . . . . . . . . . . . . . . . . . . . 7
2.1. Protocol Operation Overview. . . . . . . . . . . . . . . 9
2.2. Protocol Building Blocks . . . . . . . . . . . . . . . . 10
2.3. Design Tradeoffs . . . . . . . . . . . . . . . . . . . . 11
3. Conformance Statement . . . . . . . . . . . . . . . . . . . . 12
4. Message Formats . . . . . . . . . . . . . . . . . . . . . . . 13
4.1. NORM Common Message Header and Extensions. . . . . . . . 14
4.2. Sender Messages. . . . . . . . . . . . . . . . . . . . . 16
4.2.1. NORM_DATA Message . . . . . . . . . . . . . . . . 16
4.2.2. NORM_INFO Message . . . . . . . . . . . . . . . . 24
4.2.3. NORM_CMD Messages . . . . . . . . . . . . . . . . 26
4.3. Receiver Messages. . . . . . . . . . . . . . . . . . . . 43
4.3.1. NORM_NACK Message . . . . . . . . . . . . . . . . 43
4.3.2. NORM_ACK Message. . . . . . . . . . . . . . . . . 50
4.4. General Purpose Messages . . . . . . . . . . . . . . . . 52
4.4.1. NORM_REPORT Message . . . . . . . . . . . . . . . 52
5. Detailed Protocol Operation . . . . . . . . . . . . . . . . . 52
5.1. Sender Initialization and Transmission . . . . . . . . . 54
5.1.1. Object Segmentation Algorithm . . . . . . . . . . 55
5.2. Receiver Initialization and Reception. . . . . . . . . . 57
5.3. Receiver NACK Procedure. . . . . . . . . . . . . . . . . 57
5.4. Sender NACK Processing and Response. . . . . . . . . . . 59
5.4.1. Sender Repair State Aggregation . . . . . . . . . 60
5.4.2. Sender FEC Repair Transmission Strategy . . . . . 61
5.4.3. Sender NORM_CMD(SQUELCH) Generation . . . . . . . 62
5.4.4. Sender NORM_CMD(REPAIR_ADV) Generation. . . . . . 62
5.5. Additional Protocol Mechanisms . . . . . . . . . . . . . 63
5.5.1. Greatest Round-trip Time Collection . . . . . . . 63
5.5.2. NORM Congestion Control Operation . . . . . . . . 64
5.5.3. NORM Positive Acknowledgment Procedure. . . . . . 72
5.5.4. Group Size Estimate . . . . . . . . . . . . . . . 74
6. Security Considerations . . . . . . . . . . . . . . . . . . . 75
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 75
8. Suggested Use . . . . . . . . . . . . . . . . . . . . . . . . 75
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 76
10. References. . . . . . . . . . . . . . . . . . . . . . . . . . 76
10.1. Normative References. . . . . . . . . . . . . . . . . . 76
10.2. Informative References. . . . . . . . . . . . . . . . . 77
11. Authors' Addresses. . . . . . . . . . . . . . . . . . . . . . 79
Full Copyright Statement. . . . . . . . . . . . . . . . . . . 801. Introduction and Applicability ✨
Okay so NORM (the Negative-acknowledgment (NACK) Oriented Reliable Multicast protocol) gives you reliable transport of data from one or more senders to a whole group of receivers over an IP multicast network 🦋 The main goals are efficient, scalable, and robust bulk data transfer (things like computer files, or persistent data) across IP networks and topologies that might be heterogeneous. Iconic behaviour 💅
NORM is built for distributed multicast session participation with minimal coordination between senders and receivers. Senders and receivers can dynamically join and leave multicast sessions whenever they want, with minimal overhead for control information and timing synchronization among participants. To make that work, NORM protocol message headers carry some common information so receivers can easily synchronize to senders for the whole lifetime of a reliable multicast session 🫶
It's also self-adapting to a wide range of dynamic network conditions, with little or no pre-configuration. It's purposely tolerant of inaccurate timing estimations and lossy conditions, which happen a lot on mobile and wireless networks. And it's designed to converge and operate efficiently even with heavy packet loss and large queuing or transmission delays. Main character energy, literally 👑
This document is a product of the IETF RMT WG and follows the guidelines in RFC 3269 [1]. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [2]. Those are the words that carry the actual rule weight, so keep an eye on them 👀
Statement of Intent 🌈
Okay this memo has part of the definitions you need to fully specify a Reliable Multicast Transport protocol, following RFC 2357. As RFC 2357 says, using any reliable multicast protocol on the Internet requires an adequate congestion control scheme 🔥
While everyone waits for that scheme to be available, or for an existing one to be proven adequate, the Reliable Multicast Transport working group (RMT) is publishing this Request for Comments in the "Experimental" category. 🧚
It is the intent of RMT to re-submit this specification as an IETF Proposed Standard as soon as the above condition is met. 💜
Part 2 is coming next, stay tuned 🎀
1.1. NORM Delivery Service Model ✨
okay so a NORM protocol instance (NormSession) is basically a group of participants sending each other connectionless packets (IP or UDP) over a network, using addresses and host port numbers that were picked ahead of time 🎀. Usually they all share an IP multicast group address, but unicast can also tag along as a sidekick to multicast delivery 💅. With multicast, the participating NormNodes use a common IP multicast group address and port number that got chosen outside the context of the NormSession, which is very mysterious of them 🙈. There are other IETF data format and protocol standards that can describe and pass along that required "a priori" info for a NormSession, like Session Description Protocol (SDP) [7], Session Announcement Protocol (SAP) [8], etc. 🧚
NORM is mostly designed around one sender blasting bulk data to a whole group of receivers 🔥. But the protocol MAY operate with multiple senders inside a single NormSession. In early implementations, the senders are expected to transmit independently of one another, and receivers keep whatever state they need for each sender 💜. In future versions of NORM, though, some parts of the protocol (like round-trip time collection) might get alternate modes that work better for apps with multiple senders ⭐.
NORM gives you three types of bulk data content objects (NormObjects) to send reliably 🎁. They are:
- static computer memory data content (NORM_OBJECT_DATA type),
- computer storage files (NORM_OBJECT_FILE type), and
- non-finite streams of continuous data content (NORM_OBJECT_STREAM type).
NORM_OBJECT_DATA and NORM_OBJECT_FILE are basically the same thing, with one little difference: it's just a "hint" so receivers in sessions with mixed content know whether to set aside memory or file storage for what they get 🧠. Other than that, both give you reliable transport of finite (but potentially huge) chunks of content 🐳. These static data and file services are expected to be super useful for multicast-based cache apps that need to reliably push lots of static data around 🌸. Other static data or file delivery services might use these object types too 🦋. NORM_OBJECT_STREAM is up to the app, and it can carry static data or files as well 💖. The NORM reliable stream service also opens doors for things like serialized reliable messaging or other unbounded, maybe dynamically produced content 👑. NORM_OBJECT_STREAM gives reliable transport similar to TCP, except NORM receivers can start receiving stream content at any point in time, which is iconic but only useful depending on the app 🙌.
NORM also lets a small amount of "out-of-band" data (sent as NORM_INFO messages) ride along with the data content objects the sender transmits 📦. This ready-to-go out-of-band data helps multicast receivers figure out quickly and efficiently what kind of bulk content (data, file, or stream) is coming in 🔍. It also lets the app control whether a receiver node keeps participating in the current transport activity 🎛️. And it keeps things flexible with minimal pre-coordination between senders and receivers 🫶. The NORM_INFO content is designed to be atomic, meaning its size MUST fit into the payload portion of a single NORM message 🚨.
NORM does not provide for global or application-level identification of data content within in its message headers 🙈. The NORM_INFO out-of-band mechanism could be used by the app for that if it wants, or the identification could be embedded in the data content itself 💅. What NORM does do is identify transmitted content (NormObjects) with transport identifiers that only matter while the sender is transmitting and/or repairing that object 🎯. These transport data content identifiers (NormTransportIds) get assigned in a monotonically increasing order by each NORM sender during a NormSession 📈. Each sender keeps its own NormTransportId assignments, so each NormObject can be uniquely identified during transport by combining the sender's session-unique identifier (NormNodeId) with its assigned NormTransportId 🪪. The NormTransportIds come from a large but fixed numeric space, are assigned in increasing order, and may be reassigned during long-lived sessions 🔄. NORM also has mechanisms so the sender app can stop sending data content and tell the group about it efficiently 🛑. Other protocol control mechanisms (like session termination, receiver synchronization, etc.) are specified too, so reliable multicast app variants can build different, complete bulk transfer models for their goals 🌈.
To sum it all up: NORM gives reliable transport for different types of data content, including mixed types, all at once 💫. Senders queue up and transmit bulk content as static data or files and/or ongoing non-finite streams 📤. NORM senders also send repair transmissions of data and/or FEC content in response to NACK messages from the receiver group 🔁. The "out-of-band" info and other transport control mechanisms are there for apps to build complete reliable multicast solutions for all kinds of purposes 💜.
1.2. NORM Scalability 📊
okay, scalability is the whole reason group communication needs NACK-based protocol schemes when reliability feedback is required [9] 🥹. NORM is built around selective NACKs that request repairs of missing data 🎯. It also uses packet-level forward error correction (FEC) for efficient multicast repair and optional proactive transmission robustness [10] 🛡️. FEC-based repair can seriously cut down on how many reliable multicast repair requests and repair transmissions [11] happen in a NACK-oriented protocol ✂️. The biggest factor in NORM's scalability is how much feedback traffic the receiver set generates to keep reliability and congestion control working 📣. NORM uses probabilistic suppression of redundant feedback, based on exponentially distributed random backoff timers 🎲. How well this suppression works compared to other techniques is described in [12] 📖. NORM also measures the group's roundtrip timing on the fly to set its suppression and other protocol timers ⏱️. That's how it scales well while keeping reliable delivery with low latency, relative to the network topology it's running over 🚀.
Feedback messages can go to the whole group via multicast or get sent to the sender via unicast routing 💌. With unicast feedback, the sender "advertises" the feedback state to the group so suppression still works 📢. In typical Internet environments, NORM is expected to scale easily to group sizes in the tens of thousands of receivers 🌍. A study of how much feedback this kind of protocol generates is in [13] 📊. NORM can run on less feedback than a single TCP connection, even with a lot of receivers 🤯. So depending on the network topology, NORM might scale to even bigger group sizes 🌟. As for computer resource usage, NORM does not require state to be kept on every receiver in the group 🙅♀️. NORM senders only maintain state for receivers that are giving explicit congestion control feedback 🧾. NORM receivers, on the other hand, must maintain state for each active sender 🫠. That might limit the number of simultaneous senders in some NORM uses 🚧.
1.3. Environmental Requirements and Considerations 🌿
okay, the RMT NORM Building Block [4] and the RMT FEC Building Block [5] have their own environmental requirements and considerations, and all of them apply to NORM too 📋.
The NORM protocol SHALL be capable of working end-to-end with no help from intermediate systems beyond basic IP multicast group management, routing, and forwarding services 🛣️. NORM's techniques mostly fit "flat" end-to-end IP multicast topologies, but they can also be used in the sub-levels of hierarchical (e.g., tree-based) multicast distribution if you want 🌳. NORM can use reciprocal multicast communication (among senders and receivers) under the Any-Source Multicast (ASM) model defined in RFC 1112 [3] 🔁. But it SHALL also be capable of scalable operation in asymmetric topologies like Source Specific Multicast (SSM) [14], where there might only be unicast routing from the receivers back to the sender(s) 🪞.
NORM works with both IPv4 and IPv6 🌐. It can also work through networks using Network Address Translation (NAT), as long as the NAT device supports IP multicast and/or can cache UDP traffic source port numbers so it can remap feedback traffic from receivers to the sender(s) 🧚♀️💖.
2. Architecture Definition ✨
okay so a NormSession is basically a group chat of participants called NormNodes 💬, and each one can be a sender, a receiver, or both 💖. NORM senders push out their data as NormObjects to the session destination address 🚀, and NORM receivers try to reliably catch everything, using negative acknowledgments to ask for repairs when stuff goes missing 🩹. Every NormNode is assumed to already have a preselected unique 32-bit identifier, called the NormNodeId 🆔. Each NormNode MUST have a uniquely assigned identifier within a single NormSession, so you can tell apart multiple senders and figure out which receiver sent which feedback 💅. There are two reserved NormNodeId values: 0x00000000 is treated as invalid, and 0xffffffff is the "wildcard" NormNodeId 🙈. The protocol doesn't stop multiple senders from transmitting at the same time in one NORM session (many-to-many, babe 👯). But any real interactive coordination between senders is up to the application or a higher protocol layer 🧑🤝🧑. This document does define some optional mechanisms that can be used for that application-layer coordination ✨.
As mentioned before, NORM can reliably send three basic types of data content 🎀. The first is NORM_OBJECT_DATA, which is for static, persistent blocks of data sitting in the sender's application memory 🧠. The second is NORM_OBJECT_FILE, which covers data stored in the sender's non-volatile file system 💾. Both NORM_OBJECT_DATA and NORM_OBJECT_FILE are "NormObjects" of a finite size, though it can be really, really large 📦. The third is NORM_OBJECT_STREAM, which is an ongoing transmission with no defined length 🌊. It's kind of like the reliable stream service TCP gives you for unicast, and the format is up to the application. It can be byte oriented or message oriented 🧃. NORM lets you "flush" the stream to speed up delivery or to enforce application message boundaries 🚿. Implementations can offer in-order delivery of the stream to the receiving app, or out-of-order delivery of received segments, which is more immediate 🏃♀️. Either way, NORM senders and receivers do buffering so the stream can be repaired while it's being transported 🔁.
All NormObjects get logically split into FEC coding blocks and symbols before the sender transmits them 🧩. In NORM, an FEC encoding symbol maps directly onto the payload of a NORM_DATA message, which is also called a "segment" 📨. When systematic FEC codes are used, the NORM_DATA payloads for the first part of a FEC encoding block are source symbols, meaning actual chunks of the original user data 🥹. The rest of the symbols in that block are parity symbols made by FEC encoding 🛡️. Parity symbols are usually sent in response to repair requests, but some of them might be sent proactively at the end of each encoding block to make transmission more robust 💪. When non-systematic FEC codes are used, every symbol sent is FEC encoding parity content. In that case, the receiver must receive enough symbols to reconstruct the original user data for that block through FEC decoding 🧮. In this document, "symbol" and "segment" mean the same thing 🫶.
Transmitted NormObjects get temporarily but uniquely identified within the NormSession using the sender's NormNodeId, a NormInstanceId, and a temporary NormObjectTransportId 🪪. Depending on the implementation, each NORM sender can manage its own NormInstanceIds on its own, or everyone in the session can agree on a shared NormInstanceId to act as a session identifier 🤝. The NormObjectTransportId is assigned by the sender and is only valid for the NormObject's actual _transport_, meaning for as long as the sender is still transmitting it and providing repair for it ⏳. In a long-lived session the NormObjectTransportId can wrap around, so old identifiers can get reused 🔄. NORM does not provide globally unique identification of transported data, so if you need that, the NORM application has to handle it 🙅♀️. Each segment or symbol of a NormObject is then identified by its FEC payload identifier, which includes the coding block and symbol identifiers. More on that later in this document 🔜.
2.1. Protocol Operation Overview
okay this is the main flow 🌈. A NORM sender mostly generates NORM_DATA messages. These carry original data segments or FEC symbols, plus repair segments or symbols, for the bulk data, file, or stream NormObjects being transferred 📤. By default, the redundant FEC symbols are only sent in response to receiver repair requests (NACKs), so normally there's little to no extra transmission overhead from FEC encoding 🙌. The NORM implementation MAY be optionally configured to send some redundant FEC symbols proactively, along with the original content 🎁. That can help performance, like improving delay, but it costs extra transmission overhead 💸. This option can make sense for certain network conditions, and it can support robust, asymmetric multicast (for example unidirectional routing, satellite, or cable) [15] with less receiver feedback, or in some cases no feedback at all 🛰️.
Senders also generate NORM_INFO messages. These carry OPTIONAL "out-of-band" context information for a given transport object 🗂️. A single NORM_INFO message can be associated with one NormObject. Since it's atomic, missing NORM_INFO messages can be NACKed and repaired with a slightly lower delay than NORM's general FEC-encoded data 🏎️. NORM_INFO can be especially helpful for bulk transfer and reliable multicast apps where receivers join the group mid-stream and need context about what's currently being sent 🙋♀️. The NACK process for NORM_INFO is explained later on 📖. When the NORM_INFO message type is used, it should be sent before any NORM_DATA message for the associated NormObject ⏩.
The sender also generates NORM_CMD messages to help with protocol operations like congestion control, end-of-transmission flushing, round trip time estimation, receiver synchronization, and optional positive acknowledgment requests or application-defined commands 🛠️. A sender can transmit a NORM_CMD message in one of three ways 🎯. The first is a single, best-effort, unreliable transmission of the command. The second is repeated, redundant transmissions of the command. The third is positively-acknowledged commands ✅. Which technique gets used depends on what the command does 🤔. Several core commands are defined for basic protocol operation 🧱. Implementations MAY also want to consider providing OPTIONAL application-defined commands that take advantage of those transmission methods 🌟. That lets application-level session management make use of stuff the NORM protocol engine already knows, like round-trip timing and transmission rate 📊.
NORM receivers generate NORM_NACK or NORM_ACK messages in response to data and commands from a sender 💌. NORM_NACK messages ask for repair of detected data transmission losses 🙏. Receivers usually spot losses by tracking the sequence of transmissions from a sender, and sequencing info is embedded in the transmitted data packets and in the end-of-transmission commands 🕵️♀️. NORM_ACK messages are generated in response to certain commands from the sender. In the general, and most scalable, protocol mode, NORM_ACK messages are only sent in response to congestion control commands 📈. The feedback volume of these congestion control NORM_ACK messages is controlled with the same timer-based probabilistic suppression used for NORM_NACK messages, which avoids feedback implosion 🧯. To meet possible application needs for positive acknowledgment from receivers, other NORM_ACK messages are defined and available to use ✔️. All sender and receiver transmissions are subject to rate control, governed by a peak transmission rate that the application sets for each participant 🚦. This can be used to limit how much multicast data the group sends 🫧. When NORM's congestion control algorithm is enabled, the sender's rate gets adjusted automatically 🎚️. In some networks, it may be desirable to set minimum and maximum bounds for that rate adjustment, depending on the application, even when dynamic congestion control is on 📏. But on the general Internet, congestion control policy SHALL be observed that is compatible with coexisting TCP flows 🧡. That one's non-negotiable, bestie 🫶.
2.2. Protocol Building Blocks
NORM mostly runs on the concepts in the Nack-Oriented Reliable Multicast (NORM) Building Block document [4] 🧱. That covers the basic NORM architecture and the data transmission, repair, and feedback strategies from that document 🪄. More reliable multicast building blocks get applied to build the full NORM protocol instantiation [16] 🏗️. NORM also uses Forward Error Correction encoding for repair messaging and optional transmission robustness, as described in [10] 🔐. NORM uses the FEC Payload ID as specified in the FEC Building Block Document [5] 🆗. And for congestion control, this document includes a baseline mechanism (NORM-CC) based on the TCP-Friendly Multicast Congestion Control (TFMCC) scheme described in [19] 🚗💨.
2.3. Design Tradeoffs ✨
Okay so, the big lesson here is that there is no "one size fits all" in reliable multicast transport, bestie 🫶 Designing these things is full of engineering tradeoffs, so you have to be aware of your application and network architecture. The performance requirements that can affect the design include group size, heterogeneity (like capacity and/or delay), asymmetric delivery, data ordering, delivery delay, group dynamics, mobility, congestion control, and transport across low capacity connections. NORM has a bunch of parameters to handle lots of these different needs 💅 The NORM protocol and its mechanisms MAY be applied in multicast applications outside of bulk data transfer, but the trade-offs that set the scalability and performance in this document are built around an assumed model of bulk transfer transport service 🌸
Buffers are a whole vibe too. How reliable NORM can be depends on the buffer constraints of the sender and receiver applications. NORM protocol implementations SHOULD be designed to operate with the greatest efficiency and robustness possible within application-defined buffer constraints 🎀 Buffer requirements for reliability are, as always, a function of the delay-bandwidth product of the network topology. NORM performs best when it gets more buffering resources than typical point-to-point transport protocols do, and that's because NORM feedback suppression is based on randomly-delayed transmissions from the receiver set, not on feedback that gets sent immediately ⭐ There are definitive tradeoffs between buffer utilization, group size scalability, and efficiency of performance. Big buffers let NORM perform most efficiently in large delay-bandwidth topologies and allow for longer feedback suppression backoff timeouts, which gives you improved group size scalability 👑 NORM can operate with reduced buffering, but it costs you some efficiency (lower relative goodput) and some group size scalability 🥺
3. Conformance Statement 🧚
This Protocol Instantiation document, together with the RMT Building Block documents of [4] and [5], completely specifies a working reliable multicast transport protocol that conforms to the requirements described in RFC 2357 [17]. Iconic 🦋
This document specifies the following message types and mechanisms, which are REQUIRED in complying NORM protocol implementations. Here's the must-have crew 👑
+--------------------+-----------------------------------------------+
| Message Type | Purpose |
+--------------------+-----------------------------------------------+
|NORM_DATA | Sender message for application data |
| | transmission. Implementations must support |
| | at least one of the NORM_OBJECT_DATA, |
| | NORM_OBJECT_FILE, or NORM_OBJECT_STREAM |
| | delivery services. The use of the NORM FEC |
| | Object Transmission Information header |
| | extension is OPTIONAL with NORM_DATA |
| | messages. |
+--------------------+-----------------------------------------------+
|NORM_CMD(FLUSH) | Sender command to excite receivers for repair |
| | requests in lieu of ongoing NORM_DATA |
| | transmissions. Note the use of the |
| | NORM_CMD(FLUSH) for positive acknowledgment |
| | of data receipt is OPTIONAL. |
+--------------------+-----------------------------------------------+
|NORM_CMD(SQUELCH) | Sender command to advertise its current valid |
| | repair window in response to invalid requests |
| | for repair. |
+--------------------+-----------------------------------------------+
|NORM_CMD(REPAIR_ADV)| Sender command to advertise current repair |
| | (and congestion control state) to group when |
| | unicast feedback messages are detected. Used |
| | to control/suppress excessive receiver |
| | feedback in asymmetric multicast topologies. |
+--------------------+-----------------------------------------------+
|NORM_CMD(CC) | Sender command used in collection of round |
| | trip timing and congestion control status |
| | from group (this may be OPTIONAL if |
| | alternative congestion control mechanism and |
| | round trip timing collection is used). |
+--------------------+-----------------------------------------------+
|NORM_NACK | Receiver message used to request repair of |
| | missing transmitted content. |
+--------------------+-----------------------------------------------+
+--------------------+-----------------------------------------------+
|NORM_ACK | Receiver message used to proactively provide |
| | feedback for congestion control purposes. |
| | Also used with the OPTIONAL NORM Positive |
| | Acknowledgment Process. |
+--------------------+-----------------------------------------------+Then there's the second group of message types and their mechanisms. These are OPTIONAL for complying NORM protocol implementations, so you're not stuck with them 🌈
+----------------------+----------------------------------------------+
| Message Type | Purpose |
+----------------------+----------------------------------------------+
|NORM_INFO | Sender message for providing ancillary |
| | context information associated with NORM |
| | transport objects. The use of the NORM FEC |
| | Object Transmission Information header |
| | extension is OPTIONAL with NORM_INFO |
| | messages. |
+----------------------+----------------------------------------------+
|NORM_CMD(EOT) | Sender command to indicate it has reached |
| | end-of-transmission and will no longer |
| | respond to repair requests. |
+----------------------+----------------------------------------------+
|NORM_CMD(ACK_REQ) | Sender command to support application- |
| | defined, positively acknowledged commands |
| | sent outside of the context of the bulk data |
| | content being transmitted. The NORM Positive|
| | Acknowledgment Procedure associated with this|
| | message type is OPTIONAL. |
+----------------------+----------------------------------------------+
|NORM_CMD(APPLICATION) | Sender command containing application-defined|
| | commands sent outside of the context of the |
| | bulk data content being transmitted. |
+----------------------+----------------------------------------------+
|NORM_REPORT | Optional message type reserved for |
| | experimental implementations of the NORM |
| | protocol. |
+----------------------+----------------------------------------------+4. Message Formats 💖
As mentioned in Section 2.1, there are two main classes of NORM messages: sender messages and receiver messages 🦋 NORM_CMD, NORM_INFO, and NORM_DATA messages come from senders of data content, while NORM_NACK and NORM_ACK messages come from receivers within a NormSession. There's also an auxiliary NORM_REPORT message type for experimental purposes 🧪 This section describes the message formats the NORM protocol uses, and these messages and their fields get referenced in the detailed functional description of NORM in Section 5, so it's a whole connected web 🕸️ Individual NORM messages are designed to be compatible with the MTU limitations of the encapsulating Internet protocols, including IPv4, IPv6, and UDP. The current NORM protocol specification assumes UDP encapsulation and leverages UDP's transport features. And the best part? NORM messages don't depend on network addresses, so they work in IPv4 and IPv6 networks alike 🌸✨
4.1. NORM Common Message Header and Extensions 💖
okay bestie, every NORM message starts with the same common header 🎀 There's also a header extension mechanism so the protocol can level up without a whole new revision of this doc ✨ Here's the header every message starts with:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type | hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM Common Message Header FormatThe version field is a 4-bit value holding the protocol version number 🔢 NORM implementations SHOULD ignore received messages whose version numbers don't match their own. This number exists to tell apart upgrades that might not be interoperable 🚫 For this specification, the NORM version number is 1 ✨
The type field is a 4-bit value that tells you which NORM message type you're looking at 📬 The types are:
Message Value
NORM_INFO 1
NORM_DATA 2
NORM_CMD 3
NORM_NACK 4
NORM_ACK 5
NORM_REPORT 6The 8-bit hdr_len field counts how many 32-bit words make up the message's header 📏 That's how header extensions get tacked on. Extensions are implied whenever hdr_len is bigger than the base value for that message type 🧩
The sequence field is a 16-bit value the message originator sets, counting up by one with every NORM message sent to a given destination address 📈 A sequence number space SHOULD be kept for messages sent to the NormSession group address. Receivers can watch this value to spot packet losses coming from a sender, and use it to estimate raw packet loss for congestion control 👀 Heads up though: NORM does not use this value to detect missing reliable data, and it doesn't identify the application data or FEC payload riding along 🙅♀️ When messages are authenticated, sequence can also protect against message "replay" attacks, especially for NORM_NACK or other feedback messages 🛡️ In that case the receiver node should keep a monotonically increasing sequence space for each destination it transmits to (that could be several destinations when unicast feedback is in use). The field is sized to catch a reasonable range of packet loss within the delay-bandwidth product of expected network connections 💅
The source_id field is a 32-bit value naming the node that sent the message 🆔 A participant's NORM node identifier (NormNodeId) can be whatever the application needs, but unique identifiers have to be assigned within a single NormSession 🫶 Sometimes the host IP address, or a hash of it, is good enough. Still, you'd want to think about how identifiers get assigned, and how to resolve collisions, within a multicast session 🤔 For example, the "source identifier" mechanism in the Real-Time Protocol (RTP) specification [18] might work for NORM node identifiers. Right now the protocol makes no assumptions about how these unique identifiers actually get assigned 🦋
NORM Header Extensions ✨
When header extensions are used, they come right after the message type's base header and before any payload 🎀 Both formats start with an 8-bit het (header extension type) field 💜 The first format is for variable-length extensions, with het values from 0 through 127. The second is for fixed-length extensions (one 32-bit word), with het values from 128 through 255. Here they are:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| het <=127 | hel | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| Header Extension Content |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM Variable Length Header Extension Format
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| het >=128 | reserved | Header Extension Content |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM Fixed Length (32-bit) Header Extension FormatThe "Header Extension Content" part of these formats gets defined for each header extension type that NORM messages use 🧁 A few extensions live in this document for baseline FEC and congestion control operations 🔥
4.2. Sender Messages 🚀
Sender messages are NORM_DATA, NORM_INFO, and NORM_CMD 📤 NORM_DATA and NORM_INFO carry application data content, while NORM_CMD messages handle the protocol's various control functions 💅
4.2.1. NORM_DATA Message 💖
okay so the NORM_DATA message is literally the main character 🎀 It's the one senders are gonna be sending the most. These messages wrap up segmented data for objects of type NORM_OBJECT_DATA, NORM_OBJECT_FILE, and NORM_OBJECT_STREAM ✨ And the data inside can be original application data or FEC-encoded data, so it's giving versatility 💅
A NORM_DATA message has three logical parts 🦋 First up is a fixed-format NORM_DATA header. Next is a FEC Payload ID part, and its format depends on which FEC encoding is being used. Last is the payload part, which holds the source or encoded application data. Quick heads up: for NORM_OBJECT_STREAM objects, the payload part has some extra fields so receivers can rebuild the stream properly 🧚 Also, NORM implementations MAY also extend the NORM_DATA header with a FEC Object Transmission Information (EXT_FTI) header extension. This is so receivers can allocate resources and do FEC decoding automatically, with no pre-configuration or out-of-band info needed. Iconic, honestly 😭
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=2| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flags | fec_id | object_transport_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_payload_id |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| header_extensions (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| payload_reserved* | payload_len* |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| payload_offset* |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| payload_data* |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_DATA Message FormatNOTE: the "payload_reserved", "payload_len" and "payload_offset" fields only show up for NORM_OBJECT_STREAM objects 🌸 They let senders change the size of NORM_DATA payload segments for streams whenever they want, so apps can flush streams when they need to. For NORM_OBJECT_FILE and NORM_OBJECT_DATA objects these fields aren't needed, because receivers can work out the length and offset from "fec_payload_id" using the algorithm in Section 5.1.1 🧮 "payload_reserved" is just being kept around for future NORM stream control stuff. When systematic FEC codes (like "fec_id" = 129) are used, "payload_len" and "payload_offset" hold the real length and offset of the application data segment in NORM_DATA messages carrying source data symbols. In NORM_DATA messages that carry parity info, though, these fields aren't real lengths or offsets. They're values computed by FEC-encoding the "payload_len" and "payload_offset" fields of the _source_ data symbols in the matching coding block. Wild, I know 🤯
The "version", "type", "hdr_len", "sequence", and "source_id" fields make up the NORM Common Message Header from Section 4.1 📋 The NORM_DATA "type" field is 2. The NORM_DATA _base_ "hdr_len" is 4 (32-bit words) plus the size of the "fec_payload_id" field, and that size depends on the FEC encoding for the referenced NormObject 💜 The "fec_id" field tells you which FEC coding type is in use. For example, with small block, systematic codes you get a "fec_id" of 129, and then "fec_payload_id" is two 32-bit words. So the NORM_DATA base "hdr_len" ends up being 6 in that case. Any header extensions get added into "hdr_len" too, so the total keeps up 🧠
The "instance_id" field holds a value the sender makes up to uniquely identify its current participation in the NormSession 🌈 That's how receivers can tell when a sender has left and rejoined a session that's already going. When a sender (identified by its "source_id") shows up with a new "instance_id", the NORM receivers SHOULD drop their previous state on that sender and start receiving fresh 🫶
The "grtt" field is a non-linear quantized version of the sender's current estimate of group round-trip time (GRTT) 🔥 (It's also called R_max in [19].) Receivers use it to time the NACK repair process and other bits of protocol operation described in this document. The algorithm for encoding and decoding this field is in the RMT NORM Building Block document [4] 📖
The "backoff" field lets receivers figure out the maximum backoff timer value for timer-based NORM NACK feedback suppression ⏱️ It's a 4-bit field, so it covers 0-15, and that gets multiplied by the sender GRTT to get the max backoff timeout. It tells the receiver set what the sender's backoff factor "Ksender" is. Recommended values and how to use them are in the NORM receiver NACK procedure in Section 5.3 💅
The "gsize" field is the sender's current estimate of group size 👑 It's also 4 bits, and it can roughly cover anything from ten up to 500 million. The most significant bit is 0 or 1, meaning a mantissa of 1 or 5 respectively. The three least significant bits, plus one, are a base 10 exponent (order of magnitude). A few examples: "0x0" is 1.0e+01 (10), "0x8" is 5.0e+01 (50), "0x1" is 1.0e+02 (100), and "0xf" is 5.0e+08 🍓 For NORM feedback suppression, group size doesn't need to be super precise. It's even fine to estimate it a bit high, which keeps feedback traffic low 🌸 A default group size estimate of 10,000 ("gsize" = 0x4) is recommended for general purpose reliable multicast apps using the NORM protocol ✨
The "flags" field holds a bunch of binary flags that give hints about how the receiver should handle the object it's talking about 🦋 The defined ones are:
+--------------------+-------+-----------------------------------------+
| Flag | Value | Purpose |
+--------------------+-------+-----------------------------------------+
|NORM_FLAG_REPAIR | 0x01 | Indicates message is a repair |
| | | transmission |
+--------------------+-------+-----------------------------------------+
|NORM_FLAG_EXPLICIT | 0x02 | Indicates a repair segment intended to |
| | | meet a specific receiver erasure, as |
| | | compared to parity segments provided by |
| | | the sender for general purpose (with |
| | | respect to an FEC coding block) erasure |
| | | filling. |
+--------------------+-------+-----------------------------------------+
|NORM_FLAG_INFO | 0x04 | Indicates availability of NORM_INFO for |
| | | object. |
+--------------------+-------+-----------------------------------------+
|NORM_FLAG_UNRELIABLE| 0x08 | Indicates that repair transmissions for |
| | | the specified object will be unavailable|
| | | (One-shot, best effort transmission). |
+--------------------+-------+-----------------------------------------+
|NORM_FLAG_FILE | 0x10 | Indicates object is "file-based" data |
| | | (hint to use disk storage for |
| | | reception). |
+--------------------+-------+-----------------------------------------+
|NORM_FLAG_STREAM | 0x20 | Indicates object is of type |
| | | NORM_OBJECT_STREAM. |
+--------------------+-------+-----------------------------------------+
|NORM_FLAG_MSG_START | 0x40 | Marks the first segment of application |
| | | messages embedded in |
| | | NORM_OBJECT_STREAMs. |
+--------------------+-------+-----------------------------------------+okay bestie, we're picking up right where the flags start getting spicy 🎀✨
NORM_FLAG_REPAIR gets set when the message is a repair transmission 🔁 receivers can use it to run a join policy where anyone who just joined only starts doing the NACK thing once they've gotten new, non-repair data 🫶 no cap, that's how they avoid jumping in and asking for fixes to stuff they never even saw 💅
NORM_FLAG_EXPLICIT marks repair messages the sender sends once it's totally run out of "fresh" parity segments (the ones it hasn't sent yet) 😭 it's giving last resort. Intermediate systems could possibly use this flag to control sub-casting of repair content to different legs of a reliable multicast topology with different repair needs 🦋
NORM_FLAG_INFO is only set when optional NORM_INFO content actually exists for that object 🌸 so receivers only NACK for NORM_INFO retransmissions when there's really something to get. iconic efficiency 👑
NORM_FLAG_UNRELIABLE is set when the sender wants its object to go out as "best effort" only, and it won't send any repairs for it 💀 and here's the tea: NORM receivers SHOULD NOT execute repair requests for objects with this flag. BUT receivers can accidentally ask for repairs anyway when they're missing segments (or info content) for those objects, aka there's a gap in the object_transport_id sequence 🙈 in that case the sender should invoke the NORM_CMD(SQUELCH) process from Section 4.2.3 to shut that down.
NORM_FLAG_FILE is a "hint" from the sender that the object should be stored in non-volatile storage 💾 bestie, it's a suggestion, not a rule 🧚
NORM_FLAG_STREAM is set when the object is NORM_OBJECT_STREAM 🌊 when it's set, NORM_FLAG_MSG_START can optionally be used to mark the first data segments of application-layer messages inside the stream ✨ that's what lets receivers "synchronize" with the sender and actually make sense of the app data when they join a session that's already going 🫶 and the NORM implementation MAY set this flag on the segment sent right after the application does an explicit "flush" of the stream 💖
The fec_id field matches the FEC Encoding Identifier from the FEC Building Block document [5] 🦋 the fec_id value tells you the format of the fec_payload_id field, and together with the FEC Object Transmission Information, it tells you how to decode the FEC encoded stuff 🔥 small block, systematic codes (fec_id = 129) are expected to be the go-to for most NORM purposes, and NORM_OBJECT_STREAM needs systematic FEC codes to perform its best 👑
The object_transport_id field is a value that goes up monotonically and incrementally, assigned by the sender to the NormObjects it sends 🌈 transmissions and repair requests about an object all use the same object_transport_id value. in really long or indefinite sessions, this field may be repeated, but the 16-bit field is presumed to have a big enough sequence space to avoid object mix-ups between receivers and sources. so receivers SHOULD re-synchronize with a server when they get object sequence identifiers that are way out of range from the current state they're tracking for a given source 🧚 during its transmission in a NORM session, an object is uniquely identified by the concatenation of the sender source_id and the given object_transport_id 💜 and NORM_INFO messages for that object carry the same object_transport_id value 🎀
The fec_payload_id identifies the attached NORM_DATA "payload" content 🍓 its size and format depend on the FEC type indicated by fec_id. those formats live in the FEC Building Block document [5] and any later extensions of it. for example, here's the format for small block, systematic codes (fec_id = 129):
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_len | encoding_symbol_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Small Block, Systematic Code ("fec_id" = 129) "fec_payload_id" Formatthe source_block_number, source_block_len and encoding_symbol_id fields line up with the "Source Block Number", "Source Block Length" and "Encoding Symbol ID" fields of the FEC Payload ID format in the IETF FEC Building Block document [5] 🫶 the source_block_number tells you where the coding block sits within a NormObject 📍 for NORM_OBJECT_STREAM objects, it may wrap around in really long-lived sessions 🌀 source_block_len is the number of user data segments in that coding block, so a receiver can figure out whether an attached segment is data or parity and handle it right 🔥 encoding_symbol_id tells you exactly which symbol (segment) in the coding block the payload carries 🦋 depending on that value and the block's source_block_len, the segment could be user data or FEC parity. for systematic codes, symbols numbered below source_block_len are the original application data, and segments at or above source_block_len are parity symbols calculated for the block 💅 smashing object_transport_id::fec_payload_id together gives you a unique transport protocol data unit identifier for that segment, as far as the NORM sender's instance in a session goes 👑
Extra FEC Object Transmission Information (described in the FEC Building Block document [5]) is needed to properly receive and decode NORM transport objects 🧚 that info MAY be provided as out-of-band session information. BUT sometimes it's useful for the sender to send it in band, so receivers can get going with minimal preconfiguration 🌸 that's what the NORM FEC Object Transmission Information Header Extension (EXT_FTI) is for, and it MAY be applied to NORM_DATA and NORM_INFO messages to give receivers what they need 💖 the exact format depends on the FEC code in use, but in general it SHOULD contain any required details about that code (like the FEC Instance ID) and the byte size of the associated NormObject (for NORM_OBJECT_STREAM, that size is the stream buffer size the NORM sender keeps) 🍓 here's the format for small block systematic codes (fec_id = 129):
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| het = 64 | hel = 4 | object_length (msb) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| object_length (lsb) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_instance_id | segment_size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_max_block_len | fec_num_parity |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
FEC Object Transmission Information Header Extension (EXT_FTI) for
Small Block Systematic Codes ("fec_id" = 129)the het field for this header extension is 64 ✅ the hel length depends on the FEC code type named by fec_id, and in this example (fec_id = 129), hel is 4 💜
the 48-bit object_length field is the total size of the object in bytes for the static object types, NORM_OBJECT_FILE and NORM_OBJECT_DATA 📦 receivers use it to figure out storage needs and to allocate space for the object. if a receiver doesn't have enough storage, it may wish to skip reliable reception (aka not NACK) for that object 🙈 for NORM_OBJECT_STREAM, the sender uses object_length to tell the receiver group how big its stream buffer is, and in turn receivers SHOULD use that to allocate a stream buffer of the same size 🌊
the fec_instance_id matches the "FEC Instance ID" from the FEC Building Block document [5] 🦋 here it SHALL be a value that corresponds to the exact type of Small Block Systematic Code in use (like Reed-Solomon GF(2^8), Reed-Solomon GF(2^16), etc.) and the standardized FEC Instance ID values are described in [5] 💅 segment_size is the sender's current setting for the max message payload content (in bytes), so receivers can allocate the right buffering and process the data messages properly 🫶
fec_max_block_len is the current max number of user data segments per FEC coding block the sender uses during the session, which helps receivers allocate buffer space for the blocks they get 🍓
fec_num_parity matches the "maximum number of encoding symbols that can be generated for any source block" from the FEC Object Transmission Information for Small Block Systematic Codes in the FEC Building Block document [5] 🧚 for example, Reed-Solomon codes can be arbitrarily shortened to make different code variations for a given block length. for Reed-Solomon (GF(2^8) and GF(2^16)) codes, this value is the max number of parity segments the sender has available for its coding blocks. this field MAY be interpreted differently for other systematic codes as they get defined 🌈
and last but not least, the payload portion of NORM_DATA messages carries either source data or FEC encoded application content 💖
payload_reserved, payload_len and payload_offset are only there for transport objects of type NORM_OBJECT_STREAM 🌊. They tell you how big the application content in the payload is and where it sits in the stream 📏. If the sender is doing systematic FEC encoding, these fields hold the actual length and offset in bytes for messages carrying original data source symbols 💅. For NORM_DATA messages with calculated parity content, the values come from FEC encoding the payload_len and payload_offset values of the NORM_DATA data segments in the matching FEC coding block 🧮. So if some data goes missing, you can work out its payload_len and payload_offset once the FEC coding block is decoded 🔍. These fields do not count toward the NORM_DATA hdr_len field, fr 🙅♀️. They are not present when the flags of a NORM_DATA message say the transport object is NORM_OBJECT_FILE or NORM_OBJECT_DATA. In that case you can calculate the length and offset from fec_payload_id using the method in Section 5.1.1 🧠. And heads up: for long-lived streams, payload_offset can wrap 🌀.
The payload_data field holds the original application source or parity content for the symbol identified by fec_payload_id 🎀. Its length SHALL be limited to a maximum of the sender's NormSegmentSize bytes given in the FTI for the object 📦. Parity content always has a length of NormSegmentSize. When encoding data segments of different sizes, the FEC encoder SHALL assume zero value padding for any data segment shorter than NormSegmentSize 🧁. It is RECOMMENDED that a sender's NormSegmentSize stay constant for the whole of that sender's time in the session, though it may possibly vary on a per-object basis 💜. The NormSegmentSize is expected to be configurable by the sender application before session participation, as needed for network topology maximum transmission unit (MTU) considerations. For IPv6, MTU discovery may possibly be used at session startup to do this configuration ✨. The payload_data content may be delivered straight to the application for source symbols (when systematic FEC encoding is used) or after the FEC block is decoded. For NORM_OBJECT_FILE and NORM_OBJECT_STREAM objects, the data segment length and offset can be calculated with the algorithm in Section 5.1.1. For NORM_OBJECT_STREAM objects, the length and offset come from the segment's corresponding payload_len and payload_offset fields 🫶.
4.2.2. NORM_INFO Message ✨
The NORM_INFO message is used to convey OPTIONAL, application-defined, "out-of-band" context information for transmitted NormObjects. A good example for bulk file transfer is putting the MIME type info for the file, data, or stream object into the NORM_INFO payload 📂. Receivers may use the NORM_INFO content to decide whether to take part in reliable reception of the associated object 🤔. Each NormObject can have its own independent unit of NORM_INFO. NORM_DATA messages have a flag that shows whether NORM_INFO is available for a given NormObject. NORM receivers may NACK for retransmission of NORM_INFO when they haven't received it for a given NormObject 🙏. The NORM_INFO content is limited to the size of a single NormSegmentSize for the given sender. This atomic nature means NORM_INFO can be repaired quickly and efficiently within the NORM reliable transmission process ⚡.
When NORM_INFO content is available for a NormObject, the NORM_FLAG_INFO flag SHALL be set in NORM_DATA messages for the corresponding object_transport_id, and the NORM_INFO message shall be sent as the first message for the NormObject 🚀.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=1| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flags | fec_id | object_transport_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| header_extensions (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| payload_data |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_INFO Message FormatThe version, type, hdr_len, sequence, and source_id fields make up the NORM Common Message Header from Section 4.1 🧩. When there are no header extensions, the value of hdr_len is 4.
The instance_id, grtt, backoff, gsize, flags, fec_id, and object_transport_id fields carry the same info and do the same job as in NORM_DATA messages 🎯. They let the receiver set up buffering and so on for later transmissions from the sender when NORM_INFO is the first message it gets 🪄.
Just like with NORM_DATA messages, the NORM FTI Header Extension (EXT_FTI) may optionally be applied to NORM_INFO messages. To save on protocol overhead, some NORM implementations may want to apply EXT_FTI to NORM_INFO messages only and not to NORM_DATA messages 🦋.
The NORM_INFO payload_data field carries sender application-defined content, which receiver applications can use for the various purposes described above 💖.
4.2.3. NORM_CMD Messages 💅
NORM_CMD messages are sent by senders to run a bunch of different protocol functions 🎬. That includes round-trip timing collection, congestion control, syncing the sender/receiver repair "windows", and announcing sender status 📡. A core set of NORM_CMD messages is enumerated, and a range of command types is still open for application-specific use 🧃. Some NORM_CMD types may have dynamic content attached. Any attached content is limited to a maximum length of the sender's NormSegmentSize so commands stay atomic 🔒. Every NORM_CMD message starts with a common set of fields after the usual NORM common message header. The standard NORM_CMD fields are:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=3| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flavor | |
+-+-+-+-+-+-+-+-+ NORM_CMD Content +
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_CMD Standard FieldsThe version, type, hdr_len, sequence, and source_id fields make up the NORM Common Message Header from Section 4.1 📋. The value of hdr_len for NORM_CMD messages without header extensions depends on the flavor field 🌈.
The instance_id, grtt, backoff, and gsize fields give the same info and serve the same purpose as in NORM_DATA and NORM_INFO messages 🍓. The flavor field says what type of command comes next. The rest of the NORM_CMD message depends on the command type (flavor). NORM command flavors include:
+----------------------+-------------+---------------------------------+
| Command |Flavor Value | Purpose |
+----------------------+-------------+---------------------------------+
|NORM_CMD(FLUSH) | 1 | Used to indicate sender |
| | | temporary end-of-transmission. |
| | | (Assists in robustly initiating |
| | | outstanding repair requests from|
| | | receivers). May also be |
| | | optionally used to collect |
| | | positive acknowledgment of |
| | | reliable reception from subset |
| | | of receivers. |
+----------------------+-------------+---------------------------------+
|NORM_CMD(EOT) | 2 | Used to indicate sender |
| | | permanent end-of-transmission. |
+----------------------+-------------+---------------------------------+
|NORM_CMD(SQUELCH) | 3 | Used to advertise sender's |
| | | current repair window in |
| | | response to out-of-range NACKs |
| | | from receivers. |
+----------------------+-------------+---------------------------------+
|NORM_CMD(CC) | 4 | Used for GRTT measurement and |
| | | collection of congestion control|
| | | feedback. |
+----------------------+-------------+---------------------------------+
|NORM_CMD(REPAIR_ADV) | 5 | Used to advertise sender's |
| | | aggregated repair/feedback state|
| | | for suppression of unicast |
| | | feedback from receivers. |
+----------------------+-------------+---------------------------------+
|NORM_CMD(ACK_REQ) | 6 | Used to request application- |
| | | defined positive acknowledgment |
| | | from a list of receivers |
| | | (OPTIONAL). |
+----------------------+-------------+---------------------------------+
|NORM_CMD(APPLICATION) | 7 | Used for application-defined |
| | | purposes which may need to |
| | | temporarily preempt data |
| | | transmission (OPTIONAL). |
+----------------------+-------------+---------------------------------+4.2.3.1. NORM_CMD(FLUSH) Message ✨
okay so the NORM_CMD(FLUSH) command gets sent when the sender has reached the end of all the data content and pending repairs it queued up for transmission 💅. This might mean a temporary or permanent end of data transmission, but the sender is still totally willing to answer repair requests 🫶. The command is repeated once per 2*GRTT to excite the receiver set so they check for outstanding repair requests up to and including the transmission point in the flush message 🦋. The number of repeats equals NORM_ROBUST_FACTOR, unless there's a list of receivers that the sender expects explicit positive acknowledgment from (the acking_node_list). In that case, the acking_node_list gets updated as acknowledgments come in, and the NORM_CMD(FLUSH) is repeated using the mechanism in Section 5.5.3 🎀. The bigger the NORM_ROBUST_FACTOR, the better the odds that every applicable receiver gets excited for acknowledgment or repair requests (NACKs) and that those NACKs actually reach the sender 🔥. If a NORM_NACK message interrupts the flush process, the sender restarts the flush after any repair transmissions are finished 🌸.
Receivers also have a timeout that lets them start NACKing on their own (if they still have repair needs) when they don't get any messages of any type from a sender 🙈. This inactivity timeout is related to 2*GRTT*NORM_ROBUST_FACTOR and we'll dig into it more later. With a healthy NORM_ROBUST_FACTOR, data content gets delivered with a high level of reliability ✨. The catch is that a big NORM_ROBUST_FACTOR can mean extra NORM_CMD(FLUSH) transmissions from the sender, plus a longer wait for receivers before they start the terminal NACK process 💀.
For finite-size transport objects like NORM_OBJECT_DATA and NORM_OBJECT_FILE, the flush (if there are no further pending objects) happens at the end of the object. So FEC repair information is always ready to go for repairs requested in response to the flush command 🧚. For NORM_OBJECT_STREAM, though, the flush can happen at any time, even in the middle of an FEC coding block when systematic FEC codes are used 😭. In that case the sender can't provide FEC parity content as repair for the coding block that's in progress, so it's limited to explicitly repairing the stream data content for that block.
Applications that expect to flush stream content a lot SHOULD be judicious about the FEC coding block size (aka don't use a super large coding block size if frequent flushing is happening) 💜. For example, a reliable multicast app sending an ongoing series of small, intermittent messages will need to balance using the NORM_OBJECT_DATA paradigm against the NORM_OBJECT_STREAM paradigm, with an appropriate FEC coding block size. It's kind of like the trade-offs other transport protocols make, like picking between TCP modes such as "no delay" 🍓.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=3| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flavor = 1 | fec_id | object_transport_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_payload_id |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| acking_node_list (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_CMD(FLUSH) Message FormatBesides the NORM common message header and the standard NORM_CMD fields, the NORM_CMD(FLUSH) message has fields that show the sender's current status and logical transmit position 🌈.
The fec_id field tells you which FEC type is used for the flushing object_transport_id, and it also implies the size and format of the fec_payload_id field 🎀. Heads up: the hdr_len value for a NORM_CMD(FLUSH) message is 4 plus the size of the fec_payload_id field when there are no header extensions 🧚.
The object_transport_id and fec_payload_id fields show the sender's current logical "transmit position" 💖. They're read the same way as in the NORM_DATA message type. When receivers get a NORM_CMD(FLUSH), they're expected to check their completion state through (including) this transmission position 👑. If they have outstanding repair needs in that range, they SHALL start the NORM NACK Repair Process described in Section 5.3 ✨. If they have no outstanding repair needs, they don't send any response to the NORM_CMD(FLUSH) 🫶.
For NORM_OBJECT_STREAM objects using systematic FEC codes, receivers MUST request "explicit-only" repair of the identified source_block_number if the given encoding_symbol_id is less than the source_block_len 😭. That condition means the sender hasn't finished encoding that FEC block yet, so parity content isn't available. An "explicit-only" repair request is NACK content for the relevant source_block_number that doesn't ask for any parity-based repair 🦋. This lets NORM sender apps "flush" an ongoing stream when they need to, even in the middle of an FEC block 🍓. Once the sender picks the stream back up and passes the end of the pending coding block, later NACKs from receivers SHALL request parity-based repair as usual 💅. (Note that using a systematic FEC code is assumed here. Normal receiver NACK initiation and construction is covered in detail in Section 5.3 🧠.)
The OPTIONAL acking_node_list field has a list of NormNodeIds for receivers the sender is asking for explicit positive acknowledgment of reception, up through the transmission point identified by the object_transport_id and fec_payload_id fields 🌸. You can work out the length of the list from the length of the received NORM_CMD(FLUSH) message 🔥. When the acking_node_list is there, the lightweight positive acknowledgment process described in Section 5.5.3 SHALL be observed 💜.
4.2.3.2. NORM_CMD(EOT) Message 💌
The NORM_CMD(EOT) command gets sent when the sender reaches a permanent end-of-transmission with respect to the NormSession, and it won't answer any more repair requests 🌈. This lets receivers wrap up their operation with this sender gracefully (no timeout needed) and free up any resources they don't need anymore 🎀. The NORM_CMD(EOT) command SHOULD be sent using the same robust mechanism as NORM_CMD(FLUSH) commands, so the receiver set has a high level of assurance that it gets the message ✨.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=3| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flavor = 2 | reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_CMD(EOT) Message FormatThe hdr_len value for NORM_CMD(EOT) messages with no header extensions is 4 💖. The reserved field is saved for future use and MUST be set to an all ZERO value 🫶. Receivers MUST ignore the reserved field 🙈.
4.2.3.3. NORM_CMD(SQUELCH) Message
okay so NORM_CMD(SQUELCH) gets sent when the sender gets NORM_NACK content that's outdated or invalid 🙈 invalid NACK content means repair requests for NormObjects the sender can't or won't repair. that covers outdated objects, aborted objects, and anything the sender earlier sent with the NORM_FLAG_UNRELIABLE flag set 💅 squelch basically tells receivers what's still fixable, aka the sender's current "repair window" ✨ Receivers SHALL not generate repair requests for content that a NORM_CMD(SQUELCH) has marked as invalid 🫶
The NORM_CMD(SQUELCH) is sent once per 2*GRTT at the most 🎀 it tells everyone the current "repair window" by naming the earliest (lowest) transmission point the sender will still repair, plus an encoded list of objects from that point onward that are no longer valid for repair 🦋 this lets the sender app cancel or abort sending and/or repairing specific objects it already queued up. the list also includes the IDs of any objects inside the repair window that were sent with NORM_FLAG_UNRELIABLE set 🌸 ngl, you won't see squelch a lot. normal conditions mean it's rare, mostly used to give a reference repair window to receivers who've gone "out-of-sync" because the network was extremely bad 😭
the invalid NormObject list starts at the lowest invalid NormTransportId that's greater than the current "repair window" start, taken from the invalid NACK(s) that set off the squelch 🧚 the list length is capped by the sender's NormSegmentSize. this way receivers learn the status of the sender's repair window with minimal NORM_CMD(SQUELCH) traffic, which is iconic behavior honestly 👑 here's the message format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| version | type = 3 | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flavor = 3 | fec_id | object_transport_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_payload_id |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| invalid_object_list |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_CMD(SQUELCH) Message Formaton top of the NORM common message header and the standard NORM_CMD fields, NORM_CMD(SQUELCH) carries fields that pin down the earliest logical transmit position of the sender's current repair window, plus an "invalid object list" that starts at the logically earliest invalid repair request from the NACK that kicked off the squelch 💖
"object_transport_id" and "fec_payload_id" get concatenated to mark where the sender's current repair window begins (aka the logically earliest point in its transmission history the sender can still repair from) ✨ the "fec_id" field tells you the size and format of "fec_payload_id". this is basically a "synchronization point" advertisement so receivers know where to ask for repair 🍓 heads up, an "encoding_symbol_id" can show up in "fec_payload_id", but the sender's repair window SHOULD be aligned on FEC coding block boundaries, so "encoding_symbol_id" SHOULD be zero 🌈
the "invalid_object_list" is a list of 16-bit NormTransportIds. they're inside the sender's current repair window but no longer available for repair from the sender 🙈 for example, a sender app might dequeue an out-of-date object even though it's still inside the repair window. the total size of "invalid_object_list" content can be figured out from the packet's payload length, and it's capped at the sender's NormSegmentSize. so for really big repair windows, one NORM_CMD(SQUELCH) might not fit the whole set of invalid objects 😭 in that case the sender SHALL ensure the list starts at a NormObjectId that's greater than or equal to the lowest ordinal invalid NormObjectId from the NACK message(s) that triggered the NORM_CMD(SQUELCH). the NormObjectIds in "invalid_object_list" MUST be greater than the "object_transport_id" marking the start of the sender's repair window. this guarantees the squelch process converges, even if it takes several invalid NACK/squelch rounds 👑 this explicit list of invalid content inside the current window lets the sender app (especially for discrete "object" based transport) arbitrarily invalidate, aka dequeue, chunks of enqueued content like certain objects it no longer wants reliable transport for 💅
4.2.3.4. NORM_CMD(CC) Message
NORM_CMD(CC) messages carry fields for sender-to-receiver group greatest round-trip time (GRTT) measurement and for poking the group to send congestion control feedback 🦋 a baseline NORM congestion control scheme (NORM-CC), based on the TCP-Friendly Multicast Congestion Control (TFMCC) scheme of [19], is described in Section 5.5.2 of this document. NORM_CMD(CC) is usually sent as part of NORM-CC congestion control operation 🎀 there's a NORM header extension defined below to go with NORM_CMD(CC) to support NORM-CC. other header extensions may be defined for NORM_CMD(CC) (and/or other NORM messages as needed) to support alternative congestion control schemes in the future ✨ if NORM runs in a private network with congestion control turned off, NORM_CMD(CC) is only used for GRTT measurement and may optionally be sent less often than it would be with congestion control on 🌸
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=3| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flavor = 4 | reserved | cc_sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| send_time_sec |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| send_time_usec |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| header extensions (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| cc_node_list (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_CMD(CC) Message Formatthe NORM common message header and standard NORM_CMD fields do their usual jobs 💜
the "reserved" field is for potential future use and should be set to ZERO in this version of the NORM protocol 🙈
the "cc_sequence" field is a sequence number the sender applies 🫶 for NORM-CC operation it gives the same functionality as the "feedback round number" (fb_nr) described in [19]. receivers record the most recently received "cc_sequence" value and can feed it back to the sender in the congestion control feedback they generate for that sender 🦋 "cc_sequence" can also help NORM implementations figure out how recently a receiver has gotten NORM_CMD(CC) probes from the sender. that's handy instrumentation for complex or experimental multicast routing environments 🔥
the "send_time" field is a timestamp for when the NORM_CMD(CC) message was transmitted 🕰️ it's a 64-bit field: 32 bits of time in seconds ("send_time_sec") and 32 bits of time in microseconds ("send_time_usec"), counted from some reference time the source keeps (usually 00:00:00, 1 January 1970). the byte order is "Big Endian" network order 🌊 receivers take this timestamp, add the delay between when they got the NORM_CMD(CC) and when they respond, and use that as the "grtt_response" portion of the NORM_ACK and NORM_NACK messages they generate. this lets the sender work out round-trip times to different receivers for congestion control and other purposes (like GRTT determination) ✨
to support the baseline NORM-CC scheme in Section 5.5.2, there's a NORM-CC Rate header extension (EXT_RATE) that tells the group the sender's current transmission rate 💖 it's used together with the loss detection "sequence" field of all NORM sender messages and the NORM_CMD(CC) GRTT collection process to make NORM-CC congestion control work 👑 here's the format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| het = 128 | reserved | send_rate |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM-CC Rate Header Extension Format (EXT_RATE)the "send_rate" field is the sender's current transmission rate in bytes per second 🩵 it's a 16-bit field with 12 bits of mantissa in the most significant portion and 4 bits of base 10 exponent (order of magnitude) in the least significant portion. the 12-bit mantissa is scaled so a floating point value of 0.0 maps to 0 and a floating point value of 10.0 maps to 4096. so:
send_rate = (((int)(Value_mantissa * 4096.0 / 10.0 + 0.5)) << 4) |
Value_exponent;for example, to represent a transmission rate of 256kbps (3.2e+04 bytes per second), the lower 4 bits of the 16-bit field hold 0x04 for the exponent, while the upper 12 bits hold 0x51f, as worked out from the equation above:
Okay so here's the send_rate trick 🧚✨ The example takes 3.2, scales it by 4096/10 with a +0.5 rounding, shifts that left by 4, then ORs in 4 💅
send_rate = (((int)((3.2 * 4096.0 / 10.0) + 0.5)) << 4) | 4;
= (0x51f << 4) | 0x4
= 0x51f4To decode the "send_rate" field, you can use this equation 🎀
value = (send_rate >> 4) * 10.0 / 4096.0 *
power(10.0, (send_rate & x000f))Note the maximum transmission rate this scheme can represent is approximately 9.99e+15 bytes per second 🌈💀
When this extension is present, a "cc_node_list" may be attached as the payload of the NORM_CMD(CC) message 🦋 Its presence also means NORM receivers should respond according to the procedures in Section 5.5.2 🫶 The "cc_node_list" is a list of NormNodeIds plus their congestion control status. It includes the current limiting receiver (CLR) node, any potential limiting receiver (PLR) nodes that have been identified, and some number of other receivers whose congestion control status is being reported, most notably their current RTT measurement 👑 The list has room for at least the CLR and one other receiver, but it may be configurable for more timely feedback to the group ⭐ You can infer the list length from the length of the NORM_CMD(CC) message 🍓
Each item in the "cc_node_list" looks like this 💜
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| cc_node_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| cc_flags | cc_rtt | cc_rate |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Congestion Control Node List Item FormatThe "cc_node_id" is the NormNodeId of the receiver that the item represents 🌸
The "cc_flags" field holds flags showing the congestion control status of that receiver. These flags are defined ✨
+------------------+-------+------------------------------------------+
| Flag | Value | Purpose |
+------------------+-------+------------------------------------------+
|NORM_FLAG_CC_CLR | 0x01 | Receiver is the current limiting |
| | | receiver (CLR). |
+------------------+-------+------------------------------------------+
|NORM_FLAG_CC_PLR | 0x02 | Receiver is a potential limiting |
| | | receiver (PLR). |
+------------------+-------+------------------------------------------+
|NORM_FLAG_CC_RTT | 0x04 | Receiver has measured RTT with respect |
| | | to sender. |
+------------------+-------+------------------------------------------+
|NORM_FLAG_CC_START| 0x08 | Sender/receiver is in "slow start" phase |
| | | of congestion control operation (i.e., |
| | | The receiver has not yet detected any |
| | | packet loss and the "cc_rate" field is |
| | | the receiver's actual measured receive |
| | | rate). |
+------------------+-------+------------------------------------------+
|NORM_FLAG_CC_LEAVE| 0x10 | Receiver is imminently leaving the |
| | | session and its feedback should not be |
| | | considered in congestion control |
| | | operation. |
+------------------+-------+------------------------------------------+The "cc_rtt" holds a quantized version of the RTT that the sender measured with respect to the indicated receiver 🦋 This field only counts if the NORM_FLAG_CC_RTT flag is set in "cc_flags" 🙈 It's a one byte field, quantized using the algorithm described in the NORM Building Block document [4] 🔥
The "cc_rate" field represents the receiver's current calculated rate (during steady-state congestion control operation) or twice its measured rate (during the "slow start" phase) 💖 It's encoded and decoded using the same technique described for the NORM_CMD(CC) "send_rate" field 🧚💅
4.2.3.5. NORM_CMD(REPAIR_ADV) Message 📣
okay bestie, so the NORM_CMD(REPAIR_ADV) message is the sender spilling its repair tea 🫖. It gathers up the repair state it built from NORM_NACK messages during a repair cycle, plus any congestion control feedback it got back, and then "advertises" all of that to the group. The sender only sends this when it has been getting NORM_NACK and/or NORM_ACK(CC) messages via unicast instead of multicast 🙈. By echoing that info out to the whole receiver set, feedback suppression can still happen even when receivers are unicasting their feedback instead of multicasting it among the group [13]. iconic, honestly 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=3| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flavor = 5 | flags | reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| header extensions (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| repair_adv_payload |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_CMD(REPAIR_ADV) Message FormatThe "instance_id", "grtt", "backoff", "gsize", and "flavor" fields do the same job they do in every other NORM_CMD message ✨. When there are no extensions, the "hdr_len" value is 4.
The "flags" field tells you about the NORM_CMD(REPAIR_ADV) content 🎀. Right now there is only one flag defined:
NORM_REPAIR_ADV_FLAG_LIMIT = 0x01The sender turns this flag on when it can't squeeze its full current repair state into a single NormSegmentSize 🧩. When the flag is set, receivers should limit their NACK response to generating NACK content only up through the maximum ordinal transmission position (objectId::fecPayloadId) included in the "repair_adv_content".
When congestion control is enabled, a header extension may be attached to the NORM_CMD(REPAIR_ADV) to carry the most limiting congestion control response, meaning the one that suppresses feedback the most 🛑. This lets the message suppress receiver congestion control responses as well as NACK feedback. The field is a header extension so that other congestion control schemes can be used with NORM without changing this document, which is so smart of them 🧠. The NORM-CC Feedback Header Extension (EXT_CC) is defined to wrap congestion control feedback inside NORM_NACK, NORM_ACK, and NORM_CMD(REPAIR_ADV) messages. If another congestion control technique is used inside a NORM implementation, such as Pragmatic General Multicast Congestion Control (PGMCC) [20], an additional header extension MAY need to be defined to carry whatever feedback content it needs 🦋. The NORM-CC Feedback Header Extension format looks like this:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| het = 3 | hel = 3 | cc_sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| cc_flags | cc_rtt | cc_loss |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| cc_rate | cc_reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM-CC Feedback Header Extension (EXT_CC) FormatThe "cc_sequence" field holds the greatest "cc_sequence" value that receivers have received in NORM_CMD(CC) messages from the sender 🕰️. This gives the sender a sense of how fresh each receiver's round-trip measurement reference time is, and whether the receiver has been catching recent congestion control probes. For example, if a receiver clearly hasn't been getting recent probes (and maybe other messages from the sender either), the sender may choose to take congestion avoidance measures 🚦. For NORM_CMD(REPAIR_ADV) messages, the sender SHALL set the "cc_sequence" field to the value it set in the last NORM_CMD(CC) message it sent.
The "cc_flags" field is a set of bits that show the receiver's state for congestion control 🚩. The values it can take are the ones listed for the NORM_CMD(CC) message node list item flags. Receivers use these to control (and suppress, when needed) their congestion control feedback. For NORM_CMD(REPAIR_ADV) messages, these rules apply:
- NORM_FLAG_CC_RTT should be set only when all feedback messages received by the sender have the flag set.
- NORM_FLAG_CC_CLR or NORM_FLAG_CC_PLR should be set only when no feedback has been received from non-CLR or non-PLR receivers.
- NORM_FLAG_CC_LEAVE should be set only when all feedback messages the sender has received have this flag set.
These heuristics for setting the flags in NORM_CMD(REPAIR_ADV) are what make suppression work best for receivers sending unicast feedback 💜.
The "cc_rtt" field SHALL be set to a default maximum value, and the NORM_FLAG_CC_RTT flag SHALL be cleared, when no receiver has received any RTT measurement info yet 🌸. Once a receiver has RTT measurement info, it shall set the "cc_rtt" value accordingly and set the NORM_FLAG_CC_RTT flag in the "cc_flags" field. For NORM_CMD(REPAIR_ADV) messages, the sender SHALL set the "cc_rtt" field to the largest non-CLR/non-PLR RTT it has measured from receivers for the current feedback round ⏱️.
The "cc_loss" field is the receiver's current packet loss fraction estimate for the indicated source 📉. The loss fraction runs from 0.0 to 1.0, which maps to zero through 100 percent packet loss. The 16-bit "cc_loss" value is worked out with this formula:
"cc_loss" = decimal_loss_fraction * 65535.0For NORM_CMD(REPAIR_ADV) messages, the sender SHALL set the "cc_loss" field to the largest non-CLR/non-PLR loss estimate it has received from receivers for the current feedback round 🧮.
The "cc_rate" field is the receiver's current local congestion control rate 🏎️. During "slow start", when the receiver has detected no loss, this value is set to twice the actual rate it has measured from the corresponding sender, and the NORM_FLAG_CC_START flag is set in the "cc_flags" field. Otherwise, the receiver works out a congestion control rate from its loss measurement and its RTT measurement info (even if that info is just the default) and puts it in the "cc_rate" field. For NORM_CMD(REPAIR_ADV) messages, the sender SHALL set the "cc_loss" field value to the lowest non-CLR/non-PLR "cc_rate" report it has received from receivers for the current feedback round 🔍.
The "cc_reserved" field is reserved for future NORM protocol use 🔒. For now, senders SHALL set this field to ZERO, and receivers SHALL ignore the content of this field.
The "repair_adv_payload" has exactly the same form as the "nack_content" of NORM_NACK messages. Receivers can process it for suppression purposes in the same way, with one exception: the case where NORM_REPAIR_ADV_FLAG_LIMIT is set 🫶.
4.2.3.6. NORM_CMD(ACK_REQ) Message 🙋♀️
the NORM_CMD(ACK_REQ) message is how the sender asks a specific list of receivers to acknowledge something 💌. It's a lightweight positive acknowledgment mechanism, and the reliable multicast application can choose to use it (it's OPTIONAL) 🌈. There's a whole range of acknowledgment request types the application can pick from as it likes. Application-defined, positively-acknowledged commands are also allowed, so the application can automatically take advantage of the transmission and round-trip timing info that NORM already has 🧚. The full details of the NORM positive acknowledgment process, including sending NORM_CMD(ACK_REQ) messages and the receiver response (NORM_ACK), are in Section 5.5.3 📖. The NORM_CMD(ACK_REQ) message looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=3| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flavor = 6 | reserved | ack_type | ack_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| acking_node_list |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_CMD(ACK_REQ) Message FormatThe common NORM message header and the standard NORM_CMD fields do their usual jobs ✨. For NORM_CMD(ACK_REQ) messages with no header extension, the "hdr_len" value is 4.
The "ack_type" field says what kind of acknowledgment is being requested, and that also tells receivers how to handle the request 🎯. These "ack_type" values are defined, and they show up again in NORM_ACK messages later on:
+---------------------+--------+---------------------------------+
| ACK Type | Value | Purpose |
+---------------------+--------+---------------------------------+
|NORM_ACK_CC | 1 | Used to identify NORM_ACK |
| | | messages sent in response to |
| | | NORM_CMD(CC) messages. |
+---------------------+--------+---------------------------------+
|NORM_ACK_FLUSH | 2 | Used to identify NORM_ACK |
| | | messages sent in response to |
| | | NORM_CMD(FLUSH) messages. |
+---------------------+--------+---------------------------------+
|NORM_ACK_RESERVED | 3-15 | Reserved for possible future |
| | | NORM protocol use. |
+---------------------+--------+---------------------------------+
|NORM_ACK_APPLICATION | 16-255 | Used at application's |
| | | discretion. |The NORM_ACK_CC value is only for NORM_ACKs generated in response to the NORM_CMD(CC) messages used in congestion control 🚦. NORM_ACK_FLUSH works the same way, but for applicable NORM_CMD(FLUSH) messages 🧹. The sender SHALL NOT generate NORM_CMD(ACK_REQ) messages with an ack_type of NORM_ACK_CC or NORM_ACK_FLUSH 🙅♀️.
The NORM_ACK_RESERVED range of ack_type values is saved for possible future NORM protocol use 🗄️.
The NORM_ACK_APPLICATION range is there so NORM apps can make their own application-defined, positively-acknowledged commands 💌. Those commands can borrow the internal transmission and round-trip timing info the NORM implementation already has. Lowkey genius 🧠.
The ack_id is a sequenced identifier for each NORM_CMD(ACK_REQ) message. Receivers send it back in their NORM_ACK messages, so the sender can match each reply to the right request 🎯.
The reserved field is reserved for possible future protocol use. It SHALL be set to ZERO by senders and ignored by receivers 🤫.
The acking_node_list field has the NormNodeIds of the current NORM receivers that are supposed to send a positive acknowledgment (NORM_ACK) for this request 🙋. The packet payload length tells you how long the acking_node_list is, and it's limited to the sender's NormSegmentSize. Each NormNodeId is listed in network (Big Endian) byte order 🔢. If a receiver's NormNodeId shows up in the acking_node_list, it SHALL schedule transmission of a NORM_ACK message, as described in Section 5.5.3 📅.
4.2.3.7. NORM_CMD(APPLICATION) Message 🎀
This command lets the NORM app robustly send its own application-defined commands 💪. It preempts any data transmission that's already going, and it gets repeated up to NORM_ROBUST_FACTOR times, once per 2*GRTT. That spacing gives the app time to see any response (if responses are the whole point) before the command repeats 👀. Possible responses include starting data transmission, other NORM_CMD(APPLICATION) messages, or even application-defined, positively-acknowledged commands from other NormSession participants ✨. When these commands get scheduled, they preempt data transmission, and they may also be multiplexed with ongoing data transmission. This kind of robust command lets NORM apps build a full set of session control mechanisms with less state than FEC-encoded reliable content transfer needs, while still using NORM's transmission and round-trip timing info 🦋.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=3| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | grtt |backoff| gsize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flavor = 7 | reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Application-Defined Content |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_CMD(APPLICATION) Message FormatThe NORM common message header and NORM_CMD fields work the way we covered before 📝. When there are no header extensions, the hdr_len value for NORM_CMD(APPLICATION) is 4 🔢.
The "Application-Defined Content" area holds information in whatever format the application likes 🎨. Its size SHALL be limited to a maximum of the sender's NormSegmentSize setting 📏.
4.3. Receiver Messages 📬
The NORM message types that participating receivers generate are NORM_NACK and NORM_ACK. NORM_NACKs are sent to request repair of missing data content from sender transmissions. NORM_ACKs are generated in response to certain sender commands, including NORM_CMD(CC) and NORM_CMD(ACK_REQ) 🫶.
4.3.1. NORM_NACK Message 🙏
The main job of NORM_NACK messages is for receivers to ask for repair of sender content, using selective, negative acknowledgment whenever they detect incomplete data 🕵️♀️. They get sent under the NORM_NACK generation and suppression rules in Section 5.3. NORM_NACKs also carry extra fields that give the sender feedback for round-trip timing collection and congestion control 📈.
The payload of NORM_NACK messages has one or more repair requests, for different objects or portions of those objects. The format looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=4| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| server_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| grtt_response_sec |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| grtt_response_usec |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| header extensions (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nack_payload |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_NACK Message FormatThe NORM common message header fields do their usual jobs 🧾. For NORM_NACK messages without header extensions, the hdr_len value is 6.
The server_id field identifies the NORM sender that the NORM_NACK message is headed to 📮.
The instance_id field has the current session identifier that the sender (the one named in server_id) gave in its sender messages. The sender SHOULD ignore feedback messages that carry an invalid instance_id value 🙅.
The grtt_response fields hold an adjusted version of the timestamp from the most recent NORM_CMD(CC) message for that NORM sender ⏱️. The format matches the send_time field of NORM_CMD(CC). The grtt_response value is relative to the send_time the source gave in its matching NORM_CMD(CC). The receiver takes the source's NORM_CMD(CC) send_time and adds the time differential between when it received the NORM_CMD(CC) and when the NORM_NACK goes out, to work out the grtt_response value. That differential is the receive_to_response_differential in this formula:
"grtt_response" = NORM_CMD(CC) "send_time" +
receive_to_response_differentialThe receiver SHALL set the grtt_response to a ZERO value to show it hasn't received a NORM_CMD(CC) message from that sender yet. That tells the sender it should ignore the grtt_response in this message 0️⃣.
For NORM-CC operation, the NORM-CC Feedback Header Extension (described in the NORM_CMD(REPAIR_ADV} message description) gets added to NORM_NACK messages. It gives feedback on the receiver's current congestion control state 🚦. Alternative header extensions for congestion control feedback may be defined for other congestion control schemes for NORM use in the future 🔮.
The reserved field is for potential future NORM use and SHALL be set to ZERO for this version of the protocol 🫙.
The nack_content of the NORM_NACK message says what repairs the receiver needs from the NORM sender named in server_id 🧩. The receiver builds repair requests from the NORM_DATA and/or NORM_INFO segments it still needs, so it can finish reliable reception up to the sender's transmission position at the moment it starts the NACK Procedure (see Section 5.3). A single NORM Repair Request is a list of items, ranges, and/or FEC coding block erasure counts for the NORM_DATA and/or NORM_INFO content it needs. Multiple repair requests can be concatenated within the nack_payload field of a NORM_NACK message. A single NORM Repair Request can possibly include multiple "items", "ranges", or "erasure_counts", and the nack_payload field can contain multiple repair requests. A single NORM Repair Request looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| form | flags | length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| repair_request_items |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM Repair Request FormatThe form field tells you what kind of repair request items are in the repair_request_items list. Possible values for form are:
Form Value
NORM_NACK_ITEMS 1
NORM_NACK_RANGES 2
NORM_NACK_ERASURES 3A form of NORM_NACK_ITEMS means each repair request item in the repair_request_items list is treated as an individual request 📝. A value of NORM_NACK_RANGES means the list is made of pairs of repair request items, and each pair is an inclusive range of repair needs 🎀. And the NORM_NACK_ERASURES form means the items are still treated individually, but the encoding_symbol_id portion of the fec_payload_id field of the repair request item (see below) gets interpreted as an "erasure count" for the FEC coding block identified by that item's source_block_number 🧮.
The flags field is currently used to show the level of data content the repair request items apply to: an individual segment, an entire FEC coding block, or an entire transport object 🔍. Possible flag values include:
+------------------+-------+-----------------------------------------+
| Flag | Value | Purpose |+------------------+-------+-----------------------------------------+
|NORM_NACK_SEGMENT | 0x01 | Indicates the listed segment(s) or range|
| | | of segments are required as repair. |
+------------------+-------+-----------------------------------------+
|NORM_NACK_BLOCK | 0x02 | Indicates the listed block(s) or range |
| | | of blocks in entirety are required as |
| | | repair. |
+------------------+-------+-----------------------------------------+
|NORM_NACK_INFO | 0x04 | Indicates that NORM_INFO is required as |
| | | repair for the listed object(s). |
+------------------+-------+-----------------------------------------+
|NORM_NACK_OBJECT | 0x08 | Indicates the listed object(s) or range |
| | | of objects in entirety are required as |
| | | repair. |
+------------------+-------+-----------------------------------------+okay so these four flags are the big menu of what a receiver can ask for when it NACKs 💅 NORM_NACK_SEGMENT (0x01) means "send me these specific segments or ranges of segments," while NORM_NACK_BLOCK (0x02) means "I'm missing these whole coding blocks, so send enough to fix them completely" 🧚 NORM_NACK_INFO (0x04) is the one that's like "hey, I don't have the NORM_INFO segment for this object," and NORM_NACK_OBJECT (0x08) is the full-on "I'm missing the entire NormTransportObject, please help bestie" 😭
When NORM_NACK_SEGMENT is set, the object_transport_id and fec_payload_id fields tell the sender exactly which sets or ranges of individual NORM_DATA segments the receiver needs to fix its content 🦋 When NORM_NACK_BLOCK is set, it means the receiver has zero copies of that coding block, so it needs enough transmissions to repair the whole block from scratch 🌸 When NORM_NACK_INFO is set, the receiver is missing the NORM_INFO segment for the object_transport_id it listed. Here's the lowkey tea: NORM_NACK_INFO can be paired with NORM_NACK_BLOCK or NORM_NACK_SEGMENT, or it can show up all by itself 💜 When NORM_NACK_OBJECT is set, the receiver is missing the whole NormTransportObject named by object_transport_id. That also implicitly asks for any available NORM_INFO for that NormObject, if there is one ✨ And the fec_payload_id field gets ignored whenever NORM_NACK_OBJECT is set, so don't even look at it 🙈
The length field is the number of bytes in the repair_request_items field. 🍓
The repair_request_items field is basically a list of individual items or range pairs of transport data unit identifiers, and it looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_id | reserved | object_transport_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_payload_id |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM Repair Request Item FormatThe fec_id tells you which FEC type is in play, and that decides the layout of fec_payload_id 🎀 The reserved field is saved for future use, so it SHALL be set to a ZERO value, and NORM nodes that process NACK content just ignore it 🫶 The object_transport_id points at the NormObject that needs repair, and fec_payload_id pinpoints the exact FEC coding block and/or segment being asked for. Remember, if NORM_NACK_OBJECT is set, fec_payload_id is ignored, and if NORM_NACK_BLOCK is set, only the FEC code block identifier portion of fec_payload_id gets read 👑
The format of fec_payload_id depends on the fec_id value. Periodt 💖
Sometimes a receiver needs a mix of forms (ranges and individual items) or types (specific segments, whole blocks, or whole objects) to finish up reliable transmission. When that happens, multiple NORM Repair Requests with different form and/or flags values can be stacked together inside one NORM_NACK message ✨ Also, NORM receivers SHALL construct NORM_NACK messages with their repair requests in ordinal order based on the object_transport_id and fec_payload_id values 🌈 And the nack_payload size SHALL NOT exceed the NormSegmentSize for the sender the NORM_NACK is headed to. No going over the limit, bestie 🚫💀
NORM_NACK Content Examples
Okay, the examples are where it gets fun 🦋 For these, they assume a small block, systematic FEC code (fec_id = 129) with a user data block length of 32 segments. Example 1 is a list of individual NORM_NACK_ITEMS repair requests. Example 2 is the main character moment, with a list of NORM_NACK_RANGES requests and a single NORM_NACK_ITEMS request stacked together, to show what a full NORM_NACK message can look like 💅 Also, FEC coding block erasure counts could be included in each case, but they're not really needed, since the sender can totally work out the erasure count while it processes the NACK content. The erasure count option might still be handy when you're using other FEC codes or for intermediate system stuff 🧚
Example 1: NORM_NACK "nack_payload" for: Object 12, Coding Block 3, Segments 2,5,8
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| form = 1 | flags = 0x01 | length = 36 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_id = 129 | reserved | object_transport_id = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_number = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_length = 32 | encoding_symbol_id = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_id = 129 | reserved | object_transport_id = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_number = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_length = 32 | encoding_symbol_id = 5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_id = 129 | reserved | object_transport_id = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_number = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_length = 32 | encoding_symbol_id = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Example 2: NORM_NACK "nack_payload" for: Object 18 Coding Block 6, Segments 5, 6, 7, 8, 9, 10; and Object 19 NORM_INFO and Coding Block 1, segment 3
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| form = 2 | flags = 0x01 | length = 24 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_id = 129 | reserved | object_transport_id = 18 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_number = 6 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_length = 32 | encoding_symbol_id = 5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_id = 129 | reserved | object_transport_id = 18 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_number = 6 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_length = 32 | encoding_symbol_id = 10 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| form = 1 | flags = 0x05 | length = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_id = 129 | reserved | object_transport_id = 19 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_number = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_block_length = 32 | encoding_symbol_id = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+4.3.2. NORM_ACK Message ✨
Okay bestie, the NORM_ACK message is basically the MVP of NORM congestion control and round-trip timing 🫶. The acknowledgment type NORM_ACK_CC is the one made for that job, like we mentioned in the NORM_CMD(ACK_REQ) description. How NORM_ACK(CC) messages get generated for round-trip timing estimation and for congestion control is covered in Sections 5.5.1 and 5.5.2. But ngl, some multicast apps would love a little positive acknowledgment for certain stuff 💅. So Section 5.5.3 has a simple, scalable positive acknowledgment scheme that implementations can use when it makes sense. The NORM_CMD(FLUSH) may be used for OPTIONAL collection of positive acknowledgment of reliable reception to a certain "watermark" transmission point from specific receivers using this mechanism. The NORM_ACK type NORM_ACK_FLUSH is made for this, and its "nack_payload" format is shown below. Beyond that, there's a whole range of application-defined "ack_type" values that the app can use however it wants 🌈. Implementations using application-defined positive acknowledgments may also make use of the "nack_payload" as needed. Just stay within the rule that the "nack_payload" field can be at most the NormSegmentSize of the sender the NORM_ACK is destined for.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|version| type=5| hdr_len | sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| source_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| server_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| instance_id | ack_type | ack_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| grtt_response_sec |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| grtt_response_usec |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| header extensions (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ack_payload (if applicable) |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_ACK Message FormatThe NORM common header fields do their usual jobs 💖.
The "server_id", "instance_id", and "grtt_response" fields do the same job as the matching fields in NORM_NACK messages. Header extensions can also be applied to carry congestion control feedback or other functions in the same way.
The "ack_type" field tells you what kind of NORM_ACK this is. It matches the "ack_type" field of the NORM_CMD(ACK_REQ) message this acknowledgment answers.
The "ack_id" field is a sequence number, so the sender can check that a NORM_ACK it receives really answers a current acknowledgment request 🔍. The "ack_id" field is not used for the NORM_ACK_CC and NORM_ACK_FLUSH acknowledgment types.
The "ack_payload" format depends on the "ack_type". NORM_ACK_CC has no attached content, so only the NORM_ACK header applies. For NORM_ACK_FLUSH, there's a specific "ack_payload" format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_id | reserved | object_transport_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fec_payload_id |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
NORM_ACK_FLUSH "ack_payload" FormatThe "object_transport_id" and "fec_payload_id" are how the receiver acknowledges the applicable NORM_CMD(FLUSH) messages that were sent by the sender identified by the "server_id" field 🧚.
The "ack_payload" of NORM_ACK messages for application-defined "ack_type" values is up to the app, but it's capped at a maximum of the NormSegmentSize of the sender referenced by the "server_id" 💅.
4.4. General Purpose Messages 🌸
Some extra message formats are defined for general purposes in NORM multicast sessions. These apply whether the participant is acting as a sender, a receiver, or both within the group.
4.4.1. NORM_REPORT Message 📊
This is an optional message that NORM participants can generate. It could be used for periodic performance reports from receivers in experimental NORM implementations. The format of this message is currently undefined. Experimental NORM implementations may define NORM_REPORT formats as needed for test purposes. These report messages SHOULD be disabled for interoperability testing between different NORM implementations.
5. Detailed Protocol Operation 🦋
This section gets into the detailed interactions between senders and receivers taking part in a NORM session. Here's a quick synopsis of how protocol operation works:
- The sender periodically transmits
NORM_CMD(CC)messages as needed to initialize and collect round-trip timing and congestion control feedback from the receiver set. - The sender transmits an ordinal set of NormObjects, segmented into
NORM_DATAmessages labeled with NormTransportIds and logically identified with FEC encoding block numbers and symbol identifiers.NORM_INFOmessages may optionally precede the transmission of data content for NORM transport objects. - As receivers detect missing content from the sender, they start repair requests with
NORM_NACKmessages. Receivers track the sender's most recentobjectId::fecPayloadIdtransmit position and NACK only for content ordinally prior to that transmit position. Receivers schedule random backoff timeouts before generatingNORM_NACKmessages, and they wait an appropriate amount of time before repeating theNORM_NACKif their repair request isn't satisfied. - The sender aggregates repair requests from the receivers and logically "rewinds" its transmit position to send the appropriate repair messages. It sends repairs for the earliest ordinal transmit position first and keeps this ordinal repair transmission sequence going. Previously untransmitted FEC parity content for the applicable FEC coding block is used for repair transmissions to the greatest extent possible. If the sender uses up its available FEC parity content over multiple repair cycles for the same coding block, it falls back to an explicit repair strategy (possibly using parity content) to finish repairs. (Explicit repair is expected to be an exception in general protocol operation, but it's possible in extreme conditions 🙈.) The sender immediately starts transmitting new content once it has sent pending repairs.
- The sender transmits
NORM_CMD(FLUSH)messages when it reaches the end of enqueued transmit content and pending repairs. Receivers respond toNORM_CMD(FLUSH)messages withNORM_NACKtransmissions (following the same suppression backoff timeout strategy as for data) if they still need further repair. - Sender transmissions are subject to rate control limits set by congestion control mechanisms. In the baseline NORM-CC operation, each sender in a NormSession keeps its own independent congestion control state. Receivers provide congestion control feedback in
NORM_NACKandNORM_ACKmessages.NORM_ACKfeedback for congestion control purposes is governed by a suppression mechanism similar to the one used forNORM_NACKmessages.
While this overall concept is relatively simple, each of these aspects has details that need to be addressed for successful, efficient, robust, and scalable NORM protocol operation 🌈.
5.1. Sender Initialization and Transmission ✨
okay so the moment the NORM sender boots up, it starts blasting NORM_CMD(CC) messages 📡 to collect round trip timing and other info from the potential group. If NORM-CC congestion control is turned on, the NORM-CC Rate header extension MUST be included in these messages 💅. Congestion control SHALL be observed at all times when you're operating on the general Internet 🌐, no exceptions bestie. Even if congestion control is disabled at the sender, it can still be super useful to use the NORM_CMD(CC) messaging to gather feedback from the group with the baseline NORM-CC feedback mechanisms 🧚. This early feedback collection can be used to establish a GRTT estimate before data transmission and any potential NACK operation. literally so smart 😭
Sometimes apps want the sender to start sending data right away 🚀. Other times, the sender wants to wait until it hears some feedback or a request from the receiver set, so it knows people are actually there 👀. Ngl this part is lowkey important, because in some apps (like web push 🔔) that "hey, we're here" signal can come out-of-band, meaning through some other method outside the multicast session. As we said, the periodic NORM_CMD(CC) messages may come before the actual data transmission, so you can have an initial GRTT estimate ready to go ✨.
The OPTIONAL NORM FEC Object Transmission Information Header Extension is where things get really cute 🎀. When you include it, the sender's message headers can carry all the info receivers need to get ready for reliable reception later. That includes the FEC coding parameters, the sender NormSegmentSize, and other stuff 🧁. If this header extension isn't used, it's presumed that receivers already got the FEC Object Transmission Information some other way 🤷♀️. Apps can also use NORM_INFO messages tied to the session data objects to share application-specific context about the session and the data being sent 📝. All of this means you can run with minimal pre-coordination between senders and receivers, which is giving independence energy 💖.
The NORM sender then starts chopping up the application-enqueued data into NORM_DATA segments and sending them to the group 🍰. The segmentation algorithm is described in Section 5.1.1. The transmission rate is controlled by congestion control mechanisms, or it can be a fixed rate if you want that for closed network operations 🔒. Receivers in the multicast group give feedback to the sender as needed 🗣️. When the sender hits the end of the data it has enqueued (plus any pending repairs), it sends a series of NORM_CMD(FLUSH) messages at a rate of one per 2*GRTT ⏰. Receivers might respond to these NORM_CMD(FLUSH) messages with more repair requests 🙈. There's a protocol parameter called "NORM_ROBUST_FACTOR" that decides how many flush messages get sent 🔢. If receivers ask for repair, the repair gets provided, and then flushing happens again at the end of the repair transmission 🔁.
The sender can attach an OPTIONAL "acking_node_list" to NORM_CMD(FLUSH). This list holds the NormNodeIds of the receivers it expects explicit positive acknowledgment from 🫶. The NORM_CMD(FLUSH) message can also be used for this optional function at any time before the end of the data enqueued for transmission, with NORM_CMD(FLUSH) messages multiplexed alongside ongoing data transmissions 🧚♀️. The OPTIONAL NORM positive acknowledgment procedure is described in Section 5.5.3 💜.
5.1.1. Object Segmentation Algorithm
okay this part is the math-y queen moment 👑🧮. NORM senders and receivers must use a common algorithm for logically segmenting transport data into FEC encoding blocks and symbols 🍓. That way, appropriate NACKs can be built to request repair of missing data, no drama 💅. NORM FEC coding blocks are made of multi-byte symbols, which get sent in the payload of NORM_DATA messages 📦. Each NORM_DATA message carries one source or encoding symbol, and the NormSegmentSize sender parameter sets the maximum symbol size in bytes 📏. The FEC encoding type and its parameters decide the source block size (number of source symbols per coding block) 🧩. NORM senders and receivers use these FEC parameters, along with NormSegmentSize and the transport object size, to figure out the source block structure for transport objects 🧠. These parameters come in the FEC Transmission Information for each object 📬. The algorithm below makes all source blocks as close to equal length as possible 🎯. This helps dodge the performance problems of "short" FEC blocks 🙅♀️.
Heads up, this algorithm only applies to the statically-sized NORM_OBJECT_DATA and NORM_OBJECT_FILE transport object types, where the object size is fixed and predetermined 🔒. For NORM_OBJECT_STREAM objects, the object is segmented according to the maximum source block length given in the FEC Transmission Information, unless the FEC Payload ID shows a different size for a given block 🌊.
Here's the NORM block segmentation algorithm 🧁. For a transport object of a given length (L_obj) in bytes, a first number of FEC source blocks (N_large) is delineated at a larger block size (B_large), and a second number of source blocks (N_small) is delineated at a smaller block size (B_small). Given the maximum FEC source block size (B_max) and the sender's NormSegmentSize, the block segmentation for a given NORM transport object is worked out like this:
Inputs:
B_max= Maximum source block length (i.e., maximum number of source symbols per source block)L_sym= Encoding symbol length in bytes (i.e., NormSegmentSize)L_obj= Object length in bytes
Outputs:
N_total= The total number of source blocks into which the transport object is partitioned.N_large= Number of larger source blocks (first set of blocks)B_large= Size (in encoding symbols) of the larger source blocksN_small= Number of smaller source blocks (second set of blocks)B_small= Size (in encoding symbols) of the smaller source blocksL_final= Length (in bytes) of the last source symbol of the last source block (All other symbols are of lengthL_sym).
Algorithm:
- The total number of source symbols in the transport object is computed as:
S_total = L_obj/L_sym[rounded up to the nearest integer] - The transport object is partitioned into
N_totalsource blocks, where:N_total = S_total/B_max[rounded up to the nearest integer] - The average length of a source block is computed as:
B_ave = S_total/N_total(this may be non-integer) - The size of the first set of larger blocks is computed as:
B_large = B_ave[rounded up to the nearest integer] (Note it will always be the case thatB_large <= B_max) - The size of the second set of smaller blocks is computed as:
B_small = B_ave[rounded down to the nearest integer] (Note ifB_aveis an integerB_small = B_large; otherwiseB_small = B_large - 1) - The fractional part of
B_aveis computed as:B_fraction = B_ave - B_small - The number of larger source blocks is computed as:
N_large = B_fraction * N_total(NoteN_largeis an integer in the range 0 throughN_total - 1) - The number of smaller source blocks is computed as:
N_small = N_total - N_large - Each of the first
N_largesource blocks consists ofB_largesource symbols. Each of the remainingN_smallsource blocks consists ofB_smallsource symbols. All symbols areL_symbytes in length except for the final source symbol of the final source block, which is of length (in bytes):L_final = L_obj - (N_large*B_large + N_small*B_small - 1) * L_sym
5.2. Receiver Initialization and Reception 🌸
the NORM protocol is built so receivers can join and leave the group whenever they want 🦋. Some apps are pickier though, and need receivers to already be members of the group before data transmission starts 🫧. NORM apps can use different policies to limit the impact of new receivers joining mid-session 🛡️. For example, a useful implementation policy is for new receivers to limit or avoid repair requests for transport objects already in progress 🙅♀️. The NORM sender implementation might also want extra constraints, so receivers can't mess up reliable multicast performance by joining, leaving, and rejoining the group over and over 🌀. Different receiver "join policies" can suit different apps and scenarios 🎭. For general purpose operation, the default policy is where receivers are only allowed to request repair for coding blocks with a NormTransportId and FEC coding block number greater than or equal to the first non-repair NORM_DATA or NORM_INFO message they receive upon joining the group. This is RECOMMENDED 💖. For objects of type NORM_OBJECT_STREAM, it is RECOMMENDED that the join policy constrains receivers to start reliable reception at the current FEC coding block for which non-repair content is received 🧚♀️.
5.3. Receiver NACK Procedure 🎀
okay so when a receiver notices it's missing data from a sender's NORM transmissions, it kicks off its NACKing procedure 🙈 bestie, the NACKing procedure SHALL be initiated _only_ at FEC coding block boundaries, NormObject boundaries, and whenever a NORM_CMD(FLUSH) message shows up 💅
it starts with a random backoff timeout ⏳ the duration gets picked with the "RandomBackoff" algorithm from the NORM Building Block doc [4], using (Ksender*GRTTsender) as the "maxTime" parameter and the sender's advertised group size (GSIZEsender) as the "groupSize" parameter. senders hand over GRTTsender, Ksender and GSIZEsender through the "grtt", "backoff", and "gsize" fields of their messages ✨ the sender works out GRTTsender from the feedback it's gotten from the group, while Ksender and GSIZEsender may be determined by application requirements and expectations or ancillary info 🧚 the backoff factor "Ksender" MUST be greater than one so feedback suppression actually works 🔥 K = 4 is RECOMMENDED for Any Source Multicast (ASM), and K = 6 is RECOMMENDED for Single Source Multicast (SSM) 👑
so basically:
T_backoff = RandomBackoff(Ksender*GRTTsender, GSIZEsender)to avoid NACK implosion if the sender or network fails during SSM operation, the receiver SHALL automatically suppress its NACK and immediately enter the "holdoff" period (more on that below) when T_backoff is greater than (Ksender-1)*GRTTsender 😭 otherwise the backoff period starts and the receiver MUST accumulate external pending repair state from the NORM_NACK messages and NORM_CMD(REPAIR_ADV) messages it receives 🫶 when the backoff time ends, the receiver SHALL generate a NORM_NACK message only if both of these are true:
- The sender's current transmit position (in terms of
objectId::fecPayloadId) exceeds the earliest repair position of the receiver. - The repair state accumulated from NORM_NACK and NORM_CMD(REPAIR_ADV) messages does not equal or supersede the receiver's repair needs up to the sender transmission position at the time the NACK procedure (backoff timeout) was initiated.
if both conditions are met, the receiver immediately generates a NORM_NACK message when the backoff timeout expires 🚀 otherwise the receiver's NACK is "suppressed" and the message isn't sent 🙅 then it begins a "holdoff" period where it holds back from restarting the NACKing process. the whole point is to give the sender worst-case time to respond to the repair needs before the receiver asks for repair again 🕰️ the "holdoff" timeout (T_rcvrHoldoff), as described in [4], is:
T_rcvrHoldoff =(Ksender+2)*GRTTsenderthe NORM_NACK message holds repair request content starting from the receiver's lowest ordinal repair position, going up through the coding block right before the most recently heard ordinal transmission position for the sender 🌸 if the NORM_NACK content is bigger than the sender's NormSegmentSize, it gets truncated so the receiver only sends a single NORM_NACK message per NACK cycle for a given sender. in short, one NACK message is generated with the receiver's lowest ordinal repair needs 💖
for each partially-received FEC coding block that needs repair, the receiver SHALL, on its _first_ repair attempt for the block, request the parity portion of the FEC coding block starting at the lowest ordinal _parity_ "encoding_symbol_id" (i.e., "encoding_symbol_id" = "source_block_len"), and request the number of FEC symbols matching its data segment erasure count for the block. on _subsequent_ repair cycles for the same coding block, the receiver SHALL request only those repair symbols from the first set it hasn't received yet, up to the remaining erasure count for that coding block. ngl this part is so neat 🦋 the sender may have provided other, different, additional parity segments for other receivers that could also fill the local receiver's erasure needs ✨ when the erasure count for a partially-received FEC coding block is more than the max number of parity symbols the sender has for that block (as shown in the NORM_DATA "fec_num_parity" field), the receiver SHALL request all available parity segments plus the ordinally highest missing data segments it needs to cover its total erasure needs for the block. the goal is for the whole receiver set to ask for a lowest common denominator set of repair symbols for a given FEC coding block 💜 this lets the sender build the most efficient repair transmission segment set, and it enables effective NACK suppression among receivers even with uncorrelated packet loss. it also doesn't need any synchronization among the receiver set in their repair requests 🙌
for FEC coding blocks or NormObjects missed in their entirety, the NORM receiver builds repair requests with NORM_NACK_BLOCK or NORM_NACK_OBJECT flags set as appropriate. asking for a retransmission of NORM_INFO is done by setting the NORM_NACK_INFO flag in a matching repair request 🎀
5.4. Sender NACK Processing and Response 💅
the main goal of the sender is making forward progress on the data its application has queued up 🏃♀️ but the sender has to "rewind" its logical transmission point every so often to meet the repair needs of receivers who NACKed 😤 aggregating multiple NACKs is how it figures out an optimal repair strategy when a NACK event happens. since receivers start the NACK process on coding block or object boundaries, there's a loose level of synchronization in the repair process even when receivers experience uncorrelated data loss 🤝
5.4.1. Sender Repair State Aggregation ✨
when a sender is in its normal state of sending new data and gets a NACK, it starts a procedure to collect NACK repair state from NORM_NACK messages before it begins any repair transmissions. this aggregation period does _not_ interfere with its ongoing new data transmissions 🫧
as described in [4], the period during which the sender aggregates NORM_NACK messages equals:
T_sndrAggregate = (Ksender+1)*GRTTwhere "Ksender" is the same backoff scaling value the receivers use, and "GRTT" is the sender's current estimate of the group's greatest round-trip time 🧠
when this period ends, the sender "rewinds" by folding the accumulated repair state into its pending transmission state, and then it starts sending repair messages 🔁 once the pending repairs are done, it goes back to new transmissions of any enqueued data. at this point it also starts a "holdoff" timeout, during which it holds off on starting a new repair aggregation cycle, even if NORM_NACK messages arrive ⏸️ as described in [4], the value of this sender "holdoff" period is:
T_sndrHoldoff = (1*GRTT)if more NORM_NACK messages arrive during this sender "holdoff" period, the sender will immediately fold these "late messages" into its pending transmission state ONLY if the NACK content is ordinally greater than the sender's current transmission position 👀 this "holdoff" time gives the sender worst-case time to tell the group its current transmission sequence position, which avoids redundant repair transmissions 💖 after the holdoff timeout expires, a new NACK aggregation period can begin (when a NACK arrives), alongside pending repair and new data transmissions. remember that receivers don't start the NACK repair process until the sender's logical transmission position goes past the lowest ordinal position of their repair needs 🫣 with the new aggregation period, the sender repeats the same routine: it folds the accumulated repair state into its transmission plan, then "rewinds" to send the lowest ordinal repair data when the aggregation period ends. again, this all happens alongside ongoing new data and/or pending repair transmissions 🌈
5.4.2. Sender FEC Repair Transmission Strategy ✨
Okay bestie, the NORM sender should lean on FEC parity content for repair as much as humanly possible 💖. Quick reminder: receivers try to request a lowest common denominator of explicit repair (parity included) when they build their NORM_NACK messages 🧚. So before the sender goes and satisfies each receiver's repair needs one by one, it can use the general erasure-filling power of FEC-generated parity segments 🦋. The sender can figure out the maximum erasure filling needs for each FEC coding block from the NORM_NACK messages it gets during the repair aggregation period 📊. Then, if it has a sufficient number (less than or equal to the maximum erasure count) of previously unsent parity segments for those coding blocks, it can send those instead of the exact packets the receiver set asked for 💅. Only after it runs out of "fresh" (unsent) parity segments for a given coding block should it go back to explicit repair of the receiver set's needs 😭.
Generally, if a sufficiently powerful FEC code is used, explicit repair is the exception, and reliable multicast can get done really efficiently ✨. But being able to fall back to explicit repair is what keeps the protocol reliable even in super extreme situations 🔥.
NORM_DATA messages sent as repair transmissions SHALL be flagged with the NORM_FLAG_REPAIR flag 🚩. That way, receivers can obey any policies that limit new receivers from joining the reliable transmission when only repair transmissions have been received 🙈. Additionally, the sender SHOULD additionally flag NORM_DATA transmissions sent as explicit repair with the NORM_FLAG_EXPLICIT flag 🎀.
Now, NORM end system receivers don't actually use NORM_FLAG_EXPLICIT, but intermediate systems could totally leverage that status to "assist" NORM performance 🌸. If those systems are properly positioned with respect to reciprocal reverse-path multicast routing, they need to sub-cast only a sufficient count of non-explicit parity repairs to satisfy a multicast routing sub-tree's erasure filling needs for a given FEC coding block 🧩. When the sender has resorted to explicit repair, the intermediate systems should sub-cast all of the explicit repair packets to those portions of the routing tree that still need repair for a given coding block 💜. Note that intermediate systems will be required to conduct repair state accumulation for sub-routes, in a manner similar to the sender's repair state accumulation, so they have enough info to do the sub-casting 📚. They could also perform additional NORM_NACK suppression/aggregation as they do this repair state accumulation for NORM repair cycles ✨.
The details of this type of operation are beyond the scope of this document, but this info is provided for possible future consideration 🦋.
5.4.3. Sender NORM_CMD(SQUELCH) Generation
Okay, this one's giving "stop sending me junk" energy 💀. If the sender gets a NORM_NACK message asking for repair of data it's no longer supporting, the sender generates a NORM_CMD(SQUELCH) message 🤐. This message advertises its repair window and squelches any receivers from NACKing invalid data any further. The transmission rate of NORM_CMD(SQUELCH) messages is limited to once per 2*GRTT ⏰.
Here's the main vibe: the "invalid_object_list" (if applicable) of the NORM_CMD(SQUELCH) message SHALL begin with the lowest "object_transport_id" from the invalid NORM_NACK messages received since the last NORM_CMD(SQUELCH) transmission 👑. Lower ordinal invalid "object_transport_ids" should be included only while the NORM_CMD(SQUELCH) payload is less than the sender's NormSegmentSize parameter 🍓.
5.4.4. Sender NORM_CMD(REPAIR_ADV) Generation
So when a NORM sender gets NORM_NACK messages from receivers over unicast, it uses NORM_CMD(REPAIR_ADV) messages to advertise its accumulated repair state to the receiver set 📣. Why? Because receivers aren't directly sharing their repair needs with each other over multicast, so the sender has to be the messenger 💌. The NORM_CMD(REPAIR_ADV) message is multicast to the receiver set by the sender 🌈. Its payload has content in the same format as the NORM_NACK receiver message payload 🧾.
Receivers can then do feedback suppression the same way they would with NORM_NACK messages they got directly from other receivers 🫶. And here's a fun twist: the sender doesn't just retransmit the NACK content it got, it sends a representation of its aggregated repair state instead 🧠. Transmission of NORM_CMD(REPAIR_ADV) messages is subject to the sender transmit rate limit and the NormSegmentSize limitation ⚖️.
When the NORM_CMD(REPAIR_ADV) message is of maximum size, receivers SHALL consider the maximum ordinal transmission position value embedded in the message as the sender's "current" transmission position, and they implicitly suppress requests for ordinally higher repair 🙅♀️. For congestion control operation, the sender may also need to provide info so dynamic congestion control feedback can be suppressed as needed among receivers 🚦. This document specifies the NORM-CC Feedback Header Extension for baseline NORM-CC operation 🎀. If other congestion control mechanisms are used within a NORM implementation, other header extensions may be defined 🦋. Whatever content format is used for this should ensure that the maximum possible suppression state is conveyed to the receiver set 💅.
5.5. Additional Protocol Mechanisms ✨
Okay, so besides the main event of sending data and repairing it, NORM has some other mechanisms 🧚. These help it adapt to network conditions and play fair with other protocols that are hanging out on the same network 🤝.
5.5.1. Greatest Round-trip Time Collection
Everyone needs to be on the same clock for this to work 🕰️. For NORM receivers to scale their backoff timeouts properly, and for senders to use matching timeouts, all the participants have to agree on a common timeout basis 💜. So each NORM sender monitors the round-trip time of its active receivers and figures out the group greatest round-trip time (GRTT) 📈. The sender puts this GRTT estimate in every message it sends, so receivers always have it on hand for scaling their timers ⏱️.
To measure the current GRTT, the sender periodically sends NORM_CMD(CC) messages that carry a locally generated timestamp 🛎️. Receivers are expected to record this timestamp along with the time the NORM_CMD(CC) message arrived 📝. Then, when receivers send feedback to the sender, they embed an adjusted version of the sender's timestamp in the feedback message (NORM_NACK or NORM_ACK) 💌. The adjustment adds the amount of time the receiver held the timestamp before responding 🙃. Once the sender gets that adjusted timestamp, it can calculate the round-trip time to that receiver 🎯.
The round-trip time for each receiver goes into an algorithm that weights and smooths the values into a conservative estimate of the GRTT 🧮. The algorithm and methodology are described in the NORM Building Block document [4], in the section called "One-to-Many Sender GRTT Measurement" 📖. A conservative estimate helps feedback suppression, at a small cost in overall protocol repair delay ✨. The sender's current GRTT estimate is advertised in the grtt field found in all NORM sender messages 📬. The advertised GRTT is also limited to a minimum of the nominal inter-packet transmission time, given the sender's current transmission rate and system clock granularity ⏳. The reason for this extra limit is to keep the receiver somewhat "event driven" 🎀. It makes sure the sender has had enough time to generate any response to repair requests from receivers, given the transmit rate limits from congestion control or configuration 🌸.
When the NORM-CC Rate header extension is present in NORM_CMD(CC) messages, receivers respond to them as described in Section 5.5.2, "NORM Congestion Control Operation" 🚀. The sender generates NORM_CMD(CC) messages periodically, as described for congestion control operation 🔁. This gives proactive but controlled feedback from the group, in the form of NORM_ACK messages 💖. It also gives GRTT feedback even when no NORM_NACK messages are being sent 🫶.
If you're operating without congestion control in a closed network, the NORM_CMD(CC) messages may be sent periodically without the NORM-CC Rate header extension 🏡. In that case, receivers only provide GRTT measurement feedback when NORM_NACK messages are generated, since no NORM_ACK messages are generated 🙈. The NORM_CMD(CC) messages may be sent less frequently, perhaps as little as once per minute, to conserve network capacity 🔋. Note that the NORM-CC Rate header extension may also be used to proactively solicit RTT feedback from the receiver group, per congestion control operation, even if the sender isn't doing congestion control rate adjustment 🦋. NORM operation without congestion control should be considered only in closed networks 🔒.
5.5.2. NORM Congestion Control Operation
Alright, this section is the baseline congestion control for NORM, called NORM-CC 🚦. The supporting NORM message formats and approach here are an adaptation of the equation-based TCP-Friendly Multicast Congestion Control (TFMCC) approach described in [19] 🧪. This congestion control scheme is REQUIRED for operation within the general Internet, unless the NORM implementation is adapted to use another IETF-sanctioned reliable multicast congestion control mechanism (for example, PGMCC [20]) 👑. With this TFMCC-based approach, NORM senders control their transmissions in a rate-based manner, as opposed to the window-based congestion control algorithms in TCP 🪟. However, the NORM protocol message set may alternatively be used to support a window-based multicast congestion control scheme such as PGMCC 🔀. The details of that alternative may be described separately or in a future revision of this document 📝.
In either case (rate-based TFMCC or window-based PGMCC), successful control of sender transmission depends on collecting sender-to-receiver packet loss estimates and RTTs. That's how you identify the congestion control bottleneck path(s) within the multicast topology and adjust the sender rate to match 🎯. The receiver whose loss and RTT estimates correspond to the lowest resulting transmission rate is identified as the "current limiting receiver" (CLR) 🐢.
As described in [21], a steady-state sender transmission rate that stays "friendly" with competing TCP flows can be calculated as follows 🧮:
S
Rsender = --------------------------------------------------------------
tRTT * (sqrt((2/3)*p) + 12 * sqrt((3/8)*p) * p *
(1 + 32*(p^2)))where S is the nominal transmitted packet size 📏. In NORM, the "nominal" packet size can be determined by the sender as an exponentially weighted moving average (EWMA) of transmitted packet sizes, to account for variable message sizes 💅.
tRTT = The RTT estimate of the current "current limiting receiver" (CLR).
p = The loss event fraction of the CLR.
Okay bestie, so the sender keeps sending NORM_CMD(CC) command messages on a regular schedule 🦋 These are mixed in with the normal data and repair packets, and they're doing three jobs:
- Pulling explicit feedback out of the whole receiver set so the sender can gather congestion control info 🎀
- Telling everyone the sender's current congestion control status, including who the CLR is 👑
- Asking the CLR for instant feedback so it can keep up with congestion control dynamics on the worst path in the multicast group 💅
The format of the NORM_CMD(CC) message is described in Section 4.2.3 of this document. It carries what's needed to measure RTTs, announce the congestion control CLR, and hand out individual RTT measurements to receivers in the group ✨ It also lets the sender ask for feedback from OPTIONAL "potential limiting receiver" (PLR) nodes. These are receivers that might soon become the CLR, so getting their up-to-date feedback really helps congestion control perform 🌸 How PLRs get picked isn't covered in this document, so that's a whole other story 🙈
5.5.2.1. NORM_CMD(CC) Transmission ✨
The sender sends NORM_CMD(CC) messages periodically, right alongside its regular data 🌈 Fun fact: the repeated sending can kick off a while before any user data goes out at session startup 🧚 That head start helps the sender get a feel for the multicast topology, both group RTT and individual RTT and congestion control state 🔥
A NORM_CMD(CC) message goes out right away when the sender starts up 💖 After that, here's how the spacing gets decided:
- By default, the gap is based on the sender's current GRTT estimate. A startup GRTT of 0.5 seconds is recommended when no feedback has come in from the group yet ⭐
- If a CLR has been identified from earlier receiver feedback, the gap becomes the RTT between the sender and the CLR 🎀
- And if the normal data message interval is longer than the GRTT or RTT_clr interval, the NORM_CMD(CC) gap gets bumped up to that longer value. This makes sure control messages don't crowd out actual user data 🫶
The cc_sequence field goes up by one every time a NORM_CMD(CC) command is sent 🦋 Receivers put the highest cc_sequence they've recently seen into their feedback, so the sender can tell how old that feedback is. That's how it avoids acting on stale info during congestion avoidance 🧠
The NORM-CC Rate Header Extension is attached to the NORM_CMD(CC) message, and the sender reports its current transmission rate in the send_rate field 📈 Receivers use that number to seed their loss estimation when congestion control starts up or restarts 💜
The cc_node_list is a list of entries for receivers, showing each one's congestion control state (its flags, rtt, and loss estimates) 💅 It can be empty if the sender hasn't heard from anyone yet. Once feedback arrives, the list will have at least one entry, the CLR 👑 The NORM_FLAG_CC_CLR value in cc_flags marks the CLR's entry. It is RECOMMENDED that the CLR entry be first in the list, because that's more efficient to implement 🌟
Any extra entries are there to hand the sender's own individual RTT measurements back to receivers 🦋 How many extra entries fit depends on what percentage of traffic the sender app is okay spending on control messages compared to user data 🍓 More entries can help the sender react faster to congestion control dynamics ⚡ The list length can change dynamically based on the current transmission rate and how often NORM_CMD(CC) messages get scheduled 🌸 The maximum length matches the session's NormSegmentSize parameter 💖 Extra entries based on receiver feedback get picked using these rules:
- Receivers that haven't been given RTT feedback yet go first. Among those, receivers with the highest loss fraction get priority for a spot in the list 🔥
- After that, receivers that already got RTT feedback are included, with the ones reporting the lowest calculated congestion rate getting priority 🎀
There are other cc_flag values beyond NORM_FLAG_CC_CLR for other congestion control jobs 🧚 The NORM_FLAG_CC_PLR flag marks extra receivers that the sender wants immediate, non-suppressed feedback from 👑 These could be receivers the sender figured out algorithmically might become future CLRs, or ones that were preconfigured as likely congestion control points in the network ✨ The NORM_FLAG_CC_RTT flag says whether the cc_rtt field for that receiver is valid 💅 Usually it'll be set, since the receivers in the list are normally ones that have already sent feedback 🫶 But if the NORM sender was preconfigured with a set of PLR nodes, feedback from them might not have been collected yet. In that case the flag isn't set, and cc_rtt and cc_rate don't hold valid values 🙈
5.5.2.2. NORM_CMD(CC) Feedback Response 🦋
okay bestie, so receivers answer NORM_CMD(CC) messages by sending a NORM_ACK(RTT) message back 💌. The whole point of this feedback is to figure out which receivers have the lowest congestion control rates 👑. Receivers marked as CLR or PLR nodes in the "cc_node_list" of a NORM_CMD(CC) jump in right away and send a NORM_ACK to that message 💅. When a NORM_CMD(CC) arrives, every non-CLR and non-PLR node starts a random feedback backoff timeout 😭. It's giving the same energy as the backoff a receiver uses when it starts a repair cycle after spotting data loss (see Section 5.3) 🧚. The backoff timeout for the congestion control response is worked out like this:
T_backoff = RandomBackoff(K*GRTTsender, GSIZEsender)"RandomBackoff()" gives you a truncated exponentially distributed random number, and it's explained in the NORM Building Block document [4] 🌈. The same backoff factor K = Ksender MAY be used as with NORM_NACK suppression. BUT, if the app deliberately picks a super small Ksender backoff factor to cut down NACK repair latency (trading away some group size scalability), it can still be a good idea to keep a bigger backoff factor for congestion control feedback 🍓. Congestion control feedback usually shows up in bigger numbers than NACKs, and its latency is often fine even when reliable delivery latency isn't 🫶. As mentioned earlier, a backoff factor value of K = 4 is generally recommended for ASM operation, and K = 6 for SSM operation ⭐.
A receiver SHALL cancel the backoff timeout, and therefore its pending NORM_ACK(RTT) transmission, under the following conditions:
- The receiver generates another feedback message (
NORM_NACKor otherNORM_ACK) before the congestion control feedback timeout expires 🙈. - A
NORM_CMD(CC)or other receiver feedback with an ordinally greater"cc_sequence"field value is received before the congestion control feedback timeout expires. This is similar to the TFMCC feedback round number 🔥. - When the T_backoff is greater than 1*GRTT. This stops NACK implosion if the sender or the network fails 💀.
- "Suppressing" congestion control feedback is heard from another receiver (in a
NORM_ACKorNORM_NACK) or via aNORM_CMD(REPAIR_ADV)message from the sender. The local receiver's feedback is "suppressed" if the rate of the competing feedback (Rfb) is sufficiently close to or less than the local receiver's calculated rate (Rcalc). The local receiver's feedback gets canceled whenRcalc > (0.9 * Rfb)🦋.
Also, receivers that haven't gotten an RTT measurement from the sender yet only get suppressed by other receivers that also haven't measured RTT yet 🌸. Receivers whose RTT estimate has "aged" a lot (aka they haven't been in the NORM_CMD(CC) "cc_node_list" in a long time) might want to compete as a receiver with no prior RTT measurement once some expiration period has passed 👀.
When the backoff timer expires, the receiver SHALL generate a NORM_ACK(RTT) message to give feedback to the sender and the group 💖. This message can be multicast to the group for the best suppression in ASM topologies, or unicast to the sender, depending on how NORM is deployed and configured 🧚.
Whenever any feedback gets generated (including this NORM_ACK(RTT)), receivers put an adjusted version of the sender timestamp from the most recent NORM_CMD(CC) message, plus that command's "cc_sequence" value, into the matching NORM_ACK or NORM_NACK fields ✨. For NORM-CC operation, any generated feedback message SHALL also contain the NORM-CC Feedback header extension 🎀. The receiver shares its current "cc_rate" estimate, "cc_loss" estimate, "cc_rtt" if it knows it, and any applicable "cc_flags" through that header extension 💜.
During slow start (when the receiver hasn't detected loss from the sender yet), the receiver puts two times its measured rate from the sender into the "cc_rate" field 🚀. For steady-state congestion control operation, the receiver's "cc_rate" comes from the equation-based value using its current loss event estimate and the sender<->receiver RTT info. (The GRTT is used when the receiver hasn't measured its own individual RTT yet.) 🌈
The "cc_loss" field reflects the receiver's current loss event estimate with respect to the sender in question 🍓.
When the receiver has a valid individual RTT measurement, it SHALL include that value in the "cc_rtt" field 🫶. The NORM_FLAG_CC_RTT MUST be set when the "cc_rtt" field is valid 🔥.
After a congestion control feedback message gets generated, or when the feedback is suppressed, a non-CLR receiver starts a "holdoff" timeout period. During that time it holds back from giving congestion control feedback, even if NORM_CMD(CC) messages come in from the sender (unless the receiver gets marked as a CLR or PLR node) 🙅♀️. The holdoff timeout (T_ccHoldoff) period is:
T_ccHoldoff = (K*GRTT)So non-CLR receivers are limited to giving explicit congestion control feedback once per K*GRTT interval 🕰️. But here's the tea: as the session goes on, different receivers answer different NORM_CMD(CC) messages, so there's pretty much constant feedback of congestion control info while the sender is active 💅✨.
5.5.2.3. Congestion Control Rate Adjustment 🎀
During steady-state operation, the sender directly adjusts its transmission rate to match the rate from its currently selected CLR's feedback 💫. As noted in [19], the CLR's parameter estimates (loss and RTT) usually keep rate changes inside acceptable bounds 🧁. For rate increases, the sender SHALL observe a maximum rate of increase of one packet per RTT at all times during steady-state operation 🐢.
The sender processes congestion control feedback from the receivers and picks the CLR based on the lowest rate receiver 👑. Receiver rates are either taken straight from the slow start "cc_rate" that the receiver provided in the NORM-CC Feedback header extension, or calculated with the equation-based method using each receiver's individual RTT and loss estimates ("cc_loss") as feedback comes in 📈.
The sender can work out a current RTT for a receiver (RTT_rcvrNew) using the "grtt_response" timestamp included in feedback messages 🕰️. When the "cc_rtt" value in a response isn't valid, the sender just uses this RTT_rcvrNew value as the receiver's current RTT (RTT_rcvr) 🙃. For non-CLR and non-PLR receivers, the sender can use the "cc_rtt" value from the NORM-CC Feedback header extension as the receiver's previous RTT measurement (RTT_rcvrPrev), and smooth it like this:
RTT_rcvr = 0.5 * RTT_rcvrPrev + 0.5 * RTT_rcvrNewFor CLR receivers, where feedback comes in more regularly, the sender SHOULD keep a more smoothed RTT estimate each time new feedback arrives from the CLR, like this:
RTT_clr = 0.9 * RTT_clr + 0.1 * RTT_clrNew"RTT_clrNew" is the new RTT calculated from the timestamp in the feedback message received from the CLR 🦋. RTT_clr gets set to RTT_clrNew on the first feedback message received 🌱. The sender follows the same procedure for PLR receivers too, and if a PLR gets "promoted" to CLR status, the smoothed estimate can keep going 💕.
There are some extra periods besides steady-state operation that NORM-CC has to think about 🧠. They are:
- during session startup,
- when no feedback is received from the CLR, and
- when the sender has a break in data transmission.
During session startup, the congestion control operation SHALL observe a "slow start" procedure to quickly get close to its fair bandwidth share 🏃♀️💨. An initial sender startup rate is assumed as:
Rinitial = MIN(NormSegmentSize / GRTT, NormSegmentSize) bytes/second.The rate only goes up when feedback comes in from the receiver set 📶. The "slow start" phase keeps going until any receiver reports that loss has happened 🛑. Rate increases during slow start are applied as:
Rnew = Rrecv_minwhere "Rrecv_min" is the minimum reported receiver rate in the "cc_rate" field of congestion control feedback messages received from the group 🍭. During "slow start", receivers put two times their measured rate from the sender into the "cc_rate" field of their feedback, so keep that in mind 🧮. Rate increase adjustments are limited to once per GRTT during slow start ⏱️.
If the CLR or any receiver plans to leave the group, it sets the NORM_FLAG_CC_LEAVE flag in its congestion control feedback message. That tells the sender not to pick it as the CLR 🚪. When the CLR switches to a lower rate receiver, the sender should adjust to the new lower rate right away 🏎️. The sender is limited to increasing its rate by one additional packet per RTT toward any new, higher CLR rate 🐌.
The sender should also keep track of the "age" of the feedback it got from the CLR by comparing its current "cc_sequence" value (Seq_sender) to the last "cc_sequence" value received from the CLR (Seq_clr) 🔍. As the CLR feedback gets older with no new feedback, the sender SHALL begin reducing its rate once per RTT_clr as a congestion avoidance measure 📉. This is the algorithm for how much to cut the sender rate (Rsender bytes/sec) when the CLR feedback unexpectedly and excessively ages:
Age = Seq_sender - Seq_clr;
if (Age > 4) Rsender = Rsender * 0.5;This rate reduction is limited to the lower bound on NORM transmission rate 🧱. After NORM_ROBUST_FACTOR consecutive NORM_CMD(CC) rounds with no feedback from the CLR, the sender SHOULD assume the CLR has left the group and pick the receiver with the next lowest rate as the new CLR 🫡. This assumes the sender has no explicit knowledge that the CLR intentionally left the group 🤷♀️. If no receiver feedback comes in at all, the sender MAY wish to hold off on sending more NORM_DATA segments and only keep NORM_CMD(CC) transmissions going until feedback shows up 🫠. After a CLR timeout like that, the sender is transmitting at a minimal rate and should go back into slow start, as described here for a break in data transmission 🔄.
When the sender has a break in its data transmission, it can keep probing the group with NORM_CMD(CC) messages to keep collecting RTTs from the group 📡. That way the sender can quickly pick a good CLR when data transmission restarts 🏁. But the sender should exponentially reduce its target rate, the one used to restart transmission, as time passes during the break ⏳. The target rate SHOULD be recalculated once per RTT_clr like this:
Rsender = Rsender * 0.5;If the minimum NORM rate is reached, the sender should set the NORM_FLAG_START flag in its NORM_CMD(CC) messages when transmission restarts, and the group should observe "slow start" congestion control procedures until any receiver experiences a new loss event 🌟.
5.5.3. NORM Positive Acknowledgment Procedure ✨
Okay so this is where the sender asks the group "did you actually get that?" 💅 NORM lets the source app request positive acknowledgment (ACK) of NORM_CMD(FLUSH) and NORM_CMD(ACK_REQ) messages from group members. NORM has some specific acknowledgment requests already defined, and there's a whole range of acknowledgment request types the app gets to define itself 🎀
One predefined type is the NORM_ACK_FLUSH type. It checks whether receivers have finished reliable reception up through a specific logical transmission point in the sender's sequence of transmission. It's giving flow control vibes 🌊 because it can help the app manage flow when the sender knows about part of the receiver set. The other predefined type is NORM_ACK(CC), which gives congestion control feedback in response to NORM_CMD(CC) messages the sender sends for NORM-CC operation. Heads up, though: NORM_ACK(CC) does NOT follow the positive acknowledgment procedure described here. Different lane, bestie 🦋
The NORM_CMD(ACK_REQ) and NORM_ACK messages have an "ack_type" field that says which type of acknowledgment is being requested and provided. A range of "ack_type" values is set aside for application-defined use 💜 The app is in charge of starting the acknowledgment request and figuring out its own custom "ack_type" values, but the acknowledgment procedure SHOULD be conducted inside the protocol implementation so it can use the timing and transmission scheduling info the NORM transport already has ✨
The positive acknowledgment procedure uses polling by the sender to query the receiver group for a response. Not gonna lie, this polling is not designed to scale to very large receiver groups 😭 but it could work in a large group setting to query a critical subset of the group 🧚 Either NORM_CMD(ACK_REQ), or NORM_CMD(FLUSH) when that applies, is used for polling. Both contain a list of NormNodeIds for the receivers that should respond to the command. The app decides who goes on that list, either with "a priori" knowledge of participating nodes or through some other application-level mechanism 🍓
The ACK process gets kicked off by the sender, who generates NORM_CMD(FLUSH) or NORM_CMD(ACK_REQ) messages in periodic "rounds" 🔁 For NORM_ACK_FLUSH requests, the NORM_CMD(FLUSH) contains an object_transport_id and a fec_payload_id that mark the watermark transmission point being asked about. The receiver "echoes" that watermark point back in the matching fields of the NORM_ACK(FLUSH) it sends in reply. NORM_CMD(ACK_REQ) messages have an ack_id field that gets echoed the same way, so the sender can match each response to the right request 🫶
When receivers get a NORM_CMD(ACK_REQ), the listed receivers spread their NORM_ACK messages randomly and uniformly over a window of (1*GRTT). These NORM_ACK messages are typically unicast to the sender. (Quick aside: NORM_ACK(CC) messages SHALL be multicast or unicast in the same manner as NORM_NACK messages 📡)
The ACK process is self-limiting, which is how it avoids ACK implosion 🙌 Here's how:
- Only a single NORM_CMD(ACK_REQ) message is generated once per (2*GRTT), and,
- The size of the
"acking_node_list"of NormNodeIds that acknowledgment is requested from is limited to a maximum of the sender's NormSegmentSize setting per round of the positive acknowledgment process.
Because that list is capped at the sender's NormSegmentSize, it can take multiple NORM_CMD(ACK_REQ) rounds to hear back from every receiver on the list 🔄 The attached NormNodeId list keeps getting updated as the process moves along and NORM_ACK responses come in from the receiver set. Every time the sender gets a valid response (meaning a matching watermark point or ack_id), it SHALL remove those receivers from the next NORM_CMD(ACK_REQ) "acking_node_list" and add in any pending receiver NormNodeIds, all while staying inside the NormSegmentSize limit on list size 💖 Each receiver only gets queried a maximum number of times (NORM_ROBUST_FACTOR, by default). If a receiver still hasn't responded after that many repeated requests, it gets dropped from the payload list to make room for other receivers waiting to be acknowledged 👋 NORM_CMD(ACK_REQ) keeps getting sent until no more responses are needed or the repeat threshold has been passed for every pending receiver 🛑
Sending NORM_CMD(ACK_REQ) or NORM_CMD(FLUSH) messages for the positive acknowledgment process is multiplexed with the sender's ongoing data transmissions, so the two share the line 🎀 The NORM_CMD(FLUSH) positive acknowledgment process can get interrupted when negative acknowledgment repair requests (NACKs) come in from receivers during the acknowledgment period. Once any repairs have been transmitted, the NORM_CMD(FLUSH) positive acknowledgment process restarts for the receivers still pending acknowledgment 🔁
With NORM_CMD(FLUSH) commands that have an attached "acking_node_list", receivers will not ACK until they've received the complete transmission of all data up to and including the given watermark transmission point 🧐 All receivers SHALL interpret the watermark point provided in the request NACK for repairs if needed as for NORM_CMD(FLUSH) commands with no attached "acking_node_list" 🌸
5.5.4. Group Size Estimate 🧮
NORM sender messages carry a "gsize" field, which is a representation of the group size. It's used to scale the random backoff timer ranges. The group size estimate doesn't need to be precise, and it works reasonably well as long as it's within an order of magnitude of the actual group size 💅 By default, the NORM sender's group size estimate may be administratively configured. Also, since NORM is expected to scale well for general use, a default value of 10,000 is recommended for use as the group size estimate 🌈
It's possible that the group size could be algorithmically approximated from the volume of congestion control feedback messages that follow the exponentially weighted random backoff 🤓 However, a specification for such an algorithm is currently beyond the scope of this document, so that's a later problem 🙈
6. Security Considerations 🔐
Okay, this part is serious, so listen up 😤 The same security considerations that apply to the NORM and FEC Building Blocks also apply to the NORM protocol. On top of the vulnerabilities that any IP and IP multicast protocol implementation may generally be subject to, NORM's NACK-based feedback can be exploited by replay attacks that force the NORM sender to transmit repair information it didn't need to send 💀 This MAY be addressed by network layer IP security implementations that guard against this potential exploitation. It is RECOMMENDED that such IP security mechanisms be used when they're available 🛡️
Another possible approach is for NORM senders to use the "sequence" field from the NORM Common Message Header to detect replay attacks. This works if the NORM packets are cryptographically protected and the sender is willing to keep state on the receivers that are NACKing. A cache of receiver state may provide some protection against replay attacks 🗂️ Also, the "sequence" field of NORM messages should be incremented with independent values for different destinations (for example, group-addressed versus unicast-addressed messages versus "receiver" messages). That way, the congestion control loss estimation function of the "sequence" field can be preserved for sender messages when receiver messages are unicast to the sender 🧠
NORM is compatible with the use of the IP security (IPsec) architecture described in [22]. Important note, though: even though NORM uses FEC-based repair for scalability, that alone does NOT guarantee the integrity of received data 🚨 Application-level integrity-checking of data content is highly RECOMMENDED. Please don't skip this one, bestie 🫶
7. IANA Considerations 📋
No information in this specification is currently subject to IANA registration. However, several Header Extensions are defined within this document. If and when additional Header Extensions are developed, the first RFC MUST establish an IANA registry for them, with a "Specification Required" policy [6], and all Header Extensions, including those in the present document, MUST be registered thereafter 🗃️ Additionally, building blocks components used by NORM may introduce additional IANA considerations. In particular, the FEC Building Block used by NORM does require IANA registration of the FEC codecs it uses. The registration instructions for FEC codecs are provided in [5] 🧾
8. Suggested Use 🌟
The present NORM protocol is seen as a useful tool for reliable data transfer over generic IP multicast services 🎯 The authors didn't intend to suggest it's suitable for supporting all envisioned multicast reliability requirements, so don't go overboard with it 😅 NORM provides a simple and flexible framework for multicast applications with a degree of concern for network traffic implosion and protocol overhead efficiency. NORM-like protocols have already been successfully demonstrated within the MBone for bulk data dissemination applications, including weather satellite compressed imagery updates serving a large group of receivers and a generic web content reliable "push" application 🛰️
This framework approach also has some design features that make it attractive for bulk transfer in asymmetric and wireless internetwork applications 📶 NORM can successfully operate independent of network structure, and in environments with high packet loss, delay, and misordering. Hybrid proactive/reactive FEC-based repairing can improve protocol performance in some multicast scenarios 💫 A sender-only repair approach often makes additional engineering sense in asymmetric networks. NORM's unicast feedback capability may be suitable for asymmetric networks, or for networks where only a unidirectional multicast routing/delivery service exists 🚀 Asymmetric architectures that support multicast delivery are likely to make up an important portion of the future Internet structure (for example, DBS/cable/PSTN hybrids). Efficient, reliable bulk data transfer will be an important capability for serving large groups of subscribed receivers 🧚♀️
9. Acknowledgments (and these are not Negative) 🥰
The authors would like to thank Rick Jones, Vincent Roca, Rod Walsh, Toni Paila, Michael Luby, and Joerg Widmer for their valuable input and comments on this document 🫶 They'd also like to thank the RMT working group chairs, Roger Kermode and Lorenzo Vicisano, for their support in developing this specification, and Sally Floyd for her early input into this document 👑
10. References 📚
10.1. Normative References 📚
These are the must-read ones bestie, the stuff you literally can't skip 💅
- [1] Kermode, R. and L. Vicisano, "Author Guidelines for Reliable Multicast Transport (RMT) Building Blocks and Protocol Instantiation documents", RFC 3269, April 2002.
- [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [3] Deering, S., "Host Extensions for IP Multicasting", STD 5, RFC 1112, August 1989.
- [4] Adamson, B., Bormann, C., Handley, M., and J. Macker, "Negative-Acknowledgment (NACK)-Oriented Reliable Multicast (NORM) Building Blocks", RFC 3941, November 2004.
- [5] Luby, M., Vicisano, L., Gemmell, J., Rizzo, L., Handley, M., and J. Crowcroft, "Forward Error Correction (FEC) Building Block", RFC 3452, December 2002.
- [6] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998.
10.2. Informative References 🌸
These ones are extra reading for when you're feeling curious. Totally optional, but so worth it 🧚
- [7] Handley, M. and V. Jacobson, "SDP: Session Description Protocol", RFC 2327, April 1998.
- [8] Handley, M., Perkins, C., and E. Whelan, "Session Announcement Protocol", RFC 2974, October 2000.
- [9] S. Pingali, D. Towsley, J. Kurose, "A Comparison of Sender-Initiated and Receiver-Initiated Reliable Multicast Protocols", In Proc. INFOCOM, San Francisco CA, October 1993.
- [10] Luby, M., Vicisano, L., Gemmell, J., Rizzo, L., Handley, M., and J. Crowcroft, "The Use of Forward Error Correction (FEC) in Reliable Multicast", RFC 3453, December 2002.
- [11] Macker, J. and B. Adamson, "The Multicast Dissemination Protocol (MDP) Toolkit", Proc. IEEE MILCOM 99, October 1999.
- [12] Nonnenmacher, J. and E. Biersack, "Optimal Multicast Feedback", Proc. IEEE INFOCOMM, p. 964, March/April 1998.
- [13] J. Macker, B. Adamson, "Quantitative Prediction of Nack Oriented Reliable Multicast (NORM) Feedback", Proc. IEEE MILCOM 2002, October 2002.
- [14] H.W. Holbrook, "A Channel Model for Multicast", Ph.D. Dissertation, Stanford University, Department of Computer Science, Stanford, California, August 2001.
- [15] D. Gossink, J. Macker, "Reliable Multicast and Integrated Parity Retransmission with Channel Estimation", IEEE GLOBECOMM 98', September 1998.
- [16] Whetten, B., Vicisano, L., Kermode, R., Handley, M., Floyd, S., and M. Luby, "Reliable Multicast Transport Building Blocks for One-to-Many Bulk-Data Transfer", RFC 3048, January 2001.
- [17] Mankin, A., Romanow, A., Bradner, S., and V. Paxson, "IETF Criteria for Evaluating Reliable Multicast Transport and Application Protocols", RFC 2357, June 1998.
- [18] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.
- [19] J. Widmer and M. Handley, "Extending Equation-Based Congestion Control to Multicast Applications", Proc ACM SIGCOMM 2001, San Diego, August 2001.
- [20] L. Rizzo, "pgmcc: A TCP-Friendly Single-Rate Multicast Congestion Control Scheme", Proc ACM SIGCOMM 2000, Stockholm, August 2000.
- [21] J. Padhye, V. Firoiu, D. Towsley, and J. Kurose, "Modeling TCP Throughput: A Simple Model and its Empirical Validation", Proc ACM SIGCOMM 1998.
- [22] Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998.
11. Authors' Addresses ✨
Here's where you find the humans behind the spec, so you can slide into their inbox if you have questions 💌
Brian Adamson
Naval Research Laboratory
Washington, DC, USA, 20375
EMail: adamson@itd.nrl.navy.mil
Carsten Bormann
Universitaet Bremen TZI
Postfach 330440
D-28334 Bremen, Germany
EMail: cabo@tzi.org
Mark Handley
Department of Computer Science
University College London
Gower Street
London
WC1E 6BT
UK
EMail: M.Handley@cs.ucl.ac.uk
Joe Macker
Naval Research Laboratory
Washington, DC, USA, 20375
EMail: macker@itd.nrl.navy.milFull Copyright Statement 🎀
Copyright (C) The Internet Society (2004).
So this doc belongs to the Internet Society, but the authors still keep all their own rights, and the whole thing is under the BCP 78 rules. 🦋
The document and everything in it is handed over "AS IS", no guarantees. The contributor, the organization they represent or are sponsored by (if any), the Internet Society, and the IETF all say no to every warranty, express or implied. That includes no promise that using the info won't infringe anyone's rights, and no implied promise of merchantability or fitness for a particular purpose. 🙈
Intellectual Property 💜
The IETF doesn't take a side on whether any patents or other IP rights that might cover this tech are valid or how far they reach, and it doesn't say whether you could get a license for them. It also hasn't checked for any rights on its own. If you want the IETF's rules on rights in its documents, check out BCP 78 and BCP 79. 📖
Any IPR disclosures sent to the IETF Secretariat are available, along with any licensing assurances, or the results of any attempt to get a general license for implementers or users. You can find all of that in the IETF online IPR repository at http://www.ietf.org/ipr.
The IETF also invites anyone to flag copyrights, patents, patent applications, or other proprietary rights that might cover tech needed to implement this standard. Send that to the IETF at ietf-ipr@ietf.org 🧚
Acknowledgement 🌈
Funding for the RFC Editor function is currently provided by the Internet Society. 🫶
📦 where this page comes from
- ietf: rfc3940 2004-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms