RFC 1301

Multicast Transport Protocol

๐Ÿ“… published February 1992, from ietf

RFC 1301: Multicast Transport Protocol โœจ

Okay bestie, this is RFC 1301, titled Multicast Transport Protocol ๐Ÿฆ‹ It was published in February 1992 by S. Armstrong (Xerox), A. Freier (Apple), and K. Marzullo (Cornell). It's an informational doc, so it's a "here's how it works" memo rather than a formal standard. The header doesn't list anything it obsoletes or updates, so it's a standalone vibe ๐Ÿ’…

Quick note before we start: this rewrite is in 9 parts, so this is just part 1. More is coming, so stay tuned ๐Ÿซถ


Status of This Memo ๐Ÿ“œ

Okay so this memo gives information to the Internet community. It is NOT a standard, it doesn't specify an Internet standard at all ๐Ÿ™ˆ And distribution of it is unlimited, so share it with everyone ๐ŸŒˆ

Summary ๐ŸŒธ

This memo describes a protocol for reliable transport that uses the multicast capability of the lower layer networking architectures. Slay, right? ๐Ÿ’– The transport definition lets an arbitrary number of transport providers run realtime collaborations, and the networking clients (aka the applications) don't need to know the details of who's in the group or where they're located ๐Ÿงš It's not tied to one specific network architecture, BUT it does implicitly need some form of multicasting (or broadcasting) at the data link level, plus a way to tell the transport layer above that the capability exists ๐Ÿ”ฅ

Keywords: reliable transport, multicast, broadcast, collaboration, networking.

Table of Contents ๐Ÿ—‚๏ธ

           1. Introduction                                     2
           2. Protocol description                             3
           2.1 Definition of terms                             3
           2.2 Packet format                                   6
           2.2.1. Protocol version                             7
           2.2.2. Packet type and modifier                     7
           2.2.3. Subchannel                                   9
           2.2.4. Source connection identifier                 9
           2.2.5. Destination connection identifier           10
           2.2.6. Message acceptance                          10
           2.2.7. Heartbeat                                   12
           2.2.8. Window                                      12
           2.2.9. Retention                                   12
           2.3 Transport addresses                            12
           2.3.1. Unknown transport address                   12
           2.3.2. Web's multicast address                     13
           2.3.3. Member addresses                            13
           3. Protocol behavior                               13
           3.1. Establishing a transport                      13
           3.1.1. Join request                                14
           3.1.2. Join confirm/deny                           16
           3.2 Maintaining data consistency                   17
           3.2.1. Transmit tokens                             17
           3.2.2. Data transmission                           20
           3.2.3. Empty packets                               23
           3.2.4. Missed data                                 26
           3.2.5. Retrying operations                         26
           3.2.6. Retransmission                              27
           3.2.7. Duplicate suppression                       29
           3.2.8. Banishment                                  29
           3.3 Terminating the transport                      29
           3.3.1. Voluntary quits                             30
           3.3.2. Master quit                                 30
           3.3.3. Banishment                                  30
           3.4 Transport parameters                           30
           3.4.1. Quality of service                          30
           3.4.2. Selecting parameter values                  31
           3.4.3. Caching member information                  33
           A. Appendix: MTP as an Internet Protocol transport 34
           A.1 Internet Protocol multicast addressing         34
           A.2 Encapsulation                                  35
           A.3 Fields of the bridge protocol                  35
           A.4 Relationship to other Internet Transports      36
           References                                         36
           Footnotes                                          37
           Security Considerations                            37
           Authors' Addresses                                 38

1. Introduction ๐ŸŽ€

So this document describes a flow controlled, atomic multicasting transport protocol, which is called MTP for short ๐Ÿ’œ The whole point is to give you enough info to actually implement it, no guesswork ๐Ÿ’…

The MTP design was influenced by a big pile of networking and distributed systems literature and tech from the last decade and a half ๐Ÿ“š Some of the main sources are [Xer81], [BSTM79] and [Pos81] for transport design, and [Bog83] and [DIX82] for general broadcast and multicast concepts. [CLZ87] influenced MTP's retransmission mechanisms, and [Fre84] influenced the transport timings โฑ๏ธ MTP over IP uses mechanisms described in [Dee89]. MTP's ordering and agreement protocols were shaped by work in [CM87], [JB89] and [Cri88]. And if you want the philosophy and motivation behind MTP, check out [AFM91] ๐Ÿฆ‹

2. Protocol description ๐ŸŒท

Okay so MTP is a transport, which means it's a client of the network layer (as defined by the OSI networking model) [1] ๐Ÿซถ It gives reliable delivery of client data between one or more communicating processes, plus a predefined principal process. A collection of these processes is called a web ๐Ÿ•ธ๏ธ

Beyond just moving data reliably and efficiently, MTP also handles the synchronization web members need to agree on the order of receipt of all messages. It can even agree on delivery of a message in the face of partitions ๐Ÿ˜ญ This ordering and agreement protocol runs on serialized tokens that the master hands out to producers ๐Ÿ‘‘

The processes can have one of three levels of capability. One member has to be the master ๐Ÿ‘‘ The master sets up the web and controls how it behaves, including its membership and performance. The non-master members can be either producer/consumers or pure consumers. Producer/consumers are allowed to send user data to the whole membership (and they're expected to hear their own messages too ๐Ÿ™ˆ), while pure consumers are NOT allowed to send user data at all ๐Ÿ’€

MTP is a negative acknowledgement protocol, and it leans on how reliable local area and wide area networks are today โœจ Successful delivery is accepted silently by the consuming stations, instead of telling the producing process "hey, got it." That cuts down on the reverse traffic needed to stay in sync ๐Ÿงš

2.1 Definition of terms ๐Ÿ“–

These terms are used all over this document, and they're defined here so nobody gets confused ๐Ÿค“

  • consumer: A consumer is a transport that can ONLY receive user data ๐ŸŽง It can send control packets, like negative acknowledgements, but it may never send any requests for the transmit token, or any form of data or empty messages.
  • heartbeat: A heartbeat is an interval of time, nominally measured in milliseconds โณ It's a key parameter in the transport's state, and the transport's client can adapt it to get the quality of service it wants ๐ŸŒธ
  • master: The master is the principal member of the web ๐Ÿ‘‘ The master capability is a superset of a producer member. Its main job is handing out transmit tokens to members who want to send data, and overseeing the web's membership and operational parameters ๐Ÿ’…
  • member: A web member is any process that the master has let join the web, plus the master itself ๐Ÿซ‚
  • membership class: Every member is classified by what it intends to do when it joins the web. The membership classes are consumer, producer and master, and each one is a formal superset of the one before it ๐ŸŒˆ
  • message: An MTP message is a concatenation of the user data portions of a series of data packets, where the last packet in the series carries an end of message indication ๐Ÿ’Œ A message can have any number of bytes of user data, including zero ๐Ÿ“
  • NSAP: The network service access point. This is the network address, or the node address of the machine, where a service is available ๐Ÿ“
  • producer: A producer is a class of membership that's a formal superset of a consumer ๐Ÿ”ฅ It's permitted (and expected) to send client data, AND it consumes data sent by other producers ๐Ÿฆ‹
  • retention: Retention is one of the three fundamental parameters that make up the transport's state (along with heartbeat and window) ๐Ÿ’œ It's a number of heartbeats, and even though it applies in several different situations, it's mainly the number of heartbeats a producing client has to keep buffered data around, in case that data needs to be retransmitted ๐Ÿ”
  • token: To transmit, a producer must first be in possession of a token ๐ŸŽŸ๏ธ Tokens are granted ONLY by the master and include the message sequence number. That makes them fundamental to how the ordering and agreement protocol works in MTP โœจ
  • TSAP: The transport service access point. This is the address that uniquely defines a particular instantiation of a service ๐ŸŽ€ TSAPs are formed by logically concatenating the node's NSAP with a transport identifier (and maybe a packet/protocol type) ๐Ÿงฉ
  • user data: User data is the client information carried in MTP data packets, and the transport treats it as uninterpreted octets ๐Ÿซง The end of message and subchannel indicators are also treated as user data ๐Ÿ’…
  • web: A web is a collection of processes collaborating on the solution of a single problem ๐Ÿ•ธ๏ธ
  • window: The window is one of the fundamental elements of the transport's state, and it can be controlled to affect the quality of service given to the client ๐ŸŒท It's the number of user data carrying packets that a single member may multicast into the web during a heartbeat ๐Ÿ’–

2.2 Packet format ๐Ÿ“ฆ

An MTP packet is a transport protocol header followed by a variable amount of data ๐ŸŽ The protocol header, shown in Figure 1, is part of EVERY packet. The rest of the packet is either user data (when the packet type = data) or extra transport specific information. The fields in the header are statically defined as n-bit wide quantities. There are no undefined fields, and no fields that might have undefined values at any time ๐Ÿ™Œ Reserved fields, if they exist, must always have a value of zero ๐Ÿ’œ

Okay bestie, let's look at the packet itself ๐ŸŽ€ This is the MTP packet layout, and every field lives in one of these boxes โœจ

    0           7 8           15 16         23 24         31
   ----------------------------------------------------------    -----
   |  protocol    |    packet   |    type     |    client   |      |
   |  version     |    type     |    modifier |    channel  |      |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              source connection identifier              |      |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              destination connection identifier         |
   ---------------------------------------------------------- transport
   |                                                        |    header
   |              message acceptance criteria               |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              heartbeat                                 |      |
   ----------------------------------------------------------      |
   |                            |                           |      |
   |        window              |        retention          |      |
   ----------------------------------------------------------    -----
   |                                                        |      |
   |                                                        |      |
   |                                                        |      |
   |                   (data content and format             |
   |                   dependent on packet type             |    data
   |                   and modifier)                        |    fields
   |                                                        |
   |                                                        |      |
   |                                                        |      |
   |                                                        |      |
   ----------------------------------------------------------    -----

                        Figure 1. MTP packet format

2.2.1. Protocol version ๐Ÿ’–

The first 8 bits of the packet are the protocol version number ๐Ÿฆ‹ This document describes version 1 of the Multicast Transport Protocol, so the version field has a value of 0x01 ๐Ÿ’…

2.2.2. Packet type and modifier ๐ŸŒˆ

The second byte of the header is the packet type, and the byte right after it holds the packet type modifier โœจ Typical control message exchanges come in a request/response pair, and the modifier field makes responses super easy to build because you can reuse the incoming message with minimal changes ๐Ÿซถ The table below lists the packet type field values together with their modifiers. Modifiers only make sense in the context of their type ๐Ÿ™ˆ When we talk about the definitions later on, we'll write an entry from this table as type[modifier]. So for example, data[eow] means a packet of type data with an end of window modifier ๐Ÿ’œ

   type       modifier     description

   data(0)    data(0)      The packet is one that contains user
                           information. Only the process possessing a
                           transmit token is permitted to send data
                           unless specifically requested to retransmit
                           previously transmitted data. All packets of
                           type data are multicast to the entire web.

              eow(1)       A data packet with the eow (end of window)
                           modifier set indicates that the transmitter
                           intends to send no more packets in this
                           heartbeat either because it has sent as many
                           as permitted given the window parameter or
                           simply has no more data to send during the
                           current heartbeat. This is not client
                           information but rather a hint to be used by
                           transport providers to synchronize the
                           computation and transmission of naks.

              eom(2)       Data[eom] marks the end of the message to the
                           consumers, and the surrendering of the
                           transmit token to the master. And like a
                           data[eow] a data[eom] packet implies the end
                           of window.

   nak(1)     request(0)   A nak[request] packet is a consumer
                           requesting a retransmission of one or more
                           data packets. The data field contains an
                           ordered list of packet sequence numbers that
                           are being requested. Naks of any form are
                           always unicast.

              deny(1)      A nak[deny] message indicates that the
                           producer source of the nak[deny]) cannot
                           retransmit one or more of the packets
                           requested. The process receiving the
                           nak[deny] must report the failure to its
                           client.

   empty(2)   dally(0)     An empty[dally] packet is multicast to
                           maintain synchronization when no client data
                           is available.

              cancel(1)    If a producer finds itself in possession of a
                           transmit token and has no data to send, it
                           may cancel the token[request] by multicasting
                           an empty[cancel] message.

              hibernate(2) If the master possesses all of the web's
                           transmit tokens and all outstanding messages
                           have been accepted or rejected, the master
                           may transmit empty[hibernate] packets at a
                           rate significantly slower than indicated by
                           the web's value of heartbeat.

   join(3)    request(0)   A join[request] packet is sent by a process
                           wishing to join a web to the web's unknown
                           TSAP (see section 2.2.5).

              confirm(1)   The join[confirm] packet is the master's
                           confirmation of the destination's request to
                           join the web. It will be unicast by the
                           master (and only the master) to the station
                           that sent the join[request].

              deny(2)      A join[deny] packet indicates permission to
                           join the web was denied. It may only be
                           transmitted by the master and will be unicast
                           to the member that sent the join[request].

   quit(4)    request(0)   A quit[request] may be unicast to the master
                           by any member of the web at any time to
                           indicate the sending process wishes to
                           withdraw from the web. Any member may unicast
                           a quit to another member requesting that the
                           destination member quit the web due to
                           intolerable behavior.  The master may
                           multicast a quit[request] requiring that the
                           entire web disband. The request will be
                           multicast at regular heartbeat intervals
                           until there are no responses to retention
                           requests.

              confirm(1)   The quit[confirm] packet is the indication
                           that a quit[request] has been observed and
                           appropriate local action has been taken.
                           Quit[confirm] are always unicast.

   token(5)   request(0)   A token[request] is a producing member
                           requesting a transmit token from the master.
                           Such packets are unicast to the master.

              confirm(1)   The token[confirm] packet is sent by the
                           master to assign the transmit token to a
                           member that has requested it. token[confirm]
                           will be unicast to the member being granted
                           the token.

   isMember(6) request(0)  An isMember[request] is soliciting
                           verification that the target member is a
                           recognized member of the web. All forms of
                           the isMember packet are unicast to a specific
                           member.

              confirm(1)   IsMember[confirm] packets are positive
                           responses to isMember[requests].

              deny(2)      If the member receiving the isMember[request]
                           cannot confirm the target's membership in the
                           web, it responds with a isMember[deny].

2.2.3. Subchannel ๐ŸŒธ

The fourth byte of the transport header holds the client's subchannel value ๐Ÿ“ It starts out at zero by default. The transport client decides what the subchannel value means, so it only matters for packets of type data. Every other packet type must have a subchannel value of zero ๐Ÿ™ˆ

2.2.4. Source connection identifier ๐Ÿฆ‹

The source connection identifier is a 32 bit field holding a value that's unique to the transmitting system, assigned when the transport gets created โœจ It's how you pick out one specific transport instance, and it's part of the TSAP ๐Ÿ’… Every packet the transport sends must have this field set ๐Ÿ’–

2.2.5. Destination connection identifier ๐Ÿ‘‘

The destination connection identifier is the 32 bit identifier of the target transport ๐Ÿงš From the sending process's point of view, there are three kinds of destination connection identifiers, and they're giving very different vibes ๐Ÿ˜ญ

First up is the unknown connection identifier, 0x00000000. It's only used as the destination connection identifier in a join[request] packet ๐Ÿซถ

Next is the multicast connection identifier, which you pick up from the join[confirm] message the master sends. This one pairs with the multicast NSAP to make the destination TSAP for every packet multicast to the entire web [2] ๐ŸŒˆ

The last kind is the unicast identifier, which forms the destination TSAP when you're unicasting packets to individual members. Every member of the web has its own unicast connection identifier, and that's what builds its own unicast TSAP ๐Ÿ’œ

2.2.6. Message acceptance ๐Ÿ’…

Okay so MTP's whole thing is making sure every process agrees on which messages get accepted AND the order they get accepted in ๐Ÿซถ. The master is the one running this show. It controls the agreement by handing out transmit tokens and setting message status. Once a token for a message has been assigned (see section 3.2.1), the master sets that message's status using these rules [AFM91]:

  • If the master has seen the entire message (i.e., has seen the data[eom] and all the intervening data packets), the status is accepted โœ….
  • If the master hasn't seen the whole message but believes the sender is still operational and connected to the master (as the master decides), the status is pending ๐Ÿคž.
  • If the master hasn't seen the whole message and believes the sender has failed or partitioned away, the status is rejected ๐Ÿ’€.

Message status rides along in the message acceptance record (see Figure 2) on every packet, so processes learn the status of earlier messages just by reading it ๐Ÿ“ฆ.

The acceptance criteria is a multi-part record that carries the agreement rules. The most significant 8 bits are a flag. If it's not zero, it means synchronization is required ๐Ÿ”. This field MAY vary on a per-message basis, as directed by the producing transport's client. The default is no synchronization required.

The second part of the record is a 12-element vector that shows the status of the last 12 messages sent into the web ๐Ÿ•ธ๏ธ.

    0          7 8          15 16          23 24         31
   ---------------------------------------------------------
   |            |                                          |
   |  synchro   |         tri-state bitmask[12]            |
   ---------------------------------------------------------
   |      message             |      packet sequence       |
   |      sequence number     |      number                |
   ---------------------------------------------------------

Figure 2. Message acceptance record

Each element is two bits long and can be one of three values: accepted(0), pending(1), or rejected(2). The bit mask starts out all zeros. When the token for message m is transmitted, the left-most element of the vector is the state of message m - 1, the next element is message m - 2, and so on. This means you can see the status of the last 12 messages, and older ones just get shifted out and lost ๐ŸŒ€. Only the master is permitted to set message status. The master is not permitted to shift a pending status past the end of the vector. If that situation pops up, the master MUST instead not confirm any token[request] until the oldest message can be marked as either rejected or accepted ๐Ÿ˜ค.

Message sequence numbers are 16-bit unsigned values ๐Ÿ”ข. The master initializes the field to zero when the transport starts up and increments it by one after each token is granted. Only the master is permitted to change the message sequence number. Once granted, that number is consumed, and the message's state MUST eventually become either accepted or rejected. No transmit tokens may be granted if assigning a message sequence number would cause a pending status to be shifted past the end of the status vector.

Packet sequence numbers are unsigned 16-bit numbers assigned by the producing process on a per-message basis ๐Ÿ“ฎ. They start at zero for each new message and go up by one (consumed) for each data packet in the message. Consumers that detect missing packet sequence numbers MUST send a nak[request] to the appropriate producer to recover the missing data ๐Ÿ™.

Control packets always contain the message acceptance criteria with the synchronization flag set to zero (0x00), the highest message sequence number observed, and a packet sequence number one greater than previously observed ๐Ÿงพ. Control packets don't consume any sequence numbers. Since control messages aren't reliably delivered, the acceptance criteria SHOULD only be checked to see whether it falls within the proper range of message numbers, relative to the receiving station's current message number. The range of acceptable sequence numbers SHOULD be m-11 to m-13, inclusive, where m is the current message number ๐ŸŽฏ.

2.2.7. Heartbeat ๐Ÿ’“

Heartbeat is an unsigned 32-bit field measured in milliseconds โฑ๏ธ. Every member of the web shares the same heartbeat value. By definition, at least one packet (data, empty, or quit from the master) will be multicast into the web within every heartbeat period. Basically the web never goes silent for longer than one beat ๐Ÿซ€.

2.2.8. Window ๐ŸชŸ

The allocation window (or just window) is a 16-bit unsigned field. It tells you the maximum number of data packets a member can multicast in a single heartbeat. It counts both retransmitted and new data packets together ๐Ÿ“Š.

2.2.9. Retention ๐Ÿ—„๏ธ

Retention is a 16-bit unsigned value: the number of heartbeats for which a producer MUST retain transmitted client data and state, so it can do retransmissions ๐Ÿ”„. Keep it safe, bestie ๐Ÿ”.

2.3. Transport addresses ๐Ÿฆ‹

Each transport has three logical transport service access points (TSAPs), which are formed by concatenating a network service access point (NSAP) with a transport connection identifier ๐Ÿงฉ. They are the unknown TSAP, the web's multicast TSAP, and each individual member's TSAP.

2.3.1. Unknown transport address ๐Ÿซฃ

Stations that are just joining MUST use the multicast NSAP associated with the transport, but they don't know the web's multicast TSAP or the master process' TSAP yet ๐Ÿ™ˆ. So joining stations make up a temporary TSAP (the unknown TSAP) by using the connection identifier reserved to mean unknown (0x00000000). The join[confirm] message will come from the master's TSAP and will include the multicast transport connection identifier in its data field. Those values MUST be extracted from the join[confirm] and remembered by the joining process ๐Ÿ“.

2.3.2. Web's multicast address ๐Ÿ“ก

The multicast TSAP is formed by logically concatenating the multicast NSAP from when the transport was created with the transport connection identifier returned in the data field of the join[confirm] packet. If more than one network is involved in the web, the multicast transport address becomes a list with one entry per network. That list is supplied in the data field of token[confirm] packets ๐ŸŒ.

The multicast TSAP is the target for every message meant for the entire web, like data and empty. The master's decision to abandon the transport (quit) is also sent to the multicast transport address ๐Ÿ‘‘.

2.3.3. Member addresses ๐Ÿก

The member TSAP is formed by using the process' unicast NSAP concatenated with a locally generated unique connection identifier. That TSAP MUST be the source of every packet the process transmits, no matter where it's going, for the entire lifetime of the transport ๐Ÿ’œ.

Packets unicast to specific members MUST contain the appropriate TSAP. For producers and consumers this is pretty easy, because the only TSAPs they care about are the master and whichever station(s) are currently transmitting data ๐Ÿ’….

3. Protocol behavior ๐Ÿ’–

This section defines what the protocol implementation is expected to do โœจ. These expectations SHOULD NOT be considered guidelines or hints. They're actually part of the protocol, so no skipping ๐Ÿ™…โ€โ™€๏ธ.

3.1. Establishing a transport ๐ŸŒธ

Before any rendezvous can happen, a process MUST first acquire an NSAP that will be the service access point for the instantiation [3]. The process that first establishes itself at that NSAP is called the master of the web ๐Ÿ‘‘. The decision as to which process acts as the master MUST be made a priori, so creation stays unambiguous even when there are network partitions ๐Ÿง . The process SHOULD make a robust effort to verify that the NSAP is not already in service. It MAY do so by repeatedly sending join[requests] to the web's unknown TSAP. If there's no response after repeated transmissions, the process MAY be relatively confident that the NSAP isn't in use and can go ahead with creating the web ๐ŸŽ‰. If it does get a response, though, the creation MUST be aborted and the situation reported to its client ๐Ÿšจ.

3.1.1. Join request โœจ

Okay bestie, so new members can join the web anytime after the master is set up. The joining process sends a join[request] to the unknown TSAP ๐Ÿ’Œ It should already have a unique connection identifier for its transport instance, and that goes in the source TSAP of the join[request]. The join[request] must have zeros in every acceptance field ๐Ÿซถ The heartbeat, window and retention values get filled in however the transport provider's client asked for them. The data part carries the type, class and quality of service the client requested ๐Ÿ“ฆ

   field               class       definition

   membership class    master(0)   There can be only a single web
                                   master, and that member has all
                                   privileges of a producer class member
                                   plus those acquitted only to the
                                   master.

                       producer(1) A process that has producer class
                                   membership wishes to transmit data
                                   into the web as well as consume.

                       consumer(2) A consumer process is a read only
                                   process. It will send naks in order
                                   to reliably receive data but will
                                   never ask for or be permitted to take
                                   possession of a transmit token.

   transport class     reliable(0) Specifies a reliable transport, i.e.,
                                   one that will generate and process
                                   naks.  The implication is that the
                                   data will be reliably delivered or
                                   the failure will be detected and
                                   reported to the client.

                       unreliable(1)   The transport supports best

                                   effort delivery. Such a transport may
                                   still fail if the error rates are too
                                   high, but tolerable loss or
                                   corruption of data will be permitted
                                   [4].

   transport type      NxN(0)      The transport will accept multiple
                                   processes with producing capability.

                       1xN(1)      A 1xN transport permits only a single
                                   producer whose identity was
                                   established a priori.

Okay so the membership class is giving hierarchy ๐Ÿ‘‘ There's only ever ONE master ๐ŸŒธ and it gets every perk a producer gets, plus the master-only privileges. A producer(1) can send data into the web and read it ๐Ÿ–Š๏ธ A consumer(2) is read-only ๐Ÿ‘€ It sends naks so it can reliably receive data, but it never asks for or gets a transmit token. Not for her ๐Ÿ’…

For transport class, reliable(0) means the transport generates and processes naks. It basically promises the data gets delivered reliably, or if it fails, that failure gets detected and reported back to the client ๐Ÿฆ‹ unreliable(1) is best effort. It can still fail if error rates get too high, but a little loss or corruption is tolerated [4] ๐ŸŒˆ

For transport type, NxN(0) lets multiple processes produce data ๐ŸŽ€ 1xN(1) only allows a single producer, and that producer's identity has to be set up ahead of time (a priori) ๐Ÿ”

The client's minimum throughput, which is expressed in kilobytes per second, is the lowest value the transport will accept ๐Ÿ“Š It's calculated from the heartbeat and window parameters plus the maximum data unit size. It is NOT measured from actual traffic, so this is all math, bestie ๐Ÿงฎ Any member proposing a combo that gives unacceptable throughput gets ignored or asked to withdraw from the web ๐Ÿ’€

A joining client can also suggest a maximum data unit size ๐Ÿ“ This is a number of bytes that can fit in a data packet as client data.

If no response comes back within a single heartbeat, the join[request] should be retransmitted using the same source TSAP ๐Ÿ” That's how the master can tell a brand new process apart from a retransmission of the same join[request] ๐Ÿ•ต๏ธโ€โ™€๏ธ

3.1.2. Join confirm/deny ๐ŸŽŸ๏ธ

Only the master of the web responds to join[request] ๐Ÿ‘‘ The response either lets the new process in or denies it. A join can be denied if the new member asks for service parameters that conflict with the ones the master already set ๐Ÿ™…โ€โ™€๏ธ If the join is confirmed, the join[confirm] gets unicast by the master, with a data field holding the web's current operating parameters ๐Ÿ’Œ If those parameters don't work for the joining process, it can decide to withdraw from the web. Otherwise, the parameters must be accepted as the current operating values ๐Ÿซก

    0           7 8           15 16         23 24         31
   ----------------------------------------------------------    -----
   |  protocol    |    packet   |    type     |    client   |      |
   |  version     |    type     |    modifier |    channel  |      |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              source connection identifier              |      |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              destination connection identifier         |
   ---------------------------------------------------------- transport
   |                                                        |    header
   |              message acceptance criteria               |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              heartbeat                                 |      |
   ----------------------------------------------------------      |
   |                            |                           |      |
   |        window              |        retention          |      |
   ----------------------------------------------------------    -----
   |  member     |   transport  |  transport  |             |      |
   |  class      |   class      |  type       |  reserved   |      |
   ----------------------------------------------------------
   |        minimum             |     maximum data          |    data
   |        throughput          |     unit size             |
   ----------------------------------------------------------      |
   |                  multicast connection                  |      |
   |                  identifier                            |      |
   ----------------------------------------------------------    -----

                           Figure 3. join packet

The join[confirm] also carries the multicast connection identifier ๐Ÿงฉ This has to be used to form the TSAP that's the destination for all multicast messages for the transport ๐Ÿ“ฎ The source of the join[confirm] is the master's TSAP, and the member has to record it for later ๐Ÿ“

The master must hold all the transmit tokens when it sends a join[confirm] ๐ŸŽซ Making the master hold them means the joining member only ever sees complete messages, which is so important ๐Ÿฅน It also lets the master's client know about the join, so application state can get sent automatically to the newbie ๐ŸŽ A newly joined member might be on a network the web hasn't seen before, so a new multicast TSAP gets added to the existing list ๐ŸŒ The whole list gets sent in the data field of all later token[confirm] messages (more on those soon) ๐Ÿ’œ


3.2. Maintaining data consistency ๐Ÿง 

The transport is in charge of keeping the data consistent once producing clients submit it for delivery ๐Ÿ›ก๏ธ The actual client data is the bulk of what flows through the web, but it comes with a lot of protocol state info riding along ๐ŸŽ’ On top of the state info piggybacked onto the client data, there's also a minimum amount of protocol packets that exist purely for the transport. The transport client never sees them ๐Ÿซฅ Main character behind the scenes energy ๐ŸŽฌโœจ

3.2.1. Transmit tokens โœจ

Okay so bestie, nobody gets to send client data or state until they're holding a transmit token ๐ŸŽ€. To grab one, you send a token[request] to the master. Those requests should be unicast to the master's TSAP, and you keep re-sending them about once per heartbeat ๐Ÿ’–. The master is the one handing out tokens, so it's basically the queen of fairness ๐Ÿ‘‘. It may use some fairness algorithms to decide who gets to talk next. At the very least, requests are queued first in, first out ๐Ÿ“. If one member spams duplicate requests, those get ignored and the first unhonored request is the one that counts โœจ. When it makes sense, the master sends a token[confirm] back to a member with a request pending. The data field of that response lists every multicast TSAP represented in the current web at that moment ๐Ÿฆ‹.

If the master hears no data or heartbeat messages going into the web, it assumes the token is lost ๐Ÿ’€. That probably means the member holding it crashed or got cut off from the master ๐Ÿ™ˆ. In that case the master may check in on that process, maybe by sending isMember[request]. If the member doesn't reply, it gets removed from the active members of the web, its message is marked as rejected, and the master takes the token back for itself ๐Ÿ‘‘.

Figure 4 is a timing diagram where time runs top to bottom ๐Ÿงš. Process A holds the token, and process B asks for one even though there are no free tokens ๐ŸŒธ.

                           A    master    B
    "A" multicasts data    |             |  "B" requests
                           |\     |      |  transmit token
                           | \    |     /|
                           |  \   |    / |
                           |   \  |   /  |
    "A" multicasts data    |    \ |  /   |  "B" retransmits
    w/eom set              |\    \| /    |  token request
                           | \    \V    /|
                           |  \   |\   / |
                           |   \  | V /  |
                           |    \ |  /   |
                           |     \| /    |
                           |      \V     |
                           |      |\     |
                           |      | V    |
                           |      |\     |  Master assigns
                           |      | \    |  token to "B"
                           |      |  \   |
                           |      |   \  |
                           |      |    \ |
                           |      |     V|
                           |      |      |
                           |      |     /|  "B" multicasts
                           |      |    / |  data
                           |      |   /  |
                           |      |  /   |
                           |      | /    |
                           |      |/     |
                           |      /      |
                           |     /|      |
                           |    V |      |
                           |      |      |

Figure 4. Acquiring the token

Token packets, like other control packets, don't consume sequence numbers ๐ŸŽ€. So the master needs a different way to tell whether several token[request]s from one member are actually new asks for a separate token, or just retransmissions of the same request ๐Ÿค”. To pull that off, the master and the members need an implicit understanding of each other's state โœจ.

Here's the token packet layout ๐Ÿ’….

    0           7 8           15 16         23 24         31
   ----------------------------------------------------------    -----
   |  protocol    |    packet   |    type     |    client   |      |
   |  version     |    type     |    modifier |    channel  |      |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              source connection identifier              |      |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              destination connection identifier         |
   ---------------------------------------------------------- transport
   |                                                        |    header
   |              message acceptance criteria               |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              heartbeat                                 |      |
   ----------------------------------------------------------      |
   |                            |                           |      |
   |        window              |        retention          |      |
   ----------------------------------------------------------    -----
   |                                                        |      |
   |                                                        |      |
   |                   TSAPs of all networks                |
   |                   represented in the web               |    data
   |                   membership                           |
   |                                                        |      |
   |                                                        |      |
   ----------------------------------------------------------    -----

Figure 5. token packet

The master sees the token as being in one of three states ๐Ÿง :

  • idle: Nobody has the token right now. Its message number isn't showing up in the current message acceptance vector ๐Ÿซถ.
  • pending: The master has assigned the token with a token[confirm] packet, but it hasn't seen any data packets proving the producing member got the notification ๐Ÿ‘€.
  • busy: The token has been assigned, and the master has seen data packets carrying the assigned message number. The message those packets make up is still in the message acceptance vector ๐Ÿ’ช.

A token that isn't idle also remembers which TSAP's process owns it (or owned it before) ๐Ÿงš.

Based on that state, the master answers any process holding a token in the pending state with a reassignment of that token ๐Ÿ”. That's because it's assuming the original token[confirm] never reached the requesting process ๐Ÿ™ˆ. The other option is that the process did get the token, sent data with it, and the master just didn't see those packets. But data messages are multi-packet by design, padded with empty packets when needed, so the master missing every packet of a message is way less likely than the requester missing one single token[confirm] packet ๐Ÿ˜ญ.

The requesting process has to think about what the master did and what caused it ๐ŸŒˆ. In most cases, the master's assumptions are right. There are, however, two ambiguous situations ๐Ÿ’€. The first is the one the master is most directly dealing with: it can't tell whether the requester missed the token[confirm], or whether the master missed the data packets from the producing process. The second is that the requesting process may have timed out too quickly, so its retransmitted token[request] passed the token[confirm] in the night ๐ŸŒ™. Either way, the producing process might end up holding a token it has no need for. It can dismiss those by sending an empty[cancel] packet ๐Ÿ’….

Another possibility is that the requesting process actually used the assigned token and is now asking for another one ๐Ÿฆ‹. Unless the master has seen data using the token, the master will still consider the token pending. So a process that receives a duplicate token[confirm] should treat it as a nak and retransmit any data packets it previously sent using that token's message sequence number ๐Ÿ’–.

3.2.2. Data transmission ๐Ÿ’–

Okay so, the transport client hands over its data as uninterpreted bytes ๐Ÿซถ. Those bytes get tucked into packets right after the protocol's fixed overhead fields. A packet can carry anywhere from zero data bytes up to the maximum size of a network protocol packet, minus the network overhead and the fixed transport overhead โœจ. Every packet that uses up a sequence number MUST contain either client data or client state transitions, like the end of message indicator or a subchannel transition ๐ŸŽ€.

Packets get sent in bursts called windows ๐Ÿฆ‹. The protocol guarantees that no more than the current value of window data packets will be transmitted by a single process during a heartbeat. Every packet always carries the latest heartbeat, window and retention information ๐Ÿ’…. If full packets are unavailable [5], empty[dally] messages SHOULD be transmitted instead ๐ŸŒธ. The only packets that will be transmitted containing less than maximum capacity will be data[eom] or those containing client subchannel transitions ๐Ÿ“.

        -----     |      |
          |       |\     |
          |       | \    |
                  |\ \   |
      heartbeat   | \ \  |
                  |\ \ \ |
          |       | \ \ V|  data(n)
          |       |  \ \ |
        -----     |   \ V|  data(n+1)
                  |\   \ |
                  | \   V|  data(n+w-1) w/eow
                  |\ \   |
                  | \ \  |
                  |\ \ \ |
                  | \ \ V|  data(n+w)
                  |  \ \ |
        -----     |   \ V|  data(n+w+1)
                  |\   \ |
                  | \   V|  data(n+2w-1) w/eow
   w = window = 3 |  \   |
   r = retention = 2  |   \  |
                  |    \ |
                  |     V|  empty(n+2w)
                  |      |
        -----     |      |
                  |\     |
                  | \    |
                  |  \   |
                  |   \  |
                  |    \ |
                  |     V|  data(n+2w) w/eom
                  |      |    Packets n..n+w-1 are released,
        -----     |      |    token is surrendered.
                  |      |
                  |      |
                  |      |
                  |      |
                  |      |
                  |      |
                  |      |
        -----     |      |    Packets n+w..n+2w-1 are released.


                Figure 6. Normal data transmission

Figure 6 is a timing diagram of a process transmitting into a web (without any complicating naks) โฐ. Time goes down the figure, so keep scrolling ๐Ÿ™ˆ. The transmitting process is obligated to retransmit requested packets for at least retention heartbeat intervals after their first transmission ๐Ÿ”.

 0           7 8           15 16         23 24         31
----------------------------------------------------------    -----
|  protocol    |    packet   |    type     |    client   |      |
|  version     |    type     |    modifier |    channel  |      |
----------------------------------------------------------      |
|                                                        |      |
|              source connection identifier              |      |
----------------------------------------------------------      |
|                                                        |      |
|              destination connection identifier         |
---------------------------------------------------------- transport
|                                                        |    header
|              message acceptance criteria               |
----------------------------------------------------------      |
|                                                        |      |
|              heartbeat                                 |      |
----------------------------------------------------------      |
|                            |                           |      |
|        window              |        retention          |      |
----------------------------------------------------------    -----
|                                                        |      |
|                   uninterpreted data                   |
|                                                        |    data
|                                                        |
|                                                        |      |
----------------------------------------------------------    -----

                       Figure 7. data packet

3.2.3. Empty packets ๐Ÿงš

An empty packet is a control packet that a producer holding a transmit token multicasts into the web at regular intervals when no client data is available ๐Ÿ’œ. These are sent to keep everyone in sync and to advertise the producer's maximum sequence number ๐Ÿ“ฃ. They give consuming processes a chance to spot missed data and request retransmission, and they also show who owns a transmit token ๐Ÿ‘‘.

 0           7 8           15 16         23 24         31
----------------------------------------------------------    -----
|  protocol    |    packet   |    type     |    client   |      |
|  version     |    type     |    modifier |    channel  |      |
----------------------------------------------------------      |
|                                                        |      |
|              source connection identifier              |      |
----------------------------------------------------------      |
|                                                        |      |
|              destination connection identifier         |
---------------------------------------------------------- transport
|                                                        |    header
|              message acceptance criteria               |
----------------------------------------------------------      |
|                                                        |      |
|              heartbeat                                 |      |
----------------------------------------------------------      |
|                            |                           |      |
|        window              |        retention          |      |
----------------------------------------------------------    -----

                      Figure 8. empty packet

There are two situations where empty[dally] packets come into play ๐ŸŒˆ. The first is when the client hasn't given enough data for a full packet during a heartbeat. Partial packets SHOULD NOT be transmitted unless there's a client transition to convey, BUT something MUST be transmitted during a heartbeat or the master might think the process holding a transmit token has failed ๐Ÿ˜ญ. So empty[dally] gets sent instead of a data packet until the client provides more data to fill a packet or signals a state transition, like an end of message or subchannel transition โœจ.

The second situation is after short messages get sent ๐Ÿ“. Each message SHOULD consist of multiple packets, so consumers are more likely to see at least one packet of a message and figure out who produced it ๐Ÿ•ต๏ธโ€โ™€๏ธ. The transport parameter retention has roughly the right properties for that insurance, so a message MUST consist of at least retention packets. If the client data doesn't need that many packets, empty[dally] packets MUST be appended ๐ŸŽ€. A process that has no transmittable data and is holding a transmit token MUST send an empty[cancel] ๐Ÿ™…โ€โ™€๏ธ. Sending empty[cancel] packets hands ownership of the transmit token back to the master ๐Ÿ”™. When the master sees that control packet, it marks the referenced message as rejected, so other consumers don't think it got lost and try to recover it ๐Ÿ’….

During quiet periods (i.e., after every message has been either accepted or rejected and there are no outstanding transmit tokens), the master MAY enter hibernation mode by transmitting empty[hibernate] packets ๐Ÿ˜ด. In that mode the master increases the value of the transport parameter heartbeat to cut down on network traffic ๐Ÿ“‰. These packets indicate that the packet's heartbeat field SHOULD NOT be used for resource computation by the processes that observe it ๐Ÿ™ˆ.

3.2.4. Missed data ๐Ÿ•ต๏ธโ€โ™€๏ธ

The most common way to notice data loss is receiving a data or heartbeat message with a sequence number higher than expected from that producer ๐Ÿ‘€. The second most common way is seeing a message fragment (one missing the end of message) and then no more data or empty packets from the fragment's producer for longer than a single heartbeat ๐Ÿ˜ฌ. Either way, the consumer process sends a negative acknowledgment (nak) to the producer of the incomplete message ๐Ÿ“จ. The nak's data field holds a list of ascending sequence number pairs that the consumer needs in order to recover the missed data ๐Ÿ’–.

 0           7 8           15 16         23 24         31
----------------------------------------------------------    -----
|  protocol    |    packet   |    type     |    client   |      |
|  version     |    type     |    modifier |    channel  |      |
----------------------------------------------------------      |
|                                                        |      |
|              source connection identifier              |      |
----------------------------------------------------------      |
|                                                        |      |
|              destination connection identifier         |
---------------------------------------------------------- transport
|                                                        |    header
|              message acceptance criteria               |
----------------------------------------------------------      |
|                                                        |      |
|              heartbeat                                 |      |
----------------------------------------------------------      |
|                            |                           |      |
|        window              |        retention          |      |
----------------------------------------------------------    -----
|                            |                           |      |
|  message sequence (low)    |  packet sequence (low)    |
----------------------------------------------------------    data
|                            |                           |
|  message sequence (high)   |  packet sequence (high)   |      |
----------------------------------------------------------    -----

                       Figure 9. nak packet

3.2.5. Retrying operations ๐Ÿ”

okay so, operations HAVE to get retried, bestie, or one lost packet could take the whole transport down ๐Ÿ˜ญ The good news is the numbers you need are already sitting in the transport. The gap between retries is the transport's time constant (a.k.a. the heartbeat) ๐Ÿ’–, and the number of retries is retention โœจ.

Which operations get retried? The retriable ones (and their message types) are join, nak, token, isMember and quit ๐Ÿซถ. The heartbeat and retention also show up when you send empty messages. Empty[dally] messages go out any time data isn't available but data[eom] hasn't been sent yet ๐ŸŒธ. And if a process hasn't seen data or empty for more than retention heartbeat intervals, it's assumed to have failed or partitioned away, and the transport gets abandoned. Literally gone, no cap ๐Ÿ’€

3.2.6. Retransmission ๐Ÿฆ‹

If the producer gets a nak[request] from a consumer asking for a packet that's no longer available, the producer must send a nak[deny] to whoever asked. If that leaves the consumer in a failed state, the consumer will start withdrawing from the web ๐Ÿ˜ญ

If a producer gets a nak[request] asking for one or more packets to be resent, those packets get multicast to the entire web [6] ๐Ÿ“ก. Every one of them keeps the original client information (like subchannel and end of message state) plus the message and packet sequence numbers. BUT the retransmitted packets must carry updated protocol parameter information (heartbeat, window and retention) ๐ŸŽ€

Retransmitted packets follow the same heartbeat and window rules as the originals, so they're just as bound by those limits ๐Ÿ”ฅ That means the producer's retransmissions eat into the allocation window, which leaves less room for new data in a single heartbeat. Retransmitted packets also have priority over new data packets, meaning they should go out before any new data ๐Ÿƒโ€โ™€๏ธ๐Ÿ’จ

Here's the picture, because a diagram is worth a thousand bestie texts:

            -----     |       |     retransmission count = rx=0
              |       |\     |
              |       | \    |
              |       |\ \   |
              |       | \ \  |
              |       |\ \ \ |
              |       | \ \ V|  data(n)
              |       |  \ \ |
                      |   \ *|  data(n+1)
          heartbeat   |    \ |
                      |     V|  data(n+w-1-rx) w/eow       rx=0
              |       |      |
              |       |     /|  nak(n') of n+1
              |       |    / |
              |       |   /  |
              |       |  /   |
              |       | /    |
              |       |V     |
            -----     |      |
                      |\     |
                      | \    |
                      |\ \   |
                      | \ \  |
                      |\ \ \ |
   w = window = 3     | \ \ *|  retransmission(n+1)        rx=1
   r = retention = 1  |  \ \ |
                      |   \ V|  data(n+w)
                      |    \ |
                      |     V|  data(n+2w-1-rx) w/eow      rx=1
                      |      |
                      |     /|  nak(n') of n+1
                      |    / |
            -----     |   /  |
                      |\ /   |
                      | /    |
                      |V \   |
                      |\  \  |
                      | \  \ |
                      |\ \  V|  data(n+2w-rx)              rx=1
                      | \ \  |    Packets n..n+w-1-0 can be released.
                      |  \ \ |
                      |   \ V|  nak deny(n+1)              rx=2
                      |    \ |
                      |     V|  data(n+3w-1-rx) w/eom      rx=2
                      |      |
           -----      |      |    Packets n+w..n+2w-1-1 are released.

                  Figure 10. naks and retransmission

3.2.7. Duplicate suppression ๐Ÿ™ˆ

The consumer must be prepared to ignore duplicate packets it receives ๐Ÿ’… They'll basically always show up because the producer retransmitted something in response to another consumer's nak. Not a bug, just the vibe ๐ŸŒˆ

3.2.8. Banishment ๐Ÿšซ

If a process ever spots another one breaking the protocol, it can ask the offender to leave the web. It does this by unicasting a quit[request] to that offender, with the target field set to the offender's TSAP ๐Ÿ‘‘ And any member that has a detectable, recoverable protocol violation but still willingly responds to the quit[request] gets officially noted as having truly correct social behavior โœจ๐Ÿซถ

    0           7 8           15 16         23 24         31
   ----------------------------------------------------------    -----
   |  protocol    |    packet   |    type     |    client   |      |
   |  version     |    type     |    modifier |    channel  |      |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              source connection identifier              |      |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              destination connection identifier         |
   ---------------------------------------------------------- transport
   |                                                        |    header
   |              message acceptance criteria               |
   ----------------------------------------------------------      |
   |                                                        |      |
   |              heartbeat                                 |      |
   ----------------------------------------------------------      |
   |                            |                           |      |
   |        window              |        retention          |      |
   ----------------------------------------------------------    -----
   |                                                        |
   |              target TSAP                               |
   |                                                        |
   ----------------------------------------------------------

                          Figure 11. quit packet

3.3 Terminating the transport ๐Ÿ‘‹

Ending the transport is an advisory process, and any member of the web can start it ๐Ÿ’œ No process should intentionally quit the web while it still has retransmittable data buffered ๐Ÿ™…โ€โ™€๏ธ Stations should make every reasonable attempt to let the master know they're leaving, because their departure might collapse the topology of the web and get rid of the need to carry multicast messages across network boundaries. Tidy exit, iconic โœจ

3.3.1. Voluntary quits ๐Ÿ“

Voluntary quit[request]s are unicast to the master's TSAP. When the master gets a quit from a member, it replies with a quit[confirm] packet, and at that point the member is formally removed from the web ๐ŸŽ€ The request should be retransmitted at heartbeat intervals until the confirmation comes back from the master, or until it's been sent as many times as the web's value of retention.

3.3.2. Master quit ๐Ÿ‘‘

If the master starts the transport termination, it affects every member of the web ๐Ÿ˜ญ The master keeps all transmit tokens and refuses to assign them. Once it has the tokens, the master multicasts a quit[request] to the entire web. Every active member should acknowledge that request. If the master gets no confirmations for retention transmissions, it may assume every member has terminated its transport, and then it may follow suit ๐Ÿงš

3.3.3. Banishment ๐Ÿšซ

If the master receives any message other than a join[request] from a member it doesn't recognize, it should transmit a quit[request] with that process as the target. This covers the case where a consumer missed the termination reply and resent its original quit request, plus unannounced and rejected consumers ๐Ÿ’€

3.4 Transport parameters ๐ŸŽ›๏ธ

This section gives guidelines and reasoning for picking sensible transport quality of service parameters. It also explains some of the thinking behind the ranges of values listed ๐ŸŒธ

3.4.1. Quality of service โœจ

Active members of the web may suggest changes to the transport's quality of service parameters during the transport's lifetime ๐Ÿ’… Producers generally adjust the parameters to encourage higher throughput. Since consumers are responsible for certifying reliable delivery, they're expected to provide the force that encourages more reliability and stability. Both sides are trying to optimize the quality of service, bestie ๐Ÿซถ The negotiation that happened when members joined the web included the clients' wishes about the worst case behavior they'll tolerate. If a member can't maintain the negotiated lower bound, it may be asked to withdraw from the web. That process gets sent a unicast message (quit[request]) telling it to retire ๐Ÿ˜ญ

There are basically three parameters the transport keeps that reflect the client's quality of service requirements: heartbeat, window and retention ๐ŸŽ€ The transport can adapt these to match the capability of the members, the type of application being supported, and the network topology. When members join, they suggest values for the quality of service parameters to the master. If those parameters are acceptable, the master responds with the web's current operating values ๐Ÿ’– During the lifetime of the web, the parameters are expected to be modified by its members, though they may never result in a quality of service below the lower bounds set up during joining. Producers may try to improve performance by reducing the heartbeat interval and increasing the window size ๐Ÿ”ฅ That increases the resources committed to the transport at any time. To keep resources under control, the producer may also reduce the retention ๐Ÿฆ‹

Consumers must rely on their clients to consume the data that's occupying the transport's resources ๐Ÿ“ So the consumer transport implementation must monitor the level of committed resources to make sure it doesn't exceed its capabilities. MTP is a NAK based protocol, so the consumer is required to tell the producer if a parameter change is needed ๐Ÿ“ฃ The new information must reach the producer(s) before the consumer's resource situation becomes critical, so that no data gets missed ๐Ÿ™ˆ

For more stable operation, consumers would try to extend the heartbeat interval and reduce the window ๐ŸŒˆ To a certain degree they could also try to reduce the value of retention, which cuts the resources needed to support the transport. However, that needs a more stringent real-time capability โญ

3.4.2. Selecting parameter values โœจ

okay so the heartbeat is basically the transport time constant ๐Ÿ’–. If you model the transport as a closed loop system function (bestie, that's just a fancy way of saying feedback loop ๐Ÿซถ), then reactions to feedback should settle out in three time constants. In a transport that's stuck to a single network, the biggest source of processing delay is most likely page fault resolution time ๐Ÿ’€.

let's do an example ๐Ÿงš with a one MIP processor on an ethernet and an industry standard disk. The worst case page fault resolution takes two seeks (one to write out a dirty page, another to swap in the new page), and the average seek time is 40 milliseconds. So page fault resolution should be less than 80 milliseconds. Add some overhead and scheduling delays, and two times the worst case page fault resolution time is the minimum suitable transport time constant you could expect. So:

Heartbeat (minimum) = 160 - 200 milliseconds.

the transmit time for a full (ethernet) packet is about 1.2 milliseconds. Processing time should be less than 3 milliseconds (ignoring possible overlapped processing). Assuming disk access (with no faulting) is equivalent, and the total time per packet is the sum of the parts, that's 8.4 milliseconds. So the theoretical maximum is about 17 packets per heartbeat. The transport should be capable of about 120 packets per second, or 19.2 packets per heartbeat. main character numbers, no notes ๐Ÿ’…

Window (maximum) = 17 - 20 packets per heartbeat.

the (theoretical) throughput with these parameters in effect is 180 kilobytes per second ๐Ÿ”ฅ.

reducing retention may introduce instability, because consumers will have less chance to react to missing data. Data can get missed for a bunch of reasons ๐Ÿ˜ญ. If you're constrained to the local net, the data lost to data link corruption should be around one packet in every 50,000 (bit error rate of approximately 10-9). Telephony links (between routers, for instance) show similar characteristics. Several orders of magnitude more packets get lost at receiving processes, including packet switch routers, than over the physical links. Those losses are usually from congestion and resource starvation at lower layers, caused by processing (nearly) back to back packets. Incidental packet loss like this is basically unavoidable ๐Ÿฅฒ. You can only require that a receiving process be capable of successfully receiving some number of back to back packets, and that number must be at least greater than the value of window. Beyond that, the probability of success can be made as close to unity as required by giving the receiver the opportunity to observe the data multiple times. iykyk, repetition is the whole vibe

the receiving process must detect packet loss. The simplest method is to notice gaps in the received message/packet sequence numbers. That detection should be done after receiving an end of window or other state transition indication. So the naks cannot be transmitted, let alone received, until the following heartbeat. To make sure no single packet loss causes transport failure, the naks should have the opportunity to be transmitted at least twice ๐Ÿซถ.

when loss is detected, the nak must be transmitted and should be received at the producing process in less than two heartbeats after the data it references was transmitted. Again, it's the detection time that dominates, not the transmission of the nak. so the clock is really about detection, not sending ๐Ÿง 

Retention (minimum) = 3.

the resources committed to a producing transport under these assumptions are buffers sufficient for 80 packets of 1500 bytes each. Each buffer is committed for 600 - 800 milliseconds.

transports that span multiple networks have unique problems ๐ŸŒ. One of them: if a router drops a packet, all the processes on the remote network may try to send a nak[request] at the same time. That is not gonna enhance the router's quality of service ๐Ÿ’€. Also, it's obvious that any one nak[request] will be enough to prompt the producer to retransmit the desired packet. To cut down on the number of nak[requests] in this situation, the following scheme might be employed.

first, extend the value of retention to a minimum value of N. Then use a randomizing function that returns a value between zero and N - 2 to choose how many heartbeat intervals to dally before sending the nak[request], which spreads the transmissions out over time. For the method to be meaningful, the minimum value of retention must be adjusted.

Retention (minimum) = 5 (for internet cases)

3.4.3. Caching member information ๐Ÿ—‚๏ธ

to reduce transport member interaction and boost performance, producing members should keep a certain amount of caching going. These caches can be filled by gleaning info from reliable sources like multicast data or, when all else fails, from responses solicited from the web's master using isMember[request]. IsMember[request] requests are unicast to a member that's believed to have an accurate state of the web, at least to the degree that it can answer the question posed. The destination is usually the master. But if a process (such as the master) wants to verify that a process believes itself to be valid, it can assign the target TSAP and the destination to be the same. It is assumed that every process can verify itself ๐Ÿซถ.

if the member receiving the isMember[request] can confirm the target's active membership status in the web, it responds with a unicast isMember[confirm]. The data field holds the credibility value of the confirmation, which is the time (in milliseconds) since the information was confirmed from a reliable source.

caches are risky, because the stored info can go stale ๐Ÿ˜ฌ. So, with only a few exceptions, entries should be aged, and when they're sufficiently old, discarded. Ideally they get renewed by the same gleanable sources mentioned in the previous paragraph. If not, they're simply discarded and refilled when needed.

web membership can be gleaned from any packet that does not have a value of unknown as the destination connection identifier. A producing transport may extract the TSAP from such packets and either create or refresh local caches. Then, if while transmitting a NAK you receive one from a member whose identity is cached, no explicit request is needed to verify the source's membership โœจ.

the explicit source of membership info is the master. Info can be requested using the isMember message. Info gathered that way should be treated the same as gleaned info with respect to aging.

aging is a function of the transport's time constant (the heartbeat) and the retention. Info about a producing member must be cached at least as long as that producer has incomplete messages. It may be cached longer. The namespace for both sequence numbers and connection identifiers is intentionally long, so reuse of those namespaces will not likely collide bestie, that's called planning ahead ๐Ÿ’œ


A. Appendix: MTP as an Internet Protocol transport ๐Ÿฆ‹

MTP is a transport layer protocol, designed to be layered on top of a bunch of different network layer protocols. That underlying network layer protocol must provide certain facilities that MTP expects. In particular, it must provide "ports" or "sockets" to help address processes within a machine, plus a mechanism for multicast addressing of datagrams. These two addressing facilities are also used to formulate the NSAP for MTP on IP.

A.1. Internet Protocol multicast addressing

MTP on Internet Protocol uses the Internet Protocol multicast mechanisms defined in RFC 1112, "Host Extensions for IP Multicasting". MTP requires "Level 2" conformance described in that paper, for hosts that need to both send and receive multicast packets, both on the local net and on an internet. MTP on Internet Protocol uses the permanent host group address 224.0.1.9 that's the address you want to remember ๐ŸŒธ

A.2. Encapsulation

the Internet Protocol doesn't have a port mechanism ๐Ÿ’…. Ports are defined at the transport level instead. So to encapsulate MTP packets within Internet Protocol packets, a simple convergence or "bridge" protocol must be defined to run on top of Internet Protocol. It gives MTP the mechanism it needs to deliver packets to the proper processes. We'll call it the "MTP/Internet Protocol Bridge Protocol", or just "Bridge". The protocol header is encapsulated in the Internet Protocol data. The protocol field of the Internet Protocol packet carries the value indicating this is an MTP packet (92 decimal). The MTP packet itself is encapsulated in the Bridge data. Figure A.1 shows the positions of the fields within the MTP packet, while table A.1 defines the contents of those fields ๐ŸŽ€.

A.3. Fields of the bridge protocol

       0           7 8           15 16         23 24         31
      ----------------------------------------------------------
      |                            |                           |
      |     destination port       |     source port           |
      ----------------------------------------------------------
      |                            |                           |
      |     length                 |     checksum              |
      ----------------------------------------------------------
      |                                                        |
      |                      client data                       |
      ----------------------------------------------------------

               Figure A.1 MTP bridge protocol header fields

here's the field breakdown ๐Ÿ“

   destination port The port to which the packet is destined or sinked.

   source port The port from which the packet originates or is sourced.

   length      The length in octets of the bridged packet, including
               header and all data (the MTP packet).  The minimum value
               in this field is 8, the maximum is 65535.  The length
               does not include any padding bytes that were used to
               compute the checksum.  Note that though this field allows
               for very long packets, most networks have significantly
               shorter maximum frame sizes - the allowable and optimal
               packet size must be determined by means beyond the scope
               of this specification.

   checksum    The 16 bit one's compliment of the one's compliment sum
               of the entire bridge protocol header and data, padded
               with a zero octet (if necessary) to make multiple 16 bit
               quanities. A computed checksum of all zeros should be
               changed to all ones.  The checksum field is optional -
               all zeros in the field indicate that checksums are not in
               use.

   data        The data field is the field that carries the actual
               transport data. A single MTP packet will be carried the
               data field of each bridge packet.

A.4. Relationship to other Internet Protocol Transports

the astute reader might note that the MTP/Bridge Protocol looks a lot like the User Datagram Protocol (UDP) ๐Ÿง. UDP itself was not used because the protocol field in the Internet Protocol packet should reflect the fact that the higher level protocol of interest is MTP the protocol field is doing the talking here, periodt โœจ

References ๐Ÿ“šโœจ

okay so this is the bibliography, bestie. every paper and RFC MTP is built on, and I kept every single entry exactly as written ๐Ÿ’…

   AFM91   Armstrong, S., A. Freier and K. Marzullo, "MTP: An Atomic
           Multicast Transport Protocol", Xerox Webster Research Center
           technical report X9100359, March 1991.

   Bog83   Boggs, D., "Internet Broadcasting", Xerox PARC technical
           report CSL-83-3, October 1983.

   BSTM79  Boggs, D., J. Shoch, E. Taft, and R. Metcalfe, "Pup: An
           Internetwork Architecture", IEEE Transactions on
           Communications, COM-28(4), pages 612-624. April 1980.

   DIX82   Digital Equipment Corp., Intel Corp., Xerox Corp., "The
           Ethernet, a Local Area Network: Data Link and Physical Layer
           Specifications", September 1982.

   CLZ87   Clark, D., M. Lambert, and L. Zhang, "NETBLT: A high
           throughput transport protocol", In Proceedings of ACM SIGCOMM
           '87 Workshop, pages 353-359, 1987.

   CM87    Chang J., and M. Maxemchuck. "Atomic broadcast",  ACM
           Transactions on Computer Systems, 2(3):251-273, August 1987.

   Cri88   Cristian, F., "Reaching agreement on processor group
           membership in synchronous distributed systems",  In
           Proceedings of the 18th International Conference on Fault-
           Tolerant Computing. IEEE TOCS, 1988.

   Dee89   Deering, S., "Host Extensions for IP Multicasting", RFC 1112,
           Stanford University, August 1989.

   Fre84   Freier, A., "Compatability and interoperability", Open letter
           to XNS Interest Group, Xerox Systems Developement Division,
           December 13, 1984.

   JB89    Joseph T., and K. Birman, "Reliable Broadcast Protocols",
           pages 294-318, ACM Press, New York, 1989.

   Pos81   Postel, J., "Transmission Control Protocol - DARPA Internet
           Program Protocol Specification", RFC 793, DARPA, September
           1981.

   Xer81   Xerox Corp., "Internet Transport Protocols", Xerox System
           Integration Standard 028112, Stamford, Connecticut. December
           1981.

Footnotes ๐Ÿฆ‹

  • [1] okay so the network layer is NOT specified by MTP ๐ŸŒธ one of the goals is to specify a transport that can be implemented with equal functionality on many network architectures.
  • [2] there's only one such multicast connection identifier per web. if there are multiple processes on the same machine participating in a web, the transport must discriminate between those processes by using the connection identifier ๐ŸŽ€
  • [3] determining the network service access point (NSAP) for a given instantiation of a web is not addressed by this protocol. this document may define some policy, but the actual means are left for other mechanisms ๐Ÿงš
  • [4] best effort delivery is also known as highly reliable delivery. it's somewhat unique that the qualifying adjective "highly" weakens the definition of "reliable" in this context ๐Ÿ’€
  • [5] the resource being flow controlled is packets carrying client data. consequently, full data units provide the greatest efficiency โœจ
  • [6] there seems to be an opportunity to suppress retransmissions to networks that were not represented in the set of naks received ๐Ÿ‘€

Security Considerations ๐Ÿ”

bestie, this one is short and it is not giving much. the memo says security issues are not discussed in this memo. so there is zero security talk in here, and that's the whole tea ๐Ÿ™ˆ keep that in mind when you're building on it.


Authors' Addresses ๐Ÿซถ

the amazing humans who wrote this, plus the funding note at the bottom. verbatim, because contact info has to be exact ๐Ÿ’–

   Susan M. Armstrong
   Xerox Webster Research Center
   800 Phillips Rd. MS 128-27E
   Webster, NY 14580

   Phone: (716) 422-6437
   EMail: armstrong@wrc.xerox.com


   Alan O. Freier
   Apple Computer, Inc.
   20525 Mariani Ave. MS 3-PK
   Cupertino, CA 95014

   Phone: (408) 974-9196
   EMail: freier@apple.com


   Keith A. Marzullo
   Cornell University
   Department of Computer Science
   Upson Hall
   Ithaca, NY 14853-7501

   Phone: (607) 255-9188
   EMail: marzullo@cs.cornell.edu

      Keith Marzullo is supported in part by the Defense Advanced
      Research Projects Agency (DoD) under NASA Ames grant number NAG
      2-593, Contract N00140-87-C-8904.  The views, opinions and
      findings contained in this report are those of the authors and
      should not be construed as an official Department of Defense
      position, policy, or decision.

that's the whole RFC, bestie ๐ŸŒˆ๐Ÿ‘‘

๐Ÿ“ฆ where this page comes from