RFC 998

RFC 998: NETBLT: A Bulk Data Transfer Protocol 🦋

okay bestie, so this is RFC 998 and it's giving serious transport-layer main character energy 💅. It came out of the Network Working Group in March 1987, written by David D. Clark, Mark L. Lambert, and Lixia Zhang over at MIT 🎀. It obsoletes RFC 969, the older NIC RFC-969 spec, so 969 is basically old news now 🙈.

---

1. Status ✨

okay so this doc is a description of, and a specification for, NETBLT 💖. It's a revision of the spec from NIC RFC-969. They reworked it after doing a ton of research into how NETBLT performs over long-delay, high-bandwidth satellite channels 🛰️. Most of the changes are about how data timers get computed and used in a multiple buffering data transfer model ⏱️.

This document is published for discussion and comment, and it does not constitute a standard 🙅‍♀️. The proposal may change, and some parts of the protocol haven't been specified yet, so implementation of this document is therefore not advised 😭.

---

2. Introduction 🎀

NETBLT (NETwork BLock Transfer) is a transport level protocol meant for rapidly moving a big pile of data between computers 🚀. It's reliable and flow controlled, and it's designed to give maximum throughput over a wide variety of networks 💜. Right now NETBLT runs on top of the Internet Protocol (IP), but it should be able to operate on top of any datagram protocol that's similar in function to IP 🧚.

NETBLT's whole motivation is to get higher throughput than other protocols might offer ✨. It goes for this by trying to minimize the effect of a few network problems: network congestion, delays over satellite links, and packet loss 💅.

Its transmission rate-control algorithms handle congestion really well, its multiple-buffering capability allows high throughput over long-delay satellite channels, and its various timeout/retransmit algorithms minimize the effect of packet loss during a transfer 🫶. Most importantly, NETBLT's features give it good performance over long-delay channels without impairing performance over high-speed LANs 👑.

The protocol works by opening a connection between two "clients" (the "sender" and the "receiver"), moving the data in a series of big data aggregates called "buffers," and then closing the connection 🦋. The amount of data can be really huge, so the client is not required to hand over all of it to the protocol module at once 😳. Instead, the client provides the data in buffers. The NETBLT layer moves each buffer as a sequence of packets. Since each buffer is made up of a ton of packets, the per-buffer interaction between NETBLT and its client is way more efficient than a per-packet interaction would be 🍓.

In its simplest form, a NETBLT transfer goes like this: the sending client loads a buffer of data and calls down to the NETBLT layer to transfer it 📦. The NETBLT layer breaks the buffer up into packets and sends them across the network in Internet datagrams 📨. The receiving NETBLT layer loads these packets into a matching buffer provided by the receiving client 🪣. When the last packet in the buffer shows up, the receiving NETBLT checks that all the packets in that buffer were correctly received 🔍. If some are missing, the receiving NETBLT asks for them to be resent 🙋‍♀️. Once the buffer has been completely transmitted, the receiving client gets notified by its NETBLT layer. The receiving client disposes of the buffer and provides a new one to receive more data ♻️. The receiving NETBLT tells the sender that the new buffer is ready, and the sender prepares and sends the next buffer the same way 🔁. This keeps going until all the data has been sent. Then the sender tells the receiver that the transmission has been completed ✅, and the connection gets closed 🔒.

As described above, NETBLT is "lock-step" 🔐. Action halts after a buffer is transmitted, and begins again after confirmation is received from the receiver of the data 🛑. NETBLT provides for multiple buffering, a transfer model where the sending NETBLT can transmit new buffers while earlier buffers are still waiting for confirmation from the receiving NETBLT 🏃‍♀️💨. Multiple buffering makes the packet flow essentially continuous and markedly improves performance 🌈.

The rest of this document describes NETBLT in detail 📖. The next sections cover the philosophy behind a bunch of protocol features: packetization, flow control, transfer reliability, and connection management 🧠. The final sections describe NETBLT's packet formats 💜.

---

3. Buffers and Packets 📦

NETBLT is designed to permit transfer of a very large amount of data between two clients 🐘. During connection setup, the sending NETBLT can inform the receiving NETBLT of the transfer size. The maximum transfer length is 2**32 bytes 🤯. This limit should permit any practical application. The transfer size parameter is for the use of the receiving client, and the receiving NETBLT makes no use of it. A NETBLT receiver accepts data until told by the sender that the transfer is complete 🏁.

The data to be sent must be broken up into buffers by the client. Each buffer must be the same size, save for the last buffer. During connection setup, the sending and receiving NETBLTs negotiate the buffer size, based on limits provided by the clients. Buffer sizes are in bytes only, so the client is responsible for placing data in buffers on byte boundaries 📏.

NETBLT has been designed and should be implemented to work with buffers of any size 🌸. The only fundamental limitation on buffer size should be the amount of memory available to the client. Buffers should be as large as possible, since this minimizes the number of buffer transmissions and therefore improves performance 🚀.

NETBLT is designed to require a minimum amount of memory, so the client can allocate as much memory as possible for buffer storage 🧠. In particular, NETBLT does not keep buffer copies for retransmission purposes. Instead, data to be retransmitted is recopied directly from the client buffer. This means the client cannot release buffer storage piece by piece as the buffer is sent, but this has not been a problem in preliminary NETBLT implementations 🙈.

Buffers are broken down by the NETBLT layer into sequences of DATA packets. As with the buffer size, the DATA packet size is negotiated between the sending and receiving NETBLTs during connection setup. Unlike buffer size, DATA packet size is visible only to the NETBLT layer 🕵️‍♀️.

All DATA packets save the last packet in a buffer must be the same size. Packets should be as large as possible, since NETBLT's performance is directly related to packet size. At the same time, packets should not be so large as to cause internetwork fragmentation, since this normally causes performance degradation 🧩.

All buffers save the last buffer must be the same size; the last buffer can be any size required to complete the transfer. Since the receiving NETBLT doesn't know the transfer size in advance, it needs some way of identifying the last packet in each buffer 🕵️. For this reason, the last packet of every buffer is not a DATA packet but rather an LDATA packet. DATA and LDATA packets are identical save for the packet type 💅.

4. Flow Control ✨

okay bestie, NETBLT has two flow control strategies 💅. One lives inside the protocol, and the other lives at the client level 🫶.

The sending and receiving NETBLTs move data in buffers, so client flow control happens at the buffer level 🎀. Before a buffer can be sent, NETBLT checks that both clients have set up matching buffers, that one is ready to send, and that the other is ready to receive. That means either client can control the flow just by not providing a new buffer 🙅‍♀️. But once a buffer transfer is in progress, clients cannot stop it. no take-backs 💀

Buffers can be huge, though, so there has to be a different kind of flow control that runs during a buffer transfer. That's the job of the NETBLT layer 🧚.

Lots of flow control problems can pop up while a buffer is being sent 😭. If the sending NETBLT is pushing data faster than the receiving NETBLT can process it, the receiver's space for holding unprocessed packets can overflow, and packets get lost. A slow gateway or intermediate network can cause the same thing. Packets pile up, overflow the network's packet buffer space, and get lost inside the network. This is especially rough for NETBLT, because its buffers are usually big and made up of lots of packets 🌊.

The classic fix is a window system, where the sender is only allowed to send a certain number of packets at a time 🪟. The catch is that window-based flow control tends to give low throughput 🥲. Windows have to stay small so they don't overflow hosts and gateways, and they're hard to update, since every window change needs a full end-to-end exchange.

So to get high throughput across all kinds of networks and gateways, NETBLT uses a fresh approach called rate control 🦋. The sending and receiving NETBLTs negotiate the transmission rate during connection setup and again after each buffer transmission. The sender uses its own timers, not messages from the receiver, to keep to the negotiated rate ⏱️.

In its simplest form, rate control sets a minimum amount of time per packet transmission. That causes performance problems for a couple of reasons 🙈. First, sending a single packet takes so little time that it's often smaller than the timer's granularity. Second, keeping timers running on every single packet adds a lot of overhead, and that drags performance down.

The fix is to control the rate of groups of packets instead of single ones 💜. The sender fires off a burst of packets over a negotiated time interval, then sends another burst. That cuts the overhead by a factor of the burst size, and each burst takes long enough that the timers actually work properly. So NETBLT's rate control has two parts, a burst size and a burst rate, where (burst size)/(burst rate) equals the average transmission time per packet 🍓.

The burst size and burst rate should be based on more than just how fast each end can transmit and process packets. They also need to account for the capacity of any intermediate gateways or networks 🌐. Here are some sensible values for packet size, buffer size, burst size, and burst rate ✨.

Packet sizes can go as small as 128 bytes, but performance at that size is almost always bad, thanks to the high per-packet processing overhead 😬. Even the default Internet Protocol packet size of 576 bytes is barely big enough for decent performance. Most networks don't support packets much bigger than one or two thousand bytes, and packets that size can get fragmented while traveling across intermediate networks, which hurts performance too 🧩.

A NETBLT buffer is only limited by how much memory a client has. In theory, buffers of 100 Kbytes or more are possible 🤯. That would mean sending 50 to 100 packets per buffer.

The burst size and burst rate are super machine dependent 🖥️. Sending and receiving machines have some transmission overhead from maintaining timers and scheduling processes, and sending packets in big bursts can cut that overhead down. Burst size is also limited by how many packet buffers the operating system kernel has available. On most modern operating systems, a burst size of five to ten packets should bring overhead down to an acceptable level 👑. A preliminary NETBLT implementation for the IBM PC/AT sends packets in bursts of five. It could send more, but it's limited by the memory it has available.

The burst rate depends partly on how granular the sender's timer is, and partly on how fast the receiver and any intermediate gateways can process packets. It's also directly tied to the burst size 🔗. Burst rates of 20 to 45 milliseconds per 5-packet burst have been tried on the IBM PC/AT and Symbolics 3600 NETBLT implementations, with good results inside a single local-area network 🌸. Obviously, the right value depends on the network bandwidth and the packet buffering available.

All NETBLT flow control parameters (packet size, buffer size, burst size, and burst rate) are negotiated during connection setup, and the negotiation works the same way for every one of them 🤝. The client that starts the connection, the active end, proposes a set of values for each parameter in its connection request. The other client, the passive end, compares those values with the highest-performance values it can support. The passive end can then change any of the parameters, but it can only make them more restrictive 🚫. The modified parameters go back to the active end in its response message.

The burst size and burst rate can also be renegotiated after each buffer transmission, so the transfer rate can adjust based on how the previous buffer went 📈. The receiving end puts its burst size and burst rate values in its OK messages (more on those later). The sender compares them with what it can support. Again, it may modify any of the parameters, but only to make them more restrictive. The modified parameters are then sent to the receiver in a NULL-ACK packet, which is also covered later 💌.

Honestly, each of these parameters depends on a lot of stuff, like gateway and host processing speeds, available memory, and timer granularity 😵‍💫. Some of that stuff can't be checked by either client. So each client has to make the best guess it can, then tune for performance on later transfers 🎯.

---

5. The NETBLT Transfer Model 🚀

Every NETBLT transfer has three stages: connection setup, data transfer, and connection close 🪄. Each stage is described in detail below, along with the methods used to make sure each stage completes reliably.

5.1. Connection Setup 🤝

A NETBLT connection gets set up with an exchange of two packets between the active NETBLT and the passive NETBLT 💕. Here's the thing: either NETBLT can send or receive data. The words "active" and "passive" just tell apart the end that's making the connection request from the end that's responding to it. The active end sends an OPEN packet. The passive end can acknowledge that OPEN packet in one of two ways. It can send a REFUSED packet, which means the connection can't be completed for some reason. Or it can finish the setup by sending a RESPONSE packet. Once that happens, the transfer can begin 🎉.

Like we talked about in the last section, the OPEN and RESPONSE packets are used to negotiate flow control parameters. A couple of other parameters used in the data transfer also get negotiated 🧾. The first is the maximum number of buffers that can be sending at any one time. The second is whether or not DATA packet data will be checksummed. NETBLT automatically checksums all non-DATA/LDATA packets 🔐. If the negotiated checksum flag is set to TRUE (1), then both the header and the data of a DATA/LDATA packet get checksummed. If it's set to FALSE (0), only the header gets checksummed. The checksum value is the bitwise negation of the ones-complement sum of the 16-bit words being checksummed 🧮.

Last but not least, each end sends its death-timeout value, in seconds, in either the OPEN or the RESPONSE packet ⏳. The death-timeout value determines how often KEEPALIVE packets get sent during idle periods of an open connection. (Death timers and KEEPALIVE packets are covered in the next section, so stay tuned 👀.)

The active end picks out a passive client using a client-specific "well-known" 16 bit port number that the passive end listens on 📮. The active end identifies itself with a 32 bit Internet address and a unique 16 bit port number 🪪.

The OPEN and RESPONSE packets have an unstructured, variable-length field for any client-specific information that the two ends might need to share 🗂️. REFUSED packets have a "reason for refusal" field too 🙅.

Lost OPEN and RESPONSE packets are recovered with timers 🔁. When the active end sends an OPEN packet, it sets a timer. If the timer expires, it sends another OPEN packet, and it keeps doing that until it reaches some predetermined maximum number of OPEN packets. The timer gets cleared when a RESPONSE packet arrives 📬.

To stop OPEN and RESPONSE packets from being duplicated in a confusing way, the OPEN packet carries a 32 bit connection unique ID that has to come back in the RESPONSE packet 🆔. This keeps the initiator from mixing up the response to its current request with the response to an earlier connection request. (There can only be one connection between any two ports, btw 💅.) Any OPEN or RESPONSE packet with a destination port that matches an open connection gets its unique ID checked. If the packet's unique ID matches the connection's unique ID, then the packet type gets checked. A RESPONSE packet gets treated as a duplicate and ignored 🙄. An OPEN packet means the passive NETBLT sends another RESPONSE, since a previous RESPONSE was probably sent and lost, which made the initiating NETBLT resend its OPEN packet 🔄. A non-matching unique ID has to be treated as an attempt to open a second connection between the same port pair, so it gets rejected by sending an ABORT message 🚫✨.

5.2. Data Transfer 💖

okay so the simplest version of data transfer goes like this 💅 the sending client sets up a buffer full of data. then the receiving NETBLT sends a GO message inside a CONTROL packet to the sender, basically saying "hey bestie, i've set up my own buffer and i'm ready to receive" 🎀 once the GO is received, the sender transmits the buffer as a series of DATA packets followed by an LDATA packet. when the last packet in the buffer has arrived, the receiver sends a RESEND message inside a CONTROL packet, and that message has a list of packets that were not received. the sender resends those. this keeps going until there are no missing packets 🔁 at that point the receiver sends an OK message inside a CONTROL packet, sets up another buffer to receive data, and sends another GO message. the sender, having received the OK, sets up another buffer, waits for the GO, and repeats the process ✨

Honestly though, this lock-step model is lowkey wasteful 😭 the sending NETBLT just sits there waiting for permission to send a new buffer, and that's time literally thrown away. the fix is a more efficient model that uses multiple buffering to boost performance 🚀 multiple buffering is when the sender and receiver allocate and transmit buffers in a way that lets error recovery or successful transmission confirmation of previous buffers happen at the same time as the transmission of the current buffer. it's giving multitasking, no cap 🧚

During connection setup, one of the negotiated parameters is the number of concurrent buffers permitted during the transfer 🌸 if there's more than one buffer available, the transfer of the next buffer can start right after the current buffer finishes. here's an example:

Assume two buffers A and B in a multiple-buffer transfer, with A preceding B. When A has been transferred and the sending NETBLT is waiting for either an OK or a RESEND message for it, the sending NETBLT can start sending B immediately, keeping data flowing at a stable rate. If the receiver of data sends an OK for A, all is well; if it receives a RESEND, the missing packets specified in the RESEND message are retransmitted. 💜

In the multiple-buffer transfer model, all packets to be sent are re-ordered by buffer number (lowest number first), with the transfer rate specified by the burst size and burst rate 🍓 since buffer numbers increase monotonically, packets from an earlier buffer will always precede packets from a later buffer. so the order is never messy, periodt 💅

Having several buffers transmitting at once is actually not that much more complicated than sending one buffer at a time 🦋 the trick is to picture each buffer as a finite state machine. several buffers are just a group of finite state machines, each sitting in one of several states, and the transfer process is all about moving buffers through those states until the entire transmission is complete. it's iconic, honestly 👑

There are a few obvious flaws in the data transfer model as described above 😬 first, what if the GO, OK, or RESEND messages get lost? the sender can't act on a packet it hasn't received, so the protocol would hang. second, if an LDATA packet is lost, how does the receiver know when the buffer has been transmitted? don't worry, solutions for each of these problems are coming up right below 🫶

5.2.1. Recovering from Lost Control Messages 🦋

okay this one is the fix for lost OK, GO, and RESEND messages, and NETBLT handles it in two ways 🎀 the first way is a control timer. the receiver can send one or more control messages (OK, GO, or RESEND) within a single CONTROL packet. whenever the receiver sends a control packet, it sets a control timer. this timer is either "reset" (set again) or "cleared" (deactivated), under the following conditions:

When the control timer expires, the receiving NETBLT resends the control packet and resets the timer. the receiving NETBLT keeps resending control packets every time the control timer expires, until either the control timer is cleared or the receiving NETBLT's death timer (described later) expires, at which point it shuts down the connection 💀

Each control message includes a sequence number, which starts at one and goes up by one for each control message sent 🔢 the sending NETBLT checks the sequence number of every incoming control message against all the other sequence numbers it has received. it stores the highest sequence number below which all the other received sequence numbers are consecutive (in the following paragraphs this is called the high-acknowledged-sequence-number), and it returns this number in every packet flowing back to the receiver ✨ the receiver is permitted to clear its control timer when it receives a packet from the sender with a high-acknowledged-sequence-number greater than or equal to the highest sequence number in the control packet just sent. main character energy for the receiver, finally getting to relax 🌈

Ideally, a NETBLT implementation should be able to cope with out-of-sequence control messages 🧠 it could collect them for later processing, or even process them immediately. and if an incoming control message "fills" a "hole" in a group of message sequence numbers, the implementation could even be clever enough to detect this and adjust its outgoing sequence value accordingly. galaxy brain stuff 🌌

The sending NETBLT, upon receiving a CONTROL packet, should act on the packet as quickly as possible ⚡ it either sets up a new buffer (upon receipt of an OK message for a previous buffer), marks data for resending (upon receipt of a RESEND message), or prepares a buffer for sending (upon receipt of a GO message). if the sending NETBLT isn't in a position to send data, it should send a NULL-ACK packet, which contains its high-acknowledged-sequence-number. this lets the receiving NETBLT acknowledge any outstanding control messages, and then the sender waits until it can send more data 🫠 in all of these cases, the system overhead for a response to the incoming control message should be small and relatively constant. no drama, just vibes 💖

This small amount of message-processing overhead means accurate control timers can be set for all types of control messages with a single, simple algorithm: the network round-trip transit time, plus a variance factor 🎯 this is way more efficient than the schemes other protocols use, where timer value calculation has been a problem, because the processing time for a particular packet can vary a lot depending on the packet type 🙈

Control timer value estimation is extremely important in a high-performance protocol like NETBLT 📈 a long control timer makes the receiving NETBLT wait for long periods before retransmitting unacknowledged messages, which is so not it. a short control timer value makes the sending NETBLT receive lots of duplicate control messages, which it can reject, but rejecting them still takes time 😤

On top of control timers, NETBLT cuts down on lost control messages by using a single long-lived control packet 🎀 this packet is treated like a FIFO queue: new control messages get added at the end, and acknowledged control messages get removed from the front. the implementation places control messages in the control packet and transmits the entire control packet, which holds any unacknowledged control messages plus the new messages just added. the entire control packet is also transmitted whenever the control timer expires. since control packet transmissions are fairly frequent, unacknowledged messages may get sent several times before they're finally acknowledged 🔁 this redundant transmission of control messages provides automatic recovery for most control message losses over a noisy channel. it's giving "send it again until it lands" energy 💜

This scheme puts some burdens on the receiver of the control messages 🥲 it must be able to quickly reject duplicate control messages, since a given message may be retransmitted several times before its acknowledgement is received and it gets removed from the control packet. typically this is pretty easy to do: the sender of data just throws away any control messages with sequence numbers lower than its high-acknowledged-sequence-number 🗑️

Another problem with this scheme is that the control packet may become larger than the maximum allowable packet size if too many control messages are placed into it 😬 this hasn't been a problem in the current NETBLT implementations. a typical control packet size is 1000 bytes, RESEND control messages average about 20 bytes in length, GO messages are 8 bytes long, and OK messages are 16 bytes long. that allows 50-80 control messages to fit in the control packet, which is more than enough for reasonable transfers 🙌 other implementations can provide for multiple control packets if a single control packet isn't sufficient 🧚

The control timer value must be carefully estimated 🧮 it can have an arbitrary number as its initial value. subsequent control packets should have their timer values based on the network round-trip transit time (i.e. the time between sending the control packet and receiving the acknowledgment of all messages in the control packet) plus a variance factor. the timer value should be continually updated, based on a smoothed average of collected round-trip transit times 📊 the vibe is: keep measuring, keep adjusting, stay iconic 👑

5.2.2. Recovering from Lost LDATA Packets 🦋

Okay so, LDATA packets can get lost, and NETBLT's fix is a data timer for every buffer at the receiving end 💖 The simplest version starts a data timer when a buffer is ready to be received. If that timer expires, the receiver assumes an LDATA packet went missing and sends a RESEND asking for all the missing DATA packets in the buffer. Once every packet has arrived, the timer gets cleared ✨

Data timer values aren't based on the network's round-trip transit time. They're based on how long it takes to transfer a buffer, which is the number of DATA packet bursts in the buffer times the burst rate, plus a variance factor <1> 🧚

Getting the data timer value right is super important, bestie 🙈 A short data timer makes the receiving NETBLT send unnecessary RESEND packets, and that causes serious performance degradation because the sending NETBLT has to stop what it's doing and resend a bunch of DATA packets. Major mess 💀

Setting and clearing data timers is honestly pretty complicated, especially in a multiple-buffering transfer model 😭 The trick is to picture each buffer as a finite-state machine and look at its different states 🌸

The state sequence for a sending buffer is simple 💅 When a GO message for the buffer is received, the buffer is created, filled with data, and placed in a SENDING state. When an OK for that buffer has been received, it goes into a SENT state and is disposed of.

The state sequence for a receiving buffer is a little more complicated 👑 Assume a buffer A exists. When a control message for A is sent, the buffer moves into ACK-WAIT (it's waiting for acknowledgement of the control message).

As soon as the control message is acknowledged, buffer A moves from ACK-WAIT into ACKED (now it's waiting for DATA packets to arrive). At this point, A's data timer is set and the control message is removed from the control packet. Estimating the data timer value right now is quite difficult 🥺 In a multiple-buffer transfer model, the receiving NETBLT can send several GO messages at once. A single DATA packet from the sending NETBLT could acknowledge all of those GO messages, which would start several buffers' data timers at the same time. Each data timer has to account for that buffer's place in the order of transmission. Packets for buffer A - 1 always get transmitted before packets in A, so A's data timer has to factor in the arrival of all of A - 1's DATA packets as well as its own. That means the timer values get less accurate for higher-numbered buffers 🫠 Because this data timer value can be pretty inaccurate, it's called a "loose" data timer. The loose data timer value gets recalculated later (using the same algorithm, but with updated information), which gives a "tight" timer, as described below 🎀

When the first DATA packet for A arrives, A moves from the ACKED state to the RECEIVING state, and its data timer is set to a new "tight" value. The tight timer is calculated the same way as the loose one, but it's more accurate because time has moved forward. Buffers numbered lower than A have presumably been dealt with (or their packets would have arrived before A's), so fewer packets are left to arrive between setting the data timer and the arrival of the last DATA packet in A 🦋

The receiving NETBLT also sets the tight data timers of any buffers numbered lower than A that are in the ACKED state. This is an optimization, and it works because we know buffers are processed in order, lowest number first. If a buffer B numbered lower than A is in the ACKED state, its DATA packets should have arrived before A's. Since A's arrived first, B's must have gotten lost 😱 B's loose data timer hasn't expired yet (if it had, it would have sent a RESEND message and be in the ACK-WAIT state), so we set the tight timer, which lets us spot the missing packets sooner ⭐ An immediate RESEND isn't sent, though, because the network might have reordered A's packet ahead of B's, and B's packets could show up shortly 🙏

When all DATA packets for A have been received, A moves from the RECEIVING state to the RECEIVED state and is disposed of 🎉 If any packets had been missing, A's data timer would have expired, and A would have moved into ACK-WAIT after sending a RESEND message. From there, the state progression goes just like the example above.

Here's the big picture on the control and data timer system 🌈 Normally, the receiving NETBLT is running under one of two kinds of timers: a control timer or a data timer. There's one data timer per buffer transmission and one control timer per control packet. The data timer is active while its buffer is in either the ACKED state (the loose data timer value is used) or the RECEIVING state (the tight data timer value is used). A control timer is active whenever the receiving NETBLT has any unacknowledged control messages in its control packet ✨

5.2.3. Death Timers and Keepalive Packets 💜

The system above still leaves a few problems, babe 😤 If the sending NETBLT isn't ready to send, it sends a single NULL-ACK packet to clear any outstanding control timers at the receiving end. After that, the receiver just waits. If the sending NETBLT dies, the receiver, with its control timer cleared, would hang forever 💀 Also, the system above only puts timers on the receiving NETBLT. The sending NETBLT has no timers at all, so if the receiving NETBLT dies, the sending NETBLT hangs while waiting for control messages to arrive.

The fix for both of these is a death timer and a keepalive packet, used by both the sending and receiving NETBLTs 🫶 As soon as the connection opens, each end sets a death timer, and it gets reset every time a packet is received. When a NETBLT's death timer expires, it can assume the other end has died and close the connection.

Here's the catch: the sending or receiving NETBLT might have to wait for long stretches while its client gets buffer space and loads its buffers with data 🥱 A NETBLT waiting for buffer space is in a perfectly valid state, so the protocol needs a way to stop the other end's death timer from expiring. The solution is a KEEPALIVE packet, which gets sent repeatedly at fixed intervals whenever a NETBLT can't send other packets. Since the death timer resets whenever a packet is received, it will never expire as long as the other end keeps sending packets 🧚

Here's how the KEEPALIVE frequency gets worked out 🧮 At connection startup, each NETBLT chooses a death-timer value and sends it to the other end in either the OPEN or the RESPONSE packet. The other end takes that death-timeout value and uses it to compute how often to send KEEPALIVE packets. The KEEPALIVE frequency should be high enough that several KEEPALIVE packets can be lost before the other end's death timer expires (e.g. death timer value divided by four).

The death timer value is pretty easy to estimate 🌸 Since it keeps getting reset, it doesn't need to be based on the transfer size. Instead, it should be based at least in part on the type of application using NETBLT. User applications should have smaller death timeout values so humans don't have to wait forever for a death timeout to happen 🙃 Machine applications can have longer timeout values.

5.3. Closing the Connection 🎀

There are three ways to close a connection: a connection close, a "quit", or an "abort" 🚪

5.3.1. Successful Transfer 🥳

After a successful data transfer, NETBLT closes the connection 🎉 When the sender is transmitting the last buffer of data, it sets a "last-buffer" flag on every DATA packet in that buffer. This means no NEW data will be transmitted. The receiver knows the transfer completed successfully when all of the following are true: (1) it has received DATA packets with a "last-buffer" flag set, (2) all of its control messages have been acknowledged, and (3) it has no outstanding buffers with missing packets. At that point, the receiver is allowed to close its half of the connection. The sender knows the transfer has completed when the following are true:

(1) it has transmitted DATA packets with a "last-buffer" flag set, and (2) it has received OK messages for all of its buffers. At that point, it "dallies" for a predetermined period of time before closing its half of the connection 😌 If the NULL-ACK packet acknowledging the receiver's last OK message was lost, the receiver has time to retransmit the OK message, receive a new NULL-ACK, and recognize a successful transfer. The dally timer value MUST be based on the receiver's control timer value. It has to be long enough for the receiver's control timer to expire so the OK message can be re-sent. For this reason, all OK messages contain (in addition to new burst size and burst rate values) the receiver's current control timer value in milliseconds. The sender uses this value to compute its dally timer value 🧠

Since the dally timer value can be quite large, the receiving NETBLT is permitted to "short-circuit" the sending NETBLT's dally timer by transmitting a DONE packet 🏃‍♀️ The DONE packet gets transmitted when the receiver knows the transfer has been successfully completed. When the sender receives a DONE packet, it's allowed to clear its dally timer and close its half of the connection immediately ✨ The DONE packet isn't reliably transmitted, since failing to receive it only means the sending NETBLT takes longer to close its half of the connection (as it waits for its dally timer to clear) 🫠

5.3.2. Client QUIT 🚫

During a NETBLT transfer, one client may send a QUIT packet to the other if it thinks the other client is malfunctioning 😬 Since the QUIT happens at the client level, the QUIT transmission can only occur between buffer transmissions. The NETBLT receiving the QUIT packet can take no action other than immediately notifying its client and transmitting a QUITACK packet. The QUIT sender has to time out and retransmit until a QUITACK has been received or its death timer expires. The sender of the QUITACK dallies before quitting, so it can respond to a retransmitted QUIT 💅

5.3.3. NETBLT ABORT 🛑

An ABORT happens when a NETBLT layer thinks that it or its opposite is malfunctioning 😱 Since the ABORT comes from the NETBLT layer, it can be sent at any time. The ABORT implies that the NETBLT layer is malfunctioning, so no transmit reliability is expected, and the sender can immediately close its connection 🥀

6. Protocol Layering Structure 🧁

NETBLT is implemented directly on top of the Internet Protocol (IP). It has been assigned an official protocol number of 30 (decimal) 🔢

7. Planned Enhancements 🔮

As currently specified, NETBLT has no algorithm for determining its rate-control parameters (burst rate, burst size, etc.). In initial performance testing, these parameters were set by the person performing the test. We're now exploring ways to have NETBLT set and adjust its rate-control parameters automatically 🤖

8. Packet Formats ✨

okay so the last message type on the list is the RESEND one 🔁 RESEND Message (type 2): 💅

Okay bestie, let's look at this little packet picture 🦋 The top rows are the header, with Type taking the first byte, then Word Padding, then the 16-bit Sequence Number 🎀

                      1                   2                   3
    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
   +---------------+---------------+---------------+---------------+
   |    Type       | Word Padding  |       Sequence Number         |
   +---------------+---------------+---------------+---------------+
   |                        Buffer Number                          |
   +---------------+---------------+---------------+---------------+
   |  Number of Missing Packets    | Longword Alignment Padding    |
   +---------------+---------------+---------------+---------------+
   |       Packet Number (2 bytes) ...
   +---------------+---------------+----------
                                   |    Padding (if necessary)     |
                        -----------+---------------+---------------+

Next up is the Buffer Number, which fills a whole 32-bit row all by itself 🌸 Then comes Number of Missing Packets sharing a row with Longword Alignment Padding 💅 The Packet Number field is the one that's giving "it depends on the packet" energy, since it's 2 bytes and the diagram just trails off with ... 🙈 Anything after that is Padding (if necessary), so you only add it when the alignment actually needs it ✨

Quick definition so we're on the same page: the packet number is the 16 bit data packet identifier that shows up in every DATA packet 💜

---

NOTES

<1> Okay so this one is lowkey wild 🧚 When the buffer is big, the round trip delay variances of lots of packets can cancel each other out 🌈 That means the variance value doesn't have to be very large. The authors say this expectation still needs to be checked in further testing, so it's a theory, not a fact yet 🔥 Iykyk, science is never finished 💀