RFC 969

RFC 969 📮 NETBLT: A Bulk Data Transfer Protocol 🦋

okay so bestie, this is the very first part of a four-part rewrite of RFC 969, which is the NETBLT spec from December 1985. NETBLT is a bulk data transfer protocol 🚚. It's written by David D. Clark, Mark L. Lambert, and Lixia Zhang from the M.I.T. Laboratory for Computer Science, and it's an experimental proposal, not a standard, so it's giving "draft we're still cooking" energy 🍳. It's a Network Working Group document, and the title block says it doesn't obsolete or update anything, so it's a fresh start ✨.

Heads up: the title block does not say anything about this being a standard or what category it's in beyond the status section below, so we'll stick with what the status section says 💅.

---

1. STATUS OF THIS MEMO 📋

Okay so this part is basically the disclaimer, and it's important. This RFC suggests a proposed protocol for the ARPA-Internet community and asks for discussion and suggestions for improvements 💬. It's a preliminary discussion of NETBLT, published for discussion and comment, and it does not constitute a standard 🙅‍♀️. Since the proposal may change, implementing this document is NOT ADVISED 🚫. Distribution of this memo is unlimited 🌍.

Quick reminder: "not advised" here is the vibe, not a BCP 14 key word, so nothing in this doc is a hard requirement yet. Don't build your whole life on it 😭.

---

2. INTRODUCTION 🎀

2.1. What NETBLT Is

Okay so NETBLT stands for Network Block Transfer 🧱, and it's a transport level protocol built for rapidly moving a LOT of data between computers 💨. The big flex is that it's reliable and flow controlled, and it's structured to squeeze out maximum throughput across a wide variety of networks 🏎️💖.

2.2. How a Connection Goes

So the protocol opens a connection between two clients (the sender and the receiver) 🤝, transfers the data as a series of big chunks called buffers 📦, and then closes the connection 👋.

Here's the thing that's lowkey genius 🧠: the amount of data can be arbitrarily large, so the client does not have to hand over everything at once 😮. Instead, the client gives the protocol one buffer at a time 🫶. NETBLT then sends each buffer as a sequence of packets. Since every buffer has a LOT of packets in it, the back-and-forth between NETBLT and its client happens per buffer, not per packet, and that's way more efficient 🌸.

2.3. The Simplest Version

Okay so in its simplest form, here's how a transfer goes 🎬:

1. The sending client loads a buffer of data and calls down to the NETBLT layer to transfer it 📤.
2. The sending NETBLT layer breaks the buffer into packets and sends them across the network in Internet datagrams 💌.
3. The receiving NETBLT layer loads those packets into a matching buffer that the receiving client provided 📥.
4. When the last packet in the buffer has been transmitted, the receiving NETBLT checks that every packet in that buffer arrived 🔍.
5. If some packets are missing, the receiving NETBLT asks for them to be resent 🔁.
6. Once the whole buffer has arrived, the receiving client gets notified by its NETBLT layer 🔔.
7. The receiving client disposes of the buffer and gives a new one to receive more data 🧺.
8. The receiving NETBLT tells the sender that the buffer arrived, and the sender prepares and sends the next buffer the same way 🔄.
9. This keeps going until every buffer has been sent, and then the sender tells the receiver the transmission is complete 🏁.
10. The connection closes 💤.

2.4. Lock-Step vs. Multiple Buffering

Okay so as described above, NETBLT is "lock-step" 🔒. Action halts after a buffer is transmitted and only starts again once the receiver confirms it got the data 🛑. But NETBLT also provides multiple buffering, where several buffers can be transmitted concurrently 🚀. This makes packet flow essentially continuous and can improve performance markedly 📈✨. Literally the glow-up 💅.

2.5. What's Coming Up

The rest of the document goes deep on NETBLT 🤓. The next sections explain the philosophy behind a bunch of features: packetization, flow control, reliability, and connection management 🧩. The final sections cover the protocol format 📐.

---

3. BUFFERS AND PACKETS 📦🧚

3.1. Transfer Size

Okay so NETBLT is designed to move an essentially arbitrary amount of data between two clients 🌈. During connection setup, the sending NETBLT can optionally tell the receiving NETBLT the transfer size 📏. The maximum transfer length is set by the field width and is 2**32 bytes 💯. The original doc says that limit should permit any practical application, and honestly, same 🙌.

The transfer size is for the receiving client to use, and the receiving NETBLT doesn't use it at all 🙈. A NETBLT receiver just keeps accepting data until the sender says the transfer is complete 🛎️.

3.2. Buffer Size

The client is the one who breaks its data into buffers 🔪. Each buffer must be the same size, except for the last buffer 🎯. During connection setup, the sending and receiving NETBLTs negotiate the buffer size, based on limits the clients provide 🤝. Buffer sizes are measured in bytes only, so the client is responsible for splitting data on byte boundaries 🧮.

NETBLT is designed, and should be implemented, to work with buffers of arbitrary size 🎈. The only real limit on buffer size should be how much memory the client has 🧠. Buffers should be as large as possible, because that means fewer buffer transmissions and better performance 🚄.

3.3. Memory Use

NETBLT is designed to need as little of its own memory as possible, so the client can give as much memory as possible to buffer storage 💜. In particular, NETBLT does not keep buffer copies for retransmission 🙅‍♀️. Instead, data that needs retransmitting gets recopied straight from the client's buffer 🔁. The tradeoff is that the client cannot release buffer storage piece by piece while the buffer is being sent 😬. The original says this hasn't been a problem in the preliminary implementations, so we're doing okay 🫶.

3.4. Packets

Buffers get broken down by the NETBLT layer into sequences of DATA packets 🧃. Just like the buffer size, the packet size is negotiated between the sending and receiving NETBLTs during connection setup 🤝. The difference is that packet size is only visible to the NETBLT layer, not the client 👀.

All DATA packets, except the last packet in a buffer, must be the same size 📐. Packets should be as large as possible, because in most cases (including the preliminary implementation) performance is directly tied to packet size 🏋️‍♀️. At the same time, packets should not be so large that they cause Internet fragmentation, since that normally hurts performance 🙅‍♀️💔.

3.5. The Last Buffer and the LDATA Packet

Okay so all buffers except the last one must be the same size, and the last buffer can be any size needed to finish the transfer 🏁. The receiving NETBLT doesn't know the transfer size ahead of time, so it needs a way to spot the last packet in each buffer 👀. That's why the last packet of every buffer is not a DATA packet but an LDATA packet 🎁. DATA and LDATA packets are identical except for the packet type 🥰.

Main character moment: the LDATA packet is the receiver's little signpost for "this is where the buffer ends" 🪧✨.

---

Part 1 of 4 is done bestie 💖. Parts 2, 3, and 4 are coming next, so stay tuned 🎀🦋.

4. FLOW CONTROL ✨

okay bestie, so NETBLT has two flow control strategies 💅 one lives inside the NETBLT layer and one lives at the client level 🎀

The sending and receiving NETBLTs move data in buffers, so client flow control happens at the buffer level 🧚 Before a buffer can go out, NETBLT checks that both clients have set up matching buffers, that one side is ready to send, and that the other side is ready to receive 🫶 That means either client can control the flow just by not handing over a new buffer. But once a buffer transfer has started, clients can't stop it 😭

Buffers can be huge, so there has to be a different flow control method that runs during a buffer transfer. That's the one the NETBLT layer provides 🦋

There are a few flow control problems that can pop up while a buffer is being sent 💀 If the sending NETBLT is blasting data faster than the receiving NETBLT can process it, the receiver's space for holding unprocessed packets can overflow and packets get lost 😱 A slow gateway or an in-between network can do the same thing: packets pile up and overflow the network's packet buffer space. Then packets get lost inside the network and performance tanks 📉 This is extra intense for NETBLT because its buffers are usually big, which means they're made of tons of packets 🍓

The classic answer to packet flow control is a window system, where the sender is only allowed to send a certain number of packets at a time 🌸 The tea though: windows usually give you low throughput 😤 They have to stay small so they don't overflow hosts and gateways, and they're hard to update because every change needs a full end-to-end exchange 🙈

So to get high throughput across all kinds of networks and gateways at different speeds, NETBLT uses a totally new method called rate control 👑 The transmission rate gets negotiated by the sending and receiving NETBLTs during connection setup and again after each buffer transmission. The sender uses its own timers to keep to that rate, not messages from the receiver ⭐

In its simplest form, rate control sets a minimum amount of time between each packet 🥺 That causes performance problems for two reasons. First, one packet takes such a tiny amount of time to transmit that it's often smaller than the granularity of the timing mechanism 💅 Second, keeping timers running for every single packet has a lot of overhead, which slows things down 😩

The fix is to control the rate of groups of packets instead of single ones 🎀 The sender sends a burst of packets over a negotiated interval, then sends another burst. This way the overhead drops by a factor of the burst size, and the per-burst rate is big enough for the timing mechanism to actually work properly ✨ So NETBLT's rate control has two parts, a burst size and a burst rate, and (burst size)/(burst rate) equals the average transmission rate per packet 🧚

The burst size and burst rate should be based on more than just how fast each end can transmit and process packets. They also need to account for the capacities of the gateways and networks sitting in between 🌈 Here are some intuitive values for packet size, buffer size, burst size, and burst rate 💖

Packet sizes can be as small as 128 bytes 🙃 Performance with packets that tiny is almost always bad because the per-packet processing overhead is so high 😭 Even the default Internet Protocol packet size of 576 bytes is barely big enough for decent performance 😬 Most networks don't support packets much bigger than one or two thousand bytes, and packets that size can also get fragmented while they travel over intermediate networks, which hurts performance too 💔

The size of a NETBLT buffer is only limited by how much memory a client has 🧠 In theory, buffers of 100K bytes or more are possible 🤯 That would mean sending 50 to 100 packets per buffer 🍓

The burst size and burst rate are super machine dependent 💅 The sending and receiving machines have some transmission overhead from maintaining timers and scheduling processes ⏰ That overhead gets smaller when you send packets in large bursts 🌸 Burst size is also limited by how many packet buffers are available ✨ On most modern operating systems, a burst size of five to ten packets should bring the overhead down to an acceptable level 🫶 Fun fact: a preliminary NETBLT implementation for the IBM PC/AT sends packets in bursts of five 🎀 It could send more, but it's limited by available memory 😩

The burst rate is partly determined by the granularity of the sender's timing mechanism, and partly by the processing speed of the receiver and any intermediate gateways 🦋 It's also directly related to the burst size 💜 Burst rates from 60 to 100 milliseconds were tried on the preliminary NETBLT implementation and gave good results within a single local-area network 🌟 That value obviously depends on the network bandwidth and the packet buffering available 🧚

All NETBLT flow control parameters (packet size, buffer size, burst size, and burst rate) get negotiated during connection setup 🤝 The negotiation works the same way for every parameter. The client that starts the connection (the active end) proposes a set of values for each parameter and sends them along with its open connection request 💌 The other client (the passive end) compares those values with the highest-performance values it can support 👀 The passive end can change any parameter, but only to make it more restrictive 🙅‍♀️ The modified parameters go back to the active end in the response message 🎀

On top of that, the burst size and burst rate can be re-negotiated after each buffer transmission, so the transfer rate can adjust based on how the previous buffer performed 📈 The receiving end sends a pair of burst size and burst rate values in the OK message 💖 The sender compares those with the values it can support and, again, can only change them to be more restrictive 🔒 The modified parameters are then communicated to the receiver in a NULL-ACK packet, which gets described later 🔜

Real talk: all of these parameters depend on a ton of factors like gateway and host processing speeds, available memory, and timer granularity 🤯 Some of those can't be checked by either client 😬 So each client has to make the best guess it can and tune for performance on later transfers 🦋 Learning era, bestie 🌈

---

5. RELIABILITY 🛡️

Each NETBLT transfer has three stages: connection setup, data transfer, and connection close 💫 Every stage has to finish reliably, and the ways to make that happen are described below 🫶

5.1. Connection Setup

A NETBLT connection gets set up with an exchange of two packets between the active client and the passive client 🤝 Heads up, either client can send or receive data 💅 The words "active" and "passive" are just there to tell apart the client that starts the connection process from the client responding to the connection request 🎀 The first packet sent is an OPEN packet, and the passive end acknowledges it by sending a RESPONSE packet ✨ Once those two packets have been swapped, the transfer can begin 🚀

Like we talked about in the last section, the OPEN and RESPONSE packets are used to negotiate flow control parameters 🧚 Other parameters used in moving data also get negotiated here 🌸 Those are (1) the maximum number of buffers that can be sending at any one time (this allows multiple buffering and higher throughput) and (2) whether or not DATA/LDATA packet data gets checksummed 🔍 NETBLT automatically checksums all non-DATA/LDATA packets 💯 If the negotiated checksum flag is set to TRUE (1), then both the header and the data of a DATA/LDATA packet get checksummed 🙌 If it's set to FALSE (0), only the header gets checksummed 🤏 NETBLT uses the same checksumming algorithm that TCP uses 🧠

Last but not least, each end sends its death-timeout value in either the OPEN or the RESPONSE packet ⏳ That value decides how often KEEPALIVE packets get sent during idle periods of an open connection 💓 (Death timers and KEEPALIVE packets are covered in the next section, so stay tuned 👀)

The active end picks out a passive client using a client-specific "well-known" 16 bit port number that the passive end listens on 📮 The active end identifies itself with a 32 bit Internet address and a 16 bit port number 🪪

To let the active and passive ends share miscellaneous useful info, the OPEN and RESPONSE messages have an unstructured, variable-length field for any client-specific information that might be needed 🎁

Lost OPEN and RESPONSE packets get recovered with timers ⏰ When the active end sends an OPEN packet, it starts a timer. When that timer expires, another OPEN packet goes out, and this keeps happening until some predetermined maximum number of OPEN packets have been sent 🔁 The passive end uses a similar scheme when it sends a RESPONSE packet 🔁 When the active end receives a RESPONSE packet, it clears its timer ✅ The passive end's timer gets cleared when it receives either a GO or a DATA packet, which is explained in the data transfer section 🫶

To stop OPEN and RESPONSE packets from being duplicated, the OPEN packet carries a 32 bit connection unique ID that has to come back in the RESPONSE packet 🆔 This keeps the initiator from mixing up the response to the current request with the response to an earlier connection request (only one connection can exist between any two ports) 🚫 Any OPEN or RESPONSE packet whose destination port matches an open connection gets its unique ID checked 🕵️‍♀️ A matching unique ID means it's a duplicate packet, so it gets ignored 🙈 A non-matching unique ID must be treated as an attempt to open a second connection between the same port pair, and it must be rejected by sending an ABORT message 🛑

5.2. Data Transfer

Here's the simplest model of data transfer, bestie 🎀 The sending client sets up a buffer full of data 📦 The receiving NETBLT sends a GO message inside a CONTROL packet to the sender, saying it has set up its own buffer too and is ready to receive data into it 🟢 Once the GO message is received, the sender transmits the buffer as a series of DATA packets followed by an LDATA packet 🚚 When the last packet in the buffer has arrived, the receiver sends a RESEND message inside a CONTROL packet, listing the packets that didn't arrive 📋 The sender resends those packets 🔁 This keeps going until there are no missing packets, and then the receiver sends an OK message inside a CONTROL packet to the sender, sets up another buffer to receive data, and sends another GO message 🎉 The sender, having received the OK message, sets up another buffer, waits for the GO message, and repeats the whole process 💜

Okay so there are a few obvious flaws with this scheme 🙃 First, if the LDATA packet gets lost, how does the receiver even know when the buffer is done transmitting? 😭 Second, what if the GO, OK, or RESEND messages get lost? The sender can't act on a packet it never received, so the whole protocol just hangs 💀 Lowkey a nightmare. The fix is two kinds of timers: a data timer and a control timer ⏰✨

NETBLT handles the lost LDATA problem with a data timer on the receiving end. When the first DATA packet in a buffer shows up, the receiving NETBLT starts its data timer, and at the same time it clears its control timer (more on that in a sec) 🎀 If the data timer expires, the receiver assumes the buffer is done transmitting and every missing packet is lost. So it sends a RESEND message with a list of the missing packets 🦋

For the missing OK, GO, and RESEND messages, NETBLT uses a control timer 💅 The receiver can send one or more control messages (OK, GO, or RESEND) inside a single CONTROL packet. Every time it sends a control packet, it starts a control timer, and at the same time it clears its data timer if one has been set 🌸

Now, how does the control timer get cleared? Each control message carries a sequence number that starts at one and goes up by one for every control message sent 🔢 The sending NETBLT checks the sequence number on every incoming control message against all the other sequence numbers it has already received. It keeps track of the highest sequence number where everything below it is consecutive, and returns that number in every packet going back to the receiver 🫶 The receiver is allowed to clear the control timer of every packet with a sequence number equal to or lower than the one the sender returned. Bestie, that's giving clean bookkeeping ✨

Ideally, a NETBLT implementation should be able to handle out-of-sequence messages 🧚 It could collect them for later, or even process them right away. And if an incoming control message fills a "hole" in a run of message sequence numbers, a really clever implementation could notice that and adjust its outgoing sequence value to match 👑 Main character energy, honestly.

When the control timer expires, the receiving NETBLT resends the control message and resets the timer 🔁 After a predetermined number of resends, the receiving NETBLT can assume the sending NETBLT has died and reset the connection 💀

When the sending NETBLT gets a control message, it should act on that packet as fast as it can. It either sets up a new buffer (when an OK packet arrives for a previous buffer), resends data (when a RESEND packet arrives), or sends data (when a GO packet arrives) 🚀 If the sending NETBLT isn't in a position to send data, it sends a NULL-ACK packet containing the high-received-sequence-number described above. That lets the receiving NETBLT clear the control timers of any packets that are still outstanding. Then the sender waits until it can send more data 😌 In all of these cases, the overhead of responding to a control message should be small. The total time for a response to reach the receiving NETBLT should not be much more than the network round-trip transit time, plus a variance factor ⏱️

The timer system can be summed up like this: normally, the receiving NETBLT is running under one of two types of timers, a control timer or a data timer. There's one data timer per buffer transmission and one control timer per control packet. The data timer runs while its buffer is being transferred, and a control timer runs in between buffer transfers 📊

But there are still a couple of problems 😤 If the sending NETBLT isn't ready to send, it sends a single NULL-ACK packet to clear any outstanding control timers at the receiving end. After that, the receiver just waits. The sender could die, and then the receiver, with all its control timers cleared, would hang forever 💀 Also, this system only puts timers on the receiving NETBLT. The sending NETBLT has no timers, so if the receiving NETBLT dies, the sender just hangs waiting for control messages 🥲

The fix for both is a death timer plus a keepalive packet on both the sending and receiving NETBLTs 🛡️ As soon as the connection opens, each end sets a death timer, and it gets reset every time a packet is received. When one end's death timer expires, it can assume the other end has died and close the connection 💔

Waiting for long periods is totally valid, though. Clients can take ages to get buffer space and fill their buffers with data, and a NETBLT in that state is perfectly fine. So the protocol needs a way to stop the other end's death timer from expiring during that time 😩 The answer is a KEEPALIVE packet, sent repeatedly at fixed intervals while a NETBLT is waiting for buffer space. Since the death timer resets whenever a packet arrives, it will never expire as long as the other end keeps sending packets 🌈

Here's how often KEEPALIVE packets go out. At connection startup, each NETBLT picks a death-timeout value and sends it to the other end in either the OPEN or the RESPONSE packet. The other end takes that death-timeout value and uses it to work out how often to send KEEPALIVE packets. The KEEPALIVE frequency should be high enough that several KEEPALIVE packets can be lost before the other end's death timer expires 📬

Both ends need some way of estimating the values of the death timers, the control timers, and the data timers. Timer values obviously can't be specified in a protocol document, since they depend heavily on the machine and the network load. So they have to be computed on a per-connection basis. The protocol was designed to make that easy 🧮

The death timer is relatively easy to estimate. Since it's continually reset, it doesn't need to be based on the transfer size. Instead, it should be based at least in part on the type of application using NETBLT. User applications should have smaller death timeout values, so humans aren't stuck waiting forever for a death timeout to happen 🙈 Machine applications can have longer ones 🤖

The control timer must be estimated more carefully 🧐 Its initial value can be an arbitrary number, and that number can be used to send the first control packet. Subsequent control packets can have their timer values based on the network round-trip transit time (the time between sending the control packet and receiving the acknowledgment of the corresponding sequence number) plus a variance factor. The timer value should be continually updated based on a smoothed average of collected round-trip transit times 📈

The data timer doesn't depend on the network round-trip transit time. It depends on how long it takes to transfer a buffer of data. The time value can be computed from the burst rate and the number of bursts per buffer, plus a variance value <1>. During the RESENDing phase, the data timer value should be set according to the number of missing packets ⏳

The timers were designed to make reasonable estimation possible. In other protocols, figuring out round-trip delay has been a pain, since what the other end does on receiving a packet can vary a lot depending on the packet type. In NETBLT, the sender's action on receiving a control message is pretty much the same in every case, which keeps the round-trip delay relatively independent of the client 💯

Timer estimation is extremely important, especially in a high-performance protocol like NETBLT 🏎️ If the estimates are too low, the protocol makes lots of unneeded retransmissions, and performance tanks. A short control timer makes the sending NETBLT receive duplicate control messages. It can reject them, but that still takes time. A short data timer makes the receiving NETBLT send unnecessary RESEND packets, and that hurts way more, because the sender doesn't just toss a duplicate. It has to send a bunch of DATA packets again. Since data timers are set on each buffer transfer instead of on each DATA packet transfer, we can afford to use a small variance value without worrying about performance degradation 💜

---

5.3. Closing the Connection 🚪✨

There are three ways to close a connection: a connection close, a "quit", or an "abort" 🎬

A connection close happens after a successful data transfer. When the sending NETBLT has received an OK packet for the last buffer in the transfer, it sends a DONE packet <2>. On receipt of the DONE packet, the receiving NETBLT can close its half of the connection. The sending NETBLT dallies for a predetermined amount of time after sending the DONE packet, to allow for the possibility that the DONE packet was lost. If the DONE packet was lost, the receiving NETBLT keeps sending the final OK packet, which causes the sending end to resend the DONE packet. After the dally period expires, the sending NETBLT closes its half of the connection 🫠

During the transfer, one client may send a QUIT packet to the other if it thinks that the other client is malfunctioning. Since the QUIT happens at the client level, the QUIT can only be sent between buffer transmissions 🙅 The NETBLT receiving the QUIT packet can take no action other than to immediately notify its client and transmit a QUITACK packet. The QUIT sender must time out and retransmit until a QUITACK has been received or a predetermined number of resends have taken place. The sender of the QUITACK dallies in the manner described above 💌

An ABORT takes place when a NETBLT layer thinks that it or its opposite is malfunctioning. Since the ABORT starts in the NETBLT layer, it can be sent at any time 🚨 Since the ABORT implies that the NETBLT layer is malfunctioning, no transmit reliability is expected, and the sender can immediately close its connection 💥

6. MULTIPLE BUFFERING 🧚

okay so NETBLT is literally built to make multiple buffering a thing, because speed is the whole point bestie 💅 Multiple buffering is when the sender and receiver both allocate and send buffers in a way that lets them recover from errors in the previous buffers while the current buffer is still flying 🦋 ngl that's so efficient it's giving main character energy.

During connection setup, the two ends negotiate how many concurrent buffers are allowed for the transfer 🤝 The simplest option is one buffer at a time. That's basically lock-step, and it's so wasteful because the sender just sits there waiting for permission to send each new buffer 😩 If there's more than one buffer available, though, the next buffer can start right after the current one finishes 🌸

Here's a cute example. Say buffers A and B are allowed to go concurrently, with A coming first. The moment A finishes sending its data and is waiting for either an OK or a RESEND, B can start sending right away 🚀 Data keeps flowing at a steady rate, no cap. If A gets an OK, it's done ✅ If A gets a RESEND, then only the missing packets listed in that RESEND message get sent again 🔁

All the packets go out through a priority pipe. The priority is just the buffer number, and the transfer rate comes from the burst size and burst rate ✨ Buffer numbers always go up, so packets from an earlier buffer in the pipe will always get ahead of the later ones. Iconic 👑

There's one tweak to the timing algorithm you need to know about 💖 When the receiving NETBLT sets the data timer for a new buffer, the timer value should also account for how long it'll take to transfer all the missing packets from the previous buffers. Being thorough is the moment 🫶

Running several buffers at once is honestly not that much harder than running one. The trick is to think of each buffer as a finite state machine 🧠 Multiple buffers are just a group of these state machines, each sitting in one of several states. The whole transfer is just moving buffers through those states until everything has been sent 🦋

Now the state sequence for a send-receive buffer pair goes like this 💅 The sending and receiving buffers get created independently. The receiving NETBLT sends a GO message, which puts its buffer into a "receiving" state, and it starts its control timer ⏰ The sending NETBLT gets that GO message and moves its buffer into a "sending" state 📦 The sender keeps sending data until the whole buffer has gone out.

If the receiver's data timer goes off before it gets the last (LDATA) packet, or it gets the LDATA packet but some packets are missing, it sends a RESEND packet and moves the buffer into a "resending" state 😭 Once every DATA packet in the buffer and the LDATA packet have arrived, the receiver puts its buffer into a "received" state and sends an OK packet 🎉 The sender gets that OK packet and moves its buffer into a "sent" state. Periodt 💜

---

7. PROTOCOL LAYERING STRUCTURE 🎀

NETBLT sits directly on top of the Internet Protocol (IP), no middle layer needed 🌈 It's been given a temporary protocol number of 255. This number will change once the final protocol specification is settled, so don't get too attached to it 🙈

---

8. PACKET FORMATS 📦

okay the packet formats are where it gets really satisfying 🤩 NETBLT packets come in three categories, and they all share the same common packet header 💖

First, there are the packets that only travel from sender to receiver. These carry the control message sequence numbers the receiver uses for reliability. They're the NULL-ACK, DATA, and LDATA packets ✨

Second, there's one packet that only travels from receiver to sender, which is the CONTROL packet. Each CONTROL packet can hold any number of control messages (GO, OK, or RESEND), and each of those messages gets its own sequence number 🔢

Finally, there are the packets that either have their own special way of ensuring reliability or aren't sent reliably at all. These are QUIT, QUITACK, DONE, KEEPALIVE, and ABORT 🫧 Every one of them except DONE can be sent by both the sending and receiving NETBLTs. Lowkey the most flexible bunch 💅

Packet type numbers:

OPEN:           0
RESPONSE:       1
KEEPALIVE:      2
DONE:           3
QUIT:           4
QUITACK:        5
ABORT:          6
DATA:           7
LDATA:          8
NULL-ACK:       9
CONTROL:        10

Standard header:

local port:       2 bytes
foreign port:     2 bytes
checksum:         2 bytes
version number:   1 byte
packet type:      1 byte
packet length:    2 bytes

OPEN and RESPONSE packets:

connection unique ID:                   4 bytes
standard buffer size:                   4 bytes
transfer size:                          4 bytes
DATA packet data segment size:          2 bytes
burst size:                             2 bytes
burst rate:                             2 bytes
death timeout value in seconds:         2 bytes
transfer mode (1 = SEND, 0 = RECEIVE):  1 byte
maximum number of concurrent buffers:   1 byte
checksum entire DATA packet / checksum
DATA packet data only (1/0):         1 byte
client-specific data:                   arbitrary

DONE, QUITACK, KEEPALIVE:

standard header only

ABORT, QUIT:

reason:       arbitrary bytes

CONTROL packet format:

CONTROL packets are a standard NETBLT header of type CONTROL, followed by however many control messages you like, each in one of the formats below 🎁

Control message numbers:

GO:             0
OK:             1
RESEND:         2

OK message:

message type (OK):  1 byte
buffer number:      4 bytes
sequence number:    2 bytes
new burst size:     2 bytes
new burst interval: 2 bytes

GO message:

message type (GO):  1 byte
buffer number:      4 bytes
sequence number:    2 bytes

RESEND message:

message type (RESEND):     1 byte
buffer number:             4 bytes
sequence number:           2 bytes
number of missing packets: 2 bytes
packet numbers...:         n * 2 bytes

DATA, LDATA packet formats:

buffer number:                                4 bytes
highest consecutive sequence number received: 2 bytes
packet number within buffer:                  2 bytes
data:                                         arbitrary bytes

NULL-ACK packet format:

highest consecutive sequence number received: 2 bytes
acknowledged new burst size:                  2 bytes
acknowledged new burst interval:              2 bytes

NOTES:

<1> When the buffer size is big, the variances in the round trip delays of lots of packets can cancel each other out 🌊 That means the variance value doesn't need to be very big. This expectation can be checked in further testing 🧪

<2> The receiving end might not know the transfer size ahead of time, so it's possible it has allocated buffer space and sent GO messages for buffers past the real last buffer the sending end sent 👀 The sending end needs to be careful and ignore those extra GO messages. Not me being dramatic, this is a real gotcha 💀