RFC 632
Throughput degradations for single packet messages
Throughput Degradations for Single Packet Messages ✨
okay bestie, let's get into it 🎀 This is RFC 632, written by H. Opderbeck of UCLA-NMC (NIC # 30239) and dated 20 May 1974. It's an Internet-Draft-style doc from the Network Working Group. The title block doesn't list anything it obsoletes or updates, which makes sense because this was 1974 and there was nothing to update yet 🌸
okay so the big idea is that digitized speech over the ARPANET is a whole new vibe for packet switching 🎤 Its throughput and delay needs are totally different from interactive use or file transfers 💅 We need high throughput for small messages, because long messages make the source wait forever to fill up big buffers 😭 So the authors are studying throughput limits for single-packet messages 🦋
They admit that up until now, nobody really tried to optimize throughput for low-delay traffic. But here's the plot twist: the observed throughput for single-packet messages was often only about one fourth of what they expected 💀 This doc explains why that happens and what could be done to fix it 🔥
On April 1, 1974, they used the IMP message generator to send single-packet messages at the highest possible rate ("RFNM-driven") from the MOFFET-IMP to the SRI-IMP 🛰️ There are two three-hop paths from MOFFET to SRI, and one of them has two 230.4 kbs circuits in it ⭐ There was barely any interfering traffic, so they expected an average round-trip delay of not more than 100 msec 🍓 If there were about 3 messages in transmission between MOFFET and SRI, with a message length of about 1000 bits, the throughput should have been more than 30 kbs 🌈
It was less than 8 kbs. ugh 😩 They repeated the experiment and got the same result 🙈
A closer look at the data showed that an average of 3.5 messages were in transmission between MOFFET and SRI at the same time 👀 So interfering traffic between those two sites couldn't have been the cause 🙅♀️ Also, channel utilization for every channel involved was less than 40 percent 📉 The observed mean round-trip times were about 500 msec, and those large times weren't due to physical limitations 🧚 So they studied the flow control mechanism for single-packet messages, and found an explanation for this undesirable behavior 🔍
When a single-packet message arrives at the destination IMP out of order (meaning the logically preceding message hasn't arrived there yet), the destination IMP doesn't accept it 🚫 It's treated as a request to allocate one reassembly buffer instead 💭 The matching ALLOCATE only gets sent back to the source IMP after the RFNM for the previous message has been processed 🌊 So here's the sequence of events they describe:
MSG(i)sent from SOURCE-IMP (message i is sent from the source IMP to the destination IMP).MSG(i+1)sent from SOURCE-IMP.MSG(i+1)arrives at DEST-IMP (due to an alternate path or a line error, message (i+1) arrives at the destination IMP out of order; it is treated as a request for one reassembly buffer allocation and then discarded).MSG(i)arrives at DEST-IMP (message i arrives at the destination IMP; it is put on the proper HOST output queue).RFNM(i)sent from DEST-IMP (after message i has been accepted by the destination HOST the RFNM is sent to the source IMP).ALL(i+1)sent from DEST-IMP (only after the RFNM for message i has been processed can the ALLOCATE for message i + 1 be sent).RFNM(i)arrives at SOURCE-IMP.ALL(i+1)arrives at SOURCE-IMP.MSG(i+1)arrives at DEST-IMP (now message i+1 is put on the proper HOST output queue.)RFNM(i+1)sent from DEST-IMP.RFNM(i+1)arrives at SOURCE-IMP.
The round-trip time for message i+1 is the time between event 2 and event 11 🕰️ That means it's more than twice as large as it would've been if the message had arrived in order, everything else being the same 😤 So a line error can delay not just the message in error, but also the next single-packet message, if that one follows within 125 msec (the error retransmission timeout interval) 💔 And a faster alternate path to the destination IMP can actually slow things down, because it makes messages arrive out of order 🐢
Now it gets worse, because they also look at RFNM-driven single-packet traffic 😱 Table 1 shows one possible sequence. Again, they assume message i+1 reaches the destination IMP before message i 💅
SOURCE IMP DESTINATION IMP
MSG(i) sent
MSG(i+1) sent
MSG(i+2) sent MSG(i+1) arr
MSG(i+3) sent MSG(i) arr
RFNM(i) sent
ALL(i+1) sent
MSG(i+2) arr
MSG(i+3) arr
RFNM(i) arr
MSG(i+4) sent
ALL(i+1) arr MSG(i+4) arr
MSG(i+1) sent MSG(i+1) arr
RFNM(i+1) sent
RFNM(i+1) arr ALL(i+2) sent
MSG(i+5) sent
ALL(i+2) arr MSG(i+5) arr
MSG(i+2) sent MSG(i+2) arr
RFNM(i+2) sent
RFNM(i+2) arr ALL(i+3) sent
MSG(i+6) sent
ALL(i+3) arr MSG(i+6) arr
MSG(i+3) sent MSG(i+3) arr
RFNM(i+3) sent
RFNM(i+3) arr ALL(i+4) ent
MSG(i+7) sent
ALL(i+4) arr MSG(i+7) arr
MSG(i+4) sent MSG(i+4) arr
RFNM(i+4) sent
RFNM(i+4) arr ALL(i+5)
MSG(i+8) sent
ALL(i+5) arr
MSG(i+5) sent
Table 1.Retransmission Pattern for the Current Flow Control Scheme 🎀
(Since the traffic is RFNM-driven, the arrival of RFNM i, i+1, ... is followed by the sending of message i+4, i+5, ...)
The most iconic part is this: when message i+1 arrives before message i at the destination IMP, it doesn't just make messages i+1 retransmitted. All future messages get retransmitted too 🤯 And nothing in the future messages has to arrive out of order for that to happen. The round-trip times for message i+4 and every future message are more than four times what they'd be without these undesirable retransmissions 💀 Once this pattern sets in, the system can almost never recover, except by interrupting the input stream at the source IMP 🚨 One out-of-order arrival of any later user or control message won't change it either 😭 The normal flow only comes back if, for example, messages i+4, i+5, and i+6 are all delayed for several hundred milliseconds, so that i+1, i+2, and i+3 can be retransmitted in the meantime 🥲 The odds of that are super tiny, so the authors basically call the system trapped in this condition 🪤 The "normal" flow is only the transient behavior, because there's always a finite chance that two messages arrive out of order due to transmission errors 🎲
So, as mentioned before, the system only recovers from this throughput (and delay) degradation if the input stream of single-packet messages gets interrupted ⏸️ For speech, that might not happen for a long time 😩 That means speech systems would often have to run with only one fourth of the expected single-packet bandwidth 📞 The authors say that's clearly unacceptable, so they went looking for a change to the current flow control scheme that fixes it 🛠️ They describe two methods that could avoid the undesirable retransmission of messages 🌟
Remember, a single-packet message is rejected at the destination IMP and retransmitted later if the RFNM for the preceding message hasn't been sent to the source IMP yet 🔁 This is mainly done to prevent reassembly lockup conditions [1] 🔒 So you can't just accept every single-packet message without extra measures to prevent deadlocks 🙅♀️ Otherwise, a big pile of single-packet messages from several source IMPs could arrive out of order at their shared destination IMP 📦 Then the destination IMP might not have enough reassembly buffers for the messages that are in order, and the whole system deadlocks 💔 Any fix for the throughput degradation problem has to guarantee that every message arriving in order can be accepted by the destination IMP 🫶
One way to do that is to reject out-of-order single-packet messages only if the buffer requirement(s) of the preceding message(s) is not known 🧠 In the earlier examples, the destination IMP kept rejecting messages even though it already knew the buffer requirements for the messages that had to be delivered first 🙃 Once those buffer requirements are known, the needed number of buffers can be set aside, and future single-packet messages can be accepted without any danger of deadlock ✅ So the undesirable retransmission pattern can't take hold 🦋 Table 2 shows the sequence of events for this policy when message i+1 arrives before message i at the destination IMP 👇
SOURCE IMP DESTINATION IMP
MSG (i) sent
MSG(i+1) sent
MSG(i+2) sent MSG(i+1) arr. (rejected)
MSG(i+3) sent MSG(i) arr. (HOST output)
RFNM(i) sent
ALL (i+1) sent
MSG(i+2) arr (stored)
MSG(i+3) arr (stored)
RFNM(i) arr
MSG(i+4) sent
ALL(i+1) arr MSG(i+4) arr (stored)
MSG(i+1) sent MSG(i+1) arr (HOST output)
RFNM(i+1) sent
RFNM(i+1) arr RFNM(i+2) sent
MSG(i+5) sent RFNM(i+3) sent
RFNM(i+4) sent
Table 2.Sequence of Events for Modified Flow Control Scheme 🌸
Notice that in this modified scheme, only one message gets retransmitted: message i+1 🎯 But the IMPs have plenty of reassembly buffer space, so the authors really want to avoid even that one retransmission 🙌 This matters a lot for speech, which depends on a steady flow of data and gets disrupted by any sudden large delay 🎙️ So they suggest a second method that, in most cases, won't need any retransmissions at all ✨
Here's the idea: suppose all single-packet messages are accepted (or stored) right from the start 💫 Right now, out-of-order single-packet messages are rejected because of the possibility of deadlock. But let's take a closer look
(to be continued in Part 2 💖)
This is the deadlock question I was worried about, bestie. If every single-packet message gets accepted (or stored), the destination IMP can end up with no reassembly buffer free for the messages it needs to hand to their HOST(s) next. 🫠
But it's NOT really a lockup, and here's why 💅 The source IMPs keep a copy of every single-packet message until a RFNM comes back for it. So a single-packet message that arrived out of order but still got accepted by the destination IMP can be deleted later without losing anything. The destination IMP just has to send an ALLOCATE for each deleted single-packet message to the matching source IMP once reassembly buffer space frees up. Deferred rejection, iconic! ✨
Now a retransmission only happens if the destination IMP is truly running out of reassembly buffers. At that point we've hit the physical limits of the system, and no protocol change is going to give us big throughput gains. 🙈
Real talk: once you're out of buffers, a smarter protocol can't create more memory. Hardware is the hard ceiling here. 🧱
It's our plan to take this up with the IMP development group at BBN later on. 🦋 They agree that both of the solutions we suggested would make things better, but they've got some alternative solutions of their own in mind.
Huge thanks to Bill Naylor and Joe Katz 🫶 They helped run the experiments that led to discovering the throughput degradation in the first place, and I'm so grateful. 💜
References 📚
[1] Kleinrock, L. and H. Opderbeck. "On a Possible Lockup condition in the IMP Subnet Due to Message Sequencing", RFC #626.
[ This RFC was put into machine readable form for entry ]
[ into the online RFC archives by Alex McKenzie with ]
[ support from GTE, formerly BBN Corp. 11/99 ]📦 where this page comes from
- ietf: rfc632 1974-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms