RFC 626
On a possible lockup condition in IMP subnet due to message sequencing
On a Possible Lockup Condition in the IMP Subnet Due to Message Sequencing 🎀
📜 okay bestie, this is RFC 626, from the Network Working Group, written by L. Kleinrock (UCLA-NMC) and H. Opderbeck (UCLA-NMC). It's NIC #22161 and it's dated Mar 1974. The title block doesn't list a category, and it doesn't say this one obsoletes or updates anything, so I'm not making that up 🙅♀️
Okay so, lockups and deadlocks are highkey some of the worst malfunctions a computer system or network can have 💀 Protocols have to be designed super carefully to avoid them. The tea is that they only show up in weird situations the protocol designers didn't predict, or thought were too unlikely to bother with 🙈 (And ngl, those designers often aren't the ones who can actually work out how likely something is, with numbers.)
The most famous lockup in the ARPANET is the reassembly lockup [1] 😱 The store-and-forward lockup, which is also described in Reference 1, got avoided in the new IMPSYS by carefully following Kahn's heuristics [1]. ✨
The last lockup we know of in the subnet happened on December 21, 1973, and everyone calls it the Christmas lockup 🎄 This dormant lockup condition came to light when snapshot measurement messages were collected from all sites at the same time. What happened is that snapshot messages arrived at the UCLA IMP, which had already allocated reassembly storage for them, but no reassembly blocks were free. (A reassembly block is a piece of storage used in the actual process of reassembling packets back into messages) [2] 🧩
Deadlocks aren't just a subnet thing either. They've shown up in higher level protocols too 👀 The original design of the ICP, for example, had a flaw that only got discovered after a few months of use [3,4]. And more recently, BBN reported a deadlock problem involving the exchange of HOST status information by the RSEXEC server (RSSER) programs [5] 😭
As long as it isn't possible to design practical communication protocols that guarantee deadlock-free operation, it's vital to keep checking the protocols that are currently in use for failures, even if they seem "very unlikely" to happen. 🚨
So in this RFC we talk about a possible deadlock condition in the IMP subnet 💖 As far as we know it hasn't happened yet, and nobody has identified it yet either. We've never actually seen this problem happen, but it may happen in the future if no precautions are taken. This possible lockup condition comes from the sequencing of messages in the subset.
To avoid reassembly lockup, the flow control mechanism in the subnet got modified in some pretty significant ways 💅 Right now there's a limit of four messages that can be in transmission at the same time between any pair of source and destination IMPs. When the link-handling was removed from the old IMPSYS, it became necessary to introduce a message sequencing mechanism 🦋
Messages leave the destination IMP in the same order they entered the source IMP. (Note that the sequencing happens on an IMP-to-IMP basis, not a HOST-to-Host basis. This may introduce undesirable "sequencing delay" if two HOSTs attached to the same destination IMP receive messages from the same source IMP 🙃)
Sequencing has, in general, the potential to introduce deadlock conditions, and it's giving main character drama 🎭 The reason is that any message, say message (n+1), which is out of order (and therefore can't be delivered to its destination HOST) can use up resources that message (n) needs, and message (n) is the one that has to be delivered next. So message (n) can't reach its destination IMP, which in turn stops the other out-of-order messages (n+1), etc) from getting delivered to their destination HOST(s). That's why you have to be very careful not to allocate too many resources (e.g., buffers) to messages that are out of order 🫶
To avoid lockup conditions, the current IMPSYS takes the following two precautions:
- Requests for buffer allocation are always serviced in order of message number. In other words, no 'ALLOCATE' is returned for message (n+1) if message (n) (or a request for buffer allocation for message (n)) hasn't been received and serviced yet.
- Single packet messages (regular and priority) that arrive at the destination IMP out of order are not accepted unless they were retransmitted in response to a previous buffer allocation. These messages are treated more like a request for the allocation of one buffer (according to 1 above) and then discarded.
With these two precautions, a deadlock condition appears to be impossible to occur ✨ However, there's a second buffer allocation mechanism that isn't tied to the message sequencing, and it's the sneaky one 👀 That's the 'ALLOCATE' that gets piggy-backed on the RFNM for a multiple-packet message. The piggy-backed ALLOCATE represents a buffer allocation for the next multiple-packet message, and NOT for the next message in sequence. So if the next message in sequence is a single-packet message, the piggy-backed ALLOCATE effectively allocates buffers to a message that's out of order 😬
Let's see how this situation can lead to a deadlock condition. Assume there's a maximum of 24 reassembly buffers in each IMP.
IMPs A, B, and C continually transmit 8-packet messages to the same destination IMP D, so all 24 reassembly buffers in IMP D get used up by this multiple-packet message transmission 🌸 Now, if IMP A sends a single-packet message in the stream of 8-packet messages, it generally won't be accepted by destination IMP D, because there's no reassembly buffer space available. (There may be a free reassembly buffer if the single-packet message happens to arrive while one of the three 8-packet messages is being transmitted to its HOST.) So the single-packet message gets treated as a request for buffer allocation. That request won't be serviced before the RFNM of the previous multiple-packet message is sent. By then, though, all the free reassembly buffers have already been allocated to the next multiple-packet message through the piggy-backed ALLOCATE mechanism. The only chance the single-packet message has to get its allocation request satisfied is to grab a reassembly buffer from one of the other two 8-packet messages, and that attempt might not work, because it depends on the timing of events in the IMP ⏱️
A deadlock condition can happen if IMP B and IMP C also send a single-packet message in their streams of 8-packet messages, which can't be serviced for the same reason 💀 In that case, the three 8-packet messages that arrive later at IMP D can't be delivered to their destination HOST(s) because they're out of order. And the three single-packet messages that should be delivered next will never reach the destination IMP, since there's no reassembly space available. Yikes 😭
Here's a possible sequence of events that leads to a deadlock condition 🎀
Examples for Notation:
event: A8 arrives -> all 8 packets of the 8-packet message
from IMP A have arrived at IMP D
event: C1 arrives -> a single packet message from IMP C has
arrived at IMP D (and is treated as a
request for buffer allocation)
event: B8 complete -> the last packet of the 8-packet
message from IMP B has been received
by its destination HOST
event: A8 RFNM/ALL -> a RFNM with the piggy-backed ALLOCATE
is sent to IMP AHere's the step-by-step table, and it's a whole saga 📊
# of Allocated # of Reassembly # of Free
Reassembly Buffers Reassembly
Buffers in Use Buffers
Initially 24 0 0
1. A8 arrives 16 8 0
2. B8 arrives 8 16 0
3. C8 arrives 0 24 0
4. Al arrives 0 24 0
5. B1 arrives 0 24 0
6. C1 arrives 0 24 0
7. A8 complete 0 16 8
8. B8 complete 0 8 16
9. C8 complete 0 0 24
10. A8 RFNM/ALL 8 0 16
11. B8 RFNM/ALL 16 0 8
12. C8 RFNM/ALL 24 0 0
13. A8 arrives 16 8 0
14. B8 arrives 8 16 0
15. C8 arrives 0 24 0
16. - deadlock -Note that an ALLOCATE for one of the single-packet messages A1, B1 and C1 can only be returned to source IMP A, B, and C, respectively, after the RFNM (with its piggy-backed ALLOCATE) for the previous 8-packet message has been sent 🫠 If these RFNMs are sent in sequence, i.e., without an ALLOCATE for one of the single-packet messages in between, the temporarily freed reassembly storage (events (7) through (9)) gets implicitly allocated to the next multiple-packet messages (events (10) through (12)), and not to any of the single-packet messages. The next 8-packet messages are, however, out of order and can't be delivered to their destination HOST(s).
Right now it looks like a lockup like this can only happen if the number of reassembly buffers is a multiple of eight 🍓 And yes, the probability of a lockup in that case is much higher. However, deadlocks can also happen if the number of reassembly buffers is not a multiple of eight, so don't get too comfy 😤
Let's assume there are 26 instead of 24 reassembly buffers 🧚 Also assume that, due to alternate paths, line failure, or retransmission, the second packet of a 2-packet message arrives at IMP D before a single-packet message from the same source IMP A. The single-packet message has a smaller sequence number and therefore has to be delivered to its destination HOST before the 2-packet message. When the second packet of the 2-packet message arrives at IMP D, the IMP realizes that only 2 of the allocated 8 buffers will be needed. So 6 buffers get returned to the pool of free reassembly buffers. If there were 26-3x8=2 buffers in the pool before, the pool size goes up by 6 to 8 buffers. But these 8 buffers are just enough to send out another piggy-backed ALLOCATE 🙃 So the single-packet message finds the pool of free reassembly buffers empty, even though the total number of reassembly buffers isn't a multiple of eight. The 2-packet message can't be delivered to its destination HOST because it's out of order, and the deadlock condition we described before may happen again 😭
We agree the sequence of events above is unlikely to happen (otherwise someone would have already observed it) 🤞 This is especially true since the current maximum number of reassembly buffers (58) is much bigger than 24. But judging from past experience with computer systems and networks, even very unlikely events have a tendency to happen in the long run 🌈 Also, the probability of this deadlock condition goes up as traffic in the net increases. So it's definitely worth modifying the IMPSYS so this deadlock can't happen. The good news is that a minor modification already achieves the desired effect ✨ Recall that the deadlock described can only happen because single- and multiple-packet messages use the same pool of reassembly buffers. If we set aside a single reassembly buffer (or one for each destination HOST) that can be used only by single-packet messages, this lockup condition due to message sequencing can't happen 💖
Another solution would be to move the message sequencing from the IMP to the HOST level 🦋 That doesn't mean similar lockup problems can't happen at the HOST level, though. Also, it's currently much easier to fix this problem with a slight modification of the IMPSYS than by having to coordinate all the various HOSTs 👑 Another alternative is to discontinue the use of multiple-packet messages. However, this also involves
much more drastic changes which require careful consideration. 🫶
-5-
References 📚✨
- Kahn, R. E. and W. R. Crowther, "Flow Control in a Resource-Sharing Computer Network," IEEE Transactions on Communication, Volume COM-20,3 June 1972.
- BBN Report No. 2717, "Interface Message Processors for the ARPA Computer Network," Quarterly Technical Report No. 4, January 1974.
- Naylor, W., J. Wong, C. Kline, and J. Postel, "Regarding Proffered Official ICP," RFC 143, NIC 6728.
- Postel, J. B. "A Graph Model Analysis of Computer Communications Protocols," PhD dissertation, University of California, Los Angeles.
- BBN Report No. 2670. "Natural Communication with Computers IV," Quarterly Progress Report No. 12, October 1973.
-6-
📦 where this page comes from
- ietf: rfc626 1974-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms