RFC 636

TIP/TENEX Reliability Improvements 💖✨

Okay bestie, the header tea 🫖: this is RFC 636, also filed as NIC 30490, dated 10 June 1974. The authors are J. Burchfiel (BBN-TENEX), B. Cosell (BBN-NET), R. Tomlinson (BBN-TENEX), and D. Walden (BBN-NET). The title block doesn't list a category, and it doesn't say what it obsoletes or updates, so there's nothing to report there 🙈

So for months we've been under a LOT of pressure to make TIP/TENEX connections more reliable 😭 If connections between TENEXs and TIPs get better, the whole network's reliability looks way better too, because there are so many TENEXs and TIPs and they're super visible. But ngl, everything in here works for ANY Hosts, not just TIP/TENEX 🌈

This RFC is basically a sketch of our plan. Some big chunks are already built (TIP version 322 and TENEX version 1.32) and are in final testing before they roll out across the network 🔥 The whole implementation should be finished next quarter.

The plan has three main goals: getting and keeping TIP/TENEX connections, recovering gracefully when connections get lost, and giving the user clear messages whenever their connection state changes 💅

When the Host is down 🌸

When a TIP user tries to open a connection to any Host, that Host might be down. In that case it's super helpful to tell the user how bad it is. So we changed the IMP program to accept and use info from a Host about when it'll be back up and why it's down. TENEX is being changed to send that info before it goes down, or manually after it's already gone down. When a TIP user tries to connect to a down TENEX, the IMP local to that TENEX sends back why and for how long TENEX will be down. The TIP is being changed to show that info to the user, like this:

Host unavailable because of hardware maintenance -- expected available Tuesday at 16:30 GMT

The logger situation 🧚

The TIP's logger isn't reentrant right now, so no single user can hog it for too long. To prevent that, the TIP enforces a timeout of arbitrary length (about 60 seconds) on logger use. The problem is that a heavily loaded Host can't be guaranteed to answer a TIP login request within 60 seconds, and right now TIP users sometimes just can't connect to a heavily loaded TENEX 😩 The fix is to make the TIP logger reentrant and get rid of the timeout on logger use.

The incremental allocate soft spot 🍓

The Host/Host protocol's incremental allocate mechanism is a notorious soft spot that hurts connection reliability. Low-frequency software bugs, intermittent hardware bugs, and so on can knock the incremental allocates for a connection out of sync. When that happens, it usually looks to the user like the connection just "hung up." We made a slight addition to the Host/Host protocol so connection allocates can be resynchronized, and it's been designed and implemented for both the TIP and TENEX 🫶

Bughalts 💀

TENEX has a bunch of internal consistency checks (called "bughalts") that occasionally make TENEX halt. After system personnel diagnose the problem, TENEX can often keep going without any loss from the point of view of local users. We're adding a mechanism so TENEX can also keep going from the point of view of TIP users of TENEX ✨

Here's how that works. TENEX won't drop its ready line during a bughalt (that's the case where it can usually proceed successfully), and it won't clear its NCP tables or abort all connections. Instead, after a bughalt, TENEX will:

1. discard the message it's currently receiving, since the IMP already returned an Incomplete Transmission to the source for that message;
2. reinitialize the interface to the IMP; and
3. resynchronize, on all connections possible, Host/Host protocol allocate inconsistencies caused by lost messages, RFNMs, etc. This uses the same mechanism described above.

Real talk: this procedure is NOT guaranteed to save all data, and a tiny bit may be lost. But keeping the connection alive through the bughalt matters way more than that 😭

The TIP user has to be kept fully in the loop while TENEX halts and then continues 👀 So the TIP was changed to report "Host not responding -- connection suspended" when it senses that TENEX has halted (it figures this out by properly interpreting messages returned by the destination IMP). When TENEX resumes service after a bughalt, the procedure above tells the TIP that service is restored, and the TIP reports "Service resumed" to all users of that Host.

The other possibility is that the service interruption can't be recovered from, and TENEX has to do a total system reload and restart. In that case TENEX clears its NCP connection tables and sends a Host/Host protocol reset command to all other Hosts. When the TIP gets that reset command, it reports "Host reset -- connection closed" to all users of that Host who have suspended connections. After that, the TIP user can re-login to the TENEX or to some other Host 🔄

The half-closed connection drama 🦋

Of course, the user might not have the patience to wait for service to come back after a TENEX bughalt. So they might unilaterally connect to some other Host and ignore the suspended connection. If TENEX can keep going, its NCP still thinks the connection to the TIP is good and usable. So now you've got a connection the TIP thinks is closed and TENEX thinks is open, which is what we call a "half-closed connection." An automatic procedure for cleanly finishing the close of that kind of connection has been specified and implemented for the TIP and TENEX 🙌

Security, but make it serious 🔐

Since TENEX keeps connections alive across service interruptions, the TIP user is required to take the security procedure of telling the TIP to "forget" the suspended connection before walking away from their terminal. For example, the command @H 0 guarantees that the connection won't be reestablished when service resumes. If you skip this, your job is left at the mercy of whoever gets their hands on that terminal next 😱 Please don't skip it!!

Appendix teaser 📚

An appendix comes after this that describes the Host/Host protocol changes. These changes are backward compatible, with one exception: Hosts that haven't implemented them will sometimes receive unrecognizable Host/Host protocol commands, which they presumably discard without any harm. The protocol changes are ad hoc in nature. But given how backward compatible and potentially useful they are, ARPA okayed adding them to the TIP and TENEX NCPs, without (we believe) implying that other Hosts have to implement them, although we would

👉 part 2 coming, stay tuned 🦋

encourage their widespread implementation) 🫶 bestie, they were so generous about it!! Okay, now let's get into the good stuff. 🎀

Appendix - Ad Hoc Change to Host-Host Protocol 🎀

A.1. Introduction 💭

okay so the current Host-Host protocol (NIC #8246) has zero provisions for resynchronizing the status info kept at the two ends of each connection 😭 So if either host has a service interruption, or a control message gets lost or corrupted in an interface or in the subnet, the status info at the two ends is going to be inconsistent. 💀

And since the current protocol has no way to fix this, the NCPs at both ends just stay "confused" forever 🥺 A super frustrating symptom of this is the "lost allocate" phenomenon. That's where the receiving NCP thinks it has bit and message allocations outstanding, but the sending NCP thinks it has no allocation at all. Result? Information flow over that connection can never restart. Iconic behavior from a protocol, not it. 🙈

Using the Host-Host RST (reset) command isn't the move here, bestie, because it destroys ALL connections between the two hosts 😬 What we need is a way to resync only the affected connection without disturbing any others. ✨

A second messy symptom is the "half-closed" connection 🦋 After a service interruption or network partitioning, one NCP may think a connection is still open while the other thinks it's closed (doesn't exist). When that inconsistency is discovered, the "open" end of the connection should be closed. 💅

A.2. The RAR, RAS and RAP commands 🌸

To get resynchronization of allocation, we're adding three new commands to the Host-Host protocol. Let's look at them 👀

              8 bits   8 bits
            -------------------
            !        !        !
         16 !  RAR   !  link  !
            !        !        !
            -------------------
         Reset Allocation by Receiver

              8 bits   8 bits
            -------------------
            !        !        !
         17 !  RAS   !  link  !
            !        !        !
            -------------------
         Reset Allocation by Sender

              8 bits   8 bits
            -------------------
            !        !        !
         20 !  RAP   !  link  !
            !        !        !
            -------------------
         Reset Allocation Please

The RAS command is sent from the Host sending on "link" to the Host receiving on "link" 📨 It may be sent whenever the sending Host wants to resync the status info associated with the connection (and doesn't have a message in transit through the network). Some situations where the sending Host might do this are:

1. After a timeout when there's traffic to move but no allocation (this assumes an allocation has been lost) 😩
2. When an inconsistent event happens associated with that connection (for example, an outstanding allocation in excess of 2^32 bits or 2^16 messages) 🚨
3. After the sending host has suffered an interruption of network service 🌧️
4. In response to a RAP (see below) 🔁

The RAR command is sent from the Host receiving on "link" to the Host sending on "link," in response to a RAS 💖 It marks the completion of the connection resync. When the RAR comes back, the connection is in the known state of having no messages in transit in either direction, and the allocations are zero. The receiving Host can then start afresh with a new allocation, and normal message transmission can proceed ✨ Since the RAR may be sent ONLY in response to a RAS, there are no races in the resynchronization. All of the initiative lies with the sending Host 👑

Now, if the receiving Host spots an anomalous situation, there's no way to tell the sending Host that a resync is desirable 😤 That's where the RAP command comes in. It's basically a "suggestion" from the receiving Host that the sending Host resynchronize, and the sending Host is free to honor it or not, as it sees fit 🫠 Since there's no obligatory response to a RAP, the receiving Host can send them as often as it likes, and no harm can occur. For example, if a message in excess of the allocate arrives, the receiving Host might send RAPs every few seconds until the sending Host replies. No fears of races if one or more RAPs pass a RAS in the network 🧚

A.3. Resynchronization Procedure 🎬

The resynchronization sequence below may be initiated only by the sender, either for internally generated reasons or upon receipt of a RAP 🎬

• a) Sender - decision to resynch
  • 1) Set state to "Wait-for-RAR" (Defer transmission of message.)
  • 2) Wait until no RFNM outstanding
  • 3) Send RAS
  • 4) Zero allocation
  • 5) Ignore allocates until RAR received
  • 6) Set state to "Open" (Resume normal message transmission subject to flow control.)
• b) Receiver - receipt of RAS
  • 1) Send RAR
  • 2) Zero allocation
  • 3) Send a new allocation

When the sender is in the "Wait-for-RAR" state, it is not permitted to send new regular messages 🚫 (Steps 4 and 5 will ensure this in the normal course of events, so no sneaky sends!) Once the RAR comes back, the pipeline contains no messages and no allocates. The outstanding allocation variables at both ends are forced into agreement by setting them both to zero 💜 The receiver then reconsiders bit and message allocation, and sends an ALL command for any allocation it cares to do 🎀

A.4. The Problem of Half-closed Connections 🌙

The procedures above give us a way to resync a connection after a brief lapse by a communications component, which results in lost messages or allocates for an open connection 🩹

A longer and more severe interruption of communication can happen from a partitioning of the subnet, or from a service interruption on one of the communicating hosts 😱 It's undesirable to tie up resources indefinitely under these circumstances, so the user gets the option of freeing up those resources (including themselves) by unilaterally dissolving the connection. Here "unilaterally" means sending the CLS command and closing the connection without receiving the CLS acknowledgement. Note that this is legal ONLY if the subnet indicates that the destination is dead 💀

When service is restored after an interruption like this, the status info at the two ends of the connection is out of sync. One end believes the connection is open and may proceed to use it. The disconnecting end believes the connection is closed (doesn't exist), and may proceed to re-initialize communication by opening a new connection (RTS or STR command) using the same socket pair or same link 🙃

So the resync needed here is to properly close the open end of the connection when the inconsistency is detected. We'll accomplish this by specifying consistency checks and adding a new pair of commands 🛠️

A.5. The NXS and NXR Commands 🧩

The "missing CLS" situation described above can show up in two ways. The first way involves action taken by the NCP at the "open" end of the connection. It may keep sending regular messages on the link of the half-closed connection, or control messages referencing its link. The closed end should respond with an NXS if the message referred to a non-existent transmit link (for example, an ALL), or an NXR if the message referred to a non-existent receive link (for example, a data message). On receipt of such an NXS or NXR message, the NCP at the "open" end should close the connection by modifying its tables (without sending any CLS command), which brings both ends back into agreement 🤝

              8 bits   8 bits
            -------------------
            !        !        !
         21 !  NXR   !  link  !
            !        !        !
            -------------------
         Non-existent Receive Link

              8 bits   8 bits
            -------------------
            !        !        !
         22 !  NXS   !  link  !
            !        !        !
            -------------------
         Non-existent Send Link

A.6. Consistency Checks 🔍

A second way this inconsistency can show up involves actions initiated by the NCP at the "closed" end 👀 Thinking the connection is closed, it may send an STR or RTS to reopen it. The NCP at the "open" end should detect the inconsistency when it receives such an RTS or STR command, because the command specifies the same socket pair as an existing open connection or, in the case of an RTS, the same link. In this case, the NCP at the "open" end should close the connection (without sending any CLS command) to bring the two ends into agreement before responding to the RTS/STR 🫶

A.7. Conclusion 🏁

The scheme presented in Section A.2 to resynchronize allocation has one very important property: the data stream is preserved through the exchange 🌈 Since no data is lost, it's safe to initiate resynchronization from either end at any time. When in doubt, resynchronize!! 💖

The consistency checks for RTS and STR, plus the NXR and NXS commands, provide the synchronization needed to finish closing "half-closed" connections 🦋

The protocol changes above