RFC 816

RFC 816: FAULT ISOLATION AND RECOVERY 🦋

RFC number: 816 💖 Title: Fault Isolation and Recovery 🌸 Author: David D. Clark, MIT Laboratory for Computer Science, Computer Systems and Communications Group ✨ Date: July 1982 📅 Obsoletes / Updates: the header doesn't list anything this one obsoletes or updates, and it doesn't state a category either 🙈

---

1. Introduction ✨

okay so bestie, sometimes a network or a gateway just goes down 💀, and now the path a packet takes from source to destination has to change. Fault isolation is the teamwork between hosts and gateways where everyone figures out that something is wrong. Fault recovery is picking a backup route that reconnects the source to the destination 🔥. Gateways do most of this work, but there are a few things hosts must do too if they want decent service 💅. This doc is about the part of fault isolation and recovery that belongs to the host 🫶.

2. What Gateways Do 👑

gateways basically run an algorithm to find the best route between every pair of networks 🌈. They trade packets that hold each gateway's newest opinion on whether its neighboring networks and gateways are working. If the algorithm is running properly, you can expect a little chaos right after a network or gateway dies, but once they've negotiated for a while, they end up with a consistent, correct picture of how the internet is connected ✨.

Here's the tea 🍓: that negotiation can take several minutes right now, and lots of TCP implementations give up before then 😭. The design goal for the final algorithm is for gateways to rebuild the topology fast enough that a TCP connection can survive a route failure 🦋.

3. Host Algorithm for Fault Recovery 💜

the gateways are always trying to keep a consistent, correct view of the internetwork topology, so the host's recovery plan is honestly super simple ✨. Whenever the host feels like something's wrong, it asks the gateway for advice. If advice shows up, the host believes it completely 💖. The advice is only wrong during the short negotiation period right after an outage. Otherwise it's reliably correct 💅.

The host doesn't even have to ask, bestie, because the gateway will send advice when it's relevant 🧚. When a host sends a datagram to some distant net, it should be ready to get back either of two advisory messages the gateway might send 🎀.

• The ICMP "redirect" message says the gateway the host sent the datagram to is no longer the best gateway for that net. The gateway still forwarded the datagram, but the host should update its routing table to use a different immediate address for that net 🌸.
• The ICMP "destination unreachable" message says that because of an outage, the addressed net or host can't be reached in any way right now 💀. When that arrives, the host can either drop the connection right away without retransmitting, or resend slowly to see whether the fault gets fixed in a reasonable amount of time ⭐.

If a host could count on these two ICMP messages always showing up when something's wrong, it wouldn't need to do anything else to keep its tables in great shape 👑. Sadly, they can fail to arrive properly in two situations 😭.

First, during the transient after a failure, error messages can show up that don't match what's really happening. So hosts must treat a lone error message with some skepticism 🙈. (This transient period gets more detail below 🔍.)

Second, say the host has been sending datagrams to one particular gateway, and that gateway crashes 💀. The other gateways will rebuild the topology, but that one is still down, so it can't send any advice back. As long as the host keeps aiming datagrams at that dead gateway, they just vanish into the void, and nothing ever comes back 👻. Hosts must detect this failure 🫶.

If a gateway many hops away dies, that's not the host's problem 💅. The nearby gateways are responsible for spotting it, and they handle it in a way the host never sees 🧚. The only real problem is when the very first gateway, the one the host is sending datagrams to directly, fails 💔. So the host has exactly one job for fault isolation: it must use some strategy to detect that the gateway it's sending datagrams to is dead 🔥.

For now, let's just assume the host has some way to detect dead gateways. We'll come back to what that could be 🔁. First, let's figure out what the host should do once it knows a gateway is down. Honestly, aside from one small catch, it's super easy 💅. The host picks some other gateway and tries sending the datagram there. If that gateway is up, it'll either deliver the datagram or send back some ICMP advice ✨. Since, once the transient period is over, any gateway can give correct advice, the host is basically in the best shape it can hope for as soon as it hears back from any gateway 👑.

The catch is that the backup gateway could also be down 😬. So whatever algorithm the host uses to detect dead gateways has to keep running as the host works through every gateway it knows about, one at a time 🔁.

The hard part is figuring out how the host keeps its table of every gateway it can reach directly 🤔. Right now, the internet protocol spec doesn't define any message a host can use to ask for that table 🙈. That's because different networks can have totally different ways of filling it in 🌈.

• On a broadcast net, like an ethernet or a ringnet, every gateway can just broadcast the table now and then, and the host only has to listen 🎧.
• Some networks offer logical addressing, where a whole group of machines shares one group address, and the host can send a request for help to that address 🦋.
• If neither of those exists, the host can build its table by remembering every gateway it has ever gotten a message from 🧠.
• Sometimes the table, or at least its first few entries, has to be set up by hand by a manager or operator at the site 🛠️.

If a network gives absolutely no support for this kind of host query, someone will have to step in manually at least once so the host can learn about at least one gateway 🫶.

4. Host Algorithms for Fault Isolation 🔍

okay, back to the question from earlier: how does the host figure out it's talking to a dead gateway so it knows to switch to another one on its list? There are several algorithms that can do this 🧚. They're all pretty simple to build, but each one puts a different amount of load on the host, the gateway, and the network 💥. So which one you pick really depends on the details of your network and your host 💖.

1. Network Level Detection 🌐

lots of networks, especially the Arpanet, handle this exact job inside the network itself 🙌. If a host sends a datagram to a dead gateway on the Arpanet, the network sends back a "host dead" message. That's exactly what the host needs to know to switch to a different gateway 🔄. Some early Internet implementations on the Arpanet threw those messages away 💀. That was an extremely bad idea 🙈.

2. Continuous Polling 📡

the ICMP protocol has an echo mechanism that lets a host ask a gateway for a response 🏓. A host could just send that echo at a reasonable pace to keep checking that the gateway is still alive 💓. It works, but to catch a fault in a reasonable time, the messages have to go out pretty often, and that can mean a crazy amount of overhead for the host, the network, and the gateway 😩. This strategy is prohibited unless a specific analysis has shown the overhead is tolerable 🚫.

3. Triggered Polling ⚡

if you could limit polling to only the moments when something seems wrong, the overhead would be totally manageable ✨. That works as long as you can get the right hints from your higher-level protocols 🎯. For example, the TCP layer could be set up so that whenever it retransmits a segment more than once, it sends a hint down to the IP layer that starts polling 🔔. This doesn't have much overhead, but the host might be a bit slow to react to an error. Polling only starts after the hint, so the host can't confirm something's wrong until then, and by that point the TCP above might have already timed out 😭.

Both forms of polling have a small flaw 🙃. Hosts respond to ICMP echo messages just like gateways do. So polling can't catch the mistake where an address the host thinks is a gateway is actually a host 🤦‍♀️. That kind of mix-up can happen when machines' physical addresses get rearranged 🔀.

4. Triggered Reselection 🦋

this strategy also uses a hint from a higher level, like the previous one, but it skips polling completely 🙅‍♀️. Whenever a higher level complains that the service seems defective, the Internet layer can pick the next gateway from its list of available gateways and switch to it 🔁. If that gateway is up, a wrong switch isn't a real problem, because the worst that happens is a redirect message telling the host to go back to the original gateway 🔙. And if the original gateway really was down, you've got a new route right away, so recovery takes less time ⏩. Honestly, this one is so clever, and it's probably the most generally useful option in cases where the network itself doesn't provide fault isolation 👑. (Regretfully, I've forgotten who suggested this idea to me. It is not my invention 🙈.)

5. Higher Level Fault Detection 🎀

so far we've been talking about fault detection and recovery at the IP layer 🧠. This section looks at what higher layers like TCP should do 🫶. TCP has just one fault recovery move: it keeps retransmitting a

✨ to be continued, bestie! Part 2 picks up right here, so stay tuned 💅🌸

okay so TCP keeps a segment hanging around until either it gets an acknowledgement or its connection timer runs out ⏰ and that's the whole vibe 🫶. As we talked about, retransmission can be the trigger that asks IP for fault recovery 🔁. In the other direction, info can flow up from IP too, like ICMP Destination Unreachable messages or error messages from the attached network 📨. The only subtle question is what TCP should do when one of those errors shows up, or when the connection timer expires 🤔.

The TCP spec talks about the timer. In the open call, the timeout is an optional value the client of TCP can set 🎀. If any segment stays unacknowledged for that whole period, TCP should abort the connection 🚪. The default is 30 seconds ⏱️. Early TCPs often used a fixed timeout interval, but that didn't work well in practice, as the next bit will show 🙈.

Clients of TCP come in two flavors 💁‍♀️. There are the ones running on behalf of a human, like Telnet, and the ones supporting a program, like a mail sender 📬. Humans need a sophisticated response to errors. Depending on what went wrong, they might want to bail right away, or they might wait a long time to see if things get better 😤. Programs don't have that human impatience, but they also can't make complex decisions based on the exact details of an error 🤖. For them, a simple timeout is totally reasonable 💖.

So, at least two modes of operation are needed in TCP 👑. One mode, for programs, abandons the connection with no exceptions if the TCP timer expires 💀. The other mode, for people, never abandons the connection on its own initiative. Instead, it reports to the layer above when the timer expires 🔔. That way the human user can see error messages coming from all the relevant layers, TCP and ICMP, and can ask TCP to abort if that makes sense 🙋‍♀️. This second mode needs TCP to be able to send an asynchronous message up to its client to report the timeout, and it needs error messages arriving at lower layers to flow up through TCP as well ✨.

Above TCP, fault detection is also required 🧐. Two things can happen. First, the foreign client of TCP can fail even though TCP is still running. The data still gets acknowledged, so the timer never expires 😶. Second, the communication path can fail without the TCP timer going off, simply because the local client has nothing to send 🌫️. Both of these have caused real trouble 😭.

Sending mail is the example for the first case 📧. When you send mail with SMTP, there's an SMTP-level acknowledgement that comes back once a piece of mail is delivered successfully 💌. Several early mail receiving programs would crash right after they got all of the mail text. TCP didn't notice a timeout, because there was no unacknowledged data outstanding, but the mail hadn't been acknowledged at the SMTP level yet 💀. So early mail senders waited forever for an SMTP-level acknowledgement that never came 🥲. The obvious fix was to set a timer at the SMTP level, but the first try didn't work, because there was no simple way to pick the timer interval 🙃. If the interval was short, it expired during normal operation when sending a large file to a slow host 🐌. An interval of many minutes was needed to avoid false timeouts, but that meant failures were only detected very slowly 😩. The current solution in several mailers is to pick a timeout interval that's proportional to the size of the message 📏.

Server telnet is the example of the other kind of failure 🖥️. The communications link can easily fail while no traffic is flowing, maybe because the user is just thinking 🤔. Eventually the user tries to type something, and that's when they find out the connection is dead and abort it 😤. But the host end of the connection has nothing to send, so it never discovers anything is wrong and just waits forever 👻. On some systems there's no way for a user in a different process to destroy or take over that hanging process, so there's no way to recover 🫠.

One fix would be for the host's server telnet to check on the user end every now and then, to see if it's still up 🔍. (Telnet doesn't have an explicit query feature, but the host could negotiate some unimportant option, which should get either agreement or disagreement back 🤝.) The only problem is that a reasonable sample interval, applied to every user on a large system, can generate an unacceptable amount of traffic and system overhead 📈. A smart server telnet would only do this query when something seems wrong, maybe when there's been no user activity for a while 🧠.

In both of these cases, the general conclusion is that client-level error detection is needed, and the details of the mechanism depend heavily on the application 🧩. Application programmers have to be made aware of the problem of failures 🚨. They also need to understand that error detection at TCP or lower levels can't solve the whole problem for them 💅.

---

6. Knowing When to Give Up 🌈

It's not obvious, when error messages like ICMP Destination Unreachable arrive, whether TCP should abandon the connection 😬. The reason these errors are hard to interpret is that, as discussed above, after a failure of a gateway or network, there's a transient period when the gateways may have incorrect information. So irrelevant or incorrect error messages can sometimes come back 🌀. For example, an isolated ICMP Destination Unreachable might show up at a host if a packet gets sent while the gateways are still trying to find a new route 🛣️. Tearing down a TCP connection because of a message like that would mean ignoring one of the Internet's most valuable features: for many internal failures, it rebuilds its function without any disruption to the end points 🦋.

But if failure messages don't imply a failure, what are they even for? 🤷‍♀️ Actually, error messages serve several important purposes ✨. First, if they arrive in response to opening a new connection, they probably come from opening the connection improperly (for example, to a non-existent address), rather than from a transient network failure 🙅‍♀️. Second, they provide valuable information, after the TCP timeout has occurred, about the probable cause of the failure 🔎. Finally, certain messages, like ICMP Parameter Problem, imply a possible implementation problem 🛠️. In general, error messages give valuable information about what went wrong, but they aren't to be taken as absolutely reliable 🫢. A general alerting mechanism, like the TCP timeout discussed above, gives a good indication that whatever is wrong is a serious condition. But without the advisory messages to back up the timer, the client has no way of knowing how to respond to the error 🙈. The combination of the timer and the advice from the error messages gives the client layer a reasonable set of facts to work with 💜. It's important that error messages from all layers get passed up to the client module in a useful and consistent way 🎀.