RFC 947

1. Status of this Memo ✨

Okay bestie, let's get into it 💖 This RFC is about stretching a network's broadcast domain so it covers more than one physical network, using something called a broadcast packet repeater 🦋. The paper walks through the problem and why we cared about it, which comes from building a distributed operating system 💅. We compare different ways to extend a broadcast domain and explain why we picked the one we actually built 🔥. There's also info on the implementation and some notes on how it performs 📝.

We really hope this helps people in the Internet who have apps that use broadcasting and got stuck because they could only broadcast within a single network 🥺🫶. Everything here is accurate as of the publication date, but the specifics, especially the implementation stuff, might change later 🌸. Distribution of this memo is unlimited 🌈.

2. The Problem 💀

So communication between hosts on separate networks has mostly been handled by Internet protocols and gateways 🧚. One thing that hasn't been solved in the Internet is extending broadcasting so it reaches two or more networks 😭. Broadcasting is honestly so efficient, because you send one packet and it reaches a whole bunch of hosts 🙌. Lots of local area network (LAN) architectures support a broadcast mechanism directly 💜.

The catch is that this broadcast mechanism has a shortcoming when you're in an environment with multiple LANs connected by gateways, like the DARPA Internet 🙈. Broadcast packets only get received by hosts on the physical network where they were broadcast. So any application that uses LAN broadcasting can only broadcast to hosts on its own physical network 💀.

We leaned on broadcasting when we built the Cronus Distributed Operating System [1] 🎀. Cronus gives services and communication to processes spread across lots of different kinds of computer systems ✨. It's built around logical clusters of hosts connected to one or more high-speed LANs 🌟. Communication in Cronus is built on TCP and UDP 💅. Cronus uses broadcasting to dynamically locate resources on other hosts and to collect status information from a group of servers 🔥.

Here's the thing: Cronus's broadcast capabilities aren't meant to stop at the edge of a single LAN, so we needed a way to extend our broadcasting domain to hosts on distant LANs 🦋. That let us experiment with clusters that span several physical networks 🌈. Cronus mostly uses broadcasting to talk to a subset of the hosts that actually receive the message 🙈. A multicast mechanism would be a better fit, but it wasn't available in some of our network implementations, so we went with broadcast for the first round of Cronus utilities 🍓.

3. Our Solution 🧚

The technique we tried for the multi-network broadcasting problem is called a "broadcast repeater" 💖. A broadcast repeater is a mechanism that transparently relays broadcast packets from one LAN to another 🦋. It can also forward broadcast packets to hosts on a network that doesn't support broadcasting at the link level ✨. This gives us flexibility while still getting the convenience of LAN broadcasts 💅.

Our broadcast repeater is a process running on a network host that listens for broadcast packets 👀. The packets get picked up and retransmitted to one or more repeaters connected to distant LANs, using a simple repeater-to-repeater protocol 🫶. The repeater on the receiving end rebroadcasts the packet on its LAN and keeps the original packet's source address 🔥. The repeater can get really smart about which messages it forwards 🧠. Right now we only forward broadcast messages sent to the UDP ports Cronus uses 👑. But messages can be selected based on any field in the UDP or IP headers, or every IP-level broadcast message can be forwarded 🌸.

4. Alternatives to the Broadcast Repeater 🙈

We looked at a few other options before we settled on how to forward broadcast messages 🔍. One idea was to add functions to the Internet gateways 🚪. Gateways would listen at the link level for broadcast packets and relay them to one or more gateways on distant LANs 🌉. Those gateways could then send the same packet onto their own networks, using the local network's link-level broadcast capability if there is one 📡. Every gateway in this scheme would have to keep tables of all the other gateways that are supposed to receive broadcasts 📋. If the receiving gateway serves a network without broadcast capability, it could forward the messages directly to one or more designated hosts on that network, but again, the gateway would need to keep tables for that 😬.

We rejected putting this kind of function into gateways for several reasons 🙅‍♀️:

• (a) it would require extensions to the gateway control protocol so the lists gateways keep could be updated,
• (b) not every message (for example, LAN address-resolution messages) needs to be forwarded, so control over forwarding should sit at higher protocol levels than gateways may have access to,
• (c) Cronus could be deployed in environments where the gateways come from alternative vendors who may not implement broadcast propagation,
• (d) as part of the underlying network, gateways are likely to be controlled by a different agency from the one that configures a Cronus system, which adds bureaucratic complexity to reconfiguration 💀.

Another idea we rejected was putting broadcast functionality into the Cronus kernel 🧩. The Cronus kernel is a process that runs on each host participating in Cronus, and its job is to route all messages passed between Cronus processes 🛣️. It's the only program in the Cronus system that directly uses broadcast capability, since other parts of Cronus communicate through mechanisms the kernel provides 💅. We could either remove the Cronus kernel's dependence on broadcast entirely, or add a mechanism that emulates broadcast with serially transmitted messages when the underlying network doesn't offer a broadcast facility 🔁. Either way, every Cronus kernel process would need to know the addresses of all the other participants in a Cronus system 😤. We see that as an undesirable limit on configuration flexibility 🙅‍♀️. Plus, this would only work for Cronus, while the broadcast-repeater solution works for other broadcast-based protocols too 🌈.

5. Implementation 🛠️

The broadcast repeater is implemented as two separate processes: the forwarder and the repeater 👯‍♀️. The forwarder waits for broadcast UDP packets that come across its local network and match one or more specific port numbers (or destination addresses) 👀. When it finds one, it encapsulates the packet in a forwarder-repeater message and sends it to a repeater process on a foreign network 📦. The repeater then relays the forwarded packet onto its LAN, using that network's link-level broadcast address in the packet's destination field, but it keeps the source address from the original packet 🔥.

When the forwarder starts for the first time, it reads a configuration file 📄. That file specifies the addresses of the repeater processes and selects which packets should be forwarded to each repeater process 🎯. Different repeaters can select different sets of UDP packets 🌸. The forwarder tries to set up a TCP connection to each repeater listed in the configuration file 🔌. If a TCP link to a repeater fails, the forwarder keeps retrying the connection periodically 🔁. Non-repeater hosts can be listed in the configuration file too 🧚. For those hosts, the forwarder simply replaces the destination broadcast address in the UDP packet with the host's address and sends this new datagram directly to the non-repeater host 💌.

Now, if a repeater and a forwarder live on the same LAN, there's a problem: the forwarder might pick up packets that the repeater has already rebroadcast 😱. To guard against rebroadcasting forwarded packets ("feedback" or "ringing"), the forwarder doesn't connect to any repeater in its configuration file that's on the same network as the forwarder itself 🛡️. And to avoid a broadcast loop involving two LANs, where each has a forwarder talking to a repeater on the other LAN, forwarders don't forward packets whose source address isn't on the forwarder's LAN 🔒.

6. Experience 🏆

So far, the broadcast repeater has been implemented on a VAX running 4.2 BSD UNIX with BBN's networking software, and it's honestly been working really well for what we needed 🥹💖. Our current setup has two Ethernets that are physically separated by two other LANs 🗺️. For the past few months, the broadcast repeater has successfully stretched our broadcast domain to include both Ethernets, even though messages between them have to pass through at least two gateways 🌉. We did have to add a special capability to the BBN TCP/IP implementation, which lets privileged processes send out IP packets with another host's source address 🔐.

The repeater adds a fair amount of overhead on the shared hosts that support it 😮‍💨. That's because the forwarder process has to wake up for every UDP packet that arrives at the host, since the decision to reject a packet is made by user-level software and not in the network protocol drivers 🥱. One fix would be to do the packet filtering in the system kernel, while leaving the configuration management and rebroadcast mechanism in user code. Stanford/CMU has done exactly that in a UNIX packet filter they developed 🧠. As an alternative, we're planning to rehost the repeater function as a specialized network service on a microcomputer-based real-time system that's already part of our Cronus configuration 🖥️. That kind of machine is better suited to the job, because scheduling overhead is much lower for them than on a multi-user timesharing system ⏱️✨.

7. Reference 📚

• [1] "Cronus, A Distributed Operating System: Phase 1 Final Report", R. Schantz, R. Thomas, R. Gurwitz, G. Bono, M. Dean, K. Lebowitz, K. Schroder, M. Barrow and R. Sands, Technical Report No. 5885, Bolt Beranek and Newman, Inc., January 1985. The Cronus project is supported by the Rome Air Development Center. 🎓💜

8. Editors Note 💅

Also see RFCs 919 and 940 on this topic 🌈💖