RFC 1133

Routing between the NSFNET and the DDN

📅 published November 1989, from ietf

RFC 1133: Routing between the NSFNET and the DDN 🦋

Title block tea ☕

RFC number: 1133
Title: Routing between the NSFNET and the DDN
Authors: J. Yu and H-W. Braun
Organization: Merit Computer Network
Working group: Network Working Group
Date: November 1989
Category: It's a case study, and the memo says no standards are specified in it 📖
Obsoletes / Updates: The title block doesn't list any RFCs it obsoletes or updates. 🤷‍♀️

Status of this Memo ✨

okay bestie, this doc is a case study of how routing got built between the NSFNET and the DDN (that's the MILNET and ARPANET together 🫶). The authors hope it can help other Administrative Domains link up too. They'd love feedback and suggestions on the methods used for the interconnections. No standards are specified in this memo. And distribution is unlimited, so share away 💌

1. Definitions for this document ✨

The NSFNET is the backbone network of the National Science Foundation's computer network infrastructure. It interconnects multiple autonomously administered mid-level networks, and those mid-levels connect autonomously administered networks of campuses and research centers 🎓. The NSFNET also connects to multiple peer networks made of national network infrastructures from other federal agencies. One of those peers is the Defense Data Network (DDN). For the sake of this discussion, you should think of the DDN as the combo of the DoD's MILNET and ARPANET component networks, and both of those are national in scope 🇺🇸

It should be pointed out that network announcements in one direction result in traffic the other direction, e.g., a network announcement via a specific interconnection between the NSFNET to the DDN results in packet traffic via the same interconnection between the DDN to the NSFNET. This is the two-way street energy 🔁

2. NSFNET/DDN routing until mid '89 ✨

Until mid-1989, the NSFNET and the DDN were connected via a few intermediate routers, and those routers were connected to the ARPANET 🛣️. These routers exchanged network reachability information via the Exterior Gateway Protocol (EGP) with the NSFNET nodes as well as with the DDN Mailbridges. In the context of network routing, these Mailbridges can be viewed as route servers. They exchange external network reachability information via EGP, and they use a proprietary protocol to exchange routing information among themselves 🤫 Currently, there are three Mailbridges at east coast locations and three Mailbridges at west coast locations. Besides functioning as route servers, the Mailbridges also provide for connectivity (i.e., packet switching) between the ARPANET and the MILNET. Multitasking queen 👑

The intermediate systems between the NSFNET and the ARPANET were under separate administrative control, typically by a NSFNET mid-level network.

For a period of time, the traffic between the NSFNET and the DDN was carried by three ARPANET gateways. These ARPANET gateways were under the administrative control of a NSFNET mid-level network or a local site, and they had direct connections to both a NSFNET NSS and an ARPANET PSN. These routers had simultaneous EGP sessions with a NSFNET NSS as well as a DDN Mailbridge. This made them function as packet switches between the two peer networks. As network routes were established, packets were switched between the NSFNET and the DDN 🔀

The NSFNET used three NSFNET/ARPANET gateways, which had been provided by three different sites for redundancy purposes. Those three sites were initially Cornell University, the University of Illinois (UC), and Merit. When the ARPANET connections at Cornell University and the University of Illinois (UC) were terminated, a similar setup was introduced at the Pittsburgh Supercomputer Center and at the John von Neumann Supercomputer Center. Together with the Merit connection, that kept the redundancy going 💪

As described in RFC1092 and RFC1093, NSFNET routing is controlled by a distributed policy routing database. That database controls the acceptance and distribution of routing information, and this control also extends to the NSFNET/ARPANET gateways 🗂️

2.1. Inbound announcement -- Routes announced from the DDN to the NSFNET

In the case of the three NSFNET/ARPANET gateways, each of the associated NSSs accepted the DDN routes at a different metric. The route with the lowest metric was favored for traffic towards the specific DDN network. But if that specific gateway to the DDN ran into problems with loss of routing information, one of the redundant gateways would take over and carry the load as a fallback path 🛟 Assuming consistent DDN routing information at all three gateways (as received from the Mailbridges), only a single NSFNET/ARPANET gateway was used at a given time for traffic from the NSFNET towards the DDN. The other two stood by as hot backups 🔥 The metric for network announcements from the DDN to the NSFNET was coordinated by the Merit/NSFNET project.

2.2. Outbound announcement -- Routes announced from the NSFNET to the DDN

Each NSS involved with NSFNET/DDN routing had an EGP peer relation with the NSFNET/ARPANET gateway. Via EGP, it announced a certain set of NSFNET connected networks to its peer. Which networks got announced was, again, controlled by the distributed policy routing database 📋 The NSFNET/ARPANET gateway then redistributed the networks it had learned from the NSS to the DDN via a separate EGP session. Each of the NSFNET/ARPANET gateways used a separate Autonomous System number to communicate EGP information with the DDN. These Autonomous System numbers were also not the same as the ones the NSFNET backbone uses to communicate with directly attached client networks. The NSFNET/ARPANET gateways used the Autonomous System number of the local network 🔢

The metrics for announcing network numbers to the DDN Mailbridges were set according to the requests of the mid-level network that the specific individual network was a client of. The mid-level network also influenced which specific NSFNET/ARPANET gateway got used, including primary/secondary selection. These primary/secondary selections among the NSFNET/ARPANET gateways allowed for redundancy, while the preference of network announcements was modulated by the metric used for the announcements to the DDN from the NSFNET/ARPANET gateways. Some of the selection decisions were based on the reliability of a specific gateway, or on congestion expected in a specific PSN that connected to the NSFNET/ARPANET gateway 🧠

2.3. Administrative aspects

From an administrative point of view, the NSFNET/ARPANET gateways were administered by the institution the gateway belonged to. This has never been a real problem, thanks to the excellent cooperation received from all the involved sites 🫶

3. NSFNET/DDN routing via attached Mailbridges ✨

During the first half of 1989, a new means of interconnectivity between the NSFNET and the DDN was designed and implemented. Ethernet adapters were installed in two of the Mailbridges, which previously just connected the MILNET and the ARPANET, allowing a direct interface to NSFNET nodes. One of those two Mailbridges is on the west coast at NASA-Ames, located at Moffett Field, CA. The other is on the east coast at Mitre in Reston, VA 🌉 With this direct interconnection, the NSFNET could exchange routing information directly with the DDN route servers, with no gateway operated by a mid-level network in the middle. This also eliminated the need to traverse the ARPANET in order to reach MILNET sites. It also lets the Defense Communication Agency and the National Science Foundation exercise control over the interconnection on a need basis. For example, the connectivity can now be easily disabled from either site when network security concerns get tighter 🔒

3.1. Inbound announcement -- Routes announced from the DDN to the NSFNET

The routing setup for the direct Mailbridge connections is somewhat different from the NSFNET/ARPANET gateways we just talked about. Instead of a single NSFNET/ARPANET gateway carrying all the traffic from the DDN to the NSFNET at any moment, the distribution of network numbers is now split between the two Mailbridges. This gives a distributed load, with specific network numbers always preferring a particular Mailbridge under normal operating circumstances. If there's an outage at one of the Mailbridge connections, the other one fully takes over the load for all the involved network numbers 🦸‍♀️ For this setup, the two DDN links are known as two different Autonomous System numbers by the NSFNET. The routes learned via the NASA-Ames Mailbridges are part of Autonomous System 164, which is also the Autonomous System number the Mailbridges use by themselves during the EGP session. For the EGP sessions with the Mitre Mailbridge, the DDN-internal Autonomous System number of 164 is overwritten with a different Autonomous System number (in this case 184). That means the routes learned via the Mitre Mailbridge become part of Autonomous System 184 within the NSFNET 🔄

The NSFNET-inbound routing is controlled by the distributed policy routing database. In particular, the network number is verified against a list of legitimate networks, and a metric is associated with an authorized network number for a particular site. For example, both NSSs in Palo Alto and College Park learn net 10 (the ARPANET network number) from the Mailbridges they are connected to and have EGP sessions with. The Palo Alto NSS will accept net 10 with a metric of 10, while the College Park NSS will accept the same network number with a metric of 12. Therefore, traffic destined for net 10 will prefer the path via the Palo Alto NSS and the NASA-Ames Mailbridge. If the connection via the NASA-Ames Mailbridge stops functioning, the traffic gets re-routed via the Mailbridge link at Mitre 🗺️ Each of the two NSSs accepts half of the network routes via EGP from its co-located Mailbridge at a lower metric, and the other half at a higher metric. The half with the lower metric at the Palo Alto NSS is the same set that uses a higher metric at the College Park NSS, and vice versa. It's a whole mirror situation 🪞

There are at least three different possibilities for how the NSFNET could select a path to a DDN network via a specific Mailbridge, i.e., the one at NASA-Ames versus the one at Mitre:

Assign a primary path for all DDN networks to a single Mailbridge and use the other purely as a backup path.
Distribute the DDN networks randomly across the two Mailbridges.
Let the DDN administration inform the NSFNET which networks on the DDN are closer to a specific Mailbridge, so that the particular Mailbridge would accept these networks at a lower metric. The second Mailbridge would then function as a backup path. From a NSFNET point of view, this would mean treating the DDN like other NSFNET peer networks, such as the NASA Science network (NSN) or DOE's Energy Science Network (ESNET) 🧩

We are currently using alternative (2) as an interim solution. At this time, the DDN administration is having discussions with NSFNET about moving to alternative (3), which would give them control over how the DDN networks would be treated in the NSFNET. Stay tuned, this plot has more to come 👀✨

3.2. Outbound announcement -- Routes announced from the NSFNET to the DDN ✨

okay so the NSFNET is the one that decides the metrics for its networks when it announces them to the DDN 💅 The way it picks is based on distance between NSSs. We're talking about the NSS that brings a specific network into the NSFNET, and either one of the NSSs with a co-located Mailbridge. Here, "distance" just means hop count across the NSFNET backbone 🦋

Let's use the Princeton NSS as an example. Right now it's one hop away from the NSS co-located with the Mitre Mailbridge, but it's three hops away from the NSS with the NASA-Ames Mailbridge. So for networks whose primary path goes through the Princeton NSS, the Mitre Mailbridge gets those announcements at a lower metric than the NASA-Ames one. 🌸

This means traffic from the DDN to networks hanging off the Princeton NSS gets routed through the Mailbridge at Mitre, over to the College Park NSS, and then through the Princeton NSS to its final destination. The payoff is that traffic coming into the NSFNET from the DDN takes the shortest path to its NSFNET destination under normal operating conditions. That's giving efficiency 🧚

3.3. Administrative aspects 💜

Any of the networks connected via the NSFNET can get connectivity to the DDN through the NSFNET, but only if it's requested by the mid-level network that the specific network connects through. 🫶

For networks that don't have a DDN connection other than through the NSFNET, the NSFNET announces the nets through one of the Mailbridges at a low metric to set up a primary path (e.g., metric "1"). It then announces them through the second Mailbridge as a secondary path (e.g., metric "3").

For networks that have their own DDN connection and want to use the NSFNET as a backup to the DDN, the NSFNET announces those networks through the two Mailbridges at higher metrics. So it's basically a fallback situation 🍓

Now, if mid-level networks want their client networks announced to the DDN through the NSFNET, they have to make a specific request. 🎀 That request needs to say whether this would be a primary connection for the specific networks. If the request is for a fallback connection, it needs to state the existing metrics that are in use for announcements of the network to the DDN.

Bestie, pay attention to this part. Primary vs. fallback changes what info you have to include in your request. 🙈

4. Shortcomings of the current NSFNET/DDN interconnection routing ✨

Okay, real talk: the current setup makes full use of the two Mailbridges that connect straight to the NSFNET when it comes to redundancy and load sharing. 👑 But when it comes to performance optimization, like packet propagation delay between source/destination pairs sitting on disjoint peer networks, there are some shortcomings. These shortcomings aren't easy to fix because of the limits of the current architecture. Still, it's a worthwhile topic to discuss so we can make future improvements. 💅

To make the discussion easier, here's the setup. We're treating the NSFNET and the DDN as clouds, and the two have two connections: one on the east coast and one on the west coast. 🌈

Here's the terminology we'll be using:

      mb-east -- the Mailbridge at Mitre

      mb-west -- the Mailbridge at Ames

      NSS-east -- the NSS egp peer with mb-east

      NSS-west -- the NSS egp peer with mb-west

      DDN.east-net -- networks connected to DDN and physically closer to
                      mb-east

      DDN.west-net -- networks connected to DDN and physically closer to
                      mb-west

      NSF.east-net -- networks connected to NSFNET and physically closer
                      to NSS-east

      NSF.west-net -- networks connected to NSFNET and physically closer

                      to NSS-west

Traffic between NSFNET<->DDN falls into these two patterns:

      a) NSF.east-net <-> DDN.east-net or
         NSF.west-net <-> DDN.west-net

      b) NSF.east-net <-> DDN.west-net or
         NSF.west-net <-> DDN.east-net

The ideal traffic path for a) and b) should look like this. 🦋

   For traffic pattern a)

        NSF.east-net<-->NSS.east<-->mb-east<-->DDN.east-net

   or

        NSF.west-net<-->NSS.west<-->mb-west<-->DDN.west-net

   For traffic pattern b)

        NSF.east-net-*->NSS.west-->mb-west-->DDN.west-net-**->mb-east
                                                                    |
                                              NSF.east-net<--NSS-east

   or

        NSF.west-net-*->NSS.east-->mb-east-->DDN.east-net-**->mb-west
                                                                    |
                                              NSF.west-net<--NSS-west

Here's the legend that goes with the diagrams:

   Note:

        -*-> is used to indicate traffic transcontinentally traversing
        the NSFNET backbone

        -**-> is used to indicate traffic transcontinentally traversing
        the DDN backbone

        The traffic for a) will transcontinentally traverse NEITHER the
        NSFNET backbone, NOR the DDN backbone.

        The traffic for b) will transcontinentally traverse NSFNET once
        and DDN once and only once for each.

For the current setup, though, pattern a) traffic could end up crossing both the NSFNET and the DDN transcontinentally. 😭 And pattern b) traffic could end up crossing the DDN in both directions. 💀

To hit the ideal path, the NSFNET has to implement (3) as stated above. In other words, it has to treat the DDN like any other NSFNET peer or mid-level network. As I mentioned before, NSFNET and DDN people are already talking, and the DDN is considering giving the NSFNET the information it needs to hit these goals soon. ✨

Once that happens, packet traffic is way less likely to take unnecessary trips across national backbones. 🎀

Now here's the big idea. One of the best ways to optimize traffic between two peer networks (and it's not limited to the NSFNET and the DDN) is to avoid letting traffic cross a backbone that's comparatively slower and/or has a significantly larger diameter. 🧚 Take the NSFNET and the DDN. The NSFNET has a T1 backbone and a maximum diameter of three hops, while the DDN is a relatively slow network running largely at 56Kbps. So performance is better when traffic crosses the DDN as little as possible. Whenever traffic needs to reach a destination outside the DDN, it should find the closest Mailbridge to exit the DDN. 🌸

The current DDN routing architecture can't do this, and here's why. 🙈

First, the technology is built on a core model. It doesn't expect a single network to be announced from multiple places. An example of multiple announcements would be two NSSs announcing a single network number to the two Mailbridges at their different locations. 💅

Second, the way the existing Mailbridges swap routing info among themselves uses a protocol similar to EGP. That info then gets distributed via EGP to the DDN-external networks. The physical distance info and the locations of network numbers get lost along the way, so neither the Mailbridges nor the external gateways can do path optimization based on physical distance or propagation delay. 😭

This is hard to change, because in the DDN, link-level routing info is decoupled from IP-level routing. IP-level routing has no info about the topology of the physical infrastructure. So even if an external gateway to a DDN network learned a particular network route from two Mailbridges, it couldn't prefer one DDN exit point over the other based on distance to the respective Mailbridge. 🫶

5. Conclusions 🌈

Recent changes to the interconnection architecture between the NSFNET and DDN peer networks have brought big performance and reliability improvements. ⭐ Still, there's room for more improvement and for rationalizing this inter-peer routing. 👑 Getting there would most likely take significant architectural changes within the DDN. 💜

6. References 📚

[1] Rekhter, Y., "EGP and Policy Based Routing in the New NSFNET Backbone", RFC 1092, IBM Research, February 1989.
[2] Braun, H-W., "The NSFNET Routing Architecture", RFC 1093, Merit/NSFNET Project, February 1989.
[3] Collins, M., and R. Nitzan, "ESNET Routing", DRAFT Version 1.0, LLNL, May 1989.
[4] Braun, H-W., "Models of Policy Based Routing," RFC 1104, Merit/NSFNET Project, February 1989.

Security Considerations 🔥

Security issues are not addressed in this memo. 💀 Not gonna lie, that's a big gap, so keep it in mind. 🙈

Authors' Addresses 🎀

   Jessica (Jie Yun) Yu
   Merit Computer Network
   1075 Beal Avenue
   Ann Arbor, Michigan 48109

   Telephone:      313 936-2655
   Fax:            313 747-3745
   EMail:          jyy@merit.edu

   Hans-Werner Braun
   Merit Computer Network
   1075 Beal Avenue
   Ann Arbor, Michigan 48109

   Telephone:      313 763-4897
   Fax:            313 747-3745
   EMail:          hwb@merit.edu

📦 where this page comes from

ietf: rfc1133 1989-11
💖 girlified by claude-haiku-5-5 on 2026-10-15: a language model's rewrite of the original, under its terms