RFC 1306

Experiences Supporting By-Request Circuit-Switched T3 Networks

📅 published March 1992, from ietf

RFC 1306: Experiences Supporting By-Request Circuit-Switched T3 Networks 🎀✨

okay bestie, so this is RFC 1306 from the Network Working Group 🧚 It's dated March 1992 and written by A. Nicholson and J. Young, both from Cray Research, Inc. 💖 It's an informational doc, so it shares experiences with the Internet community instead of setting a standard 📝 The header doesn't list any obsoletes or updates, so it isn't replacing or changing any other RFC 🫶

Status of this Memo ✨

real quick: this RFC gives info to the Internet community. it does not specify an Internet standard. distribution of this memo is unlimited, so share away 💅

Abstract 🌸

so this memo is about what a project team at Cray Research, Inc. went through when they built support for circuit-switched T3 services 🦋 some of the issues might not be directly relevant to the research problems of the Internet, but researchers and implementers could still find them super interesting 🤓

Cray Research devs got a chance to use a circuit-switched T3 network for wide area networking, so they came up with an architectural model. the idea is you turn the connection on when an application starts a bulk data transfer, and you release it once the transfer is done 🔥

they tested three software implementations of this feature and wrote down the results here. there are a bunch of issues involved and more research is needed. network users are starting to see the value of this service and are planning to use by-request circuit-switched networks. a standard method of access will be needed so that vendors of circuit-switched network support products can interoperate with each other 👑

Acknowledgements 🫶

huge thank yous to the T3 project team and the rest of the Networking Group at Cray Research, Inc. 💜 Wayne Roiger, Gary Klesk, Joe Golio, John Renwick, Dave Borman and Craig Alesso, you are all iconic 🌈

Overview 💫

okay so if you use wide-area networks, you usually have to pick between low cost and high speed when you connect over long-haul links. dedicated high-speed connections cost so much money that they're basically out of reach for scientists and engineers on tight budgets 😭 for a lot of traditional apps that's fine, because the datasets live on the remote computer and the results come back as text-only, where a low-speed connection is enough. but for visualization and other data-heavy apps, this limit can seriously hurt the usefulness of high performance computing tools that are only available through long-haul network connections 💀

supercomputers are one perfect example 👑 tons of people who could benefit from them are stuck with slow network connections to a centrally located supercomputer. the fix is a circuit-switched network that gives you high speed at a reduced cost, since you only allocate the network when you actually need it ⚡

picture a researcher using a visualization app. they could use a dedicated low-speed link together with a circuit-switched high-speed link. the researcher logs in to the remote supercomputer over the low-speed link. after running whatever programs are needed to prep the visualization, the high-speed connection turns on and moves the graphics data to the researcher's workstation 🖥️✨

we built and demoed this capability in September, 1990, at the Telecommunications Association show in San Diego, using this exact type of visualization app 🎀 it'll also be available in a forthcoming release of our system software, so yay 🌷

Architectural Model 🧩

we built our circuit-switched support around a simple model of a switched network. somewhere along the path between two hosts there's a switched network connection. this connection will probably link two enterprise networks run by the same organization, since admin overlap between them is useful for accounting and configuration 📊 we believe that with further investigation, circuit-switched network support could be extended to multiple switched links in an internet environment 🔍

the switch works on a "by-request" basis (also called "on-demand") 🔘 when it receives a request to make a network connection, it does so if possible, and it breaks the connection when asked. the switch will not activate automatically if someone tries to send data over an incomplete connection 🙅

we also assumed the circuit would be switched on a connection basis rather than a packet basis 💅 once an application starts sending data over the switched connection, it sends all the data it has without stopping until it's finished, and then it releases the connection. the assumption is that the amount of data will be large enough that circuit setup time is negligible compared to the transfer period. otherwise, it's not worth the effort to support the circuit-switched network for small data transfers 😤

this model needs a request message to go to the switch right before the application starts a large bulk transfer, asking that the switched network connection be activated. once the link is up, the application starts sending, and the network routes all of its data through the switched network. as soon as everything has been sent, a message goes to the switch to turn off the switched link, and the network goes back to routing data through the slower link 🔄

the prototype we built for the TCA show was designed around this model. we connected an FDDI backbone at Cray Research in Eagan, Minnesota to the TCA show's FDDI network through 2 NSC 703 FDDI/T1/T3 routers. MCI provided a dedicated T1 line and a switched T3 line, using a DSC DS3 T3 switch located in Dallas, Texas 🗺️ together these networks gave connectivity between a Cray Research computer in Eagan and a Sun workstation on the show floor in San Diego 🌴

Alternative Solution Strategies 🧠

the first thing we had to deal with was the circuit switch itself 🔌 the DS3 switch we had was accessed through a dial-up modem, and it communicated using a subset of the CCITT Q.295 protocol. activating the switch took a 4 message exchange, and deactivating it took a 3 message exchange. we felt the protocol was awkward and might be different for other switch hardware. we also figured the dial-up part of talking to the switch had the same drawbacks. a good solution would need a cleaner way of controlling the switch from the source host requesting the switched line 😩

next up is the source host software that requests and releases the switched network. ideally, the switched network gets activated right before the data transfer and released as soon as all the data has been sent ⏱️ we considered special utility programs a user could run to control the link, special system libraries that application programs could call, or building the capability into the kernel 💻 we also thought about having these methods send messages to a daemon running on the source host, which would then talk to another entity that actually controls the switch 🤖

the last thing we looked at was choosing which switch-controlled network to use 🛤️ that involves both policy issues and routing issues. policy issues cover which users running which applications get to use the higher-cost switched links. and packets have to be routed among multiple connections offering different levels of service after they leave their source 📦

Implementations 🛠️

we developed a model for switch control through the internetwork that we believe is reasonable 💡 however, we experimented with three different source host implementations, and those are detailed here 🧪

Switch control 🎛️

our simplest design decisions were about the switch itself 🪄 we decided that the complex protocol and the dial-up line must be hidden from the source host requesting the switched link 🙈 the source host would use a simple request/release protocol, with messages sent through the regular network (as opposed to dial-up lines or other connections). some host reachable through the local network would run a program that translates those simple request and release messages into the more complicated switch protocol, and that host would also have the modem to handle the dial-up connection.

this has a variety of advantages. first, it isolates differences in switch hardware. second, multiple hosts can access the switch without needing multiple modems for the dial-up line. and it provides a central point of control for switch access 🗝️ we did not consider any alternatives to this model of switch control.

our initial implementation used a simple translator daemon running on a Sun workstation 🌞 listening on a raw IP port, it waited for switch control messages. when one arrived, it dialed up the switch and tried to handle the request, then sent back a success or failure response 📞 this host, together with the translator daemon software, is referred to as the switch controller 👑 the switch controller we used was local to our enterprise network, but it could reside anywhere in the Internet 🌍

later we designed a simple protocol for switch control, which was implemented in the translator daemon 🧩 that protocol is documented in RFC 1307, "Dynamically Switched Link Control Protocol" 📚

this problem is about deciding what entity on the source host will send the switch request and release messages to the switch controller, and when those messages get sent ⏰ we don't have very much field experience with this service yet, so we don't feel it's appropriate to recommend one method over the others 🙃 they all have advantages and disadvantages.

what we did do is build 3 different implementations of the request software and report on our experiences with each 💪 those are one set of special utility programs that communicate with the switch controller, plus 2 kernel implementations ⚙️ we did not experiment with special libraries, and we did not implement a daemon for switch control messages on the source host 🙅

Switch control user programs

Okay so this first version of switch control is the simplest one ever 💅. They wrote two little programs that send requests to the switch controller: one to activate the connection and one to deactivate it. Then the apps got tucked into shell scripts alongside those programs so they're super easy to run 🎀.

Big win: no kernel changes needed on any machine 🙌 Apps don't have to be modified either, and you can control who gets to use the circuit-switched links just by setting access permissions on the switch-control programs 🔐.

But ngl there are downsides 😭. First, the switch can stay on (and the user keeps getting billed 💸) during the dead time when the app is doing stuff that isn't bulk data transfer. Also, the on/off control only works per application, which is pretty coarse 🙈.

Another downside: most apps only use the destination host's address for transfer, and that's the only thing the transport and network layers have for routing packets 🗺️. So some other method must be used to tell apart traffic that should use the circuit-switched connection from lower-priority traffic. Route aliases (more on those below) can help with that one ✨.

Kernel switch control

Okay this one's way more intense 🔥. We made two different versions of switch control inside the operating system kernel. Both rely on the routing lookup code in the kernel to send the switch connect and tear down messages. The only difference is how they handle the time gap between asking the switch for something and getting a response back ⏳.

First up, routing table entries got expanded to hold the internet address of the switch controller plus state info for the switched connection 🧚. If a switch controller address is specified, the connection must be set up before packets may be sent on that route. We also added a separate module to handle sending and receiving the switch control messages 💌.

When a routing lookup is satisfied, the routing code checks whether the entry specifies a switch controller. If it does, the routine requesting switch setup gets called, and it sends a message over the Internet to the switch controller to set up the connection 📨.

In our first implementation, the routing lookup call returned right after sending the connection request message 🏃. That meant the transport protocol had to deal with the delay while the connection got set up, and tear it down if the switched connection couldn't be made. This had big ramifications 😬. For UDP and IP, packets had to be buffered for later transmission or they'd face almost certain extermination, since they'd probably start arriving at the switched connection before it was ready to carry traffic. So we decided this feature wouldn't be available for UDP or IP traffic 🙅.

We did make it work for TCP though 💖. TCP is already designed to buffer all data for possible later retransmission, so that wasn't a problem. Our first cut changed TCP to check that the route it was using was up if it was a switch controlled route. TCP wouldn't send any data until the route was complete, and it would close the connection if the switch didn't come up.

That didn't work well at first 😭. Every time TCP tried to send data before the switch came up, the retransmit time got reset and backed off. The rtt estimate, retransmit timeouts and the congestion control mechanism were seriously skewed before any data was ever sent. The retransmit timer could expire as many as 3 times before data could be transmitted. The fix was adding another timer just for handling the delay while the route came up, and not letting that delay affect any of the normal rtt timers ⏰✨.

Our experiences with this approach weren't particularly positive, so we tried another one 🦋. We also felt unreliable datagram protocols should be able to use the service without excessive reworking. The alternative still sends the switch control message when a routing lookup finds a controlled route. But now we suspend execution of the thread of control until a response comes back from the switch controller 🛑.

This turned out to be easier to implement in lots of ways 👑. Still, there were two major areas that needed changes outside the routing code. First, we decided that if the switch refused to activate the connection, trying again was pointless. So we changed the routing lookup interface so it could return an error specifying a permanent error condition. The transport layer could then return an appropriate error, like a host unreachable condition 🙃.

The other, more complex issue was the suspension of the thread of execution 🤯. Our operating system, UNICOS, is an ATT System V derivative, and our networking subsystem is based on the BSD tahoe and reno releases. The only way to suspend execution is to sleep. That's fine as long as there's a user context to put to sleep. But it's not a good idea to go to sleep while processing network interrupts, like when forwarding a packet 🚫.

So we used a global flag about whether it's okay for the switch control message code to sleep. If it's necessary to send a message and sleep, the flag must be set, and an error is returned if sleeping isn't allowed. User system calls that might cause a switch control message to be sent set and clear the flag on entry and exit 🎀. We also made it impossible to forward packets on a switch controlled route. We feel that's reasonable, since the overhead of switch control should only be incurred when an application program has made an explicit request to begin transferring data 💅.

One more change: we made sure TCP freed the route it was using upon entering the TIME_WAIT state 🧘. There's no point in holding the circuit open for two minutes in case we need to retransmit the final ack ⏱️. Of course, this assumes an alternate path exists for the peer to retransmit its fin 🤞.

The advantage of building this facility into the kernel is that it allows a fine degree of control over when the switch will and will not need to be activated 👑. Lots of applications that open a data connection, transmit their bulk data, and then close it won't need modifications and will make efficient use of the resource 🙌. It also opens the possibility that apps written to use type-of-service can keep the same network connection for low-bandwidth interactive traffic, change the type-of-service (thus activating the switched connection) for bulk transfers, and then release the switch when they go back to interactive traffic 🔄.

Putting this feature into the kernel also allows strong control over when and how the switched link can be used 🔒, keeps accounting information 🧾, and limits multiple use access to the switched link 🚧.

The disadvantage? Significant kernel modifications are required, and some implementation details can be very difficult to handle 💀.

Switch control libraries

The switch control programs we used were built on a library of simple switch control routines 📚. We didn't alter any standard applications to use this library, though 🙈. We did consider some advantages and disadvantages. On the plus side, it's possible to get a satisfactory degree of switch control without requiring any kernel modifications 💖.

The primary disadvantage is that all applications must be altered and recompiled 😩. That's especially inconvenient when source isn't available 🫠.

Okay so when an app wishes to send data over a circuit-switched connection, it's going to need to select the switched link over other links 🎯. This selection process may need to take place many times, depending on the local network between the source host and the bridge to the circuit switched connection 🌉.

For example, if the kernel routing code is controlling the link, there must be a way to choose a controlled route over another route 🛣️. Further downstream, there must be a way to route packets to the switched link rather than other links 🚀.

This issue has the potential to get super complex, so we avoided as much of that complexity as possible 🙃. Policy routing and local routing across multiple connections are fertile areas for work, and addressing those issues is outside the scope of this work. Instead we opted for simple answers to difficult questions ✨.

First of all, we added no special policies to link accessibility beyond what's already in UNICOS 🔐. We handled local routing issues to the NSC FDDI/T1/T3 routers with routing table manipulation and IP Type-of-Service 🧩.

We came up with three solutions for selecting a routing table entry 🎀. The first possibility is to use the type-of-service bits, which seemed natural to us 💜. We changed the routing table to include type-of-service values associated with routing entries, and the routing lookups would select using the type-of-service. UNICOS already supports a facility to mark connections with a type-of-service value 🙌. A controlled route could be marked with high throughput type-of-service, and an application wishing to transfer bulk data could set the socket for high throughput before making the connection. It could also be possible to change the type-of-service on an existing connection and start using the switched link if one is available ⚡.

Using the type-of-service bits has the advantage that downstream routers can also use this information 🧠. In our demonstration system, the NSC FDDI/T1/T3 routers were configured to transfer packets with high throughput type-of-service over the T3 connection and all others over the T1 connection 🍓.

Another possibility is to take advantage of the multiple addresses of a multi-homed host 🏠. Routing tables could be set up so that packets for one of the addresses get special treatment by traveling over the switched link. The routing table in the source host would have an entry for accessing the switch controller when sending to the high throughput destination address 🗝️.

We also derived a method we call route aliasing 🦋. Route aliasing involves associating extra addresses to a single host. However, rather than the destination being an actual multi-homed host, the alias is known only to the source host and is used as an alternative lookup key 🧚. When an application tries to connect to the alias address, the routing lookup returns an aliased route. The route alias contains the actual address of the host, but because of looking up the special address, the switch is activated ⚡. The alias may also specify a type-of-service value to send in the packets, so downstream routers can properly route the packets to the switched link 🚦.

We realize some may bemoan the waste of the limited Internet address space for aliases 😬. However, only the source host is aware of the alias, and the primary shortage is with Internet network addresses rather than host addresses 💅. In fact, we argue that this is a more efficient use of the already sparse allocation of host addresses available with each network address 🌸.

Future considerations ✨

okay so, we are SO sure that by-request services are gonna matter more and more for certain kinds of users 💅 Lots of data centers offer their high-performance resources over a wide area, and those are gonna be the first to use wide-area circuit-switched networks. It's giving main character energy for the whole concept 🦋 Some folks, like CICNet ([2]), are already super interested in deploying this, and telecom vendors are working on meeting that need too 🌸 BUT ngl there are a lot of issues involved in making this work 😭 We're working on getting others involved in this process 🫶

References

[1] Nicholson, et. al., "High Speed Networking at Cray Research", Computer Communications Review, January 1991.

[2] CICNet DS3 Working Group, "High Performance Applications on CICNet: Impact on Design and Capacity", public report, CICNet, Inc., June 1991.

[3] Young, J., and A. Nicholson, "Dynamically Switched Link Control Protocol", RFC 1307, Cray Research, Inc., March 1992.

Security Considerations

Security issues are not discussed in this memo. 💀 Not gonna lie, this one's a bit of a plot hole, so keep it in mind when you read the rest 🙈

Authors' Addresses

The two authors are both at Cray Research, Inc. 🎀 Here's where you can find them 🧚

Andy Nicholson
Cray Research, Inc.
655F Lone Oak Drive
Eagan, MN 55123

Phone: (612) 452-6650
EMail: droid@cray.com


Jeff Young
Cray Research, Inc.
655F Lone Oak Drive
Eagan, MN 55123

Phone: (612) 452-6650
EMail: jsy@cray.com
📦 where this page comes from