RFC 1152

Workshop report: Internet research steering group workshop on very-high-speed networks

📅 published April 1990, from ietf

Part 1 of 7 💖

RFC 1152 📄✨

Workshop Report: Internet Research Steering Group Workshop on Very-High-Speed Networks

Okay bestie, the title block first 🎀 This is RFC 1152, filed under the Network Working Group. Its author is C. Partridge from BBN Systems and Technologies, and it's dated April 1990. It's a workshop report, so it's informational and not a standard (more on that in a sec 👀). This copy's header doesn't list anything it obsoletes or updates.

Status of this Memo 📜

Okay so this memo is a report on a workshop that the Internet Research Steering Group sponsored 🧠 It's for information only, so it does not specify an Internet standard. Distribution of this memo is unlimited, so share it with your whole group chat 💅

Introduction 🦋

The goal of the workshop was to gather together a small number of leading researchers on high-speed networks, in an environment conducive to lively thinking ✨ The hope is that by having a workshop like this, the IRSG has helped stimulate new or improved research in the area of high-speed networks 🌈

Attendance was limited to fifty people, and attendees had to apply to get in 🎟️ Applications were reviewed by a program committee, which accepted about half of them. A few key individuals were invited directly by the program committee, without applying. The workshop was organized by Dave Clark and Craig Partridge 👑

This workshop report is derived from session writeups by each of the session chairmen, which were then reviewed by the workshop participants 📝

Session 1: Protocol Implementation (David D. Clark, Chair) 🧚

This session was all about what changes might be required in protocols to achieve very high-speed operation 🚀

The session was introduced by David Clark (MIT LCS), who claimed that existing protocols would be sufficient to go at a gigabit per second, if that were the only goal. But literally, proposals for high-speed networks usually include other requirements too, like going long distances, supporting many users, supporting new services such as reserved bandwidth, and so on 🤔 Only by examining the detailed requirements can you understand and compare various proposals for protocols. A variety of techniques have been proposed to let protocols operate at high speeds, ranging from clever implementation all the way to complete relayering of function. Clark asserted that currently even the basic problem to be solved is not clear, let alone the proper approach to the solution 💀

Mats Bjorkman (Uppsala University) described a project that used an outboard protocol processor to support high-speed operation. He asserted that his approach would permit accelerated processing of steady-state sequences of packets 🔥 Van Jacobson (LBL) reported results suggesting that existing protocols can operate at high speeds without the need for outboard processors 😭 He also argued that resource reservation can be integrated into a connectionless protocol such as IP without losing the essence of the connectionless architecture. This is in contrast to the more commonly held belief that full connection setup will be necessary in order to support resource reservation. Jacobson said that he has an experimental IP gateway that supports resource reservation for specific packet sequences today 🦋

Dave Borman (Cray Research) described high-speed execution of TCP on a Cray, where the overhead is most probably the system and I/O architecture rather than the protocol 💅 He believes protocols such as TCP would be suitable for high-speed operation if the windows and sequence spaces were large enough. He reported that the current speed of a TCP transfer between the processors of a Cray Y-MP was over 500 Mbps, which is giving speed 🏎️ Jon Crowcroft (University College London) described the current network projects at UCL. He offered a speculation that congestion could be managed in very high-speed networks by returning to the sender any packets for which transmission capacity was not available 🌸

Dave Feldmeier (Bellcore) reported on the Bellcore participation in the Aurora project, a joint experiment of Bellcore, IBM, MIT, and UPenn. Its goal is installing and evaluating two sorts of switches at gigabit speeds between those four sites 🌐 Bellcore is interested in switch and protocol design, and Feldmeier and his group are designing and implementing a 1 Gbps transport protocol and network interface. The protocol processor will have special support for things such as forward error correction to deal with ATM cell loss in VLSI. A new FEC code and chip design have been developed to run at 1 Gbps, and that's iconic ⭐

Because of the large number of speakers, there was no general discussion after this session 🙈

Session 2: High-Speed Applications (Keith Lantz, Chair) 🎀

okay so this session was all about apps and what they demand from the network 💅 Keith Lantz (Olivetti Research California) kicked things off with the portable office idea. Picture this: you take your work with you wherever you go ✨ You get the same services and the same people whether you're in the same building, at home, or at a distant site like a hotel, or even if you've got a super portable multi-media workstation you can literally carry around 🧳 So "portable" is about portability of access to services, not portability of hardware. Nobody planned it, but every presentation in this session is basically a different angle on the portable office, and I'm obsessed 🦋

Most of Lantz's talk was about desktop teleconferencing 🎥 This is traditional audio/video teleconferencing mixed with workstation-based network computing, so people in different places can collaborate in real time using lots of media (text, graphics, facsimile, audio, and video) plus every computer-based tool they've got, from the office, home, or hotel. It's giving main character energy, but it asks a LOT from the network 😭 The network needs to handle several data streams with wildly different bandwidths (from a few Kbps all the way up to 1 Gbps) while keeping delay generally low. Some streams need minimal jitter (that's isochronous), and everything has to stay synchronized with everything else (multi-channel or media synchronization). Lantz's big concern is that high-speed network researchers aren't paying enough attention to that last part 🙈 A lot of the ATM research assumes channels have independent connection request and burst statistics, and ngl, that is clearly not how desktop teleconferencing works 💀

Lantz also pushed for adaptive protocols 🦋 These are needed when the network's capacity gets exceeded, when you have to interoperate with low-speed networks, or when human factors say the quality of service should change (like bumping a video image's resolution up or down). Adaptive protocols bring two big implications. First, the interface to the network protocols has to be hardware-independent and based only on quality of service. Second, there should be a range of code conversion services available, like converting from one audio encoding scheme to another 🎶 He also highlighted some promising video approaches: variable-rate constant-quality coding, layered or embedded coding, progressive transmission, and (most recently, over at UC-Berkeley) extending structured graphics concepts to video, so the component parts of a video image stay logically separate from production all the way through to presentation 🌈

Next up, Charlie Catlett (National Center for Supercomputing Applications) analyzed a specific science app, a thunderstorm simulation, to figure out its network needs ⛈️ He looked at the data flow and how it connects to the computational algorithms, the supercomputer CPU throughput, the nature and size of the data set, and the network services available (throughput, delay, and so on). Simulation and visualization of results usually go through these steps:

  1. Simulation
  2. Tessellation (transform simulation data into three-dimensional geometric volume descriptions, or polygons)
  3. Rendering (transform polygons into raster image)

For the thunderstorm, simulation and tessellation currently happen on a Cray supercomputer, and the resulting polygons get sent to a Silicon Graphics workstation to be rendered and displayed 🖥️ The simulation spits out data at a rate of between 32 and 128 Mbps, depending on how many Cray-2 processors are working on it. Tessellation output is typically somewhere in the range of 10 to 100 Mbps, depending on how complex the visualization techniques are. The SGI workstation can display 100,000 polygons/sec, which in this example works out to up to 10 frames/sec 🎞️ Analysis tools like tracer particles and two-dimensional slices get used interactively at the workstation, working with polygon sets that were calculated ahead of time.

Looking ahead two to three years, supercomputers are projected to hit 10-30 GFLOPS, and workstations up to 1 GFLOPS with a display rate of 1 million polygons/second 🚀 The supercomputer's simulation data creation rate could climb to several Gbps for this app. With all that extra workstation power, tessellation and rendering can both happen at the workstation itself. Shared window systems will let multiple researchers on the network collaborate on one simulation, and each scientist could use their own visualization techniques while tessellation runs on their own workstation 👯 Further down the road, things like network virtual memory will let the tessellation processes on workstations access variables directly in supercomputer memory. It's so iconic, honestly 👑

Terry Crowley (BBN Systems and Technologies) kept the collaboration theme going, looking at real-time video and audio, shared multimedia workspaces, multimedia and video mail, distributed file systems, scientific visualization, network access to video and image information, transaction processing systems, and moving data and computational results between workstations and supercomputers 💜 In general, these apps help groups work together in three ways: by directly providing communication channels (like real-time video and shared multimedia workspaces), by improving and expanding the kinds of information that can be shared (like multimedia and video mail, or supercomputer data and results), and by reducing replication and the complexity of sharing (like distributed file systems and network access to video and image information).

Crowley also pointed out that actual usage patterns are hard to predict 🤷‍♀️ Real-time video might end up being used for group conferencing, video phone calls, "walking down the hall," or as a long-term shared viewport between remote locations to help build community ties. He flagged two traffic characteristics we can expect: the need to provide multiple data streams to the end user, and the need to synchronize those streams. The streams will include real-time video, access to stored video, shared multimedia workspaces, and access to other multimedia data. A presentation with multiple data streams has to be synchronized so the cross-references between them still make sense, like pointing at something in a shared multimedia workspace while saying a voice request to "delete this and save that" 🙌 Plenty of traffic will be point-to-point, but a significant amount will involve conferences between multiple sites. Honestly, a protocol that provides a multicast capability is critical 📡

Last but not least, Greg Watson (HP) gave an overview of ongoing work at the Hewlett-Packard Bristol lab 🇬🇧 Their take is that while applications for high-speed networks that use supercomputers are the technology drivers, the economic drivers will be apps needing moderate bandwidth (say 10 Mbps) that everyone on the network uses 💸 They're looking into how multimedia workstations can help distributed research teams, which are small teams of people who are geographically dispersed but need to work closely on some area of research 🧑‍🔬 Each workstation has multiple video channels, along with some distributed apps running on personal computers. The bandwidth requirement per workstation is about 40 Mbps, assuming a certain degree of compression on the video channels. Right now the video gets distributed as an analog signal over CATV equipment. Ideally, all of it would be carried over a single, unified wide-area network operating in the one-to-several Gbps range 🌟 They've already built a gigabit network prototype, and they're currently experimenting with uncompressed video carried over the same network as normal data traffic. The experiment is giving big "it works" energy 🫶

Session 3: Lightwave Technology and its Implications (Ira Richer, Chair) 🧚

Bob Kennedy (MIT) opened the session with a talk on network design in an era of excess bandwidth ✨ Kennedy's research is about multi-purpose networks where bandwidth isn't a scarce commodity, networks with bandwidths of tens of terahertz 🤯 He points out that a key challenge in these networks is that electronics can't keep up with fiber speeds. His proposal is to consider all-optical networks (where every signal is optical) with optoelectronic nodes or gateways that can recognize and capture only the traffic destined for them, using time, frequency, or code divisions of the huge bandwidth 🌊 The routing algorithms in such networks would be extremely simple, so you don't have to convert fiber-optics into slower electronic pathways just to do switching. Iconic design choice, no notes 💅

Rich Gitlin (AT&T Bell Labs) gave a talk on issues and opportunities in broadband telecommunications networks, with a focus on the role of fiber optic and photonic technology 💡 He presented a three-level architecture for a broadband telecom network. It's B-ISDN/ATM 150 (Mbps) based and has these parts:

  • Customer premises equipment (PBXs, LANs, multimedia terminals) that accesses the network via a router/gateway.
  • A Network Node, which is a high performance ATM packet switch. It acts as both a LAN-to-LAN interconnect and a packet concentrator for traffic destined for CPE attached to other Network Nodes.
  • A backbone layer that interconnects the NODES via a Digital Cross-Connect System, which provides reconfigurable SONET circuits between the NODES. Using circuits minimizes delay and avoids the need for implementing peak-transmission-rate packet switching 🔌

Within this framework, Gitlin said the most likely places for near-term application of photonics are, apart from pure transport (150 Mbps channels in a 2.4 Gbps SONET system), the Cross-Connect (a Wavelength Division Multiplexed based structure was described) and next-generation LANs that provide Gigabit per second throughputs. Those LANs get there by using multiple fibers, concurrent transmission, and new access mechanisms (such as store and forward) 🌈

Finally, a planned interlocation Bell Labs multimedia gigabit/sec research network, LuckyNet, was described 🍀 It tries to extend many of the concepts above to hit its main goals: provide a gigabit per second capability to a heterogeneous user community, stimulate applications that need Gbps throughput (the initial apps are video conferencing and LAN interconnect), and, to the extent possible, be based on standards so it can interconnect with other Gigabit testbeds. Main character energy for sure 🦋

Session 4: High Speed Networks and the Phone System 📞✨ (David Tennenhouse, Chair)

Okay bestie, David Tennenhouse (MIT) kicked things off with a report on the ATM workshop he hosted two days before this one. 🦋 That report will show up in the proceedings of his workshop, so go check there for the full tea. 🫖

Wally St. John (LANL) came next with the Los Alamos gigabit testbed. 🔥 It's built on the High Performance Parallel Interface (HPPI) and crossbar switch tech. LANL designed its own 16x16 crossbar switch and also evaluated the Network Systems 8x8 crossbar switch. The plan is to expand into the CASA gigabit testbed. The remote sites (San Diego Supercomputer Center, Caltech, and JPL) are set up pretty much like the LANL one. The long-haul link goes from HPPI to/from SONET (using ATM too, if they get it done in time). 💅

Wally also talked through some problems with building an HPPI-SONET gateway: 🧩

  • a) Flow control. HPPI on its own is only readily extensible to 64 km, because of the READY-type flow control used in the physical layer. 😬 The gateway will need larger buffers and independent flow control.
  • b) Error-rate expectations. SONET is only specified to have a 1E-10 BER on a per hop basis. That's inadequate for long links. Those in the know say SONET will be much better, but the designer is stuck with the poor BER in the SONET spec. 💀
  • c) Frame mapping. There are several interesting issues in finding a good mapping from the HPPI packet to the SONET frame. Some of them: which SONET STS levels will be available in what time frame, whether concatenated service will be available, and the error rate issue.

Dan Helman (UCSC) talked about work he's been doing with Darrell Long on interconnecting Internet networks via an ATM B-ISDN network. 🌈 Since network interfaces and packet processing are the expensive parts of high-speed networks, they figure it doesn't make sense to use the ATM backbone only for transmission. It should be used for switching too. So gateways (either shared by a subnet or integrated with fast hosts) are needed to encapsulate or convert conventional protocols to ATM format. The gateways will be responsible for caching connections to recently accessed destinations. Many short-lived, low-bandwidth connections are foreseen (for example, for mail and ftp), so routing in the ATM network (to set up connections) shouldn't be complicated. A form of static routing should be adequate. Connection performance can be monitored by the gateways, and connections get reestablished if performance is unacceptable. All decision making can be done by gateways and route servers at low packet rates, rather than at the high aggregate rate of the ATM network. One complicated issue still to be addressed: how to transparently introduce an ATM backbone alongside the existing Internet. 🧚

Session 5: Distributed Systems 🌸 (David Farber, Chair)

Craig Partridge (BBN Systems and Technologies) started this session by arguing that classic RPC doesn't scale well to gigabit-speed networks. 🚀 The gist: machines keep getting faster, but network round-trip delay stays pretty much the same, because nothing can send faster than the speed of light. ✨ So the effective cost of a simple RPC, measured in instruction cycles spent waiting at the sending machine, becomes extremely high (millions of instruction cycles spent waiting for the reply to an RPC). 😭 Also, the methods people currently use to improve RPC performance, like futures and parallel RPC, don't adequately solve this. Future requests will have to be made much much earlier if they're going to complete by the time they're needed. Parallel RPC allows multiple threads, but it doesn't fix the fact that each individual sequence of RPCs still takes a very long time.

Craig went on to suggest there are at least two possible ways out. 👑 One is to do a lot of caching (wasting bandwidth to keep the CPU fed). The catch is that at some point the cache gets so big that you have to keep it consistent with other systems' caches, and suddenly you're doing synchronization RPCs just to avoid doing normal RPCs (oops!) 🙈. The more promising approach is to consolidate RPCs being sent to the same machine into larger operations, which can be sent as a single transaction, run on the remote machine, and have the result returned. (Craig noted that he's pursuing this approach in his doctoral dissertation at Harvard. 🎓)

Ken Schroder (BBN Systems and Technologies) gave a talk on the challenges of combining gigabit networks with wide-area heterogeneous distributed operating systems. 🧠 Ken feels the key goals of wide-area distributed systems will be to support large-volume data transfers between users of conferencing and similar applications, and to deliver information to a large number of end users sharing services such as satellite image databases. These distributed systems will be motivated by the natural distribution of users, of information, and of expensive special-purpose computer resources.

Ken pointed to three of the key problems that must be addressed at the system level in these environments: how to provide high utilization, how to manage consistency and synchronization in the presence of concurrency and non-determinism, and how to construct scalable system and application services. 💅 Utilization is only key for high-performance applications, where current systems would be limited by costs like repeatedly copying messages, converting data representations, and switching between application and operating system. Concurrency can be used to improve performance, but it's also likely to show up in many programs inadvertently because of distribution. Techniques are needed both to exploit concurrency when it's needed and to limit it when non-determinism can lead to incorrect results. Extensive research on ensuring consistency and resolving resource conflicts has been done in the database area, but distributed scheduling and the need for high availability despite partial system failures introduce special problems that need additional research. Service scalability will be required to support customer needs as the size of the user community grows. It needs attention both to ensure that components don't break when they're subdivided across additional processors to support a larger user population, and to ensure that the performance each user sees can still be affordably maintained as new users are added. 🌈

Dave Cheriton (Stanford) made a bold, sweeping argument that we're making a false dichotomy between distributed operating systems and networks. 🔥 In a gigabit world, he argued, the major resource in the system is the network, and in a normal operating system we'd expect such a critical resource to be managed by the operating system. Or, put another way, the gigabit network distributed operating system should manage the network. Cheriton went on to say that if a gigabit distributed operating system is managing the network, then it's perfectly reasonable to make the network very dumb (but fast) and put the system intelligence in the operating systems on the hosts that form the distributed system. 💜

Jonathan Smith (UPenn) again raised the problem of network delay limiting RPC performance. ✨ Unlike Partridge's earlier talk, Smith argued that the appropriate approach is anticipation or caching. 🦋 He backed this up with a simple cost example. If a system is doing a page fetch between two systems that have a five millisecond round-trip network delay between them, the cost of fetching n pages is:

                         5 msec + (n-1) * 32 usec

Thus the cost of fetching an additional page is only 32 usec, but underfetching and having to make another request to get a page you missed costs 5000 usec. 😱 Based on these arguments, Smith suggested that we re-examine work in virtual memory to see if there are comfortable ways to support distributed virtual memory with anticipation. 🧚

Tommy Joseph (Olivetti), for reasons similar to those of Partridge and Smith, argued that we have to get rid of RPC and give programmers alternative programming paradigms. 🌸 He sketched out ideas for asynchronous paradigms using causal consistency, in which systems ensure that operations happen in the proper order without synchronizing through a single system. 🫶

Session 6: Hosts and Host Interfaces (Gary Delp, Chair) 🖥️✨

okay so Gary Delp (IBM Research) talked through a bunch of issues that show up when network speeds keep climbing on hosts that keep getting more powerful 🚀 here's the list:

  • Media Access: some parts are best handled by dedicated silicon, but other parts are better left to a general-purpose processor 🔧
  • Compression: some forms of compression and expansion might belong on the network interface, but most will be application-specific 🗜️
  • Forward Error Correction: the predicted big packet loss problem is packet drops from internal network congestion, not bit errors, so FEC inside a single packet may not be useful 🤔 but the latency cost of not being able to recover from bit errors is very high 😭 some proposals discussed suggested that FEC among packet groups, with dedicated hardware support, is the way to go 💅
  • Encryption/Decryption: this is a computationally intensive task 🔐 most people agree that if it's done on all traffic, some form of hardware support is helpful. so where does it fit in the protocol stack? 🧐
  • Application Memory Mapping: how much of the host memory structure should be exposed to the network interface? 👀 virtual memory and paging make this issue way more complicated 🌀
  • Communication with Other Channel Controllers: opinions ranged from totally passive network interfaces all the way to interfaces that run major portions of the operating system and bus arbitration code 🎭
  • Blocking/Segmentation: the consensus is that B/S should occur wherever the transport layer is processed 🧩
  • Routing: this is related to communications with other controllers 🔀 a routing-capable interface can reduce the bus requirements by a factor of two 📉
  • Intelligent participation in the host structure as a gateway, router, or bridge 🌉
  • Presentation Layer issues: all of the other overheads can be completely overshadowed by this one if it isn't solved well and integrated into the overall host architecture 💀 this points out the need for some standardization of representation (IEEE floating point, etc.) 🫶

Eric Cooper (CMU) gave us some first-hand experience with Nectar, a high-speed fiber-optic LAN that has been built at Carnegie Mellon 🌸 Nectar is an arbitrary mesh of crossbar switches connected by 100 Mbps fiber-optic links ✨ hosts plug into the crossbar switches through communication processor boards called CABs 🔌 the CAB presents a memory-mapped interface to user processes and off-loads all of the protocol processing from the host 🙌

The preliminary performance numbers are very revealing 👀 latency is currently limited by the number of VME operations needed by the host-to-CAB shared memory interface when sending and receiving a message. the throughput bottleneck is the speed of the VME interface: processes running on the CABs can talk over Nectar at 70 Mbps, but processes on the hosts are limited to about 25 Mbps 😤

Jeff Mogul (DEC Western Research Lab) made some really juicy observations 🍓 off-board protocol processors have been a popular way to connect a CPU to a network, but they're going to be less useful in the future 📉 in the hypothetical workstation of the late 1990s, with a 1000-MIPS CPU and a Gbps LAN, an off-board protocol processor will be of no use at all 💀 the bottleneck won't be the computation needed to implement the protocol. it'll be the cost of moving the packet data into the CPU's cache and the cost of telling the user process that the data is ready 🫠 it takes far longer (hundreds of instruction cycles) to do just the first cache miss (needed to get the packet into the cache) than to run all of the instructions needed to implement IP and TCP (perhaps a hundred instructions) 🤯

so a high-speed network interface for a reasonably-priced system must be designed with this cost structure in mind 💅 it should also get rid of as many CPU interrupts as possible, since interrupts are also very expensive 💸 it makes more sense to let a user process busy-wait on a network-interface flag register than to suspend it and then take an interrupt 🧘‍♀️ the normal CPU scheduling mechanism is more efficient than interrupts if the network interactions are rapid ⚡

David Greaves (Olivetti Research Ltd.) briefly explained why we need a total functionality interface architecture that would let us completely eliminate communication interrupts 🦋 he described the Cambridge high-speed ring as an ATM cell-like interconnect that currently runs at 500-1000 MBaud, and he claims that ATM at that speed is a done deal 🏁 Dave Tennenhouse also commented that ATM at high speeds with parallel processors is not the difficult thing that several others have been claiming 🙄

Bob Beach (Ultra Technologies) opened his talk with the iconic observation that networking could be really fast if we could just get rid of the hosts 😂 he backed that up with illustrations of 80-MByte/second transfers to frame buffers from Crays that drop to half that speed when the transfer is host-to-host 📉 using null network layers and proprietary MAC layers, the Ultra Net system can communicate application-to-application with ISO TP4 as the transport layer at seriously impressive rates 🔥 the key to high-speed host interconnects has been found to be both large packets and large (on the order of one megabyte) channel transfer requests 💜 direct DMA interfaces show much smaller transfer latencies 🧚

Derek McAuley (University Cambridge Computer Laboratory) described the work of the Fairisle project, which is producing an ATM network based on fast packet switches 🌈 a RISC processor (12 MIPS) is used in the host interface to do segmentation/reassembly/demultiplexing 🛠️ line rates of up to 150 Mbps are possible even with this modest processor 👑 Derek has promised that performance and requirement results from this system will be published in the spring 🌷

Bryan Lyles (XEROX PARC) volunteered to give an abbreviated talk in exchange for discussion rights 🎀 he reported that Xerox PARC is interested in ATM technology and wants to install an ATM LAN at the earliest possible opportunity 💖 uses will include applications like video, where guaranteed quality of service (QOS) is required 🎬 ATM technology and the desire for guaranteed QOS place a bunch of new constraints on the host interface 🚧 in particular, they believe they'll be forced towards rate-based congestion control 🏃‍♀️ because of implementation issues and burst control in the ATM switches, senders will be forced to do rate based control on a cell-by-cell basis 🧮

Don Tolmie (Los Alamos National Laboratory) described the High-Performance Parallel Interface (HPPI) of ANSI task group X3T9.3 🦄 the HPPI is a standardized basic building block for implementing, or connecting to, networks at Gbps speeds, whether they're ring, hub, cross-bar, or long-haul based ⭐ the HPPI physical layer operates at 800 or 1600 Mbps over 25-meter twisted-pair copper cables in a point-to-point configuration 🧵 the HPPI physical layer has almost finished the standards process, a companion HPPI data framing standard is under way, and a Fiber Channel standard at comparable speeds is also being developed 🌟 major companies have completed, or are working on, HPPI interfaces for supercomputers, high-end workstations, fiber-optic extenders, and networking components 🏆

The discussion at the end of the session covered a whole range of topics 💬 the appropriateness of outboard protocol processing was questioned 🤨 several people agreed that outboarding on a Cray (or similar cost/performance) machine makes economic sense 💰 Van Jacobson argued that for workstations, a simple memory-mapped network interface that makes packets visible to the host processor may well be the ideal solution 💯

Bryan Lyles reiterated several of his earlier points, insisting that when we talk about host interfaces and how to build them, we should remember we're really talking about process-to-process communication, not CPU-to-CPU communication 🫶 not all processes run on the central CPU, e.g., graphics processors and multimedia 🎨 outboard protocol processing may be a much better choice for these architectures 🦋

this is especially true when we think about how memory/bus bandwidth is often a bottleneck 🚨 when our systems run out of bandwidth, we're forced towards a NUMA model and multiple buses to localize memory traffic 🧠

and finally, because of QOS issues, the receiver must be able to tell the sender how fast it can send 🏎️ throwing away cells (packets) will not work, because unwanted packets will still clog the receiver's switch interface and host interface, and they require processing just to be thrown away 🗑️💀

Session 7: Congestion Control (Scott Shenker, Chair) 💅

okay so the congestion control session had six talks 🫶 the first two were pretty general (new approaches and old myths, iconic 🧚), and then the other four got super specific about packet (or cell) dropping: how to avoid drops, how to make them hurt less for certain apps, a calc of end-to-end throughput when drops are happening, and how rate-based flow control can cut down on buffer usage. thumbnail sketches below, bestie 💖

In the first of the general talks, Scott Shenker (XEROX PARC) explained how ideas from economics can be applied to congestion control 🤑 ngl this is so the way that it makes sense. with economics you can actually ask questions about the goals of congestion control, the minimal feedback needed to hit those goals, and the incentive structure of congestion control. Raj Jain (DEC) then went through eight myths about congestion control in high-speed networks 🙈 Among other points, Raj argued that (1) congestion problems will not become less important when memory, processors, and links become very fast and cheap, (2) window flow control is required along with rate flow control, and (3) source-based controls are required along with router-based control. not me being like "wait, the myth was that it would go away?" 💀

In the first of the more specific talks, Isidro Castineyra (BBN Communications Corporation) did a back-of-the-envelope calc on the effect of cell drops on end-to-end throughput 🧮 at extremely low drop rates, go-back-n and selective retransmission gave pretty similar end-to-end throughput, but at higher drop rates selective retransmission hit way higher throughput. 📈

Next, Tony DeSimone (AT&T) told us why high-speed networks are not just fast low-speed networks 🏎️ if the buffer/window ratio is fixed, the drop rate decreases as the network speed increases. he also showed data that adaptive rate control can greatly decrease buffer utilization, which is lowkey iconic 🦋

Jamal Golestani (Bellcore) then presented his work on stop-and-go queueing ✋ this is a simple stalling algorithm implemented at the switches that guarantees no dropped packets and greatly reduces delay jitter. the algorithm requires prior bandwidth reservation and some flow control on sources, and it is compatible with basic FIFO queues. so clean 🫧

In the last talk, Victor Frost (University of Kansas) discussed how different dropping policies affect the perceived quality of a voice connection 📞 when the source marks the drop priority of cells and the switch drops low priority cells first, the perceived quality is much higher than when cells are dropped randomly. the vibe is "protect the important stuff and everything is fine" 🎀

Session 8: Switch Architectures (Dave Sincoskie, Chair) 🌈

Dave Mills (University of Delaware) kicked things off with work on a project now under way at the University of Delaware 🌸 it's about studying architectures and protocols for a high-speed network and packet switch that can operate into the gigabit regime over distances spanning the country. it's meant for applications with very large, very fast, very bursty traffic, the kind you see in supercomputing, remote sensing, and visualizing apps. the network is assumed to be made of fiber trunks, and the switch architecture is based on a VLSI baseband crossbar design that can be configured for speeds from 25 Mbps to 1 Gbps. the hardware is giving main character energy 👑

Mills' approach involves an externally switched architecture, where the timing and routing of flows between crossbar switches are determined by sequencing tables and counters in high-speed memory local to each crossbar 🧠 the switch program is driven by a reservation-TDMA protocol and a distributed scheduling algorithm running in a co-located, general-purpose processor. the end-to-end customers are free to use any protocol or data format that is consistent with the timing of the network, so they get to pick their own vibe 🫶 his main interest in the early phases of the project is studying appropriate reservation and scheduling algorithms. he expects these algorithms to have a lot in common with the PODA algorithm used in the SATNET and WIDEBAND satellite systems, and with the algorithms being considered for the Multiple Satellite System (MSS). it's all connected, bestie 🛰️

John Robinson (JR, BBN Systems and Technologies) gave a talk called Beyond the Butterfly ✨ which described work on a design for an ATM cell switch, known as MONET. the talk covered strategies for buffering at the input and output interfaces to a switch fabric (crossbar or butterfly). the main idea was that cells should be introduced to the switch fabric in random sequence and to random fabric entry ports, so that persistent traffic patterns don't cause high cell loss in the switch fabric. losses come from contention at output ports or within the switch fabric (in the case of a butterfly), so randomizing helps 🎲 next, the relationship of this work to an earlier design for a large-scale parallel processor, the Monarch, was described. in closing, JR made the claim that this class of switch is realizable in current technology (barely) for operation over SONET OC-48 2.4 Gbps links. barely but make it work 💀🔥

Dave Sincoskie (Bellcore) reported on two topics: recent switch construction at Bellcore, and high-speed processing of ATM cells carrying VC or DG information 💅 recent switch design has resulted in a switch architecture named SUNSHINE, which is a Batcher-banyan switch that uses recirculation and multiple output banyans to resolve contention and increase throughput. a paper on this switch will be published at ISS '90, and it is available upon request from the author 📄 one of the interesting traffic results from simulations of SUNSHINE shows that per-port output queues of up to 1,000 cells (packets) may be necessary for bursty traffic patterns. also, Bill Marcus (at Bellcore) has recently produced Batcher-banyan (32x32) chips that test up to 170Mb/sec per port 🍓

The second point in this talk was that there is little difference in the switching processing of Virtual Circuit (VC) and Datagram (DG) traffic that has previously been broken into ATM cells at the network edge 🧚 the switch needs to do a header translation operation followed by some queueing (not necessarily FIFO). the header translation of VC and DG cells differs mainly in the memory organization of the address translation tables (dense vs. sparse). so basically the same, just organized differently 🫧

the discussion after the presentations seemed to wander off the topic of switching, and went back to some of the source-routing vs. network routing issues discussed earlier in the day 🙈 the group was like "wait, we were talking about switches??" lol

Session 9: Open Mike Night (Craig Partridge, Chair) 🎤

As an experiment, the workshop held an open mike session during the evening of the second day 🌙 participants were invited to speak for up to five minutes on any subject of their choice. the minutes of this session are sketchy, because the chair found himself pre-occupied with keeping speakers roughly within their time limits. honestly the chair deserves an award 🏆

Charlie Catlett (NSCA) showed a film of the thunderstorm simulations he had discussed earlier 🌩️ very cinematic.

Dave Cheriton (Stanford) made a controversial suggestion 😳 that perhaps one could manage congestion in the network simply by using a steep price curve, in which sending large amounts of data costs exponentially more than sending small amounts of data. the idea is that people would only ask for large bandwidth when they really needed it, and they would pay so much for it that we could afford to give it to them. it's a spicy take, but the logic is kinda iconic 🤑

Guru Parulkar (Washington University, St. Louis) argued that the recent discussion on the appropriateness of existing protocols and the need for new protocols (protocol architecture) for gigabit networking lacks the right focus 🎯 the emphasis should be on what the right functionality for gigabit speeds is. the list he gave was simpler per packet processing, a combination of rate and window based flow control, a smart retransmission strategy, appropriate partitioning of work among host cpu+os, off board cpu, and custom hardware, and others. he also pointed out that it is not surprising that existing protocols can be modified to include this functionality. by the same token, it is not surprising that new protocols can be designed which take advantage of lessons from existing protocols and also include other features necessary for gigabit speeds. both things can be true, periodt 💅

Raj Jain (DEC) suggested we look at new ways to measure protocol performance, since our current metrics are insufficiently informative 📊 very relatable, honestly.

Dan Helman (UCSC) asked the group to think more carefully about who exactly the users of the network will be 🧐 large consumers? or many small consumers? that's the question 🫶

Session 10: Miscellaneous Topics (Bob Braden, Chair) 🎀

As its title implies, this session covered a variety of different topics relating to high-speed networking 🌸 it was like a grab bag of goodies, bestie.

Jim Kurose (University of Massachussetts) described his studies of scheduling and discard policies for real-time (constrained delay) traffic ⏰ he showed that by enforcing local deadlines at the switches along the path, it is possible to significantly reduce overall loss for such traffic. since his results depend upon the traffic model assumptions, he ended with a plea for work on traffic models 🙏 he stated that Poisson models can sometimes lead to results that are wrong by many orders of magnitude. yikes 😭

Nachum Shacham (SRI International) discussed the importance of error correction schemes that can recover lost cells 🩹 as an example, he presented a simple scheme based upon longitudinal parity. he also showed a variant, diagonal parity, which allows a single missing cell to be recreated and its position in the stream determined. that's so clever it's giving detective energy 🕵️‍♀️

Two talks concerned high-speed LANs 🌐 Biswanath Muhkerjee (UC Davis) surveyed the various proposals for fair scheduling on unidirectional bus networks, especially those that are distance insensitive, i.e., that can achieve 100% channel utilization independent of the bus length and data rate. he described in particular his own scheme, which he calls p-i persistant. slay 💅

Howard Salwen (Proteon), speaking in place of Mehdi Massehi of IBM Zurich, who was unable to attend, also discussed high-speed LAN technologies 🫶 at 100 Mbps, a token ring has a clear advantage, but at 1 Gbps, the speed of light kills 802.6, for example. 💀 he briefly described Massehi's reservation-based scheme, CRMA (Cyclic-Reservation Multiple-Access).

Finally, Yechiam Yemeni (YY, Columbia University) discussed his work on a protocol silicon compiler 🧚 in order to exploit the potential parallelism, he is planning to use one processor per connection. so many processors, so much potential 🔥

The session closed with a spirited discussion about the relative merits of building an experimental network versus simulating it 🥊 proponents of simulation pointed out the high cost of building a prototype and the limitation on the solution space imposed by a particular hardware realization. proponents of building suggested that artificial traffic can never explore the state space of a network as well as real traffic can, and that an experimental prototype is important for validating simulations. honestly both sides are kinda right, and the debate was everything 🌈

Session 11: Protocol Architectures (Vint Cerf, Chair) ✨

okay so Nick Maxemchuk (AT&T Bell Labs) gave us the tea on circuit switching, virtual circuits, and datagrams 💅 circuits are giving steady, nearly constant rate sources, bestie. Circuit switching locks down resources for the whole service period, and you have to set up the allocation before you can use it. ngl that setup is a lot. On a 1.7 Gbps network with a 3000-mile diameter, the circuit set-up round-trip time eats 10**7 bytes, and teardown could potentially cost the same. So file transfer and facsimile transmission, which are way smaller than that, get absolutely wrecked by the 2*10**7 bytes of wasted management delay 😭 (to be fair, these costs are less dramatic if the allocated bandwidth is less than the maximum possible 🙈)

Virtual circuits allow shared use of bandwidth (multiplexing) when the main source is idle, like in Voice Time Assigned Speech Interpolation. The user just tells the network what they plan to use 🦋

Datagrams (DG) are the move when there's no prior knowledge of use statistics, or when usage is way less than the capacity you'd waste during circuit or virtual circuit set-up. You can adaptively route traffic among equivalent resources, which is so iconic 👑

In gigabit ATMs, the high service speed and decreased cell size make service requests relatively burstier. All of these traits together make DG service super attractive 🔥

Then Maxemchuk described deflection routing. Traffic gets broken into fixed-length units, lets into the network when capacity is free, and gets routed out by any available channel, with preference for the channel on the better path. Lowkey it's similar to the hot potato routing from Paul Baran's 1964 packet switching design 🍓 With one buffer, Maxemchuk hit a theoretical 90% utilization, and larger reassembly buffers give better throughput 🎀

Maxemchuk did not have an answer to the question: how do you make sure empty "slots" are available where needed? Honestly it's like the problem D. Davies at the UK National Physical Laboratory ran into with his isarythmic network design, where a finite number of crates are available for data transport throughout the network 🧚

Guru Parulkar (Washington University, St. Louis) showed a big-picture Internet architecture where part of the system runs at gigabit speeds. In his model, internet, transport, and application protocols all work end to end. The internet functions show up in gateways and the host/net interface, same as in today's Internet. But the internet would also support a new service called a congram, which tries to mix the best parts of soft connection and datagram 💜

In this setup you get a variable grade of service. Users can request congrams (UCON), or the system can set them up internally (Picons) to avoid end-to-end setup latency. The various grades could be requested by asserting different required (desired) levels of error control, throughput, delay, interarrival jitter, and so on. Gateways based on ATM switches, for example, would use packet processors at entry/exit to do internet-specific per-packet processing, which may include fragmentation and reassembly of packets (into and out of ATM cells) ✨

At the transport level, Parulkar argued for protocols that provide application-oriented flow and error control with simple per-packet processing. He also brought up a generalized RPC (GRPC), where code, data, and execution might be variously local or remote from the procedure initiator. GRPC can be implemented using network level virtual storage mechanisms 🌈

Raj Yavatkar (University of Kentucky)'s presentation was based on the idea that processes needing communication service would specify their needs in terms of peak and average data rate, plus burst parameters (frequency and size). Bandwidth for a given flow would be allocated at the effective data rate, which is computed from the flow parameters. The effective data rate sits somewhere between the peak and average data rate, based on the burst parameters. Statistical multiplexing takes up the gap between peak and effective rate when a sudden burst of traffic arrives. Bounds on packet loss rate can be computed for a given set of flow parameters and the corresponding effective data rate 🧠

This presentation sparked a discussion about deliberately disciplining inter-process communication demands to match the requested flow (service) profile. It came up because we often have little information about program behavior and might have trouble estimating the network service requirements of any particular program 🙈

Architectural Discussion ✨

An attempt was made to have a high-level discussion on various architectural questions. The discussion got a variety of opinions:

  1. The Internet would continue to exist in a form similar to its current incarnation, and gateways would be required, at least to interface the existing facilities to the high speed packet switching environment.
  2. Some participants were strongly interested in access to raw (naked ATM) services. That would let users construct their own gigabit nets, at the IP level, at any rate. David Cheriton took this to the extreme: he'd prefer to have control over routing decisions and other behavior of the ATM network.
  3. The speed of light problem (latency, round-trip delay) is not going away and will have serious impact on control of the system. Craig Partridge and Van Jacobson took the optimistic view, feeling that many of the existing network and communications management mechanisms used in the present Internet protocols would suffice, if suitably implemented, at higher speeds. A less rosy view came from David Clark, who observed (as did others) that many transactions would be serviced in much less than one round-trip time, so any end-to-end controls would be largely useless.
  4. For applications requiring fixed, periodic service, reservation of resources seemed reasonably attractive to many participants, as long as the service period dominated the set-up time (round-trip delay) by an appreciable margin.
  5. There was much discussion throughout the workshop of congestion control and flow control. Although these problems were not new, they took on a somewhat newer character in the presence of much higher round-trip delays (measured in bits outstanding). One view is that end-to-end flow control is needed, in any case, to moderate sources sending to limited bandwidth receivers. End-to-end flow control may not, however, be sufficient to protect the interior of the network from congestion problems, so additional, intra-network means are needed to cope with congestion hot spots. Eventually such conditions have to be reflected to the periphery of the network to moderate traffic sources.
  6. There was disagreement on the build or simulate question. One view was in favor of building network components so as to collect and understand live application data. Another view held that without some careful simulation, one might have little idea what to build (for example, Sincoskie's large buffer pool requirement was not apparent until the system was simulated).

Comments from Workshop Evaluation Forms ✨

At the end of the IRSG workshop, we asked attendees to fill out an evaluation form. Of the fifty-one attendees, twenty-nine (56%) turned in a form.

The evaluation form asked attendees to answer two questions:

#1. Do you feel that having attended this workshop will help you in your work on high speed networks during the next year?
#2. What new ideas, questions, or issues, did you feel were brought up in the workshop?

In this section we discuss the answers we got to both questions.

Question #1 ✨

There was a satisfying unanimity of opinion on question #1 🥰 Twenty-four attendees answered yes, often strongly (e.g., Absolutely and very much so). Of the remaining five respondents, three said they expected it to have some effect on their research, and only two said the workshop would have little or no effect.

Some forms had additional notes about why the workshop helped them. Several people mentioned that there was considerable benefit to simply meeting and talking with people they hadn't met before. A few other people noted that the workshop had broadened their perspective or improved their understanding of certain issues. A couple of people noted that they'd heard ideas they thought they could use immediately in their research 💖

Question #2 🎀

Okay so, almost everyone said they'd seen ideas at the conference that were brand new to them 😭🫶

Which new ideas mattered most totally depended on your perspective 🔮 The workshop crowd was picked to cover a huge range of specialties, so people in different lanes were obsessed with different stuff. Still, a few themes popped up on loads of forms:

  1. Limitations of our traffic models. This one showed up in some form on many forms 👀 Most attendees felt we don't really understand how network traffic would behave on a gigabit network. They were worried people might design (or worse, standardize) protocols for high speed networks that would end up inadequate once real traffic hit them 😬 People also asked about closed-loop vs. open-loop traffic models and the effects of different types of service. This worry led several people to push for building a high-speed testbed, so we can actually see some real traffic 🧪✨
  2. Congestion control. Even with those traffic model limitations, respondents thought congestion control, both at switching elements and across the whole network, was going to matter even more than it does today 🚦 The reason is the wider mix of traffic that will show up on gigabit networks. Most forms named at least one of the congestion control talks as containing a new idea. The talks by Victor Frost, Jamal Golestani and Scott Shenker got the most praise 👑 Some attendees were also into ways of keeping the lower-layer switching fabric from getting congested, and they flagged the talks by Robinson and Maxemchuk as worth a look 💜
  3. Effects of fixed delay. The reviews weren't unanimous, but a lot of people had concluded that the biggest problem in gigabit networking isn't bandwidth, it's delay ⏳😭 The workshop poked at this from several angles, and most people listed at least one aspect of fixed delay as a challenging new problem. The questions people brought up include:
    • How to avoid a one round-trip set up delay, for less than one round-trip time's worth of data? 🤔
    • How to recover from error without retransmission (and thus additional network delays)? Several people were intrigued by Nachum Shacham's work on error detection and recovery 🧠
    • Should we use window flow-control or rate-based flow control when delays were long? 🌊
    • Can we modify the idea of remote procedure calls to deal with the fact that delays are relatively long? 📞

A couple of attendees pointed out that while some of these problems look similar to today's, the subtle differences that come from running a network at gigabit speeds made them think the actual solutions would have to be very different from what we do now 🦋 Main character energy for the gigabit era, honestly 💅

Security Considerations 🔒

Security issues are not discussed in this memo. 🙈

Author's Address 💌

Craig Partridge
Bolt Beranek and Newman Inc.
50 Moulton Street
Cambridge, MA 02138

Phone: (617) 873-2459

EMail: craig@BBN.COM
📦 where this page comes from