RFC 2682

Performance Issues in VC-Merge Capable ATM LSRs

📅 published September 1999, from ietf

Performance Issues in VC-Merge Capable ATM LSRs 🎀

okay bestie, RFC 2682 is here and it's giving Performance Issues in VC-Merge Capable ATM LSRs ✨. It's from the Network Working Group. The authors are I. Widjaja (Fujitsu Network Communications) and A. Elwalid (Bell Labs, Lucent Technologies). It's dated September 1999 and sits in the Informational category, so it's sharing info, not a standard 💅. The title block doesn't list anything it obsoletes or updates.

Status of This Memo 🧾

This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.

Copyright (C) The Internet Society (1999). All Rights Reserved.

Abstract 🌸

VC merging lets lots of routes get mapped to the same VC label, so you get a scalable mapping method that can support thousands of edge routers 🤯. The catch is that VC merging needs reassembly buffers, so that cells belonging to different packets headed for the same destination don't interleave with each other 🙈. This document looks at how VC merging affects the extra buffer needed for those reassembly buffers and for other buffers too. The main result: VC merging adds only a minimal amount of buffering compared to non-VC merging 🔥. Even better, the overhead goes down as utilization goes up, or as the traffic gets more bursty 💜.

1.0 Introduction 🦋

okay so recently, several organizations have shown off some pretty radical proposals to overhaul legacy router architectures 👑. The big names are Ipsilon's IP switching [1], Cisco's Tag switching [2], Toshiba's CSR [3], IBM's ARIS [4], and the IETF's MPLS [5]. The details differ from one to the next, but they all share one core concept: map the route information to short, fixed-length labels so that next-hop routers can be found by direct indexing 🧚.

Any layer 2 switching mechanism could in principle be used, but ATM switches in the backbone network are believed to be a super attractive solution, since ATM hardware switches have been extensively studied and are widely available in many different architectures 💅. In this document, we assume layer 2 switching uses ATM technology. So each IP packet may get segmented into multiple 53-byte cells before it gets switched 🍓. Traditionally, AAL 5 has been the encapsulation method in data communications, because it's simple, efficient, and has a powerful error detection mechanism 💖. For the ATM switch to forward incoming cells to the right outputs, the IP route information has to be mapped to ATM labels that live in the VPI and/or VCI fields ✨.

The route information stored semi-permanently in the IP routing table contains the tuple (destination, next-hop router). That route information changes when the network state changes, and that usually happens slowly, except during transient cases 🌈. The word "destination" usually refers to the destination network (or CIDR prefix), but it can easily be generalized to (destination network, QoS), (destination host, QoS), or lots of other granularities. In this document, "destination" can mean any of those or any other possible granularity 🫶.

There are a few ways to map route information to ATM labels. In the simplest form, each source-destination pair is mapped to a unique VC value at a switch 🎀. This is called the non-VC merging case. It lets the receiver easily reassemble cells into their respective packets, since the VC values can be used to tell the senders apart 🙌. The downside: with n sources and destinations, each switch might have to manage O(n^2) VC labels for full-meshed connectivity 😭. For example, with 1,000 sources/destinations, the size of the VC routing table is on the order of 1,000,000 entries 💀. Clearly, this method isn't scalable to large networks 🙈.

The second method is VP merging. Here, the VP labels of cells headed for the same destination get translated to the same outgoing VP value, which cuts down on VP consumption downstream 🦋. For each VP, the VC value identifies the sender, so the receiver can rebuild packets even though cells from different packets are allowed to interleave. Each switch now has to manage O(n) VP labels, which is a huge saving from O(n^2) 🤩. Even so, VP merging is limited to only 4,096 entries at the network-to-network interface, and it needs the VC values for a given VP to be coordinated, which adds more complexity 😮‍💨.

The third method, VC merging, maps incoming VC labels for the same destination to the same outgoing VC label 👑. It's scalable and doesn't have the space constraint problem that VP merging has. With VC merging, cells for the same destination look identical at the output of a switch. So cells belonging to different packets for the same destination can't interleave with each other, or else the receiver won't be able to reassemble the packets 🧩. With VC merging, the boundary between two adjacent packets is marked by the "End-of-Packet" (EOP) marker that AAL 5 uses 🔖.

Cell interleaving could be allowed if we used the AAL 3/4 Message Identifier (MID) field to uniquely identify the sender. But that method has some serious drawbacks 🚫: 1) the MID size may not be enough to identify all senders, 2) the encapsulation method isn't efficient, 3) the CRC capability isn't as powerful as AAL 5's, and 4) AAL 3/4 isn't as widely supported as AAL 5 in data communications 💅.

Before VC merging with no cell interleaving can be crowned the most promising approach, two main issues need to be addressed 👀. First, the feasibility of an ATM switch that can merge VCs needs investigating. Second, there's widespread concern that the extra buffering needed to implement VC merging is excessive, which would make the method impractical 😱.

We're going to dispel those concerns in this document 💖. Through analysis and simulation, we show that the additional buffer requirement for VC merging is minimal for most practical purposes 🎉. Other performance issues, like the additional delay caused by VC merging, are also discussed 🧠.

2.0 A VC-Merge Capable MPLS Switch Architecture 🧚‍♀️

In principle, the reassembly buffers can sit on the input or the output side of a switch 🍒. If they're on the input, the switch fabric has to transfer all cells belonging to a given packet in an atomic manner, since cells aren't allowed to interleave. That means the fabric has to do frame switching, which isn't flexible or desirable when multiple QoSs need to be supported 🙃. On the other hand, if the reassembly buffers are on the output, the switch fabric can forward each cell independently, just like normal ATM switching 🌸. So placing the reassembly buffers at the output makes an output-buffered ATM switch the natural choice 👑.

We consider a generic output-buffered VC-merge capable MPLS switch, with VCI translation performed at the output. Other possible architectures may be adopted too 💫. The switch is made up of a non-blocking cell switch fabric and multiple output modules (OMs), each one associated with an output port 🔌. Each arriving ATM cell gets two fields appended to it: an output port number and an input port number 🏷️. Based on the output port number, the switch fabric forwards each cell to the correct output port, just like in normal ATM switches 🦋.

If VC merging isn't implemented, the OM consists of an output buffer 📦. If VC merging is implemented, the OM contains a number of reassembly buffers (RBs), followed by a merging unit and an output buffer 🧩. Each RB typically corresponds to an incoming VC value. It's important to note that each buffer is a logical buffer, and the idea is that there's a common pool of memory for the reassembly buffers and the output buffer 🫶.

The purpose of the RB is to make sure cells for a given packet don't interleave with other cells that are merged to the same VC 🎀. This mechanism, called store-and-forward at the packet level, works by storing each incoming cell for a given packet at the RB until the last cell of the packet arrives 📥. When the last cell arrives, all the cells in the packet get transferred in an atomic manner to the output buffer, ready for transmission to the next hop 🚀. It's worth pointing out that performing a cut-through mode at the RB is not recommended, since it would waste bandwidth if the subsequent cells get delayed 😬. While a packet is moving to the output buffer, the merging unit translates the incoming VCI to the outgoing VCI 🔄.

To save VC translation table space, different incoming VCIs get merged to the same outgoing VCI during translation, as long as the cells are headed for the same destination 🧠. If all traffic is best-effort, full-merging can be implemented, where all incoming VCs destined for the same destination network are mapped to the same outgoing VC 💖. But if the traffic is made up of multiple classes, it's desirable to implement partial merging instead, where incoming VCs destined for the same (destination network, QoS) are mapped to the same outgoing VC 🌈.

Whether full merging or partial merging is implemented, the output buffer may be a single FIFO buffer or multiple buffers, each one corresponding to a destination network or a (destination network, QoS) pair 🍓. If a single output buffer is used, the switch basically tries to emulate frame switching 🙈. If multiple output buffers are used, VC merging is different from frame switching, because cells of a given packet aren't bound to be transmitted back-to-back 🦋.

Fair queueing can actually be implemented, so cells from their respective output buffers get served according to some QoS requirements 💅. Cell-by-cell scheduling can be implemented with VC merging, whereas only packet-by-packet scheduling can be done with frame switching 🔥. So in summary, VC merging is more flexible than frame switching and supports better QoS control 👑✨.

3.0 Performance Investigation of VC Merging 📊

This section compares the VC-merging switch with the non-VC merging switch 🥊. The non-VC merging switch is analogous to the traditional output-buffered ATM switch, where cells from any packets are allowed to interleave 🌀. Since each cell is a distinct unit of information, the non-VC merging switch is a work-conserving system at the cell level 💪. The VC-merging switch, on the other hand, is non-work conserving, so its performance is always lower than that of the non-VC merging switch 😤. The main goal here is to study how VC merging affects the performance of MPLS switches, such as additional delay and additional buffer, under different traffic conditions 🔍.

In the simulation, the arrival process to each reassembly buffer is an independent ON-OFF process 🎲. Cells within an ON period form a single packet, and during an OFF period, the slots are idle 😴. The ON-OFF process is a general process that can model any traffic process 💜.

3.1 Effect of Utilization on Additional Buffer Requirement ✨

okay so first up, we're checking how utilization changes the extra buffer you need for a given overflow probability 💅 To compare fairly, we run both the VC-merging and non-VC merging cases with an average packet size of 10 cells. We use geometrically distributed packet sizes and packet interarrival times, and the cells of a packet show up contiguously, one right after another. (We'll try other distributions later, don't worry 🫶)

The results are very on brand: the VC-merging switch needs more buffers than the non-VC merging switch, no surprise there. When utilization is low, there can be a lot of incomplete packets sitting in the reassembly buffers at any moment, so storage gets wasted 😭 For example, at utilization 0.3, VC merging needs about 45 extra cells to hit the same overflow probability. But when utilization climbs to 0.9, that extra storage drops to about 30 cells. The reason is that as traffic intensity goes up, the VC-merging system becomes more work-conserving, and it's giving efficiency 🦋

Here's the important part, bestie. ATM switches have to be dimensioned at high utilization (somewhere in the range of 0.8-0.9) so they can survive harsh traffic conditions 🔥 At utilization 0.9, a VC-merge ATM switch needs a buffer of size 976 cells to get an overflow probability of 10^{-5}, while a non-VC merge ATM switch needs a buffer of size 946. That works out to an additional buffer requirement for VC merging of about 3%, which is hardly an extra buffering cost. Literally iconic 👑

3.2 Effect of Packet Size on Additional Buffer Requirement 🎀

Next, we vary the average packet size to see how the buffer requirement changes. We fix utilization at 0.5 and try two average packet sizes, B=10 and B=30. To reach the same overflow probability, VC merging needs about 40 extra cells (or 4 packets) compared to non-VC merging when B=10. When B=30, the extra requirement is about 90 cells (or 3 packets).

So, as expected, the extra buffer requirement in cells goes up as packet size goes up. But in terms of packets, it's roughly constant, which is a very satisfying little plot twist 🌸

3.3 Additional Buffer Overhead Due to Packet Reassembly 🧚

There may be some worry that VC merging needs way too much buffering as the number of reassembly buffers grows. That would happen if the switch gets bigger, or if cells for packets headed to different destinations are allowed to interleave. Spoiler: the worry is unfounded, because buffer sharing actually gets more efficient as the number of reassembly buffers increases 💜

To prove it, we look at the overflow probability for VC merging with several values of reassembly buffers (N), specifically N=4, 8, 16, 32, 64, and 128. Utilization is fixed at 0.8 in each case, and the average packet size is 10. For a given overflow probability, the increase in buffer requirement gets less and less pronounced as N goes up. Past a certain point (N=32), the increase becomes insignificant. The reason is that as N increases, the traffic gets thinned out and eventually approaches a limiting process 🌈

3.4 Effect of Interarrival time Distribution on Additional Buffer 🦋

Now we switch our attention to different traffic processes. First, we keep the same ON period distribution but change the OFF period distribution from geometric to hypergeometric, which has a larger Square Coefficient of Variation (SCV). The SCV is defined as the ratio of the variance to the square of the mean. Here we fix utilization at 0.5. As expected, performance gets worse as the SCV goes up, in both the VC-merging and non-VC merging cases 😬

To hit a buffer overflow probability of 10^{-4}, the additional buffer needed is about 40 cells when SCV=1, 26 cells when SCV=1.5, and 24 cells when SCV=2.6. So VC merging becomes more work-conserving as SCV increases. In short, the more bursty the interarrival times between packets get, the smaller the additional buffer requirement for VC merging becomes. Bursty is the new chill 🫶

3.5 Effect of Internet Packets on Additional Buffer Requirement 🌐

So far the packet size has been modeled as a geometric distribution with a certain parameter. For the rest of this document, we swap in a more realistic packet size distribution. Since the first VC-merge capable ATM switches will probably be deployed in the core network, it makes sense to use the packet size distribution from the Wide Area Network. For that, we rely on the data given in [6]. The data was collected on Feb 10, 1996, in FIX-West network, and it's presented as a probability mass function versus packet size in bytes. Data collected on other dates closely resembles this one.

The distribution looks bi-modal, with two big masses. One is at 40 bytes (about a third of packets), which comes from TCP acknowledgment packets. The other is at 552 bytes (about 22 percent), which comes from Maximum Transmission Unit (MTU) limitations in many routers. Other prominent sizes include 72 bytes (about 4.1 percent), 576 bytes (about 3.6 percent), 44 bytes (about 3 percent), 185 bytes (about 2.7 percent), and 1500 bytes (about 1.5 percent), which comes from the Ethernet MTU. The mean packet size is 257 bytes, and the variance is 84,287 bytes^2. That means the SCV for the Internet packet size is about 1.1 📊

To convert IP packet sizes in bytes into ATM cells, we assume AAL 5 using null encapsulation, where the additional overhead in AAL 5 is 8 bytes long [7]. With null encapsulation, the average packet size comes out to about 6.2 ATM cells.

We then look at the buffer overflow probability against buffer size using the Internet packet size distribution. The OFF period is assumed to be geometrically distributed. We find the same behavior as before, except the buffer requirement drops with Internet packets because the average packet size is smaller 💖

3.6 Effect of Correlated Interarrival Times on Additional Buffer Requirement 🔁

To model correlated interarrival times, we use the DAR(p) process (discrete autoregressive process of order p) [8]. This one has been used to model video traffic (the Star Wars movie) accurately in [9] 🎬 DAR(p) is a p-th order (lag-p) discrete-time Markov chain. The state of the process at time n depends explicitly on the states at times (n-1), ..., and (n-p).

We check the overflow probability for two cases. In the first, the interarrival time between packets is geometric and independent. In the second, the interarrival time is geometric and correlated with the previous one, with a coefficient of correlation equal to 0.9. We use the empirical Internet packet size distribution from the last section, and utilization is fixed at 0.5 in each case. The overflow probability goes up as p goes up, but the additional buffering for VC merging actually goes down as p (or equivalently, the correlation) increases. So you can pretty easily conclude that higher-order correlation or long-range dependence, which shows up in self-similar traffic, will lead to similar qualitative performance 🧠

3.7 Slow Sources 🐢

The discussions up to now have assumed that cells within a packet arrive back-to-back. When traffic shaping is implemented, adjacent cells within the same packet would typically be spaced out by idle slots. We call these "slow sources". Adjacent cells within the same packet can also get perturbed and spaced out as they travel downstream, because cells get merged and split at preceding nodes.

Here, we assume each source transmits at the rate of r_s (0 < r_s < 1), in units of link speed, to the ATM switch. To capture merging and splitting as cells travel through the network, we also assume the cell interarrival time within a packet is randomly perturbed. To model this, we stretch the original ON period by 1/r_s, and flip a Bernoulli coin with parameter r_s during the stretched ON period. In other words, a slot contains a cell with probability r_s, and is idle with probability 1-r_s during the ON period. Doing it this way means the average packet size stays the same as r_s varies. We simulated slow sources on the VC-merge ATM switch using the Internet packet size distribution, with r_s=1 and r_s=0.2. The packet interarrival time is geometrically distributed. Reducing the source rate generally reduces the stress on ATM switches, since the traffic gets smoother 🌷

With VC merging, slow sources also increase the reassembly time. At utilization 0.5, reassembly time is the more dominant effect, so the slow source (with r_s=0.2) needs more buffering than the fast source (with r_s=1). At utilization 0.8, the smoother traffic is the more dominant effect, so the slow source (with r_s=0.2) needs less buffering than the fast source (with r_s=1). This result again has practical consequences for ATM switch design, where buffer dimensioning is done at reasonably high utilization. In that situation, slow sources only help 🙌

3.8 Packet Delay ⏱️

It's super interesting to see how cell reassembly affects packet delay. Here we only consider the delay at one node. End-to-end delays are the subject of ongoing work. We define the delay of a packet as the time between the arrival of the first cell of a packet at the switch and the departure of the last cell of that same packet. We study average packet delay as a function of utilization for both VC-merging and non-VC merging switches, in the case r_s=1 (back-to-back cells in a packet). We use the Internet packet size distribution again, since it's the more realistic scenario, and the interarrival time of packets is geometrically distributed.

Worst-case delay can theoretically differ a lot between VC-merging and non-VC merging. But when we look at average delays, we consistently see the two systems differ by about one average packet time across a wide range of utilization. That difference comes from the average time needed to reassemble a packet ⏳

To see the effect of cell spacing within a packet, we simulate average packet delay again with r_s=0.2. The difference in average delays between VC merging and non-VC merging grows to a few packet times, approximately 20 cells at high utilization. It's worth noting that when a VC-merge capable ATM switch reassembles packets, it's effectively doing the work the receiver would otherwise have to do 🫠 From a practical point of view, an increase of 20 cells translates to about 60 micro seconds at OC-3 link speed. For most applications, this additional delay should be insignificant 💅

4.0 Security Considerations 🔐

There are no security considerations directly related to this document, since the document is concerned with the performance implications of VC merging. There are also no known security considerations as a result of the proposed modification of a legacy ATM LSR to incorporate VC merging 🛡️

5.0 Discussion ✨

okay so this doc was all about how VC merging affects the performance of an ATM LSR 👀 we ran a bunch of different traffic processes to see how VC-merge capable ATM LSRs actually behave, and the big takeaway is giving minimal overhead 💅 compared to non-VC merging, VC merging only needs a tiny bit of extra buffering 🫶

the overhead also goes DOWN as utilization goes UP, or when the traffic gets more bursty, and that's lowkey iconic 😭 switches are dimensioned for high utilization and stressful traffic conditions anyway, so the spot where it matters most is exactly where it's least of a problem 🔥

we only looked at FIFO scheduling for the output buffer. but after checking out slow sources, we think fair queueing won't have a significant impact on how much extra buffering you need 🦋 others may wish to investigate this further, so if you're curious, go for it 🔍

6.0 Acknowledgement 🙌

big bestie shoutout time 💜 the authors thank Debasis Mitra for his penetrating questions during the internal talks and discussions 🧚

7.0 References 📚

here's the bibliography, aka the receipts for everything above 🧾

[1] P. Newman, Tom Lyon and G. Minshall, "Flow Labelled IP:
    Connectionless ATM Under IP", in Proceedings of INFOCOM'96, San-
    Francisco, April 1996.

[2] Rekhter,Y., Davie, B., Katz, D., Rosen, E. and G. Swallow, "Cisco
    Systems' Tag Switching Architecture Overview", RFC 2105, February
    1997.

[3] Katsube, Y., Nagami, K. and H. Esaki, "Toshiba's Router
    Architecture Extensions for ATM: Overview", RFC 2098, February
    1997.

[4] A. Viswanathan, N. Feldman, R. Boivie and R. Woundy, "ARIS:
    Aggregate Route-Based IP Switching", Work in Progress.

[5] R. Callon, P. Doolan, N. Feldman, A. Fredette, G. Swallow and A.
    Viswanathan, "A Framework for Multiprotocol Label Switching",
    Work in Progress.

[6] WAN Packet Size Distribution,
    http://www.nlanr.net/NA/Learn/packetsizes.html.

[7] Heinanen, J., "Multiprotocol Encapsulation over ATM Adaptation
    Layer 5", RFC 1483, July 1993.

[8] P. Jacobs and P. Lewis, "Discrete Time Series Generated by
    Mixtures III:  Autoregressive Processes (DAR(p))", Technical
    Report NPS55-78-022, Naval Postgraduate School, 1978.

[9] B.K. Ryu and A. Elwalid, "The Importance of Long-Range Dependence
    of VBR Video Traffic in ATM Traffic Engineering", ACM SigComm'96,
    Stanford, CA, pp. 3-14, August 1996.

Authors' Addresses 🎀

these are the people who made it happen, so here's where to find them 💌

Indra Widjaja
Fujitsu Network Communications
Two Blue Hill Plaza
Pearl River, NY 10965, USA

Phone: 914 731-2244
EMail: indra.widjaja@fnc.fujitsu.com

Anwar Elwalid
Bell Labs, Lucent Technologies
600 Mountain Ave, Rm 2C-324
Murray Hill, NJ 07974, USA

Phone: 908 582-7589
EMail: anwar@lucent.com

Copyright (C) The Internet Society (1999). All Rights Reserved. 💖

okay the legal part, but make it cute 🎀 you're allowed to copy and share this document, and to make derivative works that comment on it, explain it, or help people implement it. you can publish and distribute them in whole or in part with no real restrictions 🦋 the only catch is that the copyright notice and this permission paragraph have to stay on every copy and derivative work 🔒

the document itself is a no-touchy zone 🙈 you can't modify it in any way, like removing the copyright notice or the references to the Internet Society or other Internet organizations. the exceptions are when you're developing Internet standards, in which case the copyright procedures in the Internet Standards process have to be followed, or when you're translating it into a language other than English 🌍

these permissions are perpetual, so they won't be revoked by the Internet Society or its successors or assigns ✨

and the disclaimer, which is serious business 💀

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 🌈

Funding for the RFC Editor function is currently provided by the Internet Society. 👑

📦 where this page comes from