RFC 5390
Requirements for Management of Overload in the Session Initiation Protocol
Requirements for Management of Overload in the Session Initiation Protocol 🎀
Network Working Group · Request for Comments: 5390 · Category: Informational · December 2008 🌸
Author: J. Rosenberg, Cisco 💖
Obsoletes / Updates: the title block doesn't list anything this one obsoletes or updates 🦋
Status of This Memo ✨
This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. 🫶
Copyright Notice 💜
Copyright (c) 2008 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. 📜
Abstract 🔥
Okay so overload happens in Session Initiation Protocol (SIP) networks when proxies and user agents don't have enough resources to finish processing a request. SIP has a little bit of help for this through its 503 response code, which tells an upstream element that it's overloaded. But ngl, a bunch of problems with that mechanism have been spotted. This document sums up the problems with the existing 503 mechanism, and gives some requirements for a fix. 💅
Table of Contents 🌈
1. Introduction ....................................................2
2. Causes of Overload ..............................................2
3. Terminology .....................................................4
4. Current SIP Mechanisms ..........................................4
5. Problems with the Mechanism .....................................5
5.1. Load Amplification .........................................5
5.2. Underutilization ...........................................9
5.3. The Off/On Retry-After Problem .............................9
5.4. Ambiguous Usages ..........................................10
6. Solution Requirements ..........................................10
7. Security Considerations ........................................13
8. Acknowledgements ...............................................13
9. References .....................................................14
9.1. Normative Reference .......................................14
9.2. Informative References ....................................141. Introduction 🧚
Overload happens in Session Initiation Protocol (SIP) [RFC3261] networks when proxies and user agents don't have enough resources to finish processing a request or a response. SIP gives you limited overload handling through its 503 response code. That code lets a server tell an upstream element that it's overloaded. But literally so many problems have been found with this mechanism. 😭
This doc is about the general SIP overload problem and looks at the current SIP tools for dealing with it. Then it walks through some of the problems with those tools. Finally, it gives a set of requirements for fixing them. 💖
2. Causes of Overload 🍓
Overload happens when an element, like a SIP user agent or proxy, doesn't have enough resources to successfully process all the traffic it's getting. "Resources" covers everything the element uses to process a request, including CPU processing, memory, I/O, or disk resources. It can also include external resources like a database or DNS server. In that case, the CPU, processing, memory, I/O, and disk resources of those servers are effectively part of the logical element processing the request. Overload can happen for lots of reasons, including: ✨
- Poor Capacity Planning: SIP networks need enough servers, hardware, disks, and so on, to serve the subscribers they're expected to have. Capacity planning is the process of figuring out those needs. It's based on the number of expected subscribers and the types of flows they're expected to use. If this work isn't done properly, the network might not have enough capacity for predictable usage, including regular usage and predictably high usage (like high voice calling volumes on Mother's Day). 🌸
- Dependency Failures: A SIP element can get overloaded because a resource it depends on has failed or become overloaded, which greatly cuts the element's logical capacity. In these cases, even minimal traffic might push the server into overload. Examples of these dependency overloads include DNS servers, databases, disks, and network interfaces. 💀
- Component Failures: A SIP element can get overloaded when it's part of a cluster of servers that each share the traffic load, and one or more of the other cluster members fail. In that case, the remaining elements take over the work of the failed ones. Normally, capacity planning accounts for failures like this, and servers usually run with enough spare capacity to handle another element failing. But unusual failure conditions can make many elements fail at once. This is often the case with software failures, where a bad packet or bad database entry hits the same bug in a bunch of elements in a cluster. 🔥
- Avalanche Restart: One of the most troubling sources of overload is avalanche restart. This happens when a large number of clients all try to connect to the network with a SIP registration at the same time. Avalanche restart can be caused by several events. One is the "Manhattan Reboots" scenario, where there's a power failure in a big metropolitan area, like Manhattan. When power comes back, all the SIP phones, whether they're in PCs or standalone devices, power on and start booting at the same time. They all then connect to the network and register, causing a flood of SIP REGISTER messages. 😱
Another cause of avalanche restart is the failure of a large network connection, for example, the access router for an enterprise. When it fails, SIP clients detect the failure quickly using the mechanisms in [OUTBOUND]. When connectivity is restored, that's detected too, and clients re-REGISTER, all within a short time window. Another source of avalanche restart is the failure of a proxy server. If clients had all connected to the server with TCP, its failure gets detected, followed by reconnection and re-registration to another server. Note that [OUTBOUND] does provide some remedies for this case. 🫣
- Flash Crowds: A flash crowd happens when an extremely large number of users all try to make a call at once. One example is a TV commercial that advertises a number to call to get a free gift. If the gift is compelling and lots of people see the ad, lots of calls can be made to the same number at once. This can send the system into overload. 📞
- Denial of Service (DoS) Attacks: An attacker who wants to disrupt service in the network can launch a large amount of traffic at a target server. This can come from a central source of traffic or through a distributed DoS attack. In every case, the volume of traffic far exceeds the server's capacity, sending the system into overload. 🚨
Unfortunately, overload tends to compound itself, which is giving major chaos energy 🌀 When a network goes into overload, this can often cause failures in the elements trying to process the traffic. That puts even more load on the remaining elements. Also, during overload, the overall capacity of the functional elements goes down, since so much of their resources get spent just rejecting or handling load they can't actually process. On top of that, overload tends to cause SIP messages to be delayed or lost, which triggers retransmissions, and those add even more work to the network. This compounding factor can produce substantial multipliers on the load in the system. Honestly, in the case of UDP, with as many as seven retransmits of an INVITE request before timeout, overload can multiply the already-heavy message volume by as much as seven! 😵💫
3. Terminology 🎀
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 📚
4. Current SIP Mechanisms 💅
SIP gives you very basic overload support. It defines the 503 response code, which an overloaded element sends. RFC 3261 defines it like this: 🦋
The server is temporarily unable to process the request due to
a temporary overloading or maintenance of the server. The
server MAY indicate when the client should retry the request in
a Retry-After header field. If no Retry-After is given, the
client MUST act as if it had received a 500 (Server Internal
Error) response.
A client (proxy or UAC) receiving a 503 (Service Unavailable)
SHOULD attempt to forward the request to an alternate server.
It SHOULD NOT forward any other requests to that server for the
duration specified in the Retry-After header field, if present.
Servers MAY refuse the connection or drop the request instead of
responding with 503 (Service Unavailable).The goal is to move the work of the overloaded server over to another server so the request can still get processed. The Retry-After header field, when it's present, lets a server tell an upstream element to back off for a while, so the overloaded server can work through its backlog. 🌷
RFC 3261 also tells proxies to not forward 503 responses upstream, at SHOULD NOT strength. The point is to stop the upstream server from mistakenly concluding that the proxy is overloaded, when the real problem was an element further downstream. 🙈
5. Problems with the Mechanism 💔
On the surface, the 503 mechanism seems workable. Unfortunately, it has run into lots of problems in real deployments. Those problems are described here. 🫠
Part 1 of 3 🎀 Part 2 picks up right here with Section 5's subsections 🫶
5.1. Load Amplification ✨
okay so the big problem with the 503 mechanism is that it can amplify the load in the network when everything is already overloaded 😭 which just makes the problem worse and opens up a very real chance of congestive collapse 💀. Look at this setup 👀
+------+
> | |
/ | S1 |
/ | |
/ +------+
/
/
/
/
+------+ / +------+
--------> | |/ | |
| P1 |---------> | S2 |
--------> | |\ | |
+------+ \ +------+
\
\
\
\
\
\ +------+
\ | |
> | S3 |
| |
+------+
Figure 1so proxy P1 gets SIP requests from a bunch of sources and its only job is being a load balancer 🎀 It proxies requests to S1, S2, and S3 for processing. Then the input load climbs until all three servers are overloaded 😩 S1 gets its next request and generates a 503, but since it's loaded, making that 503 takes a bit of time ⏳ If SIP is running over UDP, that delay can cause request retransmissions, which add even more work for S1 🙃 Even with TCP, if the server is loaded and the kernel can't send TCP acknowledgements fast enough, TCP retransmits can happen too 🥲 When P1 gets the 503, it retries the request on S2. S2 is overloaded as well, so it eventually sends a 503 too, but in the meantime it might get hit with retransmits as well 💅 Then P1 tries S3, which also eventually rejects it with a 503. Iconic chaos 😭
So the processing of this request, which ended in failure, involved four SIP transactions: client to P1, P1 to S1, P1 to S2, and P1 to S3 🍓 Each of those could have included a bunch of retransmissions, up to seven when UDP is in play 🫠 Under normal, unloaded conditions, one request from a client produces one request (to S1, S2, or S3) and two responses (S1 back to P1, then P1 back to the client) ✨ But when the network is overloaded, one client request, before it times out, can generate as many as 18 requests and as many responses when UDP is used 🤯 It's a little better with TCP (or any reliable transport, really), but even if not a single TCP segment got retransmitted, one client request can still generate three requests and four responses 🙈 Each server had to spend resources handling all of those messages, so more messages and more work got dumped into the network right when the elements were already drowning 😭 The 503 mechanism works well when one single element is overloaded. But when the problem is overall network load, it actually creates more messages and more work for all the servers, and the request gets rejected anyway 💀
It gets even messier if you look at the proxies upstream from P1, like in Figure 2 🌈
+------+
> | | <
/ | S1 | \\
/ | | \\
/ +------+ \\
/ \
/ \\
/ \\
/ \
+------+ / +------+ +------+
| | / | | | |
| P1 | ---------> | S2 |<----------| P2 |
| | \ | | | |
+------+ \ +------+ +------+
^ \ / ^
\ \ // /
\ \ // /
\ \ // /
\ \ / /
\ \ +------+ // /
\ \ | | // /
\ > | S3 | < /
\ | | /
\ +------+ /
\ /
\ /
\ /
\ /
\ /
\ /
\ /
\ /
+------+
| |
| PA |
| |
+------+
^ ^
| |
| |
Figure 2so here proxy PA receives requests and sends them on to P1 or P2 💖 Both P1 and P2 load balance across S1 through S3. Say again that S1 through S3 are all overloaded, and a request shows up at PA, which tries P1 first 🦋 P1 tries S1, then S2, then S3, and every one of those transactions can trigger a bunch of request retransmits if UDP is being used 😵💫 Since P1 can't eventually process the request, it rejects it. But because all of its downstream buddies are busy too, it decides to send a 503 🥀 That 503 goes up to PA, which then tries P2, and P2 goes through S1 through S3 all over again, which ends in another 503 💀 So in this case we've doubled the number of SIP transactions and the overall work in the network compared to the previous one 🙃 The real issue is that P1 knew S1 through S3 were overloaded, but that info never got passed back to PA and on to P2, so P2 just retries S1 through S3 again, no thoughts head empty 🧠💨
5.2. Underutilization ✨
okay this one is lowkey wild 😭 there are actual deployments where the network capacity got way reduced because of the overload mechanism itself 🫠 Let's look at Figure 1 again 👀 The issue is that RFC 3261 isn't clear on the scope of a 503 🤷♀️ When P1 receives one, does the proxy stop sending requests to that IP address? To the hostname? To the URI? Some implementations picked the hostname as the scope 🎀 Now picture a hostname for a URI that points to an SRV record in the DNS, and that SRV record maps to a whole cluster of downstream servers (S1, S2, and S3 in our example) 🌸 If just one of them sends a 503, the proxy thinks the entire cluster is overloaded. So proxy P1 stops sending any traffic to any element in the cluster 😱 even though some of those elements are totally underutilized 💅 Not the vibe.
5.3. The Off/On Retry-After Problem ✨
so the Retry-After mechanism lets a server tell an upstream element to stop sending it traffic for a while 🦋 The work that would have gone to that server gets sent to another server instead. The catch is that it's an all-or-nothing deal 🙃 A server can turn off all traffic toward it, or none at all. Nothing in between 💀 This tends to cause super oscillatory behavior even under mild overload 🎢
Let's say proxy P1 is balancing requests between two servers, S1 and S2 💜 The input load just hits the point where both S1 and S2 are at 100% capacity. A request arrives at P1 and gets sent to S1. S1 rejects it with a 503 and decides to use Retry-After to clear its backlog 🧹 So P1 stops sending all traffic to S1. Now S2 is getting everything, but it's seriously overloaded, sitting at 200% capacity!! 😭 It rejects a request with a 503 and a Retry-After, which now forces P1 to reject all traffic until S1's Retry-After timer runs out 🫥 Once that happens, all the load gets shunted back to S1, which hits overload again, and the whole cycle repeats 🔁 chaotic loop, periodt.
It's important to note that this problem only shows up for servers where a small number of upstream elements are sending them traffic, which is the case in these examples ✨ If a proxy is accessed by a large number of clients, each sending a small amount of traffic, the 503 mechanism with Retry-After works pretty well when it's used with a subset of the clients 🌈 That's because spreading the 503s out across the clients gives the proxy more fine-grained control over how much work it receives 🫶
5.4. Ambiguous Usages ✨
okay so the specific situations where a server is supposed to send a 503 are unclear 🙈 The result is that implementations send 503 for a bunch of reasons, and only some of them actually have to do with real overload 😭 For example, RFC 3398 [RFC3398], which specifies interworking from SIP to ISDN User Part (ISUP), defines using 503 when the gateway gets certain ISUP cause codes from downstream switches 📞 In those cases, the gateway has plenty of capacity. It's just that this specific request couldn't be processed because of a downstream problem 🥹 All the later requests might succeed if they take a different route in the Public Switched Telephone Network (PSTN) 🛤️
That causes two problems 💀 First, during overload, it makes everything worse, because it feeds additional 503s into the system and creates even more work when things are already overloaded 🫠 Second, it gets really hard for an upstream element to know whether it should retry when it gets a 503 🤷♀️ There are classes of failure where trying another server won't help, since the failure happened because a shared downstream resource is unavailable. For example, if servers S1 and S2 share a database and the database fails, a request sent to S1 will get a 503, but retrying on S2 won't help since the same database is unavailable too 🗄️😭 bestie, that's the one 🫶
6. Solution Requirements ✨
okay so this section is where the authors lay out the wishlist 🧚 for an overload control mechanism for SIP, and it's giving main character energy. every single one of these is a request that the solution has to hit, and they're literally addressing all those problems from earlier. let's gooo 💅
REQ 1: The overload mechanism shall strive to maintain the overall useful throughput (taking into consideration the quality-of-service needs of the using applications) of a SIP server at reasonable levels, even when the incoming load on the network is far in excess of its capacity. The overall throughput under load is the ultimate measure of the value of an overload control mechanism. 👑 this is the big one bestie, the whole point is keeping the server useful even when it's getting absolutely slammed 🔥
REQ 2: When a single network element fails, goes into overload, or suffers from reduced processing capacity, the mechanism should strive to limit the impact of this on other elements in the network. This helps to prevent a small-scale failure from becoming a widespread outage. 🦋 no domino effect, periodt 💀
REQ 3: The mechanism should seek to minimize the amount of configuration required in order to work. For example, it is better to avoid needing to configure a server with its SIP message throughput, as these kinds of quantities are hard to determine. 🎀 nobody wants to sit there guessing their message throughput, that's just not it 🙈
REQ 4: The mechanism must be capable of dealing with elements that do not support it, so that a network can consist of a mix of elements that do and don't support it. In other words, the mechanism should not work only in environments where all elements support it. It is reasonable to assume that it works better in such environments, of course. Ideally, there should be incremental improvements in overall network throughput as increasing numbers of elements in the network support the mechanism. 🌈 mixed networks are the real world, so it has to play nice with everyone, and the more people join the party the better it gets ✨
REQ 5: The mechanism should not assume that it will only be deployed in environments with completely trusted elements. It should seek to operate as effectively as possible in environments where other elements are malicious; this includes preventing malicious elements from obtaining more than a fair share of service. 🫶 not everyone in the group chat is trustworthy, so it has to protect against the sus ones 💀
REQ 6: When overload is signaled by means of a specific message, the message must clearly indicate that it is being sent because of overload, as opposed to other, non overload-based failure conditions. This requirement is meant to avoid some of the problems that have arisen from the reuse of the 503 response code for multiple purposes. Of course, overload is also signaled by lack of response to requests. This requirement applies only to explicit overload signals. 😭 the 503 mess is literally why this is here, nobody should be guessing what a 503 means 🙄
REQ 7: The mechanism shall provide a way for an element to throttle the amount of traffic it receives from an upstream element. This throttling shall be graded so that it is not all-or-nothing as with the current 503 mechanism. This recognizes the fact that "overload" is not a binary state and that there are degrees of overload. 🍓 overloaded isn't an on/off switch, bestie, there are levels to it, and the throttle needs to reflect that 🎚️
REQ 8: The mechanism shall ensure that, when a request was not processed successfully due to overload (or failure) of a downstream element, the request will not be retried on another element that is also overloaded or whose status is unknown. This requirement derives from REQ 1. 🧚 don't send the request from one drowning server to another drowning server, that's just chaos 🌊
REQ 9: That a request has been rejected from an overloaded element shall not unduly restrict the ability of that request to be submitted to and processed by an element that is not overloaded. This requirement derives from REQ 1. 💖 getting rejected by one server shouldn't lock the request out of every other server, it still deserves a chance somewhere healthy ✨
REQ 10: The mechanism should support servers that receive requests from a large number of different upstream elements, where the set of upstream elements is not enumerable. 🌸 for when you have a crowd and you can't even list everyone in it 🫠
REQ 11: The mechanism should support servers that receive requests from a finite set of upstream elements, where the set of upstream elements is enumerable. 🍓 and for when you have a fixed friend list you can actually count, also supported 💅
REQ 12: The mechanism should work between servers in different domains. 🌍 the mechanism shouldn't stop at the edge of one network, it should work across the whole map 🗺️
REQ 13: The mechanism must not dictate a specific algorithm for prioritizing the processing of work within a proxy during times of overload. It must permit a proxy to prioritize requests based on any local policy, so that certain ones (such as a call for emergency services or a call with a specific value of the Resource-Priority header field [RFC4412]) are given preferential treatment, such as not being dropped, being given additional retransmission, or being processed ahead of others. 🚨 the proxy gets to pick its own VIP list, but emergency calls and Resource-Priority calls can jump the line 👑 iykyk
REQ 14: The mechanism should provide unambiguous directions to clients on when they should retry a request and when they should not. This especially applies to TCP connection establishment and SIP registrations, in order to mitigate against avalanche restart. 🦋 no more mixed signals about when to retry, clients deserve clear instructions, and it stops the whole group from stampeding back at once 💀
REQ 15: In cases where a network element fails, is so overloaded that it cannot process messages, or cannot communicate due to a network failure or network partition, it will not be able to provide explicit indications of the nature of the failure or its levels of congestion. The mechanism must properly function in these cases. 🥺 when a server just goes silent you still have to handle it, no excuses 🫶
REQ 16: The mechanism should attempt to minimize the overhead of the overload control messaging. 🎀 the signals themselves shouldn't add more clutter to the network, keep it light and breezy 🪽
REQ 17: The overload mechanism must not provide an avenue for malicious attack, including DoS and DDoS attacks. 🛡️ the fix can't become the weapon, absolutely not 😤
REQ 18: The overload mechanism should be unambiguous about whether a load indication applies to a specific IP address, host, or URI, so that an upstream element can determine the load of the entity to which a request is to be sent. 📍 the load info needs to be clear about who it's actually about, no mixing up the tea ☕
REQ 19: The specification for the overload mechanism should give guidance on which message types might be desirable to process over others during times of overload, based on SIP-specific considerations. For example, it may be more beneficial to process a SUBSCRIBE refresh with Expires of zero than a SUBSCRIBE refresh with a non-zero expiration (since the former reduces the overall amount of load on the element), or to process re-INVITEs over new INVITEs. 🧠 some messages are way more useful to keep than others when things get hectic, like a SUBSCRIBE with Expires of zero that actually lightens the load, so the spec should give some guidance 💜
REQ 20: In a mixed environment of elements that do and do not implement the overload mechanism, no disproportionate benefit shall accrue to the users or operators of the elements that do not implement the mechanism. 🙅♀️ no free rides, the ones who don't implement it can't get a better deal than the ones who do 💅
REQ 21: The overload mechanism should ensure that the system remains stable. When the offered load drops from above the overall capacity of the network to below the overall capacity, the throughput should stabilize and become equal to the offered load. 🌊 once the rush is over, everything should settle back down and match what's actually coming in ✨
REQ 22: It must be possible to disable the reporting of load information towards upstream targets based on the identity of those targets. This allows a domain administrator who considers the load of their elements to be sensitive information, to restrict access to that information. Of course, in such cases, there is no expectation that the overload mechanism itself will help prevent overload from that upstream target. 🔒 if your load stats are private, you should be able to keep them that way, but just know the mechanism can't protect you from that particular upstream if you do 🙈
REQ 23: It must be possible for the overload mechanism to work in cases where there is a load balancer in front of a farm of proxies. 🧚♀️ load balancers are part of the squad too, so the mechanism has to work with them standing in front 💖
7. Security Considerations 🔐
Like all protocol mechanisms, a solution for overload handling must prevent against malicious inside and outside attacks. This document includes requirements for such security functions. 😤 literally every protocol has to guard against the bad guys, inside and out, and this doc has the requirements for that 🛡️
⚠️ Any mechanism that improves the behavior of SIP elements under load will result in more predictable performance in the face of application-layer denial-of-service attacks. 🚨 this is kind of a big deal bestie, making things work better under load also makes them way harder to knock over with app-layer DoS attacks 💀
8. Acknowledgements 🫶
The author would like to thank Steve Mayer, Mouli Chandramouli, Robert Whent, Mark Perkins, Joe Stone, Vijay Gurbani, Steve Norreys, Volker Hilt, Spencer Dawkins, Matt Mathis, Juergen Schoenwaelder, and Dale Worley for their contributions to this document. 💜 huge thank you to this whole crew, the doc is better because of y'all 🎀
9. References 📚
9.1. Normative Reference
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.9.2. Informative References
[OUTBOUND] Jennings, C. and R. Mahy, "Managing Client Initiated
Connections in the Session Initiation Protocol (SIP)",
Work in Progress, October 2008.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002.
[RFC3398] Camarillo, G., Roach, A., Peterson, J., and L. Ong,
"Integrated Services Digital Network (ISDN) User Part
(ISUP) to Session Initiation Protocol (SIP) Mapping",
RFC 3398, December 2002.
[RFC4412] Schulzrinne, H. and J. Polk, "Communications Resource
Priority for the Session Initiation Protocol (SIP)",
RFC 4412, February 2006.Author's Address 📮
Jonathan Rosenberg
Cisco
Edison, NJ
US
EMail: jdrosen@cisco.com
URI: http://www.jdrosen.net📦 where this page comes from
- ietf: rfc5390 2008-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms