RFC 7502
Methodology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic Session Setup and Registration
RFC 7502 💖
Methodology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic Session Setup and Registration
Okay bestie, this one is from the IETF (Internet Engineering Task Force) and it's dated April 2015. It's Category: Informational 🎀, so it's published for information and isn't a Standards Track spec. Its ISSN is 2070-1721. The authors are C. Davids from the Illinois Institute of Technology, V. Gurbani from Bell Laboratories, Alcatel-Lucent, and S. Poretsky from Allot Communications. The title block doesn't list any RFCs that this one obsoletes or updates, so it stands on its own ✨.
Abstract 🎀
So this doc gives you a methodology for benchmarking how well SIP devices perform 💅. The terminology lives in a companion doc, RFC 7501. Together, the two docs let you get benchmarks and compare different kinds of devices, like SIP Proxy Servers, Registrars, and Session Border Controllers 🦋. Here "performance" means how much SIP messaging the Device Under Test (DUT) can handle. Media streams only show up to study how they affect the signaling 🎶. The whole goal is a normalized set of tests so you can objectively compare how much SIP devices can handle 📊. Test setup parameters and a method are needed because SIP allows a wild range of configurations and operating conditions that can change the benchmark numbers 🙈.
Status of This Memo 💜
This doc is not an Internet Standards Track spec. It's published for information only 🌈.
It's a product of the IETF, and it represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 if you want the details 🧚.
You can find the current status, any errata, and how to send feedback at http://www.rfc-editor.org/info/rfc7502 🔗.
Copyright Notice 🌸
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved ©️.
This doc is covered by BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can read at http://trustee.ietf.org/license-info as they were on the publication date. Please actually read those, because they spell out your rights and restrictions for this doc 📜. Any Code Components pulled out of this doc have to include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Simplified BSD License describes 🫶.
Table of Contents 📑
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Benchmarking Topologies . . . . . . . . . . . . . . . . . . . 5
4. Test Setup Parameters . . . . . . . . . . . . . . . . . . . . 7
4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 7
4.2. Connection-Oriented Transport Management . . . . . . . . 7
4.3. Signaling Server . . . . . . . . . . . . . . . . . . . . 7
4.4. Associated Media . . . . . . . . . . . . . . . . . . . . 8
4.5. Selection of Associated Media Protocol . . . . . . . . . 8
4.6. Number of Associated Media Streams per SIP Session . . . 8
4.7. Codec Type . . . . . . . . . . . . . . . . . . . . . . . 8
4.8. Session Duration . . . . . . . . . . . . . . . . . . . . 8
4.9. Attempted Sessions per Second (sps) . . . . . . . . . . . 8
4.10. Benchmarking Algorithm . . . . . . . . . . . . . . . . . 9
5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . 11
5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 11
5.2. Device Benchmarks for Session Setup . . . . . . . . . . . 12
5.3. Device Benchmarks for Registrations . . . . . . . . . . . 12
6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.1. Baseline Session Establishment Rate of the Testbed . . . 13
6.2. Session Establishment Rate without Media . . . . . . . . 13
6.3. Session Establishment Rate with Media Not on DUT . . . . 13
6.4. Session Establishment Rate with Media on DUT . . . . . . 14
6.5. Session Establishment Rate with TLS-Encrypted SIP . . . . 14
6.6. Session Establishment Rate with IPsec-Encrypted SIP . . . 15
6.7. Registration Rate . . . . . . . . . . . . . . . . . . . . 15
6.8. Re-registration Rate . . . . . . . . . . . . . . . . . . 16
7. Security Considerations . . . . . . . . . . . . . . . . . . . 16
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 17
8.1. Normative References . . . . . . . . . . . . . . . . . . 17
8.2. Informative References . . . . . . . . . . . . . . . . . 17
Appendix A. R Code Component to Simulate Benchmarking Algorithm 18
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 211. Introduction 🌸
This doc describes the methodology for benchmarking Session Initiation Protocol (SIP) performance, as laid out in the Terminology doc [RFC7501] 📚. The methodology and terminology are meant for benchmarking signaling plane performance while the signaling and media load change ⚡. Media streams, when they're used, only exist to study how they impact the signaling behavior 🎧. This doc focuses on SIP session setup and SIP registrations, and that's it 💖.
The Device Under Test (DUT) is a network intermediary that's RFC 3261 [RFC3261] capable, and it can play the role of a registrar, redirect server, stateful proxy, a Session Border Controller (SBC), or a B2BUA 👑. This doc does not require the intermediary to act as a stateless proxy 🙅. You can get benchmarks and compare different kinds of devices, like a SIP proxy server, Session Border Controllers (SBC), SIP registrars, and a SIP proxy server paired with a media relay 🦋.
The test cases give you metrics for the maximum 'SIP Registration Rate' and the maximum 'SIP Session Establishment Rate' the DUT can keep up over a long stretch without failing. The "extended period of time" is defined in the algorithm in Section 4.10 ⏱️. A few cases cover encrypted SIP 🔐. The test topologies are described in the Test Setup section ✨. Both setups are considered: ones where the DUT handles media and ones where it doesn't 🎀. Measuring how well the media itself performs is out of scope for these docs 🙅♀️.
Benchmark metrics could possibly be impacted by Associated Media 🤔. The values you pick for Session Duration and Media Streams per Session let you benchmark without Associated Media, so it's not a factor 💅. Session Setup Rate could possibly be impacted by the value you choose for Maximum Sessions Attempted 📈. The benchmark for Session Establishment Rate is measured with a fixed value for maximum Session Attempts 🔒.
Overall, these tests are there to compare multiple SIP implementations, and that's the big payoff 🏆. One way to use them is to get benchmarks for SIP devices from Vendor-A, get a fresh set for similar SIP devices from Vendor-B, and then compare the Vendor-A and Vendor-B results 🆚. This doc doesn't make any claims about how to interpret those results, so you do the reading yourself 🧐.
2. Terminology 📚
In this doc, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described in BCP 14, conforming to [RFC2119], and they indicate requirement levels for compliant implementations 🫡.
RFC 2119 defines how these key words get used, so the intent of Standards Track docs stays as clear as possible ✨. This doc uses these keywords, but it isn't a Standards Track doc itself 💜.
Terms specific to SIP [RFC3261] performance benchmarking are defined in [RFC7501] 🦋.
3. Benchmarking Topologies ✨
okay so test orgs, you need to know these tests generate HUGE amounts of data 😭 so make sure your networking devices like hubs, switches, or routers can actually handle that volume. no cap, it's a lot 🌊
the test cases in Sections 6.1 to 6.6 run on two topologies 🫶 in the first one, the DUT doesn't process the media (Figure 1), and in the second one, it does (Figure 2). either way, the tester or Emulated Agent (EA) sends traffic into the DUT and soaks up traffic coming back out of it 💅 these diagrams show the logical flow of information, not how everything has to be physically wired up, so don't stress about the layout 🧚
Figure 1 has the DUT sitting between the two interfaces of the EA 🦋 if the test needs media to be exchanged, the media skips the DUT and goes straight between the two endpoints. Figure 2 also has the DUT in the middle of the two EA interfaces, but this time, if media is needed, it flows through the DUT between the endpoints ✨
+--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | |
| |------------>+ |------------>+ |
| | | | | |
| | Response | | Response | |
| Tester +<------------| DUT +<------------| Tester |
| (EA) | | | | (EA) |
| | | | | |
+--------+ +--------+ +--------+
/|\ /|\
| Media (optional) |
+==============================================+
Figure 1: DUT as an Intermediary, End-to-End Media +--------+ Session +--------+ Session +--------+
| | Attempt | | Attempt | |
| |------------>+ |------------>+ |
| | | | | |
| | Response | | Response | |
| Tester +<------------| DUT +<------------| Tester |
| (EA) | | | | (EA) |
| |<===========>| |<===========>| |
+--------+ Media +--------+ Media +--------+
(Optional) (Optional)
Figure 2: DUT as an Intermediary Forwarding Mediathe test cases in Sections 6.7 and 6.8 use the setup in Figure 3 below 💜
+--------+ Registration +--------+
| | request | |
| |------------->+ |
| | | |
| | Response | |
| Tester +<-------------| DUT |
| (EA) | | |
| | | |
+--------+ +--------+
Figure 3: Registration and Re-registration Testsduring registration or re-registration, the DUT might involve backend network elements and data stores 🌸 those aren't drawn in Figure 3, but they totally affect how long the DUT takes to send back a response, so keep that in mind 🙈
this doc splits up the registration test (Section 6.7) and the re-registration test (Section 6.8) on purpose 🎀 bc in some networks, re-registering can take a different amount of time than the first registration because of all that backend processing. the registration tests and the re-registration test are both expected to run with the same set of backend network elements, so you get a stable metric ✨
4. Test Setup Parameters 🍓
4.1. Selection of SIP Transport Protocol
test cases can be run with any transport protocol SIP supports 🦋 that includes, but isn't limited to, TCP, UDP, TLS, and websockets. the protocol used for the SIP transport must be reported with benchmarking results.
SIP lets a DUT use different transports for signaling on each side of the connection to the EAs 👀 so this doc assumes the same transport is used on both sides. if that's not the case in any test, the transport on each side of the connection MUST be reported in the test-reporting template 💅
4.2. Connection-Oriented Transport Management
SIP lets a device open one connection and send multiple requests over that same connection (responses usually come back over the same connection the request went out on) 🧚 it also lets a device open a brand new connection for every single request. your connection management strategy will seriously affect the results, especially for connection-oriented transports like TLS 😭 for those, the cryptographic handshake has to happen every time a connection is opened, so it adds up fast.
The connection management strategy, i.e., use of one connection to send all requests or closing an existing connection and opening a new connection to send each request, MUST be reported with the benchmarking result. 🫶
4.3. Signaling Server
the Signaling Server is defined in the companion terminology doc ([RFC7501], Section 3.2.2) 🌈 and yes, the Signaling Server is a DUT ✨
4.4. Associated Media
some tests need Associated Media present for each SIP session 🎀 the test topologies for benchmarking DUT performance with Associated Media are the ones shown in Figure 1 and Figure 2.
4.5. Selection of Associated Media Protocol
the test cases in this doc measure SIP performance no matter which protocol the media stream uses 🦋 any media protocol SIP supports can be used, including, but not limited to, RTP and SRTP. the protocol used for Associated Media MUST be reported with benchmarking results. 💖
4.6. Number of Associated Media Streams per SIP Session
benchmarking results can change depending on how many media streams are in each SIP session 👑 for voice testing, you use a single media stream. for voice AND video, you use two media streams. the number of Associated Media Streams MUST be reported with benchmarking results. 🔥
4.7. Codec Type
again, the test cases here measure SIP performance no matter which media stream codec you use 🌸 any codec the EAs support is fair game. the codec used for Associated Media MUST be reported with the benchmarking results. 💅
4.8. Session Duration
the DUT's benchmark values can change based on how long the SIP sessions last ⭐ Session Duration MUST be reported with benchmarking results. a Session Duration of zero seconds means a BYE gets sent right after a successful SIP establishment 🙈 setting it to '0' means the EA sends a BYE immediately after it gets a 200 OK to the INVITE. setting it to a time value greater than the test's duration means a BYE will never be sent. (yes, that last sentence is literally repeated in the original, so the same thing gets said twice 💀) setting it to a time value greater than the test's duration means a BYE is never sent.
4.9. Attempted Sessions per Second (sps)
the DUT's benchmark values can change with the Session Attempt Rate the tester offers 🧚 Session Attempt Rate MUST be reported with the benchmarking results.
the test cases in Sections 6.1 to 6.6 require the EA to be set up to send the final 2xx-class response as fast as it can 🏃♀️ this doc doesn't require the tester to add any delay between getting a request and sending a final response. main character speed only 👑
4.10. Benchmarking Algorithm ✨
okay so this section is the secret sauce for making every test case run the same way 💅 the algorithm described here should be used so that everything in Section 6 gets benchmarked uniformly 🫶 there's a casual explanation below plus pseudocode, and there's an R simulation over in Appendix A for anyone who wants to see it in action [Rtool] 🧚
the goal is to find the biggest number, R, of SIP Session Attempt Rate (measured in sessions per second, or sps) that the DUT can handle with zero errors over a defined, extended period 🎀 that period is how long it takes to attempt N SIP sessions, where N is a test parameter, at the attempt rate R. the algorithm keeps tweaking things until it settles on R, and that's the convergence part 🦋
if the DUT vendor gives you a value for R, you can just use it 💖 if they don't, or if you want to figure out R using your own local media characteristics, you run the algorithm and set "r", the session attempt rate, to whatever you like 🌸 for example, you could start with "r = 100" and then watch what value it lands on once it converges ⭐ the algorithm bumps "r" up and down as it hunts for the max sps value for R, and the weights "w" and "d" control how fast it goes up and how fast it comes down 📈
the pseudocode is below, and the R simulation is in Appendix A 💜
; ---- Parameters of test; adjust as needed
N := 50000 ; Global maximum; once largest session rate has
; been established, send this many requests before
; calling the test a success
m := {...} ; Other attributes that affect testing, such
; as media streams, etc.
r := 100 ; Initial session attempt rate (in sessions/sec).
; Adjust as needed (for example, if DUT can handle
; thousands of calls in steady state, set to
; appropriate value in the thousands).
w := 0.10 ; Traffic increase weight (0 < w <= 1.0)
d := max(0.10, w / 2) ; Traffic decrease weight
; ---- End of parameters of test
proc find_R
R = max_sps(r, m, N) ; Setup r sps, each with m media
; characteristics until N sessions have been attempted.
; Note that if a DUT vendor provides this number, the tester
; can use the number as a Session Attempt Rate, R, instead
; of invoking max_sps()
end proc
; Iterative process to figure out the largest number of
; sps that we can achieve in order to setup n sessions.
; This function converges to R, the Session Attempt Rate.
proc max_sps(r, m, n)
s := 0 ; session setup rate
old_r := 0 ; old session setup rate
h := 0 ; Return value, R
count := 0
; Note that if w is small (say, 0.10) and r is small
; (say, <= 9), the algorithm will not converge since it
; uses floor() to increment r dynamically. It is best
; to start with the defaults (w = 0.10 and r >= 100).
while (TRUE) {
s := send_traffic(r, m, n) ; Send r sps, with m media
; characteristics until n sessions have been attempted.
if (s == n) {
if (r > old_r) {
old_r = r
}
else {
count = count + 1
if (count >= 10) {
# We've converged.
h := max(r, old_r)
break
}
}
r := floor(r + (w * r))
}
else {
r := floor(r - (d * r))
d := max(0.10, d / 2)
w := max(0.10, w / 2)
}
}
return h
end proc5. Reporting Format 📋
5.1. Test Setup Report 🎀
this is the template for writing down your test setup, so every detail that matters gets recorded 💅 the parentheses hold the valid values and hints, so read them before you fill in the blanks 🧚
SIP Transport Protocol = ___________________________
(valid values: TCP|UDP|TLS|SCTP|websockets|specify-other)
(Specify if same transport used for connections to the DUT
and connections from the DUT. If different transports
used on each connection, enumerate the transports used.)
Connection management strategy for connection oriented
transports
DUT receives requests on one connection = _______
(Yes or no. If no, DUT accepts a new connection for
every incoming request, sends a response on that
connection, and closes the connection.)
DUT sends requests on one connection = __________
(Yes or no. If no, DUT initiates a new connection to
send out each request, gets a response on that
connection, and closes the connection.)
Session Attempt Rate _______________________________
(Session attempts/sec)
(The initial value for "r" in benchmarking algorithm in
Section 4.10.)
Session Duration = _________________________________
(In seconds)
Total Sessions Attempted = _________________________
(Total sessions to be created over duration of test)
Media Streams per Session = _______________________
(number of streams per session)
Associated Media Protocol = _______________________
(RTP|SRTP|specify-other)
Codec = ____________________________________________
(Codec type as identified by the organization that
specifies the codec)
Media Packet Size (audio only) = __________________
(Number of bytes in an audio packet)
Establishment Threshold time = ____________________
(Seconds)
TLS ciphersuite used
(for tests involving TLS) = ________________________
(e.g., TLS_RSA_WITH_AES_128_CBC_SHA)
IPsec profile used
(For tests involving IPsec) = _____________________5.2. Device Benchmarks for Session Setup 🚀
this part is where you write your results for session setup, like the Session Establishment Rate and whether the DUT is acting as a media relay 🦋
Session Establishment Rate, "R" = __________________
(sessions per second)
Is DUT acting as a media relay? (yes/no) = _________5.3. Device Benchmarks for Registrations 📝
and this is the spot for registration results: the Registration Rate, the Re-registration Rate, and any Notes 🌈
Registration Rate = ____________________________
(registrations per second)
Re-registration Rate = ____________________________
(registrations per second)
Notes = ____________________________________________
(List any specific backend processing required or
other parameters that may impact the rate)6. Test Cases 🧪
6.1. Baseline Session Establishment Rate of the Testbed 🌸
Objective: this one benchmarks the Session Establishment Rate of the Emulated Agent (EA) with zero failures 💖
Procedure:
- Configure the DUT in the test topology shown in Figure 1.
- Set Media Streams per Session to 0.
- Execute benchmarking algorithm as defined in Section 4.10 to get the baseline Session Establishment Rate. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of Section 5.1.
Expected Results: this is the scenario for finding the maximum Session Establishment Rate of the EA and the testbed when there's no DUT in the picture 👑 the results can be used to normalize tests run on different testbeds, or just to get a better feel for how much the DUT changes things on your testbed 🔥
6.2. Session Establishment Rate without Media 🎧
Objective: this checks the Session Establishment Rate of the DUT with no Associated Media and zero failures 💅
Procedure:
- Configure a DUT according to the test topology shown in Figure 1 or Figure 2.
- Set Media Streams per Session to 0.
- Execute benchmarking algorithm as defined in Section 4.10 to get the Session Establishment Rate. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of Section 5.1.
Expected Results: you're finding the Session Establishment Rate of the DUT when the EA isn't sending any media streams 🫶
6.3. Session Establishment Rate with Media Not on DUT 🦋
Objective: this one benchmarks the Session Establishment Rate of the DUT with zero failures when Associated Media is part of the test but the media doesn't go through the DUT ✨
Procedure:
- Configure a DUT according to the test topology shown in Figure 1.
- Set Media Streams per Session to 1.
- Execute benchmarking algorithm as defined in Section 4.10 to get the session establishment rate with media. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of Section 5.1.
Expected Results: the Session Establishment Rate with Associated Media, with any number of media streams per SIP session, is expected to be identical to the Session Establishment Rate without media when the DUT runs on a platform separate from the Media Relay 🎀
6.4. Session Establishment Rate with Media on DUT 🔥
Objective: this checks the Session Establishment Rate of the DUT with zero failures when Associated Media is included and the media runs through the DUT 💜
Procedure:
- Configure a DUT according to the test topology shown in Figure 2.
- Set Media Streams per Session to 1.
- Execute benchmarking algorithm as defined in Section 4.10 to get the Session Establishment Rate with media. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of Section 5.1.
Expected Results: the Session Establishment Rate with Associated Media may come out lower than without media when the DUT and the Media Relay are on the same platform 🌈 it's helpful to know why that happens, even though those reasons aren't test parameters themselves 🧐 for example, the drop might depend on what the DUT does with the media (like relaying vs. transcoding), the type of media (audio vs. video vs. data), and the codec used 🙈 there could also be no performance impact at all, if the DUT has dedicated media-path hardware 💅
6.5. Session Establishment Rate with TLS-Encrypted SIP 🔐
Objective: this benchmarks the Session Establishment Rate of the DUT with zero failures when TLS-encrypted SIP signaling is used 🫶
Procedure:
- If the DUT is being benchmarked as a proxy or B2BUA, then configure the DUT in the test topology shown in Figure 1 or Figure 2.
- Configure the tester to enable TLS over the transport being used during benchmarking. Note the ciphersuite being used for TLS and record it in Section 5.1.
- Set Media Streams per Session to 0 (media is not used in this test).
- Execute benchmarking algorithm as defined in Section 4.10 to get the Session Establishment Rate with TLS encryption.
Expected Results: the Session Establishment Rate with TLS-encrypted SIP may come out lower than with plaintext SIP 🥺 totally expected, since encryption does add work 🍓
6.6. Session Establishment Rate with IPsec-Encrypted SIP 🔐
Objective: okay bestie, this one checks how fast the DUT can set up sessions with zero failures when the SIP signaling is wrapped in IPsec 🦋
Procedure:
- Configure a DUT according to the test topology shown in Figure 1 or Figure 2. 🌸
- Set Media Streams per Session to 0 (media is not used in this test). No media, just signaling, periodt 💅
- Configure the tester for IPsec. Note the IPsec profile being used for IPsec and record it in Section 5.1. Future you will thank present you for writing that down 📝
- Execute the benchmarking algorithm as defined in Section 4.10 to get the Session Establishment Rate with encryption. ✨
Expected Results: the Session Establishment Rate results you get with IPsec-encrypted SIP may be lower than the ones you get with plaintext SIP. Encryption costs some work, so this is totally expected 🙈
6.7. Registration Rate 📋
Objective: this is the maximum registration rate the DUT can handle over a long stretch of time with zero failures. Main character endurance test 👑
Procedure:
- Configure a DUT according to the test topology shown in Figure 3. 🎀
- Set the registration timeout value to at least 3600 seconds. An hour minimum, bestie ⏰
- Each register request MUST be made to a distinct Address of Record (AoR). Execute the benchmarking algorithm as defined in Section 4.10 to get the maximum registration rate. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of Section 5.1. For example, the use of TLS or IPsec during registration must be noted in the reporting format. In the same vein, any specific backend processing (use of databases, authentication servers, etc.) SHOULD be recorded as well. Basically, write down everything that could change the number 🍓
Expected Results: this gives you a maximum registration rate. That's the whole point, iconic 🌈
6.8. Re-registration Rate 🔁
Objective: this one checks the re-registration rate of the DUT with zero failures, using the same backend processing and parameters you used in Section 6.7. Consistency is everything 💜
Procedure:
- Configure a DUT according to the test topology shown in Figure 3. 🧚
- Execute the test detailed in Section 6.7 to register the endpoints with the registrar and obtain the registration rate. Warm-up lap, check ✅
- After at least 5 minutes of performing Step 2, but no more than 10 minutes after Step 2 has been performed, re-register the same AoRs used in Step 3 of Section 6.7. This will count as a re-registration because the SIP AoRs have not yet expired. The timing window is very specific, so don't be late to your own party ⏳
Expected Results: note the rate you get from this test so you can compare it with the rate from Section 6.7. Side-by-side glow-up energy 🪞✨
7. Security Considerations 🔒
Okay, serious moment, because security is not a bit 💀 Documents of this type do not directly affect the security of the Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to production networks. So keep your benchmarking in the lab, bestie, never on the live network.
⚠️ Do not point these tests at production gear. Seriously. Lab only.
Security threats, and how to counter them, in SIP and the media layer are discussed in RFC 3261, RFC 3550, and RFC 3711, and various other documents. Go read those if you want the full security tea ☕ This document attempts to formalize a set of common methodology for benchmarking performance of SIP devices in a lab environment. So it's a shared recipe for lab benchmarking, nothing more and nothing less 🧁
8. References
8.1. Normative References
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>. - [RFC7501] Davids, C., Gurbani, V., and S. Poretsky, "Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic Session Setup and Registration", RFC 7501, April 2015,
<http://www.rfc-editor.org/info/rfc7501>.
8.2. Informative References
- [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002,
<http://www.rfc-editor.org/info/rfc3261>. - [Rtool] R Development Core Team, "R: A Language and Environment for Statistical Computing", R Foundation for Statistical Computing Vienna, Austria, ISBN 3-900051-07-0, 2011,
<http://www.R-project.org>.
Appendix A. R Code Component to Simulate Benchmarking Algorithm 🧮💖
Here's the R code that simulates the benchmarking algorithm. Save it, source it, and watch the convergence happen 🦋
# Copyright (c) 2015 IETF Trust and the persons identified as
# authors of the code. All rights reserved.
#
# Redistribution and use in source and binary forms, with or
# without modification, are permitted provided that the following
# conditions are met:
#
# The author of this code is Vijay K. Gurbani.
#
# - Redistributions of source code must retain the above copyright
# notice, this list of conditions and
# the following disclaimer.
#
# - Redistributions in binary form must reproduce the above
# copyright notice, this list of conditions and the following
# disclaimer in the documentation and/or other materials
# provided with the distribution.
#
# - Neither the name of Internet Society, IETF or IETF Trust,
# nor the names of specific contributors, may be used to
# endorse or promote products derived from this software
# without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
# CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
# INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
# GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
# WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
# DAMAGE.
w = 0.10
d = max(0.10, w / 2)
DUT_max_sps = 460 # Change as needed to set the max sps value
# for a DUT
# Returns R, given r (initial session attempt rate).
# E.g., assume that a DUT handles 460 sps in steady state
# and you have saved this code in a file simulate.r. Then,
# start an R session and do the following:
#
# > source("simulate.r")
# > find_R(100)
# ... debug output omitted ...
# [1] 458
#
# Thus, the max sps that the DUT can handle is 458 sps, which is
# close to the absolute maximum of 460 sps the DUT is specified to
# do.
find_R <- function(r) {
s = 0
old_r = 0
h = 0
count = 0
# Note that if w is small (say, 0.10) and r is small
# (say, <= 9), the algorithm will not converge since it
# uses floor() to increment r dynamically. It is best
# to start with the defaults (w = 0.10 and r >= 100).
cat("r old_r w d \n")
while (TRUE) {
cat(r, ' ', old_r, ' ', w, ' ', d, '\n')
s = send_traffic(r)
if (s == TRUE) { # All sessions succeeded
if (r > old_r) {
old_r = r
}
else {
count = count + 1
if (count >= 10) {
# We've converged.
h = max(r, old_r)
break
}
}
r = floor(r + (w * r))
}
else {
r = floor(r - (d * r))
d = max(0.10, d / 2)
w = max(0.10, w / 2)
}
}
h
}
send_traffic <- function(r) {
n = TRUE
if (r > DUT_max_sps) {
n = FALSE
}
n
}Acknowledgments 🫶
Huge thank-yous to Keith Drage and Daryl Malas for their contributions to this document. Dale Worley provided an extensive review that led to improvements in the documents, which is so iconic of him 🙌 We are grateful to Barry Constantine, William Cerveny, and Robert Sparks for providing valuable comments during the documents' last calls and expert reviews. Al Morton and Sarah Banks have been exemplary working group chairs, and we thank them for tracking this work to completion 👑 Tom Taylor provided an in-depth review and subsequent comments on the benchmarking convergence algorithm in Section 4.10. Absolute legends, all of them 💖
Authors' Addresses 📮
Carol Davids
Illinois Institute of Technology
201 East Loop Road
Wheaton, IL 60187
United States
Phone: +1 630 682 6024
EMail: davids@iit.edu
Vijay K. Gurbani
Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane
Rm 9C-533
Naperville, IL 60566
United States
Phone: +1 630 224 0216
EMail: vkg@bell-labs.com
Scott Poretsky
Allot Communications
300 TradeCenter, Suite 4680
Woburn, MA 08101
United States
Phone: +1 508 309 2179
EMail: sporetsky@allot.com📦 where this page comes from
- ietf: rfc7502 2015-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms