resperf(1)

📦
version 2.16.0 · 5 versions
, in nixpkgs unstable, ubuntu stonking

✨ resperf(1) — General Commands Manual ✨

okay bestie, buckle up, because this is the resperf manual page and it is a whole journey 💖 (also hi, it lives in the General Commands Manual, section 1, iconic behavior)

🎀 NAME

resperf — test the resolution performance of a caching DNS server

🌸 SYNOPSIS

resperf-report [-a local_addr] [-d datafile] [-R] [-M mode]
[-s server_addr] [-p port] [-x local_port] [-t timeout] [-b bufsize]
[-f family] [-e] [-D] [-y [alg:]name:secret] [-h] [-i interval]
[-m max_qps] [-r rampup_time] [-c constant_traffic_time] [-L max_loss]
[-C clients] [-q max_outstanding] [-F fall_behind] [-v] [-W]
[-O option=value]
resperf [-a local_addr] [-d datafile] [-R] [-M mode] [-s server_addr]
[-p port] [-x local_port] [-t timeout] [-b bufsize] [-f family] [-e]
[-D] [-y [alg:]name:secret] [-h] [-i interval] [-m max_qps]
[-P plot_data_file] [-r rampup_time] [-c constant_traffic_time]
[-L max_loss] [-C clients] [-q max_outstanding] [-F fall_behind] [-v]
[-W] [-O option=value]

🦋 DESCRIPTION

so resperf is the bestie-sibling of dnsperf, fr fr. dnsperf was originally built for benchmarking authoritative servers, and ngl it just does not vibe with caching servers that are out here talking to the actual live Internet 😭. why? because dnsperf uses a "self-pacing" approach — like, it assumes you can keep the server 100% busy just by yeeting a lil burst of back-to-back queries to fill up network buffers, and then sending a fresh query every time a response comes back. that method? slays for authoritative servers that handle queries in order, one at a time. it also works pretty well for a caching server sitting in a closed lab environment talking to a simulated Internet that's all chillin' on the same LAN ✨.

but… it does NOT work well with a caching server talking to the actual Internet, bestie, because that server might need to juggle thousands of queries in parallel just to hit its max throughput 💅. there have been literally so many attempts to use dnsperf (or its predecessor, queryperf) for benchmarking live caching servers, and they usually give mid-to-poor results 🙈. so yeah — a whole separate tool designed specifically for caching servers was needed. and that's our girl. 👑

💫 How resperf works

unlike the "self-pacing" thing dnsperf does, resperf works by sending DNS queries at a controlled, steadily increasing rate. by default, resperf sends traffic for 60 seconds, linearly ramping the traffic from zero all the way up to 100,000 queries per second (or max_qps, if you set that 👀).

during the test, resperf listens for responses from the server and keeps track of response rates, failure rates, and latencies. it will also keep listening for responses for an additional 40 seconds after it stops sending traffic, so the server actually has time to respond to the last queries it got 🫶. why 40? because that number was chosen to be longer than the overall query timeout of both Nominum CacheServe and current versions of BIND. iykyk.

if the test is successful, the query rate will at some point exceed the capacity of the server and queries will be dropped, which makes the response rate stop growing — or even go down — as the query rate keeps climbing 📉.

the result of the test is a set of measurements of the query rate, response rate, failure response rate, and average query latency, all as functions of time. periodt.

🌈 What you will need

benchmarking a live caching server is serious business. a fast caching server like Nominum CacheServe, resolving a mix of cacheable and non-cacheable queries typical of ISP customer traffic, is capable of resolving well over 1,000,000 queries per second 🔥. in the process, it will send more than 40,000 queries per second to authoritative servers on the Internet, and receive responses to most of them. assuming an average request size of 50 bytes and a response size of 150 bytes, this amounts to some 1216 Mbps of outgoing and 448 Mbps of incoming traffic 😭. if your Internet connection can't handle that bandwidth, you're not measuring the server — you're measuring the speed of your connection 💀, and you may saturate the connection, which degrades service for other users. not cute.

make sure there is no stateful firewall between the server and the Internet, because most of them can't handle the amount of UDP traffic the test will generate and will end up dropping packets, skewing your results. some will even lock up or crash 🙈.

you should run resperf on a machine separate from the server under test, on the same LAN. preferably this should be a Gigabit Ethernet network ✨. and the machine running resperf should be at least as fast as the machine being tested — otherwise it may end up being the bottleneck, and that's embarrassing 💅.

there should be no other applications running on the machine running resperf. performance testing at these traffic levels is essentially a hard real-time application — consider the fact that at a query rate of 100,000 queries per second, if resperf gets delayed by just 1/100 of a second, then 1000 incoming UDP packets will arrive in the meantime 😵. that's more than most operating systems will buffer, which means packets will be dropped. no cap.

because the granularity of timers provided by operating systems is typically too coarse to accurately schedule packet transmissions at sub-millisecond intervals, resperf will busy-wait between packet transmissions, constantly polling for responses in the meantime. therefore it is normal for resperf to consume 100% CPU during the whole test run — even during periods where query rates are relatively low. she's working hard, okay 🥺

you will also need a set of test queries in the dnsperf file format. see the dnsperf man page for instructions on how to construct this query file 📝. to make the test as realistic as possible, the queries should be derived from recorded production client DNS traffic, without removing duplicate queries or other filtering. with the default settings, resperf will use up to 3 million queries in each test run.

if the caching server being tested has a configurable limit on the number of simultaneous resolutions — like the max-recursive-clients statement in Nominum CacheServe, or the recursive-clients option in BIND 9 — you will probably have to increase it ⬆️. as a starting point, we recommend a value of 10000 for Nominum CacheServe and 100000 for BIND 9. should the limit be reached, it will show up in the plots as an increase in the number of failure responses 👀.

the server being tested should be restarted at the beginning of each test, so you know it's starting with an empty cache 🧼. if the cache already contains data from a previous test run that used the same set of queries, almost all queries will be answered from the cache, which inflates the performance numbers. that's not a flex, that's a lie.

to use the resperf-report script, you need to have gnuplot installed. make sure your installed version of gnuplot supports the png terminal driver ✨. if your gnuplot doesn't support png but does support gif, you can change the line saying terminal=png in the resperf-report script to terminal=gif. slay, adaptability. 🦋

🧚 Running the test

resperf is typically invoked via the resperf-report script, which runs resperf with its output redirected to a file and then automatically generates an illustrated report in HTML format 📊. command line arguments you give to resperf-report get passed on unchanged to resperf. easy.

when running resperf-report, you'll need to specify at least the server IP address and the query data file. a typical invocation looks like:

resperf-report -s 10.0.0.2 -d queryfile

with default settings, the test run will take at most 100 seconds (60 seconds of ramping up traffic, and then 40 seconds of waiting for responses) ⏳ — but in practice, the 60-second traffic phase will usually get cut short. to be precise, resperf can transition from the traffic-sending phase to the waiting-for-responses phase in three different ways:

  • running for the full allotted time and successfully reaching the maximum query rate (by default, 60 seconds and 100,000 qps, respectively). since this is a very high query rate, this will rarely happen (with today's hardware) — one of the other two conditions below will usually hit first 🙈.
  • exceeding 65,536 outstanding queries. this often happens as a result of (successfully 😌) exceeding the capacity of the server being tested, causing the excess queries to be dropped. the limit of 65,536 queries comes from the number of possible values for the ID field in the DNS packet. resperf needs to allocate a unique ID for each outstanding query, so it is therefore unable to send further queries if the set of possible IDs is exhausted 💀.
  • when resperf finds itself unable to send queries fast enough. resperf will notice if it's falling behind in its scheduled query transmissions, and if this backlog reaches 1000 queries, it will print a message like "Fell behind by 1000 queries" (or whatever the actual number is at the time) and stop sending traffic 🛑.

regardless of which of the above conditions caused the traffic-sending phase of the test to end, you should examine the resulting plots to make sure the server's response rate is flattening out toward the end of the test 👑. if it is not, then you are not loading the server enough. and if you're getting the "Fell behind" message, make sure that the machine running resperf is fast enough and has no other applications running.

you should also monitor the CPU usage of the server under test 👀. it should reach close to 100% CPU at the point of maximum traffic; if it does not, you most likely have a bottleneck in some other part of your test setup — for example, your external Internet connection. not the server's fault, bestie.

the report generated by resperf-report will be stored with a unique file name based on the current date and time, e.g., 20060812-1550.html ✨. the PNG images of the plots and other auxiliary files will be stored in separate files beginning with the same date-time string. to view the report, simply open the .html file in a web browser 🌐.

if you need to copy the report to a separate machine for viewing, make sure to copy the .png files along with the .html file (or simply copy all the files, e.g., using scp 20060812-1550.* host:directory/) 🫶.

💜 Interpreting the report

the .html file produced by resperf-report consists of two sections 🎀. the first section, "Resperf output", contains output from the resperf program such as progress messages, a summary of the command line arguments, and summary statistics. the second section, "Plots", contains two plots generated by gnuplot: "Query/response/failure rate" and "Latency" 📈.

The <strong>"Query/response/failure rate"</strong> plot is giving three graphs at once, bestie, and you need to know all of them 💅

  • "Queries sent per second" — this one shows how much traffic is actually being yeeted at the server. It should look like a super clean straight diagonal line, because that's the linear ramp-up of traffic doing its thing. ✨
  • "Total responses received per second" — how many of those queries got a response back. ALL responses count here, whether they succeeded (that's NOERROR or NXDOMAIN) or not (like, e.g., SERVFAIL). We don't discriminate 💖
  • "Failure responses received per second" — how many queries got a failure response. A response counts as a failure if its RCODE is <strong>neither</strong> NOERROR <strong>nor</strong> NXDOMAIN. Periodt.

So basically, if you just look at the graphs with your eyeballs 👀, you can clock how the server behaves as the load goes up. At first, "Total responses received per second" will be glued to "Queries sent per second" — like, so glued that it often renders the other graph invisible because they're literally plotted on top of each other 🫶 — but once the load exceeds what the server can handle, the total-responses graph may diverge from the queries-sent graph and flatten out, which is the plot's way of saying "bestie, some of your queries are getting dropped". 😭

The "Failure responses received per second" graph will normally show a roughly linear ramp chillin' near the bottom of the plot with some random fluctuation sprinkled in — because your typical query traffic always has some small percentage of failing queries randomly interspersed with the successful ones. As total traffic goes up, the number of failures goes up proportionally. Math is mathing. 🍓

But if that failure graph turns sharply upwards, that's another little heads-up that the load has blown past the server's capacity 🔥. This happens when the server reacts to overload by sending SERVFAIL responses instead of just dropping queries. And since <strong>Nominum CacheServe</strong> and <strong>BIND 9</strong> will both respond with SERVFAIL when they exceed their max-recursive-clients or recursive-clients limit, respectively 👑, a sudden jump in the number of failures could mean that limit needs to be increased.

Now the "Latency" plot — she has a single graph, marked <strong>"Average latency"</strong>. It shows how latency varies over the course of the test. Usually you'll see a downwards trend because the cache hit rate improves as more and more responses get cached during the test, and latency for a cache hit is way smaller than for a cache miss. 🦋 This graph is there as a lil' aid for figuring out where the server gets overloaded — which shows up as a sharp upwards turn in the graph. But heads up: the latency graph is NOT for making absolute latency measurements or for comparing servers against each other 🙈 — the latencies shown aren't representative of production latencies, because of the initially empty cache and the deliberate overloading of the server towards the end of the test. It's a vibe check, not a lab report. 💀

Important note: all measurements are displayed on the plot at the horizontal position corresponding to the point in time when the <strong>query was sent</strong>, NOT when the response (if there even was one) was received. ✨ This makes it easy to compare the query and response rates — for example, if no queries are dropped, the query and response graphs will be identical. Another example: if the plot shows 10% failure responses at t=5 seconds, that means 10% of the queries sent at t=5 seconds eventually failed, <strong>not</strong> that 10% of the responses received at t=5 seconds were failures. Read it twice, it matters. 💖

💭 Determining the server's maximum throughput 💭

Okay so, a lot of the time the whole point of running resperf is to figure out the server's maximum throughput — AKA the number of queries per second it can actually handle. 👑

And ngl, this is not always an easy task, because as a server gets driven into overload, the service it provides may deteriorate gradually, and that deterioration can show up as any of these:

  • queries being dropped 🫠
  • an increase in the number of SERVFAIL responses 😵
  • an increase in latency 🐌

So the maximum throughput can be defined as the highest level of traffic at which the server still provides an acceptable level of service — but that means you first gotta decide what "acceptable level of service" even means in terms of packet drop percentage, SERVFAIL percentage, and latency. That's a you decision, bestie. 💅

The summary statistics in the <strong>"Resperf output"</strong> section of the report contains a "Maximum throughput" value, which by default is determined from the maximum rate at which the server was able to return responses — without regard to the number of queries being dropped or failing at that point. This method of throughput measurement has the advantage of simplicity ✨, but it may or may not be appropriate for your needs. The reported value should <strong>always</strong> be validated by a visual inspection of the graphs to make sure service hasn't already deteriorated unacceptably before the maximum response rate is reached. 👀 It may also be helpful to look at the <strong>"Lost at that point"</strong> value in the summary statistics — that tells you the percentage of queries that was being dropped at the point in the test when the maximum throughput was reached.

Alternatively 😌, you can make resperf report the throughput at the point in the test where the percentage of queries dropped exceeds a given limit (or the maximum as above, if that limit is never exceeded). This can be a more realistic indication of how much the server can be loaded while still providing an acceptable level of service. And you do it with the -L command line option 🎀 — for example, specifying -L 10 makes resperf report the highest throughput reached before the server starts dropping more than 10% of the queries. Iconic.

There is no corresponding way of automatically constraining results based on the number of failed queries, because — unlike dropped queries — resolution failures will occur even when the server is not overloaded, and the number of such failures is heavily dependent on the query data and network conditions. So fr fr, you gotta manually inspect the plots to make sure there isn't an abnormal number of failures. 👀💖

🚦 GENERATING CONSTANT TRAFFIC 🚦

So besides ramping traffic up linearly, resperf can also send a constant stream of traffic. 🌈 Super useful when you're using resperf for stuff other than performance measurement — like, for example, "soak testing" a server by subjecting it to a sustained load for an extended period of time. 💜

To generate a constant traffic load, use the -c command line option, together with the -m option which specifies the desired constant query rate. 🎯 For example, to send 10000 queries per second for an hour, use:

-m 10000 -c 3600

This will include the usual 30-second gradual ramp-up of traffic at the beginning, which may be useful to avoid initially overwhelming a server that's starting with an empty cache 🧚. To start the onslaught of traffic instantly, use:

-m 10000 -c 3600 -r 0

To be precise, here's the whole choreography ✨: resperf will do a linear ramp-up of traffic from 0 to -m queries per second over a period of -r seconds, followed by a plateau of steady traffic at -m queries per second lasting for -c seconds, followed by waiting for responses for an extra 40 seconds. Either the ramp-up or the plateau can be suppressed by supplying a duration of zero seconds with -r 0 and -c 0, respectively. The latter is the default. 💅

Real talk: sending traffic at high rates for hours on end will of course require <strong>very large amounts of input data</strong> 🫠. Also, a long-running test will generate a large amount of plot data, which is kept in memory for the entire duration of the test. To reduce the memory usage and the size of the plot file, consider increasing the interval between measurements from the default of 0.5 seconds using the -i option in long-running tests. Your RAM will thank you. 🙏

When you're using resperf for long-running tests, it is <strong>important</strong> that the traffic rate specified using the -m is one that BOTH resperf itself AND the server under test can sustain. Otherwise the test is likely to get cut short 😭 — either because you run out of query IDs (which happens because of large numbers of dropped queries) or because resperf falls behind its transmission schedule. No cap.

🌈 Using DNS-over-HTTPS 🌈

When using DNS-over-HTTPS you <strong>must</strong> set the -O doh-uri=... to something that actually works with the server you're sending to. 🎀 Also note that the value for maximum outstanding queries will be used to control the maximum concurrent streams within the HTTP/2 connection. ✨

✨ OPTIONS ✨

Okay so here's the tea 🫖: because the resperf-report script just hands its command line options straight over to the resperf programs, they both take the exact same set of options — with ONE exception. resperf-report auto-slides an appropriate -P onto the resperf command line all by itself, so it doesn't take a -P option of its own. 💅

  • -d datafile — names the input data file. If you don't say anything, resperf just reads from standard input. no biggie.
  • -R — reopen the datafile if it runs out of data before testing finishes. this is the move for long-running tests on a tiny little baby query datafile 🍓
  • -M mode — picks the transport mode: "udp", "tcp", "dot" or "doh". default is "udp".
  • -s server_addr — the name or address of the server you're sending requests to. default is the loopback address, 127.0.0.1. (yes, you can absolutely flood your own localhost, live your truth 💀)
  • -p port — sets the port the DNS packets get sent to. leave it out and you get the standard DNS port: udp/tcp 53, DoT 853, DoH 443.
  • -a local_addr — the local address you're sending requests from. default is the wildcard address.
  • -x local_port — the local port you're sending requests from. default is the wildcard port (0).

If you're acting as multiple clients and the wildcard port is in play, each client gets a different random port. say a port explicitly though, and the clients use a range of ports starting from the one you specified. ✨

  • -t timeout — the request timeout value, in seconds. resperf stops waiting for a response to a particular request once this many seconds have gone by. default is 45 seconds.

so basically resperf times out unanswered requests to reclaim query IDs, so the query ID space doesn't get exhausted in a long-running test — like when you're "soak testing" a server for an entire day with -m 10000 -c 86400. lowkey the timeouts and the tuning of them are of little use in the way more typical use case of a performance test that only lasts a minute or two.

the default timeout of 45 seconds was chosen to be longer than the query timeout of current caching servers. heads up: this is longer than the matching default in dnsperf, because caching servers can take many orders of magnitude longer to answer a query than authoritative servers do. not the same vibe at all.

and if you use a short timeout, there's a real possibility resperf will get a response after the matching request already timed out — when that happens, you'll see a message like Warning: Received a response with an unexpected id: 141 printed. 😭

  • -b bufsize — sets the size of the socket's send and receive buffers, in kilobytes. don't specify it and the operating system's default is used.
  • -f family — the address family used for sending DNS packets. possible values are "inet", "inet6", or "any". if you pick "any" — which is the default — resperf will use whichever address family is appropriate for the server it's sending packets to. main character energy, adapting to the situation 🌈
  • -e — enables EDNS0 [RFC2671] by adding an OPT record to all packets sent.
  • -D — sets the DO (DNSSEC OK) bit [RFC3225] in all packets sent. this also enables EDNS0, which is required for DNSSEC. (they come as a package deal, iconic 👑)
  • -y [alg:]name:secret — adds a TSIG record [RFC2845] to all packets sent, using the specified TSIG key algorithm, name and secret. the algorithm defaults to hmac-md5 and the secret is expressed as a base-64 encoded string.
  • -h — print a usage statement and exit. the classic.
  • -i interval — the time interval between data points in the plot file. default is 0.5 seconds.
  • -m max_qps — the target maximum query rate, in queries per second. this should be higher than the expected maximum throughput of the server being tested. traffic gets ramped up at a linearly increasing rate until this value is reached, or until one of the other conditions described in the section "Running the test" happens. default is 100000 queries per second. 🔥
  • -P plot_data_file — the name of the plot data file. default is resperf.gnuplot.
  • -r rampup_time — how long traffic will be ramped up for. default is 60 seconds.
  • -c constant_traffic_time — how long traffic will be sent at a constant rate following the initial ramp-up. default is 0 seconds, which means no sending of traffic at a constant rate will be done at all.
  • -L max_loss — the maximum acceptable query loss percentage, used for determining the maximum throughput value. default is 100%, meaning resperf will measure the maximum throughput without regard to query loss. zero chill, zero filtering.
  • -C clients — act as multiple clients. requests get sent from multiple sockets. default is to act as 1 client.
  • -q max_outstanding — sets the maximum number of outstanding requests. resperf stops ramping up traffic once this many queries are outstanding. default is 64k, and the limit is 64k per client.
  • -F fall_behind — sets the maximum number of queries that can fall behind being sent. resperf will stop when this many queries should have been sent — and ngl it can be relatively easy to hit if max_qps is set too high. default is 1000, and setting it to zero (0) disables the check entirely.
  • -v — enables verbose mode, so you get reports about network readiness and congestion. yapper mode: ON 🗣️
  • -W — log warnings and errors to standard output instead of standard error, making it easier for script, test and automation to capture all the output. very considerate of them tbh 🫶
  • -O option=value — set an extended long option for various things to control different aspects of testing or protocol modules. see EXTENDED OPTIONS in dnsperf(1) for the list of available options.

🌸 THE PLOT DATA FILE 🌸

The plot data file is written by the resperf program, and it holds the data to be plotted with gnuplot. when you run resperf via the resperf-report script, you as the user never have to deal with this file directly — but its format and contents are documented right here for completeness, and in case you feel like running resperf directly and using its output for purposes other than viewing it with gnuplot. it's giving DIY energy 💅

The first line of the file is a comment identifying the fields. you can clock it as a comment by its leading hash sign (#).

Subsequent lines contain the actual plot data. for the purposes of generating the plot data file, the test run is divided into time intervals of 0.5 seconds — or some other length of time specified with the -i command line option. each line corresponds to one such interval, and carries the following values as floating-point numbers:

  • Time — the midpoint of this time interval, in seconds since the beginning of the run.
  • Target queries per second — the number of queries per second scheduled to be sent in this time interval.
  • Actual queries per second — the number of queries per second actually sent in this time interval. (target vs reality, we've all been there 😭)
  • Responses per second — the number of responses received corresponding to queries sent in this time interval, divided by the length of the interval.
  • Failures per second — the number of responses received corresponding to queries sent in this time interval and having an RCODE other than NOERROR or NXDOMAIN, divided by the length of the interval.
  • Average latency — the average time between sending the query and receiving a response, for queries sent in this time interval.
  • Connections — the number of connections done, including re-connections, during this time interval. this is only relevant to connection oriented protocols, such as TCP and DoT.
  • Average connection latency — the average time between starting to connect and having the connection ready for sending queries to, for this time interval. this is only relevant to connection oriented protocols, such as TCP and DoT.

🔍 SEE ALSO 🔍

dnsperf(1)

💖 AUTHOR 💖

Nominum, Inc.

Maintained by DNS-OARC

https://www.dns-oarc.net/

🐛 BUGS 🐛

For issues and feature requests please use:

https://codeberg.org/DNS-OARC/dnsperf/issues

For question and help please use:

admin@dns-oarc.net

and that's the whole page bestie!! this one's from resperf, version 2.16.0, filed under resperf(1) — go forth and ramp that traffic, you absolute legend 🎀✨💖

📦 where this page comes from