CMTIME(1)

💖 CMTIME(1) — the cmtime glow-up 💖

okay bestie, welcome to cmtime, straight outta librdmacm — it's the RDMA CM connection-steps timing test and she is literally here to clock every single step of your connection setup and teardown. grab a snack, we're benchmarking ✨

---

🌸 NAME 🌸

cmtime — RDMA CM connection steps timing test. that's it, that's the vibe 💅

---

🎀 SYNOPSIS 🎀

cmtime -C controller_address
          (-B bind_interface | -b bind_address)
          [-L]
          [-c connections_per_pair] [-P num_peers]
          [-p controller_port]
          [-q base_qpn] [-n num_threads]
          [-m mimic_qp_delay_us] [-r retries]
          [-t timeout_ms]
          [-S]

---

🦋 DESCRIPTION 🦋

so what does she actually do?? she figures out the min, max, and average times for all the various "steps" in RDMA CM connection setup and teardown, running between clients and servers. no cap, that's the whole job and she eats every time 🔥

to use it, you start one or more servers (that's the -L flag bestie) plus one or more clients. each client will establish -c number of connections to each server. by default, the test runs with 1 client and 1 server. simple, cute, efficient 💖

one process is gonna act as the controller process. the controller coordinates the clients and servers — she uses out-of-band sockets to collect and distribute addresses and keeps clients and servers in sync as they advance through each "step". she's the group chat admin of this operation fr fr 👑

the "steps" that get timed are, in order:

1. create ID
1. bind address
1. resolve address
1. resolve route
1. create QP
1. modify QP to INIT
1. modify QP to RTR
1. modify QP to RTS
1. CM connect
1. client establish
1. disconnect
1. destroy QP
1. destroy ID

the way that so many operations are asynchronous, allowing progress on multiple connections simultaneously?? iconic. the sum output adds the time that all connections took for a given step. the average us/conn is the sum divided by the number of connections. this is useful to identify steps which take a significant amount of time. the min and max values are the smallest and largest times that any single connection took to complete a given step.

and now the total and avg/iter times — those measure the time to complete a given step for all connections. these two values take into account asynchronous operations. for steps which are serial, the total and sum values will be roughly the same. for asynchronous steps, the total may be significantly lower than the sum, as multiple connections will be in progress simultaneously. the avg/iter is the total time divided by the number of connections. math girlies rise 🧮✨

ngl, in many cases times may not be available or only available on the clients. in such situations, the output will show 0. stats printed by each instance pertain to connections handled by it — on runs with multiple clients and/or servers (-P > 2) further external aggregation might be required. so yeah, you might have to do a lil manual math, sorry babe 😭

---

✨ OPTIONS ✨

• -C controller_address — the network name or IP address of the instance which will provide step synchronization. this option must be specified by all instances. the first instance to start which discovers this address as local will act as controller and will synchronize all other instances via an out-of-band channel as benchmark steps progress. any type of instance (client or server) can act as controller. everybody's welcome in the controller seat 🫶
• -B bind_interface — the local RDMA interface to bind to. only one of -B or -b is accepted. pick a lane bestie.
• -b bind_address — the local network address to bind to. only one of -B or -b is accepted. same energy, different flavor.
• -L — whether this instance is an RDMA CM server (set) or client (unset). (default client/unset). flip the switch and you're a server now, main character energy 👑
• -c connections_per_pair — the number of connections to establish between each client-server pair. (default 100). go big or go home.
• -P num_peers — total number of peers (clients, servers) which are going to be started. (default 2). the whole squad, counted.
• -p controller_port — the controller's port number. her personal line 💅
• -q base_qpn — the first QP number to use when creating connections without allocating hardware QPs. the test will use the values between base_qpn and base_qpn plus connections when connecting. (default 1000).
• -n num_threads — sets the number of threads to spawn used to process connection events and hardware operations. (default 1).
• -m mimic_qp_delay_us — "simulates" QP creation and modify calls by replacing them with a simple sleep function instead. this allows testing the CM at larger scale than would be practical, or even possible given system configuration settings, if HW resources needed to be allocated. fake it till you make it, the hardware never has to know 🙈
• -r retries — number of retries when resolving address or route. (default 2). she's persistent, she's trying again 💜
• -S — run connection rate test using sockets. this provides a baseline comparison of RDMA connections versus TCP connections. sockets are set to blocking mode. only supported in 1 client 1 server mode (-P 2). do not try to be slick with other peer counts, it will not work 🚫
• -t timeout_ms — timeout in millseconds (ms) when resolving address or route. (default 2000 - 2 seconds).

---

🌈 EXAMPLES 🌈

one client connecting to one server:

server$ cmtime -B enp1s0f0np0 -C 192.0.2.1 -L
client$ cmtime -B enp1s0f0np0 -C 192.0.2.1

two clients connecting to three servers, 6k connections in total:

client1$ cmtime -B enp1s0f0np0 -P 5 -C 192.0.2.1 -c 1000
client2$ cmtime -B enp1s0f0np0 -P 5 -C 192.0.2.1 -c 1000
server1$ cmtime -B enp1s0f0np0 -P 5 -C 192.0.2.1 -c 1000 -L
server2$ cmtime -B enp1s0f0np0 -P 5 -C 192.0.2.1 -c 1000 -L
server3$ cmtime -B enp1s0f0np0 -P 5 -C 192.0.2.1 -c 1000 -L

---

💀 NOTES 💀

because this test maps RDMA resources to userspace, users must ensure that they have available system resources and permissions. see the libibverbs README file for additional details. like, actually read it, don't just scroll past 👀

---

🔥 SEE ALSO 🔥

rdma_cm(7) — go read that one too, she's the lore.

---

this page lives in <strong>librdmacm</strong>, dated <strong>2025-09-03</strong> 🎀 stay slay, go measure some connections, and remember: every step deserves her moment in the spotlight ✨💖🦋