perftest(1)

perftest(1) — General Commands Manual 🫶

okay bestie, this is the perftest(1) page, and it's giving so many benchmarks. no cap, we're gonna keep every single detail, just with way more sparkle 💅

✨ NAME ✨

ib_write_bw, ib_read_bw, ib_send_bw, ib_atomic_bw, ib_write_lat, ib_read_lat, ib_send_lat, ib_atomic_lat, raw_ethernet_bw, raw_ethernet_lat, raw_ethernet_burst_lat, raw_ethernet_fs_rate — benchmarks for various types of infinabnd performance 🦋

💖 DESCRIPTION 💖

So like, Perftest is a package that includes various benchmarks that measures different metrics & verbs performance, and it comes with many different options and modes.

🏁 RUNNING TESTS 🏁

Server:

./<test name> <options>

Client:

./<test name> <options> <server IP address>

Examples:

1. Running bidirectional bandwidth test using Write verb for 5 seconds with 8388608 as a message size and 3 qps: Server: ./ib_write_bw -s 8388608 -b -D 5 -q 3 Client: ./ib_write_bw -s 8388608 -b -D 5 -q 3 1.1.1.2
2. Running latency test using Read verb for 5000 iterations with 32 as a message size: Server: ./ib_read_lat -s 32 -n 5000 Client: ./ib_read_lat -s 32 -n 5000 192.168.0.1

🚨 IMPORTANT NOTES 🚨

1. The options that specific to modes in perftest must be the same for both server and client.
2. Perftest applications may need to be ran with sudo when running from non root.
3. Perftest applications usually installed to the /usr/bin/.
4. Perftest may print some failures with syndroms to the stderr, perftest get those errors from rdma-core.

---

✨ OPTIONS ✨

• -h, --help — Lists the available options to the screen.
• -a, --all — Run sizes from 2 till 2^23. Not relevant for Atomic and RawEth.
• -A, --atomic_type=<type> — Type of atomic operation from {CMP_AND_SWAP,FETCH_AND_ADD} (default FETCH_AND_ADD). Relevant only for Atomic.
• -b, --bidirectional — Measure bidirectional bandwidth (default unidirectional). Relevant only for BW.
• -c, --connection=<RC/XRC/UC/UD/DC/SRD> — Connection type RC/XRC/UC/UD/DC/SRD (default RC). UD relevant only for Send verb. SRD relevant only for Read, Write and Send verbs. UC relevant only for Write and Send verbs. Not relevant for RawEth.
• --log_dci_streams=<log_num_dci_stream_channels> — (default 0) Run DC initiator as DCS instead of DCI with <log_num dci_stream_channels>. Not relevant for RawEth. System support required.
• --log_active_dci_streams=<log_num_active_dci_stream_channels> — (default log_num_dci_stream_channels). Not relevant for RawEth. System support required.
• --aes_xts — Runs traffic with AES_XTS feature (encryption). Not relevant for RawEth and Write latency. System support required.
• --encrypt_on_tx — Runs traffic with encryption on tx (default decryption on tx). Not relevant for RawEth and Write latency. System support required.
• --sig_before — Puts signature on data before encrypting it (default after). Not relevant for RawEth and Write latency. System support required.
• --aes_block_size=<512,520,4048,4096,4160> — (default 512). Not relevant for RawEth and Write latency. System support required.
• --data_enc_keys_number=<number of data encryption keys> — (default 1). Not relevant for RawEth and Write latency. System support required.
• --kek_path <path to the key encryption key file> — Not relevant for RawEth and Write latency. System support required.
• --credentials_path <path to the credentials file> — Not relevant for RawEth and Write latency. System support required.
• --data_enc_key_app_path <path to the data encryption key app> — Not relevant for RawEth and Write latency. System support required.
• -C, --report-cycles — Report times in cpu cycle units (default microseconds). Relevant only for latency.
• -d, --ib-dev=<dev> — Use IB device <dev> (default first device found).
• -D, --duration — Run test for a customized period of seconds.
• -e, --events — Sleep on CQ events (default poll). Not relevant for Write and RawEth.
• -X, --vector=<completion vector> — Set <completion vector> used for events. Not relevant for Write and RawEth.
• -f, --margin — measure results within margins. (default=2sec).
• -F, --CPU-freq — Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
• -g, --mcg — Send messages to multicast group with 1 QP attached to it. When there is no multicast gid specified, a default IPv6 typed gid '255:1:0:0:0:2:201:133:0:0:0:0:0:0:0:0' will be used. Relevant only for send non fsRate.
• -H, --report-histogram — Print out all results (default print summary only). Relevant only for latency and raw_ethernet_fs_rate.
• -i, --ib-port=<port> — Use port <port> of IB device (default 1).
• -I, --inline_size=<size> — Max size of message to be sent in inline. Not relevant for Read and Atomic.
• -l, --post_list=<list size> — Post list of send WQEs of <list size> size (instead of single post). Relevant only for BW and raw_ethernet_burst_lat.
• --recv_post_list=<list size> — Post list of receive WQEs of <list size> size (instead of single post). Relevant only for BW and raw_ethernet_burst_lat.
• -L, --hop_limit=<hop_limit> — Set hop limit value (ttl for IPv4 RawEth QP). Values 0-255 (default 64). Relevant only for RawEth. Not relevant for raw_ethernet_fs_rate.
• -m, --mtu=<mtu> — MTU size : 64 - 9600 (default port mtu) for RawEth else 256 - 4096. Not relevant for raw_ethernet_fs_rate.
• -M, --MGID=<multicast_gid> — In multicast, uses <multicast_gid> as the group MGID. <multicast_gid> can be either decimal or hexadecimal, e.g. regarding the IPv4 224.0.0.30 : Decimal: 0:0:0:0:0:0:0:0:0:0:255:255:224:0:0:30 , Hexadecimal: 0:0:0:0:0:0:0:0:0:0:0xff:0xff:0xe0:0:0:0x1e. Relevant only for send non fsRate.
• -n, --iters=<iters> — Number of exchanges (at least 5, default for write 5000 else 1000 ).
• -N, --noPeak — Cancel peak-bw calculation (default with peak up to iters=20000). Relevant only for bandwidth.
• -o, --outs=<num> — Relevant only for Read and Atomic.
• -O, --dualport — Run test in dual-port mode. Not relevant for RawEth. Relevant only for bandwidth. System support required.
• -p, --port=<port> — Listen on/connect to port <port> (default 18515).
• -q, --qp=<num of qp's> — Num of qp's(default 1). Relevant only for bandwidth.
• -Q, --cq-mod — Generate Cqe only after <--cq-mod> completion. Relevant only for bandwidth.
• -r, --rx-depth=<dep> — Rx queue size (default 512), if using srq, rx-depth controls max-wr size of the srq. Relevant only for send non fsRate.
• -R, --rdma_cm — Connect QPs with rdma_cm and run test on those QPs. Not relevant for RawEth.
• -s, --size=<size> — Size of message to exchange (default 65536 for bw, for lat 2). Not relevant for Atomic.
• -S, --sl=<sl> — SL (default 0). Not relevant for raw_ethernet_fs_rate.
• -t, --tx-depth=<dep> — Size of tx queue (default 128 for bw else 1). Relevant only for bw and raw_ethernet_burst_lat.
• -T, --tos=<tos value> — Set <tos_value> to RDMA-CM QPs. available only with -R flag. values 0-256 (default off). Not relevant for RawEth
• -u, --qp-timeout=<timeout> — QP timeout, timeout value is 4 usec * 2 ^(timeout), default 14.
• -U, --report-unsorted — (implies -H) print out unsorted results (default sorted). Relevant only for latency and raw_ethernet_burst_lat and raw_ethernet_fs_rate.
• -V, --version — Display perftest version number.
• -W, --report-counters=<list of counter names> — Report performance counter change (example: counters/port_xmit_data,hw_counters/out_of_buffer).
• -x, --gid-index=<index> — Test uses GID with GID index. Not relevant for RawEth.
• -z, --comm_rdma_cm — Communicate with rdma_cm module to exchange data - use regular QPs. Not relevant for RawEth.
• --out_json — Save the report in a json file.
• --out_json_file=<file> — Name of the report json file. (Default: "perftest_out.json" in the working directory).
• --cpu_util — Show CPU Utilization in report, valid only in Duration mode.
• --dlid — Set a Destination LID instead of getting it from the other side. Not relevant for raw_ethernet_fs_rate.
• --dont_xchg_versions — Do not exchange versions and MTU with other side. Not relevant for RawEth.
• --force-link=<value> — Force the link(s) to a specific type: IB or Ethernet. Not relevant for raw_ethernet_fs_rate.
• --use-srq — Use a Shared Receive Queue. --rx-depth controls max-wr size of the SRQ. Relevant only for Send.
• --ipv6 — Use IPv6 GID. Default is IPv4. Not relevant for RawEth.
• --ipv6-addr=<IPv6> — Use IPv6 address for parameters negotiation. Default is IPv4. Not relevant for RawEth.
• --bind_source_ip — Source IP of the interface used for connection establishment. By default taken from routing table. Not relevant for RawEth.
• --latency_gap=<delay_time> — delay time between each post send. Relevant only for latency.
• --mmap=file — Use an mmap'd file as the buffer for testing P2P transfers. Not relevant for RawEth.
• --mmap-offset=<offset> — The mmap offset. Not relevant for RawEth.
• --mr_per_qp — Create memory region for each qp. Relevant only for bandwidth.
• --odp — Use On Demand Paging instead of Memory Registration. System support required.
• --output=<units> — Set verbosity output level: bandwidth , message_rate, latency. Latency measurement is Average calculation. bw (bandwidth / message_rate), latency (latency).
• --payload_file_path=<payload_txt_file_path> — want a custom payload? pass a txt file with your pattern written little-endian style, like '0xaaaaaaaa, 0xbbbbbbbb, ...' and boom, that's your payload 💖 Not relevant for RawEth and Write latency, bestie — don't come for me.
• --use_old_post_send — uses the old post send flow (ibv_post_send). vintage vibes, retro moment ✨
• --perform_warm_up — runs a few iterations before it starts actually measuring, purely to warm up the memory cache 🫖 like stretching before the big game. Not relevant for raw_ethernet_fs_rate.
• --pkey_index=<pkey index> — the PKey index to use for the QP 🔑 Not relevant for raw_ethernet_fs_rate.
• --report-both — reports RX & TX results separately on Bidirectional BW tests 🎀 Relevant only for bidirectional bandwidth.
• --report_gbits — reports Max/Average BW of the test in Gbit/sec instead of MiB/sec 📊 Relevant only for bandwidth.
• --report-per-port — reports BW data on both ports when you're running Dualport and Duration mode 👀 Not relevant for RawEth. System support required, so check before you get your hopes up.
• --reversed — flips the traffic direction: Server send to client 🔄 main character energy, but reversed.
• --run_infinitely — runs the test forever, printing results every <duration> seconds ♾️ she's committed. she's not leaving.
• --retry_count=<value> — sets the retry count value in rdma_cm mode 🔁 Relevant only for rdma_cm mode. Not relevant for RawEth.
• --tclass=<value> — sets the Traffic Class in GRH (if GRH is in use) 🚦 Not relevant for raw_ethernet_fs_rate.
• --use-null-mr — allocates a null memory region for the client using ibv_alloc_null_mr(3) 🕳️ nothing but vibes in there.
• --use_cuda=<cuda device id> — uses a CUDA specific device for GPUDirect RDMA testing 💚 Not relevant for raw_ethernet_fs_rate. System support required.
• --use_cuda_bus_id=<cuda full BUS id> — uses a CUDA specific device, picked by its full PCIe address, for GPUDirect RDMA testing 🚌 Not relevant for raw_ethernet_fs_rate. System support required.
• --use_cuda_dmabuf — uses CUDA DMA-BUF for GPUDirect RDMA testing 🧚 Not relevant for raw_ethernet_fs_rate. System support required.
• --use_hl=<hl device id> — uses a HabanaLabs specific device for HW accelerator direct RDMA testing 💜 System support required.
• --use_neuron=<logical neuron core id> — uses a Neuron specific device for HW accelerator direct RDMA testing 🧠 System support required.
• --use_neuron_dmabuf — uses Neuron DMA-BUF for HW accelerator direct RDMA testing 🧠✨ System support required.
• --use_rocm=<rocm device id> — uses the selected ROCm device for GPUDirect RDMA testing ❤️ Not relevant for raw_ethernet_fs_rate. System support required.
• --use_hugepages — uses Hugepages instead of contig, memalign allocations 📄 huge pages, huge slay. Not relevant for raw_ethernet_fs_rate.
• --wait_destroy=<seconds> — waits <seconds> before destroying allocated resources (QP/CQ/PD/MR..) ⏳ she needs a moment. Relevant only for bandwidth and raw_ethernet_burst_lat.
• --disable_pcie_relaxed — disables PCIe relaxed ordering 🚫 Relevant only for bandwidth and raw_ethernet_burst_lat. System support required.
• --burst_size=<size> — sets how many messages to send in a burst when you're using the rate limiter 💥 Relevant only for bandwidth and raw_ethernet_burst_lat.
• --typical_pkt_size=<bytes> — sets the size of the packet to send in a burst 📦 only supports the PP rate limiter, no cap. Relevant only for bandwidth and raw_ethernet_burst_lat.
• --rate_limit=<rate> — sets the maximum rate of sent packages. default unit is [Gbps] — use --rate_units to change that 🏎️ Relevant only for bandwidth and raw_ethernet_burst_lat.
• --rate_units=<units> — [Mgp] sets the units for rate limit to MiBps (M), Gbps (g) or pps (p). default is Gbps (g) 📏 Relevant only for bandwidth and raw_ethernet_burst_lat.
• --rate_limit_type=<type> — [HW/SW/PP] limits the QP's by HW, PP or by SW. Disabled by default. When rate_limit is not specified, HW limit is Default ⚙️ Relevant only for bandwidth and raw_ethernet_burst_lat.
• --use_ooo — uses out of order data placement 🔀 chaos, but the good kind. System support required.
• --write_with_imm — uses the write-with-immediate verb instead of write ✍️ Write tests only.

---

🌈 RawEth only options 🌈

okay bestie this whole lil crew is RawEth exclusive, don't try them anywhere else 💅

• -B, --source_mac — Source MAC address, and it has to be in this format: XX:XX:XX:XX:XX:XX. MUST be entered — no skipping, she's required 🎀
• -E, --dest_mac — Destination MAC address, format: XX:XX:XX:XX:XX:XX. MUST be entered too, obviously 💖
• -G, --use_rss — uses RSS on the server side 🌀 need to open 2^x qps (using the -q flag. default is -q 2). open 2^x clients that transmit to this server.
• -J, --dest_ip — Destination ip address, format X.X.X.X for IPv4 or X:X:X:X:X:X for IPv6 (using to send packets with IP header) 🌐 System support required for IPv6.
• -j, --source_ip — Source ip address, format X.X.X.X for IPv4 or X:X:X:X:X:X for IPv6 (using to send packets with IP header) 📮 System support required for IPv6.
• -K, --dest_port — Destination port number (using to send packets with UDP header as default, or you can use the --tcp flag to send TCP Header) 🚪
• -k, --source_port — Source port number (using to send packets with UDP header as default, or you can use the --tcp flag to send TCP Header) 🚪✨
• -Y, --ethertype — Ethertype value in the ethernet frame, format 0xXXXX 🏷️
• -Z, --server — picks the server side for the current machine (--server/--client must be selected ) 👑
• --vlan_en — inserts vlan tag in ethernet header 🏷️💜
• --vlan_pcp — specifies the vlan_pcp value for the vlan tag, 0~7. 8 means different vlan_pcp for each packet 🎚️
• -P, --client — picks the client side for the current machine (--server/--client must be selected) 🫶 Not relevant for raw_ethernet_fs_rate.
• -v, --mac_fwd — runs the mac forwarding test 📬 Not relevant for raw_ethernet_fs_rate.
• --flows — sets the number of TCP/UDP flows, starting from <src_port, dst_port> 🌊 Not relevant for raw_ethernet_fs_rate.
• --flows_burst — sets the number of burst size per TCP/UDP flow 💥 Not relevant for raw_ethernet_fs_rate.
• --promiscuous — runs promiscuous mode 😳 she's talking to everyone. Not relevant for raw_ethernet_fs_rate.
• --reply_every — in latency test, the receiver pongs after a number of received pings 🏓 Not relevant for raw_ethernet_fs_rate.
• --sniffer — runs sniffer mode 👃 Not relevant for raw_ethernet_fs_rate. System support required.
• --flow_label=<fl0, fl1,...> — IPv6 flow label 🏷️ Not relevant for raw_ethernet_fs_rate.
• --tcp — sends TCP Packets. must include IP and Ports information 📦
• --raw_ipv6 — sends IPv6 Packets 🌐 System support required.
• --raw_mcast — 🎉 Relevant only for bandwidth.

---

👑 AUTHORS 👑

the icons who made this happen, give them their flowers fr fr 💐

• Hassan Khadour <hkhadour@nvidia.com>
• Talat Batheesh <talatb@nvidia.com>

---

and that's the whole thing bestie, you made it to the end 🎀🦋 this masterpiece lives in the perftest(1) manual page — go forth and benchmark responsibly, mwah 💅✨