netpipe(1)

🦋 netpipe(1): the NetPIPE manual, bestie 🦋

✨ NAME ✨

NetPIPE - Network Protocol Independent Performance Evaluator 💖

📜 SYNOPSIS ✨

NPtcp [-h receiver_hostname] [-b TCP_buffer_sizes] [options]

mpirun [-machinefile hostlist] -np 2 NPmpi [-a] [-S] [-z] [options]

mpirun [-machinefile hostlist] -np 2 NPmpi2 [-f] [-g] [options]

NPpvm [options]

Okay so the TESTING sections below are where the real deets live for each setup 🧚. The OPTIONS section covers the general stuff that works across all modules. If you want docs on the InfiniBand, GM, SHMEM, LAPI, and memcpy modules, check the README in the tar-ball at http://www.scl.ameslab.gov/Projects/NetPIPE/ 📖.

💅 DESCRIPTION 💅

NetPIPE basically runs a simple series of ping-pong tests across a range of message sizes, and that's how it gives you a complete read on how well a network performs 🔥. It bounces messages of growing size between two processes, whether they're talking across a network or hanging out inside one SMP system. The sizes get picked at regular intervals, with slight perturbations, so the whole communication system gets evaluated. Each data point runs a ton of ping-pong tests so the timing is accurate 🎯. Latencies are calculated by dividing the round trip time in half for small messages ( less than 64 Bytes ).

Small messages are mostly eaten up by overhead in the communication layers, which means the transmission is latency bound 💀. For bigger messages, the rate becomes bandwidth limited by some component in the communication subsystem, like the PCI bus, the network card link, or the network switch. Main character energy from the switch, honestly 👑.

These measurements can happen at the message-passing layer (MPI, MPI-2, and PVM) or at the native communications layers that run upon it (TCP/IP, GM for Myrinet cards, InfiniBand, SHMEM for the Cray T3E systems, and LAPI for IBM SP systems) 🌈. Recent work is also aimed at measuring internal system properties, like the memcpy module that measures internal memory copy rates, or a disk module under development that measures performance to various I/O devices 🍓.

Some uses for NetPIPE include:

• Comparing the latency and maximum throughput of various network cards.
• Comparing the performance between different types of networks.
• Looking for inefficiencies in the message-passing layer by comparing it to the native communication layer.
• Optimizing the message-passing layer and tuning OS and driver parameters for optimal performance of the communication subsystem.

NetPIPE comes with many modules so it can interface with a wide variety of communication layers 🦋. It's fairly easy to write new interfaces for other reliable protocols too, just use the existing modules as examples. ✨

🌸 TESTING TCP 🌸

NPtcp can be launched two ways: by manually starting NPtcp on both systems, or by using an nplaunch script 💖. To start it manually, the NetPIPE receiver has to be started first on the remote system, using:

NPtcp [options]

Then the primary transmitter gets started on the local system with:

NPtcp -h receiver_hostname [options]

Any options used must be the same on both sides 🫶.

The nplaunch script uses ssh to launch the remote receiver before starting the local transmitter. Want rsh instead? Just change the nplaunch script 🔧.

nplaunch NPtcp -h receiver_hostname [options]

The -b TCP_buffer_sizes option sets the TCP socket buffer size, and it can seriously influence the maximum throughput on some systems 🚀. If your throughput graph flattens out suddenly, that might be a sign the performance is being limited by the socket buffer sizes 👀.

🧚 TESTING MPI and MPI-2 🧚

Using the MPI interface with NetPIPE depends on which MPI implementation you're running 🌷. All of them need the number of processes specified, usually with -np 2. Cluster environments might need a list of the hosts being used every time a job runs. So put the list of hosts in hostlist, then for OpenMPI, run NetPIPE with:

mpirun --hostfile hostlist -np 2 NPmpi [NetPIPE options]

For MPICH2, use this instead:

mpirun -machinefile hostlist -np 2 NPmpi [NetPIPE options]

Wanna test the 1-sided communications of the MPI-2 standard? Compile with:

make mpi2

When you run it like above, MPI will use 1-sided MPI_Put() calls in both directions. Each receiver blocks until the last byte has been overwritten before bouncing the message back 💅. Use -f to force a fence to block instead of relying on an overwrite of the last byte. The -g option uses MP_Get() functions to transfer the data rather than MP_Put() 🦄.

🫧 TESTING PVM 🫧

Start the pvm system with:

pvm

then add a second machine with the PVM command:

add receiver_hostname

Exit the PVM command line interface using quit ✌️. Then run the PVM NetPIPE receiver on one system with:

NPpvm [options]

and run the TCP NetPIPE transmitter on the other system with:

NPpvm -h receiver hostname [options]

Any options used must be the same on both sides 💜. The nplaunch script can also be used with NPpvm, same as described above for NPtcp.

📈 TESTING METHODOLOGY 📈

NetPIPE tests network performance by sending a number of messages at each block size, starting from the lower bound on the message sizes 🎀.

The message size keeps increasing until it hits the upper bound on the message size, or until the time to transmit a block goes over one second, whichever comes first ⏱️. Message sizes are picked at regular intervals, and slight perturbations from those points get tested too, so you get a more complete evaluation of the communication subsystem 🧠.

The NetPIPE output file can be graphed with a program like gnuplot(1) 📊. The file has three columns: the number of bytes in the block, the transfer rate in bits per second, and the time to transfer the block (half the round-trip time). The first two columns are usually what you graph for throughput vs block size, and the third column gives you the latency ✨. For example, you can make the throughput versus block size graph by graphing bytes versus bits per second. Sample gnuplot(1) commands for that graph look like this:

set logscale x

plot "np.out"

🎀 OPTIONS 🎀

• -a — asynchronous mode where receives get preposted (MPI, IB modules) 🌸.
• -b TCP_buffer_sizes — sets the send and receive TCP buffer sizes (TCP module only) 🔧.
• -B — burst mode where all receives get preposted at once (MPI, IB modules) 💥.
• -f — uses a fence to block for completion (MPI2 module only) 🚧.
• -g — uses MPI_Get() instead of MPI_Put() (MPI2 module only) 🔄.
• -h hostname — the name of the receiver host you connect to (TCP, PVM, IB, GM) 🏠.
• -I — invalidates the cache so you can measure performance without cache effects (mostly affects IB and memcpy modules) 🧊.
• -i — runs an integrity check instead of a performance evaluation 🔍.
• -l starting_msg_size — sets the lower bound for the size of messages being tested 🐣.
• -n nrepeats — sets the number of repeats for each test to a constant. Otherwise, the repeat count is picked to give an accurate timing for each test. Be very careful with a low number, so the ping-pong test time still goes over the timer accuracy 😬.
• -O source_offset,dest_offset — sets the source and destination offsets of the buffers from perfect page alignment 📐.
• -o output_filename — sets the output filename (default is np.out) 📝.
• -p perturbation_size — NetPIPE picks message sizes at regular intervals, increasing them exponentially from the lower boundary to the upper one. At each point, it also tests perturbations of 3 bytes above and 3 bytes below each test point, to catch system idiosyncrasies 🕵️. You can change this value with -p, or turn it off with -p 0 . 🙅.
• -r — resets the TCP sockets after every test (TCP module only). It's needed for some streaming tests to get good measurements, since the socket window size might otherwise collapse 🫠.
• -s — streaming mode, where data only goes in one direction ➡️.
• -S — uses synchronous sends (MPI module only) 🤝.
• -u upper_bound — sets the upper boundary on the size of messages being tested. By default, NetPIPE stops when the time to transmit a block goes over one second ⛔.
• -z — receives messages using MPI_ANY_SOURCE (MPI module only) 🎯.
• -2 — bi-directional mode where both sides send and receive at the same time (supported by most modules) 🔁. You might need -a to pick asynchronous communications for MPI so you don't get freeze-ups 🧊. For TCP, the maximum test size is limited by the TCP buffer sizes.

📁 FILES 📁

• np.out — the default output file for NetPIPE. The -o option overrides it 🗂️.

💌 AUTHOR 💌

The original NetPIPE core plus the TCP and MPI modules were written by Quinn Snell, Armin Mikler, Guy Helmer, and John Gustafson 🙌. NetPIPE is currently developed and maintained by Dave Turner, with contributions from many students (Bogdan Vasiliu, Adam Oline, Xuehua Chen, and Brian Smith) 💖.

Send comments and bug reports to: <netpipe@scl.ameslab.gov> 📬.

More info about NetPIPE lives on the World Wide Web at http://www.scl.ameslab.gov/Projects/NetPIPE/ 🌐.

🚨 BUGS 🚨

Heads up! As of version 3.6.1, there is a bug that causes NetPIPE to segfault on RedHat Enterprise systems. Dave will debug it as soon as he gets access to a few of those systems. -Dave Turner (turner@ameslab.gov) 💀

---

That's the whole NetPIPE page from the June 1, 2004 edition, bestie 💜✨ Now you're literally ready to benchmark like a pro 🫶💅. netpipe(1) is iconic, periodt 🌈.