NETMAP(4)

📦 version 12.2, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 NETMAP 💖

okay bestie, welcome to the netmap(4) page — it lives in the Kernel Interfaces Manual, and it is A LOT. so grab a strawberry drink 🍓 and let's go, because this is genuinely one of the most iconic pages in the whole kernel and you're gonna understand all of it.

✨ NAME ✨

netmap — a framework for fast packet I/O 🚀

✨ SYNOPSIS ✨

device netmap

that's it. one line. the whole vibe check passed. 💅

✨ DESCRIPTION ✨

so netmap is a framework for extremely fast and efficient packet I/O — for userspace clients, for kernel clients, and for Virtual Machines. no cap. it runs on FreeBSD, Linux, and some versions of Windows 🪟✨

and it supports a whole little family of netmap ports, bestie, look:

  • physical NIC ports — to access individual queues of network interfaces 🧚
  • host ports — to inject packets into the host stack
  • VALE ports — implementing a very fast and modular in-kernel software switch/dataplane 🔥
  • netmap pipes — a shared memory packet transport channel 🎀
  • netmap monitors — a mechanism similar to bpf(4) to capture traffic 👀

and the slay is that all of these netmap ports are accessed interchangeably with the same API, and they're at least one order of magnitude faster than standard OS mechanisms (sockets, bpf, tun/tap interfaces, native switches, pipes). literally built different.

with suitably fast hardware (NICs, PCIe buses, CPUs), packet I/O using netmap on supported NICs reaches:

  • 14.88 million packets per second (Mpps) with much less than one core on 10 Gbit/s NICs
  • 35-40 Mpps on 40 Gbit/s NICs (limited by the hardware)
  • about 20 Mpps per core for VALE ports
  • over 100 Mpps for netmap pipes 🌈

and NICs without native netmap support can still use the API in emulated mode, which uses unmodified device drivers and is 3-5 times faster than bpf(4) or raw sockets. ate and left no crumbs.

Userspace clients can dynamically switch NICs into netmap mode and send and receive raw packets through memory mapped buffers. similarly, VALE switch instances and ports, netmap pipes and netmap monitors can be created dynamically, providing high speed packet I/O between processes, virtual machines, NICs and the host stack. it's giving ✨ecosystem✨

netmap supports both non-blocking I/O through ioctl(2), synchronization and blocking I/O through a file descriptor, and standard OS mechanisms such as select(2), poll(2), kqueue(2) and epoll(7). all types of netmap ports and the VALE switch are implemented by a single kernel module, which also emulates the netmap API over standard drivers. for best performance, netmap requires native support in device drivers — a list of such devices is at the end of this document. 📋

in the rest of this (long) manual page we document various aspects of the netmap and VALE architecture, features and usage. yes it's long. yes we're doing all of it. 🫶

✨ ARCHITECTURE ✨

netmap supports raw packet I/O through a port, which can be connected to a physical interface (NIC), to the host stack, or to a VALE switch. ports use preallocated circular queues of buffers (rings) residing in an mmapped region. there is one ring for each transmit/receive queue of a NIC or virtual port, and an additional ring pair connects to the host stack. 🌀

after binding a file descriptor to a port, a netmap client can send or receive packets in batches through the rings, and possibly implement zero-copy forwarding between ports. zero copy?? iconic behavior honestly. 💫

all NICs operating in netmap mode use the same memory region, accessible to all processes who own /dev/netmap file descriptors bound to NICs. independent VALE and netmap pipe ports by default use separate memory regions, but can be independently configured to share memory. 👑

✨ ENTERING AND EXITING NETMAP MODE ✨

this section is all about the system calls to create and control netmap ports (including VALE and netmap pipe ports). the simpler, higher level functions are described in the “LIBRARIES” section — that's later, bestie, don't panic. 😌

Ports and rings are created and controlled through a file descriptor, created by opening a special device:

fd = open("/dev/netmap");

and then bound to a specific port with an:

ioctl(fd, NIOCREGIF, (struct nmreq *)arg);

netmap has multiple modes of operation controlled by the struct nmreq argument. arg.nr_name specifies the netmap port name, as follows:

  • OS network interface name (e.g., 'em0', 'eth1', ...) — the data path of the NIC is disconnected from the host stack, and the file descriptor is bound to the NIC (one or all queues), or to the host stack;
  • valeSSS:PPP — the file descriptor is bound to port PPP of VALE switch SSS. switch instances and ports are dynamically created if necessary.
both SSS and PPP have the form [0-9a-zA-Z_]+ , the string cannot exceed IFNAMSIZ characters, and PPP cannot be the name of any existing OS network interface. ⚠️ (that last bit matters, don't be messy)

on return, arg indicates the size of the shared memory region, and the number, size and location of all the netmap data structures, which can be accessed by mmapping the memory:

char *mem = mmap(0, arg.nr_memsize, fd);

non-blocking I/O is done with special ioctl(2) — select(2) and poll(2) on the file descriptor permit blocking I/O. 🦋

while a NIC is in netmap mode, the OS will still believe the interface is up and running. OS-generated packets for that NIC end up into a netmap ring, and another ring is used to send packets into the OS network stack. a close(2) on the file descriptor removes the binding, and returns the NIC to normal mode (reconnecting the data path to the host stack), or destroys the virtual port. bye bye 👋✨

✨ DATA STRUCTURES ✨

the data structures in the mmapped memory region are detailed in <sys/net/netmap.h>, which is the ultimate reference for the netmap API. the main structures and fields are indicated below:

💠 struct netmap_if (one per interface)

struct netmap_if {
    ...
    const uint32_t   ni_flags;      /* properties              */
    ...
    const uint32_t   ni_tx_rings;   /* NIC tx rings            */
    const uint32_t   ni_rx_rings;   /* NIC rx rings            */
    uint32_t         ni_bufs_head;  /* head of extra bufs list */
    ...
};

Indicates the number of available rings (struct netmap_rings) and their position in the mmapped region. the number of tx and rx rings (ni_tx_rings, ni_rx_rings) normally depends on the hardware. NICs also have an extra tx/rx ring pair connected to the host stack. NIOCREGIF can also request additional unbound buffers in the same memory space, to be used as temporary storage for packets. the number of extra buffers is specified in the arg.nr_arg3 field. on success, the kernel writes back to arg.nr_arg3 the number of extra buffers actually allocated (they may be less than the amount requested if the memory space ran out of buffers). ni_bufs_head contains the index of the first of these extra buffers, which are connected in a list (the first uint32_t of each buffer being the index of the next buffer in the list). a 0 indicates the end of the list. the application is free to modify this list and use the buffers (i.e., binding them to the slots of a netmap ring). when closing the netmap file descriptor, the kernel frees the buffers contained in the list pointed by ni_bufs_head , irrespectively of the buffers originally provided by the kernel on NIOCREGIF. 🧠✨

💠 struct netmap_ring (one per ring)

struct netmap_ring {
    ...
    const uint32_t num_slots;   /* slots in each ring            */
    const uint32_t nr_buf_size; /* size of each buffer           */
    ...
    uint32_t       head;        /* (u) first buf owned by user   */
    uint32_t       cur;         /* (u) wakeup position           */
    const uint32_t tail;        /* (k) first buf owned by kernel */
    ...
    uint32_t       flags;
    struct timeval ts;          /* (k) time of last rxsync()     */
    ...
    struct netmap_slot slot[0]; /* array of slots                */
}

Implements transmit and receive rings, with read/write pointers, metadata and an array of slots describing the buffers. 💫

💠 struct netmap_slot (one per buffer)

struct netmap_slot {
    uint32_t buf_idx;           /* buffer index                 */
    uint16_t len;               /* packet length                */
    uint16_t flags;             /* buf changed, etc.            */
    uint64_t ptr;               /* address for indirect buffers */
};

Describes a packet buffer, which normally is identified by an index and resides in the mmapped region. 🎀

💠 packet buffers

Fixed size (normally 2 KB) packet buffers allocated by the kernel.

the offset of the struct netmap_if in the mmapped region is indicated by the nr_offset field in the structure returned by NIOCREGIF. from there, all other objects are reachable through relative references (offsets or indexes). macros and functions in <net/netmap_user.h> help converting them into actual pointers:

struct netmap_if *nifp = NETMAP_IF(mem, arg.nr_offset);
struct netmap_ring *txr = NETMAP_TXRING(nifp, ring_index);
struct netmap_ring *rxr = NETMAP_RXRING(nifp, ring_index);

char *buf = NETMAP_BUF(ring, buffer_index);

and that's where we're pausing, bestie — there's more coming, we're not done yet 💅

🌈 RINGS, BUFFERS AND DATA I/O 🌈

okay bestie so a ring is literally just a circular queue of packets, and it's got three lil indexes/pointers doing the lord's work: head, cur, and tail. and yes, one slot is always kept empty, that's just the rules. also do NOT assume the ring size (num_slots) is a power of two, ok? she's not always gonna be cute like that.

  • head is the first slot available to userspace;
  • cur is the wakeup point: select/poll will unblock when tail passes cur;
  • tail is the first slot reserved to the kernel.

Slot indexes must only move forward 💅 (we don't do backtracking in this house). but for convenience, this function

nm_ring_next(ring, index)

returns the next index modulo the ring size. slay.

head and cur are only modified by the user program; tail is only modified by the kernel. like, boundaries, periodt. the kernel only reads/writes the struct netmap_ring slots and buffers during the execution of a netmap-related system call. the only exception? slots (and buffers) in the range tail ... head-1, which are explicitly assigned to the kernel. those are hers. don't touch.

📤 TRANSMIT RINGS 📤

so on transmit rings, after a netmap system call, slots in the range head ... tail-1 are available for transmission. your user code should fill the slots sequentially and advance head and cur past slots ready to transmit. and cur may be moved further ahead if the user code needs more slots before further transmissions (go read “SCATTER GATHER I/O” for that one).

at the next NIOCTXSYNC/select()/poll(), slots up to head-1 are pushed to the port, and tail may advance if further slots have become available. here's the evolution of a TX ring in all her glory:

after the syscall, slots between cur and tail are (a)vailable
          head=cur   tail
           |          |
           v          v
 TX  [.....aaaaaaaaaaa.............]

user creates new packets to (T)ransmit
            head=cur tail
                |     |
                v     v
 TX  [.....TTTTTaaaaaa.............]

NIOCTXSYNC/poll()/select() sends packets and reports new slots
            head=cur      tail
                |          |
                v          v
 TX  [..........aaaaaaaaaaa........]

select() and poll() will block if there is no space in the ring, i.e.

ring->cur == ring->tail

and return when new slots have become available. so like, they'll just vibe there until there's room 💀

high speed applications may want to amortize the cost of system calls by preparing as many packets as possible before issuing them. get that bag, batch your syscalls.

a transmit ring with pending transmissions has

ring->head != ring->tail + 1 (modulo the ring size).

the function int nm_tx_pending(ring) implements this test. use her, she's handy 🫶

📥 RECEIVE RINGS 📥

on receive rings, after a netmap system call, the slots in the range head... tail-1 contain received packets. user code should process them and advance head and cur past slots it wants to return to the kernel. cur may be moved further ahead if the user code wants to wait for more packets without returning all the previous slots to the kernel. no rush bestie, hold onto what you need.

at the next NIOCRXSYNC/select()/poll(), slots up to head-1 are returned to the kernel for further receives, and tail may advance to report new incoming packets.

and here's the RX ring glow-up in motion:

after the syscall, there are some (h)eld and some (R)eceived slots
       head  cur     tail
        |     |       |
        v     v       v
 RX  [..hhhhhhRRRRRRRR..........]

user advances head and cur, releasing some slots and holding others
           head cur  tail
             |  |     |
             v  v     v
 RX  [..*****hhhRRRRRR...........]

NICRXSYNC/poll()/select() recovers slots and reports new packets
           head cur        tail
             |  |           |
             v  v           v
 RX  [.......hhhRRRRRRRRRRRR....]

🎀 SLOTS AND PACKET BUFFERS 🎀

normally, packets should be stored in the netmap-allocated buffers assigned to slots when ports are bound to a file descriptor. one packet is fully contained in a single buffer. one packet, one buffer, clean and simple 💅

these flags are the ones that affect slot and buffer processing, so pay attention:

  • NS_BUF_CHANGED — must be used when the buf_idx in the slot is changed. this is literally how you implement zero-copy forwarding, go see “ZERO-COPY FORWARDING”.
  • NS_REPORT — reports when this buffer has been transmitted. normally, netmap notifies transmit completions in batches, hence signals can be delayed indefinitely 😭. this flag helps detect when packets have been sent and a file descriptor can be closed.
  • NS_FORWARD — when a ring is in 'transparent' mode, packets marked with this flag by the user application are forwarded to the other endpoint at the next system call, thus restoring (in a selective way) the connection between a NIC and the host stack. iconic behavior honestly.
  • NS_NO_LEARN — tells the forwarding code that the source MAC address for this packet must not be used in the learning bridge code.
  • NS_INDIRECT — indicates that the packet's payload is in a user-supplied buffer whose user virtual address is in the 'ptr' field of the slot. the size can reach 65535 bytes. this is only supported on the transmit ring of VALE ports, and it helps reducing data copies in the interconnection of virtual machines. queen of efficiency ✨
  • NS_MOREFRAG — indicates that the packet continues with subsequent buffers; the last buffer in a packet must have the flag clear. last one out, flag off.

🦋 SCATTER GATHER I/O 🦋

packets can span multiple slots if the NS_MOREFRAG flag is set in all but the last slot. the maximum length of a chain is 64 buffers. this is normally used with VALE ports when connecting virtual machines, as they generate large TSO segments that are not split unless they reach a physical device. big chonky packets, she's carrying a lot.

NOTE: The length field always refers to the individual fragment; there is no place with the total length of a packet. like, the total is just not stored anywhere, do the math yourself bestie.

on receive rings the macro NS_RFRAGS(slot) indicates the remaining number of slots for this packet, including the current one. slots with a value greater than 1 also have NS_MOREFRAG set. so >1 means there's more where that came from 💜

✨ IOCTLS ✨

netmap is lowkey a two-ioctl girlie for non-blocking I/O: NIOCTXSYNC and NIOCRXSYNC. They take no argument, bestie, zero, nada. 🫶

Then there's two more ioctls, NIOCGINFO and NIOCREGIF, that you use to query and configure ports, and they take this argument:

struct nmreq {
    char      nr_name[IFNAMSIZ]; /* (i) port name                  */
    uint32_t  nr_version;        /* (i) API version                */
    uint32_t  nr_offset;         /* (o) nifp offset in mmap region */
    uint32_t  nr_memsize;        /* (o) size of the mmap region    */
    uint32_t  nr_tx_slots;       /* (i/o) slots in tx rings        */
    uint32_t  nr_rx_slots;       /* (i/o) slots in rx rings        */
    uint16_t  nr_tx_rings;       /* (i/o) number of tx rings       */
    uint16_t  nr_rx_rings;       /* (i/o) number of rx rings       */
    uint16_t  nr_ringid;         /* (i/o) ring(s) we care about    */
    uint16_t  nr_cmd;            /* (i) special command            */
    uint16_t  nr_arg1;           /* (i/o) extra arguments          */
    uint16_t  nr_arg2;           /* (i/o) extra arguments          */
    uint32_t  nr_arg3;           /* (i/o) extra arguments          */
    uint32_t  nr_flags           /* (i/o) open mode                */
    ...
};

And listen — a file descriptor you got through /dev/netmap also supports the ioctl that network devices support, so go peek at netintro(4). 💅

🌸 NIOCGINFO 🌸

Returns EINVAL if the port you named is not a netmap supporter. Otherwise it returns 0 plus some (advisory, bestie, just advisory) tea about the port. And like — all of the info below can literally change before the interface is actually put in netmap mode, so don't get attached. 🙈

  • nr_memsize — tells you how big the netmap memory region is. NICs in netmap mode all share one memory region, whereas VALE ports each get their own independent region per port. No sharing, no drama.
  • nr_tx_slots, nr_rx_slots — the size of the transmit and receive rings. That's it, that's the vibe check.
  • nr_tx_rings, nr_rx_rings — how many transmit and receive rings there are. Both the ring numbers and the sizes can be configured at runtime using interface-specific functions (like, say, ethtool(8)). Slay. 🔥

💖 NIOCREGIF 💖

Binds the port named in nr_name to the file descriptor. And if it's a physical device, it also switches it into netmap mode, which disconnects it from the host stack — so yes, that's a real thing that really happens. Multiple file descriptors can be bound to the same port, and the proper synchronization is left entirely up to you, the user. Good luck bestie, I believe in you. 🫶

The recommended way to bind a file descriptor to a port is to use the function nm_open(..) (peep the “LIBRARIES” section) because it parses names to access specific port types and turn features on. Below we document the main features, so let's get into it.

NIOCREGIF can also bind a file descriptor to one endpoint of a netmap pipe, which is made of two netmap ports with a crossover connection. A netmap pipe shares the same memory space as the parent port, and it exists so you can do the whole "master process acts as a dispatcher towards slave processes" situation.

To turn this function on, the nr_arg1 field of the struct can be used as a hint to the kernel saying how many pipes we expect to use, so it can reserve extra space in the memory region.

On return, it gives you the same info as NIOCGINFO, with nr_ringid and nr_flags telling you the identity of the rings controlled through the file descriptor.

nr_flags + nr_ringid select which rings this file descriptor controls. Possible values of nr_flags are listed below, along with the naming schemes that application libraries (like the nm_open mentioned above) can use to point at a specific set of rings. In the examples below, "netmap:foo" is any valid netmap port name. 🦋

  • NR_REG_ALL_NIC netmap:foo — (default) all hardware ring pairs, the whole buffet. 👑
  • NR_REG_SW netmap:foo^ — the ``host rings'', the ones connecting to the host stack.
  • NR_REG_NIC_SW netmap:foo+ — all hardware rings and the host rings. Everything. The full main character moment.
  • NR_REG_ONE_NIC netmap:foo-i — only the i-th hardware ring pair, where that number lives in nr_ringid.
  • NR_REG_PIPE_MASTER netmap:foo{i — the master side of the netmap pipe whose identifier (i) is in nr_ringid.
  • NR_REG_PIPE_SLAVE netmap:foo}i — the slave side of the netmap pipe whose identifier (i) is in nr_ringid.
The identifier of a pipe must be thought of as part of the pipe name, and it does not need to be sequential, ok? On return the pipe will only have a single ring pair with index 0, no matter what the value of i was. Periodt. 💅

By default, a poll(2) or select(2) call pushes out any pending packets on the transmit ring, even if no write events are specified. You can turn that feature off by or-ing NETMAP_NO_TX_POLL into the value you write to nr_ringid. And when that feature is in use, packets are transmitted only on ioctl(NIOCTXSYNC), or when select() / poll() is called with a write event (POLLOUT/wfdset) or a full ring.

When you're registering a virtual interface that's dynamically created to a VALE switch, you can specify the number of rings you want on it (1 by default, and currently up to 16) using the nr_tx_rings and nr_rx_rings fields. 🎀

⭐ NIOCTXSYNC ⭐

Tells the hardware about new packets to transmit, and updates the number of slots available for transmission.

🌈 NIOCRXSYNC 🌈

Tells the hardware about consumed packets, and asks for newly available packets.

🌟 SELECT, POLL, EPOLL, KQUEUE 🌟

select(2) and poll(2) on a netmap file descriptor process rings exactly as described in “TRANSMIT RINGS” and “RECEIVE RINGS” respectively — that's when write (POLLOUT) and read (POLLIN) events are requested. Both of them block if there are no slots available in the ring (ring->cur == ring->tail). And depending on your platform, epoll(7) and kqueue(2) are supported too. ✨

Packets in transmit rings are normally pushed out (and buffers reclaimed) even without you requesting write events. Passing the NETMAP_NO_TX_POLL flag to NIOCREGIF disables that. By default, receive rings are processed only if read events are requested — but passing the NETMAP_DO_RX_POLL flag to NIOCREGIF updates receive rings even without read events. Heads up though: on epoll(7) and kqueue(2), NETMAP_NO_TX_POLL and NETMAP_DO_RX_POLL only do anything when some event is posted for the file descriptor. iykyk. 👑

💜 LIBRARIES 💜

The netmap API is meant to be used directly, both because it's simple and because that's how you get efficient integration with applications. No wrapper needed, bestie, it's already that girl.

But for convenience, the <net/netmap_user.h> header gives you a few macros and functions to make creating a file descriptor and doing I/O with a netmap port way easier. These are loosely modeled after the pcap(3) API, so porting your libpcap-based apps over to netmap is lowkey painless. To use these extra functions, programs should

#define NETMAP_WITH_LIBS

before

#include <net/netmap_user.h>

The following functions are available:

  • struct nm_desc * nm_open(const char *ifname, const struct nmreq *req, uint64_t flags, const struct nm_desc *arg) — similar to pcap_open_live(3), it binds a file descriptor to a port.
    • ifname — is a port name, in the form "netmap:PPP" for a NIC and "valeSSS:PPP" for a VALE port.
    • req — provides the initial values for the argument to the NIOCREGIF ioctl. The nm_flags and nm_ringid values get overwritten by parsing ifname and flags, and other fields can be overridden through the other two arguments.
    • arg — points to a struct nm_desc containing arguments (e.g., from a previously open file descriptor) that should override the defaults. The fields are used as described below.
    • flags — can be set to a combination of these flags: NETMAP_NO_TX_POLL, NETMAP_DO_RX_POLL (copied into nr_ringid); NM_OPEN_NO_MMAP (if arg points to the same memory region, it avoids the mmap and uses the values from it); NM_OPEN_IFNAME (ignores ifname and uses the values in arg); NM_OPEN_ARG1, NM_OPEN_ARG2, NM_OPEN_ARG3 (uses the fields from arg); NM_OPEN_RING_CFG (uses the ring number and sizes from arg).
  • int nm_close(struct nm_desc *d) — closes the file descriptor, unmaps memory, frees resources. Cleanup queen. 🧹
  • int nm_inject(struct nm_desc *d, const void *buf, size_t size) — similar to pcap_inject(), pushes a packet to a ring, returns the size of the packet if successful, or 0 on error;
  • int nm_dispatch(struct nm_desc *d, int cnt, nm_cb_t cb, u_char *arg) — similar to pcap_dispatch(), applies a callback to incoming packets
  • u_char * nm_nextpkt(struct nm_desc *d, struct nm_pkthdr *hdr) — similar to pcap_next(), fetches the next packet

👑 SUPPORTED DEVICES 👑

netmap natively supports these devices, no cap:

On FreeBSD: cxgbe(4), em(4), iflib(4) (providing igb, em and lem), ixgbe(4), ixl(4), re(4), vtnet(4).

On Linux: e1000, e1000e, i40e, igb, ixgbe, ixgbevf, r8169, virtio_net, vmxnet3.

NICs without native support can still be used in netmap mode through emulation. Performance is inferior to native netmap mode but still significantly higher than various raw socket types (bpf, PF_PACKET, etc.). And note — for slow devices (like 1 Gbit/s and slower NICs, or several 10 Gbit/s NICs whose hardware can't sustain line rate), emulated and native mode will likely have similar or the same throughput anyway. So it's giving... fine. 💀

When emulation is in use, packet sniffer programs such as tcpdump could see received packets before they are diverted by netmap. This behaviour is not intentional — it's just an artifact of how emulation is implemented, ok, don't @ us. And note that if the netmap application subsequently moves packets received from the emulated adapter onto the host RX ring, the sniffer will intercept those packets again, since the packets are injected to the host stack as they were received by the network interface. 🙈

Emulation is also available for devices that do have native netmap support, which you can use for testing or performance comparison. The sysctl variable dev.netmap.admode globally controls how netmap mode is implemented. 🌸

✨ SYSCTL VARIABLES AND MODULE PARAMETERS ✨

okay so bestie, netmap and VALE have a whole lil control panel 💅 — on FreeBSD it's sysctl variables under dev.netmap.*, and on Linux it's module parameters in /sys/module/netmap/parameters/*. Here's the whole list, no skips:

  • dev.netmap.admode (default 0) — controls whether we use native or emulated adapter mode, and the values are:
    • 0 — just use the best option available, let it cook 🍓
    • 1 — force native mode, and fail if it's not available
    • 2 — force emulated, so it literally never fails 💖
  • dev.netmap.generic_rings (default 1) — how many rings get used for emulated netmap mode.
  • dev.netmap.generic_ringsize (default 1024) — the ring size for emulated netmap mode.
  • dev.netmap.generic_mit (default 100000) — controls interrupt moderation in emulated mode.
  • dev.netmap.fwd (default 0) — forces NS_FORWARD mode.
  • dev.netmap.txsync_retry (default 2) — number of txsync loops in the VALE flush function.
  • dev.netmap.no_pendintr (default 1) — forces recovery of transmit buffers on system calls.
  • dev.netmap.no_timestamp (default 0) — disables updating the timestamp in the netmap ring.
  • dev.netmap.verbose (default 0) — verbose kernel messages, i.e. the kernel being chatty 🗣️
  • dev.netmap.buf_num (default 163840)
  • dev.netmap.buf_size (default 2048)
  • dev.netmap.ring_num (default 200)
  • dev.netmap.ring_size (default 36864)
  • dev.netmap.if_num (default 100)
  • dev.netmap.if_size (default 1024) — these are the sizes and numbers of objects (netmap_if, netmap_ring, buffers) for the global memory region. The only one of these actually worth touching is dev.netmap.buf_num, because it's the one that changes how much total memory netmap eats. 🫶
  • dev.netmap.buf_curr_num (default 0)
  • dev.netmap.buf_curr_size (default 0)
  • dev.netmap.ring_curr_num (default 0)
  • dev.netmap.ring_curr_size (default 0)
  • dev.netmap.if_curr_num (default 0)
  • dev.netmap.if_curr_size (default 0) — these ones are the actual values in use. iykyk ✨
  • dev.netmap.priv_buf_num (default 4098)
  • dev.netmap.priv_buf_size (default 2048)
  • dev.netmap.priv_ring_num (default 4)
  • dev.netmap.priv_ring_size (default 20480)
  • dev.netmap.priv_if_num (default 2)
  • dev.netmap.priv_if_size (default 1024) — sizes and numbers of objects (netmap_if, netmap_ring, buffers) for private memory regions. Every VALE port gets its own memory region, and so does every pair of netmap pipes. main character energy for each one 💅
  • dev.netmap.bridge_batch (default 1024) — batch size used when moving packets across a VALE switch. Values above 64 generally guarantee good performance, so don't be stingy 👑
  • dev.netmap.ptnet_vnet_hdr (default 1) — allows ptnet devices to use virtio-net headers.

🦋 SYSTEM CALLS 🦋

netmap uses select(2), poll(2), epoll(7) and kqueue(2) to wake processes up when significant events happen, and mmap(2) to map memory. And ioctl(2) is what configures ports and VALE switches.

Apps might need to spin up threads and bind them to specific cores to get better performance — that's just standard OS primitives, so go look at pthread(3). Specifically, pthread_setaffinity_np(3) might be exactly your girl. 🌸

💖 EXAMPLES 💖

🎀 TEST PROGRAMS 🎀

netmap ships with a few programs you can use for testing or as simple applications. Check the examples/ directory in netmap distributions, or tools/tools/netmap/ in FreeBSD distributions.

pkt-gen(8) is your general purpose traffic source/sink. For example:

pkt-gen -i ix0 -f tx -l 60

that one generates an infinite stream of minimum size packets ✨ and:

pkt-gen -i ix0 -f rx

that one is a traffic sink. Both of them print traffic statistics, which helps you monitor how the system is doing. 👑

pkt-gen(8) has a ton of options you can use to set packet sizes, addresses, rates, and to use multiple send/receive threads and cores.

bridge(4) is another test program, and it interconnects two netmap ports. You can use it for transparent forwarding between interfaces, like:

bridge -i netmap:ix0 -i netmap:ix1

or you can even connect the NIC to the host stack using netmap:

bridge -i netmap:ix0

🌈 USING THE NATIVE API 🌈

The following code implements a traffic generator:

#include <net/netmap_user.h>
...
void sender(void)
{
    struct netmap_if *nifp;
    struct netmap_ring *ring;
    struct nmreq nmr;
    struct pollfd fds;

    fd = open("/dev/netmap", O_RDWR);
    bzero(&nmr, sizeof(nmr));
    strcpy(nmr.nr_name, "ix0");
    nmr.nm_version = NETMAP_API;
    ioctl(fd, NIOCREGIF, &nmr);
    p = mmap(0, nmr.nr_memsize, fd);
    nifp = NETMAP_IF(p, nmr.nr_offset);
    ring = NETMAP_TXRING(nifp, 0);
    fds.fd = fd;
    fds.events = POLLOUT;
    for (;;) {
        poll(&fds, 1, -1);
        while (!nm_ring_empty(ring)) {
            i = ring->cur;
            buf = NETMAP_BUF(ring, ring->slot[i].buf_index);
            ... prepare packet in buf ...
            ring->slot[i].len = ... packet length ...
            ring->head = ring->cur = nm_ring_next(ring, i);
        }
    }
}

🧚 HELPER FUNCTIONS 🧚

A simple receiver can be implemented using the helper functions:

#define NETMAP_WITH_LIBS
#include <net/netmap_user.h>
...
void receiver(void)
{
    struct nm_desc *d;
    struct pollfd fds;
    u_char *buf;
    struct nm_pkthdr h;
    ...
    d = nm_open("netmap:ix0", NULL, 0, 0);
    fds.fd = NETMAP_FD(d);
    fds.events = POLLIN;
    for (;;) {
        poll(&fds, 1, -1);
        while ( (buf = nm_nextpkt(d, &h)) )
            consume_pkt(buf, h->len);
    }
    nm_close(d);
}

💜 ZERO-COPY FORWARDING 💜

Since physical interfaces share the same memory region, you can totally do packet forwarding between ports by just swapping buffers. 🔥 The buffer from the transmit ring gets used to replenish the receive ring:

uint32_t tmp;
struct netmap_slot *src, *dst;
...
src = &src_ring->slot[rxr->cur];
dst = &dst_ring->slot[txr->cur];
tmp = dst->buf_idx;
dst->buf_idx = src->buf_idx;
dst->len = src->len;
dst->flags = NS_BUF_CHANGED;
src->buf_idx = tmp;
src->flags = NS_BUF_CHANGED;
rxr->head = rxr->cur = nm_ring_next(rxr, rxr->cur);
txr->head = txr->cur = nm_ring_next(txr, txr->cur);
...

⭐ ACCESSING THE HOST STACK ⭐

The host stack is, for all practical purposes, just a regular ring pair — and you can access it with the netmap API, e.g. with:

nm_open("netmap:eth0^", ...);

Every packet the host would send to an interface in netmap mode ends up in the RX ring, whereas every packet queued to the TX ring gets sent up to the host stack. That's the whole vibe. 💫

👑 VALE SWITCH 👑

A simple way to test the performance of a VALE switch is to attach a sender and a receiver to it — like, run these two in two different terminals:

pkt-gen -i vale1:a -f rx # receiver
pkt-gen -i vale1:b -f tx # sender

The same example works for testing netmap pipes, you just change the port names, e.g.:

pkt-gen -i vale2:x{3 -f rx # receiver on the master side
pkt-gen -i vale2:x}3 -f tx # sender on the slave side

And this next command attaches an interface and the host stack to a switch:

valectl -h vale2:em0

Other netmap clients attached to the same switch can now talk to the network card or the host. Communication slay. 🫶

🌸 SEE ALSO 🌸

  • vale(4), valectl(8), bridge(8), lb(8), nmreplay(8), pkt-gen(8)
  • http://info.iet.unipi.it/~luigi/netmap/
  • Luigi Rizzo, Revisiting network I/O APIs: the netmap framework, Communications of the ACM, 55 (3), pp.45-51, March 2012
  • Luigi Rizzo, netmap: a novel framework for fast packet I/O, Usenix ATC'12, June 2012, Boston
  • Luigi Rizzo, Giuseppe Lettieri, VALE, a switched ethernet for virtual machines, ACM CoNEXT'12, December 2012, Nice
  • Luigi Rizzo, Giuseppe Lettieri, Vincenzo Maffione, Speeding up packet I/O in virtual machines, ACM/IEEE ANCS'13, October 2013, San Jose

💅 AUTHORS 💅

The netmap framework was originally designed and implemented at the Universita` di Pisa in 2011 by Luigi Rizzo, and then extended further with help from Matteo Landi, Gaetano Catalli, Giuseppe Lettieri, and Vincenzo Maffione. absolute legends, all of them 👑

netmap and VALE have been funded by the European Commission within FP7 Projects CHANGE (257422) and OPENLAB (287581).

😭 CAVEATS 😭

No matter how fast your CPU and OS are, hitting line rate on 10G and faster interfaces requires hardware with sufficient performance. Several NICs straight up cannot sustain line rate with small packet sizes. Not enough PCIe or memory bandwidth can also tank your performance. 💀

Another super common reason for low performance is flow control on the link: a slow receiver can cap the transmit speed. So be sure to disable flow control when you're running high speed experiments. fr fr.

🦋 SPECIAL NIC FEATURES 🦋

netmap is orthogonal to some NIC features, like multiqueue, schedulers, and packet filters.

Multiple transmit and receive rings are supported natively, and you can configure them with ordinary OS tools — like ethtool(8) or device-specific sysctl variables. Same deal for Receive Packet Steering (RPS) and filtering of incoming traffic.

netmap does not use features like checksum offloading, TCP segmentation offloading, encryption, VLAN encapsulation/decapsulation, etc. So when you're using netmap to exchange packets with the host stack, make sure to disable those features. 🫶

and that's the whole page bestie, we did it 💖 this one's from the Debian crowd, dated February 6, 2020, and it's NETMAP(4) — go forth and packet-slay ✨👑

📦 where this page comes from