ENA(4)

📦
version 15.1 · 2 versions
, in freebsd 14.5, freebsd 15.1, nixpkgs unstable

✨ ENA(4) — the AWS EC2 Elastic Network Adapter driver, bestie edition ✨

okay so this is ena(4), straight out of the Kernel Interfaces Manual, and it's genuinely one of the slay-est network drivers out there — it's the AWS EC2 Elastic Network Adapter (ENA) driver. buckle up 🎀

🌸 NAME 🌸

ena — the AWS EC2 Elastic Network Adapter (ENA) driver. that's her. that's the whole name. 💅

✨ SYNOPSIS ✨

so there are two ways to get her in your life, and honestly both are valid.

option one: bake it into the kernel. drop this line into your kernel configuration file:

device ena

option two: load it as a module at boot time. for that you go into loader.conf(5) and add:

if_ena_load="YES"

pick your fighter 🫶

💖 DESCRIPTION 💖

okay so the ENA is a networking interface that's literally designed to make good use of modern CPU features and system architectures. like, she was built for this. no cap.

the ENA device exposes a lightweight management interface — and the way that it does this is with a minimal set of memory mapped registers plus an extendable command set through an Admin Queue. minimal effort, maximum vibes.

the driver supports a whole range of ENA devices, and here's the iconic part: it's link-speed independent. meaning the same driver is used for 10GbE, 25GbE, 40GbE, etc. — she doesn't play favorites. she also has a negotiated and extendable feature set, which is giving main character energy fr fr. 🌈

some ENA devices support SR-IOV. this driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. equal opportunity icon 💜

so how do the ENA devices get that high speed and low overhead network traffic processing? glad you asked, bestie:

  • multiple Tx/Rx queue pairs — and the maximum number is advertised by the device via the Admin Queue, iykyk
  • a dedicated MSI-X interrupt vector per Tx/Rx queue pair
  • CPU cacheline optimized data placement

okay and when RSS is enabled, each Tx/Rx queue pair is bound to a corresponding CPU core and its NUMA domain. the order of those bindings is based on the RSS bucket mapping. for builds with RSS support disabled, the CPU and NUMA management is left to the kernel. receive-side scaling (RSS) is supported for multi-core scaling, which is lowkey the whole point 🔥

the ena driver and its corresponding devices implement health monitoring mechanisms such as watchdog — enabling the device and driver to recover in a manner transparent to the application — plus debug logs. she heals herself and journals about it, honestly obsessed 😭

some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. several! more! microseconds! 👑

support for the netmap(4) framework is provided by the ena driver too. but — and this is important — your kernel must be built with the DEV_NETMAP option to be able to use this feature. no option, no netmap, periodt.

🦋 HARDWARE 🦋

the ena driver supports the following PCI vendor ID/device IDs:

  • 1d0f:0ec2 — ENA PF
  • 1d0f:1ec2 — ENA PF with LLQ support
  • 1d0f:ec20 — ENA VF
  • 1d0f:ec21 — ENA VF with LLQ support

all four accounted for, not a single one skipped, we love a complete list 🍓

✨ LOADER TUNABLES ✨

Okay bestie, the ena driver's behavior can be changed using run-time or boot-time sysctl arguments. The boot-time arguments can be set at the loader(8) prompt before booting the kernel, or stored in loader.conf(5). The run-time arguments can be set using the sysctl(8) command. slay, now let's get into the knobs 💅

🛠️ Boot-time tunables 🛠️

  • hw.ena.enable_9k_mbufs — Use 9k mbufs for the Rx descriptors. The default is 0. If the node value is set to 1, 9k mbufs will be used for the Rx buffers. If set to 0, page size mbufs will be used instead.

Using 9k buffers for Rx can improve Rx throughput, but in low memory conditions it might increase allocation time, as the system has to look for 3 contiguous pages. This can further lead to OS instability, together with ENA driver reset and NVMe timeouts. If network performance is critical and memory capacity is sufficient, the 9k mbufs can be used. iykyk 🫶

  • hw.ena.force_large_llq_header — Force the driver to use large (224 bytes) or regular (96 bytes) LLQ header size. The default value is 2 and the recommended LLQ header size will be used. If the node value is set to 0, the regular size LLQ header will be used, which is 96B. In some cases, the packet header can be bigger than this (for example - IPv6 with multiple extensions). In such a situation, the large LLQ header size which is 224B should be used, and can be forced by setting this node value to 1. Using large LLQ header size will take effect only if the device supports both LLQ and large LLQ headers. Otherwise, it will fallback to the no LLQ mode or regular header size.

Heads up: increasing LLQ header size reduces the size of the Tx queue by half, so it may affect the number of dropped Tx packets.

⚡ Run-time tunables ⚡

  • hw.ena.log_level — Controls extra logging verbosity of the driver. The default is 2. The higher the logging level, the more logs will be printed out. 0 means all extra logs are disabled and only error logs will be printed out. Default value (2) reports errors, warnings and is verbose about driver operation.

The possible flags are:

  • 0 - ENA_ERR — Enable driver error messages and ena_com error logs.
  • 1 - ENA_WARN — Enable logs for non-critical errors.
  • 2 - ENA_INFO — Make the driver more verbose about its actions.
  • 3 - ENA_DBG — Enable debug logs.
NOTE: In order to enable logging on the Tx/Rx data path, driver must be compiled with ENA_LOG_IO_ENABLE compilation flag. <not me forgetting the compile flag 😭>

Example: To enable logs for errors and warnings, the following command should be used: sysctl hw.ena.log_level=1

  • dev.ena.X.io_queues_nb — Number of the currently allocated and used IO queues. The default is max_num_io_queues. Controls the number of IO queue pairs (Tx/Rx). As this call has to reallocate the queues, it will reset the interface and restart all the queues - this means that everything, which was currently held in the queue, will be lost, leading to potential packet drops. fr fr, it's giving “clean slate but also maybe packet loss” energy.

This call can fail if the system isn't able to provide the driver with enough resources. In that situation, the driver will try to revert the previous number of the IO queues. If this also fails, the device reset will be triggered.

Example: To use only 2 Tx and Rx queues for the device ena1, the following command should be used: sysctl dev.ena.1.io_queues_nb=2

  • dev.ena.X.rx_queue_size — Size of the Rx queue. The default is 1024. Controls the number of IO descriptors for each Rx queue. The user may want to increase the Rx queue size if they observe a high number of Rx drops in the driver's statistics. For performance reasons, the Rx queue size must be a power of 2. periodt, powers of 2 only 💅

This call can fail if the system isn't able to provide the driver with enough resources. In that situation, the driver will try to revert to the previous number of the descriptors. If this also fails, the device reset will be triggered.

Example: To increase Rx ring size to 8K descriptors for the device ena0, the following command should be used: sysctl dev.ena.0.rx_queue_size=8192

  • dev.ena.X.buf_ring_size — Size of the Tx buffer ring (drbr). The default is 4096. Input must be a power of 2. Controls the number of mbufs that can be held in the Tx buffer ring. The drbr is used as a multiple-producer, single-consumer lockless ring for buffering extra mbufs coming from the stack in case the Tx procedure is busy sending the packets, or the Tx ring is full. Increasing the size of the buffer ring may reduce the number of Tx packets being dropped in case of a big Tx burst, which cannot be handled by the IO queue immediately. Each Tx queue has its own drbr.

It is recommended to keep the drbr with at least the default value, but in case the system lacks the resources, it can be reduced. This call can fail if the system is not able to provide the driver with enough resources. In that situation, the driver will try to revert to the previous number of the drbr and trigger the device reset.

Example: To set drbr size for interface ena0 to 2048, the following command should be used: sysctl dev.ena.0.buf_ring_size=2048

  • dev.ena.X.eni_metrics.sample_interval — Interval in seconds for updating ENI metrics. The default is 0. Determines how often (if ever) the ENI metrics should be updated. The ENI metrics are being updated asynchronously in a timer service in order to avoid admin queue overload by sysctl node reading. The value in this node controls the interval between issuing admin commands to the device, which will update the ENI metrics values.

If some application is periodically monitoring the eni_metrics, then the ENI metrics interval can be adjusted accordingly. Value 0 turns off the update completely. Value 1 is the minimum interval and is equal to 1 second. The maximum allowed update interval is 1 hour. obsessed with a clean little timer service ngl 🦋

Example: To update ENI metrics for the device ena1 every 10 seconds, the following command should be used: sysctl dev.ena.1.eni_metrics.sample_interval=10

  • dev.ena.X.rss.indir_table_size — RSS indirection table size. The default is 128. Returns the number of entries in the RSS indirection table.

Example: To read the RSS indirection table size, the following command should be used: sysctl dev.ena.0.rss.indir_table_size

  • dev.ena.X.rss.indir_table — RSS indirection table mapping. The default is x:y key-pairs of indir_table_size length. Updates selected indices of the RSS indirection table.

The entry string consists of one or more x:y keypairs, where x stands for the table index and y for its new value. Table indices that don't need to be updated can be omitted from the string and will retain their existing values.

If an index is entered more than once, the last value is used. the way that's literally just “last one wins” main character energy 💜

Example: To update two selected indices in the RSS indirection table, e.g. setting index 0 to queue 5 and then index 5 to queue 0, the following command should be used: sysctl dev.ena.0.rss.indir_table="0:5 5:0"

  • dev.ena.X.rss.key — RSS hash key. The default is 40 bytes long randomly generated hash key. Controls the RSS Toeplitz hash algorithm key value.

Only available when driver compiled without the kernel side RSS support.

Example: To change the RSS hash key value to

0x6d, 0x5a, 0x56, 0xda, 0x25, 0x5b, 0x0e, 0xc2,
0x41, 0x67, 0x25, 0x3d, 0x43, 0xa3, 0x8f, 0xb0,
0xd0, 0xca, 0x2b, 0xcb, 0xae, 0x7b, 0x30, 0xb4,
0x77, 0xcb, 0x2d, 0xa3, 0x80, 0x30, 0xf2, 0x0c,
0x6a, 0x42, 0xb7, 0x3b, 0xbe, 0xac, 0x01, 0xfa

the following command should be used:

sysctl dev.ena.0.rss.key=6d5a56da255b0ec24167253d43a38fb0d0ca2bcbae7b30b477cb2da38030f20c6a42b73bbeac01fa

💅 DIAGNOSTICS 💅

okay bestie, this is the part where the driver talks to you. every single message here is something ena can print, and like — they're all lowkey yelling about something. here's the full translation, no cap 🫶

🚨 Device initialization phase 🚨

ena%d: failed to init mmio read less

so this one means an error happened during initialization of the mmio register read request. yeah. that's the whole vibe. 😭

ena%d: Can not reset device

the device could not be reset. it may not be responding, or it's already chillin' mid-reset. awkward.

ena%d: device version is too low

the controller version is too old and the driver straight up does not support it. it's giving "we don't serve your kind here" 💀

ena%d: Invalid dma width value %d

the controller can't request the dma transaction width. either the device stopped responding, or it demanded an invalid value. rude.

ena%d: Can not initialize ena admin queue with device

initialization of the Admin Queue failed. the device may not be responding, or there was a problem with initialization of the resources. 😬

ena%d: Cannot get attribute for ena device rc: %d

failed to get attributes of the device from the controller. no attributes, no info, no nothing.

ena%d: Cannot configure aenq groups rc: %d

errors occurred when trying to configure AENQ groups. that's it, that's the message 🎀

🔧 Driver initialization/shutdown phase 🔧

ena%d: PCI resource allocation failed!
ena%d: failed to pmap registers bar
ena%d: can not allocate ifnet structure
ena%d: Error with network interface setup
ena%d: Failed to enable and set the admin interrupts
ena%d: Error, MSI-X is already enabled
ena%d: Failed to enable MSIX, vectors %d rc %d
ena%d: Not enough number of MSI-X allocated: %d
ena%d: Error with MSI-X enablement
ena%d: could not allocate irq vector: %d
ena%d: unable to allocate bus resource: registers!
ena%d: unable to allocate bus resource: msix!

resource allocation failed when initializing the device, and the driver will not be attached. like it just walks away. 😔

ena%d: ENA device init failed (err: %d)
ena%d: Cannot initialize device

device initialization failed → driver will not be attached. same energy, different line.

ena%d: failed to register interrupt handler for irq %ju: %d

an error occurred when trying to register the Admin Queue interrupt handler. 🙈

ena%d: Cannot setup mgmnt queue intr

error occurred during configuration of the Admin Queue interrupts.

ena%d: Enable MSI-X failed

configuration of the MSI-X for Admin Queue failed. there could be a lack of resources, or the interrupts could not have been configured. driver will not be attached. periodt.

ena%d: VLAN is in use, detach first

so you're trying to detach the driver while VLANs are in use. the VLANs must be detached first, and then the detach routine has to be called again. order matters, bestie 💅

ena%d: Unmapped RX DMA tag associations
ena%d: Unmapped TX DMA tag associations

error occurred when trying to destroy the RX/TX DMA tag.

ena%d: Cannot init indirect table
ena%d: Cannot fill indirect table
ena%d: Cannot fill hash function
ena%d: Cannot fill hash control
ena%d: WARNING: RSS was not properly initialized, it will affect bandwidth

error occurred during initialization of one of the RSS resources. the device will still work, but with reduced performance, because all RX packets get passed to queue 0 and there'll be no hash information. it's giving "works but sad" 😭

ena%d: LLQ is not supported. Fallback to host mode policy.
ena%d: Failed to configure the device mode. Fallback to host mode policy.
ena%d: unable to allocate LLQ bar resource. Fallback to host mode policy.

error occurred during Low-latency Queue mode setup. the device will work — just without the LLQ performance gain. fallback era 🍓

ena%d: failed to enable write combining.

error occurred while setting the Write Combining mode, which is required for the LLQ.

ena%d: failed to tear down irq: %d
ena%d: dev has no parent while releasing res for irq: %d

release of the interrupts failed. that's the whole story.

🔍 Additional diagnostic 🔍

ena%d: Invalid MTU setting. new_mtu: %d max_mtu: %d min mtu: %d

the requested MTU value is not supported and will not be set. no MTU for you 🎀

ena%d: Failed to set MTU to %d

this message appears when either the MTU change feature is not supported, or a device communication error has occurred.

ena%d: Keep alive watchdog timeout.

the device stopped responding and will be reset. it's not ghosting you, it's actually dead 💀

ena%d: Found a Tx that wasn't completed on time, qid %d, index %d.

a packet was pushed to the NIC but not sent within the given time limit. it may be caused by a hang of the IO queue.

ena%d: The number of lost tx completion is above the threshold (%d > %d). Reset the device

if too many Tx weren't completed on time, the device is going to be reset. it may be caused by a hanged queue or a hanged device. either way, reset time 🔥

ena%d: Trigger reset is on

the device will be reset. the reset is triggered either by the watchdog or because too many TX packets were not completed on time.

ena%d: device reset scheduled but trigger_reset is off

the reset task has been triggered, but the driver did not request it. so the device reset will not be performed. false alarm, everyone go home 🙈

ena%d: Device reset failed

error occurred while trying to reset the device. oof.

ena%d: Cannot initialize device
ena%d: Error, mac address are different
ena%d: Error, device max mtu is smaller than ifp MTU
ena%d: Validation of device parameters failed
ena%d: Enable MSI-X failed
ena%d: Failed to create I/O queues
ena%d: Reset attempt failed. Can not reset the device

error occurred while trying to restore the device after reset. it tried, it really did 😭

ena%d: Device reset completed successfully, Driver info: %s

okay this one's a slay 🎉 the device has been correctly restored after reset and is ready to use. main character moment fr.

ena%d: Allocation for Tx Queue %u failed
ena%d: Allocation for Rx Queue %u failed
ena%d: Unable to create Rx DMA map for buffer %d
ena%d: Failed to create io TX queue #%d rc: %d
ena%d: Failed to get TX queue handlers. TX queue num %d rc: %d
ena%d: Failed to create io RX queue[%d] rc: %d
ena%d: Failed to get RX queue handlers. RX queue num %d rc: %d
ena%d: could not allocate irq vector: %d
ena%d: failed to register interrupt handler for irq %ju: %d

IO resources initialization failed. the interface will not be brought up. nope. not happening. 💀

ena%d: LRO[%d] Initialization failed!

initialization of the LRO for the RX ring failed.

ena%d: failed to alloc buffer for rx queue
ena%d: failed to add buffer for rx queue %d
ena%d: refilled rx qid %d with only %d mbufs (from %d)

allocation of resources used on the RX path failed. if this happened during initialization of the IO queue, the interface will not be brought up.

ena%d: NULL mbuf in rx_info

error occurred while assembling mbuf from descriptors.

ena%d: tx_info doesn't have valid mbuf
ena%d: Invalid req_id: %hu
ena%d: failed to prepare tx bufs

error occurred while preparing a packet for transmission. the packet is not packet-ing 😭

ena%d: ioctl promisc/allmulti

this is the IOCTL request for the device to work in promiscuous/allmulti mode. see ifconfig(8) for more details. iykyk 💜

🫶 SUPPORT 🫶

if an issue is identified with the released source code with a supported adapter, please email the specific information related to the issue to ⟨akiyano@amazon.com⟩, ⟨osamaabb@amazon.com⟩ and ⟨darinzon@amazon.com⟩. they will literally help you, bestie 💖

👀 SEE ALSO 👀

🌸 HISTORY 🌸

the ena driver first appeared in FreeBSD 11.1. vintage era, honestly.

👑 AUTHORS 👑

the ena driver was developed by Amazon and originally written by Semihalf. iconic collaboration tbh 🎀

and that's the whole page, bestie! this one ships from the Debian crew, dated November 14, 2024, and it's ENA(4) — go forth and debug with main character energy ✨🦋💅

📦 where this page comes from