GVE(4)

📦
version 15.1 · 2 versions
, in freebsd 15.1, nixpkgs unstable

💖 GVE(4) — the gve(4) girlie's guide 💖

okay so this is the manual page for gve(4) (that's the gve driver, section 4, kernel interfaces manual ✨) and it's literally the Ethernet driver for the Google Virtual NIC, aka gVNIC. bestie, we're gonna get you networking on Google Compute Engine and it's gonna slay 🫶

✨ SYNOPSIS ✨

so if you wanna compile this driver into the kernel, you drop these lines into your kernel configuration file:

device gve

or, alternatively, if you'd rather load the driver as a module at boot time, you put this line in loader.conf(5):

if_gve_load="YES"

pick your fighter 💅

🦋 DESCRIPTION 🦋

gVNIC is a virtual network interface designed specifically for Google Compute Engine (GCE). no cap. it's required to support per-VM Tier-1 networking performance, and also for using certain VM shapes on GCE — so like, it's not optional vibes, it's a whole requirement.

gve is the driver for gVNIC. it supports the following features, and honestly? iconic lineup:

  • RX checksum offload
  • TX chesksum offload
  • TCP Segmentation Offload (TSO)
  • Large Receive Offload (LRO) in software
  • Jumbo frames
  • Receive Side Scaling (RSS)
🙈 quick heads-up: yes, "chesksum" is spelled like that up there. that's not me being chaotic, that's literally in the page. keeping it exactly as-is, bestie.

for more info on configuring this device, go see ifconfig(8) 💖

🎀 HARDWARE 🎀

gve binds to a single PCI device ID presented by gVNIC:

  • 0x1AE0:0x0042

that's it. one device. she's exclusive 💅

🌸 EXAMPLES 🌸

change the TX queue count to 4 for the gve0 interface:

sysctl dev.gve.0.num_tx_queues=4

change the RX queue count to 4 for the gve0 interface:

sysctl dev.gve.0.num_rx_queues=4

change the TX ring size to 512 for the gve0 interface:

sysctl dev.gve.0.tx_ring_size=512

change the RX ring size to 512 for the gve0 interface:

sysctl dev.gve.0.rx_ring_size=512

literally copy, paste, slay 🔥

🧚 DIAGNOSTICS 🧚

okay this section is the vibe check on all the messages the driver can throw at you. the following messages are recorded during driver initialization:

Enabled MSIX with %d vectors
Configured device resources
Successfully attached %s
Deconfigured device resources

these messages are seen if driver initialization fails. global (across-queues) allocation failures, bestie:

Failed to configure device resources: err=%d
No compatible queue formats
Failed to allocate ifnet struct
Failed to allocate admin queue mem
Failed to alloc DMA mem for DescribeDevice
Failed to allocate QPL page

irq and BAR allocation failures, coming in hot:

Failed to acquire any msix vectors
Tried to acquire %d msix vectors, got only %d
Failed to setup irq %d for Tx queue %d
Failed to setup irq %d for Rx queue %d
Failed to allocate irq %d for mgmnt queue
Failed to setup irq %d for mgmnt queue, err: %d
Failed to allocate BAR0
Failed to allocate BAR2
Failed to allocate msix table

Rx queue-specific allocation failures 😭:

No QPL left for rx ring %d
Failed to alloc queue resources for rx ring %d
Failed to alloc desc ring for rx ring %d
Failed to alloc data ring for rx ring %d

Tx queue-specific allocation failures 😭😭:

No QPL left for tx ring %d
Failed to alloc queue resources for tx ring %d
Failed to alloc desc ring for tx ring %d
Failed to vmap fifo, qpl_id = %d

the following messages are recorded when the interface detach fails:

Failed to deconfigure device resources: err=%d

if bootverbose is on, the following messages are recorded when the interface is being brought up:

Created %d rx queues
Created %d tx queues
MTU set to %d

the following messages are recorded when the interface is being brought down:

Destroyed %d rx queues
Destroyed %d tx queues

these messages are seen if errors are encountered when bringing the interface up or down:

Failed to destroy rxq %d, err: %d
Failed to destroy txq %d, err: %d
Failed to create rxq %d, err: %d
Failed to create txq %d, err: %d
Failed to set MTU to %d
Invalid new MTU setting. new mtu: %d max mtu: %d min mtu: %d
Cannot bring the iface up when detached
Reached max number of registered pages %lu > %lu
Failed to init lro for rx ring %d

these messages are seen if any admin queue command fails:

AQ command(%u): failed with status %d
AQ command(%u): unknown status code %d
AQ commands timed out, need to reset AQ
Unknown AQ command opcode %d

these messages appear if a TX timeout is detected ⏰:

Found %d timed out packet(s) on txq%d, kicking it for completions
Found %d timed out packet(s) on txq%d with its last kick %ld sec ago
which is less than the cooldown period %d. Resetting device

these messages are recorded when the device is being reset due to an error:

Scheduling reset task!
Waiting until admin queue is released.
Admin queue released

if it was the NIC that requested the reset, this message is recorded:

Device requested reset

if the reset fails during the reinitialization phase, this message is recorded:

Restore failed!

these two messages correspond to the NIC alerting the driver to link state changes:

Device link is up.
Device link is down.

apart from all these messages, the driver exposes per-queue packet and error counters as sysctl nodes. global (across queues) counters can be read using netstat(1) ✨

💜 SYSCTL VARIABLES 💜

gve exposes the following sysctl(8) variables:

  • hw.gve.driver_version — the driver version. this is read-only.
  • hw.gve.queue_format — the queue format in use. this is read-only.
  • hw.gve.disable_hw_lro — setting this boot-time tunable to 1 disables Large Receive Offload (LRO) in the NIC. the default value is 0, which means hardware LRO is enabled by default. the software LRO stack in the kernel is always used. this sysctl variable needs to be set before loading the driver, using loader.conf(5).
  • hw.gve.allow_4k_rx_buffers — setting this boot-time tunable to 1 enables support for 4K RX Buffers. the default value is 0, which means 2K RX Buffers will be used. 4K RX Buffers are only supported on DQO_RDA and DQO_QPL queue formats. when enabled, 4K RX Buffers will be used either when HW LRO is enabled or mtu is greated than 2048. this sysctl variable needs to be set before loading the driver, using loader.conf(5).
  • dev.gve.X.num_rx_queues and dev.gve.X.num_tx_queues — run-time tunables that represent the number of currently used RX/TX queues. the default value is the max number of RX/TX queues the device can support.

this call turns down the interface while setting up the new queues, which may potentially cause any new packets to be dropped 😬. this call can fail if the system isn't able to provide the driver with enough resources. in that situation, the driver will revert to the previous number of RX/TX queues. if that also fails, a device reset will be triggered 💀.

note: sysctl nodes for queue stats remain available even if a queue is removed.
  • dev.gve.X.rx_ring_size and dev.gve.X.tx_ring_size — run-time tunables that represent the current ring size for RX/TX queues. the default value is set to device defaults for ring size.

this call turns down the interface while setting up the queues with the new ring size, which may potentially cause any new packets to be dropped 😭. this call can fail if the system isn't able to provide the driver with enough resources. in that situation, the driver will try to revert to the previous ring size for RX/TX queues. if this also fails, the device will be in an unhealthy state and will need to be reloaded. this value must be a power of 2 and within the defined range. periodt.

🌈 LIMITATIONS 🌈

gve does not support the transmission of VLAN-tagged packets. all VLAN-tagged traffic is dropped. sorry bestie, that's just how it is 🫠

⭐ QUEUE FORMATS ⭐

gve features different datapath modes called queue formats, and the lore is honestly kinda deep:

  • GQI_QPL — "QPL" stands for "Queue Page List" and refers to the fact that hardware expects a fixed bounce buffer and cannot access arbitrary memory. GQI is the older descriptor format. the G in "GQI" refers to an older generation of hardware, and the "QI" stands for "Queue In-order" referring to the fact that the NIC sends Tx and Rx completions in the same order as the one in which the corresponding descriptors were posted by the driver.
  • DQO_RDA — DQO is the descriptor format required to take full advantage of next generation VM shapes. "RDA" stands for "Raw DMA Addressing" and refers to the fact that hardware can work with DMA-ed packets and does not expect them to be copied into or out of a fixed bounce buffer. the D in "DQO" refers to a newer generation of hardware, and the "QO" stands for "Queue Out-of-order" referring to the fact that the NIC might send Tx and Rx completions in an order different from the one in which the corresponding descriptors were posted by the driver.
  • DQO_QPL — the next generation descriptor format in the "QPL" mode.

iykyk 🧠✨

🍓 SUPPORT 🍓

please email gvnic-drivers@google.com with the specifics of the issue encountered. be specific bestie, they're literally asking 💌

👑 SEE ALSO 👑

netstat(1), loader.conf(5), ifconfig(8), sysctl(8)

🌸 HISTORY 🌸

the gve device driver first appeared in FreeBSD 13.3. she's a 13.3 girlie 🎀

🫶 AUTHORS 🫶

the gve driver was written by Google. main character energy fr fr.

and that's the whole page, bestie 💖 this one's from Debian, dated May 20, 2025 — go forth and network responsibly, mwah 💅✨
📦 where this page comes from