NVME(4)

📦
version 15.1 · 4 versions
, in freebsd 15.1, nixpkgs unstable

💅 NVME(4) — NVM Express core driver 💅

okay bestie, welcome to the NVME(4) page from the Kernel Interfaces Manual! this is literally the moment where we talk about the nvme driver — the absolute main character energy behind NVM Express controllers. ✨ it's giving fast, it's giving modern, and we're gonna break it all down. no cap. 💖

🌸 NAME 🌸

nvme — NVM Express core driver. periodt. 💅

🦋 SYNOPSIS 🦋

okay so to compile this driver into your kernel, you gotta drop this line into your kernel configuration file, bestie:

device nvme

or, if you're more of a load-it-as-a-module-at-boot girlie, put this in loader.conf(5):

nvme_load="YES"

most users will also want to enable nvd(4) or nda(4) to expose NVM Express namespaces as disk devices which can be partitioned. like, that's kind of the whole point, y'know? 🫶

quick lil note: in NVM Express terms, a namespace is roughly equivalent to a SCSI LUN. iykyk. 💫

✨ DESCRIPTION ✨

the nvme driver provides support for NVM Express (NVMe) controllers, and it's honestly iconic. like, it handles all of this:

  • Hardware initialization — getting your controller up and running from the jump. 🚀
  • Per-CPU IO queue pairs — one queue pair per CPU, because we love parallel vibes. ⚡
  • API for registering NVMe namespace consumers such as nvd(4) or nda(4) — so they can do their disk device things. 💾
  • API for submitting NVM commands to namespaces — sending commands straight to your namespaces. 📤
  • Ioctls for controller and namespace configuration and management — all the config and management energy you need. 🛠️

the nvme driver creates controller device nodes in the format /dev/nvmeX and namespace device nodes in the format /dev/nvmeXnsY. note that the NVM Express specification starts numbering namespaces at 1, not 0, and this driver follows that convention. so don't come for me about the indexing, okay? it's literally the spec. 🙈

🎀 CONFIGURATION 🎀

by default, nvme will create an I/O queue pair for each CPU, provided enough MSI-X vectors and NVMe queue pairs can be allocated. if not enough vectors or queue pairs are available, nvme(4) will use a smaller number of queue pairs and assign multiple CPUs per queue pair. it's giving resourceful. 💅

to force a single I/O queue pair shared by all CPUs, set this tunable value in loader.conf(5):

hw.nvme.per_cpu_io_queues=0

to assign more than one CPU per I/O queue pair, thereby reducing the number of MSI-X vectors consumed by the device, set this tunable value in loader.conf(5):

hw.nvme.min_cpus_per_ioq=X

to force legacy interrupts for all nvme driver instances, set this tunable value in loader.conf(5):

hw.nvme.force_intx=1
⚠️ note that use of INTx implies disabling of per-CPU I/O queue pairs. so like, choose wisely bestie. 💀

to control maximum amount of system RAM in bytes to use as Host Memory Buffer for capable devices, set this tunable:

hw.nvme.hmb_max

the default value is 5% of physical memory size per device. cute, right? 🍓

the nvd(4) driver is used to provide a disk driver to the system by default. the nda(4) driver can also be used instead. here's the tea on both:

  • nvd(4) — performs better with smaller transactions and few TRIM commands. it sends all commands directly to the drive immediately. like, no waiting, no queueing, just straight vibes. ⚡
  • nda(4) — performs better with larger transactions and also collapses TRIM commands giving better performance. it can queue commands to the drive; combine BIO_DELETE commands into a single trip; and use the CAM I/O scheduler to bias one type of operation over another. it's giving organized and efficient. 👑

to select the nda(4) driver, set this tunable value in loader.conf(5):

hw.nvme.use_nvd=0

this value may also be set in the kernel config file with:

options NVME_USE_NVD=0

when there is an error, nvme prints only the most relevant information about the command by default. to enable dumping of all information about the command, set this tunable value in loader.conf(5):

hw.nvme.verbose_cmd_dump=1

prior versions of the driver reset the card twice on boot. this proved to be unnecessary and inefficient, so the driver now resets drive controller only once. the old behavior may be restored in the kernel config file with:

options NVME_2X_RESET

like, why do it twice when once is enough? we love efficiency. 💖

🌈 SYSCTL VARIABLES 🌈

the following controller-level sysctls are currently implemented, and they're super handy for keeping an eye on your controller's vibe:

  • dev.nvme.0.num_cpus_per_ioq — (R) Number of CPUs associated with each I/O queue pair. like, who's paired with who. 💑
  • dev.nvme.0.int_coal_time — (R/W) Interrupt coalescing timer period in microseconds. set to 0 to disable. it's giving timing is everything. ⏱️
  • dev.nvme.0.int_coal_threshold — (R/W) Interrupt coalescing threshold in number of command completions. set to 0 to disable. because sometimes you gotta wait for a few completions before you interrupt. 🔔

the following queue pair-level sysctls are currently implemented. admin queue sysctls take the format of dev.nvme.0.adminq and I/O queue sysctls take the format of dev.nvme.0.ioq0. so like, plug in the right queue and you're good to go. 🫶

  • dev.nvme.0.ioq0.num_entries — (R) Number of entries in this queue pair's command and completion queue. 📋
  • dev.nvme.0.ioq0.num_tr — (R) Number of nvme_tracker structures currently allocated for this queue pair. the trackers are tracking. 👀
  • dev.nvme.0.ioq0.num_prp_list — (R) Number of nvme_prp_list structures currently allocated for this queue pair. prp lists are the physical region pages, and they're iconic. 💅
  • dev.nvme.0.ioq0.sq_head — (R) Current location of the submission queue head pointer as observed by the driver. the head pointer is incremented by the controller as it takes commands off of the submission queue. like, the controller is literally pulling commands off the queue. 🎣
  • dev.nvme.0.ioq0.sq_tail — (R) Current location of the submission queue tail pointer as observed by the driver. the driver increments the tail pointer after writing a command into the submission queue to signal that a new command is ready to be processed. so the driver is like "here's a new one, bestie!" 📥
  • dev.nvme.0.ioq0.cq_head — (R) Current location of the completion queue head pointer as observed by the driver. the driver increments the head pointer after finishing with a completion entry that was posted by the controller. so it's like "got it, next!" ✅
  • dev.nvme.0.ioq0.num_cmds — (R) Number of commands that have been submitted on this queue pair. the command counter is counting. 🔢
  • dev.nvme.0.ioq0.dump_debug — (W) Writing 1 to this sysctl will dump the full contents of the submission and completion queues to the console. like, it's giving debug mode, and it's a whole vibe. 🐛

in addition to the typical pci attachment, the nvme driver supports attaching to a ahci(4) device. Intel's Rapid Storage Technology (RST) hides the nvme device behind the AHCI device due to limitations in Windows. however, this effectively hides it from the FreeBSD kernel. to work around this limitation, FreeBSD detects that the AHCI device supports RST and when it is enabled. see ahci(4) for more details. like, Windows limitations? not our problem, we worked around it. 💁‍♀️

💀 DIAGNOSTICS 💀

nvme%d: System interrupt issues? — the driver found a timed-out transaction had a pending completion record, indicating an interrupt had not been delivered. the system is either not configuring interrupts properly, or the system drops them under load. this message will appear at most once per boot per controller. so like, if you see this, something's off with your interrupts. don't ignore it. 🚨

👑 SEE ALSO 👑

check out these other iconic pages, bestie:

🦋 HISTORY 🦋

the nvme driver first appeared in FreeBSD 9.2. like, it's been around for a minute. a classic. 🕰️

💖 AUTHORS 💖

the nvme driver was developed by Intel and originally written by Jim Harris <jimharris@FreeBSD.org>, with contributions from Joe Golio at EMC. the whole thing is giving collaborative excellence. 🤝

this man page was written by Jim Harris <jimharris@FreeBSD.org>. so thank him for the reading material. 💌

and that's the tea! this page is from Debian, dated June 6, 2020, and it's all about NVME(4). stay slay, bestie. 💅✨

📦 where this page comes from