STACD.CONF(5)

📦
version 2.4.1 · 8 versions
, in ubuntu resolute, ubuntu stonking

💖 NAME

stacd.conf — the config file for stacd(8), bestie. That's it, that's the name, no cap.

✨ SYNOPSIS

/etc/stas/stacd.conf

🌸 DESCRIPTION

So when stacd(8) wakes up and starts doing its thing, it reads all its settings from stacd.conf. That's the whole vibe. 💅

🎀 CONFIGURATION FILE FORMAT

OK so here's the tea: stacd.conf is a plain text file, literally just text, and it's split up into sections. Inside those sections you write your config entries in the style key=value. 🦋

The rules are so simple, like:

  • Spaces hanging out immediately before or after the = get ignored. Nobody cares about them.
  • Empty lines get ignored too. Blank space? Skipped.
  • Lines starting with # are ignored, and you can use them for commenting. Live your truth, annotate your config. ✨

👑 OPTIONS

🌈 [Global] section

These are all the options that live in the [Global] section, one by one, gather round:

  • tron= — Trace ON, babes. Takes a boolean argument. If it's true, full code tracing gets enabled. Where does the trace go? Into the system log — so like systemd's journal. Defaults to false.
  • hdr-digest= — Enables the Protocol Data Unit (PDU) Header Digest. Takes a boolean argument. NVMe/TCP supports an optional PDU Header digest, and the digests get calculated using the CRC32C algorithm. If true, Header Digests get inserted into PDUs and checked for errors. Defaults to false.
  • data-digest= — Enables the Protocol Data Unit (PDU) Data Digest. Takes a boolean argument. NVMe/TCP supports an optional PDU Data digest too, also calculated using the CRC32C algorithm. If true, Data Digests get inserted into PDUs and checked for errors. Defaults to false.
  • kato= — Keep Alive Timeout (KATO) in seconds. Takes an unsigned integer. This one sets the timeout value for the Keep Alive feature, in seconds. Defaults to 30 seconds for Discovery Controller connections and 120 seconds for I/O Controller connections.
  • ip-family= — Takes a string argument. This is where you say whether IPv4, IPv6, or both are supported when connecting to a Controller. Connections will not be attempted to IP addresses that this option disables — and that goes for whether they were discovered or manually configured with controller=. If you enter an invalid value, then the default (down below) applies. The choices are ipv4, ipv6, or ipv4+ipv6. Defaults to ipv4+ipv6.
  • nr-io-queues= — Takes a value in the range 1...N. This overrides the default number of I/O queues created by the driver.

Note: this parameter is identical to the one provided by nvme-cli.

Default: depends on the kernel and other run time factors (e.g. number of CPUs).

  • nr-write-queues= — Takes a value in the range 1...N. This adds extra queues that will be used for write I/O.

Note: this parameter is identical to the one provided by nvme-cli.

Default: depends on the kernel and other run time factors (e.g. number of CPUs).

  • nr-poll-queues= — Takes a value in the range 1...N. This adds extra queues that will be used for polling latency sensitive I/O.

Note: this parameter is identical to the one provided by nvme-cli.

Default: depends on the kernel and other run time factors (e.g. number of CPUs).

  • queue-size= — Takes a value in the range 16...1024.

This overrides the default number of elements in the I/O queues created by the driver. Heads up: this option gets ignored for discovery, but it does get passed on to the subsequent connect call.

Note: this parameter is identical to the one provided by nvme-cli.

Defaults to 128.

  • reconnect-delay= — Takes a value in the range 1 to N seconds.

This overrides the default delay before a reconnect is attempted after a connect loss.

Note: this parameter is identical to the one provided by nvme-cli.

Defaults to 10. So it retries to connect every 10 seconds.

  • ctrl-loss-tmo= — Takes a value in the range -1, 0, ..., N seconds. -1 means retry forever. 0 means do not retry.

This overrides the default controller loss timeout period (in seconds).

Note: this parameter is identical to the one provided by nvme-cli.

Defaults to 600 seconds (10 minutes).

  • disable-sqflow= — Takes a boolean argument. This disables SQ flow control, so it omits the head doorbell update for submission queues when sending nvme completions.

Note: this parameter is identical to the one provided by nvme-cli.

Defaults to false.

  • ignore-iface= — Takes a boolean argument. This option controls how connections with I/O Controllers (IOC) are made. 👀

There is no guarantee that there will be a route to reach that IOC. However, we can use the socket option SO_BINDTODEVICE to force the connection to be made on a specific interface instead of letting the routing tables decide where to make the connection.

This option determines whether stacd will use SO_BINDTODEVICE to force connections on an interface, or just rely on the routing tables. The default is to use SO_BINDTODEVICE — in other words, stacd does not ignore the interface.

BACKGROUND: By default, stacd will connect to IOCs on the same interface that was used to retrieve the discovery log pages. If stafd discovers a DC on an interface using mDNS, and stafd connects to that DC and retrieves the log pages, it is expected that the storage subsystems listed in the log pages are reachable on the same interface where the DC was discovered.

For example, let's say a DC is discovered on interface ens102. Then all the subsystems listed in the log pages retrieved from that DC must be reachable on interface ens102. If this doesn't work — for example you cannot ping -I ens102 [storage-ip] — then the most likely explanation is that proxy arp is not enabled on the switch that the host is connected to on interface ens102. Whatever you do, resist the temptation to manually set up the routing tables or to add alternate routes going over a different interface than the one where the DC is located. That simply won't work. Make sure proxy arp is enabled on the switch first. 🙈

Setting routes won't work because, by default, stacd uses the SO_BINDTODEVICE socket option when it connects to IOCs. This option is used to force a socket connection to be made on a specific interface instead of letting the routing tables decide where to connect the socket. Even if you were to manually configure an alternate route on a different interface, the connections (i.e. host to IOC) will still be made on the interface where the DC was discovered by stafd.

Defaults to false.

🦋 [I/O controller connection management] section

Connectivity between hosts and subsystems in a fabric is controlled by Fabric Zoning. Entities that share a common zone (i.e., are zoned together) are allowed to discover each other and establish connections between them. Fabric Zoning is configured on Discovery Controllers (DC). Users can add/remove controllers and/or hosts to/from zones.

Hosts have no direct knowledge of the Fabric Zoning configuration that is active on a given DC. So as a result, if a host is impacted by a Fabric Zoning configuration change, it will be notified of the connectivity configuration change by the DC via Asynchronous Event Notifications (AEN). 💫

okay bestie, quick vocab check before we dive into the ✨spicy✨ part — here's every term we're gonna throw around in this section, no cap:

Table 1. The terms, decoded:

  • AEN — Asynchronous Event Notification. A CQE (Completion Queue Entry) for an Asynchronous Event Request that was previously transmitted by the host to a Discovery Controller. AENs are how DCs tell hosts that a change happened (e.g., a connectivity configuration change, vibe shift, whatever).
  • DC — Discovery Controller.
  • DLP — Discovery Log Page. A host issues a Get Log Page command to pull the list of controllers it may connect to.
  • DLPE — Discovery Log Page Entry. The response to a Get Log Page command contains a list of DLPEs identifying each controller the host is allowed to connect with.
💅 note: DLPEs may contain both I/O Controllers (IOCs) and Discovery Controllers (DCs). DCs listed in DLPEs are called referrals. stacd only deals with IOCs. Referrals (DCs) are handled by stafd.
  • IOC — I/O Controller.
  • Manual Config — manually adding entries to stacd.conf with the controller= parameter.
  • Automatic Config — receiving configuration from a DC as DLPEs.
  • External Config — configuration done outside the nvme-stas framework, for example using nvme-cli commands.

so here's the plot: DCs notify hosts of connectivity configuration changes by sending AENs indicating a "Discovery Log" change. The host uses these AENs as a trigger to issue a Get Log Page command. The response to this command is used to update the list of DLPEs containing the controllers the host is allowed to access. Upon reception of the current DLPEs, the host determines whether DLPEs were added and/or removed — which triggers the addition and/or removal of controller connections. This happens in real time and may affect active connections to controllers, including controllers that support I/O operations (IOCs). A host that was previously connected to an IOC may suddenly be told it's no longer allowed to connect to that IOC and should disconnect. abrupt. iconic but rude. 😭

IOC connection creation. There are 3 ways to configure IOC connections on a host:

  1. Manual Config — by adding controller= entries to the [Controllers] section (see below).
  2. Automatic Config — received in the form of DLPEs from a remote DC.
  3. External Config — using nvme-cli (e.g. "nvme connect").

IOC connection removal/prevention. And there are 3 ways to remove (or prevent) connections to an IOC:

  1. Manual Config.
    1. by adding exclude= entries to the [Controllers] section (see below).
    2. by removing controller= entries from the [Controllers] section.
  2. Automatic Config. As explained above, a host gets a new list of DLPEs upon connectivity configuration changes. On DLPE removal, the host should remove the connection to the IOC matching that DLPE. This behavior is configurable using the disconnect-scope= parameter described below.
  3. External Config — using nvme-cli (e.g. "nvme disconnect" or "nvme disconnect-all").

The decision by the host to automatically disconnect from an IOC following connectivity configuration changes is controlled by two parameters: disconnect-scope and disconnect-trtypes.

🎀 disconnect-scope= 🎀

Takes one of: only-stas-connections, all-connections-matching-disconnect-trtypes, or no-disconnect.

In theory, hosts should only connect to IOCs that have been zoned for them. Connections to IOCs that a host is not zoned to have access to should simply not exist. In practice, however, users may not want hosts to disconnect from all IOCs in reaction to connectivity configuration changes (or at least for some of the IOC connections).

Some users may prefer for IOC connections to be "sticky" and only removed manually (nvme-cli or exclude=) or removed by a system reboot. Specifically, they don't want IOC connections removed unexpectedly on DLPE removal. These users may want to set disconnect-scope to no-disconnect.

⚠️ IMPORTANT, bestie: when IOC connections are removed, ongoing I/O transactions will be terminated immediately. There is no way to tell what happens to the data being exchanged when such an abrupt termination happens. If a host was in the middle of writing to a storage subsystem, there is a chance that outstanding I/O operations may not successfully complete. this is the scary part. take it seriously fr fr.

Values:

  • only-stas-connections
    • Only remove connections previously made by stacd.
    • In this mode, when a DLPE is removed as a result of connectivity configuration changes, the corresponding IOC connection will be removed by stacd.
    • Connections to IOCs made externally, e.g. using nvme-cli, will not be affected, unless they happen to be duplicates of connections made by stacd. It's simply not possible for stacd to tell that a connection was previously made with nvme-cli (or any other external tool). So, it's good practice to avoid duplicating configuration between stacd and external tools.
    • Users wanting to persist some of their IOC connections regardless of connectivity configuration changes should not use nvme-cli to make those connections. Instead, they should hard-code them in stacd.conf with the controller= parameter. Using the controller= parameter is the only way for a user to tell stacd that a connection must be made and not be deleted "no-matter-what".
  • all-connections-matching-disconnect-trtypes
    • All connections that match the transport type specified by disconnect-trtypes=, whether they were made automatically by stacd or externally (e.g., nvme-cli), will be audited and are subject to removal on DLPE removal.
    • In this mode, as DLPEs are removed as a result of connectivity configuration changes, the corresponding IOC connections will be removed by the host immediately — whether they were made by stacd, nvme-cli, or any other way. Basically, stacd audits all IOC connections matching the transport type specified by disconnect-trtypes=.
    • NOTE. This mode implies that stacd will only allow Manually Configured or Automatically Configured IOC connections to exist. Externally Configured connections using nvme-cli (or other external mechanism) that do not match any Manual Config (stacd.conf) or Automatic Config (DLPEs) will get deleted immediately by stacd. ma'am. 😳
  • no-disconnect
    • stacd does not disconnect from IOCs when a DPLE is removed or a controller= entry is removed from stacd.conf. All IOC connections are "sticky".
    • Instead, users can remove connections by issuing the nvme-cli command "nvme disconnect", add an exclude= entry to stacd.conf, or wait until the next system reboot at which time all connections will be removed.

Defaults to only-stas-connections. ← remember this one, it's the default 💖

🦋 disconnect-trtypes= 🦋

This parameter only applies when disconnect-scope is set to all-connections-matching-disconnect-trtypes. It limits the scope of the audit to specific transport types.

Can take the values tcp, rdma, fc, or a combination thereof by separating them with a plus (+) sign. For example: tcp+fc. No spaces are allowed between values and the plus (+) sign. periodt.

Values:

  • tcp
    • Audit TCP connections.
  • rdma
    • Audit RDMA connections.
  • fc — audit Fibre Channel connections.

Defaults to tcp.

  • connect-attempts-on-ncc= — okay so the NCC bit is a whole thing, let me explain. NCC stands for Not Connected to CDC, and it's a bit the CDC hands back in the EFLAGS field of the DLPE. Only CDCs will ever set the NCC bit — DDCs always clear it to 0. Basically it's the CDC's way of being like "bestie, the subsystem is not reachable from me right now," which could mean the subsystem is down or there's an outage on the chunk of network between the CDC and the subsystem. 💀

So if a host is currently failing to connect to an I/O controller, and the NCC bit for that I/O controller is asserted, the host gets to decide to stop trying to connect to that subsystem until connectivity comes back — which the CDC signals by clearing the NCC bit. The glow-up. ✨

The parameter connect-attempts-on-ncc= controls whether stacd even cares about the NCC bit when it's trying to connect to an I/O Controller. Set it to 0 and stacd ignores the NCC bit entirely and just keeps trying to connect (never gives up, iconic). Set it to a non-zero value and that number is how many connection attempts get made before stacd gives up. 🫶

Heads up though — this value should be set to something greater than 1. Like, when it's set to 1, stacd automatically uses 2 instead. And the reason is simple: it's possible that a first connect attempt just fails. 💅

Defaults to 0.

🌸 [Controllers] section 🌸

The following options are available in the [Controllers] section:

  • controller= — Controllers get specified with the controller option. You can specify this option more than once to specify more than one controller, no cap. The format is one line per Controller, made up of a little series of fields separated by semi-colons, like this:
controller=transport=[trtype];traddr=[traddr];trsvcid=[trsvcid];host-traddr=[traddr],host-iface=[iface];nqn=[nqn]

🔍 Fields

  • transport= — This is a mandatory field and it specifies the network fabric being used for a NVMe-over-Fabrics network. The current trtype values understood are:

Table 2. Transport type

  • rdma — The network fabric is an rdma network (RoCE, iWARP, Infiniband, basic rdma, etc)
  • fc — The network fabric is a Fibre Channel network.
  • tcp — The network fabric is a TCP/IP network.
  • loop — Connect to a NVMe over Fabrics target on the local host
  • traddr= — This is a mandatory field that specifies the network address of the Controller. For transports using IP addressing (e.g. rdma) this should be an IP-based address (ex. IPv4, IPv6). It could also be a resolvable host name (e.g. localhost).
  • trsvcid= — This is an optional field that specifies the transport service id. For transports using IP addressing (e.g. rdma, tcp) this field is the port number.

Depending on the transport type, this field will default to either 8009 or 4420 as follows. 👇

UDP port 4420 and TCP port 4420 have been assigned by IANA for use by NVMe over Fabrics. NVMe/RoCEv2 controllers use UDP port 4420 by default. NVMe/iWARP controllers use TCP port 4420 by default.

TCP port 4420 has been assigned for use by NVMe over Fabrics and TCP port 8009 has been assigned by IANA for use by NVMe over Fabrics discovery. TCP port 8009 is the default TCP port for NVMe/TCP discovery controllers. There is no default TCP port for NVMe/TCP I/O controllers — the Transport Service Identifier (TRSVCID) field in the Discovery Log Entry tells you which TCP port to use.

The TCP ports that may be used for NVMe/TCP I/O controllers include TCP port 4420, and the Dynamic and/or Private TCP ports (i.e., ports in the TCP port number range from 49152 to 65535). NVMe/TCP I/O controllers should not use TCP port 8009. And TCP port 4420 shall not be used for both NVMe/iWARP and NVMe/TCP at the same IP address on the same network. 🚫

Ref: IANA Service names port numbers[1]

  • nqn= — This field specifies the Controller's NVMe Qualified Name.

This field is mandatory for I/O Controllers, but is optional for Discovery Controllers (DC). For the latter, the NQN will default to the well-known DC NQN: nqn.2014-08.org.nvmexpress.discovery if left undefined.

  • host-traddr= — This is an optional field that specifies the network address used on the host to connect to the Controller. For TCP, this sets the source address on the socket.
  • host-iface= — This is an optional field that specifies the network interface used on the host to connect to the Controller (e.g. IP eth1, enp2s0, enx78e7d1ea46da). This forces the connection to be made on a specific interface instead of letting the system decide. main character energy, honestly. 👑
  • dhchap-secret= — NVMe In-band authentication host secret (i.e. key); needs to be in ASCII format as specified in NVMe 2.0 section 8.13.5.8 Secret representation. If this option is not specified, the default is read from /etc/stas/sys.conf (see the 'key' parameter under the [Host] section). In-band authentication is attempted when this is present.
  • dhchap-ctrl-secret= — This is an optional field that specifies the NVMe In-band authentication controller secret (i.e. key) for bi-directional authentication; needs to be in ASCII format as specified in NVMe 2.0 section 8.13.5.8 'Secret representation'. Bi-directional authentication will be attempted when present.
  • hdr-digest= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • data-digest= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • nr-io-queues= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • nr-write-queues= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • nr-poll-queues= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • queue-size= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • kato= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • reconnect-delay= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • ctrl-loss-tmo= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.
  • disable-sqflow= — See definition in [Global] section. This is an optional field used to override the value specified in the [Global] section.

Examples:

controller = transport=tcp;traddr=localhost;trsvcid=8009
controller = transport=tcp;traddr=2001:db8::370:7334;host-iface=enp0s8
controller = transport=fc;traddr=nn-0x204600a098cbcac6:pn-0x204700a098cbcac6
  • exclude= — Controllers that should be excluded can be specified with the exclude= option. Using mDNS to automatically discover and connect to controllers can result in unintentional connections being made (not the vibe 😭), so this keyword lets you configure the controllers that should not be connected to.

The syntax is the same as for "controller", except that only transport, traddr, trsvcid, nqn, and host-iface apply. Multiple exclude= keywords may appear in the config file to specify more than 1 excluded controller.

Note 1: A minimal match approach is used to eliminate unwanted controllers. That is, you do not need to specify all the parameters to identify a controller. Just specifying the host-iface, for example, can be used to exclude all controllers on an interface. Slay, one line and they're gone. 💅
Note 2: exclude= takes precedence over controller. A controller specified by the controller keyword, can be eliminated by the exclude= keyword.

Examples:

exclude = transport=tcp;traddr=fe80::2c6e:dee7:857:26bb # Eliminate a specific address
exclude = host-iface=enp0s8                             # Eliminate everything on this interface

👀 SEE ALSO 👀

📝 NOTES ✨

Okay bestie, sources time — because we always cite our receipts 💅

  1. IANA Service names port numbers — this is where the port numbers come from, and here's the literal link so you can go look at it yourself, no gatekeeping:

https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=nvme

Yeah it's a long one, copy it carefully, it's giving citation energy 🎀

not me making you read a whole IANA registry... but fr fr, that's the reference, and it's the only one listed here, so that's the one you get 🦋

And that's the whole page, bestie — you ate that. This one's from nvme-stas 2.4.1, and you've been reading STACD.CONF(5). Go configure something iconic. 👑💖

📦 where this page comes from