lnetctl(8)

📦 version 2.17.58, in ubuntu stonking

💖 lnetctl 💖 — configuration utilities

okay so listen up bestie, this is lnetctl(8), the configuration utility page, and it is long, so we're doing this in parts. grab a snack. 🍓

✨ SYNOPSIS ✨

lnetctl
lnetctl --list-commands
lnetctl <cmd> <subcmd> [optional parameters]

🌸 DESCRIPTION 🌸

lnetctl is the girl who configures your LNet parameters, and she comes with a whole bunch of configuration and debugging features you can just… access. icon behavior honestly.

You can invoke lnetctl in interactive mode just by issuing the lnetctl command. After that, you issue commands like this:

lnetctl command subcommand [optional parameters]

wait no — that's the non-interactive one, hold on, let me not get ahead of myself. 😭

So: to get a complete listing of available commands, type --list-commands at the lnetctl prompt. And to get basic help on the meaning and syntax of a command, type <command> --help. yes bestie, it tells you about itself, we love a self-aware queen. 👑

For non-interactive use, the lnetctl utility can be invoked from the command line like this:

lnetctl command subcommand [optional parameters]

💫 LNet Initialization 💫

  • lnetctl lnet configure [--all] — brings up the LNet Network Interface (NI) system. If the --all option is provided, it will load network interfaces defined in the modprobe files. Otherwise, it doesn't initialize any networks by default.
  • lnetctl lnet unconfigure — brings down the LNet Network Interface (NI) system, including any configured networks. putting it to bed fr 💤

🌈 Network Configuration 🌈

  • lnetctl net add — configures a network interface, either given the network name and physical interface device name, or given the ip2net parameter. Other parameters are optional.
    • --net — net name (e.g. tcp0)
    • --if — physical interface (e.g. eth0)
    • --nid — LNet network address to bring up. Cannot be combined with --net.
    • --ip2net — specify networks based on IP address patterns
    • --peer-timeout — time to wait before declaring a peer dead (in seconds). Default value for o2iblnd and socklnd is 180 seconds.
    • --peer-credits — define the max number of in-flight messages per peer.
    • --peer-buffer-credits — the max number of routed in-flight messages per peer.
    • --credits — The total number of in-flight messages over a network interface.
    • --cpt — The CPU partitions on which the created network interface is bound to. Refer to the Lustre Manual Section "Binding Network Interface Against CPU Partitions" for more details. For example to bind a Network Interface to CPU partitions 0 and 1, you would specify this parameter as --cpt [0, 1]
  • lnetctl net del — delete a network interface given the network name.
    • --net — net name (e.g. tcp0)
    • --nid — lnet network address to bring up. Cannot be combined with --net.
  • lnetctl net show — show all currently configured network interfaces if no parameters given, or filter on the network name. More details can be shown by specifying the --verbose parameter. 💅
    • --net — net name (e.g. tcp0) to filter on
    • --verbose — display detailed output per network

🦋 Peer Configuration 🦋

  • lnetctl peer add — configure an LNET peer with at least one supplied NID. The primary NID must be specified. By default, peers are marked as multi-rail capable.
    • --nid — one or more peer NIDs to add to the peer.
    • --prim_nid — Primary NID of the peer.
    • --non_mr — create this peer as not Multi-Rail capable.
    • --lock_prim — lock primary NID of the peer for the purpose of identification with Lustre.
  • lnetctl peer del — delete a peer NID. The primary NID must be specified. If the removed NID is the primary NID, the peer entry will be deleted. girl is gone gone 💀
    • --nid — one or more peer NIDs to remove from the peer.
    • --prim_nid — Primary NID of the peer.
    • --force — optional, use to delete a peer with primary NID locked.
  • lnetctl peer show — show configured peers. By default, lists all peers and associated NIDs.
    • --nid — list of primary nids to filter on
    • --verbose — Include extended statistics, including credits and counters.

⭐ Route Configuration ⭐

  • lnetctl route add — add a route.
    • --net — net name (e.g. tcp0)
    • --gateway — gateway nid (e.g. 10.1.1.2@tcp)
    • --hop — number to final destination (1 < hops < 255)
    • --priority — priority of route (0 - highest prio)
  • lnetctl route del — delete a route specified via the network and gateway.
    • --net — net name (e.g. tcp0)
    • --gateway — gateway nid (e.g. 10.1.1.2@tcp)
  • lnetctl route show — show all currently configured routes if no parameters given, or filter on given parameters. More details can be shown by specifying the --verbose parameter.
    • --net — net name (e.g. tcp0) to filter on
    • --gateway — gateway nid (e.g. 10.1.1.2@tcp) to filter on
    • --hop — number to final destination (1 < hops < 255) to filter on
    • --priority — priority of route (0 - highest prio to filter on)
    • --verbose — display detailed output per route

🎀 Routing Information 🎀

  • lnetctl routing show — show router buffers values as well as show the status of routing (IE: whether the node is set to be a router). it's giving main character router energy fr

🧚 Value Setting 🧚

Individual values can be set using the lnetctl set command. one at a time, cute and precise.

  • lnetctl set tiny_buffers value — set the number of tiny buffers in the system. This is the total number of tiny buffers for all CPU partitions.
  • lnetctl set small_buffers value — set the number of small buffers in the system. This is the total number of small buffers for all CPU partitions.
  • lnetctl set large_buffers value — set the number of large buffers in the system. This is the total number of large buffers for all CPU partitions.
  • lnetctl set routing [0, 1] — 0 value indicates to disable routing. 1 value indicates to enable routing. When routing is disabled the values of the buffers that might have been changed are not remembered, and the next time routing is enabled the default buffer count will be used. so like, no notes taken. 🙈
  • lnetctl set drop_asym_route [0, 1] — 0 value indicates to accept asymmetrical route messages. 1 value indicates to drop them. Asymmetrical route is when a message from a remote peer is coming through a router that would not be used by this node to reach the remote peer.
  • lnetctl set latency_stats [0, 1] — 0 value indicates to stop accumulating per-operation latency statistics. 1 value indicates to accumulate them (default). The latency statistics are reported by lnetctl net show -v and lnetctl peer show -v. The global counters and the per-interface element statistics are not affected by this setting.
  • lnetctl set response_tracking [0, 1, 2, 3] — set the behavior of response tracking.
    1. 0 - Only LNet pings and discovery pushes utilize response tracking.
    2. 1 - GETs are eligible for response tracking.
    3. 2 - PUTs are eligible for response tracking.
    4. 3 - Both PUTs and GETs are eligible for response tracking (default).
Note: Regardless of the value of the response_tracking parameter LNet pings and discovery pushes always utilize response tracking.
  • lnetctl set recovery_limit value — set how long LNet will attempt to recover unhealthy peer interfaces.
    1. 0 - Recover indefinitely (default).
    2. >0 - Recover for the specified number of seconds.
  • lnetctl set max_recovery_ping_interval value — set the maximum recovery ping interval. The recovery ping mechanism increases the next scheduled recovery ping attempt timeout exponentially (base 2) until it is equal to the value set. The default value is 900.

🫶 Import and Export YAML Configuration Files 🫶

LNet configuration can be represented in YAML format. A YAML configuration file can be passed to the lnetctl utility via the import command. The lnetctl utility will attempt to configure all elements defined in the YAML file. she does not skip a single one. 😤

Similarly the export command can be used to dump all supported LNet configuration to stdout. The output can be redirected to a file.

lnetctl import FILE:
lnetctl import < FILE

The import command uses the specified YAML configuration file to configure LNet parameters defined within. The import command by default adds the LNet parameters defined in the YAML file, but this default behavior can be overwritten by specifying the desired behavior.

  • --add — add configuration
  • --del — delete configuration
  • --show — show configuration
  • --exec — execute command
  • --help — display this help
lnetctl export FILE:
lnetctl export > FILE

The export command dumps the LNet configuration, state information, and stats in YAML format to stdout, which can be redirected to a normal file. The output of the export command can be used as input to the import command. round trip queen 🔁

  • --backup — dump only elements necessary to recreate the current configuration.
  • --help — display this help

💜 LNet Statistics 💜

  • lnetctl stats show — show LNET statistics ✨
    • Number of messages allocated
    • Maximum number of messages allocated
    • Number of errors encountered
    • Number of messages sent
    • Number of messages received
    • Number of messages routed
    • Total size in bytes of messages sent
    • Total size in bytes of messages received
    • Total size in bytes of messages routed
    • Total size in bytes of messages dropped

🍓 Showing Peer Credits 🍓

  • lnetctl peer_credits — show details on configured peer credits 💅
    • Peer nid
    • State
    • Reference count on the peer
    • Maximum transmit credits
    • Available transmit credits
    • Available router credits
    • Minimum router credits.

👑 UDSP Configuration 👑

  • lnetctl udsp add — add user-defined selection policy.

🌟 Adding a local network udsp. 🌟

If multiple local networks are available, each one can be assigned a priority. The one with the highest priority is selected to send on. NID and network matching is using NID-range syntax, please see the manual for more detail.

  • --src — network in NID-range syntax (e.g. tcp0 or tcp[1-3])
  • --<priority> <priority value> — optional priority value in [0-255], 0 as the highest
  • --<idx> — The index of where to insert the rule. By default append to the end of the list

…and bestie, we're not even done, the page keeps going 💖

💖 Adding a local NID udsp 💖

okay so this one gives your local NIDs a priority ranking. Once a local network has been chosen, the NI rockin' the highest priority is the one that gets picked. main character selection, honestly.

  • --src — the NID, written in NID-range syntax (so like 10.1.1.2@tcp or 10.1.1.*@tcp)
  • --<priority> <priority value> — optional, and it lives in the range [0-255], where 0 is the highest (yes bestie, lower number = more important, it's a whole thing)
  • --<idx> — the index of where to insert the rule. Left out? It just appends to the end of the list 💅

🦋 Adding a peer NID udsp 🦋

same energy, but for peer NIDs. The peer NID with the highest priority gets selected to send to.

  • --dst — the NID in NID-range syntax (e.g. 10.1.1.2@tcp)
  • --<priority> <priority value> — optional priority value in [0-255], with 0 as the highest
  • --<idx> — the index of where to insert the rule. Default behaviour is append-to-the-end, as always ✨

🌸 Adding a NID pair udsp 🌸

this one is lowkey iconic: the local NIDs that match the rule get added onto a list on the peer NIs that match the rule. When it's time to select the peer NI, whoever has the local NID being used sitting on its list gets preferred.

  • --dst — NID in NID-range syntax (e.g. 10.1.1.1@tcp)
  • --src — NID in NID-range syntax (e.g. 10.1.1.2@tcp)
  • --<idx> — the index of where to insert the rule; default is appending to the end of the list

🎀 Adding a Peer Router udsp 🎀

router edition!! The router NIDs matching the rule get added to a list on the peer NIs matching the rule. When you're sending to a remote peer, the router whose nid is on the peer NI's list is the one that wins.

  • --dst — peer NID in NID-range syntax (e.g. 10.1.1.1@tcp)
  • --rte — router NID in NID-range syntax (e.g. 10.1.2.1@tcp)
  • --<idx> — the index of where to insert the rule. No idx? Appends to the end, periodt.

🧚 lnetctl udsp del 🧚

Deletes a user-defined selection policy. bye bestie 👋

  • --idx — the index of the rule you're deleting.

💜 lnetctl udsp show 💜

Shows you every user-defined selection policy on the system. It dumps them out in YAML form so it's actually readable, we love that.

💀 Debugging 💀

lnetctl debug recovery

Lists the NIDs hanging out on the recovery queue.

  • --local — lists the NIDs on the local recovery queue
  • --peer — lists the NIDs on the peer recovery queue

lnetctl debug peer

Dumps peer debug info to the debug log and the console (both, bestie, no hiding).

  • --nid — the peer's NID

✨ OPTIONS ✨

  • --list-commands — spits out a list of every command the lnetctl utility supports. your little cheat sheet 🫶

🔥 EXAMPLES 🔥

🌟 Initializing LNet after load 🌟

  • lnetctl lnet configure
  • lnetctl lnet configure --all

🌟 Shutting down LNet 🌟

  • lnetctl lnet unconfigure

🌟 Add network 🌟

  • lnetctl net add --net tcp0 --if eth0
  • lnetctl net add --ip2net "tcp0(eth0) 192.168.0.[2,4]; tcp0 192.168.0.*; o2ib0 132.6.[1-3].[2-8/2]"

🌟 Delete network 🌟

  • lnetctl net del --net tcp0

🌟 Show network 🌟

lnetctl net show --verbose:

net:
     - nid: 0@lo
       status: up
       tunables:
          peer_timeout: 0
          peer_credits: 0
          peer_buffer_credits: 0
          credits: 0
     - nid: 192.168.205.130@tcp1
       status: up
       interfaces:
          0: eth3
          1: eth4
       tunables:
          peer_timeout: 180
          peer_credits: 8
          peer_buffer_credits: 0
          credits: 256

🌟 Add route 🌟

  • lnetctl route add --net tcp0 --gateway 10.10.10.1@tcp1 --hop 1 --priority 1

🌟 Delete route 🌟

  • lnetctl route del --net tcp0 --gateway 10.10.10.1@tcp1

🌟 Show route 🌟

  • lnetctl route show --verbose
route:
     - net: tcp
       gateway: 192.168.205.131@tcp1
       hop: 1
       priority: 0 state: down

🌟 Show routing 🌟

  • lnetctl routing show
routing:
     - cpt[0]:
       tiny:
          npages: 0
          nbuffers: 2048
          credits: 2048
          mincredits: 2048
       small:
          npages: 1
          nbuffers: 16384
          credits: 16384
          mincredits: 16384
       large:
          npages: 256
          nbuffers: 1024
          credits: 1024
          mincredits: 1024
     - enable: 1

🌟 Setting variables 🌟

  • lnetctl set tiny_buffers 2048
  • lnetctl set small_buffers 16384
  • lnetctl set large_buffers 256
  • lnetctl set routing 1

🌟 Importing YAML files for configuring 🌟

  • lnetctl import lnet.conf
  • lnetctl import < lnet.conf

🌟 Exporting LNet Configuration 🌟

  • lnetctl export lnet.conf
  • lnetctl export > lnet.conf

🌟 Showing LNet Stats 🌟

  • lnetctl stats show
statistics:
     msgs_alloc: 0
     msgs_max: 1
     errors: 0
     send_count: 89
     recv_count: 0
     route_count: 0
     drop_count: 19
     send_length: 0
     recv_length: 0
     route_length: 0
     drop_length: 0

🌟 Showing peer information 🌟

  • lnetctl peer show
peer:
    - primary nid: 10.148.0.8@o2ib
      Multi-Rail: True
      peer ni:
        - nid: 10.148.0.8@o2ib
          state: NA
    - primary nid: 10.148.0.20@o2ib
      Multi-Rail: True
      peer ni:
        - nid: 10.148.0.20@o2ib
          state: NA
        - nid: 10.148.0.25@o2ib
          state: NA

🌟 Adding a UDSP 🌟

  • lnetctl udsp add --src tcp --priority 1

🌟 Deleting a UDSP 🌟

  • lnetctl udsp del --idx 0

🌟 Show UDSPs 🌟

  • lnetctl udsp show
udsp:
    - idx: 0
    src: tcp
    dst: NA
    rte: NA
    action:
        priority: 0

🌈 SEE ALSO 🌈

and that's the whole thing bestie, you ate 🍓 this is the lnetctl(8) page from the Lustre manual, dated 2017 Jan 12. go forth and configure your networks, i'm obsessed with you 💖✨

📦 where this page comes from