LCTL(8)

📦 version 2.17.58, in ubuntu stonking

🎀 LCTL(8) — Lustre Configuration Utilities 🎀

okay bestie, welcome to lctl, aka the Lustre filesystem administrator configuration tool 💅 this is the manual page for it, and it's a big one, so buckle up 🦋

✨ NAME ✨

lctl — our beloved Lustre filesystem administrator configuration tool 👑

✨ SYNOPSIS ✨

lctl
lctl [--device DEVICE] COMMAND [ARGS]

✨ DESCRIPTION ✨

So basically, lctl is the tool a storage administrator uses to do all the things in Lustre — configuration, maintenance, and debugging features are all reachable through it 🌈

You can run it two ways:

non-interactively, where it just executes the COMMAND you gave it and dips
interactively, by running it with no arguments, so you can fire off a whole bunch of commands in one session 💬

And here's the tea: sometimes a COMMAND has to be run on a specific DEVICE. You can either name that device on the command line, or pick it interactively before you run the command 🎀

The most common commands are: debug_daemon, debug_kernel, device_list, get_param, set_param, list_nids, mark, and ping ⭐

Most sub-commands live in their own separate man pages, named like lctl-COMMAND, and you can find the list of them in the SEE ALSO section at the very end 👀 Some sub-commands don't have their own detailed man page though — those are the ones listed further down in this page. No cap.

✨ OPTIONS ✨

These are the options you can use to invoke lctl, bestie:

--device — name the DEVICE (by name or by number) that the operation should be done on. Wanna know what devices you've got locally? The device_list command literally shows you the list 🫶
help COMMAND — get the basic help on what a command means and how to actually type it out 📖
--list-commands — print out every command that the lctl utility supports 🗒️
--ignore_errors, ignore_errors — ignore errors while a script is being processed. Messy but sometimes you gotta 🙈
lustre_build_version — print the build version of the Lustre kernel modules 💜
-V, --version — print the build version of the utility itself (yes, that's a different version, don't @ me) 💅
exit, quit — dip out of an interactive session ✌️

✨ BASH COMPLETION ✨

lctl supports bash-completion(1) for tab completion of sub-command names and their options — assuming you've got the package installed properly and you're using a supporting shell 🐚✨ This lowkey makes life SO much easier for finding what commands exist, what options they take, and in lots of cases even the arguments those options want.

Here's the cheat sheet:

Type lctl TAB to list commands — or do it after you've partially typed out a command name 💫
Type COMMAND --TAB to list the long options (the descriptive, easily-understood ones, we love her) 🌸
Type COMMAND -TAB to list the short options ⚡

And some commands even do context-specific completion of the arguments for a command-line option — like generating a list of OBD devices for the --device option, or a list of OST pools for the pool_* commands. Iconic behaviour, honestly 👑

✨ COMMANDS ✨

🌈 System Configuration

This is the on-line tool set for displaying, modifying, backing up, or removing Lustre system configuration 💾 For the details, please see: lctl-get_param(8), lctl-set_param(8), lctl-llog_print(8), and lctl-lcfg_*(8) 🎀

🌈 Network Configuration

network{up|down}|{tcp|o2ib} — start or stop LNET, or pick a network type for the other lctl LNET commands 🌐 Also see lnetctl(8) for most LNet configuration commands — she's the main character for that job tbh.
conn_list — print every connected remote NID for a given network type 🔗
interface_list — print the network interface info for a given NETWORK type 🖧
list_nids — print all the Network Identifiers on the local node. Heads up: LNET must be running for this one ⚠️
peer_list — print the known peers for a given NETWORK type 👯
ping NID TIMEOUT — check LNET connectivity via an LNET ping 🏓 This uses the fabric appropriate to the NID you specified. By default lctl will try to reach the remote node for up to 120 seconds and then time out. Wanna disable the timeout? Just give it a negative timeout value, bestie 😌

careful: that's "negative", not "zero". Verbally committing to a negative number is the move here.

replace_nids DEVICE_NAME NIDS_LIST — replace the LNet NIDs. See lctl-replace_nids(8) for the details 🔁
route_list — print the complete routing table 🗺️
which_nid NIDLIST — given a list of NIDs for a remote node, show which interface communication will actually take place on 🕵️

🌈 Device Selection

device DEVNAME — this selects the specified OBD device. Every other command depends on the device having been set 🎯
device_list — show all the local Lustre OBDs. See lctl-device_list(8) 📋

🌈 Device Operations

`conf_param [-d] DEVICE|FSNAME.PARAMETER=VALUE`

Set a permanent configuration parameter for any device via the MGS. This command must be run on the MGS node 💅

BIG NOTE, BESTIE: the conf_param command is deprecated and should NOT be used for configuring Lustre parameters. Here's why: it only applies settings to targets that are present at the moment the command runs. Targets added or mounted later (like OSTs) will not automatically get those settings 🚩 For reliable and consistent configuration, use set_param instead. The set_param command immediately applies parameter changes to all relevant targets, including ones added in the future. See lctl-set_param(8) for the details ✨

For example, replace:

lctl conf_param testfs-OST0000.osc.max_rpcs_in_flight=16

with:

lctl set_param -P osc.testfs-*.max_rpcs_in_flight=16

Update any scripts or procedures to use set_param in place of conf_param 🔧 And FYI: use conf_param -d to remove existing conf_param settings — set_param -P does not remove them, periodt 👑

The -d option deletes a parameter setting (so it uses the default value the next time it restarts). A null VALUE also deletes the parameter setting. Super useful when an incorrect or obsolete parameter is sitting in the configuration 🧹

Parameters:

All the writable parameters under lctl list_param (e.g. lctl list_param -w osc.*.* or lctl list_param -F osc.*.* | grep =) can be permanently set using lctl conf_param, but the format is slightly different 🫠 With conf_param, the device comes first, then the obdtype. (See the examples below.) Wildcards are not supported. ❌

On top of that, failover nodes may be added (or removed), and some system-wide parameters may be set as well: sys.at_max, sys.at_min, sys.at_extra, sys.at_early_margin, sys.at_history, sys.timeout, sys.ldlm_timeout. And DEVICE gets completely ignored for system-wide parameters 🙈

activate — reactivate an import after deactivating it (see below). Heads up: this setting is only effective until the next restart (see conf_param) 🔄
deactivate — deactivate an import, which specifically means do not assign new file stripes to an OSC. You'd use this command on the OSC in the MDT LOV corresponding to a failed OST device, so it stops trying to talk to the failed OST 🛑
abort_recovery — abort the recovery process on a restarting MDT or OST device 💥

🌈 Changelogs

A changelog user can be registered and deregistered on a particular device. The changelog starts logging as soon as any user is registered 📝

For more details see:

lctl-changelog_register(8) — register a new changelog user on the specified MDT device with the specified parameters 🆕
lctl-changelog_deregister(8) — deregister an existing changelog user on the specified MDT 👋

🌈 Nodemap

The nodemap feature lets UIDs, GIDs, and PROJIDs from remote systems be mapped to local sets of UIDs, GIDs, and PROJIDs while retaining POSIX ownership, permissions and quota information 🌍 As a result, multiple sites with conflicting user and group identifiers can all operate on a single Lustre file system without creating collisions in the UID, GID, or PROJID space. It's giving diplomacy 💅

Nodemap commands must be run on the MGS, which is where the current nodemap configuration is asynchronously propagated from to all server nodes 📡 The nodemap configuration also persists across restarts. The exception to that? Dynamic nodemaps (see lctl-nodemap-add(8)) which are defined on the local server only and are not persistent ⚠️

Besides assigning client nodes to nodemaps, you also need to define a privileged nodemap that covers all Lustre server nodes for ID mapping to work correctly. That privileged nodemap should have both "admin" and "trusted" properties set 🔐

The "default" nodemap is a special one — it cannot be removed, and it's the fallback nodemap for all nodes not assigned to any other nodemap 🏠 Because of that special role, only some properties can be set on it, like "squash_uid". ID mapping is explicitly disabled on the "default" nodemap. So be careful when modifying the "admin" and "trusted" properties on the "default" nodemap — especially if Lustre servers aren't covered by any other nodemap 😬

🌈 Configuration logs

clear_conf DEVICE|FSNAME — this command runs on an MGS node that has the MGS device mounted with -o nosvc 🧼 It cleans up configuration files stored in the CONFIGS/ directory of any records marked SKIP. If the device name is given, then the specific logs for that filesystem (e.g. testfs-MDT0000) get processed. Otherwise, if a filesystem name is given, then all configuration files for the specified filesystem are cleared 🧽

LFSCK 💅

okay bestie, this one is the big one — an on-line Lustre consistency check and repair tool. It's here to totally replace the old lfsck tool, and it handles basically every kind of Lustre inconsistency verification you can think of, including:

corrupted or lost OI mapping
corrupted or lost link EA
corrupted or lost FID in name entry
dangling name entry
multiple referenced name entry
unmatched MDT-object and name entry pairs
orphan MDT-object
incorrect MDT-object links count
corrupted namespace
corrupted or lost LOV EA
lost OST-object
multiple referenced OST-object
unmatched MDT-object and OST-object pairs
orphan OST-object
and so on ✨

See also:

lctl-lfsck-start(8) — start LFSCK on the specified MDT or OST device with specified parameters.
lctl-lfsck-stop(8) — stop LFSCK on the specified MDT or OST device.
lctl-lfsck-query(8) — get the LFSCK global status via the specified MDT device.

Barrier 🎀

The tools set for write (modify) barrier on all MDTs. For the deets, please see:

lctl-barrier(8)

Snapshot 📸

ZFS backend based snapshot tools set 🦋 The tool loads system configuration from the file /etc/ldev.conf on the MGS, and calls related ZFS commands to maintain Lustre snapshot pieces on all targets (MGS/MDT/OST). Heads up: /etc/ldev.conf is not just for snapshot, it also does other stuff 👑

The format is:

HOST foreign/-LABEL DEVICE[JOURNAL_PATH]/- [RAIDTAB]

The format of LABEL is:

FSNAME-{ROLE}{INDEX} or {ROLE}{INDEX}

The format of DEVICE is:

[md|zfs:][POOL_DIR/]POOL/FILESYSTEM

Snapshot only uses the fields HOST, LABEL and DEVICE 💅

# cat /etc/ldev.conf
        host-mdt1 - myfs-MDT0000 zfs:/tmp/myfs-mdt1/mdt1
        host-mdt2 - myfs-MDT0001 zfs:myfs-mdt2/mdt2
        host-ost1 - OST0000 zfs:/tmp/myfs-ost1/ost1
        host-ost2 - OST0001 zfs:myfs-ost2/ost2