LFS-DF(1)

📦 version 2.17.58, in ubuntu stonking

👑 LFS-DF(1) — `lfs-df` 👑

Lustre User Utilities ✨ bestie edition, section 1 of the manual

✨ NAME ✨

lfs-df — report Lustre filesystem disk usage 💾🦋

🎀 SYNOPSIS 🎀

lfs df [-i] [-h|-H] [--lazy|-l] [--no-header|-N] [--only-summary|-s]
       [--output|-u FIELD_LIST] [--pool|-p FSNAME[.POOL]] [--mdt|-m]
       [--ost|-o] [--verbose|-v] [path]

🌸 DESCRIPTION 🌸

okay so listen. lfs df shows you filesystem usage info — and by default it does this for every single Lustre filesystem that's mounted on that node rn. Or, if you hand it a PATH, it'll show you just the filesystem that contains that PATH. no cap.

It breaks it all the way down: current usage and totals for each MDT and each OST separately, plus a per-filesystem summary that matches what df(1) gives you for each filesystem. 💅

By default, lfs df reports the SPACE usage of the OSTs (and the MDT space usage is shown only for reference, like a lil bonus). But with -i? Then it reports the INODE (object) usage for each target, and in the summary too.

Now here's the tea for ZFS-backed targets 🫖: the IUsed count accurately reflects the number of in-use objects on each target — fr fr accurate. But the Inodes total and Free inode counts are ESTIMATED, based on the current average (mean) space used per object on each target, and they may fluctuate over time to reflect the current usage pattern of the target. The estimate gets more accurate as the target gets more full, assuming the usage pattern stays consistent. so basically it gets better at guessing when it's stuffed. 😭

lfs df may also report additional target status as the last column in the display, if there's drama with that target. The states include:

D: degraded — The target has a failed drive in the RAID device, or is undergoing RAID reconstruction. This state is marked on the server automatically for ZFS targets via zed(8), or a (user-supplied) script that monitors the target device and sets lctl set_param obdfilter.TARGET.degraded=1 on the OST. This target will be avoided for new allocations, but will still be used for existing files located there, or if there aren't enough non-degraded OSTs to make up a widely-striped file. 💀
N: no-create — The target is configured to deny new OST object creation by the lctl set_param obdfilter.FSNAME-OSTXXXX.no_create=1 option, or new subdirectory creation by the lctl set_param obdfilter.FSNAME-MDTXXXX.no_create=1 parameter or the -o no_create mount option. This can be used to add an OST or MDT to the filesystem without allowing objects to be allocated on it yet, or for other reasons. very "i'm here but don't talk to me" energy 🙈
R: read-only — The target filesystem is marked read-only due to filesystem corruption detected by ldiskfs or ZFS. No modifications are allowed on this OST, and it needs to have e2fsck(8) or zpool(8) scrub run to repair the underlying filesystem. it's in its healing era, let it rest 🩹
S: out-of-space — The target filesystem has less than the minimum required free space and will not be used for new object allocations until it has more free space.
I: out-of-inodes — The target filesystem has less than the minimum required free inodes and will not be used for new object allocations until it has more free inodes.
f: flash — The target filesystem is using a flash (non-rotational) storage device. This lower-case status is only shown in conjunction with the -v option, since it is not an error condition. (lowercase means it's just a vibe, not a problem ✨)

💅 OPTIONS 💅

The various options supported by lfs df are listed and explained below — every single one, bestie, we're not skipping:

-h, --human-readable — Print output in a human readable format (e.g. 16.3T, 4.25P). Suffixes are base-2 units (i.e. 1 GiB = 1024 MiB). 🌈
-H, --si — Like -h, but suffixes are SI base-10 units (i.e. 1 GB = 1000 MB). same energy, metric flavored ⭐
-i, --inodes — Print information about the inode usage and totals for the MDTs and OSTs rather than space usage. Note that the Inodes total, IUsed, and IFree counts typically reflect the SUM of values from the MDTs. If the total number of objects available on the OSTs is smaller (factoring in the filesystem default stripe_count, as one OST object is used for each stripe in a file) then the reported Inodes and IFree values will be reduced to reflect the MINIMUM number of files that could potentially be created with the default stripe count. The actual total number of files that can be created may be different. 👀
-l, --lazy — Do not attempt to contact any OST or MDT not currently connected to the client. This avoids blocking the lfs df output if a target is offline or unreachable, and only returns the space on OSTs that can currently be accessed. we're not waiting around, we're busy 💅
-m, --mdt[=INDEX] — Filter output to show information on MDT devices. If INDEX is specified, it must be attached to the option without a space, like "-mINDEX" or "--mdt=INDEX". For example: "-m0" or "--mdt=0". Can be used with -o to show both types of devices, which is equivalent to having neither option. 🎀
-N, --no-header — Suppress column headers in the output. This is useful for scripting and automation where only the data values are needed without the header line. When combined with --output, --ost, and --mdt, this allows for clean extraction of specific field values. scripts said "no small talk" 🙈
-s, --only-summary — Display only the filesystem summary line without showing individual MDT and OST entries. This provides a consolidated view of the total filesystem usage, Useful for monitoring scripts that only need overall filesystem statistics. just the highlight reel, periodt ✨
-o, --ost[=INDEX] — Filter output to show information on OST devices. If INDEX is specified, it must be attached to the option without a space, like "-oINDEX" or "--ost=INDEX". For example: "-o0" or "--ost=0". Can be used with -m to show both types of devices, which is equivalent to having neither option. 🦋
-u, --output=FIELD_LIST — Print only the specified fields in a customizable format. This allows printing only specific fields, avoiding the need for complex awk(1) or grep(1) parsing to extract values in test scripts and automation. Headers are shown when no specific OST or MDT index is specified. When a specific target index is given (e.g., -o0 or -m1), headers are suppressed for easier scripting. Field aliases are provided to match the column headings reported by df(1). Available fields include:
- source (alias device) — Display the UUID of the target device.
- size (alias total, btotal) — Display the total block space available (in KiB by default).
- used (alias bused) — Display the used block space (in KiB by default).
- free (alias bfree) — Display the free block space (in KiB by default).
- avail (alias bavail) — Display the available block space (in KiB by default).
- pcent (alias usepct, bpct, pct) — Display the percentage of used block space.
- itotal (alias inodes) — Display the total number of inodes (in KiB by default).
- iused — Display the number of used inodes (in KiB by default).
- iavail (alias ifree) — Display the number of free inodes (in KiB by default).
- ipcent (alias iusepct, ipct) — Display the percentage of used inodes.
- domain — Display the failure-domain.
- target — Display the mount point path.
- state — Display the target os_state flags.

Fields are displayed in the order specified in FIELD_LIST. When using --output, both block and inode fields can be mixed in the same command, allowing users to see comprehensive statistics in a single view. mixing them? iconic behavior 👑

-p, --pool=[fsname.]pool — Limit the usage to report only MDTs and OSTs that are in the specified POOL. If multiple filesystems are mounted, list OSTs in POOL for every filesystem, or limit the display to only a pool for a specific filesystem if FSNAME.POOL is given. Specifying both the fsname and pool like:

lfs df --pool=FSNAME.POOL

is equivalent to specifying the mountpoint for the given FSNAME:

lfs df --pool=pool /mnt/FSNAME

-v, --verbose — Show deactivated MDTs and OSTs in the listing, along with any additional status flags for each MDT and OST. By default, any MDTs and OSTs that are deactivated by the administrator are not shown. However, targets that are only temporarily inaccessible are still shown. 🔥

✨ EXAMPLES ✨

Okay bestie, this is the part where it all clicks. Sit with me 💖

Wanna see space usage per OST and MDT for the testfs filesystem, in human-readable format? Say less:

# lfs df -hv /mnt/testfs
UUID                 bytes  Used   Avail Use% Mounted on
testfs-MDT0000_UUID  13.0G  1.2G   11.0G  10% /testfs[MDT:0]
testfs-OST0000_UUID   3.6T  2.9T  585.7G  84% /testfs[OST:0]
testfs-OST0001_UUID   3.6T  3.1T  472.5G  87% /testfs[OST:1] D
testfs-OST0002_UUID   3.6T  3.0T  570.3G  84% /testfs[OST:2] DR
OST0003           : inactive device
testfs-OST0006_UUID   5.4T  4.9T  417.8G  92% /testfs[OST:3]

filesystem_summary:  16.2T 13.8T    2.0T  88% /testfs

So what is this output even telling us 👀 The above example output shows that OST0003 is currently temporarily inactive or offline, while OST0004 and OST0005 are not shown at all — either because they are marked permanently offline by the administrator (via lctl set_param -P osc.testfs-OST000[45].active=0) or because they were never added to the filesystem. The testfs-OST0001 and testfs-OST0002 targets are currently marked Degraded (maybe they both share the same underlying storage controller? sharing is caring until it isn't 😭), while testfs-OST0002 is also marked Read-only, after detecting non-recoverable corruption in the backing filesystem. Not the glow-up we wanted.

Now for inode usage per OST and MDT across all mounted Lustre filesystems:

# lfs df -iH
UUID                Inodes  IUsed  IFree IUse% Mounted on
testfs-MDT0000_UUID 932.2k 884.6k  47.6k  95% /testfs[MDT:0]
testfs-OST0000_UUID 267.5k 179.6k  87.8k  67% /testfs[OST:0]
testfs-OST0001_UUID 268.9k 173.5k  95.4k  64% /testfs[OST:1] D
testfs-OST0002_UUID 267.5k 169.6k  97.9k  63% /testfs[OST:2] DR
OST0003           : inactive device
testfs-OST0006_UUID 426.1k 377.4k  48.7k  88% /testfs[OST:3]

filesystem_summary: 932.1k 884.6k  47.6k  95% /testfs

Just the ssd pool of testfs? We got you 🎀

# lfs df --pool ssd /mnt/testfs

Every single MDT and OST for testfs, even the ones that aren't currently connected (ghosting us, but we still count them) 💅

# lfs df -v /mnt/testfs

Only OST index 1 of testfs? Precisely. ✨

# lfs df -o1 /mnt/testfs

Custom fields, showing both block and inode stats at the same time, because why choose:

# lfs df --output=source,size,used,itotal,iused -h /mnt/lustre
UUID                       bytes        Used      Inodes       IUsed
lustre-MDT0000_UUID       122.1M        1.6M       97.7K         275
lustre-OST0000_UUID       305.8M        1.4M       97.7K         302
lustre-OST0001_UUID       305.8M        1.4M       97.7K         302

filesystem_summary:       611.5M        2.7M       97.7K         275

Only available space and usage percentages — minimal, clean, iconic:

# lfs df --output=source,avail,pcent,iavail,ipcent /mnt/lustre
UUID                   Available Use%       IFree IUse%
lustre-MDT0000_UUID       112156   2%       99725   1%
lustre-OST0000_UUID       284544   1%       99698   1%
lustre-OST0001_UUID       284544   1%       99698   1%

filesystem_summary:       569088   1%       99725   1%

Filesystem summary with the custom field order you picked yourself, main character energy:

# lfs df --output=pcent,avail,used,size,target /mnt/lustre
Use%   Available        Used   1K-blocks Mounted on
  2%      112156        1664      125056 /mnt/lustre[MDT:0]
  1%      284544        1400      313104 /mnt/lustre[OST:0]
  1%      284544        1400      313104 /mnt/lustre[OST:1]

  1%      569088        2800      626208 /mnt/lustre

Single field output with headers, for all targets:

# lfs df --output=avail /mnt/lustre
Available
    112156
    284544
    284544

    569088

Multiple fields with headers, obviously:

# lfs df --output=source,avail /mnt/lustre
UUID                   Available
lustre-MDT0000_UUID       112156
lustre-OST0000_UUID       284544
lustre-OST0001_UUID       284544

filesystem_summary:       569088

Multiple fields, but no headers and no filesystem summary — stripped down, no notes:

# lfs df --output=source,avail --no-header /mnt/lustre
lustre-MDT0000_UUID       112156
lustre-OST0000_UUID       284544
lustre-OST0001_UUID       284544

OST available space, no headers (super useful when you're hunting for the minimum, iykyk):

# lfs df --output=avail --no-header -o /mnt/lustre
     284544
     284544

Single field for target OST1, no headers — this one is made for scripting, fr fr:

# lfs df --output=avail --ost=1 /mnt/lustre
     284544

And if literally all you want is the filesystem summary and nothing else, periodt:

# lfs df --only-summary /mnt/lustre
filesystem_summary:       626208        2800      569088   1%  /mnt/lustre

AVAILABILITY 🌈

lfs df is part of the lustre(7) filesystem package since release 1.4.0. Been here since before some of us could walk 🫶