DWALK(1)

📦 version 0.12, in nixpkgs unstable

💅 DWALK(1) — the distributed walk & list moment

okay bestie, welcome to dwalk, brought to you by the mpiFileUtils fam (that's the manual name, mwah 💖). this is the page for dwalk(1) and honestly? it's giving "ls and du had a glow-up and went parallel." let's get into it.

✨ SYNOPSIS ✨

dwalk [OPTION] PATH ...

that's it. options, then one or more paths. slay.

🌸 DESCRIPTION 🌸

okay so first thing: dwalk is a parallel MPI application that recursively walks a directory and lists what's inside. like, recursively. tree mode. all the way down. 🌳

bestie it's literally ls(1) and du(1) energy combined:

  • like du(1), it reports a summary of the total number of files and bytes 👑
  • like ls(1), it sorts and prints information about individual files ✨

and the fun part? you can sort the output on different fields — name, user, group, size, etc. — and you can compute a histogram of file sizes, listing how many files fall into bins you define. custom bins. main character behavior fr fr. 💅

🎀 OPTIONS 🎀

  • -i, --input FILE — read the source list from FILE. heads up: FILE must have been generated by another tool from the mpiFileUtils suite. no random files, iykyk.
  • -o, --output FILE — write the processed list to FILE in binary format. the format can be changed with the --text option.
  • -t, --text — must be used with the --output option. writes the processed list of files to FILE in ascii text format. 📝
  • -l, --lite — walk the file system without stat. light work, no stat, no stress.
  • -s, --sort FIELD — sort output by comma-delimited fields (scroll down to SORT FIELDS, bestie 👇).
  • -d, --distribution size:SEPARATORS — print the distribution of file sizes. for example, specifying size:0,80,100 will report the number of files that have size 0 bytes, between 1-80 bytes, between 81-99 bytes, and 100 bytes or greater. bins on bins on bins 🍓
  • -f, --file-histogram — creates a file histogram without requiring the user to provide the bin sizes. the bins are created dynamically based on the max file size. the first bin is always for only zero byte files, and the rest go up until the max file size is included in the very last bin. it always goes up by orders of magnitude in powers of two. so, an example of bin separators would be: 0, 2^10, 2^20, 2^30 — assuming the max file size was somewhere within the 2^20-2^30 range. the histogram also includes both files and directories. iconic. 🌈
  • -p, --print — print files to the screen. yes queen, show us everything.
  • --no-atime — must be used with the --lite option. do not update last file access time. leave that atime alone 🙈
  • -L, --dereference — dereference symbolic links and walk the target file or directory that each symbolic link refers to. 🦋
  • --progress N — print a progress message to stdout approximately every N seconds. the number of seconds must be a non-negative integer. a value of 0 disables progress messages.
  • -v, --verbose — run in verbose mode. extra chatter, we love that for us.
  • -q, --quiet — run tool silently. no output is printed. 🤫
  • -h, --help — print usage.

🔥 SORT FIELDS 🔥

by default, the list of files dwalk captures is not sorted. (chaos, honestly.) to sort it, you can specify one or more fields in a comma-delimited list:

name,user,group,uid,gid,atime,mtime,ctime,size

a field name can be preceded with - to sort by that field in reverse order. (yes bestie, -size for the big files first, we see you 👀)

and if you give more than one field, a lexicographic sort is executed. no cap.

⭐ EXAMPLES ⭐

1. to print summary information for a directory:

mpirun -np 128 dwalk -v /dir/to/walk

2. to print a list of files, sorted by file size, then by file name:

mpirun -np 128 dwalk –print –sort size,name /dir/to/walk

3. to save the list of files:

mpirun -np 128 dwalk –output out.dwalk /dir/to/walk

4. print the file distribution for a specified histogram based on the size field from the top level directory:

mpirun -np 128 dwalk -v –print -d size:0,20,1G src/
ok real talk, that's exactly how the examples appear in the original page — including those funky dashes on –print, –sort, and –output. copying them character for character because that's the assignment 💅

👑 SEE ALSO 👑

the mpiFileUtils source code and all documentation may be downloaded from <https://github.com/hpc/mpifileutils> — go be nosy, bestie, it's a whole vibe.

💜 AUTHOR 💜

HPC

2022, LLNL/LANL/UT-Battelle/DDN

aaaand that's the page! 🎀 this was version 0.12, dated Jan 29, 2025 — so if you're reading this in the future, hi, hope the filesystem's cute. bye bestie, go walk some directories 💖✨🦋

📦 where this page comes from