RMLINT(1)

📦
version 2.10.3 · 10 versions
, in epel 10

🦋 rmlint(1) — rmlint documentation 🦋

okay so bestie. this is the manpage for rmlint, and the manual name up top is literally rmlint documentation, section 1, which is so on brand. it's a big one, so buckle up 🫶

🌸 NAME 🌸

rmlint — find duplicate files and other space waste efficiently 💅

🔥 FIND DUPLICATE FILES AND OTHER SPACE WASTE EFFICIENTLY 🔥

✨ SYNOPSIS ✨

rmlint [TARGET_DIR_OR_FILES ...] [//] [TAGGED_TARGET_DIR_OR_FILES ...]
[-] [OPTIONS]

💖 DESCRIPTION 💖

so rmlint goes digging through your filesystem to find space waste and other broken things, and honestly its main character energy is all about duplicate files and duplicate directories. 👑

here's the lint it can sniff out:

  • Duplicate files and directories (and as a by-product, unique files, cute).
  • Nonstripped binaries (binaries with debug symbols — but this one needs to be explicitly enabled, it doesn't just show up).
  • Broken symbolic links.
  • Empty files and directories (nested empty directories too, it sees all).
  • Files with broken user or group ID.

rmlint itself WILL NOT DELETE ANY FILES. not a single one. no cap. 🫶 what it does do is produce executable output (like a shell script) to help you delete stuff if that's what you want. another whole design principle: it should play nice with other tools like find. so it does not replicate features of other well-known programs — like pattern matching, or finding duplicate filenames. but it does give you a ton of convenience options for common use cases that are genuinely hard to build from scratch with standard tools. slay.

to find the lint, you hand rmlint one or more directories to traverse. if you give it no directories or files at all, it assumes the current working directory. by default rmlint will ignore hidden files and will not follow symlinks (see Traversal Options, later bestie). it first finds "other lint" and then searches the remaining files for duplicates. two-phase situation. 💫

rmlint also tries to be helpful by guessing which file in a group of duplicates is the original (i.e. the file that should not be deleted). it does this using different sorting strategies you can control with the -S option. by default it picks the first-named path on the command line. if two duplicates come from the same path, it also applies different fallback sort strategies (check the docs for the -S strategy).

you can override all that if you know one directory is full of duplicates and another is full of originals. in that case you write the original directory after a single // on the command line. everything after it is a preferred (or "tagged") directory. if there are duplicates from an unpreferred and a preferred directory, the preferred one always counts as the original. special options can also keep files in preferred directories forever (-k), or only find duplicates that exist in both given directories (-m).

new users: please have a lil look at all the options rmlint offers, and maybe test some examples before letting it run on productive data. WRONG ASSUMPTIONS ARE THE BIGGEST ENEMY OF YOUR DATA. 😭 there are extended examples at the end of this manual, and any option that isn't self-explanatory will try to give you examples too.

🎀 OPTIONS 🎀

💜 General Options 💜

  • -T, --types="list" (default: defaults) — configure what types of lint rmlint hunts for. the list string is a comma-separated list of lint types or lint groups (other separators like semicolon or space also work though, flexible queen).

one of these groups can be specified at the beginning of the list:

  • all: Enables all lint types.
  • defaults: Enables all lint types, but nonstripped.
  • minimal: defaults minus emptyfiles and emptydirs.
  • minimaldirs: defaults minus emptyfiles, emptydirs and duplicates, but with duplicatedirs.
  • none: Disable all lint types [default].

any of the following lint types can be added individually, or deselected by prefixing with a -:

  • badids, bi: Find files with bad UID, GID or both.
  • badlinks, bl: Find bad symlinks pointing nowhere valid.
  • emptydirs, ed: Find empty directories.
  • emptyfiles, ef: Find empty files.
  • nonstripped, ns: Find nonstripped binaries.
  • duplicates, df: Find duplicate files.
  • duplicatedirs, dd: Find duplicate directories (This is the same as -D!)
WARNING: it's good practice to enclose the description in single or double quotes. in obscure cases, argument parsing might fail in weird ways, especially when using spaces as separators. protect yourself. 🛡️

Examples:

$ rmlint -T "df,dd"        # Only search for duplicate files and directories
$ rmlint -T "all -df -dd"  # Search for all lint except duplicate files and dirs
  • -o, --output=spec / -O, --add-output=spec (default: -o sh:rmlint.sh -o pretty:stdout -o summary:stdout -o json:rmlint.json) — configure how rmlint spits out its results. a spec is in the form format:file or just format. a file might be an arbitrary path, or stdout, or stderr. if file is omitted, stdout is assumed. format is the name of a formatter supported by this program — for a list of formatters and their options, check the Formatters section below.

if -o is specified, rmlint's default outputs get overridden. with --O the defaults are preserved. either -o or -O may be specified multiple times to get multiple outputs, including multiple outputs of the same format. the more the merrier fr fr.

Examples:

$ rmlint -o json                 # Stream the JSON output to stdout
$ rmlint -O csv:/tmp/rmlint.csv  # Output an extra CSV file to /tmp
  • -c, --config=spec[=value] (default: none) — configure a format. this one is for fine-tuning the behaviour of the existing formatters. see the Formatters section for details on the available keys.

if the value is omitted it's set to a value meaning "enabled". ✨

Examples:

$ rmlint -c sh:link            # Smartly link duplicates instead of removing
$ rmlint -c progressbar:fancy  # Use a different theme for the progress bar
  • -z, --perms[=[rwx]] (default: no check) — only look into a file if it's readable, writable or executable by the current user. permissions can be given as arguments as one of rwx.

if no argument is given, rw is assumed. note that r does basically nothing user-visible since rmlint will ignore unreadable files anyways — it's just there for the sake of completeness. completionist behavior, honestly iconic.

by default this check is not done.

$ rmlint -z rx $(echo $PATH | tr ":" " ")  # Look at all executable files in $PATH
  • -a, --algorithm=name (default: blake2b) — choose the algorithm used to find duplicate files. it can be either paranoid (byte-by-byte file comparison) or use one of several file hash algorithms to identify duplicates. these hash families are available (in approximate descending order of cryptographic strength):

sha3, blake,

sha,

highway, md

metro, murmur, xxhash

the weaker hash functions still offer excellent distribution properties, but they're potentially more vulnerable to malicious crafting of duplicate files. 👀

the full list of hash functions (in decreasing order of checksum length) is:

  • 512-bit: blake2b, blake2bp, sha3-512, sha512
  • 384-bit: sha3-384,
  • 256-bit: blake2s, blake2sp, sha3-256, sha256, highway256, metro256, metrocrc256
  • 160-bit: sha1
  • 128-bit: md5, murmur, metro, metrocrc
  • 64-bit: highway64, xxhash.
using a 64-bit hash length for detecting duplicate files is not recommended, due to the probability of a random hash collision. don't do it bestie, it's giving risky. 💀
  • -p, --paranoid / -P, --less-paranoid (default) — increase or decrease the paranoia of rmlint's duplicate algorithm. use -p if you want byte-by-byte comparison without any hashing at all.
    • -p is equivalent to --algorithm=paranoid
    • -P is equivalent to --algorithm=highway256
    • -PP is equivalent to --algorithm=metro256
    • -PPP is equivalent to --algorithm=metro
  • -v, --loud / -V, --quiet — increase or decrease the verbosity. you can pass these options several times. this only affects rmlint's logging on stderr, not the outputs defined with -o. passing either option more than three times has no further effect (it's capped, bestie, there's only so much).
  • -g, --progress / -G, --no-progress (default) — show a progress bar with sane defaults.

convenience shortcut for -o progressbar -o summary -o sh:rmlint.sh -o json:rmlint.json -VVV.

NOTE: this flag clears all previous outputs. if you want additional outputs, specify them after this flag using -O. order matters, always has. 🎀
  • -D, --merge-directories (default: disabled) — makes rmlint use a special mode where all found duplicates are collected and then checked to see if whole directory trees are duplicates. use with caution: you should always make sure the investigated directory isn't modified while rmlint or its removal scripts run. 😭

IMPORTANT: Definition of equal: two directories are considered equal by rmlint if they contain the exact same data, no matter how the files containing the data are named. imagine rmlint creates a long, sorted stream out of the data found in the directory and compares this in a magic way to another directory. this means the layout of the directory is not considered important by default. also, empty files will not count as content. this might be surprising to some users, but remember rmlint generally cares only about content — not about any other metadata or layout. if you want to only find trees with the same hierarchy you should use --honour-dir-layout / -j.

output is deferred until all duplicates were found. duplicate directories are printed first, followed by any remaining duplicate files that are isolated or inside of any original directories.

--rank-by applies for directories too, but p or P (path index) has no defined (i.e. useful) meaning. sorting only takes place when the number of preferred files in the directory differs.

NOTES:

  • this option enables --partial-hidden and -@ (--see-symlinks) for convenience. if that's not desired, you should change it after specifying -D.
  • this feature might add some runtime cost for large datasets. she's thorough, not free. ⭐
  • When you're using this option, you cannot use the -c sh:clone option. Use -c sh:link instead — it's a great alternative, fr fr.
  • -j, --honour-dir-layout (default: disabled) — Only counts directories as dupes if they have the same path layout. Basically: every duplicate that builds the duplicate directory has to have the same path from the root of its respective directory. Lowkey this flag makes zero sense without --merge-directories, so don't come crying to me.
  • *-y, --sort-by=order* (default: none) — Sorts the found duplicate groups during output by whatever criteria order describes. order is a string, and it can hold one or more of these letters:
    • s: sort by size of group.
    • a: sort alphabetically by the basename of the original.
    • m: sort by mtime of the original.
    • p: sort by path-index of the original.
    • o: sort by natural found order (might be different on each run — chaotic, I know).
    • n: sort by number of files in the group.

You can also write the letter in uppercase (same energy as -S / --rank-by) to reverse the sorting. Heads up tho: rmlint has to hold back all results until the end of the run before it can sort and print them. So yeah, that's a thing. 💅

  • -w, --with-color (default) / -W, --no-with-color — Uses color escapes for pretty output, or disables them entirely. If you pipe rmlint's output into a file, -W is assumed automatically. Slay.
  • -h, --help / -H, --show-man — Shows a shorter reference help text (-h) or the full man page (-H). Pick your fighter.
  • --version — Prints the version of rmlint. Includes the Git revision and compile time features. Please, please include this when you give us feedback — it's giving "help us help you". 🙏

🌸 Traversal Options 🌸

  • *-s, --size=range* (default: "1") — Only considers files as duplicates if they're inside a certain size range. The format of range is min-max, where both ends can be a number with an optional multiplier. The available multipliers are:
    • C (1^1), W (2^1), B (512^1), K (1000^1), KB (1024^1), M (1000^2), MB (1024^2), G (1000^3), GB (1024^3),
    • T (1000^4), TB (1024^4), P (1000^5), PB (1024^5), E (1000^6), EB (1024^6)

The size format is basically the same one dd(1) uses. A valid example would be: "100KB-2M". That limits duplicates to a range from 100 kilobytes to 2 megabytes. Iconic.

You can also specify just one size. In that case the size is interpreted as "bigger or equal". If you want to filter for files up to that size, you can slap a - in front (-s -1M == -s 0-1M).

Edge case bestie: the default excludes empty files from the duplicate search. Normally those get treated specially by rmlint, handling them as other lint. If you want to include empty files as duplicates you should lower the limit to zero:

$ rmlint -T df --size 0
  • *-d, --max-depth=depth* (default: INF) — Only recurses up to this depth. A depth of 1 would disable recursion and is literally equivalent to a directory listing. A depth of 2 would also consider all children directories, and so on. Recursion but make it boundaries. 🦋
  • -l, --hardlinked (default) / --keep-hardlinked / -L, --no-hardlinked — Hardlinked files are treated as duplicates by default (--hardlinked). If --keep-hardlinked is given, rmlint will not delete any files that are hardlinked to an original in their respective group. Those files get displayed like originals, i.e. for the default output with a "ls" in front. The reasoning here is to maximize the number of kept files while maximizing the amount of freed space: removing hardlinks to originals won't allocate any free space. Math is math.

If --no-hardlinked is given, only one file (out of a set of hardlinked files) is considered and all the others are ignored; meaning they're not deleted and not even shown in the output. The "highest ranked" of the set is the one that gets considered. Periodt.

  • -f, --followlinks / -F, --no-followlinks / -@, --see-symlinks (default) — -f will always follow symbolic links. If filesystem loops occur, rmlint will detect this. If -F is specified, symbolic links are ignored completely. If -@ is specified, rmlint will see symlinks and treat them like small files with the path to their target inside them. The latter is the default behaviour, since it's a sensible default for --merge-directories. 💜
  • -x, --no-crossdev / -X, --crossdev (default) — Stay always on the same device (-x), or allow crossing mountpoints (-X). The latter is the default. No cap.
  • -r, --hidden / -R, --no-hidden (default) / --partial-hidden — Also traverse hidden directories? Honestly this is often not a good idea, since directories like .git/ would be investigated, possibly leading to the deletion of internal git files — which in turn breaks a repository. Genuinely tragic. 😭 With --partial-hidden, hidden files and directories are only considered if they're inside duplicate directories (see --merge-directories) and will be deleted as part of it.
  • -b, --match-basename — Only considers files as dupes if they have the same basename. See also man 1 basename. The comparison of the basenames is case-insensitive.
  • -B, --unmatched-basename — Only considers files as dupes if they do not share the same basename. See also man 1 basename. Comparison is case-insensitive here too.
  • -e, --match-extension / -E, --no-match-extension (default) — Only considers files as dupes if they have the same file extension. So like, two photos would only match if they're both a .png. The extension is compared case-insensitively, so .PNG is the same as .png. Slay.
  • -i, --match-without-extension / -I, --no-match-without-extension (default) — Only considers files as dupes if they have the same basename minus the file extension. For example: banana.png and Banana.jpeg would be considered dupes, while apple.png and peach.png won't. Comparison is case-insensitive. 🍓
  • -n, --newer-than-stamp=timestamp_filename / -N, --newer-than=iso8601_timestamp_or_unix_timestamp — Only considers files (and their size siblings for duplicates) newer than a certain modification time (mtime). The age barrier may be given as seconds since the epoch or as an ISO8601 timestamp like 2014-09-08T00:12:32+0200.

-n expects a file from which it can read the timestamp. After rmlint runs, that file gets updated with the current timestamp. If the file doesn't initially exist, no filtering is done — but the stampfile still gets written. Bestie works either way.

-N, in contrast, takes the timestamp directly and won't write anything.

Note that rmlint will find duplicates newer than timestamp even if the original is older. If you want to only find duplicates where both original and duplicate are newer than timestamp, you can use find(1):

  • find -mtime -1 -print0 | rmlint -0 # pass all files younger than a day to rmlint

Note: you can make rmlint write out a compatible timestamp with:

  • -O stamp:stdout # Write a seconds-since-epoch timestamp to stdout on finish.
  • -O stamp:stdout -c stamp:iso8601 # Same, but write as ISO8601.

👑 Original Detection Options 👑

  • -k, --keep-all-tagged / -K, --keep-all-untagged — Don't delete any duplicates that are in tagged paths (-k) or that are in non-tagged paths (-K). (Tagged paths are the ones that were listed after //.) iykyk.
  • -m, --must-match-tagged / -M, --must-match-untagged — Only looks for duplicates where at least one is in one of the tagged paths. (Paths that were named after //.)

Note that the combinations of -kM and -Km are prohibited by rmlint. See https://github.com/sahib/rmlint/issues/244 for more information. Don't try it, it's giving rule-breaking. 💀

  • *-S, --rank-by=criteria* (default: pOma) — Sorts the files in a group of duplicates into originals and duplicates by one or more criteria. Each criterion is defined by a single letter (except r and x, which expect a regex pattern after the letter). Multiple criteria may be given as a string, where the first criterion is the most important. If one criterion can't decide between original and duplicate, the next one is tried. Very "main character determines the fate of everyone else" energy.
    • m: keep lowest mtime (oldest) — M: keep highest mtime (newest)
    • a: keep first alphabetically — A: keep last alphabetically
    • p: keep first named path — P: keep last named path
    • d: keep path with lowest depth — D: keep path with highest depth
    • l: keep path with shortest basename — L: keep path with longest basename
    • r: keep paths matching regex — R: keep path not matching regex
    • x: keep basenames matching regex — X: keep basenames not matching regex
    • h: keep file with lowest hardlink count — H: keep file with highest hardlink count
    • o: keep file with lowest number of hardlinks outside of the paths traversed by rmlint.
    • O: keep file with highest number of hardlinks outside of the paths traversed by rmlint.

Alphabetical sort only uses the basename of the file and ignores its case. You can have multiple criteria, e.g.: -S am will choose first alphabetically; if tied then by mtime. Note: original path criteria (specified using //) will always take first priority over -S options. Always. No exceptions.

For more fine-grained control, it's possible to give a regular expression to sort by. This is super useful when you know a common fact that identifies original paths (like a path component being src, or a certain file ending). 🧚

To use the regular expression you simply enclose it in the criteria string by adding <REGULAR_EXPRESSION> after specifying r or x. Example: -S 'r<.*\.bak$>' makes all files that have a .bak suffix original files. Ate.

Warning: When using r or x, try to make your regex as specific as possible! Good practice includes adding a $ anchor at the end of the regex. Don't be out here writing loose regexes, bestie, it will not end well. 🙈

Tips:

  • l is useful for files like file.mp3 vs file.1.mp3 or file.mp3.bak.
  • a can be used as the last criterion to assert a defined order.
  • o/O and h/H are only useful if there are hardlinks in the traversed path.
  • o/O — these count the number of hardlinks outside the paths you're traversing (so they minimize/maximize the overall number of hardlinks). h/H, on the other hand, only counts hardlinks inside the traversed paths. When you're hardlinking files you want to link to the original file with the highest outer link count (O), because that's what actually maximizes your space cleanup. H does not maximize space cleanup — it just picks the file with the highest total hardlink count. So yeah, you usually want to specify O 💅
  • pOma is the default, and here's why each letter eats: p makes sure the first paths you gave rank as originals, O makes hardlinks get handled well, m makes the oldest file the original, and a just makes sure there's a defined ordering when no other criteria kicks in. Iconic behavior, no notes 🎀

Caching

  • --replay — reads an existing JSON file and re-outputs it. When --replay is on, rmlint does no input/output on the filesystem, even if you pass extra paths. Those paths you pass get used for filtering the --replay output instead.

This is genuinely so useful if you want to reformat, refilter, or resort the output from a previous run. Usage is simple: just pass --replay on the second run, along with whatever other changes you made to the new formatters or filters. Pass the .json files from the previous runs in addition to the paths you ran rmlint on. You can even merge several previous runs by specifying more than one .json file — in that case it merges all the files you gave it and outputs them as one big run. Obsessed 🫶

If you want to view only the duplicates of certain subdirectories, just pass them on the command line like you normally would.

Using // has the same effect as in a normal run — it can be used to prefer one .json file over another. But note: running rmlint in --replay mode includes no real disk traversal, i.e. only duplicates from previous runs get printed. So specifying new paths will simply have no effect. And as a security measure, --replay will ignore files whose mtime changed in the meantime (i.e. the mtime in the .json file differs from the current one). Those files might have been modified, and they're silently ignored. 🫥

By design, some options will not have any effect. Those are:

  • --followlinks
  • --algorithm
  • --paranoid
  • --clamp-low
  • --hardlinked
  • --write-unfinished
  • ... and all other caching options below.
NOTE: In --replay mode, a new .json file will be written to rmlint.replay.json in order to avoid overwriting rmlint.json.
  • -C, --xattr — shortcut for --xattr-read, --xattr-write, --write-unfinished. This writes a checksum and a timestamp to the extended attributes of each file that rmlint hashed, which speeds up subsequent runs on the same data set. Please note that not all filesystems may support extended attributes, and you need write support to use this feature.

See the individual options below for more details and some examples 💖

  • --xattr-read / --xattr-write / --xattr-clear — read or write cached checksums from the extended file attributes. This feature can be used to speed up consecutive runs.
CAUTION: this could potentially lead to false positives if file contents are somehow modified without changing the file modification time. rmlint uses the mtime to determine the modification timestamp if a checksum is outdated. This is not a problem if you use the clone or reflink operation on a filesystem like btrfs — there, an outdated checksum entry would simply lead to some duplicate work done in the kernel, but would do no harm otherwise.
NOTE: many tools do not support extended file attributes properly, resulting in a loss of the information when copying the file or editing it.
NOTE: you can specify --xattr-write and --xattr-read at the same time. This will read from existing checksums at the start of the run and update all hashed files at the end.

--xattr-write has no effect when --clamp-low or --clamp-top is used, to prevent false negatives in future runs without clamping.

Usage example, bestie:

$ rmlint large_file_cluster/ -U --xattr-write   # first run should be slow.
$ rmlint large_file_cluster/ --xattr-read       # second run should be faster.

# Or do the same in just one run:
$ rmlint large_file_cluster/ --xattr
  • -U, --write-unfinished — include files in the output that have not been hashed fully, i.e. files that do not appear to have a duplicate. Note that this will not include all files that rmlint traversed — only the files that were chosen to be hashed.

This is mainly useful in conjunction with --xattr-write/read. When re-running rmlint on a large dataset this can greatly speed up a re-run in some cases. Please refer to --xattr-read for an example ✨

If you want to output unique files, please look into the uniques output formatter.

Rarely used, miscellaneous options

  • -t, --threads=N (default: 16) — the number of threads to use during file tree traversal and hashing. rmlint probably knows better than you how to set this value, so just leave it as it is. And no, setting it to 1 will also not make rmlint a single threaded program 😭
  • -u, --limit-mem=size — apply a maximum amount of memory to use for hashing and --paranoid. The total amount of memory might still exceed this limit though, especially when setting it very low. In general rmlint will however consume about this amount of memory plus a more or less constant extra amount that depends on the data you are scanning.

The size-description has the same format as for --size, so you can do something like this (use this if you have 1GB of memory available):

$ rmlint -u 512M  # Limit paranoid mem usage to 512 MB
  • -q, --clamp-low=[fac.tor|percent%|offset] (default: 0) / -Q, --clamp-top=[fac.tor|percent%|offset] (default: 1.0) — the argument can be passed either as a factor (a number with a . in it), a percent value (suffixed by %), or as an absolute number or size spec, like in --size.

Only look at the content of files in the range from low to (including) high. This means if the range is less than -q 0% to -Q 100%, then only partial duplicates are searched. If the file size is less than the clamp limits, the file is ignored during traversing. Be careful when using this function — you can easily get dangerous results for small files 💀

This is useful in a few cases where a file consists of a constant sized header or footer. With this option you can just compare the data in between. It might also be useful for approximate comparison, where it suffices that the file is the same in the middle part.

Example:

$ rmlint -q 10% -Q 512M  # Only read the last 90% of a file, but
read at max. 512MB
  • -Z, --mtime-window=T (default: -1) — only consider those files as duplicates that have the same content and the same modification time (mtime) within a certain window of T seconds. If T is 0, both files need to have the same mtime. For T=1 they may differ one second, and so on. If the window size is negative, the mtime of duplicates will not be considered. T may be a floating point number 🌸

However — and this is the sneaky part — with three (or more) files, the mtime difference between two duplicates can be bigger than the mtime window T, i.e. several files may be chained together by the window. Example: if T is 1, the four files fooA (mtime: 00:00:00), fooB (00:00:01), fooC (00:00:02), fooD (00:00:03) would all belong to the same duplicate group, although the mtime of fooA and fooD differs by 3 seconds. Chain reaction, lowkey 🦋

  • --with-fiemap (default) / --without-fiemap — enable or disable reading the file extents on rotational disks in order to optimize disk access patterns. If this feature is not available, it is disabled automatically.

🌈 FORMATTERS 🌈

  • csv — output all found lint as a comma-separated-value list.

Available options:

  • no_header — do not write a first line describing the column headers.
  • unique — include unique files in the output.
  • sh — output all found lint as a shell script. This formatter is activated per default 👑

Available options:

  • cmd — specify a user defined command to run on duplicates. The command can be any valid /bin/sh expression. The duplicate path and original path can be accessed via "$1" and "$2". The command will be written to the user_command function in the sh file produced by rmlint.
  • handler — define a comma-separated list of handlers to try on duplicate files in that given order until one handler succeeds. Handlers are just the name of a way of getting rid of the file, and can be any of the following:
    • clone — for reflink-capable filesystems only. Try to clone both files with the FIDEDUPERANGE ioctl(3p) (or BTRFS_IOC_FILE_EXTENT_SAME on older kernels). This will free up duplicate extents. Needs at least kernel 4.2. Use this option when you only have read-only access to a btrfs filesystem but still want to deduplicate it — this is usually the case for snapshots.
    • reflink — try to reflink the duplicate file to the original. See also --reflink in man 1 cp. Fails if the filesystem does not support it.
    • hardlink — replace the duplicate file with a hardlink to the original file. The resulting files will have the same inode number. Fails if both files are not on the same partition. You can use ls -i to show the inode number of a file, and find -samefile <path> to find all hardlinks for a certain file 🔗
    • symlink — tries to replace the duplicate file with a symbolic link to the original. This handler never fails.
    • remove — remove the file using rm -rf (-r for duplicate dirs). This handler never fails.
    • usercmd — use the provided user-defined command (-c sh:cmd=something). This handler never fails.

Default is remove. ✨

  • link — shortcut for -c sh:handler=clone,reflink,hardlink,symlink. Use this one if you're on a reflink-capable system 🦋
    • hardlink — shortcut for -c sh:handler=hardlink,symlink. Reach for this when you want hardlinks but still want a fallback for the duplicates that happen to live on different devices 🫶
    • symlink — shortcut for -c sh:handler=symlink. Use this as your last straw, bestie 😭
  • json — prints a JSON-formatted dump of every report it found. Outputs all lint as a JSON document, and that document is a list of dictionaries where the first and last elements are the header and the footer. Everything in between? Data dictionaries ✨

Available options:

  • unique — include unique files in the output.
  • no_header=[true|false] — print the header with metadata (default: true).
  • no_footer=[true|false] — print the footer with statistics (default: true).
  • oneline=[true|false] — print one JSON document per line (default: false). Highkey useful if you're planning to parse the output line-by-line, like while rmlint is still running 👀

This formatter is extremely useful when you need to script more complex behaviour that isn't directly possible with rmlint's built-in options. A very handy tool here is jq. Here's an example that outputs all original files directly from an rmlint run:

$ rmlint -o json | jq -r '.[1:-1][] | select(.is_original) | .path'
  • py — outputs a Python script and a JSON document, just like the json formatter. The JSON document gets written to .rmlint.json, and executing the script will make it read from there. This formatter is mostly intended for complex use-cases where the lint needs special handling that you define in the Python script. That's why the Python script can be modified to do things standard rmlint isn't able to do easily 🧚
  • uniques — outputs all unique paths found during the run, one path per line. Often super useful for scripting purposes 💜

Available options:

  • print0 — don't put newlines between paths, put zero bytes instead.
  • stamp — outputs a timestamp of the time rmlint was run. See also the --newer-than and --newer-than-stamp file options ⭐

Available options:

  • iso8601=[true|false] — write ISO8601-formatted timestamps or seconds since epoch?
  • progressbar — shows a progress bar. This is meant for use with stdout or stderr [default] 🍓

See also: -g (--progress) for a convenience shortcut option.

Available options:

  • update_interval=number — number of milliseconds to wait between updates. Higher values use less resources (default 50).
  • ascii — don't attempt to use Unicode characters, which might not be supported by some terminals.
  • fancy — use a more fancy style for the progress bar 💅
  • pretty — shows all found items in realtime, nicely colored. This formatter is activated per default 👑
  • summary — shows counts of files and their respective size after the run. Also lists all written output files.
  • fdupes — prints an output similar to the popular duplicate finder fdupes(1). At first, a progress bar is printed on stderr. Afterwards the found files are printed on stdout; each set of duplicates gets printed as a block separated by newlines. Originals are highlighted in green. At the bottom a summary is printed on stderr. This is mostly useful for scripts that were set up for parsing fdupes output. We recommend the json formatter for every other scripting purpose 🌈

Available options:

  • omitfirst — same as the -f / --omitfirst option in fdupes(1). Omits the first line of each set of duplicates (i.e. the original file).
  • sameline — same as the -1 / --sameline option in fdupes(1). Doesn't print newlines between files, only a space. Newlines are printed only between sets of duplicates.

👑 OTHER STAND-ALONE COMMANDS 👑

rmlint --gui

Starts the optional graphical frontend to rmlint called Shredder 🎀

This will only work when Shredder and its dependencies were installed. See also: https://rmlint.readthedocs.io/en/latest/gui.html

The GUI has its own set of options — check --gui --help for a list. These should be placed at the end, i.e. rmlint --gui [options] when calling it from the command line.

rmlint --hash [paths...]

Makes rmlint work as a multi-threaded file hash utility, similar to the popular md5sum or sha1sum utilities, but faster and with more algorithms 🔥 A set of paths given on the command line or from stdin is hashed using one of the available hash algorithms. Use rmlint --hash -h to see options.

rmlint --equal [paths...]

Checks if the paths given on the command line all have equal content. If all paths are equal and no other error happened, rmlint will exit with an exit code 0. Otherwise it will exit with a nonzero exit code. All other options can be used as normal, but heads up: no other formatters (sh, csv etc.) will be executed by default. At least two paths need to be passed.

Note: this even works for directories and also in combination with paranoid mode (pass -p for byte comparison) — just remember rmlint does not care about the layout of the directory, only about the content of the files in it. At least two paths need to be given to the command line 💀

By default this will use hashing to compare the files and/or directories.

rmlint --dedupe [-r] [-v|-V] <src> <dest>

If the filesystem supports files sharing physical storage between multiple files, and if src and dest have the same content, this command makes the data in the src file appear in the dest file by sharing the underlying storage ✨

This command is similar to cp --reflink=always <src> <dest> except that it (a) checks that src and dest have identical data, and (b) it makes no changes to dest's metadata.

Running with the -r option will enable deduplication of read-only [btrfs] snapshots (requires root) 🙈

Tests whether file1 and file2 are reflinks (they reference the same data). This command makes rmlint exit with one of the following exit codes:

  • 0: files are reflinks
  • 1: files are not reflinks
  • 3: not a regular file
  • 4: file sizes differ
  • 5: fiemaps can't be read
  • 6: resolved absolute paths of file1 and file2 are identical
  • 7: file1 and file2 are the same file under different mountpoints
  • 8: files are hardlinks
  • 9: files are symlinks
  • 10: files are not on the same device
  • 11: other error encountered

🌸 EXAMPLES 🌸

This is a collection of common use cases and other tricks:

  • Check the current working directory for duplicates:
$ rmlint
  • Show a progress bar:
$ rmlint -g
  • Quick re-run on large datasets using different ranking criteria on the second run:
$ rmlint large_dir/ # First run; writes rmlint.json

$ rmlint --replay rmlint.json large_dir -S MaD
  • Merge together previous runs, but prefer the originals to be from b.json and make sure that no files are deleted from b.json:
$ rmlint --replay a.json // b.json -k
  • Search only for duplicates and duplicate directories:
$ rmlint -T "df,dd" .
  • Compare files byte-by-byte in current directory:
$ rmlint -p .
  • Find duplicates with the same basename (excluding extensions):
$ rmlint -b
  • Do more complex traversal using find(1):
$ find /usr/lib -iname '*.so' -type f | rmlint - # find all duplicate .so files

$ find /usr/lib -iname '*.so' -type f -print0 | rmlint -0 # as above but handles filenames with newline characters in them

$ find ~/pics -iname '*.png' | ./rmlint - # compare png files only
  • Limit file size range to investigate:
$ rmlint -s 2GB    # Find everything >= 2GB

$ rmlint -s 0-2GB  # Find everything <  2GB
  • Only find writable and executable files:
$ rmlint --perms wx
  • Reflink if possible, else hardlink duplicates to original if possible, else replace duplicate with a symbolic link:
$ rmlint -c sh:link
  • Inject a user-defined command into shell script output:
$ rmlint -o sh -c sh:cmd='echo "original:" "$2" "is the same as" "$1"'
  • Use shred to overwrite the contents of a file fully:
$ rmlint -c 'sh:cmd=shred -un 10 "$1"'
  • Use data as master directory. Find only duplicates in backup that are also in data. Do not delete any files in data:
$ rmlint backup // data --keep-all-tagged --must-match-tagged
  • Compare if the directories a, b, and c are equal:
$ rmlint --equal a b c && echo "Files are equal" || echo "Files are not equal"
  • Test if two files are reflinks:
$ rmlint --is-reflink a b && echo "Files are reflinks" || echo "Files are not reflinks".
  • Cache calculated checksums for the next run. The checksums will be written to the extended file attributes:
$ rmlint --xattr
  • Produce a list of unique files in a directory:
$ rmlint -o uniques
  • Produce a list of files that are unique, including original files ("one of each"):
$ rmlint t -o json -o uniques:unique_files | jq -r '.[1:-1][] | select(.is_original) | .path' | sort > original_files $ cat unique_files original_files
  • Sort files by a user-defined regular expression:
# Always keep files with ABC or DEF in their basename,
# dismiss all duplicates with tmp, temp or cache in their names
# and if none of those are applicable, keep the oldest files instead.
$ ./rmlint -S 'x<.*(ABC|DEF).*>X<.*(tmp|temp|cache).*>m' /some/path
  • Sort files by adding priorities to several user-defined regular expressions:
# Unlike the previous snippet, this one uses priorities:
# Always keep files in ABC, DEF, GHI by following that particular order of
# importance (ABC has top priority), dismiss all duplicates with
# tmp, temp, cache in their paths and if none of those are applicable,
# keep the oldest files instead.
$ rmlint -S 'r<.*ABC.*>r<.*DEF.*>r<.*GHI.*>R<.*(tmp|temp|cache).*>m' /some/path

💅 PROBLEMS

Okay bestie, listen up — there are like two (2) whole things you should know about before you go feral on your filesystem:

  1. False positives — depending on the options you use, there's a very slight risk of false positives (that's when files get erroneously detected as duplicates, aka the algorithm fumbles the bag 😭). The default hash function (blake2b) is very safe, but in theory it's possible for two files to have the same hash. Like, if you had 10^73 different files, all the same size, then the chance of a false positive is still less than 1 in a billion. Literally astronomically unlikely, no cap. But if you're concerned, just use the --paranoid (-p) option. That one compares all the files byte-by-byte and is not much slower than blake2b (it may even be faster 👀), although it is a lot more memory-hungry. Trade-offs, bestie.
  2. File modification during or after an rmlint run — it's possible that a file rmlint recognized as a duplicate gets modified afterwards, resulting in a different file. Rude, honestly. If you use the rmlint-generated shell script to delete the duplicates, you can run it with the -p option to do a full re-check of the duplicate against the original before it deletes the file. Safe queen behaviour 💖. But when you're using -c sh:hardlink or -c sh:symlink, care should be taken, because a modification of one file will now result in a modification of all files. Yep, they're linked, that's the whole point. This is not the case for -c sh:reflink or -c sh:clone. Use -c sh:link to minimise this risk. ✨

🦋 SEE ALSO

Reading the manpages of these tools might help with working with rmlint:

Extended documentation and an in-depth tutorial can be found at:

🫶 BUGS

If you found a bug, have a feature request, or just want to say something nice (we love nice 💖), please visit https://github.com/sahib/rmlint/issues.

Please make sure to describe your problem in detail. Always include the version of rmlint (--version). And if you experienced a crash, please include at least one of the following pieces of information with a debug build of rmlint:

  • gdb --ex run -ex bt --args rmlint -vvv [your_options]
  • valgrind --leak-check=no rmlint -vvv [your_options]

You can build a debug build of rmlint like this:

  • git clone git@github.com:sahib/rmlint.git
  • cd rmlint
  • scons GDB=1 DEBUG=1
  • sudo scons install # Optional

👑 LICENSE

rmlint is licensed under the terms of the GPLv3.

See the COPYRIGHT file that came with the source for more information.

🌈 PROGRAM AUTHORS

rmlint was written by:

Also see https://rmlint.rtfd.org for other people that helped us. 🫶

💜 AUTHOR

Christopher Pahl, Daniel Thomas, Cebtenzzre

2014-2026, Christopher Pahl, Daniel Thomas, and Cebtenzzre

And that's the whole page, bestie — you made it to the end, main character energy fr fr 🌸 This little icon of a manual page is dated Jan 17, 2026 and lives in the RMLINT(1) section of the manual, so go forth, deduplicate responsibly, and slay ✨💅🦋

📦 where this page comes from
  • epel 10: rmlint 2.10.3-6.el10_3 · GPL-3.0-only AND LicenseRef-Callaway-MIT AND LicenseRef-Callaway-BSD AND (CC0-1.0 OR Apache-2.0 OR OpenSSL) AND LicenseRef-Callaway-Public-Domain
  • 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash-1017 on : a language model's rewrite of the original, under its terms