dupd(1)

📦
version 1.7.3 · 3 versions
, in ubuntu stonking

💖 dupd(1) — the find-duplicate-files girlies' manual 💖

okay so real talk, this is dupd, from the General Commands Manual, and she is that girl. let's get into it bestie 🫶

✨ NAME ✨

dupd — find duplicate files. that's it. that's the whole job. 💅

🌸 SYNOPSIS 🌸

dupd COMMAND [OPTIONS]

🦋 DESCRIPTION 🦋

okay so dupd scans, like, all the files in the path(s) you give it, to find files that have duplicate content. no cap. 🔥

here's the tea tho: the sets of duplicate files are not displayed during a scan. instead, the duplicate info gets saved into a database, which you can totally query with subsequent commands without having to scan all the files again. it's giving efficiency. it's giving main character energy. 👑

even though dupd can be used as a simple duplicate reporting tool — similar to how other duplicate finders work, by running dupd scan ; dupd report — the real power of dupd comes from interactively exploring the filesystem for duplicates after the scan has completed. see the file, ls, dups, uniques and refresh commands. also go read the STALE DATABASE section for more context. iykyk. 💜

additional documentation and examples are available under the docs directory in the source tree. if you don't have the source tree available, see https://github.com/jvirkki/dupd/blob/master/docs/index.md 💖

👑 COMMANDS 👑

as noted in the synopsis, the first argument to dupd must be the command to run. the command is one of:

  • scan — scan files looking for duplicates
  • report — show duplicate report from last scan
  • file — check for duplicates of one file
  • ls — list info about every file
  • dups — list all duplicate files
  • uniques — list all unique files
  • refresh — remove deleted files from the database
  • validate — revalidate all duplicates in database
  • rmsh — create shell script to delete all duplicates (use with care!)
  • help — show brief usage info
  • version — show version and exit

💅 OPTIONS 💅

🍓 scan — Perform the filesystem scan for duplicates.

  • -p, --path PATH — recursively scan the directory tree starting at this path. this option can be given multiple times to specify multiple directory trees to scan. if no --path option is given, the default is to start scanning from the current directory. 🌈
  • -m, --minsize SIZE — minimum size (in bytes) to include in scan. by default, all files with 1 byte or more are scanned. in practice, duplicates in files that small are rarely interesting, so you can speed up the scan by ignoring smaller files. ✨
  • --buflimit LIMIT — limit read buffer size. LIMIT may be an integer in bytes, or include a suffix of M for megabytes or G for gigabytes. the scan animation shows the percentage of buffer space in use (%b). unless that value goes up to 100% or beyond during a scan, there is no need to adjust this limit. setting this limit to a low value will constrain dupd memory usage but possibly at a cost to performance (depends on the data set). 🧚
  • -X, --one-file-system — for each path scanned, do not cross over to a different filesystem. this is helpful, for example, if you want to scan / but want to avoid any other mounted filesystems such as NFS mounts or external drives. 🛑
  • --hidden — include hidden files (and hidden directories) in the scan. by default these are not included. 🙈
  • --db PATH — override the default database file location. the default is $HOME/.dupd_sqlite. if you override the path during scan, remember to provide this argument and the path for subsequent operations so the database can be found. ⚠️
  • -I, --hardlink-is-unique — consider hard links to the same file content as unique. by default hard links are listed as duplicates. see the HARD LINKS section below. note that if this option is given during scan, it cannot be given during interactive operations. 💀
  • --stats-file FILE — on completion, create (or append to) FILE and save some stats from the run. these are the same stats as get displayed in verbose mode but are more suitable for programmatic consumption. 📊

💜 report — Display the list of duplicates.

  • --cut PATHSEG — remove prefix PATHSEG from the file paths in the report output. this can reduce clutter in the output text if all the files scanned share a long identical prefix.
  • --minsize SIZE — report only duplicate sets which consume at least this much disk space. note this is the total size occupied by all the duplicates in a set, not their individual file size. 👀
  • --format NAME — produce the report in this output format. NAME is one of text, csv, json. the default is text.
📢 Note: the database format generated by scan is not guaranteed to be compatible with future versions. you should run report (and all the other commands below which access the database) using the same version of dupd that was used to generate the database. periodt.

🎀 file — Report duplicate status of one file.

to check whether one given file still has known duplicates, use the file operation. note that this does not do a new scan, so it will not find new duplicates. this checks whether the duplicates identified during the previous scan still exist and verifies (by hash) whether they are still duplicates. 🔍

  • --file PATH — Required: the file to check
  • --cut PATHSEG — remove prefix PATHSEG from the file paths in the report output.
  • --exclude PATH — ignore any duplicates under PATH when reporting duplicates. this is useful if you intend to delete the entire tree under PATH, to make sure you don't delete all copies of the file. 🫶
  • --hardlink-is-unique — ignore the existence of hard links to the file for the purpose of considering whether the file is unique.

🌈 ls, uniques, dups — List matching files.

while the file command checks the duplicate status of a single file, these commands do the same for all the files in a given directory tree. please note that files which are smaller than the minimum file size processed during the scan operation will not be shown. 💖

  • ls — list all files, show whether they have duplicates or not.
  • uniques — list all unique files.
  • dups — list all files which have known duplicates.

and their options:

  • --path PATH — start from this directory (default is current directory)
  • --cut PATHSEG — remove prefix $PATHSEG from the file paths in the output.
  • --exclude PATH — ignore any duplicates under PATH when reporting duplicates.
  • --hardlink-is-unique — ignore the existence of hard links to the file for the purpose of considering whether the file is unique.

⭐ refresh — Refreshing the database.

as you remove duplicate files, these are still listed in the dupd database. ideally you'd run the scan again to rebuild the database. note that re-running the scan after deleting some duplicates can be very fast because the files are in the cache, so that is the best option. ✨

however, when dealing with a set of files large enough that they don't fit in the cache, re-running the scan may take a long time. for those cases the refresh command offers a much faster alternative. 😭

the refresh command checks whether all the files in the dupd database still exist and removes those which do not.

be sure to consider the limitations of this approach. the refresh command does not re-verify whether all files listed as duplicates are still duplicates. it also, of course, does not detect any new duplicates which may have appeared since the last scan. 💀

in summary: if you have only been deleting duplicates since the previous scan, run the refresh command. it will prune all the deleted files from the database and will be much faster than a scan. however, if you have been adding and/or modifying files since the last scan, it is best to run a new scan. 👑

🧚 validate — Validating the database.

the validate operation is primarily for testing but is documented here as it may be useful if you want to reconfirm that all duplicates in the database are still truly duplicates. 🔍

in most cases you will be better off re-running the scan operation instead of using validate.

validate is fairly slow as it will fully hash every file in the database. 🐌

💀 rmsh — Create shell scrip to remove duplicate files.

as a policy, dupd never modifies the filesystem! 🛑

as a convenience for those times when it is desirable to automatically remove files, this operation can create a shell script to do so. the output is a shell script (to stdout) which you can run to delete your files (if you're feeling lucky). 🍀

Review the generated script carefully to see if it truly does what you want! 👀

automated deletion is generally not very useful because it takes human intervention to decide which of the duplicates is the best one to keep in each case. while the content is the same, one of them may have a better file name and/or location. 💅

optionally, the shell script can create either soft or hard links from each removed file to the copy being kept. the options are mutually exclusive.

  • --link — create symlinks for deleted files.
  • --hardlink — create hard links for deleted files.

🫶 Additional global options

  • -q — quiet, suppress all output. 🤫
  • -v — verbose mode. can be repeated multiple times for ever increasing verbosity. 📢
  • -V, --verbose-level N — set the logging verbosity level directly to N.
  • -h — show brief help summary.
  • --db PATH — override the default database file location.
  • -F, --hash NAME — specify an different hash function. this applies to any command which uses content hashing. NAME is one of: md5 sha1 sha512 xxhash 🔐

are hard links duplicates or not? the answer depends on "what do you mean by duplicates?" and "what are you trying to do?" 🤔

if your primary goal for removing duplicates is to save disk space, then it makes sense to ignore hardlinks. if, on the other hand, your primary goal is to reduce filesystem clutter, then it makes more sense to think of hardlinks as duplicates. 💡

by default, dupd considers hardlinks as duplicates. you can switch this around with the --hardlink-is-unique option. this option can be given either during scan or to the interactive reporting commands (file, ls, uniques, dups). 🎀

🫠 STALE DATABASE 🫠

okay bestie so the whole reason dupd is so flexible? it's because the duplicate sets (and, optionally, the unique files too) get saved into the database — and that's literally what makes all the exploratory, poke-around-and-see usage possible. iconic design choice, no notes. 💅

but. BUT. you have to know the limitations of this database, and we're gonna talk about them right now. 🫶

The database gets created during a dupd scan operation. And unless you're scanning a read-only filesystem, it is possible that files get modified, added, and deleted after that point in time — by stuff happening entirely outside of dupd's knowledge. Rude, honestly. 😭

Which means: over time, the database becomes increasingly stale compared to the actual content of the files. To discourage you from using a crusty old database, dupd prints a warning if the db is older than 3 days. But ngl that's just an arbitrary value — files may change instants after being scanned. Instants!!

So here's the vibe check on what to do:

  • If you are only deleting duplicate files from the data set, go look at the refresh command to update the database. ✨
  • But if files are being modified and/or added, then the move is to just re-run a dupd scan operation whenever the data set has changed sufficiently — and yeah, deciding what "sufficiently" means is a judgment call. your call, main character. 👑

🌸 EXAMPLES 🌸

Scan all the files in your home directory and then show the sets of duplicates that turned up:

% dupd scan --path $HOME

% dupd report

Show duplicate status (duplicate or unique) for every file in the docs subdirectory:

% dupd ls --path docs

Say I'm about to delete docs/old.doc but I want to check one last time that it's actually a duplicate, and I want to review where those duplicates live:

% dupd file --file docs/old.doc -v

And if you want more usage examples, read the documentation in the dupd 'docs' directory or the online documentation. 📚

🚪 EXIT 🚪

dupd exits with status code 0 on success, and non-zero on error. simple as that. 🎀

💜 SEE ALSO 💜

and that's the whole dupd(1) manual bestie, we made it to the end 🦋 go forth, scan responsibly, and remember: if your db is giving stale energy, refresh it or re-scan. slay 💖✨

this was the dupd(1) man page — footer and all, that's the source/version and date down there, told you I'd mention it 🫶

📦 where this page comes from