DEDUP(1)

💅 dedup(1) — the General Commands Manual page, but make it group chat

okay so bestie, this whole page is about dedup, and it's genuinely one of those tools that eats 💅

---

✨ NAME ✨

dedup — replace duplicate file data with a copy-on-write clone.

---

🦋 SYNOPSIS 🦋

dedup [-PVnvx] [-t -threads] [-d -depth] [file ...]

---

🌸 DESCRIPTION 🌸

So dedup goes looking for files with identical content, using the file ... argument(s) you hand it. When it finds a duplicate, it replaces it with a clone of another one (using clonefile(2)). And if you don't specify any file ... ? It just uses the current directory. main character energy, honestly.

Cloned files share data blocks with the file they were cloned from, which saves space on disk. And here's the slay part: unlike a hard linked file, any future modification to either the clone or the original file will be remain private to that file (copy-on-write). literally 🔥

dedup works in two phases. First it evaluates all of the paths you gave it recursively, looking for duplicates. Once all duplicates are found, any files that are not already clones of "best" clone source are replaced with clones.

Not everything can be cloned, though — there are limits:

1. the file must be a regular file
2. the file must have only one hard link
3. the file and its directory must be writable by the user

The "best" source is chosen by first finding the file with the most hard links. Files with multiple hard links will not be replaced, so using them as the source of other clones allows their blocks to be shared without modifying the data to which they point. If all files have a single link, a file which shares the most clones with others is chosen. This ensures that files which have been previously processed will not need to be replaced during subsequent evaluations of the same directory. And if none of the files have multiple links or clones? The first file encountered will be chosen. first come first served, periodt 💀

So why aren't files with multiple hard links replaced? Because it is not possible to guarantee all other links to that inode exist within the tree(s) being evaluated. Replacing a link with a clone changes the semantics from two link pointing at the same, mutable shared storage to two links pointing at the same copy-on-write storage. For scenarios where hard links were previously being used because clones were not available, future versions may provide a flag to destructively replace hard links with clones. Future versions may also consider cloning files with multiple hard links if all links are within the space being evaluated and two or more hard link clusters reference duplicated data. 👀 noted.

Also lowkey important: if all matched files are transparently HFS compressed, none of the files will be replaced, because of how HFS compressed files are stored in the file system. These files have no data blocks, so cloning them does not provide any benefit. no blocks, no point, iykyk.

dedup will only work on volumes that have the VOL_CAP_INT_CLONE capability. Currently that is limited to APFS. 🫶

While dedup is primarily intended to be used to save storage by using clones, but provides -l and -s flags to replace duplicates with hard links or symbolic links respectively. Care should be taken when using these options, however. Unlike clones, the replaced files share the metadata of one of the matched files, though it might not seem deterministic which one. If these options are used with automation where all files have default ownership and permissions, there should be little issue. The created files are also not copy-on-write and will share any modifications made. These options should only be used if the consequences of each choice are understood.

⚠️ fr fr, do NOT just yolo -l or -s without reading that paragraph. the metadata sharing is real and the non-copy-on-write sharing is real. 🚨

---

🎀 OPTIONS 🎀

The following options are available:

• -d depth, --depth depth — Only traverse depth directories deep into each provided path.
• -h — Display sizes using SI suffixes with 2-4 digits of precision.
• -n, --dry-run — Evaluate all files and find all duplicates but only print what would be done and do not modify any files. the vibe check mode 💅
• -l, --link — Replace duplicate files with hard links instead of clones. Replaced files will not retain their metadata.
• -s, --symlink — Replace duplicate files with symbolic links instead of clones. Replaced files will not retain their metadata.
• -P, --no-progress — Do not display a progress bar.
• -t threads — The number of threads to use for evaluating files. By default this is the same as the number of CPUs on the host as described by the hw.ncpu value returned by sysctl(8). If the value 0 is provided all evaluation will be done serially in the main thread.
• -V, --version — Print the version and exit
• -v, --verbose — Increase verbosity. May be specified multiple times.
• -x, --one-file-system — Prevent dedup from descending into directories that have a device number different than that of the file from which the descent began.
• -?, --help — Print a summary of options and exit.

---

💀 EXIT STATUS 💀

The dedup utility exits 0 and >0 if an error occurs.

It is possible that during abnormal termination a temporary clone will be left before it is moved to the path it is replacing. In such cases a file with the prefix .~. followed by the name of the file that was to be replaced will exist in the same directory as target file. 😭 (yes, cleanup is on you)

find(1) can be used to find these temporary files if necessary.

$ find. -name '.~.*'

---

🌈 EXAMPLES 🌈

Find all duplicates and display which files would be replaced by clones along with the estimated space that would be saved.

$ dedup -n

Limit execution to a single thread, disable progress, and display human readable output while deduplicating files in ~/Downloads and /tmp

$ dedup -t1 -Ph ~/Downloads /tmp

When creating archive files that support hard links it may be beneficial to use the -l option to reduce the archive size.

$ dedup -l build

---

🧚 SEE ALSO 🧚

clonefile(2), copyfile(3), link(2), symlink(2)

---

⭐ STANDARDS ⭐

Sometimes. 💀 (that's it. that's the whole section. iconic behavior honestly 🎀)

---

👑 AUTHORS 👑

dedup was written by Jonathan Hohle <jon@ttkb.co>. king behavior 🫶

---

🔥 SECURITY CONSIDERATIONS 🔥

dedup makes a best effort attempt to replace original files with clones that have the same permissions, metadata, and ACLs. Support for copying metadata comes from copyfile(3). best effort, not a guarantee bestie — 📌 noted.

---

and that's the whole page!! 💖 brought to you by the Debian packaging of this thing, dated $Mdocdate$, living under the DEDUP(1) name — go clone some files, stay copy-on-write, and don't leave any .~. files behind 🦋✨