SAMEFILE(1)

💖 SAMEFILE(1) 💖

okay bestie, welcome to the JS manual section, we're doing the samefile thing today and honestly? it's giving legendary. buckle up 🎀

---

✨ NAME ✨

samefile — find identical files 🔍💅

---

🌸 SYNOPSIS 🌸

samefile [-g size] [-l | -r] [-s sep] [-0aiqVvx]

---

🦋 DESCRIPTION 🦋

okay so picture this: samefile reads a list of filenames from stdin, one filename per line. slay. then for every filename pair that has literally identical contents, it spits out a line with six fields:

• the size in bytes 💖
• two filenames 📁📁
• the character = if the two files are on the same device, or X if they're not 🫶
• the link counts of the two files 🔗🔗

and the output? sorted in reverse order by size</> as the primary key, and the filenames as the secondary key. no cap. ✨

---

🎀 OPTIONS 🎀

• -0 — this one says the input list of file names is NUL terminated, like what you'd get from implementations of find(1) that support the -print0 option. without it, the file names are assumed to be newline terminated. bestie, it's all about the vibes of your terminator 💀
• -a — do not sort files with the same size alphabetically. we're going feral.
• -g size — only compare files with size greater than size bytes. default is 0. (yes, 0. iconic.)
• -i — allow files with the same device/i-node pair to be added to the binary tree. this might be useful if the output is gonna be fed into some other program. but heads up bestie: if you use this option, the statistics shown with -v will not contain the ``You have a total of x bytes in identical files'' line, because -i prohibits proper calculation of this value. sorry not sorry 😭
• -l — do not check if files with identical contents are hard links created by ln(1). by default samefile checks whether identical-content files are hard linked and, if they are, doesn't write a name pair to stdout. you get a slight speedup with this option. ⚠️ this option is incompatible with the -r option.
• -q — do not issue warning messages when open(2) fails. when you hit one of those warnings, open probably failed due to a 'permission denied' error on files or directories you don't have read permission for. useful if you're not root and wanna compare your files against files in a system directory like /etc. 🫶
• -r — report whether identical files are hard linked. the separator string followed by the [bracketed] link count gets appended to each name pair if they're hard links created with ln. ⚠️ incompatible with the -l option. also note: this kind of output has only four fields and will appear unsorted before the actual output of samefile. 💅
• -s sep — use the string sep as the output field separator, defaults to a tab character. useful if filenames contain tab characters and the output must be processed by another program, say awk(1).
• -V — print the version information and exit. short and sweet, we love a queen who knows when to leave 👑
• -v — verbose mode. writes some statistical messages about memory usage and work reduction, plus the sum of the sizes of all identical files, to stderr. ✨
• -x — switch off intelligence. yep, literally. this option prevents samefile from being smart. if file1, file2 and file3 are identical, it'll do 3 comparisons instead of just the two needed, and write more output. see the discussion under INTERNALS for why this could be useful. 🧠❌ and if you use this, the stats shown with -v won't contain the ``You have a total of x bytes in identical files'' line, because -x prohibits proper calculation of this value.

---

🌈 INTERNALS 🌈

samefile uses two stages to give optimum performance. she's efficient, she's organized, she's that girl.

Stage One 💖

in the first stage, all non-plain files are skipped (directories, devices, FIFOs, sockets, symbolic links), along with files where stat(2) fails and files that have a size less than or equal to size.

the output of the first stage (the filenames) gets written into a binary tree</> with one node for every file size. it's also at this early stage where checks for hard links happen. if hard links are found and -r is requested, the name pairs are output immediately. the whole list of hard linked name pairs will therefore appear before any output of the second stage. periodt.

for any i-node, only one filename will be added to the binary tree (unless -i was requested). 🤫

Stage Two 🦋

in the second stage, all files having the same size get compared against each other. the rules of mathematical logic are applied to reduce work and output noise (unless -x is requested): if files a, b, and c have the same size and samefile finds that a = b and a = c, then it will not compare b against c (and will not output a line for b and c) but only for a = b and a = c.

note however, that because only the first filename per i-node gets into the second stage, the output for a group of identical files with different i-node numbers is also minimized. suppose you have six identical files of size 100 in an i-node group consisting of the three i-nodes with numbers 10, 20 and 30 (the term 'i-node group' has nothing to do with the i-node group notion of some file systems — it merely refers to a set of i-nodes addressing files with identical contents):

$ ls -i
   10 file1     20 file4     30 file6
   10 file2     20 file5
   10 file3
$ ls | samefile
100     file1   file4   =       3       2
100     file1   file6   =       3       1

the sum of the sizes in the first column is the amount of disk space you could gain by making all 6 files links to only one file, or removing all but one of the files. to be precise, disk space is allocated in blocks — you'll probably gain two blocks here, rather than 200 bytes. 📦

note that it is not enough to just remove file4 and file6 (you'd gain only 100 bytes because file5 still exists). the proper way is to use the -i option. the output will look like:

100     file1   file2   =       3       3
100     file1   file3   =       3       3
100     file1   file4   =       3       2
100     file1   file5   =       3       2
100     file1   file6   =       3       1

removing all files listed in the third field will leave only file1. making all files hard links to file1 is easy. if the fourth field is a =, do a forced hard link. 🔗

if you need to know about all combinations of identical files, then you use both the -i and -x option. this produces:

$ ls | samefile -ix
100     file1   file2   =       3       3
100     file1   file3   =       3       3
100     file1   file4   =       3       2
100     file1   file5   =       3       2
100     file1   file6   =       3       1
100     file2   file3   =       3       3
100     file2   file4   =       3       2
100     file2   file5   =       3       2
100     file2   file6   =       3       1
100     file3   file4   =       3       2
100     file3   file5   =       3       2
100     file3   file6   =       3       1
100     file4   file5   =       2       2
100     file4   file6   =       2       1
100     file5   file6   =       2       1

---

🌸 EXAMPLES 🌸

find all identical files in the current working directory:

$ ls | samefile

find all identical files in my HOME directory and subdirectories, and also tell me if there are hard links:

$ find $HOME -type f | samefile -r

find all identical files in the /usr directory tree that are bigger than 10000 bytes, and write the result to usr.dups (that one is for the sysadmin folks, you may want to 'amp' — put it in the background with the ampersand & — this command because it takes a few minutes.) ⏳

$ find /usr -type f | samefile -g 10000 >usr.dups

---

💀 DIAGNOSTICS 💀

you'll see a short usage message if you use an invalid option. classic.

• malloc - free = xxxx — i didn't free the memory I've malloc(3)ed. you found a bug. please report it to the author. 😭
• Allocation failed for 'expr' ... — oops! you ran out of virtual memory. you must have a real big filename list. try to use a smaller one or increase resources available to your processes. for more information see ulimit(1) or your similar shell builtin. 💅

---

👑 SEE ALSO 👑

ln(1), find(1), rm(1), df(1)

---

🙈 BUGS 🙈

there are no known bugs. the source has been lint(1)ed and all possible care has been taken while coding. she's flawless, she's immaculate. ✨

if you find a bug (or miss a feature) please contact the author. 🫶

---

🎀 HOME 🎀

the official samefile home page www.schweikhardt.net/samefile/ is maintained by the author Jens Schweikhardt — schweikh at schweikhardt dot net 💌

---

and that's the whole page bestie!! this one's from the JS manual, dated 7 AUGUST 2005, and honestly it ate. go forth and dedupe. 💖✨🦋🎀