SIMHASH(1)

💖 SIMHASH(1) 💖

okay bestie, welcome to the General Commands Manual page for simhash, and lemme tell you, this lil tool is literally the main character of file comparison 🎀

✨ NAME ✨

simhash — file similarity hash tool 🦋

✨ SYNOPSIS ✨

simhash [ -s nshingles ] [ -f nfeatures ] [ file ]
simhash [ -s nshingles ] [ -f nfeatures ] -w file ...
simhash [ -s nshingles ] [ -f nfeatures ] -m file ...
simhash -c hashfile hashfile

✨ DESCRIPTION ✨

so. this program computes and compares similarity hashes of files, and it's giving extremely clever tbh.

A similarity hash is a chunk of data with this one specific property: some distance metric between files is proportional to some distance metric between the hashes. Translation for the girlies 💅 — the hash is usually way way smaller than the file itself, but it still knows things about the file. obsessed.

The algorithm simhash uses is Manassas' "shingleprinting" algorithm (peep BIBLIOGRAPHY below, bestie, the receipts are all there 📚): you take a hash of every m-byte subsequence of the file, and then you keep the n of those hashes that are numerically smallest. That's it. That's the whole vibe. The size of the intersection of the hash sets of two files gives you a statistically good estimate of how similar the files are as a whole. no cap 🧚

Now, the modes. In its default mode, simhash computes the similarity hash of its file argument (or stdin, if you're feeling mysterious) and writes that hash to its standard output.

• When you invoke it with the -w argument (see below), simhash computes similarity hashes of all of its file arguments in "batch mode".
• When you invoke it with the -m argument (see below), simhash compares all the given files using similarity hashes in "match mode".
• And finally, when you invoke it with the -c argument (see below), simhash reports the degree of similarity between two hashes. 🫶

✨ OPTIONS ✨

• -f feature-count — When computing a similarity hash, retain at most feature-count significant hashes from the target file. The default is 128 features. Heads up: larger feature counts give you higher resolution in the differences between files, they grow the size of the similarity hash proportionally to the feature count, and they increase similarity hash computation time slightly. so basically more features = more detail, more bytes, tiny bit more waiting ⭐
• -s shingle-size — When computing a similarity hash, use hashes of samples consisting of shingle-size consecutive bytes drawn from the target file. The default is 8 bytes, and the minimum is 4 bytes. Bigger shingle sizes emphasize the differences between files more, and they slow the similarity hash computation down proportionally to the shingle size. it's giving tradeoff energy 😭
• -c hashfile1 hashfile2 — Display the distance (normalized to the range 0..1) between the similarity hash stored in hashfile1 and the similarity hash stored in hashfile2. cute lil number, zero to one, that's the vibe 🌸
• -w file ... — Write the similarity hash of each of the file arguments to file.sim. 🎀
• -m file ... — Compute the similarity hash of each of the file arguments, and output a similarity matrix for those files. 👑

✨ AUTHOR ✨

Bart Massey <bart@cs.pdx.edu> — king behavior, honestly 👑💜

✨ BUGS ✨

This currently uses CRC32 for the hashing. A Rabin Fingerprint should be offered as a slightly slower but more reliable alternative. 💀

The shingleprinting algorithm works for text files and fairly well for other sequential filetypes, but does not work well for image files. The latter both are 2D and often undergo odd transformations. 🙈

✨ BIBLIOGRAPHY ✨

• Mark Manasse, Microsoft Research Silicon Valley. Finding similar things quickly in large collections. http://research.microsoft.com/research/sv/PageTurner/similarity.htm
• Andrei Z. Broder. On the resemblance and containment of documents. In Compression and Complexity of Sequences (SEQUENCES'97), pages 21-29. IEEE Computer Society, 1998. ftp://ftp.digital.com/pub/DEC/SRC/publications/broder/positano-final-wpnums.pdf
• Andrei Z. Broder. Some applications of Rabin's fingerprinting method. Published in R. Capocelli, A. De Santis, U. Vaccaro eds., Sequences II: Methods in Communications, Security, and Computer Science, Springer-Verlag, 1993. http://athos.rutgers.edu/~muthu/broder.ps

that's the whole page bestie!! 🦋✨ this version is from 3 January 2007, it's SIMHASH(1), and yeah it's old but it ATE and left no crumbs. go hash something. periodt 💅💖🌈