DUPEMAP(1)

💅 NAME 💅

dupemap — bestie, this one creates a database of file checksums and then uses that database to yeet the duplicates out of existence ✨

---

🦋 SYNOPSIS 🦋

dupemap [ options ] [ -d database ] operation path...

---

🌸 DESCRIPTION 🌸

okay so here's the vibe: dupemap goes through each path recursively, hunting down the checksums of what's inside your files. it does NOT care about order — directories get searched through in no particular order, it's giving chaos but make it efficient 💅

what it actually does depends on two things:

• whether you passed the -d option, and
• the operation parameter, which has to be a comma-separated list made up of scan, report, delete.

🎀 Without -d

without -d, dupemap springs into action the moment it sees the same checksum show up more than once — i.e. it's just finding duplicates recursively, no database, no drama. the action depends on operation:

• report — report which files got encountered more than once, printing their names to standard output. so you can look before you leap, queen 👑
• delete[,report] — delete the files that got encountered more than once. it prints their names if report is also given.

⚠️ WARNING: run the report operation FIRST to see what's about to get deleted. do not skip this, i am begging 😭

⚠️ WARNING: you are advised to make a backup of the target first, e.g. with "cp -al" (for GNU cp) to create hard links recursively. protect your files bestie!!

💖 With -d

with -d, the database argument to -d denotes a database file (peep the "DATABASE" section in this manual for the tea ☕) to read from or write to. in this mode, the scan operation should be run on one path, followed by the report or delete operation on another (not the same!) path.

• scan — add the checksum of each file to database. this operation must be run initially to create the database. to start over, you have to manually delete the database file(s) (again, see the "DATABASE" section).
• report — print each file name if its checksum is found in database.
• delete[,report] — delete each file if its checksum is found in database. if report is also present, print the name of each deleted file.

⚠️ WARNING: if you run dupemap delete on the same path you just ran dupemap scan on, it will delete EVERY file! the whole idea of these options is to scan one path and delete files in a second path. do not do the thing 🫠

⚠️ WARNING: use the report operation first to see what will be deleted.

⚠️ WARNING: you are advised to make a backup of the target first, e.g. with "cp -al" (for GNU cp) to create hard links recursively.

---

✨ OPTIONS ✨

• -d database — use database as an on-disk database to read from or write to. check the "DESCRIPTION" section above for how this changes the way dupemap behaves.
• -I file — reads input files from file on top of whatever you listed on the command line. if file is "-", read from standard input. each line gets interpreted as a file name. 🫶 heads up: the paths given here will NOT be scanned recursively. directories get ignored and symlinks get followed.
• -m minsize — ignore files below this size.
• -M maxsize — ignore files above this size.

---

🌈 USAGE 🌈

🍓 General usage

the easiest operations to understand are the ones where -d is NOT given. to delete all duplicate files in /tmp/recovered-files, you do:

$ dupemap delete /tmp/recovered-files

often, dupemap scan gets run to produce a checksum database of all files in a directory tree. then dupemap delete gets run on another directory, possibly following a dupemap report. for example, to delete all files in /tmp/recovered-files that already exist in $HOME, do this:

$ dupemap -d homedir.map scan $HOME
$ dupemap -d homedir.map delete,report /tmp/recovered-files

🧚 Usage with magicrescue

the main application for dupemap is taking some of the pain out of performing undelete operations with magicrescue(1). the reason is that magicrescue will extract every single file of the specified type on the block device, so undeleting files means you have to find a few files out of hundreds, which can take a long time if you do it manually. what we want is to only extract the documents that don't exist on the file system already. iykyk 💅

in the following scenario, you have accidentally deleted some important Word documents in Windows. if this were a real-world scenario, then by all means use The Sleuth Kit. however, magicrescue will work even when the directory entries were overwritten, i.e. more files were stored in the same folder later. iconic behaviour honestly ✨

you boot into Linux and change to a directory with lots of space. mount the Windows partition, preferably read-only (especially with NTFS), and create the directories we'll use.

$ mount -o ro /dev/hda1 /mnt/windows
$ mkdir healthy_docs rescued_docs

extract all the healthy Word documents with magicrescue and build a database of their checksums. it may seem a little redundant to send all the documents through magicrescue first, but the reason is that this process may modify them (e.g. stripping trailing garbage), and therefore their checksum will not be the same as the original documents. also, it will find documents embedded inside other files, such as uncompressed zip archives or files with the wrong extension. sneaky!! 🕵️

$ find /mnt/windows -type f \
  |magicrescue -I- -r msoffice -d healthy_docs
$ dupemap -d healthy_docs.map scan healthy_docs
$ rm -rf healthy_docs

now rescue all "msoffice" documents from the block device and get rid of everything that's not a *.doc.

$ magicrescue -Mo -r msoffice -d rescued_docs /dev/hda1 \
  |grep -v '\.doc$'|xargs rm -f

remove all the rescued documents that also appear on the file system, and remove duplicates.

$ dupemap -d healthy_docs.map delete,report rescued_docs
$ dupemap delete,report rescued_docs

the rescued_docs folder should now contain only a few files. this will be the undeleted files and some documents that were not stored in contiguous blocks (use that defragger ;-)) 😭

💜 Usage with fsck

in this scenario (based on a true story), you have a hard disk that's gone bad. you have managed to dd about 80% of the contents into the file diskimage, and you have an old backup from a few months ago. the disk is using reiserfs on Linux.

first, use fsck to make the file system usable again. it will find many nameless files and put them in lost+found. you need to make sure there is some free space on the disk image, so fsck has something to work with.

$ cp diskimage diskimage.bak
$ dd if=/dev/zero bs=1M count=2048 >> diskimage
$ reiserfsck --rebuild-tree diskimage
$ mount -o loop diskimage /mnt
$ ls /mnt/lost+found
(tons of files)

our strategy will be to restore the system with the old backup as a base and merge the two other sets of files (/mnt/lost+found and /mnt) into the backup after eliminating duplicates. therefore we create a checksum database of the directory we have unpacked the backup in.

$ dupemap -d backup.map scan ~/backup

next, we eliminate all the files from the rescued image that are also present in the backup.

$ dupemap -d backup.map delete,report /mnt

we also want to remove duplicates from lost+found, and we want to get rid of any files that are also present in the other directories in /mnt.

$ dupemap delete,report /mnt/lost+found
$ ls /mnt|grep -v lost+found|xargs dupemap -d mnt.map scan
$ dupemap -d mnt.map delete,report /mnt/lost+found

this should leave only the files in /mnt that have changed since the last backup or got corrupted. particularly, the contents of /mnt/lost+found should now be reduced enough to manually sort through them (or perhaps use magicsort(1)). we love a happy ending 🥹

👑 Primitive intrusion detection

you can use dupemap to see what files change on your system. this is one of the more exotic uses, and it's only included for inspiration. main character energy only 💅

first, you map the whole file system.

$ dupemap -d old.map scan /

then you come back a few days/weeks later and run dupemap report. this will give you a view of what has NOT changed. to see what HAS changed, you need a list of the whole file system. you can get this list along with preparing a new map easily. both lists need to be sorted to be compared.

$ dupemap -d old.map report /|sort > unchanged_files
$ dupemap -d current.map scan /|sort > current_files

all that's left to do is comparing these files and preparing for next week. this assumes that the dbm appends the ".db" extension to database files.

$ diff unchanged_files current_files > changed_files
$ mv current.map.db old.map.db

---

🎀 DATABASE 🎀

the actual database file(s) written by dupecheck will have some relation to the database argument, but most implementations append an extension. for example, Berkeley DB names the files database.db, while Solaris and GDBM creates both a database.dir and database.pag file.

dupecheck depends on a database library for storing the checksums. it currently requires the POSIX-standardized ndbm library, which must be present on XSI-compliant UNIXes. implementations are not required to handle hash key collisions, and a failure to do that could make dupecheck delete too many files 😭. i haven't heard of such an implementation, though. phew.

the current checksum algorithm is the file's CRC32 combined with its size. both values are stored in native byte order, and because of varying type sizes the database is not portable across architectures, compilers and operating systems. so don't go moving that db around like it's a travel vlog, bestie 🧳

---

💖 SEE ALSO 💖

magicrescue(1), weeder(1)

this tool does the same thing weeder does, except that weeder cannot seem to handle many files without crashing, and it has no largefile support. no shade but also... shade 💀

---

💀 BUGS 💀

there is a tiny chance that two different files can have the same checksum and size. the probability of this happening is around 1 to 10^14, and since dupemap is part of the Magic Rescue package, which deals with disaster recovery, that chance becomes an insignificant part of the game. you should consider this if you apply dupemap to other applications, especially if they are security-related (see next paragraph). ngl, read the next paragraph.

it is possible to craft a file to have a known CRC32. you need to keep this in mind if you use dupemap on untrusted data. a solution to this could be to implement an option for using MD5 checksums instead.

---

🌸 AUTHOR 🌸

Jonas Jensen <jbj@knef.dk>

---

⭐ LATEST VERSION ⭐

this tool is part of Magic Rescue. you can find the latest version at <https://github.com/jbj/magicrescue>

---

and that's the whole page, bestie!! this was version 1.1.10, dated 2018-10-16, brought to you by the Magic Rescue girlies 🫶 go forth and deduplicate responsibly, and remember: report before delete, always 💅✨