photodedupe(1)

📦 version 1.0.6, in ubuntu stonking

💖 photodedupe(1) — the girlies' guide to finding ur duplicate photos 💖

General Commands Manual · part 1 of 2, bestie, don't go anywhere

✨ NAME ✨

photodedupe — a command line utility for identifying duplicate photos 🫶

🌸 SYNOPSIS 🌸

photodedupe [-d|-u|-a] [DIR]

🦋 DESCRIPTION 🦋

Photodedupe is a command line utility for identifying duplicate photos, and the way it does it is honestly so iconic: it does not care whether ur images have been scaled or are in different file formats. It compares the image content visually and does not rely on metadata or file hashes to do the de-duplication. Like, it actually looks at ur pics. Main character energy fr fr. 👑

Where duplicates are identified, the images get sorted by resolution so the largest version of each image shows up first. And then u can either list only the highest resolution versions of all images, or list only the duplicates (the lower resolution versions). That means u can pull the best copies out of a photo collection, or yeet the duplicates. This sort order is lowkey perfect for stuff like removing thumbnails from image collections, or separating scaled web resolution images from the original high resolution version. 💅

The output is a list of file paths, which u can pipe to other commands so the images u want get sorted into folders (there's example usage below 👇). And no cap, photodedupe can handle large image collections — hundreds of thousands or millions of images — and it scales to any number of CPU cores. She's built different. 🔥

🎀 OPTIONS 🎀

  • -d, --duplicates — List only the detected duplicate images. Excludes the highest resolution version of each image. Also excludes unique images.
  • -u, --uniques — List only the best (highest resolution) version of each valid image, and don't list any duplicates at all. Only the queens survive. 👑
  • -a, --all — By default photodedupe lists only images that have duplicates. This option makes it list all valid image files (except those below the minimum resolution if --min-resolution is used), regardless of whether the file has a duplicate or not.
  • -c, --compare <directory of new images> — Compares a directory of new images (that's the parameter u hand to --compare) against one or more directories that make up an existing image collection (those are the arguments u pass). It tests whether each of the new images are duplicates of the existing image collection, or unique — depending on whether u used --duplicates or --uniques. When it's used with --duplicates, new images are classified as unique when they're of higher resolution than the version in the existing image collection. If u want similar images marked as duplicates in all circumstances (irrespective of resolution), also apply the --ignore-resolution option.
  • --ignore-resolution — When using --compare, always mark duplicates even when the new image is better quality. Do not mark as unique even if it's better quality.
  • --min-resolution <WidthxHeight> — Ignore all images of less than the specified resolution. Like, --min-resolution 300x200 will ignore images if either the width is less than 300 pixels or the height is less than 200 pixels. Either one fails, she's out. 🙈
  • -y, --any-file — Tests every file to see if it might be an image, regardless of file extension. Also allows image files with no extension at all. The default behaviour is to only test files with common image filename extensions, which are jpg,jpeg,png,tif,tiff,gif and webp.
  • --force-colour-diff-only — Only use the colour difference algorithm. This one is more accurate but does not perform well with large numbers of images. It's the default algorithm when there are 50,000 or fewer images. Beyond that number of images, a different perceptual hash algorithm gets used instead — slightly less accurate but much faster. 🏃‍♀️💨
  • -t, --threads <number of threads> — Number of CPU threads to use (default is 4). A higher number improves performance if u have more than 4 CPU threads available. More cores, more slay. ✨
  • --colour-diff-threshold <threshold> — Colour difference threshold. Higher value means more likely to consider images duplicates (Min:0,Max:49000,Default:256).
  • -g, --debug — Expects either one or two image file arguments. Where one file is supplied, it prints statistics about the file. Where two are supplied, it prints statistics and information about the differences found between the files. Detective mode bestie 🔍
  • -h, --help — Print help.
  • -V, --version — Print version.

💖 EXAMPLE USAGE 💖

okay bestie, here's how you actually run this thing ✨

You can hand it one or more directories straight up on the command line, and photodedupe will go rummaging recursively through every single one of them looking for images:

photodedupe dir_of_images_1/ dir_of_images_2/ dir_of_images_3/

Or you can just pass it a list of specific image files as arguments:

photodedupe image1.jpg image2.jpg image3.jpg

Or pipe a list of file paths in 🫶. This method is chef's kiss for selecting only images of a certain format by file extension:

find dir_of_photos/ -name '*.jpg' | photodedupe

By default photodedupe only inspects files with common image filename extensions. JPEG, PNG, TIFF, GIF and WebP images are supported, periodt. That extension check applies when you pipe lists of files in on stdin too. BUT — if your images are out here living their truth with inappropriate (or zero) filename extensions, the image file formats can also be auto-detected 🔍. To inspect every file regardless of extension (or lack of extension) and work out whether each one might be an image, use the --any-file option. 💅

The default output only lists images that actually have duplicates. The highest resolution version gets listed first as the “best” copy, followed by every lower resolution version listed as a duplicate. And if there are no duplicates? There's just no output on stdout. Silence. Nothing. 🦋

Best(512x341): unit_test_images/cat2_best.jpg
    Duplicate(510x340): unit_test_images/cat2_duplicate_1.jpg
    Duplicate(100x67): unit_test_images/cat2_duplicate_2.png

To list every image file found regardless of whether it has a duplicate, use the --all option.

To list only the highest resolution version of each image, use the --uniques option ✨. The output will include images that don't have any duplicates at all. This option could be used to copy the highest resolution version of each image to a different directory — main character energy honestly.

In this example photodedupe is outputting a list of filenames of the best versions of each image on stdout. The xargs command reads each filename, substitutes it for the two braces, and executes the provided command. This then causes each listed file to be copied to the specified directory 📁:

photodedupe --uniques dir_of_photos/ | xargs -i cp "{}"
unique_best_versions_dir/

To list only the lower resolution duplicate images, use the --duplicates option. This one could be used to remove duplicates from a directory, e.g:

photodedupe --duplicates dir_of_photos/ | xargs -i mv "{}"
duplicate_photos_dir/
⚠️ Not me warning you: photodedupe is performing a fuzzy match and is not 100% accurate. It is not advised to delete duplicates without manual inspection. Like, please look before you delete things, bestie. 😭

Photos below a user specified resolution can be ignored. In the following example photos will not be inspected if they have either a width of less than 150 pixels or a height of less than 100 pixels. This option is genuinely so helpful, e.g. when a web page of high resolution photos has been downloaded but the directory also includes a whole variety of other images that were on the page and that you don't need — navigation button images etc. The button images will tend to be lower resolution than the required photos, so this option immediately filters them out 🧚:

photodedupe dir_of_photos/ --min-resolution 150x100

🦋 IMAGE DIRECTORY DIFF 🦋

This feature enables a directory of new images to be compared against a pre-existing collection of photos to determine if any of the new images already appear in the collection. It's giving "have I already downloaded this?" 💀

This can be used to update a photo collection with new unique images derived from a new source. An example application might be for use with a web scraper that periodically downloads all the images from a regularly updated web page. This option can be used to determine if any of the most recently downloaded images are new or if they were downloaded on a previous occasion.

To identify which of the new images already exist in your collection, pass the directory of new images to be tested to the --compare option, then supply one or more paths to the pre-existing photo collection as arguments.

Supply the --uniques flag to show which images in new_images_dir are unique with respect to the existing collection ✨. You might then move the new images found into your collection.

photodedupe --uniques --compare new_images_dir/
collection_of_existing_images/

Supply the --duplicates flag to show which images in the new_images_dir are duplicates of the existing collection. If photos in the new images directory already exist in the collection at the same or lower resolution, the new images will be reported as duplicates. If however a photo appears in the new images directory at a higher resolution than present in the collection, it will not be reported as a duplicate 👀. This is to enable better quality versions of existing images to be discovered and added to the collection. This means you may be adding duplicates to the collection, but the duplicates added will be better quality versions of images than the versions you already have. If you do not want this behaviour, and you'd like duplicates to be reported in all circumstances, even if the new image is higher resolution, additionally supply the --ignore-resolution flag. 💅

photodedupe --duplicates --compare new_images_dir/
collection_of_existing_images/

The new images directory passed to --compare is inspected recursively and sub-directories will also be inspected for images. The same rules apply as for other directories. By default only files with common image file name extensions (such as .jpg) will be inspected, unless the --any-file option is used, in which case all files will be inspected.

It is also possible to specify a single specific image file to be compared against the entire existing photo collection. This can be used to effectively perform a database query, asking the question of whether you already have this specific image in your collection or not 🔍:

photodedupe --duplicates --compare image.jpg
collection_of_existing_images/

When using --compare, any duplicates present in the existing collection are not reported. Only duplicates present in the new images directory are reported. Reporting images found in the collection is just the default behaviour, in which case simply do not use the --compare option at all and pass the new images directory as a regular argument together with the existing photo collection.

To determine where specifically the images in the new images directory appear in the existing photo collection, use the --compare option alone (without either the --duplicates or --uniques flag). This will show only instances where there is a duplicate of an image in the new images directory. Only the highest resolution version will be shown from the collection (unless the version in the new images directory is the best version — icon behaviour tbh 👑).

💜 EXAMPLE BEHAVIOUR OF --compare 💜

In the example below the directory “new_images” is passed to the --compare option and the --duplicates flag is used. The collection directory is passed as an argument. All of the images in the below table are visually duplicates of each other 😭:

File                          Resolution   Shown As Duplicate?
───────────────────────────────────────────────────────────────
new_images/image_dupe_1.jpg   5 MP         No
collection/image_dupe_2.jpg   4 MP         No
new_images/image_dupe_3.jpg   3 MP         Yes
collection/image_dupe_4.jpg   2 MP         No
collection/image_dupe_5.jpg   1 MP         No

Image_dupe_1 is not shown as a duplicate because at 5 megapixels it exceeds the resolution of the best copy in the existing collection (Image_dupe_2), which is only 4 MP. Image_dupe_3 is displayed as a duplicate because at 3MP it is below the resolution of the best copy in the existing collection. Image_dupe_4 and Image_dupe_5 are not shown because they are in the existing collection and not in the new images directory. slay, logic understood 💅

🔥 PERFORMANCE 🔥

Photodedupe uses four threads by default to process images. The number of threads can be increased using the --threads option. Heads up though: more than the specified number of threads may actually be used, due to further multithreading within the underlying libraries. She's got layers. 💅

Up to 50,000 images, all photos found are compared to all others. However, after this number of images, the performance of this approach starts to become intractable 😭. Photodedupe will then switch to a different algorithm that is less capable of detecting duplicates, but can handle much larger numbers of images. A warning will be printed to stderr to explain when this occurs. It is possible to force use of the all to all comparison variation using the --force-colour-diff-only flag. However, this is not advised for large image sets, as the performance will decline significantly. Like, don't. 🙈

Photodedupe is not as accurate on vector art, or on images containing little variance such as very dark photos. Images are tested for variance, and where variance is below the threshold where de-duplication is likely to be reliable, the images are identified as unique — to prevent false positives ✨.

Photodedupe does not detect transformations of images as duplicates. If the image has been significantly rotated or cropped, it will be identified as unique. Soz! 🦋

The internal threshold at which a duplicate is detected can be tuned using the --colour-diff-threshold option, which accepts an integer between 0 and 49000. The default threshold is 256. Setting this value closer to zero will cause fewer duplicates to be found. At values close to 49000, virtually all images will be declared duplicates. Choose your fighter. 🎀

🌸 AUTHOR 🌸

Leon Bubb

🎀 SOURCE 🎀

https://github.com/InexplicableMagic/photodedupe

✨ LICENSE ✨

MIT

and that's the whole page, bestie 💖 this was photodedupe(1), Version 1.0.2 — go forth and de-duplicate responsibly, ily, no cap 🫶👑

📦 where this page comes from