DCMP(1)

📦 version 0.12, in nixpkgs unstable

💅 dcmp — distributed compare ✨

okay so bestie, welcome to dcmp(1) from mpiFileUtils — yes that's the manual name, i'm folding it in, you're welcome 🎀

🌈 SYNOPSIS

dcmp [OPTION] SRC DEST

🦋 DESCRIPTION

It's a parallel MPI application, bestie. Like, actually parallel. It compares two files, or it recursively compares files that have the same relative paths inside two different directories. fr fr.

dcmp gives you functionality similar to a recursive cmp(1) — it reports how many files in two different directories are the same and how many are different. Lowkey iconic.

And the fun part? dcmp can be configured to compare a whole bunch of different file properties. Not just contents. We're talking properties. Obsessed.

✨ OPTIONS ✨

  • -o, --output EXPR:FILE — writes the list of files matching expression EXPR to the specified FILE. The expression is a set of fields and states (more on those below, keep scrolling 💖). You can pass more than one -o option in a single invocation — just make sure each one has a different output file name, okay? No fighting over filenames.
  • -t, --text — changes --output so it writes files in text format instead of binary. Human-readable, slay.
  • -b, --base — enables base checks and normal stdout results even when --output is being used. So you get both, bestie.
  • --bufsize SIZE — sets the I/O buffer to SIZE bytes. Units like "MB" and "GB" may go immediately after the number with no space (e.g. 8MB). The default bufsize is 4MB.
  • --chunksize SIZE — multiple processes copy a large file in parallel by dividing it into chunks, and this sets the chunk to be at minimum SIZE bytes. Units like "MB" and "GB" can immediately follow the number without spaces (e.g. 64MB). The default chunksize is 4MB.
  • --daos-api API — specifies the DAOS API to be used. By default the API is automatically determined based on the container type, where POSIX containers use the DFS API and all other containers use the DAOS object API. Values must be in {DFS, DAOS}. No freestyling 🫶
  • -s, --direct — uses O_DIRECT to avoid caching file data. Skips the middleman, honestly.
  • --open-noatime — opens files with the O_NOATIME flag.
  • --progress N — prints a progress message to stdout approximately every N seconds. The number of seconds must be a non-negative integer. A value of 0 disables progress messages. (so --progress 0 = peace and quiet 💀)
  • -v, --verbose — runs in verbose mode, printing a list of statistics/timing data for the command: files walked, started, completed, seconds, files, bytes read, byte rate, and file rate. Numbers girlies rise 🙈
  • -q, --quiet — runs the tool silently. No output is printed. Not a single peep.
  • -l, --lite — lite mode does a comparison of file modification time and size. If modification time and size are the same, then the contents are assumed to be the same. Similarly, if the modification time or size is different, then the contents are assumed to be different. The lite mode does no comparison of data/content in the file. So like, it's trusting. Maybe too trusting. iykyk.
  • -h, --help — prints the command usage and the list of options available. The classic.

🧚 EXPRESSIONS

An expression is made up of one or more conditions, where each condition specifies a field and a state. A single condition is: a field name, an = sign, and a state name. That's the whole vibe check.

💜 Valid fields

Here's the field list, plus the property of the entry each one checks:

  • EXIST — whether the entry exists
  • TYPE — type of entry, e.g., regular file, directory, symlink
  • SIZE — size of entry in bytes, if a regular file
  • UID — user id of entry
  • GID — group id of entry
  • ATIME — time of last access
  • MTIME — time of last modification
  • CTIME — time of last status change
  • PERM — permission bits of entry
  • ACL — ACLs associated with entry, if any
  • CONTENT — file contents of entry, byte-for-byte comparision, if a regular file

💖 Valid conditions for the EXIST field

  • EXIST=ONLY_SRC — entry exists only in source path
  • EXIST=ONLY_DEST — entry exists only in destination path
  • EXIST=DIFFER — entry exists in either source or destination, but not both
  • EXIST=COMMON — entry exists in both source and destination

All other fields may only specify the DIFFER and COMMON states. That's the rule, periodt.

🌸 Combining conditions

Conditions can be joined together with AND (@) and OR (,) operators, with no spaces, to build complex expressions. For example, this expression reports entries that exist in both source and destination paths, but are of different types:

EXIST=COMMON@TYPE=DIFFER

The AND operator binds with higher precedence than the OR operator. For example, the following expression matches on entries which either (exist in both source and destination and whose types differ) or (only exist in the source):

EXIST=COMMON@TYPE=DIFFER,EXIST=ONLY_SRC

⭐ Conditions that imply other conditions

Some conditions imply others. For example, for CONTENT to be considered the same, the entry must exist in both source and destination, the types must match, the sizes must match, and finally the contents must match:

SIZE=COMMON    => EXISTS=COMMON@TYPE=COMMON@SIZE=COMMON
CONTENT=COMMON => EXISTS=COMMON@TYPE=COMMON@SIZE=COMMON@CONTENT=COMMON

A successful check on any other field also implies that EXIST=COMMON.

🎀 Expressions with -o

When used with the -o option, you also have to specify a file name at the end of the expression, separated with a :. The list of any entries that match the expression gets written to the named file. For example, to list any entries matching the above expression to a file named outfile1, you'd use this option:

-o EXIST=COMMON@TYPE=DIFFER:outfile1

👑 The default checks

If the --base option is given, or when no output option is specified, the following expressions are checked and numeric results are reported to stdout:

EXIST=COMMON
EXIST=DIFFER
EXIST=COMMON@TYPE=COMMON
EXIST=COMMON@TYPE=DIFFER
EXIST=COMMON@CONTENT=COMMON
EXIST=COMMON@CONTENT=DIFFER

🔥 EXAMPLES

  1. Compare two files in different directories:
mpirun -np 128 dcmp /src1/file1 /src2/file2
  1. Compare two directories with verbose output. The verbose output prints timing and number of bytes read:
mpirun -np 128 dcmp -v /src1 /src2
  1. Write a list of entries to outfile1 that are only in src1, or whose names exist in both src1 and src2 but whose types differ:
mpirun -np 128 dcmp -o EXIST=COMMON@TYPE=DIFFER,EXIST=ONLY_SRC:outfile1
/src1 /src2
  1. Same as above, but also write a list of entries to outfile2 that exist in either src1 or src2 but not both:
mpirun -np 128 dcmp -o EXIST=COMMON@TYPE=DIFFER,EXIST=ONLY_SRC:outfile1
-o EXIST=DIFFER:outfile2 /src1 /src2

🫶 SEE ALSO

The mpiFileUtils source code and all documentation may be downloaded from <https://github.com/hpc/mpifileutils>

💅 AUTHOR

HPC

2022, LLNL/LANL/UT-Battelle/DDN

okay that's the whole page bestie, go compare some files 💖✨ this was version 0.12, dated Jan 29, 2025 — and that's the DCMP(1) manual page, mwah 🎀🦋

📦 where this page comes from