MAILDIRSYNC(1)

📦
version 1.2 · 2 versions
, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble

👑 MAILDIRSYNC 👑

okay bestie, buckle up, because this is the online synchronizer for Maildir-format mailboxes and it's genuinely iconic. this is the manual page for maildirsync (1), and we're gonna get through it together. 💅

✨ SYNOPSIS ✨

maildirsync.pl [ --recursive ] [ --backup path ] [ --backup-tree ]     \
    [ --bzip2=bzip2 ] [ --gzip=gzip ] [ --maildirsync=maildirsync ] \
    [ --rsh=ssh ] [ --verbose ] [ --alg=md5 ] [ --delete-before ]      \
    [ --exclude=^/Folder1 ] [ --exclude=^/Fold.*er2 ]                  \
    [ --exclude-source=^/Folder3 ] [ --exclude-target=^/Folder4 ]      \
    [ --rename="s/SourceFolder/TargetFolder/" ]                        \
    [ --destination=win|lin ]                                          \
    [ -r ] [ -b path ] [ -B ] [ -R ssh ] [ -v ] [ -a ] [ -d ]          \
    source dest state_file.bz2

a simple two-way synchronization, look how cute it is:

maildirsync.pl -rvv -a md5 desktop:Maildir Maildir lib/sync_desk_note.bz2
maildirsync.pl -rvv -a md5 Maildir desktop:Maildir lib/sync_note_desk.bz2

🌸 DESCRIPTION 🌸

maildirsync is a utility for online Maildir-synchronization. it's literally designed to be used on live maildir folders, to be fail-safe, and to be optimized for minimal bandwidth. the whole vibe is: don't drop the ball, don't eat the wifi. 🦋

🧚 HOW IT WORKS 🧚

so if you call the program once, it will propagate the changes from the source side to the target side. two-way synchronization requires two state-file and two call for the program. yes, two of each. it's giving commitment. 💖

this propagation is basically two different operations from the source side:

  • New file is created or an existing file is moved to a new location (e.g flags are changed).
  • A file is deleted in the source side. (Will be deleted in the target side also).

at the first phase, the source side reads the state file (which stores the state of the last synchronization) and compares it to the current state. it collects the changes and sends them the target side.

the target side checks every file, which is marked new in the source file, and decides if:

  • The file needs to be downloaded fully
  • Its header needs to be downloaded
  • We have this file already, so we need no data.

after it decides, it sends back the requests for new files.

then the source side will send the files to the target side, which stores them into the Maildir structure.

after the send operation is completed, both operation agrees upon the commit operation. then the source side saves the new state into the state file.

note if we forget saving the state, or the program exits before it, the operation can be restarted without data loss and inconsistency, because all operations can be redone without errors. periodt. no cap. 🔥

🌈 REMOTE OPERATION 🌈

Maildir can be used in remote mode, so it can synchronize Maildir folders between computers. if you want to use it this way, you have to provide the host name before either the source or the target, like:

maildirsync.pl ... desktop:Maildir Maildir lib/maildirsync.bz2

or

maildirsync.pl ... Maildir desktop:Maildir lib/maildirsync.bz2

in remote mode, the target side must have maildirsync installed also. (see the --maildirsync command-line argument).

the state-file must be in the same system as the source, so the source file in the first example is searched in the "desktop" computer, and in the local computer in the second example.

at least source or the destination must be local, so you cannot sync maildirs on two different remote hosts. bestie. it's just not happening. 🙈

💜 COMMAND-LINE SWITCHES 💜

some command line switches has two form: a short form and a longer form. in the short form, the switches can be grouped, like: -vvvr. short options with parameters also can be grouped, but the parameter must be the following command-line argument, like:

maildirsync.pl -rvvvbR Maildir/Trash/cur ssh ...

it is the same as:

maildirsync.pl --recursive --verbose --verbose --verbose --backup \
    Maildir/Trash/cur --rsh ssh ...

long options can use '=' for assigning the parameter, or they can use the syntax above.

let's see what switches we have:

  • --recursive, -r — process the base folder as the recursive collection of Maildir folders. 🎀
  • --backup dir, -b dir — the deleted files are backed up to the specified directory (this directory does not needs to be a maildir folder). the directory is relative to the start-up directory, not the target base folder! the directory is created if it does not exists. ✨
  • --backup-tree, -B — this option is only useful when used in conjunction with the --backup option. if set, deleted messages are moved to a Maildir folders tree inside the backup directory with the same relative path. the resulting backup directory can be used with any Maildir-capable application (MUAs, MTAs, etc.). 🌳
  • --bzip2 bzip2 — path to the bzip2 utility (used only when the state file has .bz2 extension). note that using bzip2 is turned out to be quite unstable, it sometimes leave the state file empty or corrupted. 💀
  • --gzip gzip — path to the gzip utility (used only when the state file has .gz extension).
  • --maildirsync maildirsync.pl — path to the maildirsync utility on the remote machine (if we use maildirsync in remote mode).
  • --rsh ssh — path to the utility, which can be used to connect to the remote side. it defaults to "ssh". note that the protocol, which is used in remote mode, does not contain compression, but the data can be compressed well, so I suggest using the ssh compression for this purpose.
  • --destination — if it is provided, then it does transformation in the file names. the transformations are the following:
    • win: Converts ":" to ":" (since windows don't allow colon in the filename).
    • lin: Converts ":" to ":".
  • --verbose, -v — adds more verbosity to the operation. there are currently 6 different verbosity level:
    • 0 — No information at all.
    • 1 — Main operations.
    • 2 — Files sent, received, deleted, moved, md5 calculations.
    • 3 — Directories read and created.
    • 4 — Options + command echo.
    • 5 — Misc info about file transfer.
  • --alg md5, -a md5 — selects the synchronization algorithm. currently two algorithms are provided:
    • id (default) — the messages are synchronized only by the ID of a message (the ID can be determined by the message filename).
    • md5 — this method is recommended for low bandwidth operations. this mode can reduce the file transfers by checking the message moves on the target side. this mode requires an MD5 sum on the body of the message, so the first use of this mode can be quite time-consuming on both sides. ⏳

for more information about the algorithms, read the chapter about that.

  • --delete-before, -d — normally the delete operations on the target side is done after the transfers. use this switch if you don't have too much space on your hard disk. note that using this switch can reduce the chance of detecting the message moves. 🫶
  • --exclude, -x — excludes a directory regexp from the transfer and removes it from the state file. this option can be used more than once to specify more than one directories. note, that the directory, which is matched against these regexps are the relative path of the folders with a leading '/', so if you want to exclude your Trash folder in the root of your synchronization, then you have to use the following form:
--exclude=^/Trash

the excluded paths are used in either source or the target side also. so if you exclude a very large directory, you will notice speedup in the source and target side also.

note that this regexp is matched for every directory that is read from the filesystem, and every directory what is found in the state file. so if you provide the exclude pattern as ^/Trash$, then it will skip the Trash directory when traversing the directory structure, but it WON'T skip files from the Trash/cur directory when reading from the state file! so be careful if you use the exclude pattern with existing state file! 😭
  • --exclude-source, --exclude-target — these parameters can be used to exclude files only from one side of the synchronization. can be useful with the rename options (below).
  • --rename — can be used to rename files when transferred from one side to another. a perl expression is the parameter for this.

if you use this option, use it with care, because you have to provide exactly the opposite of the name-transformation if you sync to the other side, for example:

in one side, you can use (A to B):

--rename="s{^/Saved/}{/ToBeSaved/}" \
--exclude-target="^/Saved/" \

in the other side, you can use (B to A):

--exclude-source="^/Saved/" \
--rename="s{^/ToBeSaved/}{/Saved/}" \

in this case, the Saved folder in the A side will be synchronized with the ToBeSaved folder in the B side, and the Saved folder in the B side will be excluded from the synchronization. this scenario can be used when you don't want to store your emails in the server, but you want to use the "Saved" folder in the server too. in this case, the emails will be downloaded from the server (A side) to your laptop (B side), then you can move them to the Saved folder in B with a script. if this is done, then you can resynchronize, then the saved files will disappear from the server also. 🌈

🧠 ALGORITHMS 🧠

okay so bestie, right now the program ships with two algorithms and you get to pick which one does your syncing. pick wisely, they're giving very different energy 💅

✨ id (default) ✨

This one is the OG, the main character, and it's the default because it's just that girl. It works off the message-ids of your messages. The whole vibe is that a message-id shows up exactly once in both repositories with the same id — that's the assumption, no cap. And you can literally pull that id straight out of the filename.

With this algorithm you can track flag changes and track when a message gets deleted, and those changes can be propagated over to the other side too. It also handles the case where a message gets copied from the new directory into the cur directory without retransmitting the files over the network 🦋 which, slay.

This algorithm is recommended if you want a simple and quite fast operation, and if you have a not-so-slow internet connection.

💅 md5 💅

This one does further calculations, she's doing the most and honestly? Iconic. Besides all the data that the id algorithm stores, she also stores the size of the header and the md5 sum of the message body for every single message.

By using this algorithm you can track the copies and moves of a message, so you don't need to retransmit large files if you move them to a new folder. Obsessed, truly.

If you copy or move a message from one folder to another, the header is sometimes changed by the mail-reader program. That's why we can't just simply calculate the MD5 sum of the whole message — rude of the mail reader honestly 😭. And a new message in a folder will have a new identifier also, so it doesn't violate the law that a message-id is unique. The math stays mathing ✨

So here's the tea: when you copy or move a message from your INBOX to your Save folder (for example), the new message is analyzed on the source side, the header-size and md5 sum get calculated on that new message, and from the md5-hash the source side can tell the target side which messages have the same hash-value — so the target side can copy the body from that other message. If the target side successfully copies the body from one of those provided messages, then only the header needs to be transmitted across the network. If the target-side did not find the messages, then it requests for the body also. Fair enough, bestie, it tried 🙈

🌐 ONLINE OPERATION 🌐

Online operation means the software can be used in an online mailbox also. It assumes that the Maildir folder can be changed while the program is working, so it tries to be as fail-safe as can be. Very protective, very girlboss 🫶

Every new file is opened in the tmp directory, and it only gets moved to the target place when the file is fully downloaded. No half-baked files in the chat, periodt 💅

This mode of operation was the first priority, because this feature is missing from most synchronizer software — including my drsync utility also. Not throwing shade, just facts 🎀

⚡ SPEED ⚡

I am using this program to synchronize my Mailboxes. I have 9700 emails in my mailbox and the state file (bzipped) is 283K. Look at her go 👑

The first time of a two-way synchronization between a P166 server and a PIII/1200 notebook over a Cable network, where the starting position is an already synchronized directory, tooks about 10 minutes. This time is used for md5-calculation and message-id propagation.

The next two-way run tooks about 40 seconds. Now THAT is a glow-up 😭🔥

These things are measured in Debian GNU/Linux testing/unstable operating system (08 Oct 2002). Yes we're time-traveling, it's giving vintage 💜

These are only the overhead of the software, not the real transfer. If you got a very big email, it needs to be transferred at least one time on the network. But if you have it in both sides, then it does not require any more transfer if you save it to different folders. Efficiency queen ✨

📝 TODO 📝

I am currently happy with this feature set, but if I have time, I will implement these features into the software. Anyway if you have time and willingness, I accept patches also 🫶

  • Message-size limit. By limiting the maximum transmitted message, you can effectively use this software if you have very low bandwidth.
  • Config-file and cron-safe operation.

Copyright (c) 2000-2010 Szabo, Balazs (dLux)

All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. 💜

🦋 AUTHOR 🦋

dLux (Szabo, Balazs) <dlux@dlux.hu>

and that's the whole page bestie, we ate and left no crumbs 🍓 this lil manual is from perl v5.32.1, dated 2021-09-28, and it's the MAILDIRSYNC(1) page — go sync something and be iconic about it 💖✨

📦 where this page comes from