DOVEADM-SYNC(1)

💖 DOVEADM-SYNC 💖

okay bestie, welcome to the DOVEADM-SYNC(1) page — this is Dovecot's whole two-way mailbox synchronization situation, plus its one-way backup bestie doveadm-backup. it's giving mailbox migration main character energy and honestly? we're obsessed. let's get into it. 🦋

---

✨ NAME ✨

• doveadm-sync — Dovecot's two-way mailbox synchronization utility
• doveadm-backup — Dovecot's one-way mailbox synchronization utility

---

🎀 SYNOPSIS 🎀

doveadm [-Dv] sync [-u user|-A|-F file] [-S socket_path] [-1fPRU] [-l
secs] [-r rawlog path] [-m mailbox] [-g mailbox guid] [-n namespace|-N]
[-x exclude] [-a all mailbox] [-s state] [-T secs] [-t start date] [-e
end date] [-O sync flag] [-I max size] -d|destination

doveadm [-Dv] backup [-u user|-A|-F file] [-S socket_path] [-fPRU] [-l
secs] [-r rawlog path] [-m mailbox] [-g mailbox guid] [-n namespace|-N]
[-x exclude] [-a all mailbox] [-s state] [-T secs] [-t start date] [-e
end date] [-O sync flag] [-I max size] -d|destination

---

🌈 DESCRIPTION 🌈

dsync is Dovecot's mailbox synchronization utility. no cap. it can do several different use cases, and they're all kinda iconic in their own way: two-way synchronization of mailboxes, creating backups of mails, and converting mailboxes from/to different mailbox formats. all of these can be used within the same server or between different servers — via ssh(1) or tcp connections, bestie. remote mailboxes can also be accessed via IMAP protocol, which lets you use dsync for mailbox migration purposes. 💅

you can run dsync in one of three modes:

• doveadm backup performs one-way synchronization. if there are any changes in the destination they will be deleted, so the destination will look exactly like the source. periodt.
• doveadm sync performs two-way synchronization. it merges all changes without losing anything. both the mailboxes will end up looking identical after the synchronization is finished. slay.
• doveadm sync -1 performs one-way synchronization, but it merges the changes in destination without deleting anything. this doesn't currently work perfectly, so its use should be limited. its main purpose is that during mailbox migration you can run doveadm backup multiple times, then switch mails to be delivered to the new mailbox and run doveadm sync -1 once more to transfer any last new mails from the old mailbox.

the one-way algorithm is the same as two-way dsync algorithm except the source account is not modified. it fetches the message's GUID (Global UID), which is used to identify any conflicting UIDs in messages. as long as the source and destination side has matching UID<->GUID mapping, those emails are assumed to be synced correctly. only after the first mismatch will changes begin.

Example: Source mailbox has messages UID 1..5; source mailbox is sync'd using doveadm backup to the destination. Subsequently, UID 6 is delivered to the source mailbox and UID 1 is expunged from the destination mailbox. In this example, UID 1 is kept removed (in destination) because UID 1..5 have identical Date+Message-ID headers. UID 6 is not seen in destination so it's copied.

If both source and destination have UID 6, but the messages are different, the headers don't match and both the messages are kept in the destination but they're given new UIDs 7 and 8 just to be sure any client didn't get confused about what UID 6 actually was. Thus, one-way sync begins to quickly diverge from the source mailbox once changes start to occur on either side; one-way sync should therefore normally only be used within a short period of time after a doveadm backup or doveadm sync command was used to synchronize the mailboxes.

there are also three different synchronization algorithms:

• Full synchronization (-f parameter) scans through all the messages in all the mailboxes. this guarantees that everything will be synchronized, but it's unnecessarily slow for incremental synchronization. 😭
• Fast synchronization (default) first attempts to find mailboxes that have changed, and synchronize only those. this is done by checking the mailboxes' metadata (NEXTUID and HIGHESTMODSEQ). usually this works fine, especially with one-way synchronization, but if both sides do exactly the same number of changes, the metadata may end up containing the same values even if the changes were different. 💀
• Stateful synchronization (-s parameter) is the most efficient way to synchronize mailboxes. it relies on having the earlier dsync run's state saved somewhere and being passed to the next dsync run. based on this state dsync can send only the changes that happened after the previous dsync run. as long as the state or the mailboxes aren't corrupted this algorithm should work perfectly. the replicator process uses this internally to perform most of the synchronization. 👑

the syncing is done as perfectly as possible: an IMAP or a POP3 client shouldn't be able to notice any differences between the two mailboxes. two-way syncing means that it's safe to do any kind of modifications in both sides, and dsync will merge the changes without losing any changes done on either side. this is possible because dsync can access Dovecot's index logs that keep track of changes. it's of course possible to have conflicts during merging, these are resolved in a safe way. see the dsync design document for more information. 🫶

dsync uses the same configuration files as the rest of Dovecot (via doveconf(1) binary). the entire configuration can be changed by giving -c parameter to another configuration file, or using -o parameter to override specific settings. when executing a remote dsync program it works the same way: it uses its own local configuration. no drama.

dsync can be run completely standalone. it doesn't require any Dovecot server processes to be running, except when using -u parameter to do a userdb lookup from auth process. lowkey independent.

dsync can sync either one or multiple users using the -u or -A parameters. for continuous replication you can use the Dovecot replicator process, which automatically runs dsync whenever messages have changed. iconic behavior fr fr. ✨

---

to be continued, bestie… part 2 is coming and it's gonna eat 💅

✨ OPTIONS ✨

Okay bestie, first up: the global doveadm(1) options. These are the ones that work everywhere, not just here. 💅

• -D — turns on verbosity and debug messages. It's giving "tell me everything, I can handle it" energy. 🔍
• -o setting=value — overrides a config setting from /etc/dovecot/dovecot.conf and from the userdb, using the value you give it. Wanna override more than one thing? Just stack the -o option as many times as you want. No cap. ✨
• -v — enables verbosity, and yes, that includes the progress counter. Watching that counter go is lowkey therapeutic. 🦋

---

Now for the command-specific options, aka the main characters of this page: 👑

• -A — if -A is present, the command gets performed for all users. But listen, using this with system users from userdb { driver = passwd } is not recommended, because that userdb also contains users with a lower UID than whatever you configured in the first_valid_uid setting. 🙈

When the SQL userdb module is in play, make sure the iterate_query setting in /etc/dovecot/dovecot-sql.conf.ext matches your database layout. And when you're on the LDAP userdb module, make sure the iterate_attrs and iterate_filter settings in /etc/dovecot/dovecot-ldap.conf.ext match your LDAP schema. Otherwise doveadm(1) literally cannot iterate over all users. It will simply not be able to, periodt. 💀

• -F file — runs the command for all the users listed in file. It's basically the -A option's cousin, except instead of pulling the user list from the userdb, it reads them from the file you hand it. One username per line. Simple, iconic. 📄
• -1 — do a one-way synchronization instead of two-way. One direction only, we're not doing the back-and-forth today. 🎀
• -f — do a full synchronization. Everything. All of it. 🔥
• -N — synchronize all the available namespaces. By default, only namespaces that don't have an explicit location setting get synchronized. So this is your "give me the whole roster" flag. 🌈
• -P — runs a doveadm-purge(1) for the destination (remote) storage after synchronization. Cleanup crew, assemble. 🧹
• -R — do a reverse sync. Normally messages get pushed from the local system to the destination (remote). This option flips the whole flow and instead pulls messages from the remote into the local storage. Plot twist! 😭
• -S socket_path — the argument here is either an absolute path to a local UNIX domain socket, or a hostname and port (hostname:port), so you can connect to a remote host over a TCP socket. This lets an administrator execute doveadm(1) mail commands through that socket. 🫶
• -T secs — specify the time in seconds that doveadm(1) should wait for stalled I/O operations. The default timeout is 600 seconds. Tick tock, bestie. ⏳
• -U — this is used internally by replicator so that dsync notifies it when the synchronization is finished. iykyk. 🤫
• -d — use the default destination, which gets looked up from the mail_replica userdb extra field. Let the config decide. 💅
• -g mailbox_guid — same as -m, but it finds the mailbox to be synchronized by its GUID instead of by name. Names are so last season. 🎀
• -l secs — locks the dsync for this user. Waits for a maximum of secs before giving up. You should use this to avoid broken synchronization if there's any chance dsync is being run concurrently for the same user. Mutual respect for the lock, fr fr. 🔒
• -m mailbox — synchronize only this mailbox name. Just the one. ✨
• -n namespace — synchronize only the specified namespace. This parameter can be used multiple times. Stack it up. 🦋
• -a all mailbox — name for the "All mails" virtual mailbox. If you specify it, mails are attempted to be copied from this mailbox instead of being saved separately. This may reduce the total disk space usage as well as disk IO. Efficiency? Obsessed. 💖
• -t start date — skip any mails whose received-timestamp is older than the specified time. Bye, ancient mail. 👋
• -e end date — skip any mails whose received-timestamp is newer than the specified time. Not today, fresh mail. 🙅
• -O sync flag — sync only mails that have the specified flag. And if the flag name begins with "-", sync all mails except the ones with that specified flag. Sneaky little minus sign. ➖
• -I max size — skip any mails larger than the specified size. Big mail, you're not coming. 💀
• -r rawlog_path — when running dsync remotely, write the remote input/output traffic to the specified log file. Receipts, but for network traffic. 📝
• -s previous_state — use stateful synchronization. If the previous state is unknown, use an empty string. The new state is always printed to standard output. Always. No exceptions. 👑
• -u user/mask — run the command only for the given user. You can also use '*' and '?' wildcards (e.g. -u *@example.org). When neither the -A option, nor the -F file option, nor the -u user was specified, the command will be executed with the environment of the currently logged in user. So if you specify nothing, it's you. Main character energy by default. 💅
• -x mailbox_mask — exclude the specified mailbox name/mask. The mask may contain "?" and "*" wildcards. The mask can also be a special-use name (e.g. \Trash). This parameter can be used multiple times. Get that Trash out of here. 🗑️

---

🌸 ARGUMENTS 🌸

• destination — this argument specifies the synchronized destination. It can be one of the following, and bestie, they're all a vibe: 💖
  • location — same as the mail_location setting, e.g. maildir:~/Maildir
  • remote:login@host — uses the dsync_remote_cmd setting to connect to the remote host (usually via ssh)
  • remoteprefix:login@host — this is the same as remote, except "user@domain\n" is sent before the dsync protocol starts. This allows implementing a trusted wrapper script that runs doveadm dsync-server by reading the username from the first line. Fancy. 🎀
  • tcp:host[:port] — connects to a remote doveadm server via TCP. The default port is specified by the doveadm_port setting.
  • tcps:host[:port] — this is the same as tcp, but with SSL. Secure queen. 🔐
  • command [arg1 [, arg2, ...]] — runs a local command that connects its standard input & output to a dsync server.

---

💀 EXIT STATUS 💀

dsync will exit with one of the following values, so pay attention to what it's telling you:

• 0 — synchronization was done perfectly. Chef's kiss. 😘
• 2 — synchronization was done without errors, but some changes couldn't be done, so the mailboxes aren't perfectly synchronized. Running dsync again usually fixes this. Typically this occurs for message modification sequences with newly created mailboxes. It can also occur if one of the mailboxes changes during the syncing. So close, yet so far. 😭
• 1, >2 — synchronization failed. It's giving disaster. 🚨

---

🦋 EXAMPLE 🦋

SYNCHRONIZATION

Synchronize mailboxes with a remote server. Any errors get written to stderr.

doveadm sync -u username@example.com remote:server-replica.example.com

If you need more complex parameters to ssh, you can use e.g.:

doveadm sync -u username@example.com ssh -i id_dsa.dovecot \
mailuser@example.com doveadm dsync-server -u username@example.com

CONVERTING

Assuming that the mail_location setting in /etc/dovecot/conf.d/10-mail.conf is set to: mail_location = mdbox:~/mdbox, a logged in system user may convert her/his mails from its Maildir in her/his home directory to the mdbox mailbox format. The user has to execute the command:

doveadm sync maildir:~/Maildir

If you want to do this without any downtime, you can do the conversion one user at a time. Initially:

• Configuration uses mail_location = maildir:~/Maildir
• Set up the possibility of doing per-user mail location using userdb extra fields.

Then for each user:

1. Run doveadm sync once to do the initial conversion.
2. Run doveadm sync again, because the initial conversion could have taken a while and new changes could have occurred during it. This second time only applies changes, so it should be fast.
3. Update mail extra field in userdb to mdbox:~/mdbox. If you're using auth cache, you need to flush it, e.g. doveadm auth cache flush.
4. Wait for a few seconds and then kill (doveadm kick) the user's all existing imap and pop3 sessions (that are still using maildir).
5. Run doveadm sync once more to apply final changes that were possibly done. After this there should be no changes to Maildir, because the user's mail location has been changed and all existing processes using it have been killed.

Once all users have been converted, you can set the default mail_location to mdbox and remove the per-user mail locations from userdb. Slay, migration complete. 👑

---

🐛 REPORTING BUGS 🐛

Report bugs, including doveconf -n output, to the Dovecot Mailing List <dovecot@dovecot.org>. Information about reporting bugs is available at: http://dovecot.org/bugreport.html

(Yes bestie, include the doveconf -n output. They need it, don't make them ask. 🫶)

---

👀 SEE ALSO 👀

doveadm(1), doveadm-auth(1), doveadm-kick(1), doveadm-purge(1), doveconf(1)

Additional resources:

• dsync design — http://wiki2.dovecot.org/Design/Dsync

---

And that's the whole page, fr fr. This one's from Dovecot v2.3, dated 2015-08-31, and it lives under the name DOVEADM-SYNC(1) in the manual. Go forth and sync, bestie. 💖✨🦋