RFC 4549

Synchronization Operations for Disconnected IMAP4 Clients

📅 published June 2006, from ietf

RFC 4549: Synchronization Operations for Disconnected IMAP4 Clients 🦋

Okay so this is RFC 4549 🎀, and it's all about how a disconnected IMAP4 client (one that's offline or barely online) can sync its mailbox without wasting time on the phone with the server 📞. The author is A. Melnikov, Ed. (that "Ed." means editor 💅) from Isode Ltd., and it's part of the Network Working Group 🌸. The category is Informational, and the date is June 2006 📅. There's nothing in the title block about it obsoleting or updating any other RFCs, so it stands on its own ✨.

Status of This Memo ✨

This memo is here to share information with the Internet community 🌈. It is not an Internet standard of any kind, so don't go treating it like one 🙈. Distribution of this memo is unlimited, so share away 🫶.

Copyright (C) The Internet Society (2006).

Abstract 🔥

Okay, so this document tackles the stuff involved in building a disconnected IMAP4 client 💻. The main focus is the "driver" part of a sync tool, which is the code that decides which set of IMAP4 commands to send so the disconnected client gets synced in the way the human using it is most likely to be happy with 😭.

It also covers different strategies a disconnected client can use, and shows how to use the IMAP protocol to make the synchronization process as quick as possible ⚡.

Lastly, it lists IMAP extensions that a server should implement so it can give disconnected clients better synchronization tools 🧚.

Table of Contents

   1. Introduction ....................................................3
      1.1. Conventions Used in This Document ..........................3
   2. Design Principles ...............................................3
   3. Overall Picture of Synchronization ..............................4
   4. Mailbox Synchronization Steps and Strategies ....................7
      4.1. Checking UID Validity ......................................7
      4.2. Synchronizing Local Changes with the Server ................8
           4.2.1. Uploading Messages to the Mailbox ...................8
           4.2.2. Optimizing "move" and "copy" Operations .............9
           4.2.3. Replaying Local Flag Changes .......................14
           4.2.4. Processing Mailbox Compression (EXPUNGE) Requests ..15
           4.2.5. Closing a Mailbox ..................................17
      4.3. Details of "Normal" Synchronization of a Single Mailbox ...18
           4.3.1. Discovering New Messages and Changes to Old
                  Messages ...........................................18
           4.3.2. Searching for "Interesting" Messages. ..............20
           4.3.3. Populating Cache with "Interesting" Messages. ......21
           4.3.4. User-Initiated Synchronization .....................22
      4.4. Special Case: Descriptor-Only Synchronization .............22
      4.5. Special Case: Fast New-Only Synchronization ...............23
      4.6. Special Case: Blind FETCH .................................23
   5. Implementation Considerations ..................................24
      5.1. Error Recovery during Playback ............................26
      5.2. Quality of Implementation Issues ..........................28
      5.3. Optimizations .............................................28
   6. IMAP Extensions That May Help ..................................30
      6.1. CONDSTORE Extension .......................................30
   7. Security Considerations ........................................33
   8. References .....................................................33
      8.1. Normative References ......................................33
      8.2. Informative References ....................................34
   9. Acknowledgements ...............................................34

1. Introduction ✨

Lots of the recommendations in this document work for every kind of IMAP client 🌟. Still, the document mostly focuses on disconnected mail clients [IMAP-MODEL] 📬. It also suggests some IMAP extensions* that IMAP servers should implement to make life easier for disconnected clients 💖. The [UIDPLUS] extension is the star here, since it was specifically designed to streamline certain disconnected operations like expunging, uploading, and copying messages (see Sections 4.2.1, 4.2.2.1, and 4.2.4) 👑.

Readers are also strongly advised to read RFC 2683 [RFC2683] 📚. Bestie, read it, trust me 🙈.

* Note that the functionality provided by the base IMAP protocol [IMAP4] is sufficient to perform basic synchronization. So you don't need the fancy extensions just to get the basics working 💅.

1.1. Conventions Used in This Document

Okay, quick rules before we dive in 🎀. In examples, "C:" and "S:" mean lines sent by the client and server, respectively. Long lines in examples get broken up just so they're easier to read 🍓.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [KEYWORDS] 🦋. Those are the official vibes-check words, so pay attention to them 💅.

Let's call an IMAP command idempotent if running it twice in a row gives you the same result as running it just once 🔁. Think of it like hitting "like" on a post. Doing it again doesn't change anything 💜.

2. Design Principles 💅

Here's the big one 🔥. All the mailbox state and content info stored on a disconnected client should be treated strictly as a cache of the server's state 🗂️. The "master" state lives on the server, just like it would for an interactive IMAP4 client 👑. There's one exception: info about the disconnected client's own cache stays on the client. That includes flag changes made while offline and any uploads that are scheduled but not done yet ✨. So the IMAP4 server is not responsible for remembering the state of the disconnected IMAP4 client 🙅‍♀️. That's the client's job, babe 💖.

Now, what even is a disconnected client? 🤔 It's a client that, for whatever reason, wants to spend as little time as possible "on the phone" with the IMAP4 server ☎️. Often that's because it's on a dialup connection with super low bandwidth 🐢. Other times the human is just in a rush to catch a flight, or something else outside our control is going on ✈️. Whatever the reason, we assume we must make efficient use of the network connection 🚀. That means not sending spurious traffic, and it also means not letting the connection sit idle while the client or server is busy doing purely local computation or I/O 😤. A simpler way to say all this: we treat network connections as "expensive" 💸.

Real-world experience with disconnected mail systems has shown that there's no single synchronization strategy that works for every case 🌈. Different humans have different preferences 💅. And even the same human's preference can change depending on outside circumstances (like how much of a hurry they're in today) and on how much they value the messages being transferred 🫠. The bottom line is that the sync program can't magically guess exactly what the human wants, so the human has to give it some guidance 🧚.

Put those two principles together and you get this conclusion: the sync program must make its decisions based on guidance from the human 🎯. That guidance can come from picking options in the user interface or from some kind of configuration file ⚙️. Almost certainly, the program should not pause to wait for I/O with the human in the middle of the synchronization process 🙅‍♀️. And the human will almost certainly have several different configurations for different circumstances, because of course they will 💅✨.

Last but definitely not least 😭: a disconnected client has no way of knowing what changes might have happened to the mailbox while it was offline. So message numbers aren't useful to it at all 🙈. Instead, all disconnected client operations should be performed using UIDs 🔑. That way, the client can be sure that it and the server are talking about the same messages during synchronization, and that's the whole point 💜🫶.

3. Overall Picture of Synchronization ✨

okay so the basic plan for syncing is laid out below 🗺️ bestie, the real strategy can totally change from app to app, or depend on what sync mode you're in 🦋

  • a) Process any "actions" that were pending on the client and weren't tied to a mailbox. (This is mostly sending messages you wrote offline with SMTP 📨 It's not part of IMAP sync, but it's mentioned here for completeness.)
  • b) Fetch the current list of "interesting" mailboxes. (The disconnected client should let the user skip this step completely 🙌)
  • c) "Client-to-server synchronization": for each IMAP "action" that was pending on the client, do the following:
    1. If the action implies opening a new mailbox (any operation that operates on messages), open the mailbox. Check its UID validity value (see Section 4.1 for more details) returned in the UIDVALIDITY response code. If the UIDVALIDITY value returned by the server differs, the client MUST empty the local cache of the mailbox and remove any pending "actions" that refer to UIDs in that mailbox (and consider them failed) 💀 Note that this doesn't affect actions performed on client-generated fake UIDs (see Section 5).
    2. Perform the action. If the action is to delete a mailbox (DELETE), make sure that the mailbox is closed first (see also Section 3.4.12 of [RFC2683]) 🧹
  • d) "Server-to-client synchronization": for each mailbox that needs syncing, do the following:
    1. Check the mailbox UIDVALIDITY (see Section 4.1 for more details) with SELECT/EXAMINE/STATUS 🔍 If the UIDVALIDITY value returned by the server differs, the client MUST:
      • empty the local cache of that mailbox;
      • remove any pending "actions" that refer to UIDs in that mailbox and consider them failed; and
      • skip step 2-II.
    2. Fetch the current "descriptors":
      • I) Discover new messages.
      • II) Discover changes to old messages.
    3. Fetch the bodies of any "interesting" messages that the client doesn't already have 📥
  • e) Close all open mailboxes that aren't needed for further operations (if staying online), or disconnect all open connections (if going offline) 👋

Terms used, because we need to be on the same page:

"Actions" are queued requests that the human made to the client's Mail User Agent (MUA) software while the client was disconnected 📝

We define "descriptors" as a set of IMAP4 FETCH data items. Basically, a message's descriptor is the bundle of info that lets the sync program figure out what protocol actions are needed to bring the local cache to the desired state for that message 🎯 Since this decision is really up to the human, this info probably includes at least a few header fields meant for human eyes. Exactly what counts as a descriptor depends on the client implementation. At a minimum, the descriptor contains the message's UID and FLAGS. Other likely candidates are the RFC822.SIZE, RFC822.HEADER, BODYSTRUCTURE, or ENVELOPE data items 🧠

Comments (these are the tea ☕):

  1. The list of actions should be ordered. Like, if the human deletes message A1 in mailbox A, then expunges mailbox A, and then deletes message A2 in mailbox A, the human expects message A1 to be gone and message A2 to still be there, but now marked as deleted 😤

By processing all the pending actions before syncing, we avoid having to compensate for the local MUA's changes to the server's state. Once all the pending actions are processed, the steps the client takes to sync itself will be the same no matter where the changes to the server's state came from 🫶

  1. Steps a and b can be performed in parallel. Or, step a can be performed after d 🔀
  2. For step b, the set of "interesting" mailboxes pretty much has to be decided by the human 🙋‍♀️ Which mailboxes are in that set can change between IMAP4 sessions with the same server, client, and human. An interesting mailbox can be a mailbox returned by the LSUB command (see Section 6.3.9 of [IMAP4]). The special mailbox "INBOX" SHOULD be in the default set of mailboxes the client considers interesting. That said, being able to ignore INBOX for a particular session or client may be super valuable for some mail filtering strategies 🗂️
  3. On step d-2-II, the client also finds out about changes to the flags of messages it already has in its local cache, and about messages in the local cache that no longer exist on the server (meaning messages that have been expunged) 🗑️
  4. "Interesting" messages are the ones the sync program thinks the human wants cached locally, based on the configuration and the data retrieved in step b 💅
  5. A disconnected IMAP client is a special case of an IMAP client, so it MUST be able to handle any "unexpected" unsolicited responses, like EXISTS and EXPUNGE, at any time 🙃 The disconnected client MAY ignore EXPUNGE responses during the "client-to-server" synchronization phase (step c).

The rest of this discussion will focus mainly on sync issues for a single mailbox 🎀

4. Mailbox Synchronization Steps and Strategies 🧚

4.1. Checking UID Validity

The "UID validity" of a mailbox is a number returned in an UIDVALIDITY response code, inside an OK untagged response, at mailbox selection time 🔢 The UID validity value changes between sessions when UIDs fail to persist between sessions, so it's basically the mailbox's way of saying "my IDs changed, don't trust your old ones" 😭

Whenever the client selects a mailbox, it must compare the returned UID validity value with the value stored in the local cache. If the UID validity values differ, the UIDs in the client's cache are no longer valid 🚫 The client MUST then empty the local cache of that mailbox and remove any pending "actions" that refer to UIDs in that mailbox. The client MAY also issue a warning to the human ⚠️ The client MUST NOT cancel any scheduled uploads (i.e., APPENDs) for the mailbox. Those uploads are sacred, periodt 💖

Note that UIDVALIDITY isn't only returned when you select a mailbox 👀 The COPYUID and APPENDUID response codes defined in the [UIDPLUS] extension (see also 4.2.2), and the UIDVALIDITY STATUS response data item, also contain a UIDVALIDITY value for some other mailbox. The client SHOULD behave as described in the previous paragraph (but it should act on the other mailbox's cache), no matter how it got the UIDVALIDITY value 🫡

4.2. Synchronizing Local Changes with the Server 🔄

4.2.1. Uploading Messages to the Mailbox 📤

Two of the most common examples of operations that result in message uploads are:

  1. Saving a draft message ✍️
  2. Copying a message between remote mailboxes on two different IMAP servers, or between a local mailbox and a remote mailbox 🪞

Message upload is done with the APPEND command. A message scheduled to be uploaded has no UID associated with it yet, because all UIDs are assigned by the server 🙅‍♀️ The APPEND command will effectively associate a UID with the uploaded message, which can then be stored in the local cache for future reference. However, [IMAP4] doesn't describe a simple mechanism for discovering the message UID by just performing the APPEND command 😩 To figure out the UID, the client can do one of the following:

  1. Remove the uploaded message from the cache. Then, use the mechanism described in 4.3 to fetch the info about the uploaded message as if some other client had uploaded it 🕵️‍♀️
  2. Try to fetch header information as described in 4.2.2 in order to find a message that corresponds to the uploaded message. One strategy for doing this is described in 4.2.2 🔎

Option 1 is for a not particularly smart client 🙈 Here's what that looks like:

C: A003 APPEND Drafts (\Seen $MDNSent) {310}
S: + Ready for literal data
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:
S: A003 OK APPEND Completed

Fortunately, there's a much easier way to find the message UID when the [UIDPLUS] extension is around 🥹✨ Look at this:

C: A003 APPEND Drafts (\Seen $MDNSent) {310}
S: + Ready for literal data
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:
S: A003 OK [APPENDUID 1022843275 77712] APPEND completed

The UID of the appended message is the second parameter of the APPENDUID response code 💜 Just grab that number and you're done, no cap 💅

4.2.2. Optimizing "move" and "copy" Operations 🚚

Real-world experience with IMAP and other mailbox access protocols that support multiple mailboxes shows that moving a message from one mailbox to another is an extremely common operation 📦 It's giving "happens all the time," so it's worth making it smooth, obviously 🌈

4.2.2.1. Moving a Message between Two Mailboxes on the Same Server ✨

okay so in IMAP4, a "move" between two mailboxes on the same server is really a COPY plus a STORE +FLAGS (\Deleted) combo under the hood 🫶. Totally makes sense for IMAP, but it's giving awkward for a simple-minded disconnected client. It ends up deleting (and possibly expunging) its cached copy, then fetching an identical one over the network again 💀.

However, if the server supports UIDPLUS, it can help a lot 🦋:

C: A001 UID COPY 567,414 "Interesting Messages"
S: A001 OK [COPYUID 1022843275 414,567 5:6] Completed

This tells the client that the message with UID 414 in the current mailbox was successfully copied to the mailbox "Interesting Messages" and got the UID 5, and that the message with UID 567 got the UID 6 🎀.

If the server doesn't support UIDPLUS, there's a trick you can use 🧚. By including the Message-ID: header and the INTERNALDATE data item as part of the descriptor, the client can check a "new" message's descriptor against messages already in its cache and skip fetching the extra copy. Of course, checking the local cache might cost more than just fetching the message, so this technique should not be used blindly 🙈. If the MUA implements a "move" command, it makes special provisions to use this technique when it knows that a copy/delete sequence is the result of a "move" command.

Note that servers are not required (although they are strongly encouraged with "SHOULD language") to preserve INTERNALDATE when copying messages 📝.

Also, since it's theoretically possible for this algorithm to find the wrong message (given sufficiently malignant Message-ID headers), implementers should provide a way to disable this optimization, both permanently and on a message-by-message basis 🚨.

Example 1: Copying a message in the absence of UIDPLUS extension.

At some point the client has fetched the source message and cached some information:

C: C021 UID FETCH <uids> (BODY.PEEK[] INTERNALDATE FLAGS)
...
S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600"
    FLAGS (\Draft $MDNSent) BODY[] {1036}
S: ...
S: Message-Id: <20040903110856.22a127cd@chardonnay>
S: ...
S: ...message body...
S: )
...
S: C021 OK fetch completed

Later, the client decides to copy the message:

C: C035 UID COPY 123 "Interesting Messages"
S: C035 OK Completed

Since the server didn't send the COPYUID response code, the client tries the optimization described above 🔍:

C: C036 SELECT "Interesting Messages"
...
C: C037 UID SEARCH ON 31-May-2002 HEADER
    "Message-Id" "20040903110856.22a127cd@chardonnay"
S: SEARCH 12368
S: C037 OK completed

Note that if the server has returned multiple UIDs in the SEARCH response, the client MUST NOT use any of the returned UID 🙅‍♀️.

4.2.2.2. Moving a Message from a Remote Mailbox to a Local 📥

Moving a message from a remote mailbox to a local one is done with FETCH (that includes FLAGS and INTERNALDATE), followed by UID STORE <uid> +FLAGS.SILENT (\Deleted) 💅:

C: A003 UID FETCH 123 (BODY.PEEK[] INTERNALDATE FLAGS)
S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600"
    FLAGS (\Seen $MDNSent) BODY[]
S: ...message body...
S: )
S: A003 OK UID FETCH completed
C: A004 UID STORE <uid> +FLAGS.SILENT (\Deleted)
S: A004 STORE completed

Note that there's no reason to fetch the message during synchronization if it's already in the client's cache 🙌. Also, the client SHOULD preserve delivery date in the local cache 📅.

4.2.2.3. Moving a Message from a Local Mailbox to a Remote 📤

Moving a message from a local mailbox to a remote one is done with APPEND 💌:

C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
    {310}
S: + Ready for literal data
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:
S: A003 OK [APPENDUID 1022843275 77712] completed

The client SHOULD specify the delivery date from the local cache in the APPEND 🗓️.

If the [LITERAL+] extension is available, the client can save a round-trip*:

C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
    {310+}
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:
S: A003 OK [APPENDUID 1022843275 77712] completed

* Note that there is a risk that the server will reject the message due to its size. If this happens, the client will waste bandwidth transferring the whole message. If the client wouldn't have used the LITERAL+, this could have been avoided 😬:

C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2004 05:26:59 -0600"
    {16777215}
S: A003 NO Sorry, message is too big

4.2.2.4. Moving a Message between Two Mailboxes on Different Servers 🌍

Moving a message between two mailbox on two different servers is a combination of the operations described in 4.2.2.2 followed by the operations described in 4.2.2.3 🧩.

4.2.2.5. Uploading Multiple Messages to a Remote Mailbox with MULTIAPPEND 🚀

When there's a need to upload multiple messages to a remote mailbox (e.g., as per 4.2.2.3), the presence of certain IMAP extensions may significantly improve performance 📈. One of them is [MULTIAPPEND].

For some mail stores, opening a mailbox for appending might be expensive. [MULTIAPPEND] tells the server to open the mailbox once (instead of opening and closing it "n" times per "n" messages to be uploaded) and to keep it open while a group of messages is being uploaded to the server 🗝️.

Also, if the server supports both [MULTIAPPEND] and [LITERAL+] extensions, the entire upload is accomplished in a single command/response round-trip ⚡.

Note: Client implementers should be aware that [MULTIAPPEND] performs append of multiple messages atomically. This means, for example, if there is not enough space to save "n"-th message (or the message has invalid structure and is rejected by the server) after successful upload of "n-1" messages, the whole upload operation fails, and no message will be saved in the mailbox. Although this behavior might be desirable in certain situations, it might not be what you want. Otherwise, the client should use the regular APPEND command (Section 4.2.2.3), possibly utilizing the [LITERAL+] extension. See also Section 5.1 for discussions about error recovery 🛟.

Note: MULTIAPPEND can be used together with the UIDPLUS extension in a way similar to what was described in Section 4.2.1. [MULTIAPPEND] extends the syntax of the APPENDUID response code to allow for multiple message UIDs in the second parameter 👑.

Example 2:

This example demonstrates the use of MULTIAPPEND together with UIDPLUS (synchronization points where the client waits for confirmations from the server are marked with "<--->"):

C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
    {310}
<--->
S: + Ready for literal data
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:  (\Seen) " 1-Jun-2002 22:43:04 -0800" {286}
<--->
S: + Ready for literal data
C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST)
C: From: Joe Mooch <mooch@OWaTaGu.siam.EDU>
C: Subject: Re: afternoon meeting
C: To: foobar@blt.example.com
C: Message-Id: <a0434793874930@OWaTaGu.siam.EDU>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: 3:30 is fine with me.
C:

S: A003 OK [APPENDUID 1022843275 77712,77713] completed

The upload takes 3 round-trips 🐢.

Example 3:

In this example, Example 2 was modified for the case when the server supports MULTIAPPEND, LITERAL+, and UIDPLUS. The upload takes only 1 round-trip 🏎️💨.

C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600"
    {310+}
C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST)
C: From: Fred Foobar <foobar@blt.example.COM>
C: Subject: afternoon meeting
C: To: mooch@owatagu.siam.edu
C: Message-Id: <B27397-0100000@blt.example.COM>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: Hello Joe, do you think we can meet at 3:30 tomorrow?
C:  (\Seen) " 1-Jun-2002 22:43:04 -0800" {286+}
C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST)
C: From: Joe Mooch <mooch@OWaTaGu.siam.EDU>
C: Subject: Re: afternoon meeting
C: To: foobar@blt.example.com
C: Message-Id: <a0434793874930@OWaTaGu.siam.EDU>
C: MIME-Version: 1.0
C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
C:
C: 3:30 is fine with me.
C:
S: A003 OK [APPENDUID 1022843275 77712,77713] completed

4.2.3. Replaying Local Flag Changes 🔁

The disconnected client uses the STORE command to sync its local flag state with the server 🫧. The disconnected client SHOULD use +FLAGS.SILENT or -FLAGS.SILENT to set or unset flags the user changed while offline. The FLAGS form MUST NOT be used, because there's a real risk it will overwrite flags on the server that some other client changed 😱.

Example 4:

For the message with UID 15, the disconnected client stores the flags \Seen and $Highest. The flags were modified on the server by some other client: \Seen, \Answered, and $Highest. While offline, the user requested that the $Highest flag be removed and that the \Deleted flag be added. The flag synchronization sequence for the message should look like:

C: A001 UID STORE 15 +FLAGS.SILENT (\Deleted)
S: A001 STORE completed
C: A002 UID STORE 15 -FLAGS.SILENT ($Highest)
S: A002 STORE completed

If the disconnected client is able to store additional binary state information (or a piece of information that can take a value from a predefined set of values) in the local cache of an IMAP mailbox or in a local mailbox (e.g., message priority), and if the server supports storing of arbitrary keywords, the client MUST use keywords to store this state on the server 🏷️.

Example 5:

Imagine a speculative mail client that can mark a message as one of work-related ($Work), personal ($Personal), or spam ($Spam). In order to mark a message as personal, the client issues 💼:

C: A001 UID STORE 15 +FLAGS.SILENT ($Personal)
S: A001 STORE completed
C: A002 UID STORE 15 -FLAGS.SILENT ($Work $Spam)
S: A002 STORE completed

In order to mark the message as not work, not personal and not spam, the client issues 🧼:

C: A003 UID STORE 15 -FLAGS.SILENT ($Personal $Work $Spam)
S: A003 STORE completed

4.2.4. Processing Mailbox Compression (EXPUNGE) Requests ✨

okay so picture a super basic offline client that wants to tidy up its mailbox while it's disconnected 🧹. Its first instinct might be to send EXPUNGE to clear out everything flagged \Deleted. The catch is that EXPUNGE permanently erases EVERY message with the \Deleted flag, including the ones someone else flagged on the server while you were offline. Unpleasant surprise, bestie 😭

Good news: the [UIDPLUS] extension has our back here too 🫶. It adds the UID EXPUNGE command, which takes a UID set listing exactly which messages can be expunged. The server then erases only the \Deleted messages named in that list. Anything left out of the UID set is safe, even if it's flagged \Deleted. Main character energy for your mailbox 💅

Example 6:

While the user was offline, 3 messages with UIDs 7, 27, and 65 were marked \Deleted when the user requested to compress the open mailbox. Another client marked a message \Deleted on the server (UID 34). During synchronization, the disconnected client issues:

   C: A001 UID EXPUNGE 7,27,65
   S: * ... EXPUNGE
   S: * ... EXPUNGE
   S: * ... EXPUNGE
   S: A001 UID EXPUNGE completed

If another client issues UID SEARCH DELETED command (to find all messages with the \Deleted flag) before and after the UID EXPUNGE, it will get:

Before:

   C: B001 UID SEARCH DELETED
   S: * SEARCH 65 34 27 7
   S: B001 UID SEARCH completed

After:

   C: B002 UID SEARCH DELETED
   S: * SEARCH 34
   S: B002 UID SEARCH completed

No UIDPLUS? No cap, you can fake it with a sequence of commands 🎀. Just know that if another client flags more messages as deleted while you're doing this, those extra messages will get expunged too 👀

  1. Find all messages marked \Deleted on the server.
   C: A001 UID SEARCH DELETED
   S: * SEARCH 65 34 27 7
   S: A001 UID SEARCH completed
  1. Find all messages that must not be erased (for the previous example the list will consist of the message with UID 34).
  2. Temporarily remove \Deleted flag on all messages found in step 2.
   C: A002 UID STORE 34 -FLAGS.SILENT (\Deleted)
   S: A002 UID STORE completed
  1. Expunge the mailbox.
   C: A003 EXPUNGE
   S: * 20 EXPUNGE
   S: * 7 EXPUNGE
   S: * 1 EXPUNGE
   S: A003 EXPUNGE completed

Here, the message with UID 7 has message number 1, with UID 27 has message number 7, and with UID 65 has message number 20.

  1. Restore \Deleted flag on all messages found when performing step 2.
   C: A004 UID STORE 34 +FLAGS.SILENT (\Deleted)
   S: A004 UID STORE completed

4.2.5. Closing a Mailbox 🚪

when the disconnected client needs to close a mailbox, it should not use the CLOSE command, because CLOSE does a silent EXPUNGE 🙅‍♀️. (Section 4.2.4 explains why EXPUNGE should not be used by a disconnected client.) CLOSE is only safe if the mailbox was opened with EXAMINE ✅.

If the mailbox was opened with SELECT, the client can use one of these to close it implicitly and dodge that silent expunge:

  1. UNSELECT - This is a command described in [UNSELECT] that works as CLOSE, but doesn't cause the silent EXPUNGE. This command is supported by the server if it reports UNSELECT in its CAPABILITY list. 🌸
  2. SELECT <another_mailbox> - SELECT causes implicit CLOSE without EXPUNGE.
  3. If the client intends to issue LOGOUT after closing the mailbox, it may just issue LOGOUT, because LOGOUT causes implicit CLOSE without EXPUNGE as well. 🧚
  4. SELECT <non_existing_mailbox> - If the client knows a mailbox that doesn't exist or can't be selected, it MAY SELECT it.

Wanna stay in the mailbox but still avoid that implicit EXPUNGE? Use this one:

  1. EXAMINE <mailbox> - Reselect the same mailbox in read-only mode. 🔒

4.3. Details of "Normal" Synchronization of a Single Mailbox 🦋

The most common kind of sync is when the human trusts their client's copy of a mailbox's state 🥰. They just want the client's cache caught up so it matches the mailbox's current state on the server. Simple and iconic.

4.3.1. Discovering New Messages and Changes to Old Messages 👀

Let <lastseenuid> be the highest UID the client knows about in this mailbox. UIDs are handed out in strictly ascending order, so this is basically the UID of the last message the client has seen. Let <lastseenuid+1> be that UID plus one. Let <descriptors> be a list of all the FETCH data items the implementation considers part of the descriptor. At minimum that's just FLAGS, but it usually also includes BODYSTRUCTURE and RFC822.SIZE. At this step, <descriptors> SHOULD NOT include RFC822 🚫.

With no extra info, the client can send these two commands:

tag1 UID FETCH <lastseenuid+1>:* <descriptors>
tag2 UID FETCH 1:<lastseenuid> FLAGS

The first command asks for info about the "new" messages (anything the server got since the last sync) 🆕. It also lets the client build a message number to UID map, but only for the new messages. The second command lets the client:

  1. update cached flags for old messages;
  2. find out which old messages got expunged; and
  3. build a mapping between message numbers and UIDs (for old messages).

The order matters here 💅. We want the server to start sending new message descriptors ASAP, so the client can start its own FETCH commands sooner. That's why we ask for descriptors of every message the client definitely doesn't have cached yet. The second command grabs what's needed to spot changes to messages the client already has. Also, the first command should only be sent if the UIDNEXT value cached by the client differs from the one the server returned. Once both commands are out, there's nothing else to do with this mailbox until responses to the first one start rolling in ⏳. A clever sync program might use that time to load its local cache from disk, or to start syncing another mailbox 🧠.

Here's an example of the first FETCH:

C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE
    RFC822.SIZE)
Note 1: The first FETCH may result in the server's sending a huge volume of data. A smart disconnected client should use message ranges (see also Section 3.2.1.2 of [RFC2683]), so that the user is able to execute a different operation between fetching information for a group of new messages. 🫠

Example 7:

Knowing the new UIDNEXT returned by the server on SELECT or EXAMINE (<uidnext>), the client can split the UID range <lastseenuid+1>:<uidnext> into groups, e.g., 100 messages. After that, the client can issue:

C: A011 UID fetch <lastseenuid+1>:<lastseenuid+100>
    (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)
...
C: A012 UID fetch <lastseenuid+101>:<lastseenuid+200>
    (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)
...
...
C: A0FF UID fetch <lastseenuid+901>:<uidnext>
    (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE)

Heads up though: unless you run a SEARCH, you can't tell how many messages land in each subrange, because UIDs aren't necessarily contiguous 🧩.

Note 2: The client SHOULD ignore any unsolicited EXPUNGE responses received during the first FETCH command. EXPUNGE responses contain message numbers that are useless to a client that doesn't have the message-number-to-UID translation table 🙄.

The second FETCH command gives back zero or more untagged fetch responses. Each one has a matching UID FETCH data item. All messages that didn't get a matching untagged FETCH response MUST be removed from the local cache 🗑️.

For example, if <lastseenuid> was 15000 and the local cache had 3 messages with UIDs 12, 777, and 14999, then after the server sends these responses, the client must remove the message with UID 14999 from its local cache 💀.

S: * 1 FETCH (UID 12 FLAGS (\Seen))
S: * 2 FETCH (UID 777 FLAGS (\Answered \Deleted))

Note 3: If the client isn't interested in flag changes (i.e., the client only wants to know which old messages are still on the server), the second FETCH command can be swapped out for:

tag2 UID SEARCH UID 1:<lastseenuid>

This version creates less traffic 🪶. Still, an implementor should be aware that to build the message number to UID mapping table, the output of the SEARCH command MUST be sorted first. Servers have no obligation to return UIDs in SEARCH responses in any particular order, so don't skip this 🙃.

4.3.2. Searching for "Interesting" Messages. 🔍

This step happens entirely on the client (using info from 4.3.1), entirely on the server, or as a mix of both. What counts as "interesting" is up to the client software and the human 💖. One easy criterion that should probably be in any client is whether a message is "too big" for automatic retrieval, where "too big" is a parameter set in the client's configuration.

Another popular criterion is message age. For example, the client might only want messages from the last week (in this case, <date> would be today's date minus 7 days) 📅:

tag3 UID SEARCH UID <uidset> SINCE <date>

Keep in mind that a date search ignores time and time zone ⏰. The client can skip this search if it asked for INTERNALDATE in <descriptors> back in 4.3.1. If it did, it can run the search locally on its message cache, which is so much faster 🚀.

At this step, the client also decides what info about a particular message to fetch from the server. Even for a message that counts as "too big," the client MAY choose to fetch some part(s) of it 🧩. For example, if the message is a multipart/mixed with a text part and an MPEG attachment, there's no reason not to fetch the text part 📝. Which parts to grab can be based on the BODYSTRUCTURE FETCH response data item, as long as BODYSTRUCTURE was included in <descriptors> back in 4.3.1.

4.3.3. Populating Cache with "Interesting" Messages ✨

Okay so once the client knows which messages are "interesting" 👀, it can start sending FETCH commands for them, or for just the parts it actually needs 🎀.

Fetching a message into the disconnected client's local cache does not mean the human has read it, or even that they ever will 🙈. So the sync program for a disconnected client should always be careful to use the .PEEK variants of the FETCH data items, the ones that implicitly set the \Seen flag 💅.

Once the last descriptor has arrived and the last FETCH command has gone out, the client just needs to process the incoming fetch items and use them to update the local message cache 🧚.

Deadlock alert 🚨: to avoid deadlock problems, the client must give processing of received messages priority over issuing new FETCH commands during this synchronization. This might mean temporarily queuing FETCH requests that can't go out without causing a deadlock 🌸. To make the best use of the "expensive" network connection, the client will almost certainly need to pay careful attention to any flow-control information it can get from the underlying transport connection (usually a TCP connection) 🔥.

Note: The requirement in the previous paragraph might give users an unpleasant experience if it's followed blindly 😭. For example, the user might not want to wait for the client to finish synchronizing before they start doing their own stuff. A smart disconnected client should let the user perform requested operations in between the IMAP commands that are part of the synchronization process 💖. See also Note 1 in Section 4.3.1.

Example 8:

After fetching a message BODYSTRUCTURE, the client discovers a complex MIME message 👑. Then it decides to fetch the MIME headers of the nested MIME messages, plus some body parts.

C: A011 UID fetch 11 (BODYSTRUCTURE)
S: ...
C: A012 UID fetch 11 (BODY[HEADER] BODY[1.MIME] BODY[1.1.MIME]
    BODY[1.2.MIME] BODY[2.MIME] BODY[3.MIME] BODY[4.MIME]
    BODY[5.MIME] BODY[6.MIME] BODY[7.MIME] BODY[8.MIME] BODY[9.MIME]
    BODY[10.MIME] BODY[11.MIME] BODY[12.MIME] BODY[13.MIME]
    BODY[14.MIME] BODY[15.MIME] BODY[16.MIME] BODY[17.MIME]
    BODY[18.MIME] BODY[19.MIME] BODY[20.MIME] BODY[21.MIME])
S: ...
C: A013 UID fetch 11 (BODY[1.1] BODY[1.2])
S: ...
C: A014 UID fetch 11 (BODY[3] BODY[4] BODY[5] BODY[6] BODY[7] BODY[8]
    BODY[9] BODY[10] BODY[11] BODY[13] BODY[14] BODY[15] BODY[16]
    BODY[21])
S: ...

4.3.4. User-Initiated Synchronization 🌈

Once the client has finished the main synchronization process from Sections 4.3.1-4.3.3, the user can optionally ask for extra sync steps while the client is still online 🦋. This is no different from the process described in Sections 4.3.2 and 4.3.3.

Typical examples are:

  1. fetch all messages selected in UI.
  2. fetch all messages marked as \Flagged on the server.

4.4. Special Case: Descriptor-Only Synchronization 🍓

For some mailboxes, fetching the descriptors might be the entire sync step, and that's it ✨. Real-world IMAP experience has shown that a certain class of mailboxes (e.g., "archival" mailboxes) are mostly used for long-term storage of important messages. The human wants them instantly available on demand, but doesn't want them cluttering up the disconnected client's cache at any other time 📦. Messages in this kind of mailbox would only be fetched through explicit actions queued by the local MUA. So the only sync that's useful here is fetching enough descriptor information for the user to pick out messages for later download 💜.

Special mailboxes that get messages from a high-volume, low-priority mailing list might also fall into this category, at least when the human is in a hurry 🏃‍♀️.

4.5. Special Case: Fast New-Only Synchronization ⚡

Sometimes the human is in such a rush that they don't care about changes to old messages, only new ones 💨. In that case, the client can skip the UID FETCH command that gets the flags and UIDs for old messages (1:<lastseenuid>).

4.6. Special Case: Blind FETCH 🙈

Sometimes the human knows (for whatever reason) that they always want to fetch any new messages in a particular mailbox, no questions asked 🫶. In that case the client can grab the messages themselves, instead of just the descriptors, with a command like:

tag1 UID FETCH <lastseenuid+1>:* (FLAGS BODY.PEEK[])
Note that this example ignores the fact that messages can be arbitrarily long 😬. The disconnected client MUST always check for message size before downloading, unless explicitly told otherwise.

A well-behaved client should instead do something like this:

  1. Issue tag1 UID FETCH <lastseenuid+1>:* (FLAGS RFC822.SIZE).
  2. From the message sizes returned in step 1, construct UID set <required_messages>.
  3. Issue tag2 UID FETCH <required_messages> (BODY.PEEK[]).

or

  1. Issue tag1 UID FETCH <lastseenuid+1>:* (FLAGS).
  2. Construct UID set <old_uids> from the responses of step 1.
  3. Issue tag2 SEARCH UID <old_uids> SMALLER <message_limit>. Construct UID set <required_messages> from the result of the SEARCH command.
  4. Issue tag3 UID FETCH <required_messages> (BODY.PEEK[]).

or

  1. Issue tag1 UID FETCH <lastseenuid+1>:* (FLAGS BODY.PEEK[]<0.<length>>), where <length> should be replaced with the maximal message size the client is willing to download.

Note: in response to such a command, the server will only return partial data if the message is longer than <length>. It will return the full message data for any message whose size is smaller than or equal to <length>. In the former case, the client won't be able to extract the full MIME structure of the message from the truncated data, so the client should include BODYSTRUCTURE in the UID FETCH command as well 🧠.

5. Implementation Considerations 🫶

Here are some common implementation pitfalls you should keep in mind when building a disconnected client 💅.

  1. Implementing fake UIDs on the client.

A message scheduled to be uploaded has no UID yet, because the server is the one who picks UIDs 🙃. The client may make up fake UIDs internally so it can reference not-yet-uploaded messages in later operations. (For example, a message could be scheduled to be uploaded, but then get marked as deleted or copied to another mailbox.) Here, the client MUST NOT under any circumstances send these fake UIDs to the server 🚫. Also, client implementers should be reminded that according to [IMAP4], a UID is a 32-bit unsigned integer excluding 0. So both 4294967295 and 2147483648 are valid UIDs, and 0 and -1 are both invalid. Some disconnected mail clients have been known to send negative numbers (e.g., "-1") as message UIDs to servers during synchronization 💀.

Situation 1: The user starts composing a new message, edits it, saves it, keeps editing it, and saves it again ✍️.

A disconnected client may record in its replay log (the log of operations to be replayed on the server during synchronization) the sequence of operations shown below. For this situation, we assume all draft messages are stored in the mailbox called Drafts on an IMAP server. We'll also use these conventions: <old_uid> is the UID of the intermediate version of the draft when it was saved for the first time. This is a fake UID generated on the client. <new_uid> is the UID of the final version of the draft. This is another fake UID generated on the client 🎀.

1) APPEND Drafts (\Seen $MDNSent \Drafts) {<nnn>}
   ...first version of the message follows...

2) APPEND Drafts (\Seen $MDNSent \Drafts) {<mmm>}
   ...final version of the message follows...

3) STORE <old_uid> +FLAGS (\Deleted)

Step 1 is the first attempt to save the draft message, step 2 is the second attempt, and step 3 deletes the first version of the draft saved in step 1 🗑️.

A naive disconnected client might send the command in step 3 without replacing the fake client-generated <old_uid> with the value the server returned in step 1. A server will probably reject this command, which will make the client think the synchronization sequence has failed 😩.

  1. Section 5.1 discusses common implementation errors related to error recovery during playback 🔁.
  2. Don't assume that the disconnected client is the only client the user has. 👀

Situation 2: some clients may use the \Deleted flag as an indicator that the message shouldn't appear in the user's view. Using the \Deleted flag for this purpose is not safe, because other clients (e.g., online clients) might EXPUNGE the mailbox at any time 💔.

  1. Beware of data dependencies between synchronization operations. ⚠️

It might be super tempting for a client writer to optimize the playback log 🤓. Those optimizations might include removing redundant operations (for example, see optimization 2 in Section 5.3), or reordering them.

It isn't always safe to reorder or remove redundant operations during synchronization, because some operations depend on each other (as Situation 3 shows). So, if in doubt, don't do it 🙅‍♀️.

Situation 3: the user copies a message out of a mailbox and then deletes the mailbox 📂.

C: A001 SELECT Old-Mail
S: ...
C: A002 UID COPY 111 ToDo
S: A002 OK [COPYUID 1022843345 111 94] Copy completed
...
C: A015 CLOSE
S: A015 OK Completed
C: A016 DELETE Old-Mail
S: A016 OK Mailbox deletion completed successfully

If the client performs DELETE (tag A016) first and COPY (tag A002) second, then the COPY fails 😱. The message the user so carefully copied into another mailbox is lost forever 💀.

5.1. Error Recovery during Playback 💅

okay so error recovery during sync is literally one of the trickiest parts to get right 😭 below we'll go through some error conditions and suggest ways to handle them 🎀

  1. Lost connection to the server. 📡

The client MUST remember where it is in the playback (replay) log and replay from the interrupted operation (the last command the client sent but the server never acknowledged) the next time it successfully connects to the same server. If the connection dropped while the client was running a non-idempotent IMAP command (see the definition in Section 1), then once it reconnects, it MUST make sure that the interrupted command was really not executed. If it wasn't executed, the client must restart playback from the interrupted command. Otherwise, it starts from the command after that one.

After a reconnect, be careful to properly reapply logical operations that are made of multiple IMAP commands. For example, UID EXPUNGE emulation is needed when the server doesn't support UID EXPUNGE (see Section 4.2.4) 🧠

Once the client notices that the connection to the server was lost, it MUST stop replaying its log. Some existing disconnected clients pop up an error dialog for every single playback operation that fails, and users are NOT happy about it 🙈

  1. Copying or appending messages to a mailbox that doesn't exist. (The server signals this by sending the TRYCREATE response code in the tagged NO response to the APPEND or COPY command.) The user should be told about the situation and offered one of these choices:
    • a) Try to recreate the mailbox.
    • b) Copy or upload the messages to another mailbox.
    • c) Skip the copy or upload.
    • d) Abort replay.
  2. Copying messages from a mailbox that doesn't exist, or renaming a mailbox or getting/changing ACLs [ACL] on a mailbox that doesn't exist.
    • a) Skip the operation.
    • b) Abort replay.
  3. Deleting mailboxes, or deleting/expunging messages that no longer exist.

This actually isn't an error at all, so the client should ignore it ✨

  1. Performing operations on messages that no longer exist.
    • a) Skip the operation.
    • b) Abort replay.

If the client is changing flags on an expunged message, it should quietly ignore the error 🫶

Note 1: several sync operations map to multiple IMAP commands (for example, the "move" described in 4.2.2). The client has to guarantee atomicity for each of these multistep operations. For example, when doing a "move" between two mailboxes on the same server, if the server can't copy the messages, the client MUST NOT try to set the \Deleted flag on the messages being copied, let alone expunge them. However, the client MAY consider the move to have succeeded even if the server couldn't set the \Deleted flag on the copied messages 🦋

Note 2: lots of sync operations have data dependencies between them. When one operation fails, every dependent operation has to fail too. The client should check for this, and it MUST NOT blindly try to perform all the dependent operations (unless the user fixed the original problem first) 💀 For example, a message might be scheduled to be appended to a mailbox on the server, and later the appended message might be copied to another mailbox. If the APPEND operation fails, the client must not try to COPY that failed message later on. (See also Section 5, Situation 3.)

5.2. Quality of Implementation Issues 🌸

here are some quality of implementation tips for disconnected clients. They help you write a client that works correctly, syncs as fast as possible (so the user is happier and saves some money 💸), and keeps server load low 🌈

  1. Don't lose information.

No matter how smart your client is in other areas, if it loses information, users will be SO upset 😭

  1. Don't do work unless explicitly asked. Be flexible. Ask all questions BEFORE starting synchronization, if possible.
  2. Minimize traffic.

The client MUST NOT issue a command if it already received the required information from the server.

The client MUST make use of the UIDPLUS extension if the server supports it.

See also optimization 1 in Section 5.3.

  1. Minimize the number of round-trips.

Round-trips kill performance, especially on high-latency links 🐢 Sections 4.2.2.5 and 5.2 give some advice on cutting down round-trips.

See also optimization 1 in Section 5.3.

5.3. Optimizations 🦋

these are some useful optimizations. A disconnected client that supports the recommendations below will give the user a much more pleasant experience ✨

  1. The initial OK or PREAUTH responses may contain the CAPABILITY response code, as described in Section 7.1 of [IMAP4]. This response code gives the same information the CAPABILITY command returns. A disconnected client that pays attention to this response code can skip sending the CAPABILITY command and save a round-trip 🎉

* Note: some servers report extensions in the CAPABILITY response code that are only relevant in the unauthenticated state, or in all states. These servers usually send another CAPABILITY response code upon successful authentication using the LOGIN or AUTHENTICATE command (that negotiates no security layer; see Section 6.2.2 of [IMAP4]). The CAPABILITY response code sent upon successful LOGIN/AUTHENTICATE might be different from the one in the initial OK response, because extensions that are only relevant for the unauthenticated state won't be advertised, and some extra extensions that are only available in the authenticated and/or selected state will be.

Example 9:

S: * OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS
    AUTH=DIGEST-MD5 AUTH=SRP] imap.example.com ready
C: 2 authenticate DIGEST-MD5
S: 2 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN
    SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND]
    User authenticated (no layer)
  1. An advanced disconnected client may want to optimize its replay log 🧹 For example, there might be some redundant operations (this list isn't complete):
    • a) an EXPUNGE followed by another EXPUNGE or CLOSE;
    • b) changing flags (other than the \Deleted flag) on a message that gets immediately expunged;
    • c) opening and closing the same mailbox.

When optimizing, be careful about data dependencies between commands. For example, if the client is trying to optimize (see case b, above):

tag1 UID STORE <uid1> +FLAGS (\Deleted)
...
tag2 UID STORE <uid1> +FLAGS (\Flagged)
...
tag3 UID COPY <uid1> "Backup"
...
tag4 UID EXPUNGE <uid1>

it can't remove the second UID STORE command, because the message is being copied before it gets expunged 🙅‍♀️

In general, it's a good idea to keep mailboxes open during synchronization (see case c above), if possible. This is easier to do together with optimization 3 described below.

  1. Perform some synchronization steps in parallel, if possible.

Lots of sync steps don't depend on each other, so they can run in parallel. The server is usually more powerful than the client and can do some operations in parallel, so this may speed up the total sync time ⚡

To parallelize, the client has to open more than one connection to the same server. Client writers should NOT forget about the non-trivial cost of establishing a TCP connection and authenticating 🙈 The disconnected client MUST NOT use one connection per mailbox. In most cases, two connections are enough. The disconnected client SHOULD avoid selecting the same mailbox in more than one connection; see Section 3.1.1 of [RFC2683] for more details.

Any mailbox sync MUST start by checking the UIDVALIDITY, as described in Section 4.1 of this document. The client MAY use the STATUS command to check the UID Validity of a non-selected mailbox. This is way better than opening lots of connections to the same server just to sync several mailboxes at once 🫶 As described in Section 5.3.10 of [IMAP4], this SHOULD NOT be used on the selected mailbox.

6. IMAP Extensions That May Help 🧚

these extensions can save traffic and/or cut down the number of round-trips 💖

  1. The use of [UIDPLUS] is discussed in Sections 4.1, 4.2.1, 4.2.2.1 and 4.2.4.
  2. The use of the MULTIAPPEND and LITERAL+ extensions for uploading messages is discussed in Section 4.2.2.5.
  3. Use the CONDSTORE extension (see Section 6.1) for quick flag resynchronization ⭐

6.1. CONDSTORE Extension

okay so if you're building an advanced disconnected mail client, you should use the [CONDSTORE] extension whenever the server supports it 💅 the client must cache the HIGHESTMODSEQ value from the OK response code it gets when a mailbox opens, and keep that cached value updated every time the server sends MODSEQ FETCH data items 🦋

if the client gets an untagged NOMODSEQ OK response instead of HIGHESTMODSEQ, it MUST throw out the last known HIGHESTMODSEQ value in its cache and follow the more general instructions in Section 3 🫶 no cap, that's the fallback path and it's still valid

when the client opens the mailbox to sync, it checks UIDVALIDITY first, the way step d-1 in Section 3 describes 🎀 if the cached UIDVALIDITY matches what the server returned, the client MUST compare the cached HIGHESTMODSEQ with the server's value 🔥 if those match too, the client MUST NOT fetch flags for cached messages, because they haven't changed. literally nothing to redo 😭

if the server's HIGHESTMODSEQ is higher than the cached one, the client MAY use "SEARCH MODSEQ <cached-value>" to find every message whose flags changed since the client was last online with the mailbox open ✨ or it MAY use "FETCH 1:* (FLAGS) (CHANGEDSINCE <cached-value>)" instead. the second option is the lowkey iconic one, because it does the searching for changed messages and the fetching of new info in a single go 🌸

no matter which path you take, the client still has to fetch info about new messages (if the user asked for it) and figure out which messages got expunged 🙈

the presence of the CONDSTORE extension changes step d ("Server-to-client synchronization") in Section 4, so here is the amended version 🧚

d) "Server-to-client synchronization" - For each mailbox that requires synchronization, do the following:

  • 1a) Check the mailbox UIDVALIDITY (see section 4.1 for more details) with SELECT/EXAMINE/STATUS. If the UIDVALIDITY value the server returns is different, the client MUST:
    • empty the local cache of that mailbox;
    • "forget" the cached HIGHESTMODSEQ value for the mailbox;
    • remove any pending "actions" that refer to UIDs in that mailbox (note that this doesn't affect actions performed on client-generated fake UIDs; see Section 5); and
    • skip steps 1b and 2-II.
  • 1b) Check the mailbox HIGHESTMODSEQ. If the cached value is the same as the one the server returns, skip fetching message flags in step 2-II. In other words, the client only has to find out which messages got expunged 💖
  • 2) Fetch the current "descriptors".
    • I) Discover new messages.
    • II) Discover changes to old messages and flags for new messages, using FETCH 1:* (FLAGS) (CHANGEDSINCE <cached-value>) or SEARCH MODSEQ <cached-value>.
    • Discover expunged messages, for example using UID SEARCH 1:<lastseenuid>. (All messages not returned in this command are expunged 👑)
  • 3) Fetch the bodies of any "interesting" messages that the client doesn't already have.

Example 10: this one is the case where the UIDVALIDITY value is the same, but the HIGHESTMODSEQ value changed on the server while the client was offline 🍓

C: A142 SELECT INBOX
S: * 172 EXISTS
S: * 1 RECENT
S: * OK [UNSEEN 12] Message 12 is first unseen
S: * OK [UIDVALIDITY 3857529045] UIDs valid
S: * OK [UIDNEXT 201] Predicted next UID
S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited
S: * OK [HIGHESTMODSEQ 20010715194045007]
S: A142 OK [READ-WRITE] SELECT completed

after that, either:

C: A143 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 20010715194032001)
S: * 2 FETCH (UID 6 MODSEQ (20010715205008000) FLAGS (\Deleted))
S: * 5 FETCH (UID 9 MODSEQ (20010715195517000) FLAGS ($NoJunk
    $AutoJunk $MDNSent))
   ...
S: A143 OK FETCH completed

or:

C: A143 UID SEARCH MODSEQ 20010715194032001 UID 1:20
S: * SEARCH 6 9 11 12 18 19 20 23 (MODSEQ 20010917162500)
S: A143 OK Search complete
C: A144 UID SEARCH 1:20
S: * SEARCH 6 9 ...
S: A144 OK FETCH completed

7. Security Considerations 🔒

okay, the security section is dramatic, but it's also really reassuring 😌 it is believed that this document doesn't raise any new security concerns beyond the ones already in the base [IMAP4] protocol, and those issues are discussed in [IMAP4] itself 🥺 extensions mentioned in this document have their own security notes, especially [UIDPLUS], [LITERAL+], [CONDSTORE], [MULTIAPPEND], and [UNSELECT] ⚠️ so go read those too

and please, implementers, test thoroughly. like, seriously 💀🧪

8. References 📚

8.1. Normative References

  • [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [IMAP4] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1", RFC 3501, March 2003.
  • [UIDPLUS] Crispin, M., "Internet Message Access Protocol (IMAP) - UIDPLUS extension", RFC 4315, December 2005.
  • [LITERAL+] Myers, J., "IMAP4 non-synchronizing literals", RFC 2088, January 1997.
  • [CONDSTORE] Melnikov, A. and S. Hole, "IMAP Extension for Conditional STORE Operation or Quick Flag Changes Resynchronization", RFC 4551, June 2006.
  • [MULTIAPPEND] Crispin, M., "Internet Message Access Protocol (IMAP) - MULTIAPPEND Extension", RFC 3502, March 2003.
  • [UNSELECT] Melnikov, A., "Internet Message Access Protocol (IMAP) UNSELECT command", RFC 3691, February 2004.
  • [RFC2683] Leiba, B., "IMAP4 Implementation Recommendations", RFC 2683, September 1999.

8.2. Informative References

  • [ACL] Melnikov, A., "IMAP4 Access Control List (ACL) Extension", RFC 4314, December 2005.
  • [IMAP-MODEL] Crispin, M., "Distributed Electronic Mail Models in IMAP4", RFC 1733, December 1994.

9. Acknowledgements 🫶

this document is based on version 01 of the text Rob Austein wrote in November 1994 🌈

the editor appreciates the comments Mark Crispin posted to the IMAP mailing list, plus the comments, corrections, and ideas from Grant Baillie, Cyrus Daboo, John G. Myers, Chris Newman, and Timo Sirainen 💜

the editor would also like to thank the developers of Netscape Messenger and Mozilla mail clients. they gave examples of disconnected mail clients that became the base for a lot of the recommendations in this document 👑

Editor's Address 📬

Alexey Melnikov
Isode Limited
5 Castle Business Village
36 Station Road
Hampton, Middlesex
TW12 2BX
United Kingdom

Phone: +44 77 53759732
EMail: alexey.melnikov@isode.com

Copyright (C) The Internet Society (2006).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🧾

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document, or the extent to which any license under such rights might or might not be available. It also doesn't represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.

Acknowledgement 🎀

Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA).

📦 where this page comes from