RFC 8474

IMAP Extension for Object Identifiers

📅 published September 2018, from ietf

RFC 8474: IMAP Extension for Object Identifiers 🦋

Okay bestie, this one is RFC 8474, IMAP Extension for Object Identifiers 💖 It came from the IETF (Internet Engineering Task Force) and was written by B. Gondwana, Ed., from FastMail, in September 2018. It's a Standards Track document ✨ Its ISSN is 2070-1721. It updates RFC 3501 (IMAP4rev1) and doesn't obsolete anything, so think of it as a glow-up on top of the original IMAP spec 🌸

Abstract 🫶

okay so this document updates RFC 3501 (IMAP4rev1) by adding persistent identifiers to mailboxes and messages. The goal is that clients can reuse their cached data more efficiently when stuff has moved to a new place on the server 🧚

Status of This Memo 📜

This is an Internet Standards Track document, so it's a real standard and not just a vibe 💅

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It went through public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check Section 2 of RFC 7841.

Info about the current status of this document, any errata, and how to give feedback on it lives at https://www.rfc-editor.org/info/rfc8474 🔗

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. 💜

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they're in effect on the date of publication. Please actually read these, bestie, because they explain your rights and restrictions for this document. Code Components pulled out of this document must include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They come without warranty, as the Simplified BSD License describes 🎀

Table of Contents 🗂️

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
2.  Conventions Used in This Document . . . . . . . . . . . . . .   3
3.  CAPABILITY Identification . . . . . . . . . . . . . . . . . .   3
4.  MAILBOXID Object Identifier . . . . . . . . . . . . . . . . .   3
  4.1.  New Response Code for CREATE  . . . . . . . . . . . . . .   4
  4.2.  New OK Untagged Response for SELECT and EXAMINE . . . . .   4
  4.3.  New Attribute for STATUS  . . . . . . . . . . . . . . . .   5
5.  EMAILID Object Identifier and THREADID Correlator . . . . . .   6
  5.1.  EMAILID Identifier for Identical Messages . . . . . . . .   6
  5.2.  THREADID Identifier for Related Messages  . . . . . . . .   6
  5.3.  New Message Data Items in FETCH and UID FETCH Commands  .   7
6.  New Filters on SEARCH Command . . . . . . . . . . . . . . . .   9
7.  Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . .   9
8.  Implementation Considerations . . . . . . . . . . . . . . . .  10
  8.1.  Assigning Object Identifiers  . . . . . . . . . . . . . .  10
  8.2.  Interaction with Special Cases  . . . . . . . . . . . . .  11
  8.3.  Client Usage  . . . . . . . . . . . . . . . . . . . . . .  11
  8.4.  Advice to Client Implementers . . . . . . . . . . . . . .  12
9.  Future Considerations . . . . . . . . . . . . . . . . . . . .  12
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  12
11. Security Considerations . . . . . . . . . . . . . . . . . . .  13
12. References  . . . . . . . . . . . . . . . . . . . . . . . . .  13
  12.1.  Normative References . . . . . . . . . . . . . . . . . .  13
  12.2.  Informative References . . . . . . . . . . . . . . . . .  14
Appendix A.  Ideas for Implementing Object Identifiers  . . . . .  15
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . .  15
Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  16

1. Introduction 🎀

IMAP stores are often used by lots of clients at once. Each client can cache data from the server so it doesn't have to redownload stuff 🍓 [RFC3501] says a mailbox can be uniquely referenced by its name and UIDVALIDITY. A message inside that mailbox can be uniquely referenced by its mailbox (name + UIDVALIDITY) and its unique identifier (UID). The triple of mailbox name, UIDVALIDITY, and UID is guaranteed to be immutable, which is the whole point 💅

[RFC4315] defines a COPYUID response. This lets a client that copies messages know how the UIDs in the source and destination mailboxes map to each other, so it can update its local cache 🦋

If a client successfully renames a mailbox, that client knows the same messages now exist under the new mailbox name that used to live under the old one 🧚

Here's the tea ☕ The client that copies (or moves [RFC6851]) messages, or renames a mailbox, can update its own local cache. But any other client connected to the same store can't know for sure that the messages are identical, so it has to redownload everything 😭

This extension adds new properties to messages (EMAILID) and mailboxes (MAILBOXID). With these, a client can quickly figure out which messages or mailboxes were renamed by another client ✨

This extension also adds an optional thread identifier (THREADID) to messages. The server can use it to show which messages it has identified as related 🧵 A server that doesn't implement threading will return NIL to all requests for THREADID. Lowkey a very clean fallback 🫶

2. Conventions Used in This Document 📝

In the examples, "C:" marks lines sent by a client connected to a server, and "S:" marks lines sent by the server to the client 👀

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. So those capital words are the official ones, and the lowercase ones are just regular words 🙈

3. CAPABILITY Identification 🧚

IMAP servers that support this extension MUST include "OBJECTID" in the response list to the CAPABILITY command. No cap, this is how clients find out the extension exists 💖

4. MAILBOXID Object Identifier 💜

The MAILBOXID is a server-allocated unique identifier for each mailbox 🌈

The server MUST return the same MAILBOXID for a mailbox with the same name and UIDVALIDITY. Consistency is everything bestie 💅

The server MUST NOT report the same MAILBOXID for two mailboxes at the same time. Two mailboxes sharing an ID at once is a no-go 🙅

The server MUST NOT reuse the same MAILBOXID for a mailbox that doesn't obey all the invariants that [RFC3501] defines for a mailbox that doesn't change name or UIDVALIDITY. Recycling IDs is not it 🗑️

The server MUST keep the same MAILBOXID for the source and destination when it renames a mailbox in a way that keeps the same messages. BUT see [RFC3501] for the special case about renaming INBOX, which is treated as creating a new mailbox and moving the messages. INBOX is its own whole personality 👑

4.1. New Response Code for CREATE 🌸

This document extends the CREATE command so it can return a MAILBOXID response code when a mailbox is created successfully 🎀

A server that advertises the OBJECTID capability MUST include the MAILBOXID response code in the tagged OK response to all successful CREATE commands. Every successful create gets its ID, no exceptions ✨

Syntax:

"MAILBOXID" SP "(" objectid ")"

This is the response code in the tagged OK response for a successful CREATE command.

Example:

  C: 3 create foo
  S: 3 OK [MAILBOXID (F2212ea87-6097-4256-9d51-71338625)] Completed
  C: 4 create bar
  S: 4 OK [MAILBOXID (F6352ae03-b7f5-463c-896f-d8b48ee3)] Completed
  C: 5 create foo
  S: 5 NO Mailbox already exists

4.2. New OK Untagged Response for SELECT and EXAMINE 🦋

This document adds a new untagged response code to the SELECT and EXAMINE commands 🧚

A server that advertises the OBJECTID capability MUST return an untagged OK response with the MAILBOXID response code on all successful SELECT and EXAMINE commands. So SELECT and EXAMINE both spill the ID tea 🍵

Syntax:

OK SP "[" "MAILBOXID" SP "(" objectid ")" "]" SP text

This is the untagged OK response to SELECT or EXAMINE.

Example:

     C: 27 select "foo"
     [...]
     S: * OK [MAILBOXID (F2212ea87-6097-4256-9d51-71338625)] Ok
     [...]
     S: 27 OK [READ-WRITE] Completed

4.3. New Attribute for STATUS 💫

This document adds the MAILBOXID attribute to the STATUS command, using the extended syntax defined in [RFC4466] 🌟

A server that advertises the OBJECTID capability MUST support the MAILBOXID status attribute. It's not optional, it's a must-have 💅

Syntax for the attribute in the STATUS command:

"MAILBOXID"

Syntax for the response item in the STATUS response:

"MAILBOXID" SP "(" objectid ")"

The response item in the STATUS response contains the ObjectID that the server assigned to this mailbox 🎀

Example:

 C: 6 status foo (mailboxid)
 S: * STATUS foo (MAILBOXID (F2212ea87-6097-4256-9d51-71338625))
 S: 6 OK Completed
 C: 7 status bar (mailboxid)
 S: * STATUS bar (MAILBOXID (F6352ae03-b7f5-463c-896f-d8b48ee3))
 S: 7 OK Completed
 C: 8 rename foo renamed
 S: * OK rename foo renamed
 S: 8 OK Completed
 C: 9 status renamed (mailboxid)
 S: * STATUS renamed (MAILBOXID (F2212ea87-6097-4256-9d51-71338625))
 S: 9 OK Completed
 C: 10 status bar (mailboxid)
 S: * STATUS bar (MAILBOXID (F6352ae03-b7f5-463c-896f-d8b48ee3))
 S: 10 OK Completed

When the LIST-STATUS IMAP capability defined in [RFC5819] is also available, you can combine the STATUS command with the LIST command. Multitasking era 🧠

Example:

 C: 11 list "" "*" return (status (mailboxid))
 S: * LIST (\HasNoChildren) "." INBOX
 S: * STATUS INBOX (MAILBOXID (Ff8e3ead4-9389-4aff-adb1-d8d89efd8cbf))
 S: * LIST (\HasNoChildren) "." bar
 S: * STATUS bar (MAILBOXID (F6352ae03-b7f5-463c-896f-d8b48ee3))
 S: * LIST (\HasNoChildren) "." renamed
 S: * STATUS renamed (MAILBOXID (F2212ea87-6097-4256-9d51-71338625))
 S: 11 OK Completed (0.001 secs 3 calls)

5. EMAILID Object Identifier and THREADID Correlator 🌈

5.1. EMAILID Identifier for Identical Messages 💌

The EMAILID data item is an ObjectID that uniquely identifies the content of a single message. Anything that has to stay immutable on a {name, uidvalidity, uid} triple must also be the same between messages with the same EMAILID 🧚

The server MUST return the same EMAILID for the same triple, which means EMAILID is immutable. Stable, reliable, iconic 👑

The server MUST return the same EMAILID as the source message for the matching destination message in the COPYUID pairing after a COPY or MOVE command [RFC6851]. The copy is the same message, so it keeps the same ID 🦋

The server MAY assign the same EMAILID as an existing message upon APPEND. For example, that could happen if the server notices the new message has exactly identical content to an existing one 🫧

NOTE: EMAILID only identifies the immutable content of the message. In particular, it's possible for different messages with the same EMAILID to have different keywords. This document doesn't specify a way to STORE by EMAILID. So don't go assuming it does, bestie 🙈

okay so THREADID is the ObjectID that says "these messages belong together in one convo" 🧵 bestie, this is the thread grouping energy we all need 💅

The server works out THREADID based on some mix of References, In-Reply-To, and Subject, but how it does that is up to the server. [RFC5256] has some algorithms you could use, but this spec doesn't force any one strategy. so it's giving choose your own adventure 🦋

The server MUST return the same THREADID for all messages with the same EMAILID. no exceptions, that's the rule 🔥

The server SHOULD return the same THREADID for related messages, even if they're in different mailboxes. like, messages that would show up in the same thread if they lived in one mailbox SHOULD have the same THREADID, even across mailboxes 🌈

The server MUST NOT change the THREADID of a message once it's been reported. once it's told you, it's locked in 🔒💖

THREADID is OPTIONAL 🎀 if the server doesn't support it, or can't work out how messages relate, it MUST return NIL to all FETCH responses for the THREADID data item, and a SEARCH for THREADID MUST NOT match any messages. so no fake matches, ever 🙈

The server MUST NOT use the same ObjectID value for both EMAILIDs and THREADIDs. if they happen to be stored with the same value internally, the server can generate prefixed values (like the M and T prefixes in the examples below) to avoid clashes 🧚

5.3. New Message Data Items in FETCH and UID FETCH Commands 💜

okay this section is where we get the new FETCH items 🍓

This document defines two FETCH items:

Syntax: "EMAILID"

The EMAILID message data item causes the server to return EMAILID FETCH response data items.

Syntax: "THREADID"

The THREADID message data item causes the server to return THREADID FETCH response data items.

This document defines the following responses:

Syntax: "EMAILID" SP "(" objectid ")"

The EMAILID response data item contains the server-assigned ObjectID for each message.

Syntax: "THREADID" SP "(" objectid ")"

The THREADID response data item contains the server-assigned ObjectID for the set of related messages to which this message belongs.

Syntax: "THREADID" SP nil

The NIL value is returned for the THREADID response data item when the server mailbox does not support THREADID calculation.

Example:

C: 5 append inbox "20-Mar-2018 03:07:37 +1100" {733}
[...]
Subject: Message A
Message-ID: <fake.1521475657.54797@example.com>
[...]
S: 5 OK [APPENDUID 1521475658 1] Completed

C: 11 append inbox "20-Mar-2018 03:07:37 +1100" {793}
[...]
Subject: Re: Message A
Message-ID: <fake.1521475657.21213@example.org>
References: <fake.1521475657.54797@example.com>
[...]
S: 11 OK [APPENDUID 1521475658 2] Completed

C: 17 append inbox "20-Mar-2018 03:07:37 +1100" {736}
[...]
Subject: Message C
Message-ID: <fake.1521475657.60280@example.com>
[...]
S: 17 OK [APPENDUID 1521475658 3] Completed

C: 22 fetch 1:* (emailid threadid)
S: * 1 FETCH (EMAILID (M6d99ac3275bb4e) THREADID (T64b478a75b7ea9))
S: * 2 FETCH (EMAILID (M288836c4c7a762) THREADID (T64b478a75b7ea9))
S: * 3 FETCH (EMAILID (M5fdc09b49ea703) THREADID (T11863d02dd95b5))
S: 22 OK Completed (0.000 sec)

C: 23 move 2 foo
S: * OK [COPYUID 1521475659 2 1] Completed
S: * 2 EXPUNGE
S: 23 OK Completed

C: 24 fetch 1:* (emailid threadid)
S: * 1 FETCH (EMAILID (M6d99ac3275bb4e) THREADID (T64b478a75b7ea9))
S: * 2 FETCH (EMAILID (M5fdc09b49ea703) THREADID (T11863d02dd95b5))
S: 24 OK Completed (0.000 sec)
C: 25 select "foo"

C: 25 select "foo"
[...]
S: 25 OK [READ-WRITE] Completed
C: 26 fetch 1:* (emailid threadid)
S: * 1 FETCH (EMAILID (M288836c4c7a762) THREADID (T64b478a75b7ea9))
S: 26 OK Completed (0.000 sec)

Example: (no THREADID support)

          C: 26 fetch 1:* (emailid threadid)
          S: * 1 FETCH (EMAILID (M00000001) THREADID NIL)
          S: * 2 FETCH (EMAILID (M00000002) THREADID NIL)
          S: 26 OK Completed (0.000 sec)

6. New Filters on SEARCH Command 🔍

this section adds two new filters to SEARCH, EMAILID and THREADID 🌸

This document defines the filters EMAILID and THREADID on the SEARCH command.

Syntax: "EMAILID" SP objectid

Messages whose EMAILID is exactly the specified ObjectID.

Syntax: "THREADID" SP objectid

Messages whose THREADID is exactly the specified ObjectID.

Example: (as if run before the MOVE shown above when the mailbox had three messages)

          C: 27 search emailid M6d99ac3275bb4e
          S: * SEARCH 1
          S: 27 OK Completed (1 msgs in 0.000 secs)
          C: 28 search threadid T64b478a75b7ea9
          S: * SEARCH 1 2
          S: 28 OK Completed (2 msgs in 0.000 secs)

7. Formal Syntax 📐

this is the grammar part, so the exact characters matter a LOT 🤓

The following syntax specification uses the Augmented Backus-Naur Form (ABNF) [RFC5234] notation. Elements not defined here can be found in the formal syntax of the ABNF [RFC5234], IMAP [RFC3501], and IMAP ABNF extensions [RFC4466] specifications.

Except as noted otherwise, all alphabetic characters are case insensitive. The use of uppercase or lowercase characters to define token strings is for editorial clarity only. Implementations MUST accept these strings in a case-insensitive fashion.

Please note specifically that ObjectID values are case sensitive.

   capability =/ "OBJECTID"

   fetch-att =/ "EMAILID" / "THREADID"

   fetch-emailid-resp = "EMAILID" SP "(" objectid ")"
           ; follows tagged-ext production from [RFC4466]

   fetch-threadid-resp = "THREADID" SP ( "(" objectid ")" / nil )
           ; follows tagged-ext production from [RFC4466]

   msg-att-static =/ fetch-emailid-resp / fetch-threadid-resp

   objectid = 1*255(ALPHA / DIGIT / "_" / "-")
           ; characters in object identifiers are case
           ; significant

   resp-text-code =/ "MAILBOXID" SP "(" objectid ")"
           ; incorporated before the expansion rule of
           ;  atom [SP 1*<any TEXT-CHAR except "]">]
           ; that appears in [RFC3501]

   search-key =/ "EMAILID" SP objectid / "THREADID" SP objectid

   status-att =/ "MAILBOXID"

   status-att-val =/ "MAILBOXID" SP "(" objectid ")"
           ; follows tagged-ext production from [RFC4466]

8. Implementation Considerations 🛠️

8.1. Assigning Object Identifiers 🪪

All ObjectID values are allocated by the server.

In the interest of reducing the possibilities of encoding mistakes, ObjectIDs are restricted to a safe subset of possible byte values; in order to allow clients to allocate storage, they are restricted in length.

An ObjectID is a string of 1 to 255 characters from the following set of 64 codepoints: a-z, A-Z, 0-9, _, -. These characters are safe to use in almost any context (e.g., filesystems, URIs, IMAP atoms). These are the same characters defined as base64url in [RFC4648].

For maximum safety, servers should also follow defensive allocation strategies to avoid creating risks where glob completion or data type detection may be present (e.g., on filesystems or in spreadsheets). In particular, it is wise to avoid:

  • IDs starting with a dash
  • IDs starting with digits
  • IDs that contain only digits
  • IDs that differ only by ASCII case (for example, A vs. a)
  • the specific sequence of three characters NIL in any case (because this sequence can be confused with the IMAP protocol expression of the null value)

A good solution to these issues is to prefix every ID with a single alphabetical character.

8.2. Interaction with Special Cases 🎭

The case of RENAME INBOX may need special handling because it has special behavior, as defined in [RFC3501], Section 6.3.5.

It is advisable (though not required) to have MAILBOXID be globally unique, but it is only required to be unique within messages offered to a single client login to a single server hostname. For example, a proxy that aggregates multiple independent servers MUST NOT advertise the OBJECTID capability unless it can guarantee that different objects will never use the same identifiers, even if backend object identifiers collide.

8.3. Client Usage 📱

Servers that implement both RFC 6154 and this specification should optimize their execution of commands like UID SEARCH OR EMAILID 1234 EMAILID 4321.

Clients can assume that searching the all-mail mailbox using OR/ EMAILID or OR/THREADID is a fast way to find messages again if some other client has moved them out of the mailbox where they were previously seen.

Clients that cache data offline should fetch the EMAILID of all new messages to avoid redownloading already-cached message details.

Clients should fetch the MAILBOXID for any new mailboxes before discarding cache data for any mailbox that is no longer present on the server so that they can detect renames and avoid redownloading data.

8.4. Advice to Client Implementers 🧑‍💻

In cases of server failure and disaster recovery, or misbehaving servers, it is possible that a client will be sent invalid information, e.g., identical ObjectIDs or ObjectIDs that have changed where they MUST NOT change according to this document.

In a case where a client detects inconsistent ObjectID responses from a server, it SHOULD fall back to relying on the guarantees of RFC 3501. For simplicity, a client MAY instead choose to discard its entire cache and resync all state from the server.

Client authors protecting against server misbehavior MUST ensure that their design cannot get into an infinite loop of discarding cache and fetching the same data repeatedly without user interaction.

9. Future Considerations 🔮

This extension is intentionally defined to be compatible with the data model in [JMAP-MAIL].

A future extension could be proposed to give a way to SELECT a mailbox by MAILBOXID rather than name.

A future extension to [RFC5228] could allow fileinto by MAILBOXID rather than name.

An extension to allow fetching message content directly via EMAILID and message listings by THREADID could be proposed.

10. IANA Considerations 🗂️

IANA has added "OBJECTID" to the "IMAP Capabilities" registry located at <https://www.iana.org/assignments/imap-capabilities> with a reference to this document.

IANA has added "MAILBOXID" to the "IMAP Response Codes" registry located at <https://www.iana.org/assignments/imap-response-codes> with a reference to this document.

11. Security Considerations 🔐

it is STRONGLY advised that servers generate ObjectIDs that are safe to use as filesystem names and unlikely to be autodetected as numbers. see implementation considerations 🚨

If a digest is used for ID generation, it must have a collision-resistant property, so server implementations are advised to monitor current security research and choose secure digests. As the IDs are generated by the server, it will be possible to migrate to a new hash by just using the new algorithm when creating new IDs. This is particularly true if a prefix is used on each ID, which can be changed when the algorithm changes.

The use of a digest for ID generation may be used as proof that a particular sequence of bytes was seen by the server. However, this is only a risk if IDs are leaked to clients who don't have permission to fetch the data directly. Servers that are expected to handle highly sensitive data should consider this when choosing how to create IDs. 😳

See also the security considerations in [RFC3501], Section 11.

12. References 📚

12.1. Normative References ✨

These are the must-read docs, bestie 📚💅

12.2. Informative References 🌸

Appendix A. Ideas for Implementing Object Identifiers 🧚

Okay so this appendix is basically a brainstorm board for how a server could actually make these IDs. Not required stuff, just ideas bestie 💭

Ideas for calculating MAILBOXID:

  • Universally Unique Identifier (UUID) [RFC4122]
  • Server-assigned sequence number (guaranteed not to be reused)

Ideas for implementing EMAILID:

  • Digest of message content (RFC822 bytes) -- expensive unless cached 💸
  • UUID [RFC4122]
  • Server-assigned sequence number (guaranteed not to be reused)

Ideas for implementing THREADID:

  • Derive from EMAILID of first seen message in the thread.
  • UUID [RFC4122]
  • Server-assigned sequence number (guaranteed not to be reused)

Threading lowkey needs a good lookup system. There is a need to index and look up reference/in-reply-to data at message creation so the server can efficiently find matching messages for threading. Threading may be either across mailboxes or within each mailbox only. The server has significant leeway here, so it's giving freedom 🫶

Acknowledgments 💜

The author would like to thank the EXTRA working group at IETF for feedback and advice, in particular Arnt Gulbrandsen, Brandon Long, Chris Newman, and Josef Sipek. Iconic squad 👑

This document drew inspiration from the Gmail X-GM-THRID and X-GM-MSGID implementations as currently defined at <https://developers.google.com/gmail/imap/imap-extensions>, as well as the X-GUID implementation in the Dovecot server. Inspo is everything ✨

Author's Address 🎀

Bron Gondwana (editor) FastMail Level 2, 114 William St Melbourne VIC 3000 Australia

Email: brong@fastmailteam.com URI: https://www.fastmail.com

📦 where this page comes from