RFC 7377

IMAP4 Multimailbox SEARCH Extension

📅 published October 2014, from ietf

Hey, Meet RFC 7377 💖

RFC 7377 is the IMAP4 Multimailbox SEARCH Extension 🎀 It's from the Internet Engineering Task Force (IETF) 🦋. The authors are B. Leiba of Huawei Technologies and A. Melnikov of Isode Limited. It's dated October 2014, and its category is Standards Track. Its ISSN is 2070-1721. This doc obsoletes RFC 6237 and updates RFC 4466 ✨

Abstract 🌸

Okay so the IMAP4 spec only lets you search the mailbox that's currently selected 🙄 If you want to search a bunch of mailboxes, your client has to do SELECT and SEARCH over and over, waiting for each one to finish before starting the next. Literally so much waiting 😭

This extension lets a client search multiple mailboxes with one command 🎀 That cuts down on all those round trips and doesn't mess with whatever mailbox is currently selected. ESEARCH responses also get MAILBOX, UIDVALIDITY, and TAG fields, so a client can pipeline searches if it wants to 🫶 This document updates RFC 4466 and obsoletes RFC 6237.

Status of This Memo 🧚

This is an Internet Standards Track document, bestie.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 5741.

Info about the current status of this doc, any errata, and how to give feedback is at http://www.rfc-editor.org/info/rfc7377 💅

Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License 🫶

Table of Contents 🍓

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
1.1.  Conventions Used in This Document . . . . . . . . . . . . .   3
2.  New ESEARCH Command . . . . . . . . . . . . . . . . . . . . .   3
2.1.  The ESEARCH Response  . . . . . . . . . . . . . . . . . . .   4
2.2.  Source Options: Specifying Mailboxes to Search  . . . . . .   5
2.3.  Strictness in Search Matches  . . . . . . . . . . . . . . .   7
2.4.  Server-Side Limitations on Search Volume  . . . . . . . . .   7
3.  Examples  . . . . . . . . . . . . . . . . . . . . . . . . . .   7
4.  Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . .   8
5.  Security Considerations . . . . . . . . . . . . . . . . . . .   9
6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  10
7.  Changes since RFC 6237  . . . . . . . . . . . . . . . . . . .  10
8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  10
8.1.  Normative References  . . . . . . . . . . . . . . . . . . .  10
8.2.  Informative References  . . . . . . . . . . . . . . . . . .  11
Appendix A.  Acknowledgments  . . . . . . . . . . . . . . . . . .  11
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  11

1. Introduction ✨

Okay, the IMAP4 spec lets you search only the selected mailbox. Users often want to search several mailboxes, and a client that wants to support this has to issue a series of SELECT and SEARCH commands, waiting for each one to complete before moving on to the next. These commands can't be pipelined, because the server might run them in parallel, and then the untagged SEARCH responses couldn't be told apart from each other 🙈

This extension lets a client search multiple mailboxes with one command, and it adds MAILBOX and TAG fields to the ESEARCH response. Here's why that's iconic 👑

  • A single command limits the number of round trips needed to search a set of mailboxes.
  • A single command removes the need to wait for one search to finish before starting the next.
  • A single command lets the server optimize the search if it can.
  • A command that doesn't depend on the selected mailbox removes the need to disrupt the selection state or to open another IMAP connection.
  • The MAILBOX, UIDVALIDITY, and TAG fields in the responses let a client tell which responses go with which search (and which mailbox). A client can safely pipeline these search commands without any danger of confusion. Adding the MAILBOX and UIDVALIDITY fields updates the search-correlator item defined in [RFC4466] 🦋

This extension was previously published in an Experimental RFC. There's now implementation experience that gives confidence in the protocol, so this document puts the extension on the Standards Track, with some minor updates informed by that experience. A brief summary of the changes is in Section 7 💖

1.1. Conventions Used in This Document 🎀

In examples, "C:" marks lines sent by a client connected to a server, and "S:" marks lines sent by the server to the client.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 🧚

2. New ESEARCH Command 💅

Here's the syntax for the new command:

Arguments:  OPTIONAL source options
            OPTIONAL result options
            OPTIONAL charset specification (see [RFC2978])
            searching criteria (one or more)

Responses:  REQUIRED untagged response: ESEARCH
Result:     OK -- search completed
            NO -- error: cannot search that charset or criteria
            BAD -- command unknown or arguments invalid

This section defines a new ESEARCH command. It works a lot like the UID SEARCH command described in Section 2.6.1 of [RFC4466] (which was first described in Section 6.4.4 of [RFC3501] and extended by [RFC4731]) 🌈

The ESEARCH command goes further with searching by allowing optional source and result options. This document doesn't define any new result options (see Section 3.1 of [RFC4731]). A server that supports this extension includes "MULTISEARCH" in its IMAP capability string ✨

Because there's been some confusion about this, it's worth pointing out that with ESEARCH, just like any SEARCH or UID SEARCH command, it MUST NOT be considered an error if the search terms include a range of message numbers that extends (or even starts) beyond the end of the mailbox. For example, a client might want a rolling window through the search results, like this:

C: tag1 UID ESEARCH FROM "frobozz" 1:100

... followed later by this:

C: tag1 UID ESEARCH FROM "frobozz" 101:200

... and so on.

This tells the server to match only the first hundred messages in the mailbox the first time, the second hundred the second time, and so on. In fact, it might allow the server to optimize the search a lot 🔥 In the example above, whether the mailbox has 50, 150, or 250 messages, neither of the search commands shown will cause an error. It's up to the client to know when to stop moving its search window 👑

2.1. The ESEARCH Response 🦋

In response to an ESEARCH command, the server MUST return ESEARCH responses [RFC4731] (that is, not SEARCH responses). Because message numbers aren't useful for mailboxes that aren't selected, the responses MUST contain information about UIDs, not message numbers. This is true even if the source options say that only the selected mailbox should be searched 🙈

If there's a source option and no result option, that implies the "ALL" result option (see Section 3.1 of [RFC4731]). Note that this is not the same as the result from the SEARCH command described in the IMAP base protocol [RFC3501] 💖

Source options describe which mailboxes must be searched for messages. An ESEARCH command with source options does not affect which mailbox, if any, is currently selected, no matter which mailboxes get searched 🫶

For each mailbox that satisfies the source options, a single ESEARCH response MUST be returned if any messages in that mailbox match the search criteria. An ESEARCH response MUST NOT be returned for mailboxes that contain no matching messages. This is true even when result options such as MIN, MAX, and COUNT are specified (see Section 3.1 of [RFC4731]). The values returned (lowest UID matched, highest UID matched, and number of messages matched, respectively) apply to the mailbox reported in that ESEARCH response 🍓

It's possible for an ESEARCH command to return no untagged responses at all (no ESEARCH responses), if there are no matches to the search in any of the mailboxes that satisfy the source options. Clients can detect this situation by getting the tagged OK response without having received any matching untagged ESEARCH responses 💀

Each ESEARCH response MUST contain the MAILBOX, TAG, and UIDVALIDITY correlators. Correlators let clients issue several ESEARCH commands at once (pipelined) 🧚 If the SEARCHRES [RFC5182] extension is used in an ESEARCH command, that ESEARCH command MUST be executed by the server after all previous SEARCH/ESEARCH commands have completed, and before any subsequent SEARCH/ESEARCH commands are executed. The server MAY perform consecutive ESEARCH commands in parallel, as long as none of them use the SEARCHRES extension 🌸

okay so the source options are where you tell the server which mailboxes to search 📬✨ if they're there, they MUST contain a mailbox specifier from the IMAP NOTIFY extension [RFC5465], Section 6 (using the "filter-mailboxes" ABNF item) 💖 it's the same vibe with two plot twists:

  1. The "selected-delayed" specifier is not valid here 🙅‍♀️
  2. A "subtree-one" specifier gets added to the squad 🎀 "subtree" searches the mailbox AND every selectable mailbox under it, all the way down an indefinitely deep hierarchy 🌳 "subtree-one" is the chill version: it searches the mailbox plus all selectable child mailboxes, but only one hierarchy level down 🍓

bestie, this part is important 🚨 if "subtree" is specified, the server MUST defend against loops in the hierarchy (like the ones you can get from recursive file-system links inside the message store) 🔁 the server SHOULD do this by keeping track of the mailboxes it already searched and ending the traversal when it hits a repeat 🧠 if it can't do that, it MAY instead limit how deep the hierarchy goes 🪜

if the source options aren't there at all, "selected" is assumed, which means only the currently selected mailbox gets searched 🎯

the "personal" source option is so convenient for searching all of the current user's mailboxes 💅 but no cap, there's no way to use wildcard characters to search everything, because the "mailboxes" source option does NOT do wildcard expansion 🙈

now the session-state rules, which are lowkey strict 😤 if the source options include (or default to) "selected", the IMAP session MUST be in "selected" state 💜 if the source options name other mailboxes and not "selected", then the session MUST be in either "selected" or "authenticated" state 🫶 if the session isn't in a correct state, the ESEARCH command MUST return a "BAD" result 💀

duplicates are a little drama 🎭 the client SHOULD NOT provide source options that end up including the same mailbox more than once 🚫 a server can totally remove duplicates before processing, but it MAY return "BAD" to an ESEARCH command that has duplicate source mailboxes 😬

if the server supports the SEARCHRES [RFC5182] extension, the "SAVE" result option only works if "selected" is specified or defaulted to as the sole mailbox to be searched 💾 if any other source option shows up, the ESEARCH command MUST return a "BAD" result 🙅‍♀️

if the server supports the CONTEXT=SEARCH and/or CONTEXT=SORT extension [RFC5267], these extra rules apply 🧩

  • The CONTEXT return option (Section 4.2 of [RFC5267]) can be used with an ESEARCH command ✨
  • If the UPDATE return option is used (Section 4.3 of [RFC5267]), it MUST apply only to the currently selected mailbox 🎯 and if UPDATE is used with no mailbox currently selected, the ESEARCH command MUST return a "BAD" result 💅
  • The PARTIAL search return option (Section 4.4 of [RFC5267]) can be used, and it applies to each mailbox the ESEARCH command searches 🍰

and last but definitely not least, the Access Control List (ACL) [RFC4314] extension 🔐 if the server supports it, the logged-in user needs the "r" right for each mailbox she wants to search 💖 any mailboxes that aren't explicitly named (like the ones reached through "personal" or "subtree") also need the "l" right 🦋 mailboxes matching the source options where the user lacks enough rights MUST be ignored by ESEARCH processing 🙈 and ESEARCH responses MUST NOT be returned for those mailboxes 🚫 no peeking, babe! 👑

2.3. Strictness in Search Matches

okay so the base IMAP SEARCH command (Section 6.4.4. of [RFC3501]) is super strict about text searches and requires strict substring matching 📖 but lots of servers use search engines that match strings in different ways 🔍 like matching "swim" to both "swam" and "swum", or only doing full word matching (where "swim" will NOT match "swimming") 🏊‍♀️ that's covered by the "Fuzzy Search" extension to IMAP [RFC6203], and it's compatible with this one and can be combined with it 🫶

whether or not Fuzzy Search is implemented or used, this extension explicitly allows flexible searching for TEXT and BODY searches 🌈 so servers MAY use fuzzy text matching in multimailbox searches ✨

2.4. Server-Side Limitations on Search Volume

bestie, nobody wants one search hogging all the server's resources 😭 so servers MAY apply limits that go beyond loop protection 🛡️ like capping the number of mailboxes searched at once, and/or capping the number or total size of messages searched 📏 a server can apply those limits up front and respond with "NO [LIMIT]" if a limit gets exceeded (see [RFC5530] for info about response codes) 🙅‍♀️ or it can process the search and stop when a limit is hit, responding with "OK [LIMIT]" and returning partial results 🧩 but ngl, searches that return partial results can cause complexity for client implementations and confusion for users 💀

3. Examples

okay so in this example, note that two ESEARCH commands are pipelined and the server runs them in parallel 🏃‍♀️💨 it interleaves a response to the second search amid the responses to the first, so watch the tags 👀🏷️

C: tag1 ESEARCH IN (mailboxes "folder1" subtree "folder2") unseen
C: tag2 ESEARCH IN (mailboxes "folder1" subtree-one "folder2")
subject "chad"
S: * ESEARCH (TAG "tag1" MAILBOX "folder1" UIDVALIDITY 1) UID ALL
4001,4003,4005,4007,4009
S: * ESEARCH (TAG "tag2" MAILBOX "folder1" UIDVALIDITY 1) UID ALL
3001:3004,3788
S: * ESEARCH (TAG "tag1" MAILBOX "folder2/banana" UIDVALIDITY 503)
UID ALL 3002,4004
S: * ESEARCH (TAG "tag1" MAILBOX "folder2/peach" UIDVALIDITY 3) UID
ALL 921691
S: tag1 OK done
S: * ESEARCH (TAG "tag2" MAILBOX "folder2/salmon" UIDVALIDITY
1111111) UID ALL 50003,50006,50009,50012
S: tag2 OK done

4. Formal Syntax

the grammar below is written in Augmented Backus-Naur Form (ABNF) as described in [RFC5234] 📝 terms that aren't defined here come from [RFC3501], [RFC5465], or [RFC4466] 🧸

command-auth =/  esearch
        ; Update definition from IMAP base [RFC3501].
        ; Add new "esearch" command.

command-select =/  esearch
        ; Update definition from IMAP base [RFC3501].
        ; Add new "esearch" command.

filter-mailboxes-other =/  ("subtree-one" SP one-or-more-mailbox)
        ; Update definition from IMAP Notify [RFC5465].
        ; Add new "subtree-one" selector.

filter-mailboxes-selected =  "selected"
        ; Update definition from IMAP Notify [RFC5465].
        ; We forbid the use of "selected-delayed".

one-correlator =  ("TAG" SP tag-string) / ("MAILBOX" SP astring) /
        ("UIDVALIDITY" SP nz-number)
        ; Each correlator MUST appear exactly once.

scope-option =  scope-option-name [SP scope-option-value]
        ; No options defined here.  Syntax for future extensions.

scope-option-name =  tagged-ext-label
        ; No options defined here.  Syntax for future extensions.

scope-option-value =  tagged-ext-val
        ; No options defined here.  Syntax for future extensions.

scope-options =  scope-option *(SP scope-option)
        ; A given option may only appear once.
        ; No options defined here.  Syntax for future extensions.

esearch =  "ESEARCH" [SP esearch-source-opts]
        [SP search-return-opts] SP search-program

search-correlator =  SP "(" one-correlator *(SP one-correlator) ")"
        ; Updates definition in IMAP4 ABNF [RFC4466].

esearch-source-opts =  "IN" SP "(" source-mbox [SP
        "(" scope-options ")"] ")"

source-mbox =  filter-mailboxes *(SP filter-mailboxes)
        ; "filter-mailboxes" is defined in IMAP Notify [RFC5465].
        ; See updated definition of filter-mailboxes-other, above.
        ; See updated definition of filter-mailboxes-selected, above.

5. Security Considerations

okay this one is serious, so buckle up 🚨🫣 this new IMAP ESEARCH command lets a single command search many mailboxes at once 📚 a client could already do that by sending lots of IMAP SEARCH commands, but ESEARCH makes it way easier to overwork a server 😱 one command could trigger an expensive search of tens of thousands of mailboxes 💥 server implementations need to be aware of that and provide mechanisms that stop a client from adversely affecting other users 🛑 limiting the number of mailboxes searched in one command, and/or the server resources devoted to responding to a single client, are reasonable limits for an implementation to impose (see also Section 2.4) 🧱

implementations MUST, of course, apply access controls properly, limiting a user's ESEARCH access the same way their access is limited for any other IMAP commands 🔐 this extension has no data-access risks beyond what may already exist in the unextended IMAP implementation 🫶

mailboxes matching the source options where the logged-in user lacks sufficient rights MUST be ignored by ESEARCH processing (see the paragraph about this in Section 2.2) 🙈 and in particular, any attempt to tell insufficient access apart from non-existent mailboxes may expose information about the mailbox hierarchy that the client otherwise wouldn't have 👀 don't leak the hierarchy, besties! 🚫

if "subtree" is specified, the server MUST defend against loops in the hierarchy (see the paragraph about this in Section 2.2) 🔁

6. IANA Considerations

the "Internet Message Access Protocol (IMAP) Capabilities Registry" is currently located at <http://www.iana.org/assignments/imap-capabilities> 📋

IANA has changed the reference for the IMAP capability "MULTISEARCH" to point to this document 🎀

7. Changes since RFC 6237

  • Change to Standards Track. 🚀
  • Added paragraph about duplicate mailboxes. 👯‍♀️
  • Added Section 2.3 about Fuzzy Search. 🦋

8. References

8.1. Normative References ✨

Okay, these are the must-read ones 📚 bestie, you need these to really get the spec:

8.2. Informative References 🌸

These ones are more of a "good to know" vibe:

Acknowledgments 🫶

The authors gratefully acknowledge feedback provided by Timo Sirainen, Peter Coates, Arnt Gulbrandsen, and Chris Newman. Iconic group of people, honestly 💜

Authors' Addresses 💖

Barry Leiba
Huawei Technologies

Phone: +1 646 827 0648
EMail: barryleiba@computer.org
URI:   http://internetmessagingtechnology.org/


Alexey Melnikov
Isode Limited
14 Castle Mews
Hampton, Middlesex  TW12 2NP
United Kingdom

EMail: Alexey.Melnikov@isode.com
URI:   http://www.melnikov.ca/
📦 where this page comes from