RFC 7352

Sieve Email Filtering: Detecting Duplicate Deliveries

📅 published September 2014, from ietf

RFC 7352: Sieve Email Filtering: Detecting Duplicate Deliveries 💌

Okay bestie, this one's the spec for a brand new Sieve duplicate test 🎀✨ It's from September 2014 and it's a Standards Track doc, which means it's an actual IETF standard 😭🔥. The author is S. Bosch, and it's published as ISSN 2070-1721. The title block doesn't list anything this RFC obsoletes or updates, so it's standing on its own 💅

Abstract 🦋

So this doc adds a new test command called "duplicate" to the Sieve email filtering language 💖 The whole point is spotting duplicates. The big use case is those double deliveries you get from mailing lists or redirected addresses 🙈 The detection usually works by matching the message ID against a little internal list of IDs from messages that already got delivered. For fancier stuff, "duplicate" can also look at a specific header field or other parts of the message ✨

Status of This Memo 📜

This is an Internet Standards Track document 🎀

It's a product of the Internet Engineering Task Force (IETF), so it represents the consensus of the IETF community 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) 👑 If you want more info on Internet Standards, check Section 2 of RFC 5741 🌸

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc7352 💜

Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved 🌈

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stood on the day it was published 📚 Please actually read these, because they spell out your rights and restrictions for this document 🙏 Code Components pulled out of this document have to include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Simplified BSD License describes 💅

Table of Contents 🗂️

   1. Introduction ....................................................2
   2. Conventions Used in This Document ...............................3
   3. Test "duplicate" ................................................3
      3.1. Arguments ":header" and ":uniqueid" ........................5
      3.2. Argument ":handle" .........................................7
      3.3. Arguments ":seconds" and ":last" ...........................8
      3.4. Interaction with Other Sieve Extensions ....................9
   4. Sieve Capability Strings ........................................9
   5. Examples ........................................................9
      5.1. Example 1 ..................................................9
      5.2. Example 2 .................................................10
      5.3. Example 3 .................................................11
      5.4. Example 4 .................................................12
   6. Security Considerations ........................................12
   7. IANA Considerations ............................................13
   8. Acknowledgements ...............................................14
   9. References .....................................................14
      9.1. Normative References ......................................14
      9.2. Informative References ....................................15

1. Introduction 🧚

This doc specifies an extension to the Sieve filtering language, which is defined in RFC 5228 [SIEVE] 💖 It adds a test that tracks whether a text string was already seen by the delivery agent during an earlier run of the Sieve script 👀 That's how you detect and handle duplicate deliveries 🎀

Duplicate deliveries are super common when you're subscribed to a mailing list 😭 Like, if a list member replies to both you and the list itself, you'll usually get one copy directly and another through the list 💀 And if someone crossposts to several lists you're on, you'll probably get a copy from each one 🙈 Another classic: you have a bunch of redirected addresses all pointing to your main account, and if one of your contacts emails more than one of them, you'll likely get more than one copy ✨ With the "duplicate" extension, you can detect those and handle them, like discarding them, marking them as "seen", or putting them in a special folder 🫶

Duplicate messages are normally detected with the Message-ID header field, which is required to be unique for each message 🔥 But the "duplicate" test is flexible enough to use other criteria for what counts as a duplicate, like the subject line or parts of the message body 🌸 This new test can be used for other stuff too, as long as the tracked unique value is a string 💜

2. Conventions Used in This Document 📝

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [KEYWORDS] 💅

Notation conventions are the same as in Section 1.1 of [SIEVE], including the use of the "Usage:" label for defining action and tagged arguments syntax 🎀

3. Test "duplicate" 💖

Here's the usage syntax, copied exactly:

   Usage: "duplicate" [":handle" <handle: string>]
                      [":header" <header-name: string> /
                          ":uniqueid" <value: string>]
                      [":seconds" <timeout: number>] [":last"]

So the "duplicate" test identifies a message by a "unique ID" ✨ It uses that unique ID to keep track of which messages it saw during an earlier Sieve execution 👀 In its basic form, the test grabs the unique ID from the content of the message's Message-ID header field 🫶 The test evaluates to "true" if the message was seen before, and "false" if it wasn't 💅

Here's the tea 🍓 The test also adds the unique ID to an internal duplicate-tracking list, but only once the Sieve execution finishes successfully 🔥 The first time a particular unique ID shows up, the message isn't a duplicate, and the ID gets added to the tracking list 🌈 If a future execution sees a message whose unique ID is already on that list, the test evaluates to "true" and the message gets treated as a duplicate 😭

Heads up, though: this side effect only happens when the "duplicate" test is actually evaluated 🙈 If the test is nested inside a control structure, or it isn't the first item of an "allof" or "anyof" test list, its evaluation depends on the results of the tests before it, and that can produce unexpected results 💀

Implementations MUST only update the internal duplicate-tracking list when the Sieve script execution finishes successfully. If failing script executions add the unique ID to the duplicate-tracking list, all "duplicate" tests in the Sieve script would erroneously yield "true" for the next delivery attempt of the same message. Depending on the action taken for a duplicate, this can easily lead to discarding the message without further notice. 😱

Okay but deferring the real update to the end of a successful run isn't totally problem-free 🥺 It can cause a race condition when a duplicate message gets delivered in parallel before the tracking list is updated 😵 Then a duplicate could slip past the "duplicate" test. More complex implementations could use a locking mechanism to stop this 🔒 But no matter which implementation you pick, situations where the "duplicate" test erroneously yields "true" MUST be prevented 🛑

The "duplicate" test MUST only check for duplicates among unique ID values seen in previous executions of the Sieve script 💜 It MUST NOT consider ID values seen earlier in the current execution as potential duplicates 🙅‍♀️ So all "duplicate" tests in one Sieve script execution, including the ones in scripts pulled in with the "include" [INCLUDE] extension, MUST always yield the same result when their arguments are identical 🦋

The Message-ID header field is assumed to be globally unique, as required in Section 3.6.4 of RFC 5322 [IMAIL] 🌸 In practice, that assumption doesn't always hold up, and the "duplicate" test doesn't handle that situation, so false duplicates may get detected 😬 The good news is that users can handle this by picking a different way to identify messages, using the ":header" or the ":uniqueid" argument, which the next section covers 🧚✨

3.1. Arguments ":header" and ":uniqueid" 🦋

Okay so, duplicate tracking has one big job: figure out the unique ID for a message, then check if that ID is already sitting in the duplicate-tracking list 👀. The unique ID gets picked like this:

  • When neither :header nor :uniqueid is used, the unique ID is the content of the message's Message-ID header field.
  • When :header is used, the unique ID is the content of that specified header field in the message. The header field name is NOT part of the ID, so it's only the field value. 💅
  • When :uniqueid is used, the unique ID is the string parameter you give with the argument. ✨

:header and :uniqueid are mutually exclusive, so putting both on one "duplicate" test command MUST trigger an error. No mixing them, bestie 🙅‍♀️

The header name parameter of :header follows the syntax rules in Section 2.4.2.2 of RFC 5228 [SIEVE]. Here's the vibe check: implementations MUST NOT trigger an error for an invalid header name. Instead, the "duplicate" test MUST yield "false" unconditionally in that case. The parameter of :uniqueid can be any string at all, so go wild 🌈

When the tracked unique ID comes straight from a header field (so :uniqueid isn't used), these three operations MUST be done before the actual duplicate check:

  • Unfold the header line as described in Section 2.2.3 of RFC 5322 [IMAIL] (see also Section 2.4.2.2 of RFC 5228 [SIEVE]).
  • If possible, convert the header value to Unicode, encoded as UTF-8 (see Section 2.7.2 of RFC 5228 [SIEVE]). If conversion isn't possible, the value is left unchanged.
  • Trim leading and trailing whitespace from the header value (see Section 2.2 of RFC 5228 [SIEVE]).

These rules also apply to the Message-ID header field used by the basic "duplicate" test with no :header or :uniqueid argument. When :uniqueid IS used, any normalization has to happen in the Sieve script itself, since that's where the unique ID gets created 🛠️

If the header field named by :header shows up multiple times in the message, extracting the unique ID MUST use only the first occurrence. This holds whether or not multiple occurrences are allowed by Section 3.6 of RFC 5322 [IMAIL]. If the specified header field isn't in the message at all, the "duplicate" test MUST yield "false" unconditionally. In that case the duplicate-tracking list is left unmodified by this test, because there's no unique ID value to work with 🫠 Same deal for Message-ID on the basic "duplicate" test with no :header or :uniqueid, since that header could also be missing or show up more than once.

The string parameter of :uniqueid can be built from arbitrary text pulled out of the message using the "variables" [VARIABLES] extension. To grab text from the message body, the "foreverypart" and "extracttext" [SIEVE-MIME] extensions need to be used too. Total control over how the unique ID gets made, iconic 👑

The unique ID MUST be matched case-sensitively against the contents of the duplicate-tracking list, no matter how the unique ID was determined. Want case-insensitive behavior with :uniqueid? Use the "set" command from the "variables" [VARIABLES] extension to normalize the value to upper or lower case 🔠

3.2. Argument ":handle" 🧚

The "duplicate" test MUST track a unique ID value independent of its source. So every value in the duplicate-tracking list should be used for duplicate testing, whether it came from the Message-ID header field, an arbitrary header named with :header, or explicitly from :uniqueid. These three examples are equivalent and match the same entry in the list:

require "duplicate";
if duplicate {
  discard;
}
require "duplicate";
if duplicate :header "message-id" {
  discard;
}
require ["duplicate", "variables"];
if header :matches "message-id" "*" {
  if duplicate :uniqueid "${0}" {
    discard;
  }
}

The :handle argument can override this default behavior 🎀 It separates a "duplicate" test from other "duplicate" tests with a different or omitted :handle argument. That means unrelated tests can't mess with each other: a message only counts as a duplicate when its tracked unique ID was seen before, in an earlier script execution, by a "duplicate" test with the same :handle argument. Main character energy for each test 💖

NOTE: The mechanism needed to track duplicate messages is super similar to the one needed for tracking duplicate responses for the "vacation" action [VACATION]. So one way to build the mechanism for the "duplicate" test is to store a hash of the tracked unique ID and, if provided, the :handle argument. 🤓

3.3. Arguments ":seconds" and ":last" ⏰

Implementations SHOULD let entries in the tracking list expire after a short period of time. Users can control how long that is with the :seconds argument, which takes an integer for the timeout in seconds. If :seconds is left out, an appropriate default value MUST be used, and a default expiration of around 7 days is usually appropriate 📅 Sites SHOULD impose a maximum limit on expiration time. If :seconds goes over that limit, the maximum value MUST be silently substituted, and going over the limit MUST NOT produce an error. If :seconds is zero, the "duplicate" test MUST yield "false" unconditionally.

When :last is omitted, the expiration time for entries in the duplicate-tracking list MUST be measured from the moment the entry was first created. That's the end of the successful script execution during which the "duplicate" test returned "false" for a message with that unique ID value. So later duplicates have zero influence on when the entry finally expires 🙅‍♀️

When :last IS specified, the expiration time MUST be measured from the last script execution in which the "duplicate" test checked that entry's unique ID value. Basically, the entry won't expire as long as duplicates with that unique ID keep showing up at intervals shorter than the expiration time. Lowkey so clever 🧠

It's possible to write Sieve scripts where, in one execution, more than one "duplicate" test runs with the same unique ID value and :handle argument but different :seconds or :last arguments. This spec leaves the resulting behavior undefined, so please avoid those constructs 🙈 Implementations MAY choose to use the :seconds and :last arguments from the "duplicate" test that was evaluated last.

3.4. Interaction with Other Sieve Extensions 🫶

The "duplicate" test doesn't support the "index" [DATE-INDEX] or "mime" [SIEVE-MIME] extensions directly. So none of :index, :mime, or their associated arguments get added to the "duplicate" test when those extensions are active. The "variables" [VARIABLES] extension combined with :uniqueid can get the same result indirectly 🔁

Normally, Sieve scripts run at final delivery. But with the "imapsieve" [IMAPSIEVE] extension, scripts get invoked when the IMAP [IMAP] server does stuff to the message store, like when messages are uploaded, flagged, or moved somewhere else. The "duplicate" test was devised for final delivery, and its semantics in the "imapsieve" context are left undefined. So implementations SHOULD NOT allow the "duplicate" test to be used in the context of "imapsieve". 🚫

4. Sieve Capability Strings 💜

A Sieve implementation that defines the "duplicate" test command will advertise the capability string "duplicate". Easy peasy 🌸

5. Examples 🍓

5.1. Example 1

In this basic example, message duplicates get detected by tracking the Message-ID header field. Duplicate deliveries are stored in a special folder inside the user's Trash folder. If that folder doesn't exist, it's created automatically using the "mailbox" [MAILBOX] extension, so the user still has a chance to recover messages when needed 🗑️ Messages that aren't recognized as duplicates land in the user's inbox as normal.

require ["duplicate", "fileinto", "mailbox"];

if duplicate {
  fileinto :create "Trash/Duplicate";
}

5.2. Example 2

This one shows a more complex use of the "duplicate" test 🔍 The user gets network alerts from a bunch of remote automated monitoring systems. Several notifications can come in about the same event from different monitoring systems. The Message-ID header field is different on each, since they're all distinct messages from different senders. So to avoid getting notified more than once about the same event, the user writes this script:

require ["duplicate", "variables", "imap4flags",
  "fileinto"];

if header :matches "subject" "ALERT: *" {
  if duplicate :seconds 60 :uniqueid "${1}" {
    setflag "\\seen";
  }
  fileinto "Alerts";
}

The subjects of these notifications follow a predictable pattern that includes a description of the event. In the script, the "duplicate" test detects duplicate alert events. The subject gets matched against a pattern, and the event description is pulled out using the "variables" [VARIABLES] extension. If a message with that event in the subject came in before, but more than a minute ago, it's not detected as a duplicate because of the :seconds argument 🕐 In the event of a duplicate, the message is marked as "seen" using the "imap4flags" [IMAP4FLAGS] extension ✅ All alert messages go into the "Alerts" mailbox, whether they're duplicates or not 📬

5.3. Example 3 🎀

okay bestie, this example is about using the "duplicate" test to limit how often "enotify" [NOTIFY] sends notifications 🔔✨ Picture this: a mail user gets Extensible Messaging and Presence Protocol (XMPP) notifications [NOTIFY-XMPP] about new mail through Sieve, but sometimes one contact sends like twenty messages in a few minutes 💀 The user wants to stop getting pinged for every single one. So they only want to hear from each person once per 30 minutes 🕐 and they write this script:

require ["variables", "envelope", "enotify", "duplicate"];

if envelope :matches "from" "*" { set "sender" "${1}"; }
if header :matches "subject" "*" { set "subject" "${1}"; }

if not duplicate :seconds 1800 :uniqueid "${sender}"
{
  notify :message "[SIEVE] ${sender}: ${subject}"
    "xmpp:user@im.example.com";
}

The example above uses the message envelope sender instead of the Message-ID header field as the unique ID for duplicate tracking 💅

It can be extended too 🦋 So the user can get more messages from the same sender in close succession, as long as the subject is different. Here's how:

require ["variables", "envelope", "enotify", "duplicate"];

if envelope :matches "from" "*" { set "sender" "${1}"; }
if header :matches "subject" "*" { set "subject" "${1}"; }

# account for 'Re:' prefix
if string :comparator "i;ascii-casemap"
  :matches "${subject}" "Re:*"
{
  set "subject" "${1}";
}
if not duplicate :seconds 1800
  :uniqueid "${sender} ${subject}"
{
  notify :message "[SIEVE] ${sender}: ${subject}"
    "xmpp:user@im.example.com";
}

This one uses a combo of the envelope sender and the subject as the unique ID for duplicate tracking 😭🫶 Sender + subject together is so iconic.

5.4. Example 4 🌸

In this example, the mail user uses the "duplicate" test for two separate jobs 🧚 First, to toss out duplicate events from a notification system. Second, to mark certain follow-up messages in a software support mailing list as "seen" using the "imap4flags" [IMAP4FLAGS] extension.

Here's the tea ☕ Both "duplicate" tests look at a different header to identify messages: "X-Event-ID" and "X-Ticket-ID". But these two headers can have similar values (like both being based on a timestamp, for example), so one "duplicate" test could accidentally match IDs that the other one is tracking 🙈 The user doesn't want that to happen in either direction. So the fix is giving each test its own ":handle" argument. Iykyk 💜

require ["duplicate", "imap4flags"];

if duplicate :header "X-Event-ID" :handle "notifier" {
  discard;
}
if allof (
  duplicate :header "X-Ticket-ID" :handle "support",
  address "to" "support@example.com",
  header :contains "subject" "fileserver")
{
  setflag "\\seen";
}

6. Security Considerations 🔐

A flood of unique messages could make the duplicate-tracking list grow forever 😱 So implementations SHOULD limit the number of entries in that list. And when they do limit it, implementations SHOULD discard the oldest entries first 🗑️

Scripts that use the "duplicate" test should be aware that message IDs are not necessarily unique 🙃 This can happen through the fault of well-meaning generators, or because an attacker injects a message with the properties the duplicate Sieve filter looks at, at some point before the filter runs. So scripts are well advised to be conservative about what actions they take when duplicates are identified only by message ID 🚨

The list of unique IDs used for duplicate tracking can hold privacy-sensitive stuff, like message ID values, subject line content, and content pulled out of message bodies 👀 Implementations SHOULD protect that info by obscuring it through hashing (see the note at the end of Section 3.2) and/or by storing it with the same level of access control as the messages themselves. Lock it down bestie 🔒

These measures will not stop an entity that can access the duplicate-tracking list from asking whether messages with certain unique ID values were received. That question is basically the whole point of the "duplicate" test, so it can't be prevented, and it may go against what the user expects. For example, a user who deletes a message from the server may expect no record of it to remain there. That won't be true if its message ID is kept on the server in the duplicate-tracking list 💀

It's notable, though 🤔 server logs will often store the same info that's on the duplicate-tracking list anyway, and they would probably expose plaintext message IDs for a much longer time than this mechanism would. Users of email services that intentionally delete such logs to limit traceability should be made aware that using the duplicate-tracking mechanism re-exposes this info for the duration of the expiry interval. So a shorter default expiry interval may be appropriate in those situations ⏳

7. IANA Considerations 📋

The following template specifies the IANA registration of the Sieve extension specified in this document 💖

   To: iana@iana.org
   Subject: Registration of new Sieve extension

   Capability name:  duplicate
   Description:      Adds test 'duplicate' that can be used to test
                     whether a particular message is a duplicate,
                     i.e., whether a copy of it was seen before by
                     the delivery agent that is executing the Sieve
                     script.
   RFC number:       RFC 7352
   Contact address:  Sieve mailing list <sieve@ietf.org>

This information has been added to the list of Sieve extensions given on <http://www.iana.org/assignments/sieve-extensions> ✨

8. Acknowledgements 🫶

Huge thanks to Brian Carpenter, Cyrus Daboo, Arnt Gulbrandsen, Tony Hansen, Kristin Hubner, Barry Leiba, Alexey Melnikov, Subramanian Moonesamy, Tom Petch, Hector Santos, Robert Sparks, Aaron Stone, and Stefan Winter for their reviews and suggestions 💜 Special thanks to Ned Freed for his guidance and support 👑

9. References 📚

9.1. Normative References

  • [DATE-INDEX] Freed, N., "Sieve Email Filtering: Date and Index Extensions", RFC 5260, July 2008.
  • [IMAIL] Resnick, P., Ed., "Internet Message Format", RFC 5322, October 2008.
  • [IMAPSIEVE] Leiba, B., "Support for Internet Message Access Protocol (IMAP) Events in Sieve", RFC 6785, November 2012.
  • [INCLUDE] Daboo, C. and A. Stone, "Sieve Email Filtering: Include Extension", RFC 6609, May 2012.
  • [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [SIEVE] Guenther, P. and T. Showalter, "Sieve: An Email Filtering Language", RFC 5228, January 2008.
  • [SIEVE-MIME] Hansen, T. and C. Daboo, "Sieve Email Filtering: MIME Part Tests, Iteration, Extraction, Replacement, and Enclosure", RFC 5703, October 2009.
  • [VARIABLES] Homme, K., "Sieve Email Filtering: Variables Extension", RFC 5229, January 2008.

9.2. Informative References

  • [IMAP] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1", RFC 3501, March 2003.
  • [IMAP4FLAGS] Melnikov, A., "Sieve Email Filtering: Imap4flags Extension", RFC 5232, January 2008.
  • [MAILBOX] Melnikov, A., "The Sieve Mail-Filtering Language -- Extensions for Checking Mailbox Status and Accessing Mailbox Metadata", RFC 5490, March 2009.
  • [NOTIFY] Melnikov, A., Leiba, B., Segmuller, W., and T. Martin, "Sieve Email Filtering: Extension for Notifications", RFC 5435, January 2009.
  • [NOTIFY-XMPP] Saint-Andre, P. and A. Melnikov, "Sieve Notification Mechanism: Extensible Messaging and Presence Protocol (XMPP)", RFC 5437, January 2009.
  • [VACATION] Showalter, T. and N. Freed, "Sieve Email Filtering: Vacation Extension", RFC 5230, January 2008.

Author's Address 🌈

Stephan Bosch Enschede NL

EMail: stephan@rename-it.nl 💌

📦 where this page comes from