RFC 7103
Advice for Safe Handling of Malformed Messages
RFC 7103: Advice for Safe Handling of Malformed Messages ✨
Authors: M. Kucherawy, G. Shapiro, and N. Freed 💖 Category: Informational 🎀 Date: January 2014 🌸 ISSN: 2070-1721 Obsoletes/Updates: The title block doesn't list anything this one obsoletes or updates 🦋
Abstract ✨
Okay so, Internet message formats have been precisely defined since the 1970s, right? But authoring and handling software is often only mildly conformant to those specs, bestie 💅 The malformed messages that come out of that are non-standard. Still, decades of experience show that being a little tolerant when handling those malformations is often acceptable, and it's better than rejecting the messages outright as nonconformant 🫶 This document is a collection of the best advice available for a bunch of common malformed mail situations, and it's meant to be used as implementation guidance 📬
Status of This Memo 🌈
This document is not an Internet Standards Track specification. It's published for informational purposes 🧚
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 5741 🍓
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc7103 ✨
Copyright Notice 💜
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License 🔥
Table of Contents 🎀
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. The Purpose of This Work . . . . . . . . . . . . . . . . 3
1.2. Not the Purpose of This Work . . . . . . . . . . . . . . 4
1.3. General Considerations . . . . . . . . . . . . . . . . . 4
2. Document Conventions . . . . . . . . . . . . . . . . . . . . 5
2.1. Examples . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5
4. Invariant Content . . . . . . . . . . . . . . . . . . . . . . 5
5. Mail Submission Agents . . . . . . . . . . . . . . . . . . . 6
6. Line Termination . . . . . . . . . . . . . . . . . . . . . . 7
7. Header Anomalies . . . . . . . . . . . . . . . . . . . . . . 8
7.1. Converting Obsolete and Invalid Syntaxes . . . . . . . . 8
7.1.1. Host-Address Syntax . . . . . . . . . . . . . . . . . 8
7.1.2. Excessive Angle Brackets . . . . . . . . . . . . . . 8
7.1.3. Unbalanced Angle Brackets . . . . . . . . . . . . . . 8
7.1.4. Unbalanced Parentheses . . . . . . . . . . . . . . . 9
7.1.5. Commas in Address Lists . . . . . . . . . . . . . . . 9
7.1.6. Unbalanced Quotes . . . . . . . . . . . . . . . . . . 10
7.1.7. Naked Local-Parts . . . . . . . . . . . . . . . . . . 10
7.2. Non-Header Lines . . . . . . . . . . . . . . . . . . . . 10
7.3. Unusual Spacing . . . . . . . . . . . . . . . . . . . . . 12
7.4. Header Malformations . . . . . . . . . . . . . . . . . . 13
7.5. Header Field Counts . . . . . . . . . . . . . . . . . . . 13
7.5.1. Repeated Header Fields . . . . . . . . . . . . . . . 14
7.5.2. Missing Header Fields . . . . . . . . . . . . . . . . 15
7.5.3. Return-Path . . . . . . . . . . . . . . . . . . . . . 16
7.6. Missing or Incorrect Charset Information . . . . . . . . 16
7.7. Eight-Bit Data . . . . . . . . . . . . . . . . . . . . . 18
8. MIME Anomalies . . . . . . . . . . . . . . . . . . . . . . . 18
8.1. Missing MIME-Version Field . . . . . . . . . . . . . . . 19
8.2. Faulty Encodings . . . . . . . . . . . . . . . . . . . . 19
9. Body Anomalies . . . . . . . . . . . . . . . . . . . . . . . 19
9.1. Oversized Lines . . . . . . . . . . . . . . . . . . . . . 19
10. Security Considerations . . . . . . . . . . . . . . . . . . . 20
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.1. Normative References . . . . . . . . . . . . . . . . . . 20
11.2. Informative References . . . . . . . . . . . . . . . . . 20
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 231. Introduction 🌸
1.1. The Purpose of This Work 💅
The history of email standards, going back to [RFC733] and beyond, is a pretty rigid evolution of specs. But implementations in that culture have also had an undercurrent known formally as "the robustness principle," and informally as "Postel's Law": "Be liberal in what you accept, and conservative in what you send" [RFC1122] 🦋
Jon Postel's directive is often read as meaning that any deviance from a specification is acceptable. We don't think that's what he meant, though. We believe it was only meant to cover legitimate differences in interpretation within specifications, plus basic transit errors, like bit errors. Taken to its unintended extreme, excessive tolerance would mean there are no limits on the liberties a sender might take, while putting the burden on a receiver to guess "correctly" at what any such variation means. It gets messier because receiver software, the end users' mail readers, is also sometimes flawed, so senders end up crafting messages (sometimes bending the rules) to get around those flaws 🙈
Honestly, this served the email ecosystem well, since it let a few implementation errors slide without blocking anyone from playing the game. The bar was set low, lowkey on purpose. But as we've moved into the current era, some of these lenient stances have started to expose openings that malefactors can exploit. Various email-based applications rely on strict application of these standards for simple security checks, while the very basic building blocks of that infrastructure, which were meant to be robust, completely fail to enforce those standards 💀
The distributed, non-interactive nature of email has often led to adjustments in receiving software to handle these variations, instead of trying to get senders to conform better. That's because the receiving operator is mainly driven by complaints from recipient users and has no authority over the sending side of the system. All that flexibility comes at a cost, since mail software has to decide whether to let non-conforming messages continue toward their destinations unaltered, adjust them to conform (possibly losing some of the original message), or reject them outright 🔥
This document is a collection of the best advice available for a variety of common malformed mail situations, and it's meant to be used as implementation guidance. These malformations are typically based on loose interpretations or implementations of specifications such as the Internet Message Format [MAIL] and Multipurpose Internet Mail Extensions [MIME] ✨
1.2. Not the Purpose of This Work 🙅♀️
Important to understand: this work is not an effort to endorse or standardize common malformations, not even a little bit 🚫 The code and culture that put such messages into the mail stream need to be repaired, because the security penalty we're paying for this lax processing arguably outweighs the savings in support costs for end users who aren't expected to understand the standards. Sadly, though, the reality is that this won't be fixed quickly 😭
So, given that, it's beneficial to give implementers guidance on the safest or most effective way to handle malformed messages when they arrive. That means weighing the trade-offs of the available choices, especially how various actors in the email ecosystem respond to such messages in terms of handling, parsing, or rendering to end users 🎀
1.3. General Considerations 🧠
Many deviations from message format standards are considered by some receivers to be strong indications that a message is undesirable, such as spam or something containing malware. These receivers quickly decide the best handling choice is to simply reject or discard the message. That means malformations caused by innocent misunderstandings or ignorance of proper syntax can cause messages with no ill intent to fail to be delivered 😤
Senders who want their messages delivered are best advised to adhere strictly to the relevant standards (including, but not limited to, [MAIL], [MIME], and [DKIM]), and to observe other industry best practices that the IETF or others may publish from time to time 🍓
Receivers who haven't got the luxury of strictly enforcing the standards on inbound messages are usually best served by following these guidelines for handling malformed messages 👑
- Whenever possible, mitigation of syntactic malformations should be guided by an assessment of the most likely semantic intent. For example, it is reasonable to conclude that multiple sets of angle brackets around an address are simply superfluous and can be dropped. 💖
- When the intent is unclear, or when it is clear but also impractical to change the content to reflect that intent, mitigation should be limited to cases where not taking any corrective action would clearly lead to a worse outcome. 🌷
- Security issues, when present, need to be addressed and may force mitigation strategies that are otherwise suboptimal. 🔐
2. Document Conventions 🦄
2.1. Examples 📝
Examples of message content include a number within braces at the end of each line. These are line numbers for use in subsequent discussion, and they are not actually part of the message content presented in the example 🌈
Blank lines are not numbered in the examples 🫧
3. Background 📚
The reader would benefit from reading [EMAIL-ARCH] for some general background about the overall email architecture. Of particular interest is the Internet Message Format, detailed in [MAIL]. Throughout this document, the use of the term "message" should be assumed to mean a block of text conforming to the Internet Message Format 💜
4. Invariant Content ✨
okay bestie, so an agent handling a message can have a few different versions of it floating around 🫧. One is the internal representation, which is whatever setup makes processing easier for the agent, like separate blocks of storage for the header and body, some header or body alterations, or tables indexed by header name. The other is the representation passed along to the next agent in the handling chain. That one might be totally identical to the message that came in, or it might have some changes, like added or reordered header fields or body elisions to remove malicious content 👀
Message handling is usually most effective when each module in a sequence gets the same content to analyze. A module that "fixes" or otherwise alters the content passed to later modules can stop those later modules from spotting malicious or other content that exposes the end user to harm 😭 It's super important that all processing modules can make consistent assertions about the content. Modules that run one after another sometimes add private header fields to pass info downstream for later filters to use (and possibly remove), or they may have out-of-band ways of doing that. However, even the presence of private header fields can affect a downstream handling agent that doesn't know their local meaning, so an out-of-band method is always preferable 💅
The above is less of a concern when multiple analysis modules run in parallel, independent of one another ✨
Abuse reporting systems often can only act effectively when a complaint or report contains the original message exactly as it was generated. Messages that have been altered by handling modules might make a complaint not actionable, since the system getting the report may be unable to identify the original message as one of its own 🙈
Some message changes alter syntax without changing semantics. For example, Section 7.4 describes a situation where an agent removes additional header whitespace. That's a syntax change with no change in semantics, though some systems (such as DKIM) are sensitive to changes like that. Message system developers need to be aware of the downstream impact of making either kind of change 🧚
Where a change to content between modules is unavoidable, it's a good idea to add standard trace data to show that a "visible" handoff between modules has happened. The only advisable way to do this is to prepend Received fields with the appropriate information, as described in Section 3.6.7 of [MAIL] 🎀
There will always be local handling exceptions, but these guidelines should be useful for building integrated message processing environments 🌸
In most cases, this document only discusses techniques used on internal representations. It's occasionally necessary to make changes between the input and output versions, and those cases will be called out explicitly 🦋
5. Mail Submission Agents
Within the email context, the single most influential component that can reduce malformed items in the email system is the Mail Handling Service (MHS; see [EMAIL-ARCH]), which includes the Mail Submission Agent (MSA). This is the component that's basically the interface between end users creating content and the mail stream 💌
MHSs need to get way stricter about enforcing all relevant email standards, especially [MAIL] and the [MIME] family of documents 🔥
More strict conformance by relaying Mail Transfer Agents (MTAs) will also help. Stopping malformed messages from spreading is desirable, but rejecting mail that's already in transit also has a support cost -- namely, creating a [DSN] that many end users might not understand 🥺
6. Line Termination
For interoperable Internet Mail messages, the only valid line separation sequence during a typical SMTP session is ASCII 0x0D ("carriage return", or CR) followed by ASCII 0x0A ("line feed", or LF), commonly called "CRLF" 💖 This is not the case for binary mode SMTP (see [BINARYSMTP]).
Common UNIX tools, though, typically only use LF for internal line termination. That means a protocol engine that converts between UNIX and Internet message formats has to convert between these two end-of-line representations before sending a message or after receiving one 🌈
Non-compliant implementations can create messages with mixed line terminations, like LF everywhere except CRLF only at the very end of the message. According to [SMTP] and [MAIL], this means the entire message actually exists on a single line 💀
Within modern Internet Mail, it's highly unlikely that an isolated CR or LF is valid in common ASCII text. Plus, when content actually needs to contain such an unusual sequence, [MIME] provides mechanisms for encoding it in an SMTP-safe manner 🫶
Thus, it will typically be safe and helpful to treat an isolated CR or LF as equivalent to a CRLF when parsing a message ✨
Note that this advice applies only to the raw SMTP data and not to decoded MIME entities. As noted above, when MIME encoding mechanisms are used, the unusual character sequences aren't visible in the raw SMTP stream.
7. Header Anomalies ✨
This section covers common syntactic and semantic anomalies found in a message header and presents suggested methods of mitigation 🧐
7.1. Converting Obsolete and Invalid Syntaxes 💅
A message using an obsolete header syntax (see Section 4 of [MAIL]) might confuse an agent that's trying to be robust in its handling of syntax variations. A bad actor could exploit that weakness to get abusive or malicious content past a filter. This section shows some examples of such variations. Messages including these variations ought to be rejected; where that isn't possible, recommended internal interpretations are provided 🚫
7.1.1. Host-Address Syntax 🦋
The following obsolete syntax attempts to specify source routing:
To: <@example.net:fran@example.com>This means "send to fran@example.com via the mail service at example.net". It can safely be interpreted as:
To: <fran@example.com>7.1.2. Excessive Angle Brackets 🌸
The following overuse of angle brackets:
To: <<<user2@example.org>>>can safely be interpreted as:
To: <user2@example.org>7.1.3. Unbalanced Angle Brackets 🙈
The following use of unbalanced angle brackets:
To: <another@example.netcan usually be treated as:
To: <another@example.net>The following:
To: second@example.org>can usually be treated as:
To: second@example.org7.1.4. Unbalanced Parentheses 🎀
The following use of unbalanced parentheses:
To: (Testing <fran@example.com>can safely be interpreted as:
To: (Testing) <fran@example.com>Likewise, this case:
To: Testing) <sam@example.com>can safely be interpreted as:
To: "Testing)" <sam@example.com>In both cases, it's obvious where the active email address in the string can be found. The former case keeps the active email address in the string by completing what looks like a comment that was meant to be there. The intent in the latter case is less obvious, so the leading string is interpreted as a display name 🧠
7.1.5. Commas in Address Lists 🍓
This use of an errant comma:
To: <third@example.net, fourth@example.net>can usually be interpreted as ending an address, so the above is usually best interpreted as:
To: third@example.net, fourth@example.net7.1.6. Unbalanced Quotes 💜
The following use of unbalanced quotation marks:
To: "Joe <joe@example.com>leaves software with no unambiguous interpretation. One possible interpretation is:
To: "Joe <joe@example.com>"@example.netwhere "example.net" is the domain name or host name of the handling agent making the interpretation. However, the more obvious and likely best interpretation is simply:
To: "Joe" <joe@example.com>7.1.7. Naked Local-Parts 🫠
[MAIL] defines a local-part as the user portion of an email address, and the display-name as the "user-friendly" label that accompanies the address specification 📬
Some broken submission agents might introduce messages with only a local-part or only a display-name and no properly formed address. For example:
To: JoeA submission agent ought to reject this or, at a minimum, append "@" followed by its own host name or some other valid name likely to enable a reply to be delivered to the correct mailbox. Where this is not done, an agent receiving such a message will probably be successful by synthesizing a valid header field for evaluation using the techniques described in Section 7.5.2 (the rescue plan is giving main character energy 👑)
7.2. Non-Header Lines ✨
okay so sometimes a message has a line chilling in the header that isn't a valid header field of any kind 💀 literally the worst. Like this:
From: user@example.com {1}
To: userpal@example.net {2}
Subject: This is your reminder {3}
about the football game tonight {4}
Date: Wed, 20 Oct 2010 20:53:35 -0400 {5}
Don't forget to meet us for the tailgate party! {7}The usual culprit is a bug in some message generator 🙈 Line {4} was supposed to be a continuation of line {3}, so it should have been indented with whitespace like Section 2.2.3 of [MAIL] says. Oops.
This anomaly hits different software in different ways, depending on the implementation 🫠
- Some agents only split the header from the body at the first empty line (so a CRLF immediately followed by another CRLF).
- Some agents decide the body starts at line {4}, since the header is assumed to end when they hit something that isn't a valid header field or a folded part of one.
- Some agents think it's an intended header fold and just stick a single space character (ASCII 0x20) plus the content of line {4} onto line {3}.
- Some agents reject the whole thing outright, because line {4} is neither a valid header field nor a folded continuation of one before an empty line.
This is where it gets sus 😤 It can be exploited if one agent in the handling chain does one thing and the next agent does another. Picture a message filter scanning headers for abusive or malicious stuff, sitting behind a Mail Transfer Agent (MTA) that uses option 2 above. An attacker could put this malformation above the property they care about, knowing the MTA won't count that stuff as header. So the MTA never hands it to the filter, and it slips right past detection 🥷 Meanwhile the Mail User Agent (MUA), the part that actually shows the content to a real person, uses option 1 or 3, and that causes some undesirable effects.
A few implementations pick option 4, and honestly? Valid. Any reputable message generator gets header folding right, so something this blatant is probably an error caused by a malefactor 🚩
If option 4 isn't the route you're taking, the preferred heuristic when this malformation shows up goes like this 🧠
- Search forward for an empty line. If you find one, apply option 3 above to the anomalous line, and keep going.
- Search forward for another line that looks like a new header field (a name followed by a colon). If you find one, apply option 3 above to the anomalous line, and keep going.
7.3. Unusual Spacing 🌸
This message is totally valid per [MAIL], no cap:
From: user@example.com {1}
To: userpal@example.net {2}
Subject: This is your reminder {3}
{4}
about the football game tonight {5}
Date: Wed, 20 Oct 2010 20:53:35 -0400 {6}
Don't forget to meet us for the tailgate party! {8}Line {4} is just a single whitespace. The intention is that lines {3}, {4}, and {5} are one continued header field 🧵 BUT some agents are super aggressive about stripping trailing whitespace, so line {4} gets read as an empty line, which means it's now the separator between header and body. Chaos, and it messes with the header-specific processing we just talked about 😭
Fun fact: this example was legal in older versions of the Internet message format standard, but it got made obsolete as of [RFC2822], because line {4} could be read as the separator between header and body. Rude 💅
The best way to handle this? A message parsing engine should act like line {4} isn't even there, and a message creation engine should emit the message with line {4} removed ✨
7.4. Header Malformations 🎀
A super common malformation is whitespace showing up in weird places, like this:
From: user@example.com {1}
To: userpal@example.net {2}
Subject: This is your reminder {3}
MIME-Version : 1.0 {4}
Content-Type: text/plain {5}
Date: Wed, 20 Oct 2010 20:53:35 -0400 {6}
Don't forget to meet us for the tailgate party! {8}Look at line {4}: there's extra whitespace after the field name but before the colon that separates the name from the value 👀
Here's the tea. The obsolete grammar in Section 4 of [MAIL] actually allows that extra whitespace, so it can't be called invalid. BUT most implementations prefer to strip it out anyway 🧹 Nothing about the meaning of the field changes, because that whitespace is semantically meaningless. So the move is to remove all whitespace after the field name and before the colon, and then emit the field in that cleaned-up form. Glow-up complete ✨
7.5. Header Field Counts 🔢
Section 3.6 of [MAIL] spells out specific header field counts for a valid message. Here's the thing though: very few agents actually enforce that. If a message goes over one or more of those limits, it usually just gets through 🙃 They do tend to add any missing required fields, though.
And honestly, few agents that take messages as input (including MUAs that actually show messages to users) bother checking that the input is valid before moving on. Some popular open-source filtering programs and some popular Mailing List Management (MLM) packages pick either the first or the last instance of a field, like From, to decide who sent the message 😬 So if an attacker knows the filter only looks at one instance, they can craft a message with multiple copies of the same field, and the user gets shown the other ones. Sneaky, sneaky 🦹
It gets even messier when message validity gets checked with enhanced authentication methods like DomainKeys Identified Mail [DKIM]. Those methods might cover one instance of a constrained field and not another, so they could mark the wrong one as "good" or "safe". For example, an MUA could show a user the first of two From fields as "good" or "safe", while the authentication method only actually verified the second one 💀
To counter this, one of the following strategies can be used 🛡️
- Reject outright, or refuse to process further, any input message that doesn't conform to Section 3.6 of [MAIL].
- Remove any instances of a header field whose presence goes over a limit prescribed in Section 3.6 of [MAIL] when generating output. If it's an MUA, it can refuse to render those instances instead.
- Where a field can contain multiple distinct values (like From) or is free-form text (like Subject), combine them into a single header field of the same name that means the same thing (see Section 7.5.1).
- Alter the name of any header field whose presence goes over a limit prescribed in Section 3.6 of [MAIL] when generating output, so later agents can produce a consistent result. Any alteration that's likely to get the field ignored by downstream agents is fine. A common approach is prefixing the field names with a string like "BAD-".
When you're picking a mitigation action (or some other action) from that list, an operator needs to think about their own needs and what their user base is like 🫶
7.5.1. Repeated Header Fields 🧁
Sometimes you'll spot repeated fields where only one was expected. Here are two examples, bestie 💜 First:
From: reminders@example.com {1}
To: jqpublic@example.com {2}
Subject: Automatic Meeting Reminder {3}
Subject: 4pm Today -- Staff Meeting {4}
Date: Wed, 20 Oct 2010 08:00:00 -0700 {5}
Reminder of the staff meeting today in the small {6}
auditorium. Come early! {7}That message has two Subject fields, which breaks Section 3.6 of [MAIL]. A safe way to read it is to treat the two Subject field values as concatenated, as long as they aren't identical:
From: reminders@example.com {1}
To: jqpublic@example.com {2}
Subject: Automatic Meeting Reminder {3}
4pm Today -- Staff Meeting {4}
Date: Wed, 20 Oct 2010 08:00:00 -0700 {5}
Reminder of the staff meeting today in the small {6}
auditorium. Come early! {7}Second:
From: president@example.com {1}
From: vice-president@example.com {2}
To: jqpublic@example.com {3}
Subject: A note from the E-Team {4}
Date: Wed, 20 Oct 2010 08:00:00 -0700 {5}
This memo is to remind you of the corporate dress {6}
code. Attached you will find an updated copy of {7}
the policy. {8}
...Same deal as the first one: there's a violation in how many From fields there are. A likely safe reading is to combine them into a comma-separated address list inside one From field 📬
From: president@example.com, {1}
vice-president@example.com {2}
To: jqpublic@example.com {3}
Subject: A note from the E-Team {4}
Date: Wed, 20 Oct 2010 08:00:00 -0700 {5}
This memo is to remind you of the corporate dress {6}
code. Attached you will find an updated copy of {7}
the policy. {8}
...7.5.2. Missing Header Fields 🥺
Kind of like the last section, there are messages out in the wild that are missing certain required header fields. In particular, [MAIL] requires that a From and a Date field be present in all messages 📅
When an MTA gets a message that's missing these fields, it might do one of the following:
- Make no changes.
- Add an instance of the missing field(s) using synthesized content based on data in other parts of the protocol.
Option 2 is recommended for this case. Handling agents should add these fields for internal handling if they're missing, but should not add them to the external representation. Why? Some filter modules would treat the absence of those fields as a special situation (like rejection). So if an upstream filter adds them somewhere visible to other components, those modules lose their effectiveness 🫣
The synthesized fields should hold a best guess at what should have been there 🔍 For From, you can use the address from the SMTP MAIL command (if it isn't null), or a placeholder address followed by an address literal, like unknown@[192.0.2.1]. For Date, a date pulled from a Received field is a reasonable pick ✨
One other important case is a missing Message-ID field. An MTA that runs into a message without this field should synthesize a valid one and add it to the external representation. Why so serious? Lots of deployed tools use the content of that field as a unique message reference, so without it, correlating message processing gets really hard 🔗 Section 3.6.4 of [MAIL] describes advisable practice for synthesizing this field when it's absent, and it establishes a requirement that it be globally unique 🌍
7.5.3. Return-Path ✨
okay so legit messages can have more than one Return-Path header field 🙈 but honestly? most of the time that's a mistake, not a proper message with multiple header field blocks (see Section 3.6 of [MAIL] for those). so when you spot multiple Return-Path fields, ignore everything except the topmost one 💅. the topmost one is the most recent, since it got stuck on right near the mailbox that received the message. iconic behaviour, honestly 👑
7.6. Missing or Incorrect Charset Information 🌸
okay this one is giving textbook 📚. MIME lets you include text in charsets ("charsets") other than US-ASCII 🌍 and that text is required to have an identified charset. you can name it with a "charset" parameter in the Content-Type header field, put a charset label inside the MIME entity itself, or the Content-Type can imply the charset (see [CHARSET]) 🧚
the tea is that charset info is often missing or wrong in textual MIME entities 😭 so processing agents should run basic sanity checks, like:
- US-ASCII only has bytes between 1 and 127 inclusive (colloquially, "7-bit" data) 🔢, so anything with bytes outside that range ("8-bit" data) is necessarily not US-ASCII. (See Section 2.1 of [MAIL].)
- [UTF-8] has a very specific syntactic structure that other 8-bit charsets are unlikely to follow 🦋
- Null bytes (ASCII 0x00) are not allowed in either 7-bit or 8-bit data 🚫
- Not all 7-bit material is US-ASCII 🙃 the escape sequences used for character switching can hint at which charset you're dealing with, based on ISO/IEC 2022 [ISO-2022], like those defined in [ISO-2022-CN], [ISO-2022-JP], and [ISO-2022-KR] 🎀
when a character set error shows up, processing agents should either:
- apply heuristics to figure out the most likely character set and, if that works, carry on using that info; or
- refuse to process the malformed MIME entity 🙅♀️
null bytes are a whole vibe and need extra special handling 🌈 a null byte inside a textual MIME entity can make typical string processing functions misidentify where a string ends, and bad actors can use that to hide malicious content from analysis processes 💀 so yeah, they get their own treatment.
a few null bytes in isolation is probably just poor message construction 🤷♀️ those nulls should be silently dropped.
lots of null bytes usually means binary material that's been improperly encoded, improperly labelled, or both 📦. that stuff is likely damaged beyond recovery, so the best move is to refuse to process it 😤
and finally, null bytes might be a sign of possible malicious intent 👀 keep your eyes open, bestie 🫶
7.7. Eight-Bit Data 💜
standards-compliant email messages don't contain any non-ASCII data without saying so through published SMTP extensions 📮. otherwise, MIME encodings are usually what convert non-ASCII data to ASCII, in a way other handling agents or end users can reverse 🔄
the best way to handle non-compliant 8-bit material depends on where it shows up 📍
non-compliant 8-bit material in MIME entity content should just be processed as if the necessary SMTP extensions had been used to transfer the message ✨ also, improperly labelled 8-bit material in textual MIME entities may need the treatment from Section 7.6.
non-compliant 8-bit material in message or MIME entity header fields can be handled like this:
- occurrences in unstructured text fields, comments, and phrases can be converted into encoded-words (see [MIME3] if a likely character set can be determined) 🍓. or, 8-bit characters can be removed or replaced with some other character.
- occurrences in header fields whose syntax is unknown may be handled by dropping the field entirely, or by removing/replacing the 8-bit character as described above 🗑️
- occurrences in addresses are especially problematic 😩 agents supporting [EAI] may, if the 8-bit material conforms to 8-bit syntax, choose to treat the message as an EAI message and process it accordingly 🌸. otherwise, in most cases, it's best to exclude the address from any sort of processing (which might mean dropping it entirely), since any attempt to fix it for real is unlikely to work 💅
8. MIME Anomalies 🦄
the five-part set of MIME specifications includes a mechanism of message extensions for providing text in character sets other than ASCII, non-text attachments to messages, multipart message bodies, and similar facilities 📎
some anomalies with MIME-compliant generation are pretty common too 🙈 this section talks about some of those and presents preferred ways to mitigate them.
8.1. Missing MIME-Version Field 🔍
any message that uses [MIME] constructs is required to have a MIME-Version header field 📋. without it, the Content-Type and associated fields have no semantic meaning 🫠
it's often observed that a message has a complete MIME structure but still lacks this header field 😅 it's prudent to disregard that absence and analyse the message as if the field were present, especially for agents trying to spot malicious material 🕵️♀️
and the absence of MIME-Version might be a sign of malicious intent, so extra scrutiny of the message may be warranted 🚩 these omissions aren't expected from compliant message generators.
8.2. Faulty Encodings 🧁
there have been a few different specifications of base64 over the years 🕰️ the implementation defined in [MIME] tells decoders to discard characters that aren't part of the base64 alphabet. other implementations consider an encoded body containing such characters to be completely invalid 🙅♀️. very early specifications of base64 (see [PEM89], for example, which was later obsoleted by [PEM93]) even allowed email-style comments inside base64-encoded data 🤯
the attack vector here is building a base64 body whose meaning changes depending on which possible decoding is used 😈 if a security analysis module wants to be thorough, it should consider scanning the possible outputs of the known decoding dialects, trying to anticipate how the MUA will interpret the data 🔬
9. Body Anomalies 🎀
9.1. Oversized Lines 📏
a message with a line of content that goes past 998 characters plus the line terminator (1000 total) violates Section 2.1.1 of [MAIL] 🙃 some handling agents may not look at anything in a single line past the first 998 bytes, which gives bad actors a chance to hide malicious content 🫣
there's no specified way to handle these messages, other than noting that they're non-compliant and either rejecting them or rewriting the oversized line so the message becomes compliant 🤷♀️
to make sure long lines don't stop analysis of potentially malicious data, handling agents are strongly encouraged to take one of these actions:
- break such lines into multiple lines at a position that doesn't change the semantics of the text being altered 🪓. for example, break an oversized line at a spot where a [URI] doesn't span two lines (splitting it could stop the URI from being identified properly) 🔗
- rewrite the MIME part (or the entire message, if it isn't MIME) that holds the excessively long line, using a content encoding that breaks the line during transmission but still leaves the line intact when decoded for the user to see 🧩. both of the encodings declared in [MIME] can do this.
10. Security Considerations 🔐
okay this section is the main character 👑 the discussions of the anomalies above and their prescribed solutions are themselves security considerations 😭. the practices in this document are generally seen as attempts to fix security considerations that already exist, rather than adding new ones 🛡️. however, some of the attacks described here may not have shown up in previous email specifications 💀 so stay alert, bestie 🚨
11. References 📚
11.1. Normative References 💖
- [EMAIL-ARCH] Crocker, D., "Internet Mail Architecture", RFC 5598, July 2009.
- [MAIL] Resnick, P., "Internet Message Format", RFC 5322, October 2008.
- [MIME] Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996.
11.2. Informative References 🧚
- [BINARYSMTP] Vaudreuil, G., "SMTP Service Extensions for Transmission of Large and Binary MIME Messages", RFC 3030, December 2000.
- [CHARSET] Melnikov, A. and J. Reschke, "Update to MIME regarding "charset" Parameter Handling in Textual Media Types", RFC 6657, July 2012.
- [DKIM] Crocker, D., Ed., Hansen, T., Ed., and M. Kucherawy, Ed., "DomainKeys Identified Mail (DKIM) Signatures", RFC 6376, September 2011.
- [DSN] Moore, K. and G. Vaudreuil, "An Extensible Message Format for Delivery Status Notifications", RFC 3464, January 2003.
- [EAI] Yang, A., Steele, S., and N. Freed, "Internationalized Email Headers", RFC 6532, February 2012.
- [ISO-2022-CN] Zhu, HF., Hu, DY., Wang, ZG., Kao, TC., Chang, WCH., and M. Crispin, "Chinese Character Encoding for Internet Messages", RFC 1922, March 1996.
- [ISO-2022-JP] Murai, J., Crispin, M., and E. van der Poel, "Japanese Character Encoding for Internet Messages", RFC 1468, June 1993.
- [ISO-2022-KR] Choi, U., Chon, K., and H. Park, "Korean Character Encoding for Internet Messages", RFC 1557, December 1993.
- [ISO-2022] ISO/IEC, "Information technology -- Character code structure and extension techniques", ISO/IEC 2022, 1994,
<http://www.iso.org/iso/catalogue_detail.htm?csnumber=22747>. - [MIME3] Moore, K., "MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text", RFC 2047, November 1996.
- [PEM89] Linn, J., "Privacy Enhancement for Internet Electronic Mail: Part I -- Message Encipherment and Authentication Procedures", RFC 1113, August 1989.
- [PEM93] Linn, J., "Privacy Enhancement for Internet Electronic Mail: Part I: Message Encryption and Authentication Procedures", RFC 1421, February 1993.
- [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -- Communication Layers", RFC 1122, October 1989.
- [RFC2822] Resnick, P., Ed., "Internet Message Format", RFC 2822, April 2001.
- [RFC733] Crocker, D., Vittal, J., Pogran, K., and D. Henderson, Jr., "Standard for the Format of Internet Text Messages", RFC 733, November 1977.
- [SMTP] Klensin, J., "Simple Mail Transfer Protocol", RFC 5321, October 2008.
- [URI] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", RFC 3986, January 2005.
- [UTF-8] Yergeau, F., "UTF-8, a transformation format of ISO 10646", RFC 3629, 2003.
Appendix A. Acknowledgements ✨
okay bestie, the authors gave a shoutout to the people who reviewed this proposal and gave constructive criticism 🫶 that's Dave Cridland, Dave Crocker, Jim Galvin, Tony Hansen, John Levine, Franck Martin, Alexey Melnikov, and Timo Sirainen. literally the squad that made this better 💖
Authors' Addresses 🎀
Murray S. Kucherawy EMail: superuser@gmail.com
Gregory N. Shapiro EMail: gshapiro@proofpoint.com
Ned Freed EMail: ned.freed@mrochek.com
that's the whole doc, bestie!! 🌈💅
📦 where this page comes from
- ietf: rfc7103 2014-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms