RFC 886

RFC 886: Proposed Standard for Message Header Munging ✨

okay bestie, this is RFC 886 🎀 It's a Proposed Standard (so the category is Proposed Standard) and it's dated Thu Dec 15 03:37:52 1983. It was written by Marshall T. Rose of the Department of Information and Computer Science, University of California, Irvine, Irvine, CA 92717, and you can reach him at MRose.UCI@Rand-Relay 💌 The doc doesn't list anything it obsoletes or updates, so there's no drama there 🙈

This memo proposes a standard for the ARPA Internet community. If this proposal is adopted, hosts on the ARPA Internet that do message translation would be expected to adopt and implement this standard 🌈

Introduction 🦋

This memo explains the rules for when mail gets transformed from one standard format into another 💅 It's only about text messages (computer network mail, aka electronic mail) that travel across the ARPA Internet. It is NOT a replacement or amendment for the "Standard for the Format of ARPA Internet Text Messages", RFC822 🙅‍♀️ Instead it zooms in on one particular aspect of mail and gives implementors of transport agents and user agents that do message munging a conceptual and practical foundation to work with ✨

Even though this memo was made to be used with the 822 standard, you don't need to know 822 to use it 🫶 The rest of this section recaps some key 822 concepts and explains how they connect to what we're doing here.

So messages are basically an envelope plus contents 📬 The envelope gets handled only by transport agents, and it holds everything the transport agents need to deliver the message to its recipients. This memo doesn't address the envelope directly, but we'll see that some of the rules we discuss later also apply to it 👀

The contents are a rigorously structured part called the headers, followed by a freely formatted part called the body. Honestly the body is completely uninteresting to us 😂 We only care about the headers. Each header is a field, its value, and a terminating end-of-line sequence. 822 covers things like continuation lines, the syntax that tells fields apart from values, and the syntax and semantics of values for various fields. For our purposes, we only need the idea that the headers section is one or more headers, which are split into one or more field/value pairs 💖

Now the term "message munging" means the actions a transport or user agent takes to change a message's contents from following one standard format to following another 🔥 822 calls this "Network-Specific Transformation". You might also hear "header munging" or "mail filtering". Whatever you call it, the key idea is that the message goes from its current format (the source message) to the structure the target standard requires ✨ For this memo, a "munging agent" is any entity that does message munging, and it can be part of either a transport or user agent 💅

Background 🌸

As more networks plug into the ARPA Internet community, their users will swap computer mail with other Internet hosts 💌 The 822 standard has to be strictly adhered to for mail traveling the ARPA Internet, but other networks might not adopt it internally 🤷‍♀️ Even so, it's really desirable to let mail flow between hosts that internally follow the standard and hosts that don't 💜 The 822 standard is very clear about this:

"This standard is NOT intended to dictate the internal formats used by sites, the specific message system features that they are expected to support, or any of the characteristics of user interface programs that create or read messages."

So basically, even hosts inside the ARPA Internet are allowed to use something other than 822 internally 🦋 But when they send mail to other ARPA Internet hosts, they are expressly required to use 822 😤 That's why it's not hard to picture message munging becoming a super common thing for transport and user agents 🔥

There are other reasons munging could get really widespread too, and CSnet is a perfect example 🌈 The CSnet relays give authorized mail access to the ARPA Internet for CSnet phonenet sites. CSnet sites aren't registered with the NIC, so they're often missing from the host tables of ARPA Internet sites 🙈 That means the addresses of mailboxes on CSnet phonenet sites are unknown to ARPA Internet sites. From an ARPA Internet site, sending to those addresses would be impossible, since the local transport agent has no handle on the destination hosts of the phonenet mailboxes 💀 And replying to such a message is simply not possible either 😭

To fix this, the transport agents on the CSnet relays do message munging on mail headed for the ARPA Internet 🎀 Phonenet addresses of the form mbox@host get transformed into mbox.host@relay, where relay is the ARPA Internet host name of the relay doing the transformation. Other addresses are left alone ✨ Now agents across the ARPA Internet can process these addresses, because the host-part is a known ARPA Internet host 🫶

The source-routing solution is hopefully going to be replaced by domain handling once domains are implemented in the ARPA Internet 🧚 When that happens, phonenet addresses of the form mbox@host will become mbox@host.CSNET. Even with that change (which can only be for the better, since source-routing causes a plethora of problems 🙄), message munging will still happen, because it will most likely be necessary to add domain names during message transmission (see section 6.2.2 of the 822 standard) 🍓

For a different reason, it's not unlikely that users will want to transform mail from their archives that follows an older standard into the current one 📼 There could be lots of reasons, but a common one is that a user wants to re-introduce the message into the transport system 🔁 The old message was perfectly valid when it was written (e.g., back in the days of the 733 standard), but it might no longer conform to the current standard (i.e., 822) 😅 In that case, a user agent would have to do message munging to make the message acceptable to the local transport agent 💅

To sum up, even in the most "homogeneous" environments, transport and user agents will still need message munging under certain conditions 🌸

Section 3.4.10 of the 822 standard briefly covers "Network-Specific Transformations" 🦋 Basically, 822 imagines a message traveling from net-A to reach net-B going through three phases:

• Transformation: The message is made to conform to net-A's standards
• Transformation Reversal: Net-A's idiosyncrasies are removed and the message now conforms to the 822 standard
• Transformation: The message is made to conform to net-B's standards

This memo only deals with this section of the 822 standard 🤓 822 uses end-of-line sequences as an example of where transformation might happen. That's a valid concern, but we're focused on higher-level constructs: fields and structured field values 💖

Scope 💅

This memo does NOT specify the exact transformation rules to use when munging a message from one standard to another 🙅‍♀️ What it does do is spell out the policies that should be followed when implementing any munging agent for the ARPA Internet ✨ Working out the formulas for munging between two given standards is left to the implementors of those munging systems, or to the writers of future RFCs 📝 So you can think of this memo as the philosophy and conceptual basis of message munging in the ARPA Internet 🌈

NOTE: It is critical that this position be understood. The actual policies used by domain-specific munging agents is completely beyond the scope of this memo.

For ease of explanation, some examples here use message munging between the ARPA Internet and the USENET distribution network 🗞️ But this memo should NOT be considered to specify how that particular munging activity should take place 🙈 That context was picked just for its familiarity and simplicity 🫶

Message Decomposition 🎀

A munging agent is all about transforming a message that follows a source standard into a message that follows a target standard 💫 This transformation happens at various levels, and four of them are covered here 🔥

• Field Transformation: For two standards, some fields may convey identical semantics but have different names. As standards progress, for example, the names of fields may change, but the presence of those fields and their contents continue to have the same meaning. For example, prior to 822 standard, some mailers considered the Remailed- prefix to have semantics equivalent to the 822 standard's Resent- prefix. In this circumstance, one aspect of message munging would be to simply substitute the field names. 🔄
• Value Transformation: The value of certain fields may be viewed as containing structured components. The syntax and semantics of these components may differ significantly between two formats. In this circumstance, one aspect of message munging would be to transform components from one representation to another. 🧩
• Field/Value Combination: The semantics of a given header in a particular standard may not be directly expressed using a single header from another standard. In this circumstance, one aspect of message munging would be to map the field/value of a header in the source message to any number of headers in the target message (or vice-versa). As expected, further complication could result by performing value transformation in addition to one-to-many or many-to-one field transformation. 🌀
• Header Ordering: Some standards may require that fields appear in a particular order in the headers part of the message. Others make no requirements as to the order in which the fields appear. In this circumstance, one aspect of message munging from the latter to the former standard would be to capture the essential information from the source message in order to construct the first field of the target message. As expected, further complication could result by requiring several field/values be consulted in the source message before sufficient context is present to construct the first field of the target message. 🧠

part 1 of 3 is done, bestie 🦋 more coming up next 💖

Canonical Forms ✨

okay so this is the big idea that makes the whole transformation thing work, bestie 💅. For a given message standard, every field and structured field value can be thought of as an object with a particular semantics, and that object can be written out as one or more strings. All of those strings have the same semantics because they all point at the same object 🦋. For example, in terms of the 822 standard, these two strings are semantically equivalent:

    The Protocol Police <NetSer@UCI>

    NetSer@UCI

For the purposes of this memo, a fully-qualified canonical form of an object is the simplest string that represents the full and complete meaning of that object. What "simple" means is, of course, open to interpretation 🤷‍♀️. Sometimes "simplest" means "shortest". Other times a longer, unabbreviated string is the "simpler" one. Either way, a canonical form is a representation of an object, and it holds the smallest amount of information required to fully describe what the object means. 🌸

It is not hard to figure out what a canonical form should describe for different objects. In terms of the 822 standard, the following should be considered as minimal definitions of canonical forms:

    object        specifies    contains
    ------        ---------    --------
    field        field-name    name
    address        mailbox        local-part
                    domain-part
    date        date-time     date-part
                    time-part

In terms of USENET, the following might be considered as minimal definitions of canonical forms:

    object        specifies    contains
    ------        ---------    --------
    field        field-name    name
    address        mailbox        user
                    route
    date        date-time     date-part
                    time-part

NOTE: This memo clearly has no authority to specify the minimal canonical forms for USENET. The above table is listed solely for the benefit of the examples which follow. 🙈

Conceptually, transformation of fields and structured field values happens between canonical forms. So to transform an address, you reduce the string representing the object down to its canonical form, which captures the essence of its meaning. That form is then transformed, somehow, into the equivalent canonical form for the target standard. This target canonical form can later be output as a string representation. 🧚

NOTE: This memo does not require that canonical forms be represented or otherwise implemented as strings. Nor does this standard require that strings be used during the transformation process. Thinking of a canonical form as a string is a convenient formalism only, not an implementational requirement. 💖

---

Transformation Rules 🎀

All of the forms of message decomposition discussed above can now be viewed as transformation between canonical forms. So it becomes necessary to think about how canonical forms should be manipulated during transformation. What rules get followed when you build an equivalent canonical form? There are several guidelines that must be followed, and we will characterize them in the following fashion:

Preservation of information

All attempts must be made to preserve all information contained in the original canonical form. This information can be super useful to the recipients of munged messages 🫶. Obviously, for two widely-differing formats this may not be possible. For example, some standards may not have a group addressing notation like the one present in the 822 standard, e.g., the notation

    List: Support@UCI, ZOTnet@UCI;

might not be permitted. If one were to consider membership in a group as part of an address' canonical form, then this portion of the canonical form could not be transformed to the other standard. 😭

The 822 standard supports a liberal commenting convention which might prove quite useful in preserving information. Implementors may wish to consider capturing the original information in commentary text. For example, if the USENET address

    mark@cbosgd.UUCP (Mark Horton)

had the USENET canonical of

     user:  mark
    route:  ucbvax!ihnp4!cbosgd

and if the corresponding 822 canonical was

      local-part:    ucbvax!ihnp4!cbosgd!mark
     domain-part:    USENET.UCI

then it would not be unreasonable for an implementation to output this canonical form as

    "mark@cbosgd.UUCP" <ucbvax!ihnp4!cbosgd!mark@USENET.UCI>

NOTE: Implementors should exercise extreme caution in using a policy such as this. Information placed between commentary delimiters must still conform to the target standard at the syntactic level. ⚠️

Note however that in the above example, the commentary information "(Mark Horton)" was discarded. This practice is strongly discouraged 💀. Although the canonical form for an object does not rely on commentary information as a necessary part, implementors are encouraged to preserve this information whenever possible.

Finally, preservation of information requires preservation of case at all costs. Only under the most restrictive of circumstances should an implementation change the case of the strings output for a canonical form. 👑

Re-Formatting

Ideally, the target message should have the exact horizontal and vertical padding as the source message 🍓. Because a string representing the source canonical form of an object may not be the same length as the string representing the target canonical form, the number of characters on each physical and logical line in the headers may be different.

The 822 standard supports a header folding convention which permits long field/value pairs to be represented on more than one physical line. When a new canonical form is output to the target message, the resulting field/value pair may be longer than the number of characters that antiquated display devices can show on a single line. The 822 standard suggests 65 or 72 characters-per-line as a metric for this limitation. Although not required, message munging agents may re-fold headers if (and only if) this limitation is exceeded. Note however that under no circumstances should a header be re-folded if it was not munged. Refolding without munging may occur on behalf of some transport or user agent, but it may not occur on behalf of a munging agent. Put more simply, this memo does not authorize or forbid such activity, although it does discourage it. 🙃

Error Recovery

The preceding discussion has been made under the assumption that the objects composing the field/value pairs of the source message have conformed to the source standard. It is an unfortunate reality that this may not be the case 😩. In fact, for those standards which are poorly specified (if at all), determining that an object is improperly constructed might be quite difficult. In addition, it is possible, though hopefully extremely improbable, that a target canonical form does not exist for a particular source canonical form. In these cases, munging agents must be able to recover.

At this point, we introduce two extension fields for the 822 standard. As such, these fields are hereby designated as "reserved" and may not be used for other purposes. These fields are:

    Illegal-Field
    Illegal-Object

The syntax of these fields is as follows:

    munge-field =
         "Illegal-Field"    ":" *text
      /  "Illegal-Object"    ":" *text

    munge-object =
         <a "field-name", the exact field-names which are
          valid will be presented later>

The semantics of these fields are as follows:

An Illegal-Field header should be introduced when a header-line which does not conform to the source standard is found in the source message. Illegal-Field should be used only when a header-line is so poorly formed as to prevent recognition of the field in the header-line. For example, if the line lacks a colon, or has a poorly formed field-name, then it should not be output to the target message and a new header-line should be introduced in its place. This header-line has Illegal-Field as its field and contains the offending line as its value. Illegal-Field should not be used if the field can be identified but the value is poorly formed. 🧐

An Illegal-Object header should be introduced when an object in the source message can not be parsed into a canonical form, or if the canonical form it represents has no corresponding target canonical form. The offending object should not be output to the target message in the header-line in which it occurs. If the header-line now contains no objects, then the header-line should not be output to the target message either. Then, an Illegal-Object field should be introduced into the target message. The value of this Illegal-Object field should at the very minimum contain the name of the field that contained the object, the object in question, and an explanation as to why the object was illegal. Alternately, the value of this Illegal-Object field should consist of the entire header-line (field and value) that contained the object in question, along with an explanation as to why the object was illegal. 📝

NOTE: In the circumstance where multiple objects exist in a single header-line in the source message, and one of those objects is determined to be illegal, the actual policy used in determining how much information can be considered to be "uncorrupted" is left to the implementors. Munging agents which use sophisticated parsers may attempt to recover in mid-stream (so to speak) and continue parsing objects on the header-line. Other agents may wish to continue recovering with the next header-line in the source message. Regardless of the policy used, the agent must present the contents of the entire header-line in the associated Illegal-Object header. 🧩

Implementations should not take extraordinary measures to perform syntax/semantics checking of the source message -- only those fields which must be examined should be rigorously checked. This memo strongly discourages any additional examination. It is not the intention of this memo to suggest that composing agents should produce messages which do not conform to the source standard. A composing agent should not expect a munging agent to enforce adherence to the source standard. 🫠

Introduction of Information

Munging agents are authorized to introduce a "Received" header into the target message when a message is transformed. 📬

NOTE: Adding a "Received" header is entirely optional. This memo strongly recommends that this header be introduced whenever some munging (translation of addresses and/or dates) occurs.

NOTE: Although this memo does not specify the position that the introduced header should have in relation to the other fields in the target message, it is strongly recommended that the introduced header be grouped with the other "Received" headers, at the very beginning of the message. 🌈

When introducing a "Received" field, three phrases, which are normally optional in such a field, should be specified by the munging agent. These are:

"from" domain        # the source domain
"by"   domain        # the target domain
"with" protocol        # the munging agent's host

Okay bestie, here's a super clear example 🦋 Say we have a munging agent on the UCI host, and that agent looks after a USENET/ARPA boundary. When UCI gets a message from the USENET domain that's headed for the ARPA domain, this happens:

First, the UCI mailsystem prepends this header:

Received: from nma by UCI with UUCP; 15 Dec 83 03:53:00 PST

Second, the munging agent prepends this one when it transforms the message:

Received: from USENET by ARPA with UCI; 15 Dec 03:54:00 PST

Finally, the UCI mailsystem delivers the message to the right ARPA mailsystem, which prepends this:

Received: from UCI by ISIF with SMTP; 15 Dec 83 03:55:00 PST

Ngl, this example gets a little clearer if the domains are qualified a bit more 💅 The first three lines of the message could look like this:

Received: from UCI.ARPA by ISIF.ARPA; 15 Dec 83 03:55:00 PST
Received: from USENET by ARPA with UCI; 15 Dec 03:54:00 PST
Received: from nma.USENET by UCI.USENET; 15 Dec 83 03:53:00 PST

The key thing to notice is that the munging agent used the "from" and "by" clauses to show which domain boundary was crossed, and used the "with" clause to name itself 🎀 Since the agent munges the message according to some set of transformation rules, it's actually running a "mail protocol", so it's totally justified in identifying itself in the "with" clause ✨

---

Objects of Interest

At the moment, only three types of objects are of interest: fields, addresses, and dates 🌸 In the context of the 822 standard, header-lines containing the following fields are viewed as appropriate for transformation:

Address Fields:    From, Sender, Reply-To, To, cc, Bcc,
        and any of these fields with the Resent- prefix

Date Fields:    Date, Resent-Date

So the definition of munge-object, in 822 terms, is:

munge-object =
     "From"
  /  "Sender"
  /  "Reply-To"
  /  "To"
  /  "cc"
  /  "Bcc"
  /  "Resent-From"
  /  "Resent-Sender"
  /  "Resent-Reply-To"
  /  "Resent-To"
  /  "Resent-cc"
  /  "Resent-Bcc"
  /  "Date"
  /  "Resent-Date"

NOTE: The list of munge-objects is extensible. For the purposes of this memo, the above fields are defined as the MINIMUM list of munge-objects for the 822 standard. Implementors are encouraged to introduce other fields to the list of munge-objects as their munging agents require. These additions should also be registered with the revisions of this memo as they gain popularity.

For the rest of this memo, an address header-line is any header-line in the source message whose field component is one of the fields above listed as an address field. Similarly, a date header-line is any header-line in the source message whose field component is one of the fields above listed as a date field 🦋

If address munging is performed, then all addresses contained in all address header-lines must be munged. It is expressly forbidden to perform address munging on the source message without performing address munging on every address header-line. Further, it is expressly forbidden to munge some, but not all, of the addresses in any address header-line. All addresses in all of the message's address header-lines must be address munged. If address munging is not performed, then these header-lines need not be considered for munging. Literally no half-munging allowed 🙈

A similar requirement is made for date munging. If date munging is performed, then all instances of header-lines whose field is Date or Resent-Date must be fully date munged 🫶

NOTE: Certain fields are to be excluded from munging of any sort, and all munging agents must preserve their contents exactly. At present, there is one such field: "Received". The contents of this field should ALWAYS be preserved for trace and debugging purposes 🔥

---

Bibliography

D.H. Crocker, "Standard for the Format of ARPA Internet Text Messages", RFC 822, (August, 1982).

M.R. Horton, "Standard for the Interchange of USENET Messages", RFC 850, (June, 1983).

M.T. Rose, "Achieving Interoperability between two Domains -- Connecting the ZOTnet and UUCP Computer Mail Networks", Technical Report Number 201, Department of Information and Computer Science, University of California, Irvine, (January, 1983).

P.V. Mockapetris, "Domain Names -- Concepts and Facilities", RFC 882, (November, 1983).

---

Appendices

Minimal Canonical Forms

This memo defines the minimal canonical forms for the 822 standard. Implementations may wish to augment these forms with additional information that may be present in the source message. An earlier example suggested that group membership might be part of an address' canonical form. Further, since the 822 standard permits routes to be specified in addresses, e.g.,

Fred Rated <@ISI-Troll.ARPA,@UCI-750a.UCI: FRated@UCI-750b>

Perhaps they too should be considered part of the 822 address' canonical form? 🤔

This memo makes no such requirement. If implementations wish to make use of this additional information, then they are free to do so. This practice is neither encouraged nor discouraged. In short, the spirit of this memo is to require those minimal components required by the 822 standard, nothing more 💜