RFC 5987

Character Set and Language Encoding for Hypertext Transfer Protocol (HTTP) Header Field Parameters

📅 published August 2010, from ietf

RFC 5987 Glow-Up 💖

okay bestie, let's get into it 🫶 this one is RFC 5987: Character Set and Language Encoding for Hypertext Transfer Protocol (HTTP) Header Field Parameters. It's by J. Reschke from greenbytes, it's a Standards Track doc, and it's dated August 2010. The ISSN is 2070-1721. The header doesn't list any obsoletes or updates, so it's standing on its own 🌸

Abstract ✨

so HTTP header parameters normally can't carry characters outside ISO-8859-1, which is kind of a bummer 🥲 RFC 2231 has a fix for this, but it's built for MIME headers. This doc gives us an encoding for HTTP header fields that works with a profile of RFC 2231's encoding 💅

Status of This Memo 🎀

this is an Internet Standards Track document. It came from the Internet Engineering Task Force (IETF), it has the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication. More on Internet Standards is in Section 2 of RFC 5741, and the current status, errata, and feedback info lives at http://www.rfc-editor.org/info/rfc5987.

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were on the publication date. Read those carefully, because they spell out your rights and restrictions. Code Components pulled out of this doc must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License 📜

Table of Contents 🧚

1. Introduction ....................................................2
2. Notational Conventions ..........................................2
3. Comparison to RFC 2231 and Definition of the Encoding ...........3
   3.1. Parameter Continuations ....................................3
   3.2. Parameter Value Character Set and Language Information .....3
        3.2.1. Definition ..........................................3
        3.2.2. Examples ............................................6
   3.3. Language Specification in Encoded Words ....................6
4. Guidelines for Usage in HTTP Header Field Definitions ...........7
   4.1. When to Use the Extension ..................................7
   4.2. Error Handling .............................................7
5. Security Considerations .........................................8
6. Acknowledgements ................................................8
7. References ......................................................8
   7.1. Normative References .......................................8
   7.2. Informative References .....................................9

1. Introduction 🌈

by default, message header field parameters in HTTP ([RFC2616]) messages can't carry characters outside the ISO-8859-1 character set ([ISO-8859-1]). RFC 2231 ([RFC2231]) has an encoding mechanism for MIME headers, and this doc gives us an encoding that works in HTTP header fields and is compatible with a profile of RFC 2231's encoding 🎀

Note: in the rest of this doc, RFC 2231 only shows up to explain why we picked the features we did. So those references are purely informative. 📝
Note: this encoding does NOT apply to message payloads sent over HTTP, like when you're using the media type "multipart/form-data" ([RFC2388]). Payloads are a different vibe 🙅‍♀️

2. Notational Conventions 💅

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this doc are to be interpreted as described in [RFC2119]. Those are the official BCP 14 words, so pay attention to them 👑

this spec uses the ABNF (Augmented Backus-Naur Form) notation from [RFC5234]. It includes these core rules by reference, as defined in [RFC5234], Appendix B.1: ALPHA (letters), DIGIT (decimal 0-9), HEXDIG (hexadecimal 0-9/A-F/a-f), and LWSP (linear whitespace) ✨

quick note on terminology: this spec says "character set" to match other IETF specs like RFC 2277 (see [RFC2277], Section 3). A more accurate term would be "character encoding", which is a mapping of code points to octet sequences 🤓

3. Comparison to RFC 2231 and Definition of the Encoding 🦋

RFC 2231 defines several extensions to MIME. The sections below talk about whether and how those apply to HTTP header fields 💖

in short:

  • Parameter Continuations aren't needed (Section 3.1),
  • Character Set and Language Information are useful, so a simple subset is specified (Section 3.2), and
  • Language Specifications in Encoded Words aren't needed (Section 3.3).

3.1. Parameter Continuations 🧩

Section 3 of [RFC2231] has a mechanism for the length limits that apply to MIME headers. Those limits don't apply to HTTP ([RFC2616], Section 19.4.7), so we're good 🙌

so parameter continuations are NOT part of the encoding this spec defines 🚫

3.2. Parameter Value Character Set and Language Information 🌸

Section 4 of [RFC2231] explains how to embed language info into parameter values and how to encode non-ASCII characters, while dealing with the restrictions in both MIME and HTTP header parameters 💅

BUT RFC 2231 doesn't name a mandatory-to-implement character set, so senders have a hard time picking one. So recipients implementing this spec MUST support the character sets "ISO-8859-1" [ISO-8859-1] and "UTF-8" [RFC3629]. No more guessing, bestie 🫶

also, RFC 2231 lets you leave out the character set info. This encoding doesn't let you do that 🙅‍♀️

3.2.1. Definition 📜

the syntax for parameters is defined in Section 3.6 of [RFC2616] (with the RFC 2616 implied LWS translated to RFC 5234 LWSP):

  parameter     = attribute LWSP "=" LWSP value

  attribute     = token
  value         = token / quoted-string

  quoted-string = <quoted-string, defined in [RFC2616], Section 2.2>
  token         = <token, defined in [RFC2616], Section 2.2>

to add character set and language info, this spec modifies the RFC 2616 grammar to be:

  parameter     = reg-parameter / ext-parameter

  reg-parameter = parmname LWSP "=" LWSP value

  ext-parameter = parmname "*" LWSP "=" LWSP ext-value

  parmname      = 1*attr-char

  ext-value     = charset  "'" [ language ] "'" value-chars
                ; like RFC 2231's <extended-initial-value>
                ; (see [RFC2231], Section 7)

  charset       = "UTF-8" / "ISO-8859-1" / mime-charset

  mime-charset  = 1*mime-charsetc
  mime-charsetc = ALPHA / DIGIT
                / "!" / "#" / "$" / "%" / "&"
                / "+" / "-" / "^" / "_" / "`"
                / "{" / "}" / "~"
                ; as <mime-charset> in Section 2.3 of [RFC2978]
                ; except that the single quote is not included
                ; SHOULD be registered in the IANA charset registry

  language      = <Language-Tag, defined in [RFC5646], Section 2.1>

  value-chars   = *( pct-encoded / attr-char )

  pct-encoded   = "%" HEXDIG HEXDIG
                ; see [RFC3986], Section 2.1

  attr-char     = ALPHA / DIGIT
                / "!" / "#" / "$" / "&" / "+" / "-" / "."
                / "^" / "_" / "`" / "|" / "~"
                ; token except ( "*" / "'" / "%" )

so a parameter is either a regular one (reg-parameter), as defined back in Section 3.6 of [RFC2616], or an extended one (ext-parameter) 🦋

extended parameters are the ones where the left-hand side of the assignment ends with an asterisk character ⭐

the value part of an extended parameter (ext-value) is a token with three pieces, separated by single quote characters: the REQUIRED character set name (charset), the OPTIONAL language info (language), and a character sequence for the actual value (value-chars). Both character set names and language tags are restricted to US-ASCII and are matched case-insensitively (see [RFC2978], Section 2.3 and [RFC5646], Section 2.1.1). Main character energy 👑

inside the value part, any character that isn't in attr-char gets encoded into an octet sequence using the specified character set. That octet sequence is then percent-encoded as specified in Section 2.1 of [RFC3986] 🍓

producers MUST use either the "UTF-8" ([RFC3629]) or the "ISO-8859-1" ([ISO-8859-1]) character set. Extension character sets (mime-charset) are reserved for future use 🔮

Note: recipients should be ready to handle encoding errors, like malformed or incomplete percent escape sequences, or octet sequences that can't be decoded, in a robust way. This spec doesn't mandate any specific behavior. For instance, all of these strategies are acceptable: - ignoring the parameter, - stripping a non-decodable octet sequence, - substituting a non-decodable octet sequence with a replacement character, such as the Unicode character U+FFFD (Replacement Character).
Note: the RFC 2616 token production ([RFC2616], Section 2.2) is different from the one RFC 2231 uses (imported from Section 5.1 of [RFC2045]) because curly braces ("{" and "}") are excluded. So those two characters are excluded from the attr-char production too 🙈
Note: the <mime-charset> ABNF defined here is different from the one in Section 2.3 of [RFC2978] because it does NOT allow the single quote character (see also RFC Errata ID 1912 [Err1912]). In practice, no character set names using that character have been registered as of this writing 💀

3.2.2. Examples 🧁

non-extended notation, using "token":

  foo: bar; title=Economy

non-extended notation, using "quoted-string":

  foo: bar; title="US-$ rates"

extended notation, using the Unicode character U+00A3 (POUND SIGN):

  foo: bar; title*=iso-8859-1'en'%A3%20rates
Note: the Unicode pound sign U+00A3 got encoded into the single octet A3 using ISO-8859-1, then percent-encoded. The space character also got encoded as %20, since it isn't in attr-char. 🙌

extended notation, using the Unicode characters U+00A3 (POUND SIGN) and U+20AC (EURO SIGN):

  foo: bar; title*=UTF-8''%c2%a3%20and%20%e2%82%ac%20rates
Note: the Unicode pound sign U+00A3 got encoded into the octet sequence C2 A3 using UTF-8, then percent-encoded. The Unicode euro sign U+20AC was encoded into E2 82 AC the same way, then percent-encoded. Also, HEXDIG allows both lowercase and uppercase, so recipients MUST understand both. And the language info is optional, but the character set is not. 🫶

3.3. Language Specification in Encoded Words

Okay so, section 5 of [RFC2231] stretches the encoding from [RFC2047] so you can also specify a language inside encoded words 🌸. HTTP/1.1 does mention RFC 2047 ([RFC2616], Section 2.2), but it's honestly kinda unclear which header field that's supposed to apply to, and nobody's sure if it's actually used in the wild 🙈. You can read more in the tracker ticket at <http://tools.ietf.org/wg/httpbis/trac/ticket/111>.

So this spec just doesn't include that feature 💅.

4. Guidelines for Usage in HTTP Header Field Definitions ✨

Bestie, if your header field definition uses the extensions from Section 3.2, you ought to say so clearly 🫶. The easiest way to do that is to normatively reference this spec and drop the ext-value production into the ABNF for your header field 💖.

Here's a cute example:

  foo-header  = "foo" LWSP ":" LWSP token ";" LWSP title-param
  title-param = "title" LWSP "=" LWSP value
              / "title*" LWSP "=" LWSP ext-value
  ext-value   = <see RFC 5987, Section 3.2>
Note: the Parameter Value Continuation feature from Section 3 of [RFC2231] makes it impossible to have multiple extended parameters with identical parmname components, because the continuations would get ambiguous 😭. So if you're using this extension, you're advised to disallow that case so you stay compatible with RFC 2231 🎀.

4.1. When to Use the Extension 🦋

Section 4.2 of [RFC2277] says protocol elements with human-readable text need to be able to carry language info 🌈. So the ext-value production ought to always be used when the parameter value is textual and you actually know its language 💜.

It also ought to be used whenever the parameter value needs characters that aren't in the US-ASCII ([USASCII]) character set 🍓. Like, it would be totally unacceptable to define a new parameter that only works with a subset of Unicode 🙅‍♀️.

4.2. Error Handling 🧚

Header field specs have to define whether multiple instances of parameters with identical parmname components are allowed, and how they get processed 👑. This spec suggests that a parameter using the extended syntax takes precedence ⭐. That way, producers can send both formats without breaking recipients that don't understand the extended syntax yet 🫶.

Example:

  foo: bar; title="EURO exchange rates";
            title*=utf-8''%e2%82%ac%20exchange%20rates

So in this case, the sender includes an ASCII version of the title for legacy recipients, but also an internationalized version for recipients that understand this spec 🔥. Those newer recipients obviously ought to prefer the new syntax over the old one 💅.

Note: when this was written, lots of implementations failed to ignore the form they didn't understand, or prioritized the ASCII form even though the extended syntax was there 💀.

5. Security Considerations 🚨

Okay this one is serious 😭. The format here lets you transport non-ASCII characters, which means "spoofing" is possible, where what's displayed looks like something other than what it actually is 👀.

There are also known attack scenarios involving UTF-8 decoding 💀.

For more on both of these, see Section 10 of [RFC3629] 📚.

Also, the extension in this document lets you transport multiple language variants for a single parameter. That might allow spoofing attacks where the different language versions of the same parameter aren't actually equivalent 🙈. Whether that's a real attack depends on which parameter we're talking about 🫣.

6. Acknowledgements 🫶

Huge thanks to Martin Duerst and Frank Ellermann for helping figure out the ABNF details 💖, to Graham Klyne and Alexey Melnikov for general review 🌸, to Chris Newman for pointing out an RFC 2231 incompatibility 👑, and to Benjamin Carlyle and Roar Lauritzsen for implementer feedback 🧚. Iconic team 💅✨.

7. References 📚

7.1. Normative References

[ISO-8859-1]  International Organization for Standardization,
              "Information technology -- 8-bit single-byte coded
              graphic character sets -- Part 1: Latin alphabet No.
              1", ISO/IEC 8859-1:1998, 1998.

[RFC2119]     Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2616]     Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
              Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
              Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.

[RFC2978]     Freed, N. and J. Postel, "IANA Charset Registration
              Procedures", BCP 19, RFC 2978, October 2000.

[RFC3629]     Yergeau, F., "UTF-8, a transformation format of ISO
              10646", RFC 3629, STD 63, November 2003.

[RFC3986]     Berners-Lee, T., Fielding, R., and L. Masinter,
              "Uniform Resource Identifier (URI): Generic Syntax",
              RFC 3986, STD 66, January 2005.

[RFC5234]     Crocker, D., Ed. and P. Overell, "Augmented BNF for
              Syntax Specifications: ABNF", STD 68, RFC 5234,
              January 2008.

[RFC5646]     Phillips, A., Ed. and M. Davis, Ed., "Tags for
              Identifying Languages", BCP 47, RFC 5646,
              September 2009.

[USASCII]     American National Standards Institute, "Coded Character
              Set -- 7-bit American Standard Code for Information
              Interchange", ANSI X3.4, 1986.

7.2. Informative References 🌈

[Err1912]     RFC Errata, Errata ID 1912, RFC 2978,
              <http://www.rfc-editor.org>.

[RFC2045]     Freed, N. and N. Borenstein, "Multipurpose Internet
              Mail Extensions (MIME) Part One: Format of Internet
              Message Bodies", RFC 2045, November 1996.

[RFC2047]     Moore, K., "MIME (Multipurpose Internet Mail
              Extensions) Part Three: Message Header Extensions for
              Non-ASCII Text", RFC 2047, November 1996.

[RFC2231]     Freed, N. and K. Moore, "MIME Parameter Value and
              Encoded Word Extensions: Character Sets, Languages, and
              Continuations", RFC 2231, November 1997.

[RFC2277]     Alvestrand, H., "IETF Policy on Character Sets and
              Languages", BCP 18, RFC 2277, January 1998.

[RFC2388]     Masinter, L., "Returning Values from Forms: multipart/
              form-data", RFC 2388, August 1998.

Author's Address 🖋️

Julian F. Reschke
greenbytes GmbH
Hafenweg 16
Muenster, NW  48155
Germany

EMail: julian.reschke@greenbytes.de
URI:   http://greenbytes.de/tech/webdav/

That's the whole thing, bestie 🌈💖✨ RFC 5987 is a wrap!!

📦 where this page comes from