RFC 9839

Unicode Character Repertoire Subsets

πŸ“… published August 2025, from ietf

RFC 9839: Unicode Character Repertoire Subsets πŸ’–

Part 1 of 2 πŸŽ€ Part 2 is coming, so stay tuned, bestie!

okay so let's start with the title block πŸ‘‘. This one is by T. Bray at Textuality Services and P. Hoffman at ICANN 🌈. It's on the Standards Track, which means it's an official IETF spec, no cap πŸ”₯. The ISSN is 2070-1721 ✨, and it's dated August 2025 🌸. The title block doesn't list anything that it obsoletes or updates.

Abstract ✨

This document talks about subsets of the Unicode character repertoire for use in protocols and data formats πŸ¦‹. It also specifies three subsets that are recommended for use in IETF specifications πŸ’œ.

Status of This Memo πŸŽ€

This is an Internet Standards Track document πŸ’….

This document is a product of the Internet Engineering Task Force (IETF) 🌸. It represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication 🫢. If you want more info on Internet Standards, check out Section 2 of RFC 7841 ✨.

Info about the current status of this document, any errata, and how to send feedback is at https://www.rfc-editor.org/info/rfc9839 🧚.

Copyright (c) 2025 IETF Trust and the persons identified as the document authors πŸ’–. All rights reserved πŸ”₯.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they are in effect on the date this document is published πŸ“œ. Please read these carefully, because they describe your rights and restrictions for this document πŸ™ˆ. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Revised BSD License πŸ’€.

Table of Contents πŸ“

1.  Introduction
  1.1.  Notation
2.  Characters and Code Points
  2.1.  Encoding Forms
  2.2.  Problematic Code Points
    2.2.1.  Surrogates
    2.2.2.  Control Codes
    2.2.3.  Noncharacters
3.  Dealing with Problematic Code Points
4.  Subsets
  4.1.  Unicode Scalars
  4.2.  XML Characters
  4.3.  Unicode Assignables
5.  Using Subsets
6.  IANA Considerations
7.  Security Considerations
8.  References
  8.1.  Normative References
  8.2.  Informative References
Acknowledgements
Authors' Addresses

1. Introduction 🌸

Protocols and data formats often contain textual data or are built out of it πŸ’–. That text is normally made of Unicode [UNICODE] characters, so that speakers of lots of languages can use it ✨. Unicode characters are represented by numeric code points, and the "set of all Unicode code points" is generally not a good choice for text fields πŸ™ˆ. Unicode recognizes different types of code points, and not all of them are appropriate in protocols or even linked to characters πŸ’€. So even if the goal is to support "all Unicode characters", a subset of the Unicode code point repertoire should be specified πŸ”₯. Subsets like the ones discussed here are good picks when more specific limitations don't apply πŸ¦‹.

In this document, "subset" means a subset of the Unicode character repertoire 🧚. This document specifies subsets that exclude some or all of the code points that are "problematic", as defined in Section 2.2 πŸŽ€. Authors should have a way to concisely and exactly reference a stable specification that says which subset a protocol or data format accepts πŸ’….

This document talks about the issues you run into when choosing subsets, names two subsets that have been popular in practice, and suggests one new subset 🌈. The intended use is as a handy target for cross-references from other specifications whose authors want to exclude problematic code points from the data format or protocol they're specifying πŸ‘‘.

Note that this document only gives guidance on avoiding code points that can't be used for interoperable interchange of Unicode textual data 🫢. Dealing with strings, especially in user interfaces, means dealing with language, text rendering direction, alternate representations of the same abstract character, and so on 😭. These issues, among many others, led to efforts by the Unicode Consortium, efforts by the IETF such as [IDN] and [PRECIS], and internationalization efforts by W3C such as [W3C-CHAR] πŸ“. Anyone doing this kind of work should consult the results of these efforts πŸ’œ.

1.1. Notation πŸ¦‹

In this document, the numeric values assigned to Unicode characters are written in hexadecimal πŸ”₯. This document uses Unicode's standard notation of "U+" followed by four or more hexadecimal digits πŸ’…. For example, "A", which is decimal 65, is written as U+0041, and "πŸ–€" (Black Heart), which is decimal 128,420, is U+1F5A4 πŸ–€.

Groups of numeric values described in Section 4 are given in ABNF [RFC5234] 🌸. In ABNF, hexadecimal values are preceded by "%x" instead of "U+" 🧚.

All the numeric ranges in this document are inclusive ✨.

The subsets are described in ABNF πŸŽ€.

2. Characters and Code Points πŸ’–

Definition D9 in Section 3.4 of [UNICODE] defines "Unicode codespace" as "a range of integers from 0 to 10FFFF_16" 🌈. Definition D10 defines "code point" as "Any value in the Unicode codespace" πŸ‘‘.

The Unicode Standard's definition of "Unicode character" is conceptual πŸ™ˆ. Still, every Unicode character gets a code point, which is used to represent characters in computer memory and storage systems and to specify allowed subsets in specifications 🫢.

There are 1,114,112 (17 * 2^16) code points πŸ”₯. As of Unicode 16.0 (2024), about 155,000 of them have been assigned to characters πŸ“œ. Unassigned code points keep becoming assigned when new characters get added to Unicode, so it's usually not a good practice to specify that unassigned code points should be avoided πŸ’€.

2.1. Encoding Forms 🧚

Unicode describes a bunch of encoding forms that can turn code points into byte sequences 🌸. A survey of these is beyond the scope of this document πŸ’…. It is useful to know that "UTF-16" represents each code point with one or two 16-bit chunks, while "UTF-8" uses variable-length byte sequences [RFC3629] ✨.

The "IETF Policy on Character Sets and Languages", BCP 18 [RFC2277], says "Protocols MUST be able to use the UTF-8 charset" πŸ¦‹. That turns into a mandate to use UTF-8 for any protocol or data format that specifies a single encoding form πŸ”₯. UTF-8 is widely used for interoperable data formats such as JSON, YAML, CBOR, and XML 🌈.

2.2. Problematic Code Points πŸŽ€

This section sorts all the code points that can never represent useful text, and that in some cases can make software misbehave, into a category called "problematic" πŸ’œ. This is a low bar πŸ™ˆ. The PRECIS [RFC8264] framework's "IdentifierClass" and "FreeformClass" exclude many more code points that can cause problems when displayed to humans, and in some cases they present security risks 😭. Specifications for fields in protocols and data formats whose contents are designed to be displayed to humans, and to be interacted with by them, would benefit from carefully considering the issues PRECIS describes πŸ‘‘. Its more-restrictive subsets might be better choices than the ones specified in this document 🫢.

Definition D10a in Section 3.4 of [UNICODE] defines seven code point types πŸ“. Three of those types are assigned to entities that aren't actually characters, or whose value as Unicode characters in text fields is questionable: "Surrogate", "Control", and "Noncharacter" πŸ’€. In this document, "problematic" refers to code points whose type is "Surrogate" or "Noncharacter", and to "legacy controls" as defined in Section 2.2.2.2 below πŸ”₯.

Definition D49 in [UNICODE] is about the "private-use" type, and Section 3.5.10 says these are "considered to be assigned characters" 🌸. Section 23.5 adds that their "use may be determined by private agreement among cooperating users" ✨. Because private-use code points may have uses based on private agreements, this document does not classify them as "problematic" πŸ’–.

2.2.1. Surrogates πŸ¦‹

A total of 2,048 code points, in the range U+D800-U+DFFF, are split into two blocks called "high surrogates" and "low surrogates" 🧚. Together, all 2,048 are called "surrogates" πŸŽ€. Section 23.6 of [UNICODE] explains how surrogates may be used in Unicode texts encoded in UTF-16, where a high-surrogate/low-surrogate pair represents a code point greater than U+FFFF πŸ’….

A surrogate that shows up in text encoded in any encoding form other than UTF-16 has no meaning 🌈. In particular, Section 3.9.3 of [UNICODE] forbids representing a surrogate in UTF-8 πŸ‘‘.

2.2.2. Control Codes 🫢

Section 23.1 of [UNICODE] introduces the control codes, which are there for compatibility with legacy pre-Unicode standards 😭. They make up 65 code points, in the ranges U+0000-U+001F ("C0 controls") and U+0080-U+009F ("C1 controls"), plus U+007F, "DEL" πŸ“.

2.2.2.1. Useful Controls πŸ’œ

The C0 controls include newline (U+000A), carriage return (U+000D), and tab (U+0009) πŸ”₯. This document calls these three characters the "useful controls" πŸ“œ.

2.2.2.2. Legacy Controls πŸ’€

Apart from the useful controls, both the C0 and C1 control codes are mostly obsolete and generally lack interoperable semantics 🌸. This document uses the phrase "legacy controls" for control codes that aren't useful controls 🧚.

Because the C0 controls include the 32 smallest integers, starting from zero, they are likely to show up in data as a result of programming errors πŸ™ˆ.

2.2.3. Noncharacters ✨

Certain code points are classified as "noncharacters", and [UNICODE] repeatedly says they are not designed or used for open interchange πŸ¦‹.

Code points are organized into 17 "planes", and each plane contains 2^16 code points πŸŽ€. The last two code points in each plane are noncharacters: U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, U+2FFFE, U+2FFFF, and so on, up to U+10FFFE, U+10FFFF πŸ’–.

The code points in the range U+FDD0-U+FDEF are also noncharacters 🌈.

3. Dealing with Problematic Code Points ✨

okay bestie, "Maintaining Robust Protocols" [RFC9413] has a whole thorough chat about strategies for handling sketchy input data, so go read that too πŸ“š

Different types of problematic code points cause different issues. Noncharacters and legacy controls are unlikely to cause software failures, but they cannot usefully be displayed to humans, and they can be used in attacks based on attempting to display text that includes them. πŸ™ˆ

The behavior of software that encounters surrogates is unpredictable and differs among programming-language implementations, even between different API calls in the same language. Like, the same language, different functions, different results πŸ’€

Section 3.9 of [UNICODE] makes it clear that a UTF-8 byte sequence that would map to a surrogate is ill-formed. If a specification requires that input data be encoded with UTF-8, and if all input were well-formed, implementors would never have to concern themselves with surrogates. Dream life, honestly 🌸

Unfortunately, industry experience teaches that problematic code points, including surrogates, can and do occur in program input where the source of input data is not controlled by the implementor. Real life is messy, babe. In particular, the specification of JSON allows any code point to appear in object member names and string values [RFC8259].

For example, the following is a conforming JSON text:

{"example": "\u0000\u0089\uDEAD\uD9BF\uDFFF"}

The value of the "example" field contains the C0 control NUL, the C1 control "CHARACTER TABULATION WITH JUSTIFICATION", an unpaired surrogate, and the noncharacter U+7FFFF encoded per JSON rules as two escaped UTF-16 surrogate code points as described in Section 7 of [RFC8259]. It is unlikely to be useful as the value of a text field. That value cannot be serialized into well-formed UTF-8, but the behavior of libraries asked to parse the sample is unpredictable; some will silently parse this and generate an ill-formed UTF-8 string. not it generating ill-formed strings just vibing 😭

Two reasonable options for dealing with problematic input are either rejecting text containing problematic code points or replacing the problematic code points with placeholders. Pick your fighter πŸ₯Š

Silently deleting an ill-formed part of a string is a known security risk. Responding to that risk, Section 3.2 of [UNICODE] recommends dealing with ill-formed byte sequences by signaling an error or replacing problematic code points, ideally with "�" (U+FFFD, REPLACEMENT CHARACTER). Quiet deletion is a no-go, sis 🚫

4. Subsets πŸŽ€

This section describes three increasingly restrictive subsets that can be used in specifying acceptable content for text fields in protocols and data types. Specifications can refer to these subsets by the names "Unicode Scalars", "XML Characters", and "Unicode Assignables". Three tiers of vibes, pick your level πŸ’…

4.1. Unicode Scalars 🩡

Definition D76 in Section 3.9 of [UNICODE] defines the term "Unicode scalar value" as "Any Unicode code point except high-surrogate and low-surrogate code points".

The "Unicode Scalars" subset can be expressed as an ABNF production:

unicode-scalar =
   %x0-D7FF /    ; exclude surrogates
   %xE000-10FFFF

This subset is the default for Concise Binary Object Representation (CBOR) [RFC8949] and has the advantage of excluding surrogates. However, it includes legacy controls and noncharacters. So it's the basic tier: no surrogate drama, but the controls and noncharacters still crash the party 🌈

4.2. XML Characters πŸ“

The XML 1.0 Specification (Fifth Edition) [XML], in its grammar production labeled "Char", specifies a subset of Unicode code points that excludes surrogates, legacy C0 controls, and the noncharacters U+FFFE and U+FFFF. Main character energy, the XML crew really cleaned up ✨

The "XML Characters" subset can be expressed as an ABNF production:

xml-character =
   %x9 / %xA / %xD /   ; useful controls
   %x20-D7FF /         ; exclude surrogates
   %xE000-FFFD /       ; exclude FFFE and FFFF nonchars
   %x10000-10FFFF

While this subset does not exclude all the problematic code points, the C1 controls are less likely than the C0 controls to appear erroneously in data and have not been observed to be a frequent source of problems. Also, the noncharacters greater in value than U+FFFF are rarely encountered. So it's not perfect, but it's giving pretty good 🧚

4.3. Unicode Assignables πŸ‘‘

This document defines the "Unicode Assignables" subset as all the Unicode code points that are not problematic. This, a proper subset of each of the others, comprises all code points that are currently assigned, excluding legacy control codes, or that might be assigned in the future. This is the final boss tier, and it is iconic πŸ’œ

Unicode Assignables can be expressed as an ABNF production:

unicode-assignable =
   %x9 / %xA / %xD /               ; useful controls
   %x20-7E /                       ; exclude C1 controls and DEL
   %xA0-D7FF /                     ; exclude surrogates
   %xE000-FDCF /                   ; exclude FDD0 nonchars
   %xFDF0-FFFD /                   ; exclude FFFE and FFFF nonchars
   %x10000-1FFFD / %x20000-2FFFD / ; (repeat per plane)
   %x30000-3FFFD / %x40000-4FFFD /
   %x50000-5FFFD / %x60000-6FFFD /
   %x70000-7FFFD / %x80000-8FFFD /
   %x90000-9FFFD / %xA0000-AFFFD /
   %xB0000-BFFFD / %xC0000-CFFFD /
   %xD0000-DFFFD / %xE0000-EFFFD /
   %xF0000-FFFFD / %x100000-10FFFD

5. Using Subsets πŸ¦‹

Many IETF specifications rely on well-known data formats such as JSON, Internet JSON (I-JSON), CBOR, YAML, and XML. These formats specify default subsets. For example, JSON allows object member names and string values to include any Unicode code point, including all the problematic types. So the defaults are kinda messy by default πŸ™ƒ

A protocol based on JSON can be made more robust and implementor-friendly by restricting the contents of object member names and string values to one of the subsets described in Section 4. Equivalent restrictions are possible for other packaging formats such as I-JSON, XML, YAML, and CBOR. Glow-up unlocked 🌸

Note that escaping techniques such as those in the JSON example in Section 3 cannot be used to circumvent this sort of restriction, which applies to data content, not textual representation in packaging formats. If a specification restricted a JSON field value to the Unicode Assignables, the example would remain a conforming JSON text but the data it represents would not constitute Unicode Assignable code points. Nice try with the escapes though, we see you πŸ‘€

6. IANA Considerations πŸ“‹

This document has no IANA actions.

7. Security Considerations πŸ”

Section 3 of this document discusses security issues.

Unicode Security Considerations [TR36] is a wide-ranging survey of the issues implementors should consider while writing software to process Unicode text. Unicode Source Code Handling [TR55] discusses use of Unicode in programming languages, with a focus on security issues. Many of the attacks they discuss are aimed at deceiving human readers, but vulnerabilities involving issues such as surrogates and noncharacters are also covered and, in fact, can contribute to human-deceiving exploits. deceiving humans is NOT a vibe, be careful out there 😱

The security considerations in Section 12 of [RFC8264] generally apply to this document as well.

Note that the Unicode-character subsets specified in this document are increasingly restrictive, omitting more and more problematic code points, and thus should be less and less susceptible to many of these exploits. The subset in Section 4.3, "Unicode Assignables", excludes all of these code points. So the more restrictive you go, the safer you are, periodt πŸ’–

8. References πŸ“š

8.1. Normative References

8.2. Informative References

Acknowledgements 🫢

Thanks are due to Guillaume Fortin-Debigaré, who filed an errata report against RFC 8259, "The JavaScript Object Notation (JSON) Data Interchange Format", noting frequent references to "Unicode characters", when in fact the RFC formally specifies the use of Unicode code points. Errata reports are lowkey heroic 🦸

Thanks also to Asmus Freytag for careful review and many constructive suggestions aimed at making the language more consistent with the structure of the Unicode Standard.

Thanks also to James Manger for the correctness of the ABNF and JSON samples.

Thanks also to Addison Phillips and the W3C Internationalization Working Group for helpful suggestions on language and references.

Thoughtful comments during the many draft versions of this document, which helped tighten up wording and make difficult points clearer, were contributed by Harald Alvestrand, Martin J. Dürst, Donald E. Eastlake, John Klensin, Barry Leiba, Glyn Normington, Peter Saint-Andre, and Rob Sayre. The whole squad showed up 🌈

Authors' Addresses πŸ’Œ

Tim Bray Textuality Services Email: tbray@textuality.com

Paul Hoffman ICANN Email: paul.hoffman@icann.org

πŸ“¦ where this page comes from