RFC 7940
Representing Label Generation Rulesets Using XML
Representing Label Generation Rulesets Using XML 🎀
bestie, this is RFC 7940 💅 Published by the IETF in August 2016 as a Standards Track document (ISSN 2070-1721). The authors are K. Davies from ICANN and A. Freytag from ASMUS, Inc. ✨ The header doesn't list anything it obsoletes or updates, so it stands on its own 🌸
Abstract 💭
okay so this doc is all about how to write down rules for checking identifier labels, plus alternate forms of those labels, using XML 🧚 These policies are called "Label Generation Rulesets" (LGRs), and they're used for things like Internationalized Domain Names (IDNs) 🌈 A ruleset is basically the policy that says which labels and Unicode code points are allowed for registration, which code points count as variants of each other, and what can happen to labels containing those variants 🦋
Status of This Memo 📋
this is an Internet Standards Track document 💖
it's a product of the Internet Engineering Task Force (IETF), it represents consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication 🫶 if you want more on Internet Standards, check Section 2 of RFC 7841 🎀
for the current status, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc7940 ✨
Copyright Notice ©️
Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. 💜
this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info), as they stand on the publication date. please read these carefully, because they spell out your rights and restrictions for this document 🙈 any Code Components pulled out of this document have to include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License 💀
Table of Contents 📜
1. Introduction ....................................................4
2. Design Goals ....................................................5
3. Normative Language ..............................................6
4. LGR Format ......................................................6
4.1. Namespace ..................................................7
4.2. Basic Structure ............................................7
4.3. Metadata ...................................................8
4.3.1. The "version" Element ...............................8
4.3.2. The "date" Element ..................................9
4.3.3. The "language" Element ..............................9
4.3.4. The "scope" Element ................................10
4.3.5. The "description" Element ..........................10
4.3.6. The "validity-start" and "validity-end" Elements ...11
4.3.7. The "unicode-version" Element ......................11
4.3.8. The "references" Element ...........................12
5. Code Points and Variants .......................................13
5.1. Sequences .................................................14
5.2. Conditional Contexts ......................................15
5.3. Variants ..................................................16
5.3.1. Basic Variants .....................................16
5.3.2. The "type" Attribute ...............................17
5.3.3. Null Variants ......................................18
5.3.4. Variants with Reflexive Mapping ....................19
5.3.5. Conditional Variants ...............................20
5.4. Annotations ...............................................22
5.4.1. The "ref" Attribute ................................22
5.4.2. The "comment" Attribute ............................23
5.5. Code Point Tagging ........................................23
6. Whole Label and Context Evaluation .............................23
6.1. Basic Concepts ............................................23
6.2. Character Classes .........................................25
6.2.1. Declaring and Invoking Named Classes ...............25
6.2.2. Tag-Based Classes ..................................26
6.2.3. Unicode Property-Based Classes .....................26
6.2.4. Explicitly Declared Classes ........................28
6.2.5. Combined Classes ...................................29
6.3. Whole Label and Context Rules .............................30
6.3.1. The "rule" Element .................................31
6.3.2. The Match Operators ................................32
6.3.3. The "count" Attribute ..............................33
6.3.4. The "name" and "by-ref" Attributes .................34
6.3.5. The "choice" Element ...............................34
6.3.6. Literal Code Point Sequences .......................35
6.3.7. The "any" Element ..................................35
6.3.8. The "start" and "end" Elements .....................35
6.3.9. Example Context Rule from IDNA Specification .......36
6.4. Parameterized Context or When Rules .......................37
6.4.1. The "anchor" Element ...............................37
6.4.2. The "look-behind" and "look-ahead" Elements ........38
6.4.3. Omitting the "anchor" Element ......................40
7. The "action" Element ...........................................40
7.1. The "match" and "not-match" Attributes ....................41
7.2. Actions with Variant Type Triggers ........................41
7.2.1. The "any-variant", "all-variants", and
"only-variants" Attributes .........................41
7.2.2. Example from Tables in the Style of RFC 3743 .......44
7.3. Recommended Disposition Values ............................45
7.4. Precedence ................................................45
7.5. Implied Actions ...........................................45
7.6. Default Actions ...........................................46
8. Processing a Label against an LGR ..............................47
8.1. Determining Eligibility for a Label .......................47
8.1.1. Determining Eligibility Using Reflexive
Variant Mappings ...................................47
8.2. Determining Variants for a Label ..........................48
8.3. Determining a Disposition for a Label or Variant Label ....49
8.4. Duplicate Variant Labels ..................................50
8.5. Checking Labels for Collision .............................50
9. Conversion to and from Other Formats ...........................51
10. Media Type ....................................................51
11. IANA Considerations ...........................................52
11.1. Media Type Registration ..................................52
11.2. URN Registration .........................................53
11.3. Disposition Registry .....................................53
12. Security Considerations .......................................54
12.1. LGRs Are Only a Partial Remedy for Problem Space .........54
12.2. Computational Expense of Complex Tables ..................54
13. References ....................................................55
13.1. Normative References .....................................55
13.2. Informative References ...................................56
Appendix A. Example Tables ........................................58
Appendix B. How to Translate Tables Based on RFC 3743 into the
XML Format ............................................63
Appendix C. Indic Syllable Structure Example ......................68
C.1. Reducing Complexity .......................................70
Appendix D. RELAX NG Compact Schema ...............................71
Acknowledgements ..................................................82
Authors' Addresses ................................................821. Introduction ✨
okay bestie, this section sets the scene 🌷 It explains how to use XML to describe Label Generation Rulesets (LGRs) 💖 An LGR is an algorithm that decides whether a given identifier label is allowed, and under which conditions, based on the code points it contains and the context they show up in 🧚 Each one is made of three things: a list of permitted code points, mappings from code points to their variants, and a set of rules that act on those code points and mappings 🎀 LGRs are part of an administrator's policies. When people were deploying Internationalized Domain Names (IDNs), these were also called IDN tables or variant tables 🌈
there are other kinds of label policies that LGRs don't usually cover, so this XML format can't always represent them. Examples include, but aren't limited to, policies about trademarks or banning fraudulent or objectionable words 🙈
administrators of top-level domain registry zones have historically published their LGRs as ASCII text or HTML 📄 That formatting was loosely based on the Language Variant Table format from [RFC3743] 🦋 [RFC4290] also gives a "model table format" that does something pretty similar 💅 The catch is that the algorithms used to evaluate the data in these formats are either implicit or specified somewhere else 😭
through the first decade of IDN deployment, it became clear that LGRs built from these different formats were hard to implement consistently and hard to compare 💀 A universal format, like a structured XML one, helps with machine readability, consistency, reusability, and maintainability 🫶
when you're only listing a simple set of permitted code points, the format is really straightforward 🍓 Yes, the resulting file gets a bit more complicated, but in exchange you can handle conditional variants in a more sophisticated way, which matches what current zone administrator policies actually need 👑
another big perk: the format makes a lot of the algorithms explicit and machine-implementable ✨ There's still a small set of implicit algorithms, and this document describes those too, so everyone implements them the same way 🔥
the main use case is IDN label policy, but the format isn't limited to that. It can also describe ASCII domain name label rulesets, or other kinds of identifier labels beyond domain names 🌸
- Design Goals
Okay so, bestie, let's talk about what this format is actually trying to do 💅✨ The goals are kind of the whole vibe, so here they are:
o The format needs to be implementable in a reasonably straightforward manner in software. 🧚
o The format should be able to be automatically checked for formatting errors, so that common mistakes can be caught. 🫶
o An LGR needs to be able to express the set of valid code points that are allowed for registration under a specific administrator's policies. 🎀
o An LGR needs to be able to express computed alternatives to a given identifier based on mapping relationships between code points, whether one-to-one or many-to-many. These computed alternatives are commonly known as "variants". 🦋
o Variant code points should be able to be tagged with explicit dispositions or categories that can be used to support registry policy (such as whether to allocate the computed variant or to merely block it from usage or registration). 💜
o Variants and code points must be able to be stipulated based on contextual information. For example, some variants may only be applicable when they follow a certain code point or when the code point is displayed in a specific presentation form. 🌸
o The data contained within an LGR must be able to be interpreted unambiguously, so that independent implementations that utilize the contents will arrive at the same results. 🔥
o To the largest extent possible, policy rules should be able to be specified in the XML format without relying on hidden or built-in algorithms in implementations. 👑
o LGRs should be suitable for comparison and reuse, such that one could easily compare the contents of two or more to see the differences, to merge them, and so on. 🍓
o As many existing IDN tables as practicable should be able to be migrated to the LGR format with all applicable interpretation logic retained. ⭐
These requirements are partly derived from reviewing the existing corpus of published IDN tables, plus the requirements of ICANN's work to implement an LGR for the DNS root zone [LGR-PROCEDURE]. In particular, Section B of that document identifies five specific requirements for an LGR methodology.
The syntax and rules in [RFC5892] and [RFC3743] were also reviewed.
It is explicitly not the goal of this format to stipulate what code points should be listed in an LGR by a zone administrator. Which registration policies are used for a particular zone are outside the scope of this memo. 🙈
3. Normative Language ✨
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 📚
4. LGR Format 🎀
An LGR is expressed as a well-formed XML document [XML] that conforms to the schema defined in Appendix D. 💖
As XML is case sensitive, an LGR MUST be authored with the correct casing. For example, the XML element names MUST be in lowercase as described in this specification, and matching of attribute values is only performed in a case-sensitive manner. 🔠
A document that is not well-formed, is non-conforming, or violates other constraints specified in this specification MUST be rejected. 🚫
4.1. Namespace
The XML Namespace URI is urn:ietf:params:xml:ns:lgr-1.0.
See Section 11.2 for more information. 🌈
4.2. Basic Structure 🧱
The basic XML framework of the document is as follows:
<?xml version="1.0"?>
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
...
</lgr>The lgr element contains up to three sub-elements or sections. First is an optional meta element that contains all metadata associated with the LGR, such as its authorship, what it is used for, implementation notes, and references. This is followed by a required data element that contains the substantive code point data. Finally, an optional rules element contains information on rules for evaluating labels, if any, along with action elements providing for the disposition of labels and computed variant labels. 🌟
<?xml version="1.0"?>
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
<meta>
...
</meta>
<data>
...
</data>
<rules>
...
</rules>
</lgr>A document MUST contain exactly one lgr element. Each lgr element MUST contain zero or one meta element, exactly one data element, and zero or one rules element; and these three elements MUST be in that order. 📏
Some elements that are direct or nested child elements of the rules element MUST be placed in a specific relative order to other elements for the LGR to be valid. An LGR that violates these constraints MUST be rejected. In other cases, changing the ordering would result in a valid, but different, specification. 🔢
In the following descriptions, required, non-repeating elements or attributes are generally not called out explicitly, in contrast to "OPTIONAL" ones, or those that "MAY" be repeated. For attributes that take lists as values, the elements MUST be space-separated. 🫧
4.3. Metadata 📝
The meta element expresses metadata associated with the LGR, and the element SHOULD be included so that the associated metadata are available as part of the LGR and cannot become disassociated. The following subsections describe elements that may appear within the meta element. 💅
The meta element can be used to identify the author or relevant contact person, explain the intended usage of the LGR, and provide implementation notes as well as references. Detailed metadata allow the LGR document to become self-documenting -- for example, if rendered in a human-readable format by an appropriate tool. 🦋
Providing metadata pertaining to the date and version of the LGR is particularly encouraged to make it easier for interoperating consumers to ensure that they are using the correct LGR. 🎀
With the exception of the unicode-version element, the data contained within is not required by software consuming the LGR in order to calculate valid labels or to calculate variants. If present, the unicode-version element MUST be used by a consumer of the table to identify that it has the correct Unicode property data to perform operations on the table. This ensures that possible differences in code point properties between editions of the Unicode Standard do not impact the product of calculations utilizing an LGR. 🌸
4.3.1. The "version" Element 🔖
The version element is OPTIONAL. It is used to uniquely identify each version of the LGR. No specific format is required, but it is RECOMMENDED that it be the decimal representation of a single positive integer, which is incremented with each revision of the file. 🍓
An example of a typical first edition of a document:
<version>1</version>The version element MAY have an OPTIONAL comment attribute.
<version comment="draft">1</version>4.3.2. The "date" Element 📅
The OPTIONAL date element is used to identify the date the LGR was posted. The contents of this element MUST be a valid ISO 8601 "full-date" string as described in [RFC3339].
Example of a date:
<date>2009-11-01</date>4.3.3. The "language" Element 🌍
Each OPTIONAL language element identifies a language or script for which the LGR is intended. The value of the language element MUST be a valid language tag as described in [RFC5646]. The tag MAY refer to a script plus undefined language if the LGR is not intended for a specific language.
Example of an LGR for the English language:
<language>en</language>If the LGR applies to a script rather than a specific language, the "und" language tag SHOULD be used followed by the relevant script subtag from [RFC5646]. For example, for a Cyrillic script LGR:
<language>und-Cyrl</language>If the LGR covers a set of multiple languages or scripts, the language element MAY be repeated. However, for cases of a script-specific LGR exhibiting insignificant admixture of code points from other scripts, it is RECOMMENDED to use a single language element identifying the predominant script. In the exceptional case of a multi-script LGR where no script is predominant, use Zyyy (Common):
<language>und-Zyyy</language>4.3.4. The "scope" Element 🔭
This OPTIONAL element refers to a scope, such as a domain, to which this policy is applied. The type attribute specifies the type of scope being defined. A type of "domain" means that the scope is a domain that represents the apex of the DNS zone to which the LGR is applied. For that type, the content of the scope element MUST be a domain name written relative to the root zone, in presentation format with no trailing dot. However, in the unique case of the DNS root zone, it is represented as ".".
<scope type="domain">example.com</scope>There MAY be multiple scope tags used -- for example, to reflect a list of domains to which the LGR is applied.
No other values of the type attribute are defined by this specification; however, this specification can be used for applications other than domain names. Implementers of LGRs for applications other than domain names SHOULD define the scope extension grammar in an IETF specification or use XML namespaces to distinguish their scoping mechanism distinctly from the base LGR namespace. An explanation of any custom usage of the scope in the "description" element is RECOMMENDED.
<scope xmlns="http://example.com/ns/scope/1.0">
... content per alternate namespace ...
</scope>4.3.5. The "description" Element 🗒️
The description element is an OPTIONAL, free-form element that contains any additional relevant description that is useful for the user in its interpretation. Typically, this field contains authorship information, as well as additional context on how the LGR was formulated and how it applies, such as citations and references that apply to the LGR as a whole. 🫶
This field should not be relied upon for providing instructions on how to parse or utilize the data contained elsewhere in the specification. Authors of tables should expect that software applications that parse and use LGRs will not use the description element to condition the application of the LGR's data and rules. ⚠️
The element has an OPTIONAL type attribute, which refers to the Internet media type [RFC2045] of the enclosed data. Typical types would be "text/plain" or "text/html". The attribute SHOULD be a valid media type. If supplied, it will be assumed that the contents are of that media type. If the description lacks a type value, it will be assumed to be plain text ("text/plain"). 🌷
4.3.6. The "validity-start" and "validity-end" Elements ⏳
The validity-start and validity-end elements are OPTIONAL elements that describe the time period from which the contents of the LGR become valid (are used in registry policy) and time when the contents of the LGR cease to be used, respectively.
The dates MUST conform to the "full-date" format described in Section 5.6 of [RFC3339].
<validity-start>2014-03-12</validity-start>4.3.7. The "unicode-version" Element 🔢✨
okay bestie, if your LGR relies on character properties from a specific version of the Unicode Standard, you MUST list that version number in the form x.y.z, where x, y, and z are positive decimal integers (see [Unicode-Versions]). 💅 And if any software processing the table doesn't have access to character property data of the requisite version, it MUST NOT do any whole-label evaluation that relies on Unicode character properties (Section 6.2.3). not me panicking over version mismatches 😭
The value of a Unicode character property can change between versions of the Unicode Character Database [UAX44], unless that change has been explicitly disallowed in [Unicode-Stability]. It is RECOMMENDED to only reference properties that are defined as stable or immutable. 🧚 Instead of referencing the property, you can also spell the information out explicitly in the LGR.
<unicode-version>6.3.0</unicode-version>You don't need a unicode-version element for LGRs that don't use Unicode character properties, but it is RECOMMENDED anyway 🌸
4.3.8. The "references" Element 📚💜
An LGR can define a list of references that link various individual elements to one or more normative references. Common uses are annotating that code points belong to an externally defined collection or standard, or giving normative references for rules. 🦋
References go in an OPTIONAL references element that holds one or more reference elements, each with a unique id attribute. It is RECOMMENDED that the id attribute be a zero-based integer. But it MAY also contain uppercase letters A-Z, as well as a period, hyphen, colon, or underscore, on top of the digits 0-9. The value of each reference element SHOULD be the citation of a standard, dictionary, or other specification in any suitable format. On top of the id attribute, a reference element MAY have a comment attribute for an optional free-form annotation. ✨
<references>
<reference id="0">The Unicode Consortium. The Unicode
Standard, Version 8.0.0, (Mountain View, CA: The Unicode
Consortium, 2015. ISBN 978-1-936213-10-8)
http://www.unicode.org/versions/Unicode8.0.0/</reference>
<reference id="1">Big-5: Computer Chinese Glyph and Character
Code Mapping Table, Technical Report C-26, 1984</reference>
<reference id="2" comment="synchronized with Unicode 6.1">
ISO/IEC
10646:2012 3rd edition</reference>
...
</references>
...
<data>
<char cp="0620" ref="0 2" />
...
</data>A reference gets linked to an element by using its id as part of an optional ref attribute (see Section 5.4.1). The ref attribute can go on lots of elements in the data or rules sections of the LGR, most notably the ones defining code points, variants, and rules. However, a ref attribute may not occur on certain kinds of elements, including references to named character classes or rules. See below for the description of these elements. 🙈
5. Code Points and Variants 🫧✨
The bulk of an LGR is a description of which set of code points is eligible for a given label. For rulesets that perform operations resulting in potential variants, the code point-level relationships between those variants also need to be described. 💖
The code point data is collected inside the data element. Within it, a series of char and range elements describe eligible code points or ranges of code points, respectively. Together, these are known as the repertoire. 🌈
Discrete permissible code points or code point sequences (see Section 5.1) are declared with a char element. Here's a minimal example declaration for a single code point, with the code point value given in the cp attribute: 🎀
<char cp="002D"/>As described below, a full declaration for a char element, whether it's used for a single code point or for a sequence (see Section 5.1), may have optional child elements that define variants. Both the char and range elements can take a number of optional attributes for conditional inclusion, commenting, cross-referencing, and character tagging, as described below. 👑
Ranges of permissible code points may be declared with a range element, like in this minimal example: 🍓
<range first-cp="0030" last-cp="0039"/>The range is inclusive of the first and last code points. Any additional attributes defined for a range element act as if they were applied to each code point within it. A range element has no child elements. 🧚
It's always possible to swap a range out for a list of individually specified code points. The reverse isn't necessarily the case. Whenever that substitution is possible, it makes no difference in processing the data. Tools that read or write the LGR format are free to group sequences of consecutive code points with the same properties into range elements. 💅
Code points MUST be represented according to the standard Unicode convention, but without the prefix U+. They're expressed in uppercase hexadecimal and zero-padded to a minimum of 4 digits. 😭
The rationale for not allowing other encoding formats, including native Unicode encoding in XML, is explored in [UAX42]. The XML conventions used in this format, like element and attribute names, mirror this document where that's practical and reasonable. It is RECOMMENDED to list all char elements in ascending order of the cp attribute. Skipping that makes it unnecessarily hard for authors and reviewers to spot errors, like duplications, or to compare against code point listings in other documents and specifications. 🔍
All char elements in the data section MUST have distinct cp attributes. The range elements MUST NOT specify code point ranges that overlap either another range or any single code point char element. An LGR that defines the same code point more than once, through any combination of char or range elements, MUST be rejected. 🚫💀
5.1. Sequences 🦋
A sequence of two or more code points may be specified in an LGR -- for example, when defining the source for n:m variant mappings. Sequences are also useful when the exact order of code points has to occur for the constituent elements to be eligible, like when a code point is only eligible when it's preceded or followed by a certain code point. The following would define the eligibility of the MIDDLE DOT (U+00B7) only when it's both preceded and followed by the LATIN SMALL LETTER L (U+006C): 🌸
<char cp="006C 00B7 006C" comment="Catalan middle dot"/>All sequences defined this way must be distinct, but sub-sequences may be defined. So the sequence defined here can coexist with single code point definitions like this one: ✨
<char cp="006C" />As an alternative to using sequences to define a required context, a char or range element can specify a conditional context using an optional when attribute, as described below in Section 5.2. Using a conditional context is more flexible, because a context isn't limited to a specific sequence of code points. It also lets you choose between specifying either a prohibited or a required context. 🎀
5.2. Conditional Contexts 🧚♀️
A conditional context is specified by a rule that must be satisfied (or, alternatively, must not be satisfied) for a code point in a given label, often at a particular location in a label. 💜
To specify a conditional context, either a when or not-when attribute may be used. The value of each when or not-when attribute is a context rule, as described below in Section 6.3. That rule can be a rule evaluating the whole label or a parameterized context rule. The context condition is met when the rule specified in the when attribute is matched, or when the rule in the not-when attribute fails to match. It is an error to reference a rule that is not actually defined in the rules element. 🙅♀️
A parameterized context rule (see Section 6.4) defines the context immediately surrounding a given code point. Unlike a sequence, the context isn't limited to a specific fixed code point. It can, for example, designate any member of a certain character class, or a code point that has a certain Unicode character property. 🌈
Given a suitable definition of a parameterized context rule named "follows-virama", this example specifies that a ZERO WIDTH JOINER (U+200D) is restricted to immediately follow any of several code points classified as virama: 🦋
<char cp="200D" when="follows-virama" />For a complete example, see Appendix A. 📖
In contrast, a whole label rule (see Section 6.3) specifies a condition to be met by the entire label -- for example, that it must contain at least one code point from a given script anywhere in the label. In the following example, no digit from either range may occur in a label that mixes digits from both ranges: 🔢
<data>
<range first-cp="0660" last-cp="0669" not-when="mixed-digits"
tag="arabic-indic-digits" />
<range first-cp="06F0" last-cp="06F9" not-when="mixed-digits"
tag="extended-arabic-indic-digits" />
</data>(See Section 6.3.9 for an example of the "mixed-digits" rule.) The OPTIONAL when or not-when attributes are mutually exclusive. They MAY be applied to both char and range elements in the data element, including char elements defining sequences of code points, as well as to var elements (see Section 5.3.5). 🫶
If a label contains one or more code points that fail to satisfy a conditional context, the label is invalid (see Section 7.5). For variants, the conditional context restricts the definition of the variant to the case where the condition is met. Outside the specified context, a variant is not defined. 🚨
5.3. Variants 🧁
Most LGRs typically only determine simple code point eligibility. For those, the elements described so far would be the only ones required for their data section. Others additionally specify a mapping of code points to other code points, known as "variants". What counts as a variant code point is a matter of policy and varies for each implementation. The following examples are meant to demonstrate the syntax; they are not necessarily typical. 🌟
5.3.1. Basic Variants
okay so variant code points get specified with one or more "var" elements 🎀 nested inside a "char" element 💅. The target mapping goes in the "cp" attribute, and there are some other optional attributes on "var" that we'll get into below ✨
For example, to map LATIN SMALL LETTER V (U+0076) as a variant of LATIN SMALL LETTER U (U+0075):
<char cp="0075">
<var cp="0076"/>
</char>A sequence of multiple code points can be a variant of a single code point too 🦋 For instance, the sequence of LATIN SMALL LETTER O (U+006F) followed by LATIN SMALL LETTER E (U+0065) could hypothetically be a variant for LATIN SMALL LETTER O WITH DIAERESIS (U+00F6), like this:
<char cp="00F6">
<var cp="006F 0065"/>
</char>The source and target of a variant mapping can both be sequences, but they can't be ranges 🙅♀️
If the source of one mapping is a prefix sequence of the source for another, both variant mappings get considered at the same spot in the input label when you're generating permuted variant labels 🌀 Honestly, if an LGR is poorly designed, it could end up generating multiple copies of the same variant label for the same original label, but with possibly different dispositions. That's a whole mess 💀 Any duplicate variant labels encountered MUST be treated as an error (see Section 8.4).
The "var" element only specifies variant mappings in one direction, even though the variant relation is usually seen as symmetric 🪞 So if A is a variant of B, then B should also be a variant of A. The format needs the inverse of the variant to be written out explicitly if you want to fully specify a symmetric variant relation in the LGR. The nice side effect is that the symmetry becomes super obvious ✨
<char cp="006F 0065">
<var cp="00F6"/>
</char>Variant relations are normally symmetric and also transitive 🔗 If A is a variant of B and B is a variant of C, then A is also a variant of C. Just like with symmetry, these transitive relations are only part of the LGR if you spell them out explicitly. Implementations that need an LGR to be symmetric and transitive should verify this mechanically 🤖
All variant mappings are unique 💜 For a given "char" element, all "var" elements MUST have a unique combination of "cp", "when", and "not-when" attributes. It is RECOMMENDED to list the "var" elements in ascending order of their target code point sequence. (For "when" and "not-when" attributes, see Section 5.3.5.)
5.3.2. The "type" Attribute
Variants can be tagged with an OPTIONAL "type" attribute 🏷️ The value can be any non-empty value that doesn't start with an underscore and doesn't contain spaces. This value is used to resolve the disposition of any variant labels created using a given variant. (See Section 7.2.)
By default, the "type" values directly describe the target policy status (disposition) for a variant label that was generated using a particular variant. Any variant label gets the disposition that matches the most restrictive variant type 😤 Several conventional disposition values are predefined below in Section 7. Whenever those values can represent the policy you want, they SHOULD be used ✨
<char cp="767C">
<var cp="53D1" type="allocatable"/>
<var cp="5F42" type="blocked"/>
<var cp="9AEA" type="blocked"/>
<var cp="9AEE" type="blocked"/>
</char>By default, if a variant label contains any instance of a variant of type "blocked", the whole label is blocked. But if it only contains instances of variants that are to be allocated, it can be allocated 🍓 See the discussion about implied actions in Section 7.6.
The XML format makes the relation between the "type" values on variants and the resulting disposition of variant labels totally explicit 👑 See the discussion in Section 7.2. This explicitness lets the "type" attribute go from directly reflecting dispositions to a more differentiated intermediate value, which is then used when resolving the label's disposition. Instead of just applying the most restrictive disposition to the whole label, this kind of generalized resolution can hit extra goals, like limiting the set of allocatable variant labels or carrying out other policies found in existing LGRs (see, for example, Appendix B) 🧠
Because variant mappings MUST be unique, you can't define the same variant for the same "char" element with different "type" attributes (however, see Section 5.3.5) 🙅♀️
5.3.3. Null Variants
A null variant is a variant string that maps to no code point 🫥 This comes in handy when a particular code point sequence is considered discretionary in the context of a whole label. To specify a null variant, use an empty "cp" attribute. For example, to mark a string with a ZERO WIDTH NON-JOINER (U+200C) as the same string without the ZERO WIDTH NON-JOINER:
<char cp="200C">
<var cp=""/>
</char>This is useful for saying that some code points in a label should get mapped away when you generate a canonical variant of the label 🧹 Just be careful, though: in tables designed to have symmetric variant mappings, this can cause a combinatorial explosion if you don't handle it well 💥
The symmetric form of a null variant looks like this:
<char cp="">
<var cp="200C" type="invalid" />
</char>A "char" element with an empty "cp" attribute MUST specify at least one variant mapping. It is strongly RECOMMENDED to use a type of "invalid" or something equivalent when defining variant mappings from null sequences. That way, those mappings get removed during variant label generation (see Section 5.3.2) 🗑️✨
5.3.4. Variants with Reflexive Mapping
At first glance, there seems to be no reason to add variant mappings where the source and target code points are the same 🤔 Those are reflexive mappings, aka identity mappings. But honestly? Reflexive mappings show up a lot in LGRs that follow [RFC3743] 👀
Adding a "var" element lets you specify both a type and a reference id for it. The reference id isn't used in processing, but the variant's type can trigger actions 💫 When you permute a label to generate all possible variants, the type linked to a reflexive variant mapping gets applied to any of the permuted labels that contain the original code point.
In the example below, let's pretend the goal is to only allocate labels that contain a variant considered "preferred" in some way 🌟 The code point U+3473 exists both as a variant of U+3447 and as a variant of itself (a reflexive mapping). Given an original label of "U+3473 U+3447", the permuted variant "U+3473 U+3473" would be the reflexive variant of U+3473 followed by a variant of U+3447. With the variant mappings defined this way, both variant mappings used to generate that permutation would have the value "preferred":
<char cp="3447" ref="0">
<var cp="3473" type="preferred" ref="1 3" />
</char>
<char cp="3473" ref="0">
<var cp="3447" type="blocked" ref="1 3" />
<var cp="3473" type="preferred" ref="0" />
</char>Once the variant types are set up like this, you could define a set of actions that return a disposition of "allocatable" or "activated" for a label made up entirely of variants with type "preferred", for example 🎯 (For details on defining actions based on variant types, see Section 7.2.1.)
In general, using reflexive variant mappings like this makes it possible to calculate disposition values with one uniform approach for all labels, whether they're made of mapped variant code points, original code points, or a mix of both 🧩 Two otherwise identical labels can even have different dispositions, depending on which variant mappings were run to generate each of them 😳 (For details on generating variants and evaluating dispositions, see Section 8.)
Another useful convention that uses reflexive variants is described below in Section 7.2.1. 💖
5.3.5. Conditional Variants 🎀
Okay so, variants are basically mappings between two code point sequences, right? 💖 But sometimes a variant relationship only exists if you look at context outside the sequence itself 👀. Position is the big one! 🧚
Arabic is the classic example. Arabic letters change shape depending on where they sit in a word, and some letters share the same shape in certain positions. That makes them variants of each other in exactly those positions 🌸. The context can't come from the code point alone, because the code point is the same no matter which form it's drawn in.
As we saw in Section 5.2, you get an OPTIONAL "when" or "not-when" attribute on any "var" element, and it lets you say which contextual conditions must be met (or must not be met) for the variant to be defined 💅.
Let's say the "rules" element has suitable rules called "arabic-isolated" and "arabic-final". The example below marks ARABIC LETTER ALEF WITH WAVY HAMZA BELOW (U+0673) as a variant of ARABIC LETTER ALEF WITH HAMZA BELOW (U+0625), but only in its isolated or final forms ✨:
<char cp="0625">
<var cp="0673" when="arabic-isolated"/>
<var cp="0673" when="arabic-final"/>
</char>A "var" element MUST NOT contain multiple conditions. It's only allowed a single "when" or "not-when" attribute 🙅♀️. But you MAY specify multiple "var" elements using the same mapping, each with a different "when" or "not-when" attribute. The combo of mapping plus conditional context is what defines a unique variant 🦋.
Careful here, bestie! For each variant label, at most one of the contextual conditions can be met for variants sharing the same mapping. If you mess this up, you end up with duplicate variant labels for the same input label 😭. Any such duplicate variant labels MUST be treated as an error. See Section 8.4 for that.
Contexts can also be complementary. This next example shows ARABIC LETTER TEH MARBUTA (U+0629) as a variant of ARABIC LETTER HEH (U+0647), but with two different types 💜:
<char cp="0647" >
<var cp="0629" not-when="arabic-final" type="blocked" />
<var cp="0629" when="arabic-final" type="allocatable" />
</char>The idea here is that a label using U+0629 instead of U+0647 in a final position should count as essentially the same label, so it's allocatable to the same entity 🫶. Meanwhile, the same swap in a non-final position makes labels that are different but confusable. In that case, only one of them, not both, should be delegatable 🙈.
For symmetry, the reverse mappings must exist, and they must agree in their "when" or "not-when" attributes. Symmetry doesn't extend to the other attributes though 👀. Here are the potential reverse mappings for the example above:
<char cp="0629" >
<var cp="0647" not-when="arabic-final" type="allocatable" />
<var cp="0647" when="arabic-final" type="allocatable" />
</char>Both variants have the same "type" attribute here. You might be tempted to notice that the "when" and "not-when" attributes are complementary in this case, so between them they cover every possible context. But it is strongly RECOMMENDED to use the format shown in the example, because it makes the symmetry easy for parsers and tools to verify 🧠. (The same applies to entries created for transitivity.)
Arabic is a script where these conditional variants have already been implemented, based on the joining contexts of Arabic code points 🌙. The mechanism here also supports other kinds of conditional variants that other scripts might need ✨.
5.4. Annotations 📝
Two attributes, "ref" and "comment", can be used to annotate individual elements in the LGR. They get ignored when the LGR is processed by machines 🤖. The "ref" attribute is for formal annotations, and "comment" is for free-form ones. The "comment" attribute can be applied more widely, too 💬.
5.4.1. The "ref" Attribute 📚
Reference information MAY optionally be specified with a "ref" attribute, which is a space-delimited sequence of reference identifiers (see Section 4.3.8) 🎀.
<char cp="5220" ref="0">
<var cp="5220" ref="5"/>
<var cp="522A" ref="2 3"/>
</char>This is mostly used to give source information for code points or variant relations. It gets ignored when a machine processes an LGR 🤖. If you use it on a range, the "ref" attribute applies to every code point in that range 🌈. All reference identifiers MUST come from the set declared in the "references" element (see Section 4.3.8). Repeating a reference identifier inside the same "ref" attribute is an error 🚫. It is RECOMMENDED that identifiers be listed in ascending order 🔢.
Along with "char", "range", and "var" elements in the "data" section, a "ref" attribute can also show up on a few element types inside the "rules" element, as described below. Those are actions and literals (a "char" inside a rule), plus definitions of rules and classes. It can't go on references to named character classes or rules that use the "by-ref" attribute, which is defined below. (The "by-ref" and "ref" attributes are mutually exclusive.) None of the elements in the metadata take a "ref" attribute. If you need to add extra info there, use the "description" element instead 💡.
5.4.2. The "comment" Attribute 💬
Any "char", "range", or "variant" element in the "data" section may contain an OPTIONAL "comment" attribute 🌸. The contents of a "comment" attribute are free-form plain text. Machines ignore comments when processing the table 🙈. "comment" attributes MAY also be placed on all elements in the "rules" section of the document, like actions and match operators, plus definitions of classes and rules. The one exception is child elements of the "class" element, which can't have them 🚫. Finally, in the metadata, only the "version" and "reference" elements MAY have "comment" attributes. That matches the syntax in [RFC3743] 🫶.
5.5. Code Point Tagging 🏷️
Usually, LGRs are used to explicitly designate the code points that are allowed. Any label containing a code point that isn't explicitly listed in the LGR is an ineligible label according to the ruleset 🚫.
For more complicated registry rules, you might need to tell apart one or more subsets of code points. You can do that by adding an OPTIONAL "tag" attribute to "char" or "range" elements that are child elements of the "data" element 🧁. Collecting code points that share the same tag value lets you define character classes (see Section 6.2.2). Those classes can then be used in parameterized context or whole label rules (see Section 6.3.2) ✨.
Each "tag" attribute MAY contain multiple values, separated by white space. A tag value is an identifier that can also include certain punctuation marks, like a colon. Formally, it MUST correspond to the XML 1.0 Nmtoken (Name token) production (see [XML] Section 2.3). Duplicating a value within the same "tag" attribute is an error 🙅♀️. A "tag" attribute on a "range" element applies to all code points in that range. Code point sequences aren't proper members of a set of code points, so a "tag" attribute MUST NOT be present in a "char" element that defines a code point sequence 💀.
6. Whole Label and Context Evaluation 🔍
6.1. Basic Concepts 🧠
The "rules" element holds the specification for both context-based and whole label rules. Together, these are called Whole Label Evaluation (WLE) rules (Section 6.3). The "rules" element also contains the character classes (Section 6.2) they depend on, plus any actions (Section 7) that assign dispositions to labels based on rules or variant mappings 💅.
A whole label rule gets applied to the entire label. It's used to validate both original labels and any variant labels computed from them 🎯.
A rule that implements a conditional context (as discussed in Section 5.2) doesn't necessarily apply to the whole label. It can be specific to the context around a single code point or code point sequence 🦋. Sometimes certain code points in a label need to satisfy context-based rules. For example, that might be needed for the label to be valid, or to satisfy the context for a variant mapping (see the description of the "when" attribute in Section 6.4) 💜.
Say a rule gets referenced in the "when" attribute of a variant mapping. Then it describes the conditional context under which that particular variant mapping exists 🌈.
Each rule is defined in a "rule" element. A rule can contain the following as child elements:
- literal code points or code point sequences
- character classes, which define sets of code points to be used for context comparisons
- context operators, which define when character classes and literals may appear
- nested rules, whether defined in place or invoked by reference
Together, these are called "match operators", and they're listed in Section 6.3.2. An LGR containing rules or match operators that:
- are incorrectly defined or nested,
- have invalid attributes, or
- have invalid or undefined attribute values
MUST be rejected. Note that not all of the constraints defined here are validated by the schema 🚫.
6.2. Character Classes 🎨
Character classes are sets of characters that often share a particular property. They work like sets in every way, even supporting the usual set operators. They're called "character classes" here as a nod to how that term is used in regular expression syntax. (This also avoids confusion with "character set" in the sense of character encoding 🤓.)
Character classes can be specified in several ways:
- by defining the class via matching a tag in the code point data. All characters with the same "tag" attribute are part of the same class;
- by referencing a value of one of the Unicode character properties defined in the Unicode Character Database;
- by explicitly listing all the code points in the class; or
- by defining the class as a set combination of any number of other classes.
6.2.1. Declaring and Invoking Named Classes 🏷️
A character class has an OPTIONAL "name" attribute, which is a single identifier with no spaces in it 💖. All class names must be unique. If you leave out the "name" attribute, the class is anonymous, and it only exists inside the rule or combined class where it's defined. A named character class is defined independently, and any rules or other character class definitions can reference it by name ✨.
<class name="example" comment="an example class definition">
0061 4E00
</class>
...
<rule>
<class by-ref="example" />
</rule>An empty "class" element with a "by-ref" attribute is a reference to an existing named class 🔗. The "by-ref" attribute MUST NOT be used in the same "class" element with any of these attributes: "name", "from-tag", "property", or "ref" 🙅♀️. The "name" attribute MUST be present if and only if the class is a direct child element of the "rules" element 🎯. It's an error to reference a named class whose definition hasn't been seen yet 😭.
6.2.2. Tag-Based Classes ✨
okay so the char or range elements inside data MAY have a tag attribute 💅 It holds one or more tag values separated by spaces, like this:
<char cp="0061" tag="letter lower"/>
<char cp="4E00" tag="letter"/>That gives U+0061 two tags, "letter" and "lower". Now watch this 👀 Bestie can make two named classes from those tags:
<class name="letter" from-tag="letter" />
<class name="lower" from-tag="lower" />The first class, "letter", holds every code point with that tag, so 0061 and 4E00 are both in it. The second, "lower", holds 0061 but not 4E00. The name attribute can be left out if you want an anonymous, in-place, tag-based class nested somewhere else 🦋
Tag values are usually identifiers, plus a few punctuation marks like a colon. Formally, they MUST correspond to the XML 1.0 Nmtoken production. A tag attribute can hold a list of tag values, but the from-tag attribute MUST always contain a single tag value 🫶
If the document has no char or range elements with a matching tag, the character class is the empty set 🫧 That's allowed, so a shared rules element can be reused across files. Still, it is RECOMMENDED that implementations show a warning, so people can tell that referring to an undefined tag this way was on purpose 💜
6.2.3. Unicode Property-Based Classes 🌈
A class can be defined by a Unicode property. You give the Unicode property alias, then the property value or property value alias, separated by a colon:
<class name="virama" property="ccc:9" />This one picks every code point whose Unicode Canonical Combining Class (ccc) value is 9. That ccc value is given to all the code points that encode viramas 🍓
Unicode property values MUST be written as a composite of the attribute name and the value, as defined for the property value in [UAX42], with a colon between them. Loose matching of property values and names, as described in [UAX44], is not right for an XML schema, so it isn't supported. It isn't supported in the XML representation [UAX42] of the Unicode Character Database either 🙅♀️
A property-based class MAY be anonymous. Or, when it's an immediate child of the rules element, it MAY get a name, which connects the formal property definition to how it gets used. For example, you can use the value 9 for ccc to mark a virama (or halant) across different scripts ✨
Unicode properties may, in principle, change between versions of the Unicode Standard. The values assigned for one version are fixed, though 🧚 If Unicode properties are used, a Unicode version MUST be declared in the unicode-version element in the header. (Note: some Unicode properties are stable across versions by definition, and they don't change once assigned. See [Unicode-Stability].)
All implementations that process LGR files SHOULD support this minimal set of Unicode properties 💅
- General Category (gc)
- Script (sc)
- Canonical Combining Class (ccc)
- Bidi Class (bc)
- Arabic Joining Type (jt)
- Indic Syllabic Category (InSC)
- Deprecated (Dep)
The short name for each property is in the parentheses.
If a program uses an LGR to check whether a label is valid, and it runs into a property it doesn't support, it MUST abort with an error. No cap, that's a hard stop 🚫
6.2.4. Explicitly Declared Classes 💖
You can also declare a class by listing every code point in it. This comes in handy when tagging doesn't work, usually because the code points only show up inside code point sequences and aren't listed one by one as eligible code points for the LGR. Sad, but it happens 😭
To define a class from an explicit list, write the hexadecimal code point values separated by spaces:
<class name="abcd">0061 0062 0063 0064</class>This makes a class called "abcd" with the code points for the characters "a", "b", "c", and "d" 🌸 The order doesn't matter, but it is RECOMMENDED to list them in ascending order. If you don't, it gets way harder for people to spot duplicates or to compare the class against other specifications 🙈
For ranges, use a hexadecimal start and end value joined by a hyphen. This is the same as the one above:
<class name="abcd">0061-0064</class>Ranges and single code points can be mixed freely too:
<class name="abcd">0061 0062-0063 0064</class>Here's the difference between a class and a repertoire. A repertoire MAY contain sequences as elements, but a class MUST NOT. Classes look a lot like the character classes you'd find in regular expressions 💜
6.2.5. Combined Classes 🎀
Classes can be combined with operators for set complement, union, intersection, difference (the elements of the first class that aren't in the second), and symmetric difference (the elements in either class but not both). Classes work like sets, so the union of several character classes is also a class, for example 🦋
+-------------------+----------------------------------------------+
| Logical Operation | Example |
+-------------------+----------------------------------------------+
| Complement | <complement><class by-ref="xxx"></complement>|
+-------------------+----------------------------------------------+
| Union | <union> |
| | <class by-ref="class-1"/> |
| | <class by-ref="class-2"/> |
| | <class by-ref="class-3"/> |
| | </union> |
+-------------------+----------------------------------------------+
| Intersection | <intersection> |
| | <class by-ref="class-1"/> |
| | <class by-ref="class-2"/> |
| | </intersection> |
+-------------------+----------------------------------------------+
| Difference | <difference> |
| | <class by-ref="class-1"/> |
| | <class by-ref="class-2"/> |
| | </difference> |
+-------------------+----------------------------------------------+
| Symmetric | <symmetric-difference> |
| Difference | <class by-ref="class-1"/> |
| | <class by-ref="class-2"/> |
| | </symmetric-difference> |
+-------------------+----------------------------------------------+
Set OperatorsYou can nest these elements inside each other, with one rule: a complement element MUST contain precisely one class or one of the operator elements. An intersection, symmetric-difference, or difference element MUST contain precisely two. A union element MUST contain two or more of these elements 🔥
You can define an anonymous combined class right inside a rule, or inside any of the match operator elements that allow child elements (see Section 6.3.2), by using the set combination as the outer element:
<rule>
<union>
<class by-ref="xxx"/>
<class by-ref="yyy"/>
</union>
</rule>This is an anonymous combined class for the union of the classes "xxx" and "yyy". You don't need to wrap the union in another class element. In fact, set combination elements MUST NOT be nested inside a class element 🙅♀️
If you want a named combined class that other classes or rules can reference as <class by-ref="xxxyyy"/>, give the set combination element a name attribute, like <union name="xxxyyy" />, and put it at the top level, right below the rules element (see Section 6.2.1):
<rules>
<union name="xxxyyy">
<class by-ref="xxx"/>
<class by-ref="yyy"/>
</union>
...
</rules>Since a combination of classes is a class too, no matter which set operators built it, you always reference it with the class element from Section 6.2.1. A named class is always referenced through an empty class element with the by-ref attribute, holding the name of the class you want 💅
6.3. Whole Label and Context Rules 🌸
Each rule is a series of matching operators. They all have to be satisfied, in order, before you know whether a label meets a given condition. Rules can reference other rules or character classes defined elsewhere in the table 🧚
6.3.1. The "rule" Element 👑
A matching rule is defined by a rule element. Its child elements are match operators from Section 6.3.2. When a rule is evaluated, each child element is matched in order. rule elements MAY be nested inside each other and inside certain match operators 🫶
Here's a simple rule that matches a label where every character is a member of a class called "preferred-codepoint":
<rule name="preferred-label">
<start />
<class by-ref="preferred-codepoint" count="1+"/>
<end />
</rule>Rules come with explicit and implied actions, and those actions trigger when a rule matches a label. For example, a simple explicit action for the rule above would be:
<action disp="allocatable" match="preferred-label" />With that, a label made up entirely of preferred code points gets its policy disposition set to "allocatable". Explicit actions are covered in Section 7, and implicit actions in Section 7.5. Rules are also used to define conditional contexts for code points and variants, as discussed in Sections 5.2 and 5.3.5 ✨
A rule that is an immediate child of the rules element MUST be named with a name attribute holding a single identifier string with no spaces. A named rule can be pulled into another rule by reference, and it can also be referenced by an action element, a when attribute, or a not-when attribute. If the name attribute is left out, the rule is anonymous and MUST be nested inside another rule or match operator 🦋
6.3.2. The Match Operators ✨
okay so the child elements of a rule are a series of match operators 💅. here they are, sorted by type and name, with a basic example or two 🎀
+------------+-------------+------------------------------------+
| Type | Operator | Examples |
+------------+-------------+------------------------------------+
| logical | any | <any /> |
| +-------------+------------------------------------+
| | choice | <choice> |
| | | <rule by-ref="alternative1"/> |
| | | <rule by-ref="alternative2"/> |
| | | </choice> |
+--------------------------+------------------------------------+
| positional | start | <start /> |
| +-------------+------------------------------------+
| | end | <end /> |
+--------------------------+------------------------------------+
| literal | char | <char cp="0061 0062 0063" /> |
+--------------------------+------------------------------------+
| set | class | <class by-ref="class1" /> |
| | | <class>0061 0064-0065</class> |
+--------------------------+------------------------------------+
| group | rule | <rule by-ref="rule1" /> |
| | | <rule><any /></rule> |
+--------------------------+------------------------------------+
| contextual | anchor | <anchor /> |
| +-------------+------------------------------------+
| | look-ahead | <look-ahead><any /></look-ahead> |
| +-------------+------------------------------------+
| | look-behind | <look-behind><any /></look-behind> |
+--------------------------+------------------------------------+
Match Operatorsany element that defines an anonymous class can be a match operator 🦋, and that includes the set combination operators (see Section 6.2.5) plus references to named classes 💜.
ngl, all the match operators that show up as empty elements in the Examples column of the table above don't get child elements of their own 🙅. otherwise, match operators MAY be nested 🪆. anonymous "rule" elements are especially good for grouping ✨.
6.3.3. The "count" Attribute 🔢
the OPTIONAL "count" attribute (totally optional, bestie 🫶) tells you the minimum required or maximum allowed number of times a match operator gets used to match input. if the "count" attribute is:
n— the match operator matches the input exactly n times, where n is 1 or greater.n+— the match operator matches the input at least n times, where n is 0 or greater.n:m— the match operator matches the input at least n times, where n is 0 or greater, but matches the input up to m times in total, where m > n. If m = n and n > 0, the match operator matches the input exactly n times.
if there's no "count" attribute, the match operator just matches the input exactly once 🎯.
when it comes to matching, greedy evaluation is used in the sense defined for regular expressions 🍝. past the required number of times, the input gets matched as many times as possible, but not so often that it blocks a match of the rest of the rule.
big rule 🚫: a "count" attribute MUST NOT go on any element that has a "name" attribute, but it MAY go on operators like "class" that declare anonymous classes (combined classes included) or call any predefined classes by reference 💜. and the "count" attribute MUST NOT go on any "class" element, or any element defining a combined class, when it's nested inside a combined class 💀.
a "count" attribute MUST NOT be applied to match operators of type "start", "end", "anchor", "look-ahead", or "look-behind", and it also MUST NOT go on any operators, like "rule" or "choice", that contain a nested instance of them 🙅♀️. this limit applies recursively, and it doesn't matter whether a "rule" element holding these nested instances is declared in place or used by reference.
however 🌸, the "count" attribute MAY be applied to any other instances of either an anonymous "rule" element or a "choice" element, including the ones nested inside other match operators. it MAY also be applied to the elements "any" and "char", when they're used as match operators ✨.
6.3.4. The "name" and "by-ref" Attributes 🏷️
like classes (see Section 6.2.1), rules declared as immediate child elements of the "rules" element MUST be named with a unique "name" attribute, and all other instances MUST NOT be named 📛. anonymous rules and classes, or references to named rules and classes, can be nested inside other match operators by reference 🧚.
to reference a named rule or class inside a rule or match operator, use a "rule" or "class" element with an OPTIONAL "by-ref" attribute holding the name of the thing you're referencing. it's an error to reference a rule or class whose complete definition hasn't been seen yet. in other words, recursive rules or class definitions are explicitly not possible 🙃. the "by-ref" attribute MUST NOT appear in the same element as the "name" attribute, or in an element that has any child elements.
here's an example with a few named classes and a named rule that calls some of them by name 🎀:
<class name="letter" property="gc:L"/>
<class name="combining-mark" property="gc:M"/>
<class name="digit" property="gc:Nd" />
<rule name="letter-grapheme">
<class by-ref="letter" count="1+"/>
<class by-ref="combining-mark" count="0+"/>
</rule>6.3.5. The "choice" Element 🔀
the "choice" element is how you show a list of two or more alternatives 🍓:
<rule name="ldh">
<choice count="1+">
<class by-ref="letter"/>
<class by-ref="digit"/>
<char cp="002D" comment="literal HYPHEN"/>
</choice>
</rule>each child element of a "choice" element is one alternative 🫧. the first matching alternative decides the match for the "choice" element. if you want a choice where an alternative is itself a sequence of elements, the sequence has to be wrapped in an anonymous rule 🎁.
6.3.6. Literal Code Point Sequences 🔤
a literal code point sequence matches a single code point or a sequence 💫. it's defined by a "char" element, with the code point or sequence to match given by the "cp" attribute. when it's used as a literal, a "char" element MAY contain a "count" attribute in addition to the "cp" attribute, plus OPTIONAL "comment" or "ref" attributes. no other attributes or child elements are permitted 🚫.
6.3.7. The "any" Element 🎲
the "any" element is an empty element that matches any single code point 🌈. it MAY have a "count" attribute. for an example, check out Section 6.3.9.
unlike a literal, the "any" element MUST NOT have a "ref" attribute 🙅.
6.3.8. The "start" and "end" Elements 🚪
to match the beginning or end of a label, use the "start" or "end" element 🎬. an empty label would match this rule:
<rule name="empty-label">
<start/>
<end/>
</rule>conceptually, whole label rules evaluate the label as a whole, but in practice, lots of rules don't actually need to be written to match the entire label 🧠. for example, if you want to say a label can't start with a digit, your rule only needs to describe the initial part of the label 🔢.
this example uses the rules from before, together with "start" and "end" elements, to define a rule that requires an entire label to be well-formed 💅. for this example, that means it has to start with a letter, and it has no leading digits or combining marks, nor combining marks placed on digits 🌷.
<rule name="leading-letter" >
<start />
<rule by-ref="letter-grapheme" count="1"/>
<choice count="0+">
<rule by-ref="letter-grapheme" count="0+"/>
<class by-ref="digit" count="0+"/>
</choice>
<end />
</rule>each "start" or "end" element occurs at most once in a rule 1️⃣, except when it's nested inside a "choice" element in such a way that, while matching, each alternative encounters at most one occurrence of each. otherwise it's an error, and so is any case where a "start" or "end" element isn't the first or last element to be matched, respectively, while matching a rule 😬. "start" and "end" elements are empty elements that don't have a "count" attribute or any attribute other than "comment". it's an error for any match operator enclosing a nested "start" or "end" element to have a "count" attribute 💀.
6.3.9. Example Context Rule from IDNA Specification 🌍
this is an example of the WLE rule from [RFC5892] that forbids mixing the Arabic-Indic and extended Arabic-Indic digits in the same label 🕌. it's implemented as a whole label rule tied to the code point ranges using the "not-when" attribute, which defines an impermissible context. the example also shows a few instances of anonymous rules being used for grouping 🎀.
<data>
<range first-cp="0660" last-cp="0669" not-when="mixed-digits"
tag="arabic-indic-digits" />
<range first-cp="06F0" last-cp="06F9" not-when="mixed-digits"
tag="extended-arabic-indic-digits" />
</data>
<rules>
<rule name="mixed-digits">
<choice>
<rule>
<class from-tag="arabic-indic-digits"/>
<any count="0+"/>
<class from-tag="extended-arabic-indic-digits"/>
</rule>
<rule>
<class from-tag="extended-arabic-indic-digits"/>
<any count="0+"/>
<class from-tag="arabic-indic-digits"/>
</rule>
</choice>
</rule>
</rules>as the example says, a label with a code point from either of the two digit ranges is invalid for any label matching the "mixed-digits" rule 🚨. that means any time a code point from the other range shows up too. heads up 👀: invalidating the label is not the same as invalidating the definition of the "range" elements. the definition of the tag values doesn't depend on the "when" attribute, so you're good 💖.
6.4. Parameterized Context or When Rules 🎀
to recap 📝: when a rule is meant to give context for evaluating whether a code point or variant mapping is valid, it gets invoked by the "when" or "not-when" attributes described in Section 5.2. for "char" and "range" elements, an action implied by a context rule always has a disposition of "invalid" whenever the rule given by the "when" attribute isn't matched (see Section 7.5). flip it around, and a "not-when" attribute gives a disposition of "invalid" whenever the rule does match 🔁. when a rule is used this way, it's called a context or "when" rule 🏷️.
the example in the previous section shows a whole label rule being used as a context rule, which basically makes the whole label the context 🫧. the next sections describe several match operators you can use to specify a context more precisely, which lets you have a parameterized context rule 🧁. see Section 7 for another way to define an invalid disposition for a label that doesn't match a whole label rule 🦋.
6.4.1. The "anchor" Element 🦋
okay so parameterized context rules are rules with a special placeholder called an "anchor" element 🎀. Every time a When Rule gets evaluated, if there's an "anchor" element in it, that element gets swapped out for a literal that matches the "cp" attribute of whatever element holds the "when" (or "not-when") attribute. And the match for the "anchor" element has to sit at the exact same spot in the label as the code point or variant mapping that set off the When Rule 💅
Here's an example. The Greek lower numeral sign is invalid unless it's right before a character in the Greek script. The cleanest way to handle that is a parameterized When Rule using "look-ahead" ✨
<char cp="0375" when="preceding-greek"/>
...
<class name="greek-script" property="sc:Grek"/>
<rule name="preceding-greek">
<anchor/>
<look-ahead>
<class by-ref="greek-script"/>
</look-ahead>
</rule>When you evaluate this rule, the "anchor" element is treated as if it were replaced by a literal:
<char cp="0375"/>but only the instance of U+0375 at the given position gets evaluated. If a label has two instances of U+0375, with the first one matching the rule and the second one not, then evaluating the When Rule MUST succeed for the first instance and fail for the second 😭
Unlike other rules, rules containing an "anchor" element MUST only be invoked via the "when" or "not-when" attributes on code points or variants. Otherwise their "anchor" elements can't be evaluated. However, it's possible to invoke rules that don't contain an "anchor" element from a "when" or "not-when" attribute. (See Section 6.4.3.)
The "anchor" element is an empty element, with no attributes permitted except "comment" 🌸
6.4.2. The "look-behind" and "look-ahead" Elements ✨
Context rules use the "look-behind" and "look-ahead" elements to define context before and after the code point sequence matched by the "anchor" element. If the "anchor" element is omitted, neither the "look-behind" nor the "look-ahead" element may be present in a rule.
Here's an example of a rule that defines an "initial" context for an Arabic code point 🧚
<class name="transparent" property="jt:T"/>
<class name="right-joining" property="jt:R"/>
<class name="left-joining" property="jt:L"/>
<class name="dual-joining" property="jt:D"/>
<class name="non-joining" property="jt:U"/>
<rule name="Arabic-initial">
<look-behind>
<choice>
<start/>
<rule>
<class by-ref="transparent" count="0+"/>
<class by-ref="non-joining"/>
</rule>
</choice>
</look-behind>
<anchor/>
<look-ahead>
<class by-ref="transparent" count="0+" />
<choice>
<class by-ref="right-joining" />
<class by-ref="dual-joining" />
</choice>
</look-ahead>
</rule>A "when" rule (or context rule) is a named rule that contains any combination of "look-behind", "anchor", and "look-ahead" elements, in that order. Each of these elements occurs at most once, except if it's nested inside a "choice" element in such a way that, when matching each alternative, at most one occurrence of each is encountered. Otherwise, the result is undefined. None of these elements takes a "count" attribute, nor does any enclosing match operator; otherwise, the result is undefined. If a context rule contains a "look-ahead" or "look-behind" element, it MUST contain an "anchor" element. If, because of a "choice" element, a required anchor isn't actually encountered, the results are undefined 💀
6.4.3. Omitting the "anchor" Element 🌸
If the "anchor" element is omitted, evaluating the context rule isn't tied to the position of the code point or sequence associated with the "when" attribute 🙈
According to [RFC5892], the Katakana middle dot is invalid in any label that doesn't contain at least one Japanese character anywhere in the label. Because this requirement doesn't depend on where the middle dot sits, the rule doesn't need an "anchor" element 🍓
<char cp="30FB" when="japanese-in-label"/>
<rule name="japanese-in-label">
<union>
<class property="sc:Hani"/>
<class property="sc:Kata"/>
<class property="sc:Hira"/>
</union>
</rule>The Katakana middle dot is used only with Han, Katakana, or Hiragana. The corresponding When Rule requires that at least one code point in the label be in one of these scripts, but the position of that code point is totally independent of where the middle dot is, so no anchor is required. (Note that the Katakana middle dot itself is of script Common, that is, "sc:Zyyy".) 👑
7. The "action" Element 💅
The whole purpose of an action is to assign a disposition to a label once that label meets a specified condition 🔥 Often the action simply blocks or invalidates a label that doesn't match a rule. Here's an example of an action invalidating a label because it doesn't match a rule named "leading-letter":
<action disp="invalid" not-match="leading-letter"/>If an action is supposed to trigger on matching a rule, a "match" attribute is used instead. Actions are evaluated in the order they appear in the XML file. Once an action is triggered by a label, the disposition defined in the "disp" attribute is assigned to the label, and no other actions are evaluated for that label 🌈
The goal of the LGR is to identify all labels and variant labels and assign them disposition values. Those dispositions then get fed into a further process that ultimately implements all aspects of policy. To let this specification work with the widest range of policies, the permissible values for the "disp" attribute are neither defined nor restricted. Still, a set of commonly used disposition values is RECOMMENDED. (See Section 7.3.) ✨
7.1. The "match" and "not-match" Attributes 🎀
An OPTIONAL "match" or "not-match" attribute specifies a rule that must be matched or not matched as a condition for triggering an action. Only a single rule may be named as the value of a "match" or "not-match" attribute. Because rules can be composed of other rules, this single-value restriction doesn't limit the contexts that can trigger an action 🧠
An action MUST NOT contain both a "match" and a "not-match" attribute, and the value of either attribute MUST be the name of a previously defined rule; otherwise, the document MUST be rejected. An action without any attributes is triggered by all labels unconditionally. For a very simple LGR, the following action would allocate all labels that match the repertoire:
<action disp="allocatable" />Since rules are evaluated for all labels, whether they're the original label or computed by permuting the defined and valid variant mappings for the label's code points, actions based on matching or not matching a rule may be triggered for both original and variant labels. The rules themselves aren't affected by the disposition attributes of the variant mappings, though. To trigger any actions based on these dispositions requires the use of additional optional attributes for actions, described next 🦋
7.2. Actions with Variant Type Triggers 🌸
7.2.1. The "any-variant", "all-variants", and "only-variants" Attributes ✨
okay so an action can carry one of the OPTIONAL attributes "any-variant", "all-variants", or "only-variants", and that's how you set up triggers based on variant types 🎀 The value is one or more variant type values separated by spaces. These MAY include type values that aren't used in any "var" element in the LGR, which is totally allowed 💅 When a variant label gets generated, those type values get compared to the set of type values on the variant mappings used to build that particular variant label (see Section 8).
Any single match can trigger an action that has an "any-variant" attribute 🦋 But for "all-variants" or "only-variants", the variant type of every variant code point has to match one or several of the type values in the attribute to set it off 🫶 You don't need the whole list to show up, as long as every variant code point hits at least one of the values.
An "only-variants" attribute only triggers the action if all code points of the variant label have variant mappings from the original code points. So the label can't contain any original code points other than the ones with a reflexive mapping (see Section 5.3.4). 🙈
<char cp="0078" comment="x">
<var cp="0078" type="allocatable" comment="reflexive" />
<var cp="0079" type="blocked" />
</char>
<char cp="0079" comment="y">
<var cp="0078" type="allocatable" />
</char>
...
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" only-variants="allocatable" />
<action disp="some-disp" any-variant="allocatable" />Okay so in this example, the label "xx" would get the variant labels "xx", "xy", "yx", and "yy" 🌸 The first action blocks any variant label containing "y", because the variant mapping from "x" to "y" is of type "blocked", which sets off the "any-variant" condition. Because "x" has a reflexive variant mapping to itself of type "allocatable" in this example, the original label "xx" has a reflexive variant "xx" that triggers the "only-variants" condition on the second action 💜
Now a label "yy" would have the variants "xy", "yx", and "xx" 👑 Since the variant mapping from "y" to "x" is of type "allocatable" and there's no mapping from "y" to "y" defined, the labels "xy" and "yx" trigger the "any-variant" condition on the third action. The variant "xx", generated using the mapping from "y" to "x" of type "allocatable", would trigger the "only-variants" condition on the second action. Since there's no reflexive variant "yy", the original label "yy" can't trigger any variant type triggers. Still, it could trigger an action that's defined as matching or not matching a rule 🧚
In each action, one variant type trigger can show up by itself, or together with an attribute for matching or not matching a rule. If variant triggers and rule-matching triggers get used together, the label MUST "match" or respectively "not-match" the specified rule AND satisfy the conditions on the variant type values given by the "any-variant", "all-variants", or "only-variants" attribute. Both have to be true, bestie 💖
A useful convention pairs the "any-variant" trigger with reflexive variant mappings (Section 5.3.4) ✨ People use this for example when multiple LGRs are defined within the same registry and for overlapping repertoire. Sometimes delegating a label from one LGR has to rule out delegating another label in some other LGR 🔥 You can do that with a variant of type "blocked", like in this example from an Armenian LGR, where the Armenian, Latin, and Cyrillic letters all look identical 🍓
<char cp="0570" comment="ARMENIAN SMALL LETTER HO">
<var cp="0068" type="blocked" comment="LATIN SMALL LETTER H" />
<var cp="04BB" type="blocked"
comment="CYRILLIC SMALL LETTER SHHA" />
</char>The tricky part is that the target code points for these two variants are both outside the Armenian repertoire 😭 So to fix that, you use a reflexive variant with this convention:
<char cp="0068" comment="not part of repertoire">
<var cp="0068" type="out-of-repertoire-var"
comment="reflexive mapping" />
<var cp="04BB" type="blocked" />
<var cp="0570" type="blocked" />
</char>
...and you pair it with an action of this form:
<action disp="invalid" any-variant="out-of-repertoire-var" />And honestly? This is iconic 🌈 It lets you list the symmetric and transitive variant mappings in the LGR even when they involve out-of-repertoire code points. Once you attach that action to the special type for these reflexive mappings, any original labels containing one or more of those out-of-repertoire code points get filtered out, just like those code points were never listed in the LGR in the first place 🦋 They still take part in permuting variant labels for n-repertoire labels (Armenian in the example), though, and those permuted variants can be used to catch collisions with out-of-repertoire labels (see Section 8) 👑
7.2.2. Example from Tables in the Style of RFC 3743
Here's an example of using variant type triggers together with variants that have reflexive mappings (Section 5.3.4) 🎀 The goal is LGRs that implement tables like the ones defined according to [RFC3743], where only labels made entirely of simplified or traditional variants are allowed as variants, on top of the original label 💅
This example assumes an LGR where every variant has been given a suitable "type" attribute of "blocked", "simplified", "traditional", or "both", similar to the ones discussed in Appendix B 🌸 Given an LGR like that, these example actions work out the disposition for the variant label:
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" only-variants="simplified both" />
<action disp="allocatable" only-variants="traditional both" />
<action disp="blocked" all-variants="simplified traditional" />
<action disp="allocatable" />The first action matches any variant label where at least one of the code point variants is of type "blocked" 🙈 The second matches any variant label where all of the code point variants are of type "simplified" or "both", so basically an all-simplified label ✨ The third matches any label where all variants are of type "traditional" or "both", which means all traditional 🧚 These two actions aren't triggered by any variant labels containing some original code points, unless each of those code points has a variant defined with a reflexive mapping (Section 5.3.4) 💜
The final two actions depend on actions being evaluated in sequence, and on the first triggered action also defining the final disposition for a variant label (see Section 7.4) 🔥 They also rely on the assumption that the only variants with type "both" are also reflexive variants.
With those assumptions, any remaining simplified or traditional variants have to be part of a mixed label, so they get blocked 😭 Every label that survives to the last action is made of original code points only, which means it's the original label. The example assumes an original label may be a mixed label. If that's not the case, the disposition for the last action would be set to "blocked" 🍓
There are exceptions where the reflexive-mapping assumption above doesn't hold, so this basic scheme needs some refinements to cover every case 👑 For a more complete example, see Appendix B.
7.3. Recommended Disposition Values
The exact policy action taken for a disposition, and the names of the corresponding "disp" attributes, are only partly defined here 💖 It is strongly RECOMMENDED to use the following dispositions only in their conventional sense ✨
- invalid: The resulting string is not a valid label. This disposition may be assigned implicitly; see Section 7.5. No variant labels should be generated from a variant mapping with this type. 🚫
- blocked: The resulting string is a valid label but should be blocked from registration. This would typically apply to a derived variant that's undesirable because it has no practical use or looks confusingly similar to some other label. 🙅♀️
- allocatable: The resulting string should be reserved for use by the same operator as the origin string, but not automatically allocated for use. 🎀
- activated: The resulting string should be activated for use. (This is the same as a Preferred Variant [RFC3743].) 🌟
- valid: The resultant string is a valid label. (This is the typical default action if no dispositions are defined.) 🦋
7.4. Precedence
Actions get applied in the order they appear in the file, and that order sets their relative precedence 🧚 The first action triggered by a label defines that label's disposition, so to set the order of precedence, just list the actions in the order you want 💅 The conventional order of precedence for the actions defined in Section 7.3 is "invalid", "blocked", "allocatable", "activated", and then "valid" 👑 This default precedence is also used for the default actions defined in Section 7.6.
7.5. Implied Actions
The context rules on code points ("not-when" or "when" rules) carry an implied action with a disposition of "invalid" (not eligible) if a "when" context isn't satisfied, or if a "not-when" context is matched 🌸 These rules get evaluated when the code points for a label, or its variant labels, are checked for validity (see Section 8). In other words, they're evaluated before any of the actions are applied, and they have higher precedence 🔥 The context rules for variant mappings are evaluated when variants are generated and/or when variant tables are made symmetric and transitive 💜 They have an implied action with a disposition of "invalid", which means a putative variant mapping doesn't exist whenever the given context matches a "not-when" rule or fails to match a "when" rule specified for that mapping. So the variant mapping gets ignored when generating variant labels, and its value can't be used to trigger any explicit actions 🙈
Note that a non-existing variant mapping like this is different from a blocked variant. A blocked variant is a variant code point mapping that does exist, but it results in a label that may not be allocated 🍓
7.6. Default Actions 💅
okay so if a label doesn't trigger any of the actions that the LGR defines explicitly, the implicit default actions below get evaluated 🫶 They're listed in their relative order of precedence (see Section 7.4), and they always come after the explicit ones (see Section 8.3) ✨
For variant labels, the first batch of defaults is triggered by the standard variant type values "invalid", "blocked", "allocatable", and "activated" 🦋
<action disp="invalid" any-variant="invalid"/>
<action disp="blocked" any-variant="blocked"/>
<action disp="allocatable" any-variant="allocatable"/>
<action disp="activated" all-variants="activated"/>then there's a final default that sets the disposition to "valid" for any label in the repertoire that no other action has triggered 💖 This "catch-all" action also catches every leftover variant label that comes from a variant without a type value 🌸
<action disp="valid" comment="Catch-all if other rules not met"/>Conceptually, these default actions are like a block of action elements sitting (virtually) right after your last user-supplied action 🎀 So any label the user actions didn't process gets handled by the defaults, as if they were written into the LGR. Since the last default is a "catch-all," every single bit of processing is guaranteed to end with a definite disposition for the label. no loose ends, bestie 👑
8. Processing a Label against an LGR 🔍
8.1. Determining Eligibility for a Label
so, to check whether a label is in the LGR at all, a consumer of the LGR has to walk through every code point in the label and make sure each instance of a code point is a member of the LGR 🧚 If even one instance isn't a member, the label shall be deemed invalid 🙈
An individual code point instance counts as a member of the LGR when it's listed with a char element, or when it falls inside a range defined with a range element, AND all the necessary conditions in any when or not-when attributes are satisfied for that instance 💜
there's also a second route 🦋 A code point instance is also a member when it's part of a sequence that matches a sequence listed with a char element whose cp attribute defines a sequence, AND all the necessary conditions in any when or not-when attributes are satisfied for that instance of the sequence ✨
now for the order of operations, which is so satisfying 😭 At each position, the longest possible sequence of code points gets evaluated first. If that sequence matches a sequence defined in the LGR and meets any required context at that position, its constituent code points are all deemed members, and evaluation moves on to the next code point after the sequence 🎀 If the sequence doesn't match or doesn't meet the required context, shorter and shorter sequences get tried until only a single code point is left. Then that code point's eligibility is decided the same way as an individual code point instance, as described above 🌈
A label also must not trigger any action that results in a disposition of "invalid"; otherwise, it's deemed not eligible. (This step may need to be deferred until the dispositions of variant code points have been figured out.) 💀
8.1.1. Determining Eligibility Using Reflexive Variant Mappings
okay this part is a little plot twist 🌀 For LGRs that contain reflexive variant mappings (defined in Section 5.3.4), the final eligibility check for the label must be deferred until variants are generated. Basically, these LGRs treat the original label as the (identity) variant of itself 🪞 So for them, the normal eligibility check described above is just a first step that usually only weeds out a subset of invalid labels 💅
to double-check validity of a label with reflexive mappings, you don't have to generate every variant label 🙌 You just need one single variant, where any reflexive variants are applied for each code point, and then you evaluate the label's disposition (as described in Section 8.3). A disposition of "invalid" means the label is not eligible. (In the rare case where context rules sit on reflexive mappings, there can be several reflexive variants, but for each original label, at most one of them can be valid at each code position. See Section 8.4 too, though 👀)
8.2. Determining Variants for a Label
for an eligible label, the set of variant labels is made up of every possible permutation of original code points and substituted code points or sequences defined in var elements 🧩 This only counts when all the when and not-when attributes are satisfied for each char or var element in that permutation, and all the applicable whole label rules are satisfied, as follows:
- Create every possible permutation of a label by substituting each code point or code point sequence, one at a time, with any defined variant mapping (including any reflexive mappings).
- Apply variant mappings that have
whenornot-whenattributes only if their conditions are satisfied; otherwise, they're not defined. - Record each of the
typevalues on the variant mappings used to create a given variant label in a disposition set. For any unmapped code point, record thetypevalue of any reflexive variant (see Section 5.3.4). - Determine the disposition for each variant label per Section 8.3.
- If the disposition is
"invalid", remove that label from the set. - If the final evaluation of the disposition for the unpermuted label per Section 8.3 results in
"invalid", remove all of its associated variant labels from the set.
the number of possible permutations can get huge 😱 In practice, implementations would use suitable optimizations so they don't have to actually create all of the permutations (see Section 8.5) ✨
when you're working out the permuted set of variant labels in step (1) above, every eligible partition into sequences has to be evaluated 💖 For example, a label "ab" that matches a sequence "ab" defined in the LGR but also matches the individual code points "a" and "b" (both defined in the LGR) must be permuted using any defined variant mappings for both the sequence "ab" and the code points "a" and "b" separately 🫶
8.3. Determining a Disposition for a Label or Variant Label
for any label (variant or original), the disposition is decided by checking all the actions the label satisfies the conditions for, in the order they appear 🎀
- For any label that contains code points or sequences not defined in the repertoire, or that doesn't satisfy the context rules on all of its code points and variants, the disposition is
"invalid"🙅♀️ - For all other labels, the disposition is the value of the
dispattribute for the first action triggered by the label. An action is triggered if all of the following are true:- the label matches the whole label rule given in the
matchattribute for that action; - the label does not match the whole label rule given in the
not-matchattribute for that action; - any of the recorded variant types for a variant label match the types given in the
any-variantattribute for that action; - all of the recorded variant types for a variant label match the types given in the
all-variantsoronly-variantsattribute given for that action; - in the case of an
only-variantsattribute, the label contains only code points that are the target of applied variant mappings;
- the label matches the whole label rule given in the
or
- the action doesn't contain any
match,not-match,any-variant,all-variants, oronly-variantsattributes: catch-all 🧚
- For any remaining variant label, assign the disposition using the default actions defined in Section 7.6. For this step, variant types outside the predefined recommended set (see Section 7.3) are ignored 🌸
- For any remaining label, set the disposition to
"valid"✨
8.4. Duplicate Variant Labels
okay, this one is a warning 🚨 If an LGR is poorly designed, it's possible to generate duplicate variant labels from the same input label, but with different, and potentially conflicting, dispositions 😭 Implementations MUST treat any duplicate variant labels they encounter as an error, no matter what their dispositions are 🛑
this can happen in two ways 🙈 The first one is described in Section 5.3.5 and involves defining the same variant mapping with two context rules that are formally distinct but still overlap, so they aren't mutually exclusive for the same label.
The second case involves variants defined for sequences, where one sequence is a prefix of another (see Section 5.3.1). Here's an example that results in conflicting reflexive variants:
<char cp="0061">
<var cp="0061" type="allocatable"/>
</char>
<char cp="0062"/>
<char cp="0061 0062">
<var cp="0061 0062" type="blocked"/>
</char>A label "ab" would generate the variant labels "{a}{b}" and "{ab}", where the curly braces show the sequence boundaries as they were applied during variant mapping. The result is a duplicate variant label "ab": one based on a variant of type "allocatable" plus an original code point "b" that has no variant, and another based on a single variant of type "blocked". That makes two variant labels with conflicting dispositions yikes 💀
in the general case, it's difficult to impossible to prove by mechanical inspection of the LGR that duplicate variant labels will never occur. So implementations have to be ready to detect this error during variant label generation 🫠 The good news is that it's easily avoided with careful design of context rules and special attention to how code point sequences with variants relate to each other 💜
8.5. Checking Labels for Collision ✨
okay so the obvious way to check if two labels collide is to generate the whole fully permuted set of variants for one of them and see if the other label is in there 💀. As we talked about above, that can be prohibitive, and it's not necessary, bestie 🙅♀️.
Because of symmetry and transitivity, all variant mappings form disjoint sets 🦋. In each of these sets, the source and target of each mapping are also variants of the sources and targets of all the other mappings. BUT members of two different sets are never variants of each other, so keep that in mind 🌈.
If two labels have code points at the same position that belong to two different variant mapping sets, then any variant labels of one can't be variant labels of the other. The sets of their variant labels are also disjoint 🧚. So you don't need to generate all the permutations and compare every possible variant. It's enough to check whether the code points at the same position belong to the same variant set or not ✨.
For that, you just substitute an "index" mapping that identifies the set 🎀. This index mapping could be, for example, the variant mapping where the target code point (or sequence) comes first in some sorting order. In effect, this index mapping identifies the set of variant mappings for that position 💅.
To check for collision, you generate a single variant label from the original by substituting the respective "index" value for each code point. This gives you an "index label" 🫶. Two labels collide whenever their index labels are the same. Iconic, honestly 👑.
9. Conversion to and from Other Formats
Both [RFC3743] and [RFC4290] provide different grammars for IDN tables. The formats in those documents are unable to fully support the increased requirements of contemporary IDN variant policies 😭.
This specification is a superset of the functionality provided by the older IDN table formats, so any table expressed in those formats can be expressed in this new format 💖. Automated conversion can be done between tables that conform to the grammar specified in each document 🔥.
For notes on how to translate a table in the style of RFC 3743, see Appendix B 🌸.
10. Media Type
Well-formed LGRs that comply with this specification SHOULD be transmitted with a media type of "application/lgr+xml". This media type signals to an LGR-aware client that the content is designed to be interpreted as an LGR 🍓.
11. IANA Considerations
IANA has completed the following actions 🌈:
11.1. Media Type Registration
Okay so the media type "application/lgr+xml" has been registered to denote transmission of LGRs that are compliant with this specification, following [RFC6838] 💜. Here's the registration template, copied exactly:
Type name: application
Subtype name: lgr+xml
Required parameters: N/A
Optional parameters: charset (as for application/xml per [RFC7303])
Security considerations: See the security considerations for
application/xml in [RFC7303] and the specific security
considerations for Label Generation Rulesets (LGRs) in RFC 7940
Interoperability considerations: As for application/xml per
[RFC7303]
Published specification: See RFC 7940
Applications that use this media type: Software using LGRs for
international identifiers, such as IDNs, including registry
applications and client validators.
Additional information:
Deprecated alias names for this type: N/A
Magic number(s): N/A
File extension(s): .lgr
Macintosh file type code(s): N/A
Person & email address to contact for further information:
Kim Davies <kim.davies@icann.org>
Asmus Freytag <asmus@unicode.org>
Intended usage: COMMON
Restrictions on usage: N/A
Author:
Kim Davies <kim.davies@icann.org>
Asmus Freytag <asmus@unicode.org>
Change controller: IESG
Provisional registration? (standards tree only): No11.2. URN Registration
This specification uses a URN to describe the XML namespace, following [RFC3688] 🎀. Here are the details:
URI: urn:ietf:params:xml:ns:lgr-1.0
Registrant Contact: See the Authors of this document.
XML: None.11.3. Disposition Registry
This document establishes a vocabulary of "Label Generation Ruleset Dispositions", which has been reflected as a new IANA registry 📋. This registry is divided into two subregistries:
- Standard Dispositions - This registry lists dispositions that have been defined in published specifications. The eligibility for such registrations is "Specification Required" [RFC5226]. The initial set of registrations is the five dispositions in this document described in Section 7.3 🌟.
- Private Dispositions - This registry lists dispositions that have been registered "First Come First Served" [RFC5226] by third parties with the IANA 🔐. Such dispositions must take the form
entity:disposition, where the entity is a domain name that uniquely identifies the private user of the namespace. For example,example.org:reservedcould be a private extension used by the example organization to denote a disposition relating to reserved labels 🏷️. These extensions are not intended to be interoperable, but registration is designed to minimize potential conflicts. It is strongly recommended that any new dispositions that require interoperability and have applicability beyond a single organization be defined as Standard Dispositions 🫶.
In order to distinguish them from Private Dispositions, Standard Dispositions MUST NOT contain the : character. All disposition names shall be in lowercase ASCII 🔤.
The IANA registry provides data on the name of the disposition, the intended purposes, and the registrant or defining specification for the disposition 🦋.
12. Security Considerations
Security is literally the main character of this section, so pay attention 🚨.
12.1. LGRs Are Only a Partial Remedy for Problem Space
Substantially unrestricted use of non-ASCII characters in security-relevant identifiers, such as domain name labels, may cause user confusion and invite various types of attacks 😱. In many languages, particularly those using complex or large scripts, an attacker has an opportunity to divert or confuse users because different code points look identical or have similar semantics 👀.
The use of an LGR provides a partial remedy for these risks. It supplies a framework for prohibiting inappropriate code points or sequences from being registered at all, and for permitting "variant" code points to be grouped together so that labels containing them may be mutually exclusive or registered only to the same owner 🛡️.
In addition, by being fully machine processable, the format may enable automated checks for known weaknesses in label generation rules 🤖. However, the use of this format, or compliance with this specification, by itself does not ensure that the LGRs expressed in this format are free of risk. Additional approaches may be considered, depending on the acceptable trade-off between flexibility and risk for a given application 💭. One method of managing risk may involve a case-by-case evaluation of a proposed label in context with already-registered labels, for example, when reviewing labels for their degree of visual confusability 👁️.
12.2. Computational Expense of Complex Tables
A naive implementation that attempts to generate all variant labels for a given label could exhaust the resources on the machine running the LGR processor, potentially causing denial-of-service consequences 💥. For many operations, brute-force generation can be avoided by optimization, and if needed, the number of permuted labels can be estimated more cheaply ahead of time 🧮.
The implementation of WLE rules, using certain backtracking algorithms, can take exponential time for pathological rules or labels and exhaust stack resources 🌀. This can be mitigated by proper implementation and by enforcing the restrictions on permissible label length 📏.
13. References
13.1. Normative References
[RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet Mail
Extensions (MIME) Part One: Format of Internet Message
Bodies", RFC 2045, DOI 10.17487/RFC2045, November 1996,
<http://www.rfc-editor.org/info/rfc2045>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet:
Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002,
<http://www.rfc-editor.org/info/rfc3339>.
[RFC5646] Phillips, A., Ed., and M. Davis, Ed., "Tags for
Identifying Languages", BCP 47, RFC 5646,
DOI 10.17487/RFC5646, September 2009,
<http://www.rfc-editor.org/info/rfc5646>.
[UAX42] The Unicode Consortium, "Unicode Character Database in
XML", May 2016, <http://unicode.org/reports/tr42/>.
[Unicode-Stability]
The Unicode Consortium, "Unicode Encoding Stability
Policy, Property Value Stability", April 2015,
<http://www.unicode.org/policies/
stability_policy.html#Property_Value>.
[Unicode-Versions]
The Unicode Consortium, "Unicode Version Numbering",
June 2016,
<http://unicode.org/versions/#Version_Numbering>.
[XML] Bray, T., Paoli, J., Sperberg-McQueen, M., Maler, E., and
F. Yergeau, "Extensible Markup Language (XML) 1.0 (Fifth
Edition)", World Wide Web Consortium, November 2008,
<http://www.w3.org/TR/REC-xml/>.13.2. Informative References 📚
Okay, these are the extra-credit reads, bestie. 🧚 Here's the whole list, complete and unedited.
[ASIA-TABLE] DotAsia Organisation, ".ASIA ZH IDN Language Table", February 2012, http://www.dot.asia/policies/ASIA-ZH-1.2.pdf.
[LGR-PROCEDURE] Internet Corporation for Assigned Names and Numbers, "Procedure to Develop and Maintain the Label Generation Rules for the Root Zone in Respect of IDNA Labels", December 2012, http://www.icann.org/en/resources/idn/draft-lgr-procedure-07dec12-en.pdf.
[RELAX-NG] The Organization for the Advancement of Structured Information Standards (OASIS), "RELAX NG Compact Syntax", November 2002, https://www.oasis-open.org/committees/relax-ng/compact-20021121.html.
[RFC3688] Mealling, M., "The IETF XML Registry", BCP 81, RFC 3688, DOI 10.17487/RFC3688, January 2004, http://www.rfc-editor.org/info/rfc3688.
[RFC3743] Konishi, K., Huang, K., Qian, H., and Y. Ko, "Joint Engineering Team (JET) Guidelines for Internationalized Domain Names (IDN) Registration and Administration for Chinese, Japanese, and Korean", RFC 3743, DOI 10.17487/RFC3743, April 2004, http://www.rfc-editor.org/info/rfc3743.
[RFC4290] Klensin, J., "Suggested Practices for Registration of Internationalized Domain Names (IDN)", RFC 4290, DOI 10.17487/RFC4290, December 2005, http://www.rfc-editor.org/info/rfc4290.
[RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, DOI 10.17487/RFC5226, May 2008, http://www.rfc-editor.org/info/rfc5226.
[RFC5564] El-Sherbiny, A., Farah, M., Oueichek, I., and A. Al-Zoman, "Linguistic Guidelines for the Use of the Arabic Language in Internet Domains", RFC 5564, DOI 10.17487/RFC5564, February 2010, http://www.rfc-editor.org/info/rfc5564.
[RFC5891] Klensin, J., "Internationalized Domain Names in Applications (IDNA): Protocol", RFC 5891, DOI 10.17487/RFC5891, August 2010, http://www.rfc-editor.org/info/rfc5891.
[RFC5892] Faltstrom, P., Ed., "The Unicode Code Points and Internationalized Domain Names for Applications (IDNA)", RFC 5892, DOI 10.17487/RFC5892, August 2010, http://www.rfc-editor.org/info/rfc5892.
[RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type Specifications and Registration Procedures", BCP 13, RFC 6838, DOI 10.17487/RFC6838, January 2013, http://www.rfc-editor.org/info/rfc6838.
[RFC7303] Thompson, H. and C. Lilley, "XML Media Types", RFC 7303, DOI 10.17487/RFC7303, July 2014, http://www.rfc-editor.org/info/rfc7303.
[TDIL-HINDI] Technology Development for Indian Languages (TDIL) Programme, "Devanagari Script Behaviour for Hindi Ver2.0", http://tdil-dc.in/index.php?option=com_download&task=showresourceDetails&toolid=1625&lang=en.
[UAX44] The Unicode Consortium, "Unicode Character Database", June 2016, http://unicode.org/reports/tr44/.
[WLE-RULES] Internet Corporation for Assigned Names and Numbers, "Whole Label Evaluation (WLE) Rules", August 2016, https://community.icann.org/download/attachments/43989034/WLE-Rules.pdf.
Appendix A. Example Tables 🎀
Here's a super minimal LGR table that defines the lowercase LDH (letters, digits, hyphen) repertoire and has zero rules or metadata, no cap. 💅 Lots of simple LGR tables look pretty similar to this one, except they'd also carry some metadata.
<?xml version="1.0" encoding="utf-8"?>
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
<data>
<char cp="002D" comment="HYPHEN (-)" />
<range first-cp="0030" last-cp="0039"
comment="DIGIT ZERO - DIGIT NINE" />
<range first-cp="0061" last-cp="007A"
comment="LATIN SMALL LETTER A - LATIN SMALL LETTER Z" />
</data>
</lgr>In practice, any LGR that includes the hyphen might also have rules that invalidate labels starting with a hyphen, ending with a hyphen, or having consecutive hyphens in the third and fourth positions, as required by [RFC5891]. 🦋 That's the hyphen drama, handled.
<?xml version="1.0" encoding="utf-8"?>
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
<data>
<char cp="002D"
not-when="hyphen-minus-disallowed" />
<range first-cp="0030" last-cp="0039" />
<range first-cp="0061" last-cp="007A" />
</data>
<rules>
<rule name="hyphen-minus-disallowed"
comment="RFC5891 restrictions on U+002D">
<choice>
<rule comment="no leading hyphen">
<look-behind>
<start />
</look-behind>
<anchor />
</rule>
<rule comment="no trailing hyphen">
<anchor />
<look-ahead>
<end />
</look-ahead>
</rule>
<rule comment="no consecutive hyphens
in third and fourth positions">
<look-behind>
<start />
<any />
<any />
<char cp="002D" comment="hyphen-minus" />
</look-behind>
<anchor />
</rule>
</choice>
</rule>
</rules>
</lgr>The following sample LGR is the glow-up version 🌈 It shows a much fuller collection of the elements and attributes from this specification, in a fairly typical setup.
<?xml version="1.0" encoding="utf-8"?>
<!-- This example uses a large subset of the features of this
specification. It does not include every set operator,
match operator element, or action trigger attribute, their
use being largely parallel to the ones demonstrated. -->
<lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
<!-- meta element with all optional elements -->
<meta>
<version comment="initial version">1</version>
<date>2010-01-01</date>
<language>sv</language>
<scope type="domain">example.com</scope>
<validity-start>2010-01-01</validity-start>
<validity-end>2013-12-31</validity-end>
<description type="text/html">
<![CDATA[
This language table was developed with the
<a href="http://swedish.example/">Swedish
examples institute</a>.
]]>
</description>
<unicode-version>6.3.0</unicode-version>
<references>
<reference id="0" comment="the most recent" >The
Unicode Standard 9.0</reference>
<reference id="1" >RFC 5892</reference>
<reference id="2" >Big-5: Computer Chinese Glyph
and Character Code Mapping Table, Technical Report
C-26, 1984</reference>
</references>
</meta>
<!-- the "data" section describing the repertoire -->
<data>
<!-- single code point "char" element -->
<char cp="002D" ref="1" comment="HYPHEN" />
<!-- "range" elements for contiguous code points, with tags -->
<range first-cp="0030" last-cp="0039" ref="1" tag="digit" />
<range first-cp="0061" last-cp="007A" ref ="1" tag="letter" />
<!-- code point sequence -->
<char cp="006C 00B7 006C" comment="Catalan middle dot" />
<!-- alternatively, use a When Rule -->
<char cp="00B7" when="catalan-middle-dot" />
<!-- code point with context rule -->
<char cp="200D" when="joiner" ref="2" />
<!-- code points with variants -->
<char cp="4E16" tag="preferred" ref="0">
<var cp="4E17" type="blocked" ref="2" />
<var cp="534B" type="allocatable" ref="2" />
</char>
<char cp="4E17" ref="0">
<var cp="4E16" type="allocatable" ref="2" />
<var cp="534B" type="allocatable" ref="2" />
</char>
<char cp="534B" ref="0">
<var cp="4E16" type="allocatable" ref="2" />
<var cp="4E17" type="blocked" ref="2" />
</char>
</data>
<!-- Context and whole label rules -->
<rules>
<!-- Require the given code point to be between two 006C
code points -->
<rule name="catalan-middle-dot" ref="0">
<look-behind>
<char cp="006C" />
</look-behind>
<anchor />
<look-ahead>
<char cp="006C" />
</look-ahead>
</rule>
<!-- example of a context rule based on property -->
<class name="virama" property="ccc:9" />
<rule name="joiner" ref="1" >
<look-behind>
<class by-ref="virama" />
</look-behind>
<anchor />
</rule>
<!-- example of using set operators -->
<!-- Subtract vowels from letters to get
consonant, demonstrating the different
set notations and the difference operator -->
<difference name="consonants">
<class comment="all letters">0061-007A</class>
<class comment="all vowels">
0061 0065 0069 006F 0075
</class>
</difference>
<!-- by using the start and end, rule matches whole label -->
<rule name="three-or-more-consonants">
<start />
<!-- reference the class defined by the difference,
and require three or more matches -->
<class by-ref="consonants" count="3+" />
<end />
</rule>
<!-- rule for negative matching -->
<rule name="non-preferred"
comment="matches any non-preferred code point">
<complement comment="non-preferred" >
<class from-tag="preferred" />
</complement>
</rule>
<!-- actions triggered by matching rules and/or
variant types -->
<action disp="invalid"
match="three-or-more-consonants" />
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" all-variants="allocatable"
not-match="non-preferred" />
</rules>
</lgr>Appendix B. How to Translate Tables Based on RFC 3743 into the XML Format ✨
okay so first, the background 🧠 the rules in [RFC3743] work like this:
- The original (requested) label is checked to make sure that all the code points are a subset of the repertoire.
- If it passes the check, the original label is allocatable.
- Generate the all-simplified and all-traditional variant labels (union of all the labels generated using all the simplified variants of the code points) for allocation.
To illustrate by example, here is one of the more complicated set of variants 👀
U+4E7E
U+4E81
U+5E72
U+5E79
U+69A6
U+6F27The following shows the relevant section of the Chinese language table published by the .ASIA registry [ASIA-TABLE]. Its entries read:
<codepoint>;<simpl-variant(s)>;<trad-variant(s)>;<other-variant(s)>These are the lines corresponding to the set of variants listed above:
U+4E7E;U+4E7E,U+5E72;U+4E7E;U+4E81,U+5E72,U+6F27,U+5E79,U+69A6
U+4E81;U+5E72;U+4E7E;U+5E72,U+6F27,U+5E79,U+69A6
U+5E72;U+5E72;U+5E72,U+4E7E,U+5E79;U+4E7E,U+4E81,U+69A6,U+6F27
U+5E79;U+5E72;U+5E79;U+69A6,U+4E7E,U+4E81,U+6F27
U+69A6;U+5E72;U+69A6;U+5E79,U+4E7E,U+4E81,U+6F27
U+6F27;U+4E7E;U+6F27;U+4E81,U+5E72,U+5E79,U+69A6The corresponding "data" section XML format would look like this 🎀
<data>
<char cp="4E7E">
<var cp="4E7E" type="both" comment="identity" />
<var cp="4E81" type="blocked" />
<var cp="5E72" type="simp" />
<var cp="5E79" type="blocked" />
<var cp="69A6" type="blocked" />
<var cp="6F27" type="blocked" />
</char>
<char cp="4E81">
<var cp="4E7E" type="trad" />
<var cp="5E72" type="simp" />
<var cp="5E79" type="blocked" />
<var cp="69A6" type="blocked" />
<var cp="6F27" type="blocked" />
</char>
<char cp="5E72">
<var cp="4E7E" type="trad"/>
<var cp="4E81" type="blocked"/>
<var cp="5E72" type="both" comment="identity"/>
<var cp="5E79" type="trad"/>
<var cp="69A6" type="blocked"/>
<var cp="6F27" type="blocked"/>
</char>
<char cp="5E79">
<var cp="4E7E" type="blocked"/>
<var cp="4E81" type="blocked"/>
<var cp="5E72" type="simp"/>
<var cp="5E79" type="trad" comment="identity"/>
<var cp="69A6" type="blocked"/>
<var cp="6F27" type="blocked"/>
</char>
<char cp="69A6">
<var cp="4E7E" type="blocked"/>
<var cp="4E81" type="blocked"/>
<var cp="5E72" type="simp"/>
<var cp="5E79" type="blocked"/>
<var cp="69A6" type="trad" comment="identity"/>
<var cp="6F27" type="blocked"/>
</char>
<char cp="6F27">
<var cp="4E7E" type="simp"/>
<var cp="4E81" type="blocked"/>
<var cp="5E72" type="blocked"/>
<var cp="5E79" type="blocked"/>
<var cp="69A6" type="blocked"/>
<var cp="6F27" type="trad" comment="identity"/>
</char>
</data>Here, the simplified variants have been given a type of "simp" and the traditional variants one of "trad", and all other ones are given "blocked".
Because some variant mappings show in more than one column, while the XML format allows only a single type value, they have been given the type of "both" 🤓
Note that some variant mappings map to themselves (identity); that is, the mapping is reflexive (see Section 5.3.4). In creating the permutation of all variant labels, these mappings have no effect, other than adding a value to the variant type list for the variant label containing them.
In the example so far, all of the entries with type="both" are also mappings where source and target are identical. That is, they are reflexive mappings as defined in Section 5.3.4.
Given a label "U+4E7E U+4E81", the following labels would be ruled allocatable per [RFC3743], based on how that standard is commonly implemented in domain registries:
Original label: U+4E7E U+4E81
Simplified label 1: U+4E7E U+5E72
Simplified label 2: U+5E72 U+5E72
Traditional label: U+4E7E U+4E7EHowever, if allocatable labels were generated simply by a straight permutation of all variants with type other than type="blocked" and without regard to the simplified and traditional variants, we would end up with an extra allocatable label of "U+5E72 U+4E7E". This label is composed of both a Simplified Chinese character and a Traditional Chinese code point and therefore shouldn't be allocatable 🚫
To more fully resolve the dispositions requires several actions to be defined, as described in Section 7.2.2, that will override the default actions from Section 7.6. After blocking all labels that contain a variant with type "blocked", these actions will set to "allocatable" labels based on the following variant types: "simp", "trad", and "both". Note that these variant types do not directly relate to dispositions for the variant label, but that the actions will resolve them to the Standard Dispositions on labels, i.e., "blocked" and "allocatable".
To resolve label dispositions requires five actions to be defined (in the "rules" section of the XML document in question); these actions apply in order, and the first one triggered defines the disposition for the label. The actions are as follows:
- Block all variant labels containing at least one blocked variant.
- Allocate all labels that consist entirely of variants that are "simp" or "both".
- Also allocate all labels that are entirely "trad" or "both".
- Block all surviving labels containing any one of the dispositions "simp" or "trad" or "both", because they are now known to be part of an undesirable mixed simplified/traditional label.
- Allocate any remaining label; the original label would be such a label.
The rules declarations would be represented as:
<rules>
<!--"action" elements - order defines precedence-->
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" only-variants="simp both" />
<action disp="allocatable" only-variants="trad both" />
<action disp="blocked" any-variant="simp trad" />
<action disp="allocatable" comment="catch-all" />
</rules>Up to now, variants with type "both" have occurred only associated with reflexive variant mappings. The "action" elements defined above rely on the assumption that this is always the case. However, consider the following set of variants 🌀
U+62E0;U+636E;U+636E;U+64DA
U+636E;U+636E;U+64DA;U+62E0
U+64DA;U+636E;U+64DA;U+62E0The corresponding XML would be:
<char cp="62E0">
<var cp="636E" type="both" comment="both, but not reflexive" />
<var cp="64DA" type="blocked" />
</char>
<char cp="636E">
<var cp="636E" type="simp" comment="reflexive, but not both" />
<var cp="64DA" type="trad" />
<var cp="62E0" type="blocked" />
</char>
<char cp="64DA">
<var cp="636E" type="simp" />
<var cp="64DA" type="trad" comment="reflexive" />
<var cp="62E0" type="blocked" />
</char>To make such variant sets work requires a way to selectively trigger an action based on whether a variant type is associated with an identity or reflexive mapping, or is associated with an ordinary variant mapping. This can be done by adding a prefix "r-" to the "type" attribute on reflexive variant mappings. For example, the "trad" for code point U+64DA in the preceding figure would become "r-trad".
With the dispositions prepared in this way, only a slight modification to the actions is needed to yield the correct set of allocatable labels:
<action disp="blocked" any-variant="blocked" />
<action disp="allocatable" only-variants="simp r-simp both r-both" />
<action disp="allocatable" only-variants="trad r-trad both r-both" />
<action disp="blocked" all-variants="simp trad both" />
<action disp="allocatable" />The first three actions get triggered by the same labels as before.
The fourth action blocks any label that combines an original code point with any mix of ordinary variant mappings; however, no labels that are a combination of only original code points (code points having either no variant mappings or a reflexive mapping) would be affected. These are the original labels, and they are allocated in the last action.
Using this scheme of assigning types to ordinary and reflexive variants, all tables in the style of RFC 3743 can be converted to XML. By defining a set of actions as outlined above, the LGR will yield the correct set of allocatable variants: all variants consisting completely of variant code points preferred for simplified or traditional, respectively, will be allocated, as will be the original label. All other variant labels will be blocked 💖
Appendix C. Indic Syllable Structure Example ✨
okay so for LGRs covering Indic scripts 🪷, it can be super useful to only allow labels made of valid Indic syllables, aka aksharas. This appendix gives a sample set of rules that enforces exactly that 💅
Here's an example BNF for an akshara, taken from "Devanagari Script Behaviour for Hindi" [TDIL-HINDI]. The rules for other languages and scripts used in India are expected to be generally similar 🌸
For Hindi, the BNF looks like this:
V[m]|{C[N]H}C[N](H|[v][m])Here's what each symbol means:
V(uppercase) is any independent vowelmis any vowel modifier (Devanagari Anusvara, Visarga, and Candrabindu)Cis any consonant (with inherent vowel)Nis NuktaHis a halant (or virama)v(lowercase) is any dependent vowel sign (matra){}encloses items that may be repeated one or more times[ ]encloses items that may or may not be present|separates items, out of which only one can be present
Now the fun part 🎀 By using the Unicode character property "InSC" or "Indic_Syllabic_Category", which lines up pretty directly with the character classes in the BNF above, we can translate the BNF into a set of WLE rules that match the definition of an akshara 🦋
<rules>
<!--Character class definitions go here-->
<class name="halant" property="InSC:Virama" />
<union name="vowel-modifier">
<class property="InSC:Visarga" />
<class property="InSC:Bindu" comment="includes anusvara" />
</union>
<!--Whole label evaluation and context rules go here-->
<rule name="consonant-with-optional-nukta">
<class by-ref="InSC:Consonant" />
<class by-ref="InSC:Nukta" count="0:1"/>
</rule>
<rule name="independent-vowel-with-optional-modifier">
<class by-ref="InSC:Vowel_Independent" />
<class by-ref="vowel-modifier" count="0:1" />
</rule>
<rule name="optional-dependent-vowel-with-opt-modifier" >
<class by-ref="InSC:Vowel_Dependent" count="0:1" />
<class by-ref="vowel-modifier" count="0:1" />
</rule>
<rule name="consonant-cluster">
<rule count="0+">
<rule by-ref="consonant-with-optional-nukta" />
<class by-ref="halant" />
</rule>
<rule by-ref="consonant-with-optional-nukta" />
<choice>
<class by-ref="halant" />
<rule by-ref="optional-dependent-vowel-with-opt-modifier" />
</choice>
</rule>
<rule name="akshara">
<choice>
<rule by-ref="independent-vowel-with-optional-modifier" />
<rule by-ref="consonant-cluster" />
</choice>
</rule>
<rule name="WLE-akshara-or-other" comment="series of one or
more aksharas, possibly alternating with other types of
code points such as digits">
<start />
<choice count="1+">
<class property="InSC:other" />
<rule by-ref="akshara" />
</choice>
<end />
</rule>
<!--"action" elements go here - order defines precedence-->
<action disp="invalid" not-match="WLE-akshara-or-other" />
</rules>With those rules and classes in place, the final action slaps a disposition of "invalid" on every label that isn't made of a sequence of well-formed aksharas 🙅♀️ (optionally mixed up with other characters, like digits, for example) ✨
The relevant Unicode character property could also be replicated by tagging repertoire values directly in the LGR. That would remove the dependency on any specific version of the Unicode Standard 🧚
Generally, dependent vowels may only follow consonant expressions. BUT for some scripts, like Bengali, the Unicode Standard supports sequences of dependent vowels or their application on independent vowels 👀 which makes the definition of akshara less restrictive. no cap, that's a plot twist 😭
C.1. Reducing Complexity
okay real talk: as presented in this example, the rules are pretty complex 😵💫 Super useful for showing off the features of the XML format, but that level of complexity would be a total vibe-killer in an actual LGR 💀
It's possible to reduce the complexity by defining alternate rules that simply describe the permissible pair-wise context of adjacent code points by character class 🌈 For example, a rule that a halant can only follow a (nuktated) consonant. These pair-wise contexts are easier to understand, implement, and verify. They also have the bonus of letting tools pinpoint why a label failed to validate 🔍 They also tend to match up more directly with the kinds of well-formedness requirements that matter most for DNS security, like limiting the application of a combining mark (such as a vowel modifier) to only selected base characters (in this case, vowels) 💜 (See the example and discussion in [WLE-RULES].)
Appendix D. RELAX NG Compact Schema
This schema is provided in RELAX NG Compact format [RELAX-NG] 🎀
<CODE BEGINS>
#
# LGR XML Schema 1.0
#
default namespace = "urn:ietf:params:xml:ns:lgr-1.0"
#
# SIMPLE TYPES
#
# RFC 5646 language tag (e.g., "de", "und-Latn")
language-tag = xsd:token
# The scope to which the LGR applies. For the "domain" scope type,
# it should be a fully qualified domain name.
scope-value = xsd:token {
minLength = "1"
}
## a single code point
code-point = xsd:token {
pattern = "[0-9A-F]{4,6}"
}
## a space-separated sequence of code points
code-point-sequence = xsd:token {
pattern = "[0-9A-F]{4,6}( [0-9A-F]{4,6})+"
}
## single code point, or a sequence of code points, or empty string
code-point-literal = code-point | code-point-sequence | ""
## code point or sequence only
non-empty-code-point-literal = code-point | code-point-sequence
## code point sent represented in short form
code-point-set-shorthand = xsd:token {
pattern = "([0-9A-F]{4,6}|[0-9A-F]{4,6}-[0-9A-F]{4,6})"
~ "( ([0-9A-F]{4,6}|[0-9A-F]{4,6}-[0-9A-F]{4,6}))*"
}
## dates are used in information fields in the meta
## section ("YYYY-MM-DD")
date-pattern = xsd:token {
pattern = "\d{4}-\d\d-\d\d"
}
## variant type
## the variant type MUST be non-empty and MUST NOT
## start with a "_"; using xsd:NMTOKEN here because
## we need space-separated lists of them
variant-type = xsd:NMTOKEN
## variant type list for action triggers
## the list MUST NOT be empty, and entries MUST NOT
## start with a "_"
variant-type-list = xsd:NMTOKENS
## reference to a rule name (used in "when" and "not-when"
## attributes, as well as the "by-ref" attribute of the "rule"
## element).
rule-ref = xsd:IDREF
## a space-separated list of tags. Tags should generally follow
## xsd:Name syntax. However, we are using the xsd:NMTOKENS here
## because there is no native XSD datatype for space-separated
## xsd:Name
tags = xsd:NMTOKENS
## The value space of a "from-tag" attribute. Although it is closer
## to xsd:IDREF lexically and semantically, tags are not unique in
## the document. As such, we are unable to take advantage of
## facilities provided by a validator. xsd:NMTOKEN is used instead
## of the stricter xsd:Names here so as to be consistent with
## the above.
tag-ref = xsd:NMTOKEN
## an identifier type (used by "name" attributes).
identifier = xsd:ID
## used in the class "by-ref" attribute to reference another class of
## the same "name" attribute value.
class-ref = xsd:IDREF
## "count" attribute pattern ("n", "n+", or "n:m")
count-pattern = xsd:token {
pattern = "\d+(\+|:\d+)?"
}
## "ref" attribute pattern
## space-separated list of "id" attribute values for
## "reference" elements. These reference ids
## must be declared in a "reference" element
## before they can be used in a "ref" attribute
ref-pattern = xsd:token {
pattern = "[\-_.:0-9A-Z]+( [\-_.:0-9A-Z]+)*"
}
#
# STRUCTURES
#
## Representation of a single code point or a sequence of code
## points
char = element char {
attribute cp { code-point-literal },
attribute comment { text }?,
attribute when { rule-ref }?,
attribute not-when { rule-ref }?,
attribute tag { tags }?,
attribute ref { ref-pattern }?,
variant*
}
## Representation of a range of code points
range = element range {
attribute first-cp { code-point },
attribute last-cp { code-point },
attribute comment { text }?,
attribute when { rule-ref }?,
attribute not-when { rule-ref }?,
attribute tag { tags }?,
attribute ref { ref-pattern }?
}
## Representation of a variant code point or sequence
variant = element var {
attribute cp { code-point-literal },
attribute type { xsd:NMTOKEN }?,
attribute when { rule-ref }?,
attribute not-when { rule-ref }?,
attribute comment { text }?,
attribute ref { ref-pattern }?
}
#
# Classes
#
## a "class" element that references the name of another "class"
## (or set-operator like "union") defined elsewhere.
## If used as a matcher (appearing under a "rule" element),
## the "count" attribute may be present.
class-invocation = element class { class-invocation-content }
class-invocation-content =
attribute by-ref { class-ref },
attribute count { count-pattern }?,
attribute comment { text }?
## defines a new class (set of code points) using Unicode property
## or code points of the same tag value or code point literals
class-declaration = element class { class-declaration-content }
class-declaration-content =
# "name" attribute MUST be present if this is a "top-level"
# class declaration, i.e., appearing directly under the "rules"
# element. Otherwise, it MUST be absent.
attribute name { identifier }?,
# If used as a matcher (appearing in a "rule" element, but not
# when nested inside a set-operator or class), the "count"
# attribute may be present. Otherwise, it MUST be absent.
attribute count { count-pattern }?,
attribute comment { text }?,
attribute ref { ref-pattern }?,
(
# define the class by property (e.g., property="sc:Latn"), OR
attribute property { xsd:NMTOKEN }
# define the class by tagged code points, OR
| attribute from-tag { tag-ref }
# text node to allow for shorthand notation
# e.g., "0061 0062-0063"
| code-point-set-shorthand
)
class-invocation-or-declaration = element class {
class-invocation-content | class-declaration-content
}
class-or-set-operator-nested =
class-invocation-or-declaration | set-operator
class-or-set-operator-declaration =
# a "class" element or set-operator (effectively defining a class)
# directly in the "rules" element.
class-declaration | set-operator
#
# set-operators
#
complement-operator = element complement {
attribute name { identifier }?,
attribute comment { text }?,
attribute ref { ref-pattern }?,
# "count" attribute MUST only be used when this set-operator is
# used as a matcher (i.e., nested in a "rule" element but not
# inside a set-operator or class)
attribute count { count-pattern }?,
class-or-set-operator-nested
} union-operator = element union {
attribute name { identifier }?,
attribute comment { text }?,
attribute ref { ref-pattern }?,
# "count" attribute MUST only be used when this set-operator is
# used as a matcher (i.e., nested in a "rule" element but not
# inside a set-operator or class)
attribute count { count-pattern }?,
class-or-set-operator-nested,
# needs two or more child elements
class-or-set-operator-nested+
}
intersection-operator = element intersection {
attribute name { identifier }?,
attribute comment { text }?,
attribute ref { ref-pattern }?,
# "count" attribute MUST only be used when this set-operator is
# used as a matcher (i.e., nested in a "rule" element but not
# inside a set-operator or class)
attribute count { count-pattern }?,
class-or-set-operator-nested,
class-or-set-operator-nested
}
difference-operator = element difference {
attribute name { identifier }?,
attribute comment { text }?,
attribute ref { ref-pattern }?,
# "count" attribute MUST only be used when this set-operator is
# used as a matcher (i.e., nested in a "rule" element but not
# inside a set-operator or class)
attribute count { count-pattern }?,
class-or-set-operator-nested,
class-or-set-operator-nested
}
symmetric-difference-operator = element symmetric-difference {
attribute name { identifier }?,
attribute comment { text }?,
attribute ref { ref-pattern }?,
# "count" attribute MUST only be used when this set-operator is
# used as a matcher (i.e., nested in a "rule" element but not
# inside a set-operator or class)
attribute count { count-pattern }?,
class-or-set-operator-nested,
class-or-set-operator-nested
}
## operators that transform class(es) into a new class.
set-operator = complement-operator
| union-operator
| intersection-operator
| difference-operator
| symmetric-difference-operator
#
# Match operators (matchers)
#
any-matcher = element any {
attribute count { count-pattern }?,
attribute comment { text }?
}
choice-matcher = element choice {
## "count" attribute MUST only be used when the choice-matcher
## contains no nested "start", "end", "anchor", "look-behind",
## or "look-ahead" operators and no nested rule-matchers
## containing any of these elements
attribute count { count-pattern }?,
attribute comment { text }?,
# two or more match operators
match-operator-choice,
match-operator-choice+
}
char-matcher =
# for use as a matcher - like "char" but without a "tag" attribute
element char {
attribute cp { non-empty-code-point-literal },
# If used as a matcher (appearing in a "rule" element), the
# "count" attribute may be present. Otherwise, it MUST be
# absent.
attribute count { count-pattern }?,
attribute comment { text }?,
attribute ref { ref-pattern }?
}
start-matcher = element start {
attribute comment { text }?
}
end-matcher = element end {
attribute comment { text }?
}
anchor-matcher = element anchor {
attribute comment { text }?
}
look-ahead-matcher = element look-ahead {
attribute comment { text }?,
match-operators-non-pos
}
look-behind-matcher = element look-behind {
attribute comment { text }?,
match-operators-non-pos
}
## non-positional match operator that can be used as a direct child
## element of the choice-matcher.
match-operator-choice = (
any-matcher | choice-matcher | start-matcher | end-matcher
| char-matcher | class-or-set-operator-nested | rule-matcher
)
## non-positional match operators do not contain any "anchor",
## "look-behind", or "look-ahead" elements.
match-operators-non-pos = (
start-matcher?,
(any-matcher | choice-matcher | char-matcher
| class-or-set-operator-nested | rule-matcher)*,
end-matcher?
)
## positional match operators have an "anchor" element, which may be
## preceded by a "look-behind" element, or followed by a "look-ahead"
## element, or both.
match-operators-pos =
look-behind-matcher?, anchor-matcher, look-ahead-matcher?
match-operators = match-operators-non-pos | match-operators-pos
#
# Rules
#
# top-level rule must have "name" attribute
rule-declaration-top = element rule {
attribute name { identifier },
attribute comment { text }?,
attribute ref { ref-pattern }?,
match-operators
}
## "rule" element used as a matcher (either "by-ref" or contains
## other match operators itself)
rule-matcher =
element rule {
## "count" attribute MUST only be used when the rule-matcher
## contains no nested "start", "end", "anchor", "look-behind",
## or "look-ahead" operators and no nested rule-matchers
## containing any of these elements
attribute count { count-pattern }?,
attribute comment { text }?,
attribute ref { ref-pattern }?,
(attribute by-ref { rule-ref } | match-operators)
}
#
# Actions
#
action-declaration = element action {
attribute comment { text }?,
attribute ref { ref-pattern }?,
# dispositions are often named after variant types or vice versa
attribute disp { variant-type },
( attribute match { rule-ref }
| attribute not-match { rule-ref } )?,
( attribute any-variant { variant-type-list }
| attribute all-variants { variant-type-list }
| attribute only-variants { variant-type-list } )?
}
# DOCUMENT STRUCTURE
start = lgr
lgr = element lgr {
meta-section?,
data-section,
rules-section?
}
## Meta section - information recorded with an LGR that generally
## does not affect machine processing (except for "unicode-version").
## However, if any "class-declaration" uses the "property" attribute,
## a "unicode-version" element MUST be present.
meta-section = element meta {
element version {
attribute comment { text }?,
text
}?
& element date { date-pattern }?
& element language { language-tag }*
& element scope {
# type may by "domain" or an application-defined value
attribute type { xsd:NCName },
scope-value
}*
& element validity-start { date-pattern }?
& element validity-end { date-pattern }?
& element unicode-version {
xsd:token {
pattern = "\d+\.\d+\.\d+"
}
}?
& element description {
# this SHOULD be a valid MIME type
attribute type { text }?,
text
}?
& element references {
element reference {
attribute id {
xsd:token {
# limit "id" attribute to uppercase letters,
# digits, and a few punctuation marks; use of
# integers is RECOMMENDED
pattern = "[\-_.:0-9A-Z]*"
minLength = "1"
}
},
attribute comment { text }?,
text
}*
}?
}
data-section = element data { (char | range)+ }
## Note that action declarations are strictly order dependent.
## class-or-set-operator-declaration and rule-declaration-top
## are weakly order dependent; they must precede first use of the
## identifier via "by-ref".
rules-section = element rules {
( class-or-set-operator-declaration
| rule-declaration-top
| action-declaration)*
}
<CODE ENDS>Acknowledgements ✨
Okay bestie, huge shoutout first 🫶 This format totally builds on the work that a bunch of different registry operators did documenting IDN tables. 💅 The big one is a super thorough language table for Chinese, Japanese, and Korean made by the "Joint Engineering Team" [RFC3743], and honestly that table is the basis for a ton of registry policies. 🌸 There's also a set of guidelines for Arabic script registrations [RFC5564] that the Arabic-language community put together. 🕌
Then there's the whole squad of people whose contributions shaped this document 💖 Francisco Arias, Julien Bernard, Mark Davis, Martin Duerst, Paul Hoffman, Sarmad Hussain, Barry Leiba, Alexander Mayrhofer, Alexey Melnikov, Nicholas Ostler, Thomas Roessler, Audric Schiltknecht, Steve Sheng, Michel Suignard, Andrew Sullivan, Wil Tan, and John Yunker. 👑 Iconic cast, no notes. 🎀
Authors' Addresses 🦋
Here are the authors' contact details, copied exactly so you can reach them 💌
Kim Davies
Internet Corporation for Assigned Names and Numbers
12025 Waterfront Drive
Los Angeles, CA 90094
United States of America
Phone: +1 310 301 5800
Email: kim.davies@icann.org
URI: http://www.icann.org/
Asmus Freytag
ASMUS, Inc.
Email: asmus@unicode.org📦 where this page comes from
- ietf: rfc7940 2016-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms