RFC 3492
Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)
Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA) 🦋
Okay bestie, this is RFC 3492 💖 It's a Network Working Group document (Request for Comments: 3492), written by A. Costello from Univ. of California, Berkeley, and it's on the Standards Track 👑. It's dated March 2003 🎀. The header doesn't list anything it obsoletes or updates, so there's nothing to add there ✨.
Status of this Memo 📜
This document is an Internet standards track protocol for the Internet community, and it's asking for discussion and suggestions for improvements 🌸. If you want the current standardization state and status of this protocol, check the latest edition of the "Internet Official Protocol Standards" (STD 1) 🧚. Distribution of this memo is unlimited 🌈.
Copyright Notice ©️
Copyright (C) The Internet Society (2003). All Rights Reserved. 💜
Abstract 🫶
Okay, so Punycode is a simple and efficient transfer encoding syntax made for Internationalized Domain Names in Applications (IDNA) 💅. It uniquely and reversibly turns a Unicode string into an ASCII string, no cap 🔥. ASCII characters stay literal in the Unicode string, and non-ASCII characters get represented by ASCII characters that are allowed in host name labels (letters, digits, and hyphens) ✨.
This document defines a general algorithm called Bootstring 🦋. It lets a string of basic code points uniquely represent any string of code points from a bigger set, which is iconic 🙈. Punycode is an instance of Bootstring that uses the particular parameter values this document specifies, and those values are tailored for IDNA 💖.
Table of Contents 🎀
1. Introduction...............................................2
1.1 Features..............................................2
1.2 Interaction of protocol parts.........................3
2. Terminology................................................3
3. Bootstring description.....................................4
3.1 Basic code point segregation..........................4
3.2 Insertion unsort coding...............................4
3.3 Generalized variable-length integers..................5
3.4 Bias adaptation.......................................7
4. Bootstring parameters......................................8
5. Parameter values for Punycode..............................8
6. Bootstring algorithms......................................9
6.1 Bias adaptation function.............................10
6.2 Decoding procedure...................................11
6.3 Encoding procedure...................................12
6.4 Overflow handling....................................13
7. Punycode examples.........................................14
7.1 Sample strings.......................................14
7.2 Decoding traces......................................17
7.3 Encoding traces......................................19
8. Security Considerations...................................20
9. References................................................21
9.1 Normative References.................................21
9.2 Informative References...............................21
A. Mixed-case annotation.....................................22
B. Disclaimer and license....................................22
C. Punycode sample implementation............................23
Author's Address.............................................34
Full Copyright Statement.....................................351. Introduction 🧚
Okay so [IDNA] describes the architecture for supporting internationalized domain names 🌍. Labels that have non-ASCII characters can be represented by ACE labels 💖. Those start with a special ACE prefix and only contain ASCII characters ✨. The rest of the label after that prefix is a Punycode encoding of a Unicode string that meets certain constraints 🎀. For the details on the prefix and constraints, check [IDNA] and [NAMEPREP] 💅.
Punycode is an instance of a more general algorithm called Bootstring 🦋. Bootstring lets strings made from a small set of "basic" code points uniquely represent any string of code points from a larger set 🌸. So Punycode is basically Bootstring with particular parameter values that fit IDNA perfectly 👑.
1.1 Features 💅
Bootstring was designed with these features, and they're giving main character energy 🎀:
- Completeness: Every extended string (a sequence of arbitrary code points) can be represented by a basic string (a sequence of basic code points). Restrictions on which strings are allowed, and on length, can be imposed by higher layers 🌈.
- Uniqueness: There is at most one basic string that represents a given extended string 💜.
- Reversibility: Any extended string mapped to a basic string can be recovered from that basic string 🔥.
- Efficient encoding: The ratio of basic string length to extended string length is small. This matters a lot for domain names, because RFC 1034 [RFC1034] limits a domain label to 63 characters 🙈.
- Simplicity: The encoding and decoding algorithms are reasonably simple to implement. Efficiency and simplicity are kind of at odds, so Bootstring aims for a good balance between them ✨.
- Readability: Basic code points that show up in the extended string are represented as themselves in the basic string. (The main goal is efficiency, not readability, so this one's a bonus 🍓.)
Punycode can also do something extra that [IDNA]'s ToASCII and ToUnicode operations don't use 🧚. When extended strings get case-folded before encoding, the basic string can use mixed case to show how to turn the folded string back into a mixed-case string 🦋. See appendix A "Mixed-case annotation" for the deets 💖.
1.2 Interaction of protocol parts 🫶
Punycode is used by the IDNA protocol [IDNA] to convert domain labels into ASCII 💅. It's not designed for any other purpose, and it's explicitly not designed for processing arbitrary free text 🙈. Not me trying to use it for everything 💀.
2. Terminology 📚
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [RFC2119] 🎀.
A code point is an integral value associated with a character in a coded character set 🔢.
As in the Unicode Standard [UNICODE], Unicode code points are written as U+ followed by four to six hexadecimal digits 🌸. A range of code points is written as two hexadecimal numbers separated by .., with no prefixes 💜.
The operators div and mod do integer division 🧮. (x div y) is the quotient of x divided by y, with the remainder thrown away, and (x mod y) is that remainder. So (x div y) * y + (x mod y) == x always works out ✨. Bootstring only uses these operators with nonnegative operands, so the quotient and remainder are always nonnegative 🫶.
The break statement jumps out of the innermost loop, same as in C 🔥.
An overflow is an attempt to compute a value that's bigger than the maximum value of an integer variable 💀.
3. Bootstring description 🦋
Bootstring turns an arbitrary sequence of code points (the "extended string") into a sequence of basic code points (the "basic string") 💖. This section describes that representation 🎀. Section 6 "Bootstring algorithms" gives the algorithms as pseudocode 🧚. Sections 7.1 "Decoding traces" and 7.2 "Encoding traces" trace the algorithms for sample inputs 📜.
There are four techniques in Bootstring, and I'm obsessed with how they fit together 😭. First, "Basic code point segregation" is a super simple, efficient encoding for the basic code points in the extended string. They just get copied all at once 🌈. Next, "Insertion unsort coding" encodes the non-basic code points as deltas. It processes the code points in numerical order instead of order of appearance, which usually gives smaller deltas 💅. The deltas are written as "generalized variable-length integers", which use basic code points to represent nonnegative integers 🔢. Last, "bias adaptation" tunes the parameters of that integer representation as it goes, so consecutive deltas with similar magnitudes encode efficiently 👑.
3.1 Basic code point segregation 🧺
All the basic code points in the extended string are written literally at the start of the basic string, in their original order 🍓. If there's at least one basic code point, a delimiter follows them, and if there aren't any, there's no delimiter at all 💜. The delimiter is one particular basic code point that never shows up in the rest of the basic string 🔥. So the decoder can find the end of the literal part, if there is one, by scanning for the last delimiter 🦋.
3.2 Insertion unsort coding 🌸
The rest of the basic string (everything after the last delimiter, if there is one) is a sequence of nonnegative integral deltas written as generalized variable-length integers, which are described in section 3.3 ✨. The meaning of the deltas makes the most sense if you think about it from the decoder's side 🧚.
The decoder builds the extended string bit by bit 🎀. At first, the extended string is just a copy of the literal part of the basic string (not counting the last delimiter) 💖. Then the decoder inserts one non-basic code point for each delta, and that finally gives the decoded string 👑.
The heart of all this is a state machine with two state variables: an index i and a counter n 🫶. The index i points to a position in the extended string. It goes from 0 (the first position) up to the current length of the extended string, which can point just past the current end 🌈. If the current state is <n,i>, the next state is <n,i+1> when i is less than the length of the extended string. When i equals that length, the next state is <n+1,0> 💅. So each state change bumps i, wrapping around to zero when it needs to, and n counts how many wrap-arounds have happened 🔢.
The state always moves forward, so the decoder can never go back to an earlier state 🦋. In each state, an insertion either happens or doesn't, and at most one insertion happens per state 🙈. An insertion puts the value of n at position i in the extended string 💜. The deltas are a run-length encoding of these events. Each delta is the length of a run of non-insertion states before an insertion state 🍓. So for each delta, the decoder does delta state changes, then an insertion, then one more state change 🌈. (An implementation doesn't have to do each state change one at a time. It can use division and remainder calculations to jump straight to the next insertion state 🧮.) If the inserted code point turns out to be a basic code point, that's an error, since basic code points were supposed to be segregated as described in section 3.1 💀.
The encoder's main job is to work out the sequence of deltas that makes the decoder build the string the encoder wants 🧚. It does this by scanning the extended string again and again for the next code point the decoder would need to insert, and counting the state changes the decoder would have to do to get there ✨. It has to keep in mind that the decoder's extended string only has the code points that were already inserted 🫶. Section 6.3 "Encoding procedure" has the precise algorithm 📜.
to be continued... 🦋 (part 1 of 6, more parts coming, bestie!)
3.3. Generalized variable-length integers ✨
okay bestie, so in a conventional integer the base is just how many distinct digit symbols you get, with values 0 through base-1 🔢 Let's call digit_0 the least significant digit, then digit_1 the next one up, and so on 💅. The value is the sum over j of digit_j * w(j), where w(j) = base^j is the weight (the scale factor) for position j. Like, in base 8 the number 437 has digits 7, 3, and 4, the weights are 1, 8, and 64, so the value is 7 + 3*8 + 4*64 = 287 🎀. But this setup has two big problems, and I'm not here for them 😭:
- First, every value has multiple encodings, because you can just keep tossing extra zeros into the most significant spots. That's a mess when you need a unique encoding 💀
- Second, the integer isn't self-delimiting, so if you smoosh several integers together, the boundaries between them are gone forever 🙈
The generalized variable-length representation fixes both of these, periodt 👑 The digit values are still 0 through base-1, but now the integer is self-delimiting thanks to thresholds t(j), each in the range 0 through base-1 🦋 Exactly one digit, the most significant one, satisfies digit_j < t(j). So if a bunch of integers are concatenated, you can totally separate them 🫶 You start with the first one if they're little-endian (least significant digit first), or with the last one if they're big-endian (most significant digit first). The value is still the sum over j of digit_j * w(j), but the weights are different now 🌈:
w(0) = 1 w(j) = w(j-1) * (base - t(j-1)) for j > 0
Let's do an example, bestie 🧚 Take the little-endian sequence of base 8 digits 734251... with thresholds 2, 3, 5, 5, 5, 5... The weights become 1, 1*(8-2) = 6, 6*(8-3) = 30, 30*(8-5) = 90, 90*(8-5) = 270, and so on 💖 Now, 7 is not less than 2, and 3 is not less than 3, but 4 is less than 5, so 4 is the last digit ✨ The value of 734 is 7*1 + 3*6 + 4*30 = 145 🍓 The next integer is 251, with value 2*1 + 5*6 + 1*30 = 62 💜
Decoding is lowkey very similar to decoding a regular integer 🌸 Start with a current value of N = 0 and a weight w = 1. Fetch the next digit d and increase N by d * w. If d is less than the current threshold (t), then stop. Otherwise, increase w by a factor of (base - t), update t for the next position, and repeat 🔥
Encoding is also similar to a conventional integer, iykyk 🙈 If N < t, then output one digit for N and stop. Otherwise, output the digit for t + ((N - t) mod (base - t)), then replace N with (N - t) div (base - t), update t for the next position, and repeat 💅
For any particular set of values of t(j), there is exactly one generalized variable-length representation of each nonnegative integral value 👑 Like, one and only one, no cap 💀
Bootstring uses little-endian ordering so that the deltas can be separated starting with the first one 🦋 The t(j) values are defined in terms of the constants base, tmin, and tmax, plus a state variable called bias:
t(j) = base * (j + 1) - bias, clamped to the range tmin through tmax
The clamping means that if the formula gives a value less than tmin or greater than tmax, then t(j) = tmin or tmax, respectively 🎀 (In the pseudocode in section 6 "Bootstring algorithms", the expression base * (j + 1) is denoted by k for performance reasons ⭐) These t(j) values make the representation favor integers within a particular range, which is set by the bias 🌈
3.4. Bias adaptation 💖
After each delta is encoded or decoded, bias gets set for the next delta like this 🧚
- Delta is scaled so it doesn't overflow in the next step:
let delta = delta div 2
But when this is the very first delta, the divisor isn't 2. Instead it's a constant called damp 🫶 This makes up for the fact that the second delta is usually way smaller than the first one 🥹
- Delta gets increased to make up for the fact that the next delta will be inserting into a longer string 🌸
let delta = delta + (delta div numpoints)
numpoints is the total number of code points encoded or decoded so far. That includes the one for this delta itself, and the basic code points too ✨
- Delta gets divided over and over until it falls within a threshold, so we can predict the minimum number of digits needed for the next delta 🔥
while delta > ((base - tmin) * tmax) div 2 do let delta = delta div (base - tmin)
- The bias is set 👑
let bias = (base * the number of divisions performed in step 3) + (((base - tmin + 1) * delta) div (delta + skew))
The whole point of this procedure is that the current delta gives a hint about how big the next delta is likely to be 💅 So t(j) is set to tmax for the more significant digits, starting with the one expected to be last. It's tmin for the less significant digits, up through the one expected to be third-last. And for the digit expected to be second-last, it's somewhere between tmin and tmax, balancing the hope that the expected-last digit isn't needed against the danger that it's not enough 😭🦋
4. Bootstring parameters 🎀
Given a set of basic code points, one of them needs to be designated as the delimiter 🍓 The base can't be greater than the number of distinguishable basic code points left over. The digit-values in the range 0 through base-1 need to be associated with distinct non-delimiter basic code points 💜 Sometimes multiple code points need to share the same digit-value. For example, uppercase and lowercase versions of the same letter need to be equivalent if basic strings are case-insensitive 🌈
The initial value of n can't be greater than the minimum non-basic code point that could show up in extended strings ✨
The remaining five parameters (tmin, tmax, skew, damp, and the initial value of bias) need to satisfy these constraints 🫶:
0 <= tmin <= tmax <= base-1 skew >= 1 damp >= 2 initial_bias mod base <= base - tmin
Once the constraints are satisfied, these five parameters only affect efficiency, not correctness 🧚 They're best chosen empirically 💅
If you want mixed-case annotation support (see appendix A), make sure the code points for 0 through tmax-1 all have both uppercase and lowercase forms 👑
5. Parameter values for Punycode 🦋
Punycode uses these Bootstring parameter values 🌸:
base = 36 tmin = 1 tmax = 26 skew = 38 damp = 700 initial_bias = 72 initial_n = 128 = 0x80
Punycode only restricts the input integers to be nonnegative, but these parameters are specially designed to work well with Unicode [UNICODE] code points 🌈 Those are integers in the range 0..10FFFF, except D800..DFFF, which are reserved for the UTF-16 encoding of Unicode 💖 The basic code points are the ASCII [ASCII] code points (0..7F). Of those, U+002D (-) is the delimiter, and some of the others have digit-values like this ✨:
code points digit-values ------------ ---------------------- 41..5A (A-Z) = 0 to 25, respectively 61..7A (a-z) = 0 to 25, respectively 30..39 (0-9) = 26 to 35, respectively
Using hyphen-minus as the delimiter means the encoded string can end with a hyphen-minus only if the Unicode string is made up entirely of basic code points. But IDNA forbids those strings from being encoded 🙈 The encoded string can begin with a hyphen-minus, but IDNA prepends a prefix. So IDNA using Punycode conforms to the RFC 952 rule that host name labels neither begin nor end with a hyphen-minus [RFC952] 🫶
A decoder MUST recognize the letters in both uppercase and lowercase forms, including mixtures of both 💅 An encoder SHOULD output only uppercase forms or only lowercase forms, unless it uses mixed-case annotation (see appendix A) 🎀
Most users probably won't write or type encoded strings by hand (as opposed to cutting and pasting them), but the ones who do need to watch out for these visually ambiguous characters 👀:
G 6 I l 1 O 0 S 5 U V Z 2
Those ambiguities are usually sorted out by context, but in a Punycode encoded string there's no context that humans can see 😭💀
6. Bootstring algorithms 🧚
Some parts of the pseudocode can be left out if the parameters satisfy certain conditions (which Punycode qualifies for) 💖 These parts are wrapped in {braces}, and notes right after the pseudocode explain when they can be omitted 🌸
Formally, code points are integers, so the pseudocode assumes arithmetic can be done directly on code points. In some programming languages, you might need an explicit conversion between code points and integers 🔥
6.1. Bias adaptation function 👑
function adapt(delta,numpoints,firsttime): if firsttime then let delta = delta div damp else let delta = delta div 2 let delta = delta + (delta div numpoints) let k = 0 while delta > ((base - tmin) * tmax) div 2 do begin let delta = delta div (base - tmin) let k = k + base end return k + (((base - tmin + 1) * delta) div (delta + skew))
It doesn't matter whether the changes to delta and k inside adapt() affect variables with the same names inside the encoding or decoding procedures, because after calling adapt() the caller doesn't read those variables before overwriting them 🦋✨
6.2 Decoding procedure ✨
okay bestie, the decoder is where the magic happens 🔮 it reads the string one piece at a time and rebuilds the original code points. here's the full recipe:
let n = initial_n
let i = 0
let bias = initial_bias
let output = an empty string indexed from 0
consume all code points before the last delimiter (if there is one)
and copy them to output, fail on any non-basic code point
if more than zero code points were consumed then consume one more
(which will be the last delimiter)
while the input is not exhausted do begin
let oldi = i
let w = 1
for k = base to infinity in steps of base do begin
consume a code point, or fail if there was none to consume
let digit = the code point's digit-value, fail if it has none
let i = i + digit * w, fail on overflow
let t = tmin if k <= bias {+ tmin}, or
tmax if k >= bias + tmax, or k - bias otherwise
if digit < t then break
let w = w * (base - t), fail on overflow
end
let bias = adapt(i - oldi, length(output) + 1, test oldi is 0?)
let n = n + i div (length(output) + 1), fail on overflow
let i = i mod (length(output) + 1)
{if n is a basic code point then fail}
insert n into output at position i
increment i
endthe whole brace-wrapped check (seeing if n is a basic code point) can be skipped if initial_n is bigger than every basic code point, which is true for Punycode, because n never drops below initial_n 💅
about the t assignment: the + tmin bit can always be left out 🙈 that makes the clamping go wrong when bias < k < bias + tmin, but that can't actually happen, thanks to how bias gets computed and the parameter constraints.
here's the cute part 🦋 the decoder state only ever moves forward, and each delta has exactly one representation. so only one encoded string can stand for any given sequence of integers. the only errors are invalid code points, unexpected end-of-input, overflow, and basic code points encoded as deltas instead of appearing literally. if the decoder fails on those errors like shown above, it can never produce the same output from two different inputs. without that property, you'd have to re-encode the output and check it matches the input just to make sure the encoding is unique, and nobody wants that extra work 😭
6.3 Encoding procedure 🌸
now the encoder, which goes the other way and turns your code points into the Punycode string:
let n = initial_n
let delta = 0
let bias = initial_bias
let h = b = the number of basic code points in the input
copy them to the output in order, followed by a delimiter if b > 0
{if the input contains a non-basic code point < n then fail}
while h < length(input) do begin
let m = the minimum {non-basic} code point >= n in the input
let delta = delta + (m - n) * (h + 1), fail on overflow
let n = m
for each code point c in the input (in order) do begin
if c < n {or c is basic} then increment delta, fail on overflow
if c == n then begin
let q = delta
for k = base to infinity in steps of base do begin
let t = tmin if k <= bias {+ tmin}, or
tmax if k >= bias + tmax, or k - bias otherwise
if q < t then break
output the code point for digit t + ((q - t) mod (base - t))
let q = (q - t) div (base - t)
end
output the code point for digit q
let bias = adapt(delta, h + 1, test h equals b?)
let delta = 0
increment h
end
end
increment delta and n
endthe brace-enclosed check (is there a non-basic code point below n in the input?) can be dropped if every code point below initial_n is basic, which is true for Punycode if code points are unsigned 🧚
the brace-enclosed bits "non-basic" and "or c is basic" can also go if initial_n is bigger than every basic code point (again, true for Punycode), since the code point being tested is never below initial_n ✨
same deal with t as before: the + tmin can always be omitted. it breaks the clamp when bias < k < bias + tmin, but that's impossible because of how bias works and the parameter constraints 💖
the overflow checks are a must-have, though 💀 they stop you from making invalid output when the input has really huge values or is super long.
the delta increment at the bottom of the outer loop can't overflow, since delta < length(input) before the increment, and length(input) is already assumed to be representable. the increment of n could overflow, but only when h == length(input), and at that point the procedure is done anyway 🫶
6.4 Overflow handling 🎀
okay so overflow is a whole thing, and the good news is that IDNA is pretty chill about it 😌 26-bit unsigned integers are enough to handle every valid IDNA label without overflowing, because any string that needed a 27-bit delta would have to go past either the code point limit (0..10FFFF) or the label length limit (63 characters). BUT overflow handling still matters, since the inputs aren't necessarily valid IDNA labels 👑
if your programming language doesn't do overflow detection, you can use this trick 🔥 say A, B, and C are representable nonnegative integers and C is nonzero. then A + B overflows if and only if B > maxint - A, and A + (B * C) overflows if and only if B > (maxint - A) div C. here, maxint is the biggest integer where maxint + 1 can't be represented. check appendix C "Punycode sample implementation" for demos of this in C, it's giving clean code 💜
the decoding and encoding algorithms in sections 6.2 and 6.3 handle overflow by catching it the moment it happens. another option is to put limits on the inputs so overflow never shows up in the first place. for example, if the encoder checks that no input code point goes above M and the input length doesn't go above L, then no delta could ever exceed (M - initial_n) * (L + 1), so no overflow could happen as long as your integer variables can hold values that big. this prevention approach puts more restrictions on the input than the detection approach does, but some programming languages might make it feel simpler 🌈
in theory, the decoder could do something similar by limiting how many digits a variable-length integer can have (aka limiting the iterations in the innermost loop). BUT the number of digits that's enough to represent a delta can sometimes stand for much bigger deltas, because of the adaptation, so this would probably need integers wider than 32 bits 🙈
one more approach for the decoder: let overflow happen, then re-encode the final output string and compare it to the decoder input. if and only if they don't match (using a case-insensitive ASCII comparison), overflow has occurred. this delayed-detection approach doesn't put any more restrictions on the input than the immediate-detection approach, and some languages might make it simpler too 🦋
and finally, if the decoder is only used inside the IDNA ToUnicode operation [IDNA], it doesn't need to check for overflow at all 🥹 ToUnicode does a higher-level re-encoding and comparison already, and a mismatch has the same result as if the Punycode decoder had failed. so it's all covered, periodt 💅
7. Punycode examples ✨
7.1. Sample strings ✨
okay so this is where we get to see Punycode in action 🦋 the ACE prefix is not shown in these examples, bestie, so don't go hunting for xn-- 🙈 and the backslashes are just where line breaks got added because some strings were too long for one line 💅
The first several examples are all translations of the sentence "Why can't they just speak in <language>?" (courtesy of Michael Kaplan's "provincial" page [PROVINCIAL] 💜). Word breaks and punctuation have been removed, as is often done in domain names 🍓
(A) Arabic (Egyptian):
u+0644 u+064A u+0647 u+0645 u+0627 u+0628 u+062A u+0643 u+0644
u+0645 u+0648 u+0634 u+0639 u+0631 u+0628 u+064A u+061F
Punycode: egbpdaj6bu4bxfgehfvwxn
(B) Chinese (simplified):
u+4ED6 u+4EEC u+4E3A u+4EC0 u+4E48 u+4E0D u+8BF4 u+4E2D u+6587
Punycode: ihqwcrb4cv8a8dqg056pqjye
(C) Chinese (traditional):
u+4ED6 u+5011 u+7232 u+4EC0 u+9EBD u+4E0D u+8AAA u+4E2D u+6587
Punycode: ihqwctvzc91f659drss3x8bo0yb
(D) Czech: Pro<ccaron>prost<ecaron>nemluv<iacute><ccaron>esky
U+0050 u+0072 u+006F u+010D u+0070 u+0072 u+006F u+0073 u+0074
u+011B u+006E u+0065 u+006D u+006C u+0075 u+0076 u+00ED u+010D
u+0065 u+0073 u+006B u+0079
Punycode: Proprostnemluvesky-uyb24dma41a
(E) Hebrew:
u+05DC u+05DE u+05D4 u+05D4 u+05DD u+05E4 u+05E9 u+05D5 u+05D8
u+05DC u+05D0 u+05DE u+05D3 u+05D1 u+05E8 u+05D9 u+05DD u+05E2
u+05D1 u+05E8 u+05D9 u+05EA
Punycode: 4dbcagdahymbxekheh6e0a7fei0b
(F) Hindi (Devanagari):
u+092F u+0939 u+0932 u+094B u+0917 u+0939 u+093F u+0928 u+094D
u+0926 u+0940 u+0915 u+094D u+092F u+094B u+0902 u+0928 u+0939
u+0940 u+0902 u+092C u+094B u+0932 u+0938 u+0915 u+0924 u+0947
u+0939 u+0948 u+0902
Punycode: i1baa7eci9glrd9b2ae1bj0hfcgg6iyaf8o0a1dig0cd
(G) Japanese (kanji and hiragana):
u+306A u+305C u+307F u+3093 u+306A u+65E5 u+672C u+8A9E u+3092
u+8A71 u+3057 u+3066 u+304F u+308C u+306A u+3044 u+306E u+304B
Punycode: n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa
(H) Korean (Hangul syllables):
u+C138 u+ACC4 u+C758 u+BAA8 u+B4E0 u+C0AC u+B78C u+B4E4 u+C774
u+D55C u+AD6D u+C5B4 u+B97C u+C774 u+D574 u+D55C u+B2E4 u+BA74
u+C5BC u+B9C8 u+B098 u+C88B u+C744 u+AE4C
Punycode: 989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5j\
psd879ccm6fea98c
(I) Russian (Cyrillic):
U+043F u+043E u+0447 u+0435 u+043C u+0443 u+0436 u+0435 u+043E
u+043D u+0438 u+043D u+0435 u+0433 u+043E u+0432 u+043E u+0440
u+044F u+0442 u+043F u+043E u+0440 u+0443 u+0441 u+0441 u+043A
u+0438
Punycode: b1abfaaepdrnnbgefbaDotcwatmq2g4l
(J) Spanish: Porqu<eacute>nopuedensimplementehablarenEspa<ntilde>ol
U+0050 u+006F u+0072 u+0071 u+0075 u+00E9 u+006E u+006F u+0070
u+0075 u+0065 u+0064 u+0065 u+006E u+0073 u+0069 u+006D u+0070
u+006C u+0065 u+006D u+0065 u+006E u+0074 u+0065 u+0068 u+0061
u+0062 u+006C u+0061 u+0072 u+0065 u+006E U+0045 u+0073 u+0070
u+0061 u+00F1 u+006F u+006C
Punycode: PorqunopuedensimplementehablarenEspaol-fmd56a
(K) Vietnamese:
T<adotbelow>isaoh<odotbelow>kh<ocirc>ngth<ecirchookabove>ch\
<ihookabove>n<oacute>iti<ecircacute>ngVi<ecircdotbelow>t
U+0054 u+1EA1 u+0069 u+0073 u+0061 u+006F u+0068 u+1ECD u+006B
u+0068 u+00F4 u+006E u+0067 u+0074 u+0068 u+1EC3 u+0063 u+0068
u+1EC9 u+006E u+00F3 u+0069 u+0074 u+0069 u+1EBF u+006E u+0067
U+0056 u+0069 u+1EC7 u+0074
Punycode: TisaohkhngthchnitingVit-kjcr8268qyxafd2f1b9gThe next several examples are all names of Japanese music artists, song titles, and TV programs, just because the author happens to have them handy 🎀 (but Japanese is super useful for showing single-row text, two-row text, ideographic text, and all the mixtures in between 🌸)
(L) 3<nen>B<gumi><kinpachi><sensei>
u+0033 u+5E74 U+0042 u+7D44 u+91D1 u+516B u+5148 u+751F
Punycode: 3B-ww4c5e180e575a65lsy2b
(M) <amuro><namie>-with-SUPER-MONKEYS
u+5B89 u+5BA4 u+5948 u+7F8E u+6075 u+002D u+0077 u+0069 u+0074
u+0068 u+002D U+0053 U+0055 U+0050 U+0045 U+0052 u+002D U+004D
U+004F U+004E U+004B U+0045 U+0059 U+0053
Punycode: -with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n
(N) Hello-Another-Way-<sorezore><no><basho>
U+0048 u+0065 u+006C u+006C u+006F u+002D U+0041 u+006E u+006F
u+0074 u+0068 u+0065 u+0072 u+002D U+0057 u+0061 u+0079 u+002D
u+305D u+308C u+305E u+308C u+306E u+5834 u+6240
Punycode: Hello-Another-Way--fc4qua05auwb3674vfr0b
(O) <hitotsu><yane><no><shita>2
u+3072 u+3068 u+3064 u+5C4B u+6839 u+306E u+4E0B u+0032
Punycode: 2-u9tlzr9756bt3uc0v
(P) Maji<de>Koi<suru>5<byou><mae>
U+004D u+0061 u+006A u+0069 u+3067 U+004B u+006F u+0069 u+3059
u+308B u+0035 u+79D2 u+524D
Punycode: MajiKoi5-783gue6qz075azm5e
(Q) <pafii>de<runba>
u+30D1 u+30D5 u+30A3 u+30FC u+0064 u+0065 u+30EB u+30F3 u+30D0
Punycode: de-jg4avhby1noc0d
(R) <sono><supiido><de>
u+305D u+306E u+30B9 u+30D4 u+30FC u+30C9 u+3067
Punycode: d9juau41awczczpThe last example is an ASCII string that breaks the existing rules for host name labels 💀 (it's not a realistic example for IDNA, because IDNA never encodes pure ASCII labels, so this one is purely for vibes 🙃)
(S) -> $1.00 <-
u+002D u+003E u+0020 u+0024 u+0031 u+002E u+0030 u+0030 u+0020
u+003C u+002D
Punycode: -> $1.00 <--7.2. Decoding traces 🔥
okay so in these traces, the decoder's evolving state is shown as a sequence of hex values, which are the code points in the extended string 🧚 there's an asterisk right after the most recently inserted code point, and that asterisk tells you both n (the value right before it) and i (the position of the value just after it) 👑 everything else is decimal, so don't mix them up bestie 💅
Decoding trace of example B from section 7.1:
n is 128, i is 0, bias is 72
input is "ihqwcrb4cv8a8dqg056pqjye"
there is no delimiter, so extended string starts empty
delta "ihq" decodes to 19853
bias becomes 21
4E0D *
delta "wc" decodes to 64
bias becomes 20
4E0D 4E2D *
delta "rb" decodes to 37
bias becomes 13
4E3A * 4E0D 4E2D
delta "4c" decodes to 56
bias becomes 17
4E3A 4E48 * 4E0D 4E2D
delta "v8a" decodes to 599
bias becomes 32
4E3A 4EC0 * 4E48 4E0D 4E2D
delta "8d" decodes to 130
bias becomes 23
4ED6 * 4E3A 4EC0 4E48 4E0D 4E2D
delta "qg" decodes to 154
bias becomes 25
4ED6 4EEC * 4E3A 4EC0 4E48 4E0D 4E2D
delta "056p" decodes to 46301
bias becomes 84
4ED6 4EEC 4E3A 4EC0 4E48 4E0D 4E2D 6587 *
delta "qjye" decodes to 88531
bias becomes 90
4ED6 4EEC 4E3A 4EC0 4E48 4E0D 8BF4 * 4E2D 6587Decoding trace of example L from section 7.1:
n is 128, i is 0, bias is 72
input is "3B-ww4c5e180e575a65lsy2b"
literal portion is "3B-", so extended string starts as:
0033 0042
delta "ww4c" decodes to 62042
bias becomes 27
0033 0042 5148 *
delta "5e" decodes to 139
bias becomes 24
0033 0042 516B * 5148
delta "180e" decodes to 16683
bias becomes 67
0033 5E74 * 0042 516B 5148
delta "575a" decodes to 34821
bias becomes 82
0033 5E74 0042 516B 5148 751F *
delta "65l" decodes to 14592
bias becomes 67
0033 5E74 0042 7D44 * 516B 5148 751F
delta "sy2b" decodes to 42088
bias becomes 84
0033 5E74 0042 7D44 91D1 * 516B 5148 751F7.3. Encoding traces 🔥
in these ones the code point values are hex, and everything else is decimal 🌈 this is the encoder's side of the story, so you can watch each code point get its delta and bias update in real time 🎀
Encoding trace of example B from section 7.1:
bias is 72
input is:
4ED6 4EEC 4E3A 4EC0 4E48 4E0D 8BF4 4E2D 6587
there are no basic code points, so no literal portion
next code point to insert is 4E0D
needed delta is 19853, encodes as "ihq"
bias becomes 21
next code point to insert is 4E2D
needed delta is 64, encodes as "wc"
bias becomes 20
next code point to insert is 4E3A
needed delta is 37, encodes as "rb"
bias becomes 13
next code point to insert is 4E48
needed delta is 56, encodes as "4c"
bias becomes 17
next code point to insert is 4EC0
needed delta is 599, encodes as "v8a"
bias becomes 32
next code point to insert is 4ED6
needed delta is 130, encodes as "8d"
bias becomes 23
next code point to insert is 4EEC
needed delta is 154, encodes as "qg"
bias becomes 25
next code point to insert is 6587
needed delta is 46301, encodes as "056p"
bias becomes 84
next code point to insert is 8BF4
needed delta is 88531, encodes as "qjye"
bias becomes 90
output is "ihqwcrb4cv8a8dqg056pqjye"Encoding trace of example L from section 7.1:
bias is 72
input is:
0033 5E74 0042 7D44 91D1 516B 5148 751F
basic code points (0033, 0042) are copied to literal portion: "3B-"
next code point to insert is 5148
needed delta is 62042, encodes as "ww4c"
bias becomes 27
next code point to insert is 516B
needed delta is 139, encodes as "5e"
bias becomes 24
next code point to insert is 5E74
needed delta is 16683, encodes as "180e"
bias becomes 67
next code point to insert is 751F
needed delta is 34821, encodes as "575a"
bias becomes 82
next code point to insert is 7D44
needed delta is 14592, encodes as "65l"
bias becomes 67
next code point to insert is 91D1
needed delta is 42088, encodes as "sy2b"
bias becomes 84
output is "3B-ww4c5e180e575a65lsy2b"8. Security Considerations 🔐
okay bestie, this part is SERIOUS 😭 users expect every domain name in DNS to be controlled by a single authority 🫶 if a Unicode string meant for use as a domain label could map to multiple ACE labels, then one internationalized domain name could turn into multiple ASCII domain names, each controlled by a different authority, and some of those could be spoofs that hijack service requests meant for someone else 💀 that's why Punycode is designed so that each Unicode string has a unique encoding 🔒
however, there can still be multiple Unicode representations of the "same" text, depending on what you mean by "same" 🙈
this problem gets handled to some extent by the Unicode standard under the topic of canonicalization, and this work is leveraged for domain names by Nameprep [NAMEPREP] 💜
9. References 📚
9.1. Normative References 💅
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.9.2. Informative References 💖
[RFC952] Harrenstien, K., Stahl, M. and E. Feinler, "DOD Internet
Host Table Specification", RFC 952, October 1985.
[RFC1034] Mockapetris, P., "Domain Names - Concepts and
Facilities", STD 13, RFC 1034, November 1987.
[IDNA] Faltstrom, P., Hoffman, P. and A. Costello,
"Internationalizing Domain Names in Applications
(IDNA)", RFC 3490, March 2003.
[NAMEPREP] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep
Profile for Internationalized Domain Names (IDN)", RFC
3491, March 2003.
[ASCII] Cerf, V., "ASCII format for Network Interchange", RFC
20, October 1969.
[PROVINCIAL] Kaplan, M., "The 'anyone can be provincial!' page",
http://www.trigeminal.com/samples/provincial.html.
[UNICODE] The Unicode Consortium, "The Unicode Standard",
http://www.unicode.org/unicode/standard/standard.html.A. Mixed-case annotation ✨
okay so, if you want Punycode to represent strings that don't care about caps (case-insensitive, bestie 💅), the higher layers need to case-fold the strings before Punycode encoding. Then the encoded string can still carry mixed case as an annotation, basically a little note that says how to turn the folded string back into a mixed-case one for display 🎀. BUT heads up: the ToASCII and ToUnicode operations in [IDNA] don't use this annotation at all, so if you're an IDNA implementor you can just skip this appendix 🙈
Basic code points can be mixed case directly, because the decoder copies them over verbatim: lowercase stays lowercase and uppercase stays uppercase 🦋. Each non-basic code point is represented by a delta, which is a sequence of basic code points, and the last one in that sequence is the annotation. If it's uppercase, it's a suggestion to map the non-basic code point to uppercase (if possible) 💜. If it's lowercase, it's a suggestion to map it to lowercase (if possible) 🌸
These annotations don't change the code points that decoders return. They come back separately, and the caller can use them or ignore them 🫶 Encoders can accept annotations on top of code points too, but the annotations never change the output, except for influencing the uppercase/lowercase form of ASCII letters ✨
Punycode encoders and decoders don't have to support these annotations, and higher layers don't have to use them either 💖
B. Disclaimer and license 📜
okay this is the legal-ish part but make it cute 🧚 The author makes no guarantees about this entire document (pseudocode and C code included) and isn't responsible for any damage from using it 😭 Anyone gets irrevocable permission to use, modify, and distribute it in any way, as long as that doesn't take away anyone else's right to do the same 🌈 Redistributed derivative works just can't contain misleading author or version info 🙈 And derivative works don't need to be licensed under similar terms, so no pressure 💅
C. Punycode sample implementation 💻
here's the full sample C code, straight from the source 🎀 (the comments up top even say where it lives, iykyk)
/*
punycode.c from RFC 3492
http://www.nicemice.net/idn/
Adam M. Costello
http://www.nicemice.net/amc/
This is ANSI C code (C89) implementing Punycode (RFC 3492).
*/
/************************************************************/
/* Public interface (would normally go in its own .h file): */
#include <limits.h>
enum punycode_status {
punycode_success,
punycode_bad_input, /* Input is invalid. */
punycode_big_output, /* Output would exceed the space provided. */
punycode_overflow /* Input needs wider integers to process. */
};
#if UINT_MAX >= (1 << 26) - 1
typedef unsigned int punycode_uint;
#else
typedef unsigned long punycode_uint;
#endif
enum punycode_status punycode_encode(
punycode_uint input_length,
const punycode_uint input[],
const unsigned char case_flags[],
punycode_uint *output_length,
char output[] );
/* punycode_encode() converts Unicode to Punycode. The input */
/* is represented as an array of Unicode code points (not code */
/* units; surrogate pairs are not allowed), and the output */
/* will be represented as an array of ASCII code points. The */
/* output string is *not* null-terminated; it will contain */
/* zeros if and only if the input contains zeros. (Of course */
/* the caller can leave room for a terminator and add one if */
/* needed.) The input_length is the number of code points in */
/* the input. The output_length is an in/out argument: the */
/* caller passes in the maximum number of code points that it */
/* can receive, and on successful return it will contain the */
/* number of code points actually output. The case_flags array */
/* holds input_length boolean values, where nonzero suggests that */
/* the corresponding Unicode character be forced to uppercase */
/* after being decoded (if possible), and zero suggests that */
/* it be forced to lowercase (if possible). ASCII code points */
/* are encoded literally, except that ASCII letters are forced */
/* to uppercase or lowercase according to the corresponding */
/* uppercase flags. If case_flags is a null pointer then ASCII */
/* letters are left as they are, and other code points are */
/* treated as if their uppercase flags were zero. The return */
/* value can be any of the punycode_status values defined above */
/* except punycode_bad_input; if not punycode_success, then */
/* output_size and output might contain garbage. */
enum punycode_status punycode_decode(
punycode_uint input_length,
const char input[],
punycode_uint *output_length,
punycode_uint output[],
unsigned char case_flags[] );
/* punycode_decode() converts Punycode to Unicode. The input is */
/* represented as an array of ASCII code points, and the output */
/* will be represented as an array of Unicode code points. The */
/* input_length is the number of code points in the input. The */
/* output_length is an in/out argument: the caller passes in */
/* the maximum number of code points that it can receive, and */
/* on successful return it will contain the actual number of */
/* code points output. The case_flags array needs room for at */
/* least output_length values, or it can be a null pointer if the */
/* case information is not needed. A nonzero flag suggests that */
/* the corresponding Unicode character be forced to uppercase */
/* by the caller (if possible), while zero suggests that it be */
/* forced to lowercase (if possible). ASCII code points are */
/* output already in the proper case, but their flags will be set */
/* appropriately so that applying the flags would be harmless. */
/* The return value can be any of the punycode_status values */
/* defined above; if not punycode_success, then output_length, */
/* output, and case_flags might contain garbage. On success, the */
/* decoder will never need to write an output_length greater than */
/* input_length, because of how the encoding is defined. */
/**********************************************************/
/* Implementation (would normally go in its own .c file): */
#include <string.h>
/*** Bootstring parameters for Punycode ***/
enum { base = 36, tmin = 1, tmax = 26, skew = 38, damp = 700,
initial_bias = 72, initial_n = 0x80, delimiter = 0x2D };
/* basic(cp) tests whether cp is a basic code point: */
#define basic(cp) ((punycode_uint)(cp) < 0x80)
/* delim(cp) tests whether cp is a delimiter: */
#define delim(cp) ((cp) == delimiter)
/* decode_digit(cp) returns the numeric value of a basic code */
/* point (for use in representing integers) in the range 0 to */
/* base-1, or base if cp is does not represent a value. */
static punycode_uint decode_digit(punycode_uint cp)
{
return cp - 48 < 10 ? cp - 22 : cp - 65 < 26 ? cp - 65 :
cp - 97 < 26 ? cp - 97 : base;
}
/* encode_digit(d,flag) returns the basic code point whose value */
/* (when used for representing integers) is d, which needs to be in */
/* the range 0 to base-1. The lowercase form is used unless flag is */
/* nonzero, in which case the uppercase form is used. The behavior */
/* is undefined if flag is nonzero and digit d has no uppercase form. */
static char encode_digit(punycode_uint d, int flag)
{
return d + 22 + 75 * (d < 26) - ((flag != 0) << 5);
/* 0..25 map to ASCII a..z or A..Z */
/* 26..35 map to ASCII 0..9 */
}
/* flagged(bcp) tests whether a basic code point is flagged */
/* (uppercase). The behavior is undefined if bcp is not a */
/* basic code point. */
#define flagged(bcp) ((punycode_uint)(bcp) - 65 < 26)
/* encode_basic(bcp,flag) forces a basic code point to lowercase */
/* if flag is zero, uppercase if flag is nonzero, and returns */
/* the resulting code point. The code point is unchanged if it */
/* is caseless. The behavior is undefined if bcp is not a basic */
/* code point. */
static char encode_basic(punycode_uint bcp, int flag)
{
bcp -= (bcp - 97 < 26) << 5;
return bcp + ((!flag && (bcp - 65 < 26)) << 5);
}
/*** Platform-specific constants ***/
/* maxint is the maximum value of a punycode_uint variable: */
static const punycode_uint maxint = -1;
/* Because maxint is unsigned, -1 becomes the maximum value. */
/*** Bias adaptation function ***/
static punycode_uint adapt(
punycode_uint delta, punycode_uint numpoints, int firsttime )
{
punycode_uint k;
delta = firsttime ? delta / damp : delta >> 1;
/* delta >> 1 is a faster way of doing delta / 2 */
delta += delta / numpoints;
for (k = 0; delta > ((base - tmin) * tmax) / 2; k += base) {
delta /= base - tmin;
}
return k + (base - tmin + 1) * delta / (delta + skew);
}
/*** Main encode function ***/
enum punycode_status punycode_encode(
punycode_uint input_length,
const punycode_uint input[],
const unsigned char case_flags[],
punycode_uint *output_length,
char output[] )
{
punycode_uint n, delta, h, b, out, max_out, bias, j, m, q, k, t;
/* Initialize the state: */
n = initial_n;
delta = out = 0;
max_out = *output_length;
bias = initial_bias;
/* Handle the basic code points: */
for (j = 0; j < input_length; ++j) {
if (basic(input[j])) {
if (max_out - out < 2) return punycode_big_output;
output[out++] =
case_flags ? encode_basic(input[j], case_flags[j]) : input[j];
}
/* else if (input[j] < n) return punycode_bad_input; */
/* (not needed for Punycode with unsigned code points) */
}
h = b = out;
/* h is the number of code points that have been handled, b is the */
/* number of basic code points, and out is the number of characters */
/* that have been output. */
if (b > 0) output[out++] = delimiter;
/* Main encoding loop: */
while (h < input_length) {
/* All non-basic code points < n have been */
/* handled already. Find the next larger one: */
for (m = maxint, j = 0; j < input_length; ++j) {
/* if (basic(input[j])) continue; */
/* (not needed for Punycode) */
if (input[j] >= n && input[j] < m) m = input[j];
}
/* Increase delta enough to advance the decoder's */
/* <n,i> state to <m,0>, but guard against overflow: */
if (m - n > (maxint - delta) / (h + 1)) return punycode_overflow;
delta += (m - n) * (h + 1);
n = m;
for (j = 0; j < input_length; ++j) {
/* Punycode does not need to check whether input[j] is basic: */
if (input[j] < n /* || basic(input[j]) */ ) {
if (++delta == 0) return punycode_overflow;
}
if (input[j] == n) {
/* Represent delta as a generalized variable-length integer: */
for (q = delta, k = base; ; k += base) {
if (out >= max_out) return punycode_big_output;
t = k <= bias /* + tmin */ ? tmin : /* +tmin not needed */
k >= bias + tmax ? tmax : k - bias;
if (q < t) break;
output[out++] = encode_digit(t + (q - t) % (base - t), 0);
q = (q - t) / (base - t);
}
output[out++] = encode_digit(q, case_flags && case_flags[j]);
bias = adapt(delta, h + 1, h == b);
delta = 0;
++h;
}
}
++delta, ++n;
}
*output_length = out;
return punycode_success;
}
/*** Main decode function ***/okay bestie, the decoder is where the magic happens 🧚✨ here's the rest of it, plus the test wrapper (the bit that lets you play with it in a terminal) 💻
enum punycode_status punycode_decode(
punycode_uint input_length,
const char input[],
punycode_uint *output_length,
punycode_uint output[],
unsigned char case_flags[] )
{
punycode_uint n, out, i, max_out, bias,
b, j, in, oldi, w, k, digit, t;
/* Initialize the state: */
n = initial_n;
out = i = 0;
max_out = *output_length;
bias = initial_bias;
/* Handle the basic code points: Let b be the number of input code */
/* points before the last delimiter, or 0 if there is none, then */
/* copy the first b code points to the output. */
for (b = j = 0; j < input_length; ++j) if (delim(input[j])) b = j;
if (b > max_out) return punycode_big_output;
for (j = 0; j < b; ++j) {
if (case_flags) case_flags[out] = flagged(input[j]);
if (!basic(input[j])) return punycode_bad_input;
output[out++] = input[j];
}
/* Main decoding loop: Start just after the last delimiter if any */
/* basic code points were copied; start at the beginning otherwise. */
for (in = b > 0 ? b + 1 : 0; in < input_length; ++out) {
/* in is the index of the next character to be consumed, and */
/* out is the number of code points in the output array. */
/* Decode a generalized variable-length integer into delta, */
/* which gets added to i. The overflow checking is easier */
/* if we increase i as we go, then subtract off its starting */
/* value at the end to obtain delta. */
for (oldi = i, w = 1, k = base; ; k += base) {
if (in >= input_length) return punycode_bad_input;
digit = decode_digit(input[in++]);
if (digit >= base) return punycode_bad_input;
if (digit > (maxint - i) / w) return punycode_overflow;
i += digit * w;
t = k <= bias /* + tmin */ ? tmin : /* +tmin not needed */
k >= bias + tmax ? tmax : k - bias;
if (digit < t) break;
if (w > maxint / (base - t)) return punycode_overflow;
w *= (base - t);
}
bias = adapt(i - oldi, out + 1, oldi == 0);
/* i was supposed to wrap around from out+1 to 0, */
/* incrementing n each time, so we'll fix that now: */
if (i / (out + 1) > maxint - n) return punycode_overflow;
n += i / (out + 1);
i %= (out + 1);
/* Insert n at position i of the output: */
/* not needed for Punycode: */
/* if (decode_digit(n) <= base) return punycode_invalid_input; */
if (out >= max_out) return punycode_big_output;
if (case_flags) {
memmove(case_flags + i + 1, case_flags + i, out - i);
/* Case of last character determines uppercase flag: */
case_flags[i] = flagged(input[in - 1]);
}
memmove(output + i + 1, output + i, (out - i) * sizeof *output);
output[i++] = n;
}
*output_length = out;
return punycode_success;
}
/******************************************************************/
/* Wrapper for testing (would normally go in a separate .c file): */
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* For testing, we'll just set some compile-time limits rather than */
/* use malloc(), and set a compile-time option rather than using a */
/* command-line option. */
enum {
unicode_max_length = 256,
ace_max_length = 256
};
static void usage(char **argv)
{
fprintf(stderr,
"\n"
"%s -e reads code points and writes a Punycode string.\n"
"%s -d reads a Punycode string and writes code points.\n"
"\n"
"Input and output are plain text in the native character set.\n"
"Code points are in the form u+hex separated by whitespace.\n"
"Although the specification allows Punycode strings to contain\n"
"any characters from the ASCII repertoire, this test code\n"
"supports only the printable characters, and needs the Punycode\n"
"string to be followed by a newline.\n"
"The case of the u in u+hex is the force-to-uppercase flag.\n"
, argv[0], argv[0]);
exit(EXIT_FAILURE);
}
static void fail(const char *msg)
{
fputs(msg,stderr);
exit(EXIT_FAILURE);
}
static const char too_big[] =
"input or output is too large, recompile with larger limits\n";
static const char invalid_input[] = "invalid input\n";
static const char overflow[] = "arithmetic overflow\n";
static const char io_error[] = "I/O error\n";
/* The following string is used to convert printable */
/* characters between ASCII and the native charset: */
static const char print_ascii[] =
"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
" !\"#$%&'()*+,-./"
"0123456789:;<=>?"
"@ABCDEFGHIJKLMNO"
"PQRSTUVWXYZ[\\]^_"
"`abcdefghijklmno"
"pqrstuvwxyz{|}~\n";
int main(int argc, char **argv)
{
enum punycode_status status;
int r;
unsigned int input_length, output_length, j;
unsigned char case_flags[unicode_max_length];
if (argc != 2) usage(argv);
if (argv[1][0] != '-') usage(argv);
if (argv[1][2] != 0) usage(argv);
if (argv[1][1] == 'e') {
punycode_uint input[unicode_max_length];
unsigned long codept;
char output[ace_max_length+1], uplus[3];
int c;
/* Read the input code points: */
input_length = 0;
for (;;) {
r = scanf("%2s%lx", uplus, &codept);
if (ferror(stdin)) fail(io_error);
if (r == EOF || r == 0) break;
if (r != 2 || uplus[1] != '+' || codept > (punycode_uint)-1) {
fail(invalid_input);
}
if (input_length == unicode_max_length) fail(too_big);
if (uplus[0] == 'u') case_flags[input_length] = 0;
else if (uplus[0] == 'U') case_flags[input_length] = 1;
else fail(invalid_input);
input[input_length++] = codept;
}
/* Encode: */
output_length = ace_max_length;
status = punycode_encode(input_length, input, case_flags,
&output_length, output);
if (status == punycode_bad_input) fail(invalid_input);
if (status == punycode_big_output) fail(too_big);
if (status == punycode_overflow) fail(overflow);
assert(status == punycode_success);
/* Convert to native charset and output: */
for (j = 0; j < output_length; ++j) {
c = output[j];
assert(c >= 0 && c <= 127);
if (print_ascii[c] == 0) fail(invalid_input);
output[j] = print_ascii[c];
}
output[j] = 0;
r = puts(output);
if (r == EOF) fail(io_error);
return EXIT_SUCCESS;
}
if (argv[1][1] == 'd') {
char input[ace_max_length+2], *p, *pp;
punycode_uint output[unicode_max_length];
/* Read the Punycode input string and convert to ASCII: */
fgets(input, ace_max_length+2, stdin);
if (ferror(stdin)) fail(io_error);
if (feof(stdin)) fail(invalid_input);
input_length = strlen(input) - 1;
if (input[input_length] != '\n') fail(too_big);
input[input_length] = 0;
for (p = input; *p != 0; ++p) {
pp = strchr(print_ascii, *p);
if (pp == 0) fail(invalid_input);
*p = pp - print_ascii;
}
/* Decode: */
output_length = unicode_max_length;
status = punycode_decode(input_length, input, &output_length,
output, case_flags);
if (status == punycode_bad_input) fail(invalid_input);
if (status == punycode_big_output) fail(too_big);
if (status == punycode_overflow) fail(overflow);
assert(status == punycode_success);
/* Output the result: */
for (j = 0; j < output_length; ++j) {
r = printf("%s+%04lX\n",
case_flags[j] ? "U" : "u",
(unsigned long) output[j] );
if (r < 0) fail(io_error);
}
return EXIT_SUCCESS;
}
usage(argv);
return EXIT_SUCCESS; /* not reached, but quiets compiler warning */
}Author's Address ✨
Adam M. Costello University of California, Berkeley http://www.nicemice.net/amc/
Full Copyright Statement 💖
Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement 🫶
Funding for the RFC Editor function is currently provided by the Internet Society. 🌸
📦 where this page comes from
- ietf: rfc3492 2003-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms