PERLUNICODE(1perl)
PERLUNICODE(1perl) 💖✨ — Unicode support in Perl
okay bestie, this is perlunicode from the Perl Programmers Reference Guide, and we are about to get u fully UNI-fied. no cap. 👑
NAME
perlunicode— Unicode support in Perl. periodt. 🦋
DESCRIPTION
Before you dive in, if you haven't already, go get familiar with both perlunitut and perluniintro. They're the pre-reqs, bestie. 💅
Unicode's whole thing is to UNI-fy the en-CODE-ings of every character set in the world into one single Standard. Back when Unicode first showed up, a lot of the coding standards around then could be converted by just adding a constant to each code point in the original standard—and converting back meant subtracting that same constant. For ASCII and ISO-8859-1, that constant is 0. For ISO-8859-5 (Cyrillic), it's 864. For Hebrew (ISO-8859-8), it's 1488. For Thai (ISO-8859-11), it's 3424. And so on and so forth. This made conversions easy and helped Unicode get adopted. ✨
And it ate. These days, those legacy standards are barely used. Almost everyone is on Unicode now. 🌈
Unicode is a comprehensive standard—it specifies a bunch of stuff outside Perl's scope, like how to display sequences of characters. For the full tea on every aspect of Unicode, see <https://www.unicode.org>. 👑
Important Caveats
Some of this section might not click on the first read, and that's okay. But we think it's important enough to spotlight the gotchas before we go deeper. So here goes: 🫶
Unicode support is an extensive requirement. Perl does not implement the Unicode standard or its accompanying technical reports cover to cover, but Perl does support many Unicode features.
Also, Unicode can bring security issues that aren't obvious. See "Security Implications of Unicode" below. 🚨
Safest if you use feature 'unicode_strings' 💅
To keep backward compatibility, Perl does not turn on full internal Unicode support unless you specify the pragma use feature 'unicode_strings'. (This gets automatically selected if you use v5.12 or higher.) If you don't do this, you can get unexpected surprises. See "The "Unicode Bug"" below.
This pragma does not affect I/O. It also doesn't change the internal representation of strings—only their interpretation. There are still several places where Unicode isn't fully supported, like in filenames. 😭
Input and Output Layers
Use the :encoding(...) layer to read from and write to filehandles using the encoding you specify. (See open.) 🎀
You must convert your non-ASCII, non-UTF-8 Perl scripts to be UTF-8.
The encoding module has been deprecated since perl 5.18, and the perl internals it needs have been removed with perl 5.26. 💀
use utf8 still needed to enable UTF-8 in scripts
If your Perl script is itself encoded in UTF-8, you have to explicitly include the use utf8 pragma to enable recognition of that—in string or regular expression literals, or in identifier names. This is the only time you need an explicit use utf8. (See utf8.)
If a Perl script begins with the bytes that form the UTF-8 encoding of the Unicode BYTE ORDER MARK ("BOM", see "Unicode Encodings"), those bytes are completely ignored. 🦋
UTF-16 scripts autodetected
If a Perl script begins with the Unicode "BOM" (UTF-16LE, UTF16-BE), or if the script looks like non-"BOM"-marked UTF-16 of either endianness, Perl will correctly read the script as the appropriate Unicode encoding. ✨
Byte and Character Semantics
Before Unicode, most encodings used 8 bits (a single byte) to encode each character. So a character was a byte, a byte was a character, and you could have only 256 or fewer possible characters. That's what "Byte Semantics" in this section's title means. Nobody needed to distinguish between "Byte" and "Character". 💖
Then Unicode pulled up with room for over a million characters (and Perl allows even more). That means a character might need more than a single byte to represent it, so the two terms are no longer equivalent. What matters is the characters as whole entities, not usually the bytes that make them up. That's the "Character Semantics" in the title. 👑
Perl had to change internally to decouple "bytes" from "characters". And you need to change your mindset too, if you haven't already, so "byte" and "character" no longer mean the same thing in your brain. 🧚
The basic building block of Perl strings has always been a "character". The changes basically come down to: the implementation no longer thinks a character is always just a single byte.
There are various things to note:
- String handling functions mostly still operate in terms of characters.
length(), for example, returns the number of characters in a string, just like before. But that number is no longer necessarily the same as the number of bytes in the string (there can be more bytes than characters). The other functions like this includechop(),chomp(),substr(),pos(),index(),rindex(),sort(),sprintf(), andwrite(). 💅
The exceptions are:
- the bit-oriented
vec - the byte-oriented
pack/unpackCformat
But the W specifier does operate on whole characters, and so does the U specifier.
- some operators that interact with the platform's operating system
Operators dealing with filenames are examples.
- when the functions are called from within the scope of the
use bytespragma
Likely you should only use this for debugging anyway.
- Strings—including hash keys—and regular expression patterns can contain characters with ordinal values larger than 255.
If you use a Unicode editor to edit your program, Unicode characters can appear directly inside literal strings in UTF-8 encoding, or UTF-16. (The former requires a use utf8; the latter may require a BOM.)
"Creating Unicode" in perluniintro gives other ways to put non-ASCII characters in your strings. 🌸
- The
chr()andord()functions work on whole characters. - Regular expressions match whole characters. For example,
.matches a whole character instead of only a single byte. - The
tr///operator translates whole characters. (Note that thetr///CUfunctionality has been removed. For similar functionality, seepack('U0', ...)andpack('C0', ...).) scalar reverse()reverses by character rather than by byte.- The bit string operators,
& | ^ ~and (starting in v5.22)&. |. ^. ~.can operate on bit strings encoded in UTF-8, but this can give unexpected results if any of the strings contain code points above 0xFF. Starting in v5.28, it is a fatal error to have such an operand. Otherwise, the operation is performed on a non-UTF-8 copy of the operand. If you're not sure about the encoding of a string, downgrade it before using any of these operators; you can useutf8::utf8_downgrade(). 🔥
Bottom line: Perl has always practiced "Character Semantics", but with Unicode in the picture, that's now different from "Byte Semantics". 💜
ASCII Rules versus Unicode Rules
Before Unicode, when a character was a byte was a character, Perl only knew about the 128 characters defined by ASCII—code points 0 through 127 (except under use locale). That left code points 128 to 255 as unassigned, free for whatever a program wanted to do with them. Their only semantics are their ordinal numbers, and the fact that they're members of none of the non-negative character classes. None of them are considered to match \w, for example, but all of them match \W. 👀
Unicode, obviously, gives each of those code points a particular meaning (along with ones above 255). To preserve backward compatibility, Perl only uses the Unicode meanings when something indicates that Unicode is what's intended; otherwise those non-ASCII code points stay treated as if they're unassigned.
Here are the ways Perl knows a string should be treated as Unicode:
- Within the scope of
use utf8
If the whole program is Unicode (signified by using 8-bit Unicode Transformation Format), then all literal strings inside it must be Unicode.
- Within the scope of
use feature 'unicode_strings'
This pragma was made so you can explicitly tell Perl that operations running inside its scope use Unicode rules. More operations are affected in newer perls. See "The "Unicode Bug"". ✨
- Within the scope of
use v5.12or higher
This implicitly turns on use feature 'unicode_strings'.
- Within the scope of
use locale 'not_characters', oruse localeand the current locale is a UTF-8 locale.
The former is defined to imply Unicode handling; the latter signals a Unicode locale, which means a Unicode interpretation of all strings inside it.
- When the string contains a Unicode-only code point
Perl has never accepted code points above 255 unless they're Unicode, so using them implies Unicode for the whole string.
- When the string contains a Unicode named code point
\N{...}
The \N{...} construct explicitly refers to a Unicode code point, even if it's one that's also in ASCII. So the string containing it must be Unicode.
- When the string has come from an external source marked as Unicode
The -C command line option can specify that certain inputs to the program are Unicode, and your Perl code can read the values of this; see ${^UNICODE} in perlvar. 💖
- When the string has been upgraded to UTF-8
The function utf8::utf8_upgrade() can be explicitly used to permanently (unless a later utf8::utf8_downgrade() is called) cause a string to be treated as Unicode.
- There are additional methods for regular expression patterns
A pattern compiled with the /u or /a modifiers is treated as Unicode (though there are some restrictions with /a). Under the /d and /l modifiers, there are several other indications for Unicode; see "Character set modifiers" in perlre. 🎀
Note that all of the above gets overridden within the scope of use bytes; but you should only be using that pragma for debugging.
Also note that some interactions with the platform's operating system never use Unicode rules.
When Unicode rules are in effect:
- Case translation operators use the Unicode case translation tables. 👑
Note bestie that uc(), or "\U" inside interpolated strings, translates stuff to uppercase, while ucfirst, or "\u" inside interpolated strings, translates to titlecase in the languages that actually make that distinction (which is literally just uppercase in the languages that don't). 🫶
There's a CPAN module, Unicode::Casing, that lets you define your own mappings to be used in lc(), lcfirst(), uc(), ucfirst(), and fc (or their double-quoted-string inlined versions like "\U"). 💅 (Historically, before Perl 5.16, this functionality was partially provided right in the Perl core, but it had a whole pile of insurmountable drawbacks, so the CPAN module got written instead. The more you know. 🌈)
- 🔤 Character classes in regular expressions match based on the character properties specified in the Unicode properties database.
"\w" can be used to match a Japanese ideograph, for instance; and [[:digit:]] a Bengali number. Not me matching scripts I don't even read 😭 so iconic.
- 🏷️ Named Unicode properties, scripts, and block ranges may be used (like bracketed character classes) by using the
"\p{}""matches property" construct and the"\P{}"negation, "doesn't match property".
See "Unicode Character Properties" for more details.
You can also define your own character properties and use them in the regular expression with the "\p{}" or "\P{}" construct. See "User-Defined Character Properties" for more details. Customization queen 👑
✨ Extended Grapheme Clusters (Logical characters) ✨
Okay so picture a character, let's say "H". It could show up with all kinds of marks around it — an acute accent, a circumflex, hooks, circles, arrows, all of that — sitting above it, below it, off to one side, whatever. The world's languages have so many possibilities here. Honestly the number of combos is astronomical, and if Unicode made a separate character for every single combination it would burn through its more-than-a-million possible characters immediately. So Unicode took a different route: there's a character for the base "H", and a character for each of the possible marks, and you combine them however you need to get your final logical character. 💫
So a logical character — the thing that looks like one single character to your eyeballs — can actually be a sequence of more than one individual character. Unicode calls these "extended grapheme clusters" (which is the glow-up version of "grapheme cluster", a term that's not really used much anymore). And Perl hands you the "\X" regular expression construct so you can match those sequences in their entirety. 🎀
But here's the ✨plot twist✨: Unicode's whole intent is to unify all the existing character set standards and practices, and several standards that came before it already have single characters meaning the same thing as some of these combos. Like ISO-8859-1, which has quite a few of them. For example, "LATIN CAPITAL LETTER E WITH ACUTE" was already in this standard when Unicode came along. So Unicode added it to its repertoire as that single character. 🥰 But Unicode considers this character to be equivalent to the sequence consisting of the character "LATIN CAPITAL LETTER E" followed by the character "COMBINING ACUTE ACCENT".
"LATIN CAPITAL LETTER E WITH ACUTE" is called a "pre-composed" character, and its equivalence with the "E" + "COMBINING ACCENT" sequence is called canonical equivalence. Every pre-composed character is said to have a decomposition (into the equivalent sequence), and the decomposition type is also called canonical. A string may consist as much as possible of precomposed characters, or it may consist of entirely decomposed characters. Unicode calls these respectively "Normalization Form Composed" (NFC) and "Normalization Form Decomposed". The Unicode::Normalize module contains functions that convert between the two. 🔄 A string may also have both composed and decomposed characters mixed together; this module can be used to make it all one or the other.
So yeah, you may be handed strings in any of these equivalent forms. And currently there is nothing in Perl 5 that ignores the differences. 😭 So you'll have to specially handle it. The usual advice? Convert your inputs to "NFD" before processing further. Periodt.
For more detailed information, see <http://unicode.org/reports/tr15/>.
💜 Unicode Character Properties 💜
(Quick disclaimer: the only time Perl considers a sequence of individual code points as a single logical character is in the "\X" construct, already mentioned up there. 👆 So when this discussion says "character", it means a single Unicode code point. Cool? Cool.)
Very nearly all Unicode character properties are accessible through regular expressions by using the "\p{}" "matches property" construct and the "\P{}" "doesn't match property" for its negation. 🫶
For instance, "\p{Uppercase}" matches any single character with the Unicode "Uppercase" property, while "\p{L}" matches any character with a "General_Category" of "L" (letter) property (peep "General_Category" below). Brackets are not required for single letter property names, so "\p{L}" is equivalent to "\pL". Easy.
More formally: "\p{Uppercase}" matches any single character whose Unicode "Uppercase" property value is "True", and "\P{Uppercase}" matches any character whose "Uppercase" property value is "False" — so those could have been written as "\p{Uppercase=True}" and "\p{Uppercase=False}", respectively. ✨
This formality is needed when properties aren't binary — that is, when they can take on more values than just "True" and "False". For example, the "Bidi_Class" property (see "Bidirectional Character Types" below) can take on several different values, such as "Left", "Right", "Whitespace", and others. To match those, you need to specify both the property name ("Bidi_Class") AND the value you're matching against ("Left", "Right", etc.). And you do that, just like in the examples above, by having the two components separated by an equal sign (or, interchangeably, a colon), like "\p{Bidi_Class: Left}". 💅
All Unicode-defined character properties may be written in these compound forms of "\p{property=value}" or "\p{property:value}", but Perl provides some additional properties that are written only in the single form, as well as single-form short-cuts for all binary properties and certain others described below, in which you may omit the property name and the equals or colon separator. 👑
Most Unicode character properties have at least two synonyms (or aliases, if that's your vibe): a short one that's easier to type, and a longer one that's more descriptive and hence easier to understand. Thus the "L" and "Letter" properties above are equivalent and can be used interchangeably. Likewise, "Upper" is a synonym for "Uppercase", and we could have written "\p{Uppercase}" equivalently as "\p{Upper}". Also, there are typically various synonyms for the values the property can be. For binary properties, "True" has 3 synonyms: "T", "Yes", and "Y"; and "False" correspondingly has "F", "No", and "N".
⚠️ But be careful, bestie. A short form of a value for one property may not mean the same thing as the short form spelled the same for another. Thus, for the"General_Category"property,"L"means"Letter", but for the"Bidi_Class"property,"L"means"Left". 😵 A complete list of properties and synonyms is in perluniprops.
Upper/lower case differences in property names and values are irrelevant; thus "\p{Upper}" means the same thing as "\p{upper}" or even "\p{UpPeR}". Similarly, you can add or subtract underscores anywhere in the middle of a word, so these are also equivalent to "\p{U_p_p_e_r}". And white space is generally irrelevant adjacent to non-word characters, such as the braces and the equals or colon separators, so "\p{ Upper }" and "\p{ Upper_case : Y }" are equivalent to these as well. In fact, white space and even hyphens can usually be added or deleted anywhere. So even "\p{ Up-per case = Yes}" is equivalent. 😌 All of this is called "loose-matching" by Unicode. The "name" property has some restrictions on this due to a few outlier names. Full details are given in <https://www.unicode.org/reports/tr44/tr44-24.html#UAX44-LM2>.
The few places where stricter matching is used is in the middle of numbers, the "name" property, and in the Perl extension properties that begin or end with an underscore. Stricter matching cares about white space (except adjacent to non-word characters), hyphens, and non-interior underscores. 🙈
You can also use negation in both "\p{}" and "\P{}" by introducing a caret ("^") between the first brace and the property name: "\p{^Tamil}" is equal to "\P{Tamil}". 🎀
Almost all properties are immune to case-insensitive matching. That is, adding a /i regular expression modifier does not change what they match. But there are two sets that are affected. 👇
- The first set is
"Uppercase_Letter","Lowercase_Letter", and"Titlecase_Letter", all of which match"Cased_Letter"under/imatching. - And the second set is
"Uppercase","Lowercase", and"Titlecase", all of which match"Cased"under/imatching. This set also includes its subsets"PosixUpper"and"PosixLower", both of which under/imatch"PosixAlpha".
(The difference between these sets is that some things, such as Roman numerals, come in both upper and lower case so they are "Cased", but aren't considered letters, so they aren't "Cased_Letter"'s. 🏛️)
See "Beyond Unicode code points" for special considerations when matching Unicode properties against non-Unicode code points.
🌸 General_Category 🌸
Every Unicode character is assigned a general category, which is the "most usual categorization of a character" (from <https://www.unicode.org/reports/tr44>). It's giving ✨organized✨.
The compound way of writing these is like "\p{General_Category=Number}" (short: "\p{gc:n}"). But Perl furnishes shortcuts in which everything up through the equal or colon separator is omitted. So you can instead just write "\pN". Lazy girl math, we love it 💅
Here are the short and long forms of the values the "General Category" property can have:
L—LetterLC,L&—Cased_Letter(that is:[\p{Ll}\p{Lu}\p{Lt}])Lu—Uppercase_LetterLl—Lowercase_LetterLt—Titlecase_LetterLm—Modifier_LetterLo—Other_LetterM—MarkMn—Nonspacing_MarkMc—Spacing_MarkMe—Enclosing_MarkN—NumberNd—Decimal_Number(alsoDigit)Nl—Letter_NumberNo—Other_NumberP—Punctuation(alsoPunct)Pc—Connector_PunctuationPd—Dash_PunctuationPs—Open_PunctuationPe—Close_PunctuationPi—Initial_Punctuation(may behave likePsorPedepending on usage)Pf—Final_Punctuation(may behave likePsorPedepending on usage)Po—Other_PunctuationS— Symbol</p>Sm—Math_SymbolSc—Currency_SymbolSk—Modifier_SymbolSo—Other_SymbolZ— Separator</p>Zs—Space_SeparatorZl—Line_SeparatorZp—Paragraph_SeparatorC— Other</p>Cc—Control(alsoCntrl) 🙈Cf—FormatCs—SurrogateCo—Private_UseCn—Unassigned
k so here's a lil cheat code bestie: single-letter properties match every character in any of the two-letter sub-properties that start with that same letter. iconic. but LC and L& are special — they're both aliases for the set of everything matched by Ll, Lu, and Lt. no cap.
Bidirectional Character Types 🔄
Real talk: scripts don't all go the same direction. Hebrew and Arabic? Written right to left, babes. Left to right is not universal, it's giving main character syndrome. So Unicode supplies a Bidi_Class property to keep track of all that. Some of the values it can have are:
L— Left-to-RightLRE— Left-to-Right EmbeddingLRO— Left-to-Right OverrideR— Right-to-LeftAL— Arabic LetterRLE— Right-to-Left EmbeddingRLO— Right-to-Left OverridePDF— Pop Directional FormatEN— European NumberES— European SeparatorET— European TerminatorAN— Arabic NumberCS— Common SeparatorNSM— Non-Spacing MarkBN— Boundary NeutralB— Paragraph SeparatorS— Segment SeparatorWS— WhitespaceON— Other Neutrals
This property is always written in the compound form. Like, for example, \p{Bidi_Class:R} matches characters that are normally written right to left. 💅
But heads up: unlike the General_Category property, this one can have more values added in a future Unicode release. The list above was the complete set for many Unicode releases, but more got added in Unicode 6.3 — and you can always find out what the current ones are in perluniprops. And <https://www.unicode.org/reports/tr9/> describes how to actually use them.
Scripts 🦋
The world's languages are written in so many different scripts and honestly it's beautiful. This sentence (unless you're reading it in translation) is written in Latin, while Russian is written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in Hiragana or Katakana. There are many more. 💖
The Unicode Script and Script_Extensions properties tell you what script a given character is in. And Script_Extensions is like the glow-up version of Script, as demonstrated below. Either property can be specified with the compound form like \p{Script=Hebrew} (short: \p{sc=hebr}), or \p{Script_Extensions=Javanese} (short: \p{scx=java}).
Plus, Perl furnishes shortcuts for all Script_Extensions property names. You can yeet everything up through the equals (or colon) and simply write \p{Latin} or \P{Cyrillic}. 🎀
Important though: this is not true forScript, which is required to be written in the compound form. Prior to Perl v5.26, the single form returned the plain oldScriptversion, but that got changed becauseScript_Extensionsgives better results. bestie upgraded.
The difference between these two properties involves characters that are used in multiple scripts. For example the digits '0' through '9' are used in many parts of the world — those live in a script named Common. Other characters are used in just a few scripts. For example, the KATAKANA-HIRAGANA DOUBLE HYPHEN is used in both Japanese scripts, Katakana and Hiragana, but nowhere else. 👑
So: the Script property places all characters that are used in multiple scripts in the Common script, while Script_Extensions places those that are used in only a few scripts into each of those scripts; while still using Common for those used in many scripts. Thus both of these match:
"0" =~ /\p{sc=Common}/ # Matches
"0" =~ /\p{scx=Common}/ # Matchesand only the first of these match:
"\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Common} # Matches
"\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Common} # No matchAnd only the last two of these match:
"\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Hiragana} # No match
"\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{sc=Katakana} # No match
"\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Hiragana} # Matches
"\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}" =~ /\p{scx=Katakana} # MatchesSo Script_Extensions is thus an improved Script, in which there are fewer characters in the Common script, and correspondingly more in other scripts. It's new in Unicode version 6.0, and its data are likely to change significantly in later releases, as things get sorted out. New code should probably be using Script_Extensions and not plain Script. ✨
And two edge cases, because of course there are edge cases bestie:
- If you compile perl with a Unicode release that doesn't have
Script_Extensions, the single form Perl extensions will instead refer to the plainScriptproperty. - If you compile with a version of Unicode that doesn't have the
Scriptproperty, these extensions will not be defined at all. 😭
(Actually, besides Common, the Inherited script also contains characters that are used in multiple scripts. These are modifier characters which inherit the script value of the controlling character. Some of these are used in many scripts, and so go into Inherited in both Script and Script_Extensions. Others are used in just a few scripts, so are in Inherited in Script, but not in Script_Extensions.)
Also worth stressing: there are several different sets of digits in Unicode that are equivalent to 0-9 and are matchable by \d in a regular expression. If they're used in a single language only, they're in that language's Script and Script_Extensions. If they're used in more than one script, they'll be in sc=Common — but only if they're used in many scripts should they be in scx=Common.
The explanation above has omitted some detail; refer to UAX#24 "Unicode Script Property": <https://www.unicode.org/reports/tr24>.
A complete list of scripts and their shortcuts is in perluniprops. 📚
Use of the "Is" Prefix ⭐
For backward compatibility (with ancient Perl 5.6, grandmacore fr fr), all properties writable without using the compound form mentioned so far may have Is or Is_ prepended to their name. So \P{Is_Lu}, for example, is equal to \P{Lu}, and \p{IsScript:Arabic} is equal to \p{Arabic}. cute, right?
Blocks 🧱
In addition to scripts, Unicode also defines blocks of characters. The difference between scripts and blocks is that the concept of scripts is closer to natural languages, while the concept of blocks is more of an artificial grouping based on groups of Unicode characters with consecutive ordinal values. 😌
For example, the Basic Latin block is all the characters whose ordinals are between 0 and 127, inclusive; in other words, the ASCII characters. The Latin script contains some letters from this as well as several other blocks, like Latin-1 Supplement, Latin Extended-A, etc., but it does not contain all the characters from those blocks. It does not, for example, contain the digits 0-9, because those digits are shared across many scripts, and hence are in the Common script. 🔍
For more about scripts versus blocks, see UAX#24 "Unicode Script Property": <https://www.unicode.org/reports/tr24>
The Script_Extensions or Script properties are likely to be the ones you want to use when processing natural language; the Block property may occasionally be useful in working with the nuts and bolts of Unicode. 🔧
Block names are matched in the compound form, like \p{Block: Arrows} or \p{Blk=Hebrew}. Unlike most other properties, only a few block names have a Unicode-defined short name.
Perl also defines single form synonyms for the block property in cases where these do not conflict with something else. But don't use any of these, because they are unstable. 😭 Since these are Perl extensions, they are subordinate to official Unicode property names; Unicode doesn't know nor care about Perl's extensions. It may happen that a name that currently means the Perl extension will later be changed without warning to mean a different Unicode property in a future version of the perl interpreter that uses a later Unicode release, and your code would no longer work. 💀
The extensions are mentioned here for completeness, so let's go: take the block name and prefix it with one of:
In— for example\p{Blk=Arrows}can currently be written as\p{In_Arrows}- sometimes
Is— like\p{Is_Arrows} - or sometimes no prefix at all — like
\p{Arrows}
As of this writing (Unicode 9.0) there are no conflicts with using the In_ prefix, but there are plenty with the other two forms. For example, \p{Is_Hebrew} and \p{Hebrew} mean \p{Script_Extensions=Hebrew} which is NOT the same thing as \p{Blk=Hebrew}. 🙈
Our advice used to be to use the In_ prefix as a single form way of specifying a block. But Unicode 8.0 added properties whose names begin with In, and it's now clear that it's only luck that's so far prevented a conflict. Using In is only marginally less typing than Blk:, and the latter's meaning is clearer anyway, and guaranteed to never conflict.
So don't take chances bestie. Use \p{Blk=foo} for new code. And be sure that block is what you really really want to do. In most cases scripts are what you want instead. periodt.
A complete list of blocks is in perluniprops. 📖
Other Properties 🍓
There are many more properties than the very basic ones described here. A complete list is in perluniprops.
Unicode defines all its properties in the compound form, so all single-form properties are Perl extensions. Most of these are just synonyms for the Unicode ones, but some are genuine extensions, including several that are in the compound form. And quite a few of these are actually recommended by Unicode (in <https://www.unicode.org/reports/tr18>). ✨
This section gives some details on all extensions that aren't just synonyms for compound-form Unicode properties (for those properties, you'll have to refer to the Unicode Standard <https://www.unicode.org/reports/tr44>. ) — okay notably the original sentence has a stray unclosed paren, we're keeping the vibe faithful but the point stands: go read the Unicode Standard.
\p{All}— This matches every possible code point. It is equivalent toqr/./s. And unlike all the other non-user-defined\p{}property matches, no warning is ever generated if this is property is matched against a non-Unicode code point (see "Beyond Unicode code points" below). 💀✨\p{Alnum}— This matches any\p{Alphabetic}or\p{Decimal_Number}character.\p{Any}— This matches any of the 1_114_112 Unicode code points. It is a synonym for\p{Unicode}. 👑
"\p{ASCII}" ✨
Matches any of the 128 characters in the US-ASCII character set. Which is, btw, a whole subset of Unicode. iconic behavior fr.
"\p{Assigned}" 💖
Matches any assigned code point. Meaning: any code point whose general category is not "Unassigned" — or, same thing, not "Cn".
"\p{Blank}" 🎀
Literally the same as "\h" and "\p{HorizSpace}": a character that changes the spacing horizontally. side to side. not up and down. ok bestie.
"\p{Decomposition_Type: Non_Canonical}" (short: "\p{Dt=NonCanon}") 🦋
Matches a character that has any of the non-canonical decomposition types. Canonical decompositions got introduced up in the "Extended Grapheme Clusters (Logical characters)" section, so go read that. But a lot more characters have a different kind of decomposition, generically called "compatible" decompositions, or "non-canonical". And, key point: the sequences that form these decompositions are not considered canonically equivalent to the pre-composed character. no cap.
Example time: the "SUPERSCRIPT ONE". It's kinda like a regular digit 1, but not exactly — its decomposition into the digit 1 is called a "compatible" decomposition, specifically a "super" (for "superscript") decomposition. There are several of these compatibility decompositions (see <https://www.unicode.org/reports/tr44>). And "\p{Dt: Non_Canon}" is a Perl extension that uses just one name to refer to the union of all of them. slay.
Most Unicode characters don't have a decomposition at all, so their decomposition type is "None". Which means "Non_Canonical" is equivalent to:
qr/(?[ \P{DT=Canonical} - \p{DT=None} ])/(Little aside: one of the non-canonical decompositions is named"compat", which honestly could've been named"miscellaneous"instead. It's just the stuff Unicode couldn't figure out a better generic name for. we've all been there 💀)
"\p{Graph}" 🔥
Matches any character that is graphic. Theoretically that means a character that, on a printer, would cause ink to be used. cute.
"\p{HorizSpace}" 💅
Same as "\h" and "\p{Blank}": a character that changes the spacing horizontally.
"\p{In=*}" ⭐
This is a synonym for "\p{Present_In=*}". just so you know.
"\p{PerlSpace}" 🫶
Same as "\s", but restricted to ASCII — namely "[ \f\n\r\t]", and starting in Perl v5.18, a vertical tab too.
Mnemonic: Perl's (original) space.
"\p{PerlWord}" 🌸
Same as "\w", restricted to ASCII, namely "[A-Za-z0-9_]".
Mnemonic: Perl's (original) word.
"\p{Posix...}" 💜
There are several of these, and they're equivalents — using the "\p{}" notation — for Posix classes. They're described in "POSIX Character Classes" in perlrecharclass. go look, bestie.
"\p{Present_In: *}" (short: "\p{In=*}") 👑
This property is your girl when you need to know in what Unicode version(s) a character exists.
The "*" up there stands for some Unicode version number, like 1.1 or 12.0; or "*" can be "Unassigned". This property matches the code points whose final disposition has been settled as of the Unicode release given by that version number. And "\p{Present_In: Unassigned}" matches those code points whose meaning has yet to be assigned. still pending fr.
For example: "U+0041" "LATIN CAPITAL LETTER A" was present in the very first Unicode release available, which is 1.1, so this property is true for all valid "*" versions. On the other hand, "U+1EFF" wasn't assigned until version 5.1, when it became "LATIN SMALL LETTER Y WITH LOOP", so the only "*" values that would match it are 5.1, 5.2, and later.
Unicode furnishes the "Age" property, and this one is derived from it. The problem with Age is that a strict interpretation of it (which Perl takes, obviously) has it matching the precise release a code point's meaning is introduced in. So "U+0041" would match only 1.1; and "U+1EFF" only 5.1. And that's… usually not what you want. 🙈
Some non-Perl implementations of the Age property may change its meaning to be the same as Perl's "Present_In" property. Just be aware of that, ok?
Another confusion with both these properties: the definition isn't that the code point has been assigned — it's that the meaning of the code point has been determined. Why? Because 66 code points will always be unassigned, so the "Age" for them is the Unicode version in which the decision to make them so was made. Example: "U+FDD0" is to be permanently unassigned to a character, and the decision to do that was made in version 3.1. So "\p{Age=3.1}" matches this character, and so do "\p{Present_In: 3.1}" and up. ✨
"\p{Print}" 🎀
Matches any character that is graphical or blank — except controls. periodt.
"\p{SpacePerl}" 🦋
Same as "\s", but including beyond ASCII.
Mnemonic: Space, as modified by Perl. (Heads up: it doesn't include the vertical tab until v5.18 — even though both the Posix standard and Unicode consider it white space. petty drama, honestly.)
"\p{Title}" and "\p{Titlecase}" 👑
Under case-sensitive matching, both of these match the same code points as "\p{General Category=Titlecase_Letter}" ("\p{gc=lt}"). The difference shows up under /i caseless matching: these match the same as "\p{Cased}", whereas "\p{gc=lt}" matches "\p{Cased_Letter". (yes that's exactly what the original says, no I'm not fixing it, moving on 💅)
"\p{Unicode}" 🌈
Matches any of the 1_114_112 Unicode code points. Same as "\p{Any}". the whole squad.
"\p{VertSpace}" 💜
Same as "\v": a character that changes the spacing vertically. up and down, bestie.
"\p{Word}" 💖
Same as "\w", including over 100_000 characters beyond ASCII. that's a lot of letters.
"\p{XPosix...}" ⭐
There are several of these — they're the standard Posix classes extended to the full Unicode range. Described in "POSIX Character Classes" in perlrecharclass.
🌈 Comparison of "\N{...}" and "\p{name=...}" ✨
Starting in Perl 5.32, you can specify a character by its name in regular expression patterns using "\p{name=...}". This is in addition to the longstanding method of using "\N{...}". So here's the tea on how these two differ:
can interpolate—\N{...}: only witheval;\p{Name=...}: yes [1]custom names—\N{...}: yes;\p{Name=...}: no [2]name aliases—\N{...}: yes;\p{Name=...}: yes [3]named sequences—\N{...}: yes;\p{Name=...}: yes [4]name value parsing—\N{...}: exact;\p{Name=...}: Unicode loose [5]
Let's break those footnotes down:
- The whole interpolation thing means you can do something like
qr/\p{na=latin capital letter $which}/and specify $which somewhere else. multitasking queen.
- You can create your own names for characters, and override official ones, when using
"\N{...}". See "CUSTOM ALIASES" in charnames. - Some characters have multiple names. synonyms, basically. they contain multitudes.
- Some particular sequences of characters are given a single name, in addition to their individual ones.
- Exact name value matching means you have to specify case, hyphens, underscores, and spaces precisely in the name you want. Loose matching follows the Unicode rules <https://www.unicode.org/reports/tr44/tr44-24.html#UAX44-LM2>, where those are mostly irrelevant. Except for a few outlier character names, these are the same rules already used for any other
"\p{...}"property. so it's not even new behavior, bestie.
🍓 Wildcards in Property Values 🍓
Starting in Perl 5.30, you can do something like this:
qr!\p{numeric_value=/\A[0-5]\z/}!or, by abbreviating and adding /x:
qr! \p{nv= /(?x) \A [0-5] \z / }!This matches all code points whose numeric value is one of 0, 1, 2, 3, 4, or 5. And this particular example could instead have been written as
qr! \A [ \p{nv=0}\p{nv=1}\p{nv=2}\p{nv=3}\p{nv=4}\p{nv=5} ] \z !xxin earlier perls — so in this case the feature just makes things easier and shorter to write. 🌸
If we hadn't included the "\A" and "\z", these would've matched things like "1/2", because that contains a 1 (as well as a 2). 💀 As written, it matches things like subscripts that have these numeric values. And if we only wanted the decimal digits with those numeric values, we could say:
qr! (?[ \d & \p{nv=/[0-5]/ ]) }!xThe "\d" gets rid of needing to anchor the pattern, since it forces the result to only match "[0-9]", and the "[0-5]" further restricts it.
The text in the above examples enclosed between the / characters can be just about any regular expression. It's independent of the main pattern, so it doesn't share any capturing groups, etc. The delimiters for it must be ASCII punctuation, but it may NOT be delimited by "{", nor "}" nor contain a literal "}", since that delimits the end of the enclosing "\p{}". Like any pattern, certain other delimiters are terminated by their mirror images — these are "(", "[", and "<". And if the delimiter is any of "-", "_", "+", or "\", or is the same delimiter as is used for the enclosing pattern, it must be preceded by a backslash escape, both fore and aft. yes both. don't skip one. 😭
Beware of using"$"to indicate matching the end of the string! It can too easily be interpreted as being a punctuation variable, like$/. terrifying. be careful.
No modifiers may follow the final delimiter. Instead, use "(?adlupimnsx-imnsx)" in perlre and/or "(?adluimnsx-imnsx:pattern)" in perlre to specify modifiers. However — certain modifiers are illegal in your wildcard subpattern. The only character set modifier you can specify is "/aa"; any other character set, plus "-m", "p", and "s" are all illegal. Specifying modifiers like "qr/.../gc" that aren't legal in the "(?...)" notation normally raises a warning, but with wildcard subpatterns, their use is an error. And the "m" modifier is ineffective — everything that matches will be a single line.
By default, your pattern is matched case-insensitively, as if "/i" had been specified. You can change this by saying "(?-i)" in your pattern.
There are also certain operations that are illegal. You can't nest "\p{...}" and "\P{...}" calls within a wildcard subpattern, and "\G" doesn't make sense, so it's prohibited too. sorry bestie.
And the "*" quantifier (or its equivalent "(0,}") is illegal. no infinite loops for you.
This feature is not available when the left-hand side is prefixed by "Is_", nor for any form that is marked as "Discouraged" in "Discouraged" in perluniprops. 🚫
This is an experimental feature that's been added to begin implementing <https://www.unicode.org/reports/tr18/#Wildcard_Properties>. Using it will raise a (default-on) warning in the "experimental::uniprop_wildcards" category. And — important — we reserve the right to change its operation as we gain experience. so don't build your whole personality on it. 💀
So like, your subpattern can literally be anything you want, but for it to actually do something useful, it should match when it's called with either or both of:
- a) the full name of the property value with underscores (and/or spaces in the
Blockproperty) and some stuff in uppercase - b) the property value all lowercase, with spaces and underscores squeezed out
For example:
qr!\p{Blk=/Old I.*/}!
qr!\p{Blk=/oldi.*/}!both of those match the same things. 💅
Another example, and this one shows that inside \p{...} you don't need /x to have spaces:
qr!\p{scx= /Hebrew|Greek/ }!ngl, to be safe we should have anchored the above example so it doesn't accidentally match something like "Hebrew_Braille" — but there are no script names like that so far, so we're chilling. 🧚 A warning gets issued if none of the legal values for a property are matched by your pattern. And it's likely a future release will raise a warning if your pattern ends up causing every possible code point to match. 😭
Starting in 5.32, the Name, Name Aliases, and Named Sequences properties are allowed to be matched. They're treated as one single combination property, just like \N{} has been forever. Loose matching doesn't work exactly the same way for these as it does for the values of other properties. The rules live here: <https://www.unicode.org/reports/tr44/tr44-24.html#UAX44-LM2>. As a result, Perl doesn't do loose matching for you like it does for other properties — you're on your own, bestie. All letters in names are uppercase, but you can add (?i) to your subpattern to ignore case. If you're not sure where a blank goes, you can drop ? in your subpattern. No character name contains an underscore, so don't even bother trying to match one. Hyphens are particularly cursed; go read that link. But note that, as of Unicode 13.0, the only script in modern usage with weirdness here is Tibetan; plus the two Korean characters U+116C HANGUL JUNGSEONG OE and U+1180 HANGUL JUNGSEONG O-E. And Unicode makes zero promises about not adding more hyphen-problematic names later. 💀
Using wildcards on these is resource intensive, because you're checking against hundreds of thousands of legal names.
An example of using Name property wildcards:
qr!\p{name=/(SMILING|GRINNING) FACE/}!Another one:
qr/(?[ \p{name=\/CJK\/} - \p{ideographic} ])/which is the 200-ish (as of Unicode 13.0) CJK characters that aren't ideographs. ✨
There are certain properties that wildcard subpatterns don't currently work with. These are:
Bidi Mirroring Glyph
Bidi Paired Bracket
Case Folding
Decomposition Mapping
Equivalent Unified Ideograph
Lowercase Mapping
NFKC Case Fold
Titlecase Mapping
Uppercase MappingAnd the @unicode_property@ form is not implemented either. 🙈
Here's a complete example of matching IPV4 internet protocol addresses in any (single) script:
no warnings 'experimental::uniprop_wildcards';
# Can match a substring, so this intermediate regex needs to have
# context or anchoring in its final use. Using nt=de yields decimal
# digits. When specifying a subset of these, we must include \d to
# prevent things like U+00B2 SUPERSCRIPT TWO from matching
my $zero_through_255 =
qr/ \b (*sr: # All from same script
(?[ \p{nv=0} & \d ])* # Optional leading zeros
( # Then one of:
\d{1,2} # 0 - 99
| (?[ \p{nv=1} & \d ]) \d{2} # 100 - 199
| (?[ \p{nv=2} & \d ])
( (?[ \p{nv=:[0-4]:} & \d ]) \d # 200 - 249
| (?[ \p{nv=5} & \d ])
(?[ \p{nv=:[0-5]:} & \d ]) # 250 - 255
)
)
)
\b
/x;
my $ipv4 = qr/ \A (*sr: $zero_through_255
(?: [.] $zero_through_255 ) {3}
)
\z
/x;🌸 User-Defined Character Properties 🌸
okay so you can define your own binary character properties by defining subroutines whose names begin with "In" or "Is". (The regex sets feature (?[ ]) in perlre gives you an alternative that allows more complex definitions, if you wanna go fancier.) The subroutines can be defined in any package. Like all official Unicode properties, their names must be ASCII only, and they override any Unicode properties using the same names. You can use your user-defined properties in the regular expression \p{} and \P{} constructs; and if you're pulling a user-defined property from a package other than the one you're currently in, you must specify its package in the \p{} or \P{} construct.
# assuming property IsForeign defined in Lang::
package main; # property package name required
if ($txt =~ /\p{Lang::IsForeign}+/) { ... }
package Lang; # property package name not required
if ($txt =~ /\p{IsForeign}+/) { ... }The subroutines get passed a single parameter, which is 0 if case-sensitive matching is in effect and non-zero if caseless matching is in effect. The subroutine may return different values depending on the value of the flag. But the subroutine is never called more than once for each flag value (zero vs non-zero). The return value is saved and used instead of calling the sub ever again. If the sub is defined at the time the pattern is compiled, it gets called then; if not, it gets called the first time its value (for that flag) is needed during execution. 👑
Note that if the regular expression is tainted, then Perl will die rather than calling the subroutine when the name of the subroutine is determined by the tainted data.
The subroutines must return a specially-formatted string, with one or more newline-separated lines. Each line must be one of the following:
- A single hexadecimal number denoting a code point to include.
- Two hexadecimal numbers separated by horizontal whitespace (space or tabular characters) denoting a range of code points to include. The second number must not be smaller than the first.
- Something to include, prefixed by
"+": a built-in character property (prefixed by"utf8::") or a fully qualified (including package name) user-defined character property, to represent all the characters in that property; two hexadecimal code points for a range; or a single hexadecimal code point. - Something to exclude, prefixed by
"-": an existing character property (prefixed by"utf8::") or a fully qualified (including package name) user-defined character property, to represent all the characters in that property; two hexadecimal code points for a range; or a single hexadecimal code point. - Something to negate, prefixed
"!": an existing character property (prefixed by"utf8::") or a fully qualified (including package name) user-defined character property, to represent all the characters in that property; two hexadecimal code points for a range; or a single hexadecimal code point. - Something to intersect with, prefixed by
"&": an existing character property (prefixed by"utf8::") or a fully qualified (including package name) user-defined character property, for all the characters except the characters in the property; two hexadecimal code points for a range; or a single hexadecimal code point.
For example, to define a property that covers both the Japanese syllabaries (hiragana and katakana), you can define:
sub InKana {
return <<END;
3040\t309F
30A0\t30FF
END
}Imagine the here-doc end marker is at the beginning of the line. Now you can use \p{InKana} and \P{InKana}. 🎀
You could also have used the existing block property names:
sub InKana {
return <<'END';
+utf8::InHiragana
+utf8::InKatakana
END
}Suppose you wanted to match only the allocated characters, not the raw block ranges — like, you wanna remove the unassigned characters:
sub InKana {
return <<'END';
+utf8::InHiragana
+utf8::InKatakana
-utf8::IsCn
END
}The negation is useful for defining (surprise!) negated classes.
sub InNotKana {
return <<'END';
!utf8::InHiragana
-utf8::InKatakana
+utf8::IsCn
END
}This will match all non-Unicode code points, since every single one of them is not in Kana. You can use intersection to exclude these if you want, like this modified example shows:
sub InNotKana {
return <<'END';
!utf8::InHiragana
-utf8::InKatakana
+utf8::IsCn
&utf8::Any
END
}&utf8::Any must be the last line in the definition. periodt.
Intersection is used generally for getting the common characters matched by two (or more) classes. It's important to remember not to use "&" for the first set — that would be intersecting with nothing, resulting in an empty set. (Similarly, using "-" for the first set does nothing.) 💀
Unlike non-user-defined \p{} property matches, no warning is ever generated if these properties are matched against a non-Unicode code point (see "Beyond Unicode code points" below).
💜 User-Defined Case Mappings (for serious hackers only) 💜
This feature has been removed as of Perl 5.16. The CPAN module "Unicode::Casing" provides better functionality without the drawbacks that this feature had. If you're on a Perl earlier than 5.16, this feature was most fully documented in the 5.14 version of this pod: <http://perldoc.perl.org/5.14.0/perlunicode.html#User-Defined-Case-Mappings-%28for-serious-hackers-only%29>
🦋 Character Encodings for Input and Output 🦋
See Encode.
⭐ Unicode Regular Expression Support Level ⭐
The following list of Unicode supported features for regular expressions describes all features currently directly supported by core Perl. The references to "Level N" and the section numbers refer to UTS#18 "Unicode Regular Expressions" <https://www.unicode.org/reports/tr18>, version 18, October 2016.
Level 1 - Basic Unicode Support
RL1.1 Hex Notation - Done [1]
RL1.2 Properties - Done [2]
RL1.2a Compatibility Properties - Done [3]
RL1.3 Subtraction and Intersection - Done [4]
RL1.4 Simple Word Boundaries - Done [5]
RL1.5 Simple Loose Matches - Done [6]
RL1.6 Line Boundaries - Partial [7]
RL1.7 Supplementary Code Points - Done [8][1]— the literal ways to spell a code point:\N{U+...}and\x{...}.[2]—\p{...}and\P{...}. This requirement is only asking for a minimal list of properties, and bestie, Perl has those. For the rest of them, run to R2.7. 👑[3]— Perl's got\d,\D,\s,\S,\w,\W,\X,[:prop:]and[:^prop:], plus literally every property listed over athttps://www.unicode.org/reports/tr18/#Compatibility_Properties. All of them are explained above in "Other Properties".[4]— the regex sets feature(?[...]), which showed up in v5.18, handles this one. Go peep(?[ ])in perlre. ✨[5]—\band\Bclear most of this requirement but not all of it, no cap.\b{wb}and\B{wb}clear all of it, and they also handle the stricter R2.3.[6]— heads up, bestie: Perl does Full case-folding in matching, not Simple.
So like, U+1F88 is equivalent to U+1F00 U+03B9, instead of just U+1F80. This whole difference only really matters for certain Greek capital letters with certain modifiers: Full case-folding takes the letter apart, while Simple case-folding would just map it onto a single character.
[7]— the reason this one is only partially implemented is that Perl hasqr/\b{lb}/andUnicode::LineBreak, and both of those are conformant with UAX#14 "Unicode Line Breaking Algorithm"https://www.unicode.org/reports/tr14. The regex construct gives you default behavior; the heavier-weight module gives you customizable line breaking.
BUT. Perl treats \n as the start- and end-line delimiter, whereas Unicode says a whole bunch more characters should be read that way.
The list is:
VT U+000B (\v in C)
FF U+000C (\f)
CR U+000D (\r)
NEL U+0085
LS U+2028
PS U+2029^ and $ in regular expression patterns are supposed to match all of these, but they don't. 😭 These characters also don't affect <>, $. and script line numbers, even though they should.
Also, lines should not be split within CRLF (meaning there is no empty line between \r and \n). For CRLF, try the :crlf layer (see PerlIO).
[8]— UTF-8/UTF-EBDDIC as used in Perl allows not onlyU+10000toU+10FFFFbut also beyondU+10FFFF. iconic tbh.
Level 2 - Extended Unicode Support 🦋
- RL2.1 — Canonical Equivalents — Retracted by Unicode
[9] - RL2.2 — Extended Grapheme Clusters and Character Classes with Strings — Partial
[10] - RL2.3 — Default Word Boundaries — Done
[11] - RL2.4 — Default Case Conversion — Done
- RL2.5 — Name Properties — Done
- RL2.6 — Wildcards in Property Values — Partial
[12] - RL2.7 — Full Properties — Partial
[13] - RL2.8 — Optional Properties — Partial
[14] [9]— Unicode rewrote this part of UTS#18 to say that getting canonical equivalence (see UAX#15 "Unicode Normalization Forms"https://www.unicode.org/reports/tr15) is basically something the programmer handles. So use NFD to write both your regular expressions and the text you're matching them against (you can use Unicode::Normalize for that). 💅[10]— Perl has\Xand\b{gcb}. Unicode retracted their "Grapheme Cluster Mode" and recently added string properties, which Perl does not yet support.[11]— see UAX#29 "Unicode Text Segmentation"https://www.unicode.org/reports/tr29,[12]— see "Wildcards in Property Values" above.[13]— Perl supports all the properties in the Unicode Character Database (UCD). It does not yet support the listed properties that come from other Unicode sources.[14]— the only optional property that Perl supports is Named Sequence. None of these properties are in the UCD.
Level 3 - Tailored Support
This has been retracted by Unicode. that's it, that's the whole level. 💀
Unicode Encodings 🌈
Unicode characters get assigned to code points, and code points are just abstract numbers. To actually use those numbers you need encodings. Several of them, obviously.
UTF-8 💖
UTF-8 is a variable-length (1 to 4 bytes), byte-order independent encoding. In most of Perl's documentation — including elsewhere in this very document — the term "UTF-8" also means "UTF-EBCDIC". But in this section, "UTF-8" refers only to the encoding used on ASCII platforms. It's a superset of 7-bit US-ASCII, so anything encoded in ASCII has the identical representation when it's encoded in UTF-8.
The following table is from Unicode 3.2.
Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
U+0000..U+007F 00..7F
U+0080..U+07FF * C2..DF 80..BF
U+0800..U+0FFF E0 * A0..BF 80..BF
U+1000..U+CFFF E1..EC 80..BF 80..BF
U+D000..U+D7FF ED 80..9F 80..BF
U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++
U+E000..U+FFFF EE..EF 80..BF 80..BF
U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF
U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
U+100000..U+10FFFF F4 80..8F 80..BF 80..BFNotice the gaps marked by * before several of those byte entries up there. Those gaps exist because legal UTF-8 avoids non-shortest encodings: technically you could UTF-8-encode a single code point in different ways, but that's explicitly forbidden, and the shortest possible encoding should always be used (and that is exactly what Perl does). periodt.
Another way to look at it is via bits:
Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
0aaaaaaa 0aaaaaaa
00000bbbbbaaaaaa 110bbbbb 10aaaaaa
ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaaSee how the continuation bytes all begin with 10, and the leading bits of the start byte tell you how many bytes there are in the encoded character? chef's kiss. 🔥
The original UTF-8 specification allowed up to 6 bytes, so it could encode numbers up to 0x7FFF_FFFF. Perl continues to allow those, and has extended that up to 13 bytes to encode code points up to what can fit in a 64-bit word. However, Perl will warn if you output any of these as being non-portable; and under strict UTF-8 input protocols, they are forbidden. On top of that, it is now illegal to use a code point larger than what a signed integer variable on your system can hold. On 32-bit ASCII systems, that means 0x7FFF_FFFF is the legal maximum (much higher on 64-bit systems).
fr fr do not sleep on that portability warning, bestie. 🫶
UTF-EBCDIC
Like UTF-8, but EBCDIC-safe, in the way that UTF-8 is ASCII-safe. That means all the basic characters (which includes all of those that have ASCII equivalents — like A, 0, %, etc.) are the same in both EBCDIC and UTF-EBCDIC.
UTF-EBCDIC is used on EBCDIC platforms. It generally requires more bytes to represent a given code point than UTF-8 does; the largest Unicode code points take 5 bytes to represent (instead of 4 in UTF-8), and, extended for 64-bit words, it uses 14 bytes instead of 13 bytes in UTF-8.
UTF-16, UTF-16BE, UTF-16LE, Surrogates, and BOM's (Byte Order Marks) 🎀
The following items are mostly for reference and general Unicode knowledge — Perl doesn't use these constructs internally.
Like UTF-8, UTF-16 is a variable-width encoding, but where UTF-8 uses 8-bit code units, UTF-16 uses 16-bit code units. All code points occupy either 2 or 4 bytes in UTF-16: code points U+0000..U+FFFF are stored in a single 16-bit unit, and code points U+10000..U+10FFFF in two 16-bit units. That second case is using surrogates, with the first 16-bit unit being the high surrogate and the second being the low surrogate.
Surrogates are code points set aside to encode the U+10000..U+10FFFF range of Unicode code points in pairs of 16-bit units. The high surrogates are the range U+D800..U+DBFF and the low surrogates are the range U+DC00..U+DFFF. The surrogate encoding is
$hi = ($uni - 0x10000) / 0x400 + 0xD800;
$lo = ($uni - 0x10000) % 0x400 + 0xDC00;and the decoding is
$uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);Because of all that 16-bitness, UTF-16 is byte-order dependent. UTF-16 itself can be used for in-memory computations, but if you need storage or transfer, you have to pick either UTF-16BE (big-endian) or UTF-16LE (little-endian).
Which introduces another problem: what if you just know your data is UTF-16, but you don't know which endianness it is? Byte Order Marks, or BOM's, are the solution. A special character has been reserved in Unicode to function as a byte order marker: the character with the code point U+FEFF is the BOM.
The trick is that if you read a BOM, you'll know the byte order — because if it was written on a big-endian platform you'll read the bytes 0xFE 0xFF, but if it was written on a little-endian platform you'll read the bytes 0xFF 0xFE. (And if the originating platform was writing in ASCII platform UTF-8, you'll read the bytes 0xEF 0xBB 0xBF.)
The way this trick works is that the character with the code point U+FFFE is not supposed to be in input streams, so the sequence of bytes 0xFF 0xFE is unambiguously "BOM, represented in little-endian format" and cannot be "U+FFFE, represented in big-endian format". the way that actually works?? obsessed. 😌
Surrogates have no meaning in Unicode outside their use in pairs to represent other code points. However, Perl allows them to be represented individually internally, for example by saying chr(0xD801), so that all code points — not just those valid for open interchange — are representable. Unicode does define semantics for them, such as their General_Category is Cs. But because their use is somewhat dangerous, Perl will warn (using the warning category surrogate, which is a sub-category of utf8) if an attempt is made to do things like take the lower case of one, or match case-insensitively, or to output them. (But don't try this on Perls before 5.14.) 🙈
UTF-32, UTF-32BE, UTF-32LE
The UTF-32 family is pretty much like the UTF-16 family, except that the units are 32-bit, and therefore the surrogate scheme is not needed. UTF-32 is a fixed-width encoding. The BOM signatures are 0x00 0x00 0xFE 0xFF for BE and 0xFF 0xFE 0x00 0x00 for LE.
UCS-2, UCS-4 🦋
- Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit encoding. Unlike UTF-16, UCS-2 is not extensible beyond
U+FFFF, because it does not use surrogates. UCS-4 is a 32-bit encoding, functionally identical to UTF-32 (the difference being that UCS-4 forbids neither surrogates nor code points larger than0x10_FFFF). UTF-7— a seven-bit safe (non-eight-bit) encoding, which is useful if the transport or storage is not eight-bit safe. Defined by RFC 2152.
💀 Noncharacter code points 💀
okay so this is a whole thing and it's lowkey fascinating. Unicode set aside 66 code points as "noncharacter code points". All of them have the "Unassigned" (Cn) General_Category, and no character will ever be assigned to any of them — like, ever, it's not happening, periodt. They are the 32 code points between U+FDD0 and U+FDEF inclusive, plus these 34 code points:
U+FFFE U+FFFF
U+1FFFE U+1FFFF
U+2FFFE U+2FFFF
...
U+EFFFE U+EFFFF
U+FFFFE U+FFFFF
U+10FFFE U+10FFFFNow here's the glow-up story. Until Unicode 7.0, the noncharacters were "forbidden for use in open interchange of Unicode text data", so code that was chewing through those streams could use these code points as sentinels that could be mixed right in with actual character data, and would always be tellable-apart from that data. (Just so you know: the emphasis in the para above and in the next one was added in this document, not by the Unicode people. 💅)
Then Unicode 7.0 was like "mm, actually" and reworded it to "not recommended for use in open interchange of Unicode text data". And the 7.0 Standard goes on to say:
"If a noncharacter is received in open interchange, an application is not required to interpret it in any way. It is good practice, however, to recognize it as a noncharacter and to take appropriate action, such as replacing it withU+FFFDreplacement character, to indicate the problem in the text. It is not recommended to simply delete noncharacter code points from such text, because of the potential security issues caused by deleting uninterpreted characters. (See conformance clause C7 in Section 3.2, Conformance Requirements, and Unicode Technical Report #36, "Unicode Security Considerations"https://www.unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences)."
Why the switch-up, you ask? Because it turned out that a bunch of commercial tools — editors, source code control, that whole vibe — had been written so they would not handle program files that used these code points, which basically precluded their use almost entirely!! 😭 And that was never the intent. They've always been meant to be usable within an application, or a cooperating set of applications, at will. slay.
So, real talk, if you're writing code — say an editor — that's supposed to be able to handle any Unicode text data, then you yourself shouldn't be using these code points, and instead you should allow them in the input. If you need sentinels, use something that isn't legal Unicode. For UTF-8 data you can use the bytes 0xC0 and 0xC1 as sentinels, since they never appear in well-formed UTF-8. (There are equivalents for UTF-EBCDIC.) You can also stash your Unicode code points in integer variables and use negative values as sentinels. easy.
If you're not writing that kind of tool, then whether you accept noncharacters as input is up to you (though the Standard recommends you not 🫣). If you do strict input stream checking with Perl, these code points continue to be forbidden. That's for backward compatibility — otherwise potential security holes could open up, because an unsuspecting application written assuming the noncharacters would be filtered out before reaching it could now, without warning, start getting them. To do strict checking, you can use the layer :encoding('UTF-8').
Perl continues to warn (using the warning category nonchar, which is a sub-category of utf8) if an attempt is made to output noncharacters. 👮♀️
🌌 Beyond Unicode code points 🌌
The maximum Unicode code point is U+10FFFF, and Unicode only defines operations on code points up through that. But Perl? Perl works on code points up to the maximum permissible signed number available on the platform. iconic. However, Perl will not accept these from input streams unless lax rules are being used, and will warn (using the warning category non_unicode, which is a sub-category of utf8) if any are output.
Since Unicode rules are not defined on these code points, if a Unicode-defined operation is done on them, Perl uses what we believe are sensible rules, while generally warning, using the non_unicode category. For example, uc("\x{11_0000}") will generate such a warning, returning the input parameter as its result, since Perl defines the uppercase of every non-Unicode code point to be the code point itself. (All the case changing operations, not just uppercasing, work this way.)
The situation with matching Unicode properties in regular expressions — the \p{} and \P{} constructs — against these code points is not as clear cut, and how these are handled has changed as we've gained experience. 🧠
One possibility is to treat any match against these code points as undefined. But since Perl doesn't have the concept of a match being undefined, it converts this to failing or FALSE. This is almost, but not quite, what Perl did from v5.14 (when use of these code points became generally reliable) through v5.18. The difference is that Perl treated all \p{} matches as failing, but all \P{} matches as succeeding.
And that leads to some truly confusing, "wait, what 😭" results:
chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Failed on <= v5.18
chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Failed! on <= v5.18Like, it treated both matches as undefined, and converted that to false (raising a warning on each). The first case is the expected result, but the second is probably counterintuitive: "How could both be false when they are complements?" 🙈 Another problem was that the implementation optimized many Unicode property matches down to already existing simpler, faster operations, which don't raise the warning. And we chose not to forgo those optimizations — they help the vast majority of matches — just to generate a warning for the unlikely event that an above-Unicode code point is being matched against.
So, because of all that mess, starting in v5.20, what Perl does is treat non-Unicode code points as just typical unassigned Unicode characters, and matches accordingly. (Heads up: Unicode does have atypical unassigned code points. For instance it has noncharacter code points, and ones that, when they do get assigned, are destined to be written Right-to-left, like Arabic and Hebrew. Perl assumes that no non-Unicode code point has any atypical properties.)
Perl, in most cases, will raise a warning when matching an above-Unicode code point against a Unicode property when the result is TRUE for \p{}, and FALSE for \P{}. For example:
chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails, no warning
chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Succeeds, with warningIn both those examples the character being matched is non-Unicode, so Unicode doesn't define how it should match. It clearly isn't an ASCII hex digit, so the first example clearly should fail — and so it does, with no warning. But it's arguable that the second example should have an undefined, hence FALSE, result. So a warning gets raised for it.
So the warning now fires for way fewer cases than in earlier Perls, and only when what the result is could be arguable. And it turns out that none of the optimizations made by Perl (or ever likely to be made) cause the warning to be skipped — so it solves both problems of Perl's earlier approach. The most commonly used property affected by this change is \p{Unassigned}, which is a short form for \p{General_Category=Unassigned}. Starting in v5.20, all non-Unicode code points are considered Unassigned. In earlier releases the matches failed because the result was considered undefined.
The only place where the warning isn't raised when it might ought to have been is if optimizations cause the whole pattern match to not even be attempted. For example, Perl may figure out that for a string to match a certain regular expression pattern, the string has to contain the substring foobar. Before attempting the match, Perl may look for that substring, and if it's not found, immediately fail the match without actually trying it — so no warning gets generated even if the string contains an above-Unicode code point. 🤷♀️
This behavior is more "Do what I mean" than in earlier Perls for most applications. But it catches fewer issues for code that needs to be strictly Unicode compliant. So there's an additional mode of operation available to accommodate such code. This mode is enabled if a regular expression pattern is compiled within the lexical scope where the non_unicode warning class has been made fatal, say by:
use warnings FATAL => "non_unicode"(see warnings). In this mode of operation, Perl will raise the warning for all matches against a non-Unicode code point (not just the arguable ones), and it skips the optimizations that might cause the warning to not be output. (It currently still won't warn if the match isn't even attempted, like in the foobar example above.)
So, summary time, bestie 📝: Perl now normally treats non-Unicode code points as typical Unicode unassigned code points for regular expression matches, raising a warning only when it is arguable what the result should be. However, if this warning has been made fatal, it isn't skipped.
And there's exactly one exception to all of this. \p{All} looks like a Unicode property, but it's a Perl extension that is defined to be true for all possible code points, Unicode or not — so no warning is ever generated when matching this against a non-Unicode code point. (Prior to v5.20, it was an exact synonym for \p{Any}, matching code points 0 through 0x10FFFF.)
🔐 Security Implications of Unicode 🔐
The security implications of Unicode are, ngl, quite complicated — as you might expect from something trying to handle all the world's scripts. There's an introduction in Unicode Security Methods https://www.unicode.org/reports/tr39.
Here are a few examples of pitfalls. 🕳️
Confusables
Many characters in Unicode look similar enough to other characters that they could be easily confused with each other. This is true even within the same script — for example English, where the digit 0 and a capital letter O may look like each other. But people who use that script know to look out for that. 👀
Okay so picture this, bestie: in Unicode, a digit in one script can be confusable with a digit that has a completely different numeric value in another script. 😭 A malicious website could absolutely use that to make it look like the price of something is less than what you actually get charged for. (You can use "Script Runs" in perlre to make sure such digits are not being inter-mixed, iykyk.)
This is a general problem with internet addresses, fr fr. The people who give out domain names need to be careful to not give out ones that spoof other ones (examples in "Script Runs" in perlre).
And computer program identifier names can be such that they look like something they're not, and hence could fool a code reviewer, for example. 👀 Script runs on the individual identifiers can catch many of these, but not all. All the letters in the ASCII word "scope" have look-alikes in Cyrillic, though those do not form a real word. Using those Cyrillic letters in that order would almost certainly be an attempt at spoofing. 💀
💀 Malformed text 💀
Successful attacks have been made against websites and databases by passing strings to them that aren't actually legal; the receiver fails to realize this; and performs an action it otherwise wouldn't, based on what it thinks the input meant. Such strings are said to be "malformed" or "illformed".
Vast sums of money have been lost to such attacks. 💸 It became important to not fall for them, which involves detecting malformed text and taking appropriate action (or inaction).
The Unicode REPLACEMENT CHARACTER (U+FFFD) is crucial to detecting and handling these. It has no purpose other than to indicate it is a substitute for something else. It is generally displayed as a white question mark on a dark background that is shaped like a diamond (a rectangle rotated 45 degrees). 🔷
When a malformed string is encountered, the code processing it should substitute the REPLACEMENT CHARACTER for it. There are now strict rules as to what parts get replaced. You should never try to infer what the replaced part was meant to be. To do so could be falling into an attacker's trap. 🙈
Many of the attack vectors were not originally envisioned by Unicode's creators, nor by its implementers, such as Perl. Rules about what were acceptable strings were originally laxer, tightened as the school of hard knocks dictated. 😮💨
Unfortunately, the Perl interpreter's methods of working with Unicode strings were developed when we too were naive about the possibility of attacks. Because of concerns about breaking existing code and continued naivety about the consequences, there has been resistance to changing it, and so our implementation has lagged behind Unicode's requirements and recommendations. But, over the years, various improvements have been added to minimize the issues.
So it is super important to use the latest version of the Perl interpreter when working with Unicode. Not unil Perl v5.44 is it fully hardened against known attack vectors. 🛡️ And who knows what new ideas clever atackers may come up with in the future that we will have to change to counter. (Although no new ones have become known recently that the Unicode Standard wasn't prepared for.) And CPAN modules can easily lag behind the interpreter itself. 😬
Note that finding a REPLACEMENT CHARACTER in your string doesn't necessarily mean there is an attack. It is a perfectly legal input character, for whatever reason. 🫶
One of those reasons is when there is an encoding for which there are Unicode equivalents for most, but not all characters in it. You just use the REPLACEMENT CHARACTER for the missing ones. As long as most of the text is translatable, the results could be intelligible to a human reader. 🧚
This, in fact, was one of the main reasons (besides malformation substitution) for the creation of the REPLACEMENT CHARACTER in the very first version of the Unicode Standard (often abbreviated TUS). Back then, many characters were missing that have since been added (the first release had 40,000 characters; Unicode 16.0 has nearly 300K). ⭐
And, transmission errors where bits get dropped or disk sector failures can also cause malformations. The REPLACEMENT CHARACTER gets substituted, and the results may be legible to a human as long as the error rate is low enough.
Now that Unicode is far more complete, and the odds of finding a character that Unicode doesn't know about are far lower, the primary use of the REPLACEMENT CHARACTER is to substitute for malformations in strings.
When you are programming in pure Perl, you end up relying on the underlying interpreter and modules to handle these kinds of nuances. You are responsible, however, for knowing the encoding(s) needed for your program to interact with the outside world. For example, a common encoding for files is UTF-8. You could use the following to read one: 👇
use PerlIO::encoding;
my $path = "path-to-UTF-8-file";
open my $fh, "<:encoding(UTF-8)", $path
or die "Couldn't open $path: $!";This, behind-the-scenes, uses the Encode module to translate the contents of $path to something Perl can understand. "Encode" knows how to handle a wide variety of encodings. Use this paradigm as well to output to a file; 💾
use PerlIO::encoding;
my $out_path = "path-to-UTF-8-file";
open my $fh, ">:encoding(UTF-8)", $out_path
or die "Couldn't open $out_path: $!";(You can also use "binmode" in perlfunc to change the encoding of an already-open file.)
(There are fewer options to specifying the encoding of arguments passed to your Perl program or to interact with environment variables. See "PERL_UNICODE" in perlrun.)
✨ Skip to the end of this item unless you are writing in XS. ✨
When writing in XS, and manipulating Unicode strings, you need to know more about the internals. "Unicode Support" in perlapi lists the available API elements for working with it. UTF-8 is how Unicode strings are currently stored internally by the interpreter. You need to be using functions in the utf8_to_uv family when parsing a string encoded in UTF-8. This avoids the pitfalls of earlier API functions, whose names contained to_uvchr instead of plain to_uv.
🚫 Illegal code points 🚫
Unicode considers many code points to be illegal, or to be avoided. Perl generally accepts them anyway, once they have passed through any input filters that may try to exclude them. 😅 These have been discussed above (see "Surrogates" under UTF-16 in "Unicode Encodings", "Noncharacter code points", and "Beyond Unicode code points").
If you are writing in XS, the utf8_to_uv in perlapi family of functions has ones that can exclude common varieties of these. In particular, strict_utf8_to_uv, excludes all but the most restrictive set defined by TUS. 👑
✨ Regular Expressions ✨
Regular expression pattern matching may surprise you if you're not accustomed to Unicode. 😳 Starting in Perl 5.14, several pattern modifiers are available to control this, called the character set modifiers. Details are given in "Character set modifiers" in perlre.
🌍 Unicode in Perl on EBCDIC 🌍
Unicode is supported on EBCDIC platforms. See perlebcdic.
Unless ASCII vs. EBCDIC issues are specifically being discussed, references to UTF-8 encoding in this document and elsewhere should be read as meaning UTF-EBCDIC on EBCDIC platforms. See "Unicode and UTF" in perlebcdic.
Because UTF-EBCDIC is so similar to UTF-8, the differences are mostly hidden from you; use utf8 (and NOT something like use utfebcdic) declares the script is in the platform's "native" 8-bit encoding of Unicode. (Similarly for the :utf8 layer.)
🗺️ Locales 🗺️
See "Unicode and UTF-8" in perllocale
😶🌫️ When Unicode Does Not Happen 😶🌫️
There are still many places where Unicode (in some encoding or another) could be given as arguments or received as results, or both, in Perl, but it is not, in spite of Perl having extensive ways to input and output in Unicode, and a few other "entry points" like the @ARGV array (which can sometimes be interpreted as UTF-8).
The following are such interfaces. Also, see "The "Unicode Bug"". For all of these interfaces Perl currently (as of v5.16.0) simply assumes byte strings both as arguments and results, or UTF-8 strings if the (deprecated) "encoding" pragma has been used. 😬
One reason that Perl does not attempt to resolve the role of Unicode in these situations is that the answers are highly dependent on the operating system and the file system(s). For example, whether filenames can be in Unicode and in exactly what kind of encoding, is not exactly a portable concept. Similarly for qx and system: how well will the "command-line interface" (and which of them?) handle Unicode? 🤷♀️
chdir,chmod,chown,chroot,exec,link,lstat,mkdir,rename,rmdir,stat,symlink,truncate,unlink,utime,-X%ENVglob(aka the<*>)open,opendir,sysopenqx(aka the backtick operator),systemreaddir,readlink
🐛 The "Unicode Bug" 🐛
The term, "Unicode bug" has been applied to an inconsistency with the code points in the "Latin-1 Supplement" block, that is, between 128 and 255. Without a locale specified, unlike all other characters or code points, these characters can have very different semantics depending on the rules in effect. (Characters whose code points are above 255 force Unicode rules; whereas the rules for ASCII characters are the same under both ASCII and Unicode rules.)
Under Unicode rules, these upper-Latin1 characters are interpreted as Unicode code points, which means they have the same semantics as Latin-1 (ISO-8859-1) and C1 controls.
As explained in "ASCII Rules versus Unicode Rules", under ASCII rules, they are considered to be unassigned characters.
This can lead to unexpected results. 💀 For example, a string's semantics can suddenly change if a code point above 255 is appended to it, which changes the rules from ASCII to Unicode. As an example, consider the following program and its output:
$ perl -le'
no feature "unicode_strings";
$s1 = "\xC2";
$s2 = "\x{2660}";
for ($s1, $s2, $s1.$s2) {
print /\w/ || 0;
}
'
0
0
1So like, if there's no \w in "s1" and, again, no \w in "s2"… why on earth does smushing them together suddenly give you one? 😭
This lil' anomaly is literally just Perl trying so hard not to disturb older programs that never used Unicode, while also wanting to add Unicode support all seamless-like. Sadly it did not turn out seamless, bestie. 🙈 (Oh and btw, you can choose to get warned when stuff like this goes down — see "encoding::warnings".)
So use feature 'unicode_strings' got added, starting in Perl v5.12, to fix this whole mess. It affects these things, so stay with me:
- Changing the case of a scalar, k that's
uc(),ucfirst(),lc(), andlcfirst(), or"\L","\U","\u"and"\l"in double-quotish contexts, like regular expression substitutions.
Under unicode_strings starting in Perl 5.12.0, Unicode rules are generally used. See "lc" in perlfunc for the deets on how this plays with various other pragmas.
- Using caseless (
"/i") regular expression matching.
Starting in Perl 5.14.0, regular expressions compiled within the scope of unicode_strings use Unicode rules even when executed or compiled into larger regular expressions outside the scope.
- Matching any of several properties in regular expressions.
These properties are "\b" (without braces), "\B" (without braces), "\s", "\S", "\w", "\W", and all the Posix character classes except "[[:ascii:]]".
Starting in Perl 5.14.0, regular expressions compiled within the scope of unicode_strings use Unicode rules even when executed or compiled into larger regular expressions outside the scope.
- In
"quotemeta"or its inline equivalent"\Q".
Starting in Perl 5.16.0, consistent quoting rules are used within the scope of unicode_strings, as described in "quotemeta" in perlfunc. Prior to that, or outside its scope, no code points above 127 are quoted in UTF-8 encoded strings, but in byte encoded strings, code points between 128-255 are always quoted.
- In the
".."or range operator.
Starting in Perl 5.26.0, the range operator on strings treats their lengths consistently within the scope of unicode_strings. Prior to that, or outside its scope, it could produce strings whose length in characters exceeded that of the right-hand side, where the right-hand side took up more bytes than the correct range endpoint.
- In
"split"'s special-case whitespace splitting.
Starting in Perl 5.28.0, the "split" function with a pattern specified as a string containing a single space handles whitespace characters consistently within the scope of unicode_strings. Prior to that, or outside its scope, characters that are whitespace according to Unicode rules but not according to ASCII rules were treated as field contents rather than field separators when they appear in byte-encoded strings.
You can see from all of the above that the effect of unicode_strings increased over several Perl releases. (And Perl's support for Unicode keeps getting better; it's best to use the latest available release so you get the most complete and accurate results possible, fr fr.) Note that unicode_strings is automatically chosen if you use v5.12 or higher. 💅
For Perls earlier than those described above, or when a string is passed to a function outside the scope of unicode_strings, see the next section. 👀
Forcing Unicode in Perl (Or Unforcing Unicode in Perl) 💪
Sometimes (see "When Unicode Does Not Happen" or "The "Unicode Bug"") there are situations where you simply need to force a byte string into UTF-8, or the other way around. The standard module Encode can be used for this, or the low-level calls utf8::upgrade($bytestring) and "utf8::downgrade($utf8string[, FAIL_OK])".
Note that utf8::downgrade() can fail if the string contains characters that don't fit into a byte. 🫠
Calling either function on a string that already is in the desired state is a no-op.
"ASCII Rules versus Unicode Rules" gives all the ways that a string is made to use Unicode rules.
Using Unicode in XS 🧚
See "Unicode Support" in perlguts for an introduction to Unicode at the XS level, and "Unicode Support" in perlapi for the API details.
Hacking Perl to work on earlier Unicode versions (for very serious hackers only) 🧑💻
Perl by default comes with the latest supported Unicode version built-in, but the goal is to allow you to change to use any earlier one. In Perls v5.20 and v5.22, however, the earliest usable version is Unicode 5.1. Perl v5.18 and v5.24 are able to handle all earlier versions.
Download the files in the desired version of Unicode from the Unicode web site <https://www.unicode.org>). These should replace the existing files in lib/unicore in the Perl source tree. Follow the instructions in README.perl in that directory to change some of their names, and then build perl (see INSTALL).
Porting code from perl-5.6.X 🕰️
Perls starting in 5.8 have a different Unicode model from 5.6. In 5.6 the programmer was required to use the "utf8" pragma to declare that a given scope expected to deal with Unicode data and had to make sure that only Unicode data were reaching that scope. If you have code that is working with 5.6, you will need some of the following adjustments to your code. The examples are written such that the code will continue to work under 5.6, so you should be safe to try them out.
- A filehandle that should read or write UTF-8
if ($] > 5.008) {
binmode $fh, ":encoding(UTF-8)";
}- A scalar that is going to be passed to some extension
Be it "Compress::Zlib", "Apache::Request" or any extension that has no mention of Unicode in the manpage, you need to make sure that the UTF8 flag is stripped off. Note that at the time of this writing (January 2012) the mentioned modules are not UTF-8-aware. Please check the documentation to verify if this is still true.
if ($] > 5.008) {
require Encode;
$val = Encode::encode("UTF-8", $val); # make octets
}- A scalar we got back from an extension
If you believe the scalar comes back as UTF-8, you will most likely want the UTF8 flag restored:
if ($] > 5.008) {
require Encode;
$val = Encode::decode("UTF-8", $val);
}- Same thing, if you are really sure it is UTF-8
if ($] > 5.008) {
require Encode;
Encode::_utf8_on($val);
}- A wrapper for DBI
"fetchrow_array"and"fetchrow_hashref"
When the database contains only UTF-8, a wrapper function or method is a convenient way to replace all your "fetchrow_array" and "fetchrow_hashref" calls. A wrapper function will also make it easier to adapt to future enhancements in your database driver. Note that at the time of this writing (January 2012), the DBI has no standardized way to deal with UTF-8 data. Please check the DBI documentation to verify if that is still true.
sub fetchrow {
# $what is one of fetchrow_{array,hashref}
my($self, $sth, $what) = @_;
if ($] < 5.008) {
return $sth->$what;
} else {
require Encode;
if (wantarray) {
my @arr = $sth->$what;
for (@arr) {
defined && /[^\000-\177]/ && Encode::_utf8_on($_);
}
return @arr;
} else {
my $ret = $sth->$what;
if (ref $ret) {
for my $k (keys %$ret) {
defined
&& /[^\000-\177]/
&& Encode::_utf8_on($_) for $ret->{$k};
}
return $ret;
} else {
defined && /[^\000-\177]/ && Encode::_utf8_on($_) for $ret;
return $ret;
}
}
}
}- A large scalar that you know can only contain ASCII
Scalars that contain only ASCII and are marked as UTF-8 are sometimes a drag to your program. If you recognize such a situation, just remove the UTF8 flag:
utf8::downgrade($val) if $] > 5.008;💀 BUGS 💀
Okay bestie, before we get into it — go peek at "The "Unicode Bug"" up above ☝️ because that one's a whole thing and it lives there, not here.
✨ Interaction with Extensions ✨
So here's the tea 🍵 when Perl talks to an extension: the extension is supposed to understand the UTF8 flag and act accordingly. If it doesn't recognize that flag? Yeah, it's probably gonna hand you back data with the wrong flag on it. Incorrectly-flagged data. Not the vibe. 😭
So if you're working with Unicode data, consult the documentation of every module you're using to check if there are any issues with Unicode data exchange. And if the docs don't mention Unicode at all? Suspect the worst, bestie 🚩 and honestly go read the source to learn how the module is actually implemented. Modules written completely in Perl shouldn't cause problems. It's the modules that directly or indirectly access code written in other programming languages that are at risk. 👀
For affected functions, the simple strategy to avoid data corruption is to always make the encoding of the exchanged data explicit. Choose an encoding that you know the extension can handle. Convert arguments passed to the extensions to that encoding, and convert results back from that encoding. Then write wrapper functions that do the conversions for you, so you can later change those functions when the extension finally catches up. Main character foresight. 👑
To give you an example, let's say the popular Foo::Bar::escape_html function doesn't deal with Unicode data yet. The wrapper function would convert the argument to raw UTF-8 and convert the result back to Perl's internal representation, like so:
sub my_escape_html ($) {
my($what) = shift;
return unless defined $what;
Encode::decode("UTF-8", Foo::Bar::escape_html(
Encode::encode("UTF-8", $what)));
}Sometimes, when the extension does not convert data but just stores and retrieves it, you'll be able to use the otherwise dangerous Encode::_utf8_on() function. ⚠️ Let's say the popular Foo::Bar extension, written in C, provides a param method that lets you store and retrieve data according to these prototypes:
$self->param($name, $value); # set a scalar
$value = $self->param($name); # retrieve a scalarIf it does not yet provide support for any encoding, one could write a derived class with a param method like this:
sub param {
my($self,$name,$value) = @_;
utf8::upgrade($name); # make sure it is UTF-8 encoded
if (defined $value) {
utf8::upgrade($value); # make sure it is UTF-8 encoded
return $self->SUPER::param($name,$value);
} else {
my $ret = $self->SUPER::param($name);
Encode::_utf8_on($ret); # we know, it is UTF-8 encoded
return $ret;
}
}Some extensions provide filters on data entry/exit points, such as DB_File::filter_store_key and family. 👯 Look out for such filters in the documentation of your extensions — they can make the transition to Unicode data much easier. Slay. 💅
🌸 Speed 🌸
Lowkey some functions are slower when working on UTF-8 encoded strings than on byte encoded strings. All the functions that need to hop over characters — like length(), substr() or index() — or matching regular expressions can work much faster when the underlying data are byte-encoded. 🏃♀️💨
In Perl 5.8.0 the slowness was often quite spectacular 😳; in Perl 5.8.1 a caching scheme was introduced which improved the situation. In general though, operations with UTF-8 encoded strings are still slower.
As an example: the Unicode properties (character classes) like \p{Nd} are known to be quite a bit slower (5-20 times) than their simpler counterparts like [0-9]. That said — and this is the important part — there are hundreds of Unicode characters matching Nd, compared with the 10 ASCII characters matching [0-9]. So like… you're getting a lot more for the wait, okay? 🤷♀️
🔍 SEE ALSO 🔍
perlunitutperluniintroperlunipropsEncodeopenutf8bytesperlretut${^UNICODE}inperlvarhttps://www.unicode.org/reports/tr44
aaaand that's a wrap on PERLUNICODE(1perl) bestie!! 🎀🦋 you read the whole thing, which is genuinely iconic of you, no cap. this was perl v5.44.0, dated 2026-10-17 — go forth and encode responsibly, mwah 💖✨
📦 where this page comes from
- arch rolling: perl 5.44.0-1 · Artistic-1.0-Perl AND GPL-1.0-or-later
- 💖 girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms