PERLUNITUT(1perl)

💖 NAME 💖

perlunitut — the Perl Unicode Tutorial, aka the glow-up your string handling has been begging for ✨

---

🦋 DESCRIPTION 🦋

okay so bestie, the era of just flinging strings around with zero thoughts?? over. done. dead. 💀

It's like, well established at this point that modern programs need to be able to handle funny accented letters and euro symbols and all that gorgeousness. Which means programmers need some new habits, no cap. Programming Unicode-capable software is genuinely easy — but doing it right? That takes discipline. 🫶

There's a LOT to know about character sets and text encodings. Ideally you'd dedicate a whole day to it. But honestly? The basics? You can learn those in minutes. fr fr.

That said — these are not the very basics. This assumes you already know the difference between bytes and characters, and that you've fully realized (and accepted!!) that there are many different character sets and encodings, and that your program has to be explicit about which is which. Required reading before this: "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" by Joel Spolsky, over at <https://joelonsoftware.com/articles/Unicode.html>. 📚

Also just so we're clear — this tutorial speaks in pretty absolute terms and gives you only a limited view of the whole glorious wealth of character-string-related features Perl has. For most projects though, this info will probably suffice. 💅

✨ Definitions ✨

Okay, first things first, we need to set a few things straight. This is literally the most important part of the whole tutorial. And ngl, this view may conflict with other stuff you've found on the web — but that's mostly because a bunch of sources out there are just wrong. 🙈

You may have to re-read this entire section a few times...

Unicode

Unicode is a character set with room for lots of characters. The ordinal value of a character is called a code point. (But in practice, the distinction between code point and character is blurred, so the terms often get used interchangeably.)

There are many, many code points, but computers work with bytes, and a byte has room for only 256 values. Unicode has way more characters than that, so you need a method to make all of them accessible.

Unicode is encoded using several competing encodings, of which UTF-8 is the most used. In a Unicode encoding, multiple subsequent bytes can be used to store a single code point — or simply: a character.

UTF-8

UTF-8 is a Unicode encoding. Many people think Unicode and UTF-8 are the same thing?? They are not. 😭 There are more Unicode encodings, but much of the world has standardized on UTF-8.

UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. Those take only one byte per character. Every other character is encoded as two to four bytes using a complex scheme. Fortunately, Perl handles all that for us, so we don't have to worry about it. 🎀

Text strings (character strings)

Text strings, or character strings, are made of characters. Bytes are irrelevant here, and so are encodings. Each character is just that: the character.

On a text string, you would do things like:

$text =~ s/foo/bar/;
if ($string =~ /^\d+$/) { ... }
$text = ucfirst $text;
my $character_count = length $text;

The value of a character (ord, chr) is the corresponding Unicode code point.

Binary strings (byte strings)

Binary strings, or byte strings, are made of bytes. Here you don't have characters, just bytes. All communication with the outside world (anything outside of your current Perl process) is done in binary.

On a binary string, you would do things like:

my (@length_content) = unpack "(V/a)*", $binary;
$binary =~ s/\x00\x0F/\xFF\xF0/;  # for the brave :)
print {$fh} $binary;
my $byte_count = length $binary;

Encoding

Encoding (as a verb) is the conversion from text to binary. To encode, you have to supply the target encoding, for example iso-8859-1 or UTF-8. Some encodings, like the iso-8859 ("latin") range, do not support the full Unicode standard; characters that can't be represented are lost in the conversion. 💔

Decoding

Decoding is the conversion from binary to text. To decode, you have to know what encoding was used during the encoding phase. And most of all, it must be something decodable. Decoding a PNG image into a text string doesn't make much sense, bestie. 🖼️

Internal format

Perl has an internal format, an encoding it uses to encode text strings so it can store them in memory. All text strings are in this internal format. In fact, text strings are never in any other format!

You shouldn't worry about what this format specifically is, because conversion happens automatically when you decode or encode. 😌

🧰 Your new toolkit

Add this line to your standard heading:

use Encode qw(encode decode);

Or, if you're lazy (no judgment, we love a shortcut), just:

use Encode;

🌈 I/O flow (the actual 5 minute tutorial)

The typical input/output flow of a program is:

1. Receive and decode
2. Process
3. Encode and output

If your input is binary, and is supposed to remain binary, then obviously you shouldn't decode it to a text string. But in all other cases, you should decode it.

Decoding can't happen reliably if you don't know how the data was encoded. And if you get to choose, it's a good idea to standardize on UTF-8.

my $foo   = decode('UTF-8', get 'http://example.com/');
my $bar   = decode('ISO-8859-1', readline STDIN);
my $xyzzy = decode('Windows-1251', $cgi->param('foo'));

Processing happens exactly as you knew it before. The only difference is that you're now using characters instead of bytes. Which is super useful when you use things like substr or length. 🔥

It's important to realize there are no bytes in a text string. Yes, Perl has its internal encoding to store the string in memory, but ignore that. If you have to do anything involving the number of bytes, it's probably best to move that part to step 3, right after you've encoded the string. Then you know exactly how many bytes it'll be in the destination string.

The syntax for encoding text strings to binary strings is as simple as decoding:

$body = encode('UTF-8', $body);

And if you needed to know the length of the string in bytes? Now's the perfect time. Because $body is now a byte string, length will report the number of bytes instead of the number of characters. The number of characters is no longer known, because characters only exist in text strings. 👑

my $byte_count = length $body;

And if the protocol you're using supports a way of letting the recipient know which character encoding you used — please help the receiving end by using that feature! For example, E-mail and HTTP support MIME headers, so you can use the Content-Type header. They can also have Content-Length to indicate the number of bytes, which is always a good idea to supply if the number is known.

"Content-Type: text/plain; charset=UTF-8",
"Content-Length: $byte_count"

---

⭐ SUMMARY ⭐

Decode everything you receive, encode everything you send out. (If it's text data, obviously. 😘)

---

💬 Q and A (or FAQ) 💬

After reading this document, you ought to read perlunifaq too, then perluniintro.

---

🫶 ACKNOWLEDGEMENTS 🫶

Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the Amsterdam Perl Mongers meetings got me interested and determined to find out how to use character encodings in Perl in ways that don't break easily. 😤

Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild" (Dutch Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.

Thanks to the people who asked about this kind of stuff in several Perl IRC channels, and have constantly reminded me that a simpler explanation was needed.

Thanks to the people who reviewed this document for me, before it went public. They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan Gray. 💜

---

👑 AUTHOR 👑

Juerd Waalboer <#####@juerd.nl>

---

🌸 SEE ALSO 🌸

perlunifaq, perlunicode, perluniintro, Encode

---

and that's a wrap babes ✨ this tutorial ships with perl v5.44.0 and is dated 2026-10-17 — go forth and decode responsibly, ily 💖