Encode::PerlIO(3perl)

💖 NAME 💖

Encode::PerlIO — the whole entire detailed doc on Encode and PerlIO, bestie. no cap, this is the deep lore 💅

---

✨ Overview ✨

okay so picture this: you wanna do encoding transformations while you're reading or writing files, network connections, pipes, etc. super common, everyone wants it. and if your Perl is configured to use the new 'perlio' IO system, then Encode hands you a "layer" (peep PerlIO for the deets) that can transform data as it's read or written. iconic behaviour honestly 🦋

here's how the blind poet would modernise the encoding, and yes this is real:

use Encode;
open(my $iliad,'<:encoding(iso-8859-7)','iliad.greek');
open(my $utf8,'>:utf8','iliad.utf8');
my @epic = <$iliad>;
print $utf8 @epic;
close($utf8);
close($illiad);

and bonus: the new IO system can also be set up to read/write UTF-8 encoded characters (like I said above, it's efficient, we love that):

open(my $fh,'>:utf8','anything');
print $fh "Any \x{0021} string \N{SMILEY FACE}\n";

either one of those "layer" specification forms can be made the default for a lexical scope using the use open ... pragma. see open for that one.

once a handle is open, you can still change its layers with binmode. flexible queen 💅

now — no configuration at all, or Perl itself built using the system's own IO? then write operations just assume the file handle accepts only bytes and will literally die if you write a character bigger than 255 to it. and when reading, every octet from the handle becomes a byte-in-a-character. note that this default is exactly the same behaviour bytes-only languages have (including Perl before v5.6), and it's enough to handle native 8-bit encodings, e.g. iso-8859-1, EBCDIC etc., plus any legacy mechanisms for handling other encodings and binary data. periodt.

in other cases tho, it is 100% the program's responsibility to transform characters into bytes using the API above before doing writes, and to transform the bytes read from a handle into characters before doing "character operations" (e.g. lc, /\W+/, ...). your problem now bestie 🙈

you can also flex PerlIO to convert larger amounts of data you don't wanna bring into memory. for example, converting between ISO-8859-1 (Latin 1) and UTF-8 (or UTF-EBCDIC in EBCDIC machines):

open(F, "<:encoding(iso-8859-1)", "data.txt") or die $!;
open(G, ">:utf8",                 "data.utf") or die $!;
while (<F>) { print G }

# Could also do "print G <F>" but that would pull
# the whole file into memory just to write it out again.

more examples, because we stay fed:

open(my $f, "<:encoding(cp1252)")
open(my $g, ">:encoding(iso-8859-2)")
open(my $h, ">:encoding(latin9)")       # iso-8859-15

also peep encoding for how to change the default encoding of the data in your script. iykyk ✨

---

🎀 How does it work? 🎀

here's a crude diagram of how filehandle, PerlIO, and Encode interact. it's giving science class but make it cute:

filehandle <-> PerlIO        PerlIO <-> scalar (read/printed)
                     \      /
                      Encode

so when PerlIO receives data from either direction, it fills a buffer (currently with 1024 bytes) and passes that buffer to Encode. Encode tries to convert the valid part and passes it back to PerlIO, leaving the invalid parts (usually a partial character) sitting in the buffer. then PerlIO appends more data to the buffer, calls Encode again, and so on and so on until the data stream ends. the grind never stops 💀

to pull that off, PerlIO always calls the (de|en)code methods with CHECK set to 1. this makes sure the method stops at the right place when it bumps into a partial character. and here's what happens when PerlIO and Encode try to encode (from utf8) more than 1024 bytes and the buffer boundary just happens to land in the middle of a character. rude:

   A   B   C   ....   ~     \x{3000}    ....
  41  42  43   ....  7E   e3   80   80  ....
  <- buffer --------------->
  << encoded >>>>>>>>>>
                       <- next buffer ------

Encode converts from the beginning all the way to \x7E, leaving \xe3 in the buffer because it's invalid (partial character). caught in 4k.

unfortunately — and this is the sad part 😭 — this whole scheme does not work well with escape-based encodings such as ISO-2022-JP.

---

🌸 Line Buffering 🌸

now let's see what goes down when you try to decode from ISO-2022-JP and the buffer ends right in the middle of a character. the drama:

              JIS208-ESC   \x{5f3e}
   A   B   C   ....   ~   \e   $   B  |DAN | ....
  41  42  43   ....  7E   1b  24  41  43  46 ....
  <- buffer --------------------------->
  << encoded >>>>>>>>>>>>>>>>>>>>>>>

see how the next buffer begins with \x43? but \x43 is 'C' in ASCII, which is wrong here, because we're now in JISX 0208 area so it has to convert \x43\x46, not \x43. unlike utf8 and EUC, in escape-based encodings you literally cannot tell if a given octet is a whole character or just part of one. it's giving ambiguous 💀

good news tho: PerlIO also supports line buffer if you tell PerlIO to use one instead of the fixed buffer. and since ISO-2022-JP is guaranteed to revert to ASCII at the end of the line, a partial character will never happen when line buffer is used. slay 🎀

to tell PerlIO to use line buffer, implement the ->needs_lines method for your encoding object. see Encode::Encoding for all the details.

thanks to these efforts, most encodings that come with Encode support PerlIO — but that still leaves the following ones out in the cold:

iso-2022-kr
MIME-B
MIME-Header
MIME-Q

fortunately iso-2022-kr is hardly ever used (according to Jungshik, so you know it's true) and MIME-* are very unlikely to be fed to PerlIO because they're for mail headers. see Encode::MIME::Header for that whole situation.

🧚 How can I tell whether my encoding fully supports PerlIO ? 🧚

as of this writing, any encoding whose class belongs to Encode::XS and Encode::Unicode works. the Encode module has a "perlio_ok" method you can use before applying PerlIO encoding to the filehandle. example time:

my $use_perlio = perlio_ok($enc);
my $layer = $use_perlio ? "<:raw" : "<:encoding($enc)";
open my $fh, $layer, $file or die "$file : $!";
while(<$fh>){
  $_ = decode($enc, $_) unless $use_perlio;
  # ....
}

vibe check passed, you're welcome 💜

---

👑 SEE ALSO 👑

Encode::Encoding, Encode::Supported, Encode::PerlIO, encoding, perlebcdic, "open" in perlfunc, perlunicode, utf8, the Perl Unicode Mailing List <perl-unicode@perl.org> — go read them, bestie, all of them, they're all linked together like a little friend group 🫶

---

okay that's the whole page, mwah 💖 this was written for perl v5.44.0 and dated 2026-10-17, Encode::PerlIO(3perl) — stay encoded, stay iconic, and remember: partial characters are not your fault, they're the buffer's fault ✨💅