Unicode::UTF8(3pm)

📦
version 0.75 · 11 versions
, in ubuntu stonking

💖 Unicode::UTF8 💖

okay so bestie, welcome to Unicode::UTF8 — it's the Perl module (a 3pm manual page, in case the vibe check needed confirming) that does Encoding and decoding of UTF-8 encoding form. pull up a chair, grab a strawberry 🍓, we're doing this.

✨ SYNOPSIS ✨

like, here's the whole thing in one screenshot-worthy block:

use Unicode::UTF8 qw[decode_utf8 encode_utf8 read_utf8];

use warnings FATAL => 'utf8'; # fatalize encoding glitches
$string = decode_utf8($octets);
$octets = encode_utf8($string);

$count = read_utf8($fh, $buf, $length);

🌈 DESCRIPTION 🌈

bestie. this module hands you functions to encode and decode the UTF-8 encoding form exactly as specified by Unicode and ISO/IEC 10646:2011. no cap. that's the whole assignment and it ate. 👑

🦋 FUNCTIONS 🦋

decode_utf8

$string = decode_utf8($octets);
$string = decode_utf8($octets, $fallback);

gives you back a decoded representation of $octets in UTF-8 encoding, as a character string. cute, right?

$fallback is an optional CODE reference and it's your lil error-handling mechanism — it lets you customize how errors get handled. the default error-handling mechanism? it replaces any ill-formed UTF-8 sequences with REPLACEMENT CHARACTER (U+FFFD). iconic behavior honestly.

$string = $fallback->($octets, $is_usv, $position);

so $fallback gets invoked with three arguments: $octets, $is_usv and $position. lemme break it down:

  • $octets is a sequence of one or more octets containing the maximal subpart of the ill-formed subsequence or encoded code point which can't be interchanged.
  • $is_usv is a boolean indicating whether or not $octets represent a encoded Unicode scalar value.
  • $position is an unsigned integer containing the zero based octet position at which the error occurred within the octets provided to decode_utf8().

$fallback must return a character string consisting of zero or more Unicode scalar values. and what counts as a Unicode scalar value? code points in the range U+0000..U+D7FF and U+E000..U+10FFFF. periodt. 💅

encode_utf8

$octets = encode_utf8($string);
$octets = encode_utf8($string, $fallback);

returns an encoded representation of $string in UTF-8 encoding as an octet string. the other direction, we love a symmetrical queen.

$fallback is an optional CODE reference which provides a error-handling mechanism, allowing customization of error handling. the default error-handling mechanism is to replace any code points which can't be interchanged or represented in UTF-8 encoding form with REPLACEMENT CHARACTER (U+FFFD).

$string = $fallback->($codepoint, $is_usv, $position);

$fallback is invoked with three arguments: $codepoint, $is_usv and $position.

  • $codepoint is a unsigned integer containing the code point which can't be interchanged or represented in UTF-8 encoding form.
  • $is_usv is a boolean indicating whether or not $codepoint is a Unicode scalar value.
  • $position is a unsigned integer containing the zero based character position at which the error occurred within the string provided to encode_utf8().

$fallback must return a character string consisting of zero or more Unicode scalar values. Unicode scalar values consist of code points in the range U+0000..U+D7FF and U+E000..U+10FFFF. (yes we said it twice, it's that important 💀)

read_utf8

$count = read_utf8($fh, $buf, $length);
$count = read_utf8($fh, $buf, $length, $offset);

reads up to $length UTF-8 encoded characters (code points) from the file handle $fh, decoding and validating them in place, and stores the result in $buf. returns the number of characters actually read, 0 at end of file, or undef on a read error (with $! set).

because read_utf8 reads and validates the octets directly, there is no need to apply a PerlIO encoding layer (such as :encoding(UTF-8) or :utf8) to $fh. the handle should be a plain byte handle; the bytes are validated and decoded by read_utf8 itself. it does its own thing, no help needed, main character energy fr. ✨

if $offset is specified, the read data is written into $buf starting at that character offset, preserving the existing content before it. a negative $offset counts back from the end of $buf. if the offset is past the end of the string, $buf is zero-filled up to the offset first.

ill-formed and truncated input is not fatal: each maximal ill-formed subpart is replaced with the Unicode replacement character U+FFFD and a warning is emitted in the utf8 warnings category. the returned count includes the substituted code points.

tied file handles are not supported. nope. not happening. 🙈

since version 0.71.

slurp_utf8

$string = slurp_utf8($filename);

reads the entire file named $filename and returns its contents decoded from UTF-8 as a character string. the file is read using unbuffered (:unix) IO, and the octets are validated and decoded directly.

there is no upper bound on the amount read; the whole file is loaded into memory. when handling untrusted or potentially large files, check the size first (for example with -s $filename) before calling slurp_utf8. bestie please check the size first, we don't want your RAM crying. 😭

ill-formed and truncated input is not fatal: each maximal ill-formed subpart is replaced with the Unicode replacement character U+FFFD and a warning is emitted in the utf8 warnings category.

throws an exception if the file cannot be opened or read. 🚨

since version 0.73.

valid_utf8

$boolean = valid_utf8($octets);

returns a boolean indicating whether or not the given $octets consist of well-formed UTF-8 sequences. yes or no, no essay required.

since version 0.60.

🎀 EXPORTS 🎀

None by default. all functions can be exported using the :all tag or individually. you gotta ask for them, they're not just gonna show up uninvited. 💅

👑 DIAGNOSTICS 👑

Can't decode a wide character string

(F) Wide character in octets.

Can't validate a wide character string

(F) Wide character in octets.

Can't decode ill-formed UTF-8 octet sequence <%s> in position %u

(W utf8) Encountered an ill-formed UTF-8 octet sequence. <%s> contains a hexadecimal representation of the maximal subpart of the ill-formed subsequence.

Can't represent surrogate code point U+%X in position %u

(W utf8, surrogate) Surrogate code points are designated only for surrogate code units in the UTF-16 character encoding form. Surrogates consist of code points in the range U+D800 to U+DFFF.

Can't represent super code point \x{%X} in position %u

(W utf8, non_unicode) Code points greater than U+10FFFF. Perl's extended codespace.

Can't decode ill-formed UTF-X octet sequence <%s> in position %u

(F) Encountered an ill-formed octet sequence in Perl's internal representation of wide characters.

and a lil note bestie: the sub-categories surrogate and non_unicode is only available on Perl 5.14 or greater. see perllexwarn for available categories and hierarchies. iykyk.

🌸 COMPARISON 🌸

here's a summary of features for comparison with Encode's UTF-8 implementation, because a queen knows her competition:

  • Simple API which makes use of Perl's standard warning categories.
  • Implements Unicode's recommended practice for using U+FFFD.
  • Better diagnostics in warning messages
  • Detects and reports inconsistency in Perl's internal representation of wide characters (UTF-X)
  • Preserves taintedness of decoded $octets or encoded $string
  • Better performance ~ 600% - 1200% (JA: 600%, AR: 700%, SV: 900%, EN: 1200%, see benchmarks directory in git repository)

like that last bullet?? six hundred to twelve hundred percent?? it's giving speed. 🔥

⭐ CONFORMANCE ⭐

it's the author's belief that this UTF-8 implementation is conformant with the Unicode Standard Version 6.0. any deviations from the Unicode Standard is to be considered a bug.

🫶 SEE ALSO 🫶

💜 SUPPORT 💜

BUGS

please report any bugs through the web interface at <https://github.com/chansen/p5-unicode-utf8/issues>. you will be automatically notified of any progress on the request by the system. so you'll literally get pinged, no refreshing required. 🦋

SOURCE CODE

this is open source software. the code repository is available for public review and contribution under the terms of the license.

<http://github.com/chansen/p5-unicode-utf8>

git clone http://github.com/chansen/p5-unicode-utf8

🌟 AUTHOR 🌟

Christian Hansen chansen@cpan.org

Copyright 2011-2026 by Christian Hansen.

this is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

okay and that's the whole page bestie!! 🎉 before you go: this lil manual was written for perl v5.42.3 and the page itself is dated 2026-08-20 — no cap, that's the version and the date, noted for the record. mwah, go encode something gorgeous 💖🌸💅

📦 where this page comes from