Encode::Guess(3)

🌸 Encode::Guess(3) — like, the Perl manual page, but make it make sense 💅

okay so bestie, this one's from User Contributed Perl Documentation and honestly? it's giving "detective module" energy. it's Encode::Guess, and it's here to look at a blob of bytes and be like "hmm… what encoding are you, bestie?" 🔍✨

---

✨ NAME ✨

Encode::Guess — literally guesses the encoding from the data. that's the whole job. icon behaviour fr.

---

🎀 SYNOPSIS 🎀

  # if you are sure $data won't contain anything bogus

  use Encode;
  use Encode::Guess qw/euc-jp shiftjis 7bit-jis/;
  my $utf8 = decode("Guess", $data);
  my $data = encode("Guess", $utf8);   # this doesn't work!

  # more elaborate way
  use Encode::Guess;
  my $enc = guess_encoding($data, qw/euc-jp shiftjis 7bit-jis/);
  ref($enc) or die "Can't guess: $enc"; # trap error this way
  $utf8 = $enc->decode($data);
  # or
  $utf8 = decode($enc->name, $data)

did you catch that comment? # this doesn't work! — yeah, encoding into "Guess" is not a thing, bestie. guessing is a one-way street 🙈

---

🦋 ABSTRACT 🦋

Encode::Guess lets you guess what encoding some data is in — or, at the very least, it tries. lowkey tries. we respect the effort 🫶

---

💖 DESCRIPTION 💖

so by default it only checks ascii, utf8, and UTF-16/32 with BOM. that's the starter pack ✨

  use Encode::Guess; # ascii/utf8/BOMed UTF

to actually make it useful, you gotta hand it the names of the encodings you want it to check — those are your suspects (yes, that's the real terminology, it's giving murder mystery 🕵️‍♀️). the names of the suspects can be either canonical names or aliases. either one slays.

CAVEAT: unlike UTF-(16|32), the BOM in utf8 is NOT AUTOMATICALLY STRIPPED. no cap. it just sits there. 😭

 # tries all major Japanese Encodings as well
  use Encode::Guess qw/euc-jp shiftjis 7bit-jis/;

and listen — if $Encode::Guess::NoUTFAutoGuess is set to a true value, then no heuristics get applied to UTF8/16/32, and the result is limited to the suspects plus "ascii". so basically you turned off the vibes-based guessing. valid.

👑 Encode::Guess->set_suspects

you can also swap out the internal suspects list using the set_suspects method. very main character energy.

  use Encode::Guess;
  Encode::Guess->set_suspects(qw/euc-jp shiftjis 7bit-jis/);

🌈 Encode::Guess->add_suspects

orrr you can use add_suspects. the difference? set_suspects flushes the current suspects list, while add_suspects just adds to it. one is a clean slate, the other is a group chat invite 🎀

  use Encode::Guess;
  Encode::Guess->add_suspects(qw/euc-jp shiftjis 7bit-jis/);
  # now the suspects are euc-jp,shiftjis,7bit-jis, AND
  # euc-kr,euc-cn, and big5-eten
  Encode::Guess->add_suspects(qw/euc-kr euc-cn big5-eten/);

💜 Encode::decode("Guess" ...)

once you're happy with your suspects list, you can literally just do:

  my $utf8 = Encode::decode("Guess", $data);

🔥 Encode::Guess->guess($data)

but heads up — it will croak if:

• two or more suspects remain 😬
• no suspects are left 💀

so instead, try this:

  my $decoder = Encode::Guess->guess($data);

on success, $decoder is an object documented in Encode::Encoding. so now you can do:

  my $utf8 = $decoder->decode($data);

on failure, $decoder now contains an error message, so the whole thing looks like:

  my $decoder = Encode::Guess->guess($data);
  die $decoder unless ref($decoder);
  my $utf8 = $decoder->decode($data);

that unless ref($decoder) check is the vibe check, bestie — if it's not a reference, it's a string, and strings mean the guessing flopped 🫠

🌸 guess_encoding($data, [, list of suspects])

you can also use the guess_encoding function, which is exported by default (no qw needed, we love convenience ✨). it takes $data to check, and it optionally takes the list of suspects too. important: that optional suspect list is not reflected back into the internal suspects list. it's a one-time thing, iykyk.

  my $decoder = guess_encoding($data, qw/euc-jp euc-kr euc-cn/);
  die $decoder unless ref($decoder);
  my $utf8 = $decoder->decode($data);
  # check only ascii, utf8 and UTF-(16|32) with BOM
  my $decoder = guess_encoding($data);

---

💀 CAVEATS 💀

• because of the algorithm it uses, ISO-8859 series and other single-byte encodings do not work well unless one of ISO-8859 is the only suspect (besides ascii and utf8). no cap, this is real:

  use Encode::Guess;
  # perhaps ok
  my $decoder = guess_encoding($data, 'latin1');
  # definitely NOT ok
  my $decoder = guess_encoding($data, qw/latin1 greek/);

the reason? Encode::Guess guesses encoding by trial and error. it first splits $data into lines, then tries to decode each line for every suspect. it keeps going until all but one encoding is eliminated from the suspects list. and ISO-8859 series is just… too successful at most cases (because it fills almost all code points in \x00-\xff). it ate too hard and now it's useless 😭

• do not mix national standard encodings with the corresponding vendor encodings. like:

  # a very bad idea
  my $decoder
     = guess_encoding($data, qw/shiftjis MacJapanese cp932/);

because vendor encoding is usually a superset of the national standard, so it becomes way too ambiguous for most cases. it's giving "same outfit, three people" 🫠

• on the other hand, mixing various national standard encodings automagically works — unless $data is too short to allow for guessing:

 # This is ok if $data is long enough
 my $decoder =
  guess_encoding($data, qw/euc-cn
                           euc-jp shiftjis 7bit-jis
                           euc-kr
                           big5-eten/);

• DO NOT PUT TOO MANY SUSPECTS! don't you dare try something like this:

  my $decoder = guess_encoding($data,
                               Encode->encodings(":all"));

because it is, after all, just a guess. you should always be explicit when it comes to encodings. but there are some environments — especially Japanese ones — where guess-coding is a must. so use this module with care 🧚

---

🍓 TO DO 🍓

Encode::Guess does not work on EBCDIC platforms. that's the whole to-do list. short and honest 💅

---

⭐ SEE ALSO ⭐

Encode, Encode::Encoding

---

and that's the page, bestie 🫶 go forth and guess responsibly — remember, it's literally called Guess, not Know 💖✨

this masterpiece is from perl v5.42.0, dated 2024-03-08, and it's the Encode::Guess(3) page — mwah, iconic, no notes 🎀