Encode...pCP1252(3pm)

📦
version 0.40 · 3 versions
, in debian bookworm, debian trixie, ubuntu noble

💖 Name 💖

Encode::ZapCP1252 — okay bestie, it literally zaps Windows Western gremlins ✨

✨ Synopsis ✨

use Encode::ZapCP1252;

# Zap or fix in-place.
zap_cp1252 $latin1_text;
fix_cp1252 $utf8_text;

# Zap or fix copy.
my $clean_latin1 = zap_cp1252 $latin1_text;
my $fixed_utf8   = fix_cp1252 $utf8_text;

🌸 Description 🌸

So picture this: you're processing a Web form submit for a feed, you're being so good, assuming the incoming text is encoded exactly like the Content-Type header said, or like the XML declaration said, and then — BOOM 💀 — a whole bunch of junk shows up because somebody pasted in content from Microsoft Word. Not their fault, not your fault, it's just... the vibes were off.

The tea is this: Microsoft uses a superset of the Latin-1 encoding called "Windows Western" or "CP1252". So if the specified encoding is Latin-1, mostly things come out right, but a few things — like curly quotes, m-dashes, ellipses, and the like — may not. These differences are well-known, fr fr; there's a nice chart documenting them over on Wikipedia <https://en.wikipedia.org/wiki/Windows-1252> 👑

But lowkey, that chart won't actually help you. What will help you is to quit using Latin-1 and switch to UTF-8, periodt. Then you can just convert from CP1252 to UTF-8 without losing a single thing, like this:

use Encode;
$text = decode 'cp1252', $text, 1;

That's the dream. But I know there are those of you out there who are stuck with Latin-1 and who do not want any junk characters from Word users. That's where this module comes in, bestie. Its zap_cp1252 function will zap those CP1252 gremlins for you, turning them into their appropriate ASCII approximations. It's giving cleanup crew energy 🧹✨

Another case that can occasionally come up is when you're reading in text that claims to be UTF-8, but it still ends up with some CP1252 gremlins mixed in with properly encoded characters. Not me seeing examples of exactly this when processing GMail messages and attempting to insert them into a UTF-8 database, as well as in some feeds processed by, say, Yahoo! Pipes 😭 Doesn't work so well. For those cases, there's fix_cp1252, which converts those CP1252 gremlins into their UTF-8 equivalents. Iconic behaviour, honestly.

🦋 Usage 🦋

This module exports two subroutines: zap_cp1252() and fix_cp1252(), each of which accept a single argument:

zap_cp1252 $text;
fix_cp1252 $text;

When called in a void context, like in these examples, zap_cp1252() and fix_cp1252() perform in place conversions of any CP1252 gremlins into their appropriate ASCII approximations or UTF-8 equivalents, respectively. But heads up 👀 — because the conversion happens in place, the data to be converted cannot be a string constant; it must be a scalar variable. No exceptions, bestie.

When called in a scalar or list context, on the other hand, a copy will be modified and returned. The original string will be unchanged 💅

my $clean_latin1 = zap_cp1252 $latin1_text;
my $fixed_utf8   = fix_cp1252 $utf8_text;

In this case, even constant values can be processed. Either way, undefs will be ignored. We simply do not perceive them ✨

In Perl 5.10 and higher, the functions may optionally be called with no arguments, in which case $_ will be converted, instead:

zap_cp1252; # Modify $_ in-place.
fix_cp1252; # Modify $_ in-place.
my $zapped = zap_cp1252; # Copy $_ and return zapped
my $fixed = zap_cp1252; # Copy $_ and return fixed

In Perl 5.8.8 and higher, the conversion will work even when the string is decoded to Perl's internal form (usually via decode 'ISO-8859-1', $text) or the string is encoded (and thus simply processed by Perl as a series of bytes). The conversion will even work on a string that has not been decoded but has had its utf8 flag flipped anyway (usually by an injudicious use of Encode::_utf8_on()). This is to enable the highest possible likelihood of removing those CP1252 gremlins no matter what kind of processing has already been executed on the string. Maximum gremlin-removal energy 🫶

That said, although fix_cp1252() takes a conservative approach to replacing text in Unicode strings, it should be used as a very last option. Really, avoid that situation if you can. Like, genuinely. 🙈

🎀 Conversion Table 🎀

Here's how the characters are converted to ASCII and UTF-8. The ASCII conversions are not perfect, but they should be good enough for general cleanup. If you want perfect, switch to UTF-8 and be done with it! ⭐

   Hex | Char  | ASCII | UTF-8 Name
------+-------+-------+-------------------------------------------
0x80 |   X   |   e   | EURO SIGN
0x82 |   X   |   ,   | SINGLE LOW-9 QUOTATION MARK
0x83 |   X   |   f   | LATIN SMALL LETTER F WITH HOOK
0x84 |   X   |   ,,  | DOUBLE LOW-9 QUOTATION MARK
0x85 |   X   |  ...  | HORIZONTAL ELLIPSIS
0x86 |   X   |   +   | DAGGER
0x87 |   X   |   ++  | DOUBLE DAGGER
0x88 |   X   |   ^   | MODIFIER LETTER CIRCUMFLEX ACCENT
0x89 |   X   |   %   | PER MILLE SIGN
0x8a |   X   |   S   | LATIN CAPITAL LETTER S WITH CARON
0x8b |   X   |   <   | SINGLE LEFT-POINTING ANGLE QUOTATION MARK
0x8c |   X   |   OE  | LATIN CAPITAL LIGATURE OE
0x8e |   X   |   Z   | LATIN CAPITAL LETTER Z WITH CARON
0x91 |   X   |   '   | LEFT SINGLE QUOTATION MARK
0x92 |   X   |   '   | RIGHT SINGLE QUOTATION MARK
0x93 |   X   |   "   | LEFT DOUBLE QUOTATION MARK
0x94 |   X   |   "   | RIGHT DOUBLE QUOTATION MARK
0x95 |   X   |   *   | BULLET
0x96 |   X   |   -   | EN DASH
0x97 |   X   |   --  | EM DASH
0x98 |   X   |   ~   | SMALL TILDE
0x99 |   X   |  (tm) | TRADE MARK SIGN
0x9a |   X   |   s   | LATIN SMALL LETTER S WITH CARON
0x9b |   X   |   >   | SINGLE RIGHT-POINTING ANGLE QUOTATION MARK
0x9c |   X   |   oe  | LATIN SMALL LIGATURE OE
0x9e |   X   |   z   | LATIN SMALL LETTER Z WITH CARON
0x9f |   X   |   Y   | LATIN CAPITAL LETTER Y WITH DIAERESIS

💜 Changing the Tables 💜

Don't like these conversions? Bestie, you can modify them to your heart's content by accessing this module's internal conversion tables. For example, if you wanted zap_cp1252() to use an uppercase E for the euro sign, just do this:

local $Encode::ZapCP1252::ascii_for{"\x80"} = 'E';

Or if, for some reason, you wanted the UTF-8 equivalent for a bullet converted by fix_cp1252() to be a black square, you can assign the bytes (never a Unicode string) like so:

local $Encode::ZapCP1252::utf8_for{"\x95"} = Encode::encode_utf8('X');
Just remember: without local, this would be a global change. In that case, be careful if your code zaps CP1252 elsewhere. Of course, it shouldn't really be doing that. These functions are just for cleaning up messes in one spot in your code, not for making a fundamental part of your text handling. For that, use Encode. 🔥

🌈 See Also 🌈

🍓 Support 🍓

This module is stored in an open GitHub repository <https://github.com/theory/encode-zapcp1252/>. Feel free to fork and contribute! We love a collab 👑

Please file bug reports via GitHub Issues <https://github.com/theory/encode-zapcp1252/issues/> or by sending mail to bug-Encode-CP1252@rt.cpan.org <mailto:bug-Encode-CP1252@rt.cpan.org>.

💅 Author 💅

David E. Wheeler <david@justatheory.com>

🫶 Acknowledgments 🫶

My thanks to Sean Burke for sending me his original method for converting CP1252 gremlins to more-or-less appropriate ASCII characters, and to Karl Williamson for more correct handling of Unicode strings. Legends, both of them ✨

Copyright (c) 2005-2020 David E. Wheeler. Some Rights Reserved.

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

and that's the whole page, bestie 💖 this one's from perl v5.36.0, dated 2022-12-11 — go forth and zap those gremlins, no cap 🎀🦋

📦 where this page comes from