Map(3pm)

📦
version 0.112 · 10 versions
, in ubuntu questing, ubuntu resolute

✨ Unicode::Map 💖 (Map(3pm)) ✨

okay bestie, this is the manual page for Unicode::Map, from the User Contributed Perl Documentation collection 📚🦋 it's the Perl module that converts strings to and from UTF-16 Unicode 💅

🎀 NAME 🎀

Unicode::Map - maps charsets from and to utf16 unicode 🌸

🪄 SYNOPSIS 🪄

use Unicode::Map();

$Map = new Unicode::Map("ISO-8859-1");

$utf16 = $Map -> to_unicode ("Hello world!");
  => $utf16 == "\0H\0e\0l\0l\0o\0 \0w\0o\0r\0l\0d\0!"

$locale = $Map -> from_unicode ($utf16);
  => $locale == "Hello world!"

so the vibe is: make a Map object for a charset, call to_unicode to go into utf16, and from_unicode to come back out 💜 there's a more detailed breakdown below, don't worry 🫶

2do: short note about perl's Unicode perspectives. 🙈

💜 DESCRIPTION 💜

okay so this module converts strings from and to 2-byte Unicode UCS2 format ✨ every mapping goes through 2-byte UTF16 encodings, NOT through 1-byte UTF8 encoding. if you need to transform stuff into or out of those, use Unicode::String 🧚

for historical reasons, this module lives alongside Unicode::Map8 🕰️ please use Unicode::Map8 unless you actually need to handle two-byte character sets, like chinese GB2312 🇨🇳 honestly though, if you stick to the basic functionality (see the documentation) you can use either module and it's the same energy 💅

ngl this module will probably disappear eventually, because Unicode mapping support needs to make its way into perl's core 😭 if you're into working on this area, please don't hesitate to contact Gisle Aas!

this module can't deal with utf8 directly 🙅‍♀️ use Unicode::String to convert utf8 to utf16 and back again.

character mapping follows the data in the binary mapfiles inside the Unicode::Map hierarchy 🗂️ you can also create binary mapfiles with this module, which means you can install your own custom character sets 🎨 check out mkmapfile or the file REGISTRY in the Unicode::Map hierarchy for more.

👑 CONVERSION METHODS 👑

these are probably the only methods you'll ever need from this module 💖 their usage is compatible with Unicode::Map8, so no stress 🦋

🌈 new

$Map = new Unicode::Map("GB2312-80")

makes and returns a new Map object for the GB2312-80 encoding ✨

🌈 from_unicode

$dest = $Map -> from_unicode ($src)

makes a string in locale charset representation from the utf16 encoded string $src 🍓

🌈 to_unicode

$dest   = $Map -> to_unicode ($src)

makes a string in utf16 representation from $src 🧚

🌈 to8

an alias for from_unicode. it's here for compatibility with Unicode::Map8 🫶

🌈 to16

an alias for to_unicode. also here for compatibility with Unicode::Map8 💅

⚠️ WARNINGS ⚠️

you can tell Unicode::Map to throw warnings when you use something deprecated or incompatible 🙈 the constants are WARN_DEFAULT, WARN_DEPRECATION or WARN_COMPATIBILITY. the latter two can be ored together, so you can combine them 🔥

no special warnings:

$Unicode::Map::WARNINGS = Unicode::Map::WARN_DEFAULT

warnings for deprecated usage:

$Unicode::Map::WARNINGS = Unicode::Map::WARN_DEPRECATION

warnings for incompatible usage:

$Unicode::Map::WARNINGS = Unicode::Map::WARN_COMPATIBILITY

🛠️ MAINTAINANCE METHODS 🛠️

note: these methods are ONLY for maintaining Unicode::Map itself 🔧 using any of them will lead to programs that are incompatible with Unicode::Map8, so be careful bestie 💀

🌷 alias

@list = $Map -> alias ($csid)

returns a list of alias names for the character set $csid 🌸

🌷 mapping

$path = $Map -> mapping ($csid)

returns the absolute path of the binary character mapping for the character set $csid, according to the REGISTRY file of Unicode::Map 🗺️

🌷 id

$real_id||"" = $Map -> id ($test_id)

returns a valid character set identifier $real_id if $test_id is a valid character set name or alias name, according to the REGISTRY file of Unicode::Map ✨ (if it isn't valid, you get "" back 🙈)

🌷 ids

@ids = $Map -> ids()

returns a list of all the character set names defined in the REGISTRY file 🌈

🌷 read_text_mapping

1||0 = $Map -> read_text_mapping ($csid, $path, $style)

reads a text mapping of style $style, named $csid, from the filename $path 📖 you can then save the mapping to a file with the method write_binary_mapping. $style can be:

  • "unicode" — a text mapping as of ftp://ftp.unicode.org/MAPPINGS/
  • "" — same as "unicode"
  • "reverse" — similar to unicode, but both columns are switched 🔄
  • "keld" — a text mapping as of ftp://dkuug.dk/i18n/charmaps/

🌷 src

$path = $Map -> src ($csid)

returns the path of the textual character mapping for the character set $csid, according to the REGISTRY file of Unicode::Map 📄

🌷 style

$path = $Map -> style ($csid)

returns the style of the textual character mapping for the character set $csid, according to the REGISTRY file of Unicode::Map 🎀

🌷 write_binary_mapping

1||0 = $Map -> write_binary_mapping ($csid, $path)

stores a mapping that you've loaded with read_text_mapping into the file $path 💾

🪦 DEPRECATED METHODS 🪦

some functionality isn't promoted anymore 🙃

☠️ noise

deprecated! don't use this any longer.

☠️ reverse_unicode

deprecated! use Unicode::String::byteswap instead.

🧩 BINARY MAPPINGS 🧩

this is the structure of binary mapfiles 🤓

unicode character mapping tables have sequences of sequential keys and sequential values, and this property is what makes it possible to crunch the maps easily 🍬 n (0<n<256) sequential characters are stored as a bytecount n plus the first character code, key_start. for these subsequences, the matching value sequences get crunched together too 💜 the value 0 is used to start an extended information block (which is only partially implemented, though 😭).

there are two ways you could make a binary mapfile 🧠 the first method would be to write a list of all the key codes first, and THEN a list of all the value codes. the second method, which is the one used here, appends the crunched value code lists right after each partial key code list. this keeps value codes a little closer to their key codes 🫶

heads up: the file format is still in a very liquid state 🫠 don't rely on it staying like this, don't assume the description is bug-free, and don't assume every feature is implemented.

🗂️ STRUCTURE:

<main>:
       offset  structure     value

       0x00    word          0x27b8   (magic)
       0x02    @(<extended> || <submapping>)

the mapfile ends with extended mode <end> in the main stream 🏁

<submapping>:
       0x00    byte != 0     charsize1 (bits)
       0x01    byte          n1 number of chars for one entry
       0x02    byte          charsize2 (bits)
       0x03    byte          n2 number of chars for one entry
       0x04    @(<extended> || <key_seq> || <key_val_seq)

       bs1=int((charsize1+7)/8), bs2=int((charsize2+7)/8)

one submapping ends when a <mapend> entry shows up 🔚

<key_val_seq>:
       0x00    size=0|1|2|4  n, number of sequential characters
       size    bs1           key1
       +bs1    bs2           value1
       +bs2    bs1           key2
       +bs1    bs2           value2
       ...

key_val_seq ends if either the file ends (that's n = infinite mode) or n pairs have been read 🌊

<key_seq>:
       0x00    byte          n, number of sequential characters
       0x01    bs1           key_start, first character of sequence
       1+bs1   @(<extended> || <val_seq>)

a key sequence starts with a byte count telling you how long the sequence is 📏 it's followed by the key start code, and after that comes a list of value sequences. that list ends once sum(m) equals n ✨

<val_seq>:
       0x00    byte          m, number of sequential characters
       0x01    bs2           val_start, first character of sequence
<extended>:
       0x00    byte          0
       0x01    byte          ftype
       0x02    byte          fsize, size of following structure
       0x03    fsize bytes   something

for future extensions or private use, you can slot in 1..255 byte long streams here 🧁 ftype can be 30..255, and values 0..29 are reserved 🚫 the modi aren't fully defined yet and could change, and they'll be explained later on 🔮

🚧 TO BE DONE 🚧

  • ✨ something clever for when a character has no translation
  • 🔁 direct charset to charset mapping
  • 🏎️ better performance
  • 📜 support for mappings according to RFC 1345

🔗 SEE ALSO 🔗

🫶 AUTHOR 🫶

Martin Schwartz <martin@nacho.de> 💜

and that's the whole tea, bestie ☕💖 this was the Map(3pm) page from perl v5.40.0, dated 2024-10-20, and it's the manual for Unicode::Map 🧚✨ go be iconic 👑🦋

📦 where this page comes from