Unicod...apanese(3pm)

πŸ’– Unicode::Japanese(3pm) πŸ’–

from the User Contributed Perl Documentation, bestie 🫢

✨ NAME ✨

Unicode::Japanese β€” Convert encoding of japanese text. That's the whole vibe: it moves Japanese text between encodings. πŸ’…

πŸŽ€ SYNOPSIS πŸŽ€

Okay so first you load it up:

use Unicode::Japanese;
use Unicode::Japanese qw(unijp);

# convert utf8 -> sjis

print Unicode::Japanese->new($str)->sjis;
print unijp($str)->sjis; # same as above.

# convert sjis -> utf8

print Unicode::Japanese->new($str,'sjis')->get;

# convert sjis (imode_EMOJI) -> utf8

print Unicode::Japanese->new($str,'sjis-imode')->get;

# convert zenkaku (utf8) -> hankaku (utf8)

print Unicode::Japanese->new($str)->z2h->get;

πŸ’… DESCRIPTION πŸ’…

The Unicode::Japanese module converts encoding of japanese text from one encoding to another. Like, it's the translator bestie for Japanese text between encodings. ✨

🌈 FEATURES 🌈

  • An instance of Unicode::Japanese internally holds a string in UTF-8. 🧠
  • This module is implemented in two ways: XS and pure perl. If efficiency is important for you, you should build and install the XS module. If you don't want to, or if you can't build the XS module, you may use the pure perl module instead. In that case, only you have to do is to copy Japanese.pm into somewhere in @INC. πŸ› οΈ
  • This module can convert characters from zenkaku (full-width) form to hankaku (half-width) form, and vice versa. Conversion between hiragana (one of two sets of japanese phonetical alphabet) and katakana (another set of japanese phonetical alphabet) is also supported. πŸ”
  • This module has mapping tables for emoji (graphic characters) defined by various japanese mobile phones; DoCoMo i-mode, ASTEL dot-i and J-PHONE J-Sky. Those letters are mapped on Unicode Private Use Area so unicode strings it outputs are still valid even if they contain emoji, and you can safely pass them to other software that can handle Unicode. 🫢
  • This module can map some emoji from one set to another. Different mobile phones define different sets of emoji, so mapping each other is not always possible. But since some emoji exist in two or more sets with similar appearance, this module considers those emoji to be the same. 😭
  • This module uses the mapping table for MS-CP932 instead of the standard Shift_JIS. The Shift_JIS encoding used by MS-Windows (MS-SJIS/MS-CP932) slightly differs from the standard. πŸ’»
  • When the module converts strings from Unicode to Shift_JIS, EUC-JP or ISO-2022-JP, unicode letters which can't be represented in those encodings will be encoded in "&#dddd;" form (decimal character reference). Note, however, that letters in Unicode Private Use Area will be replaced with '?' mark ('QUESTION MARK'; U+003F) instead of being encoded. In addition, encoding to character sets for mobile phones makes every unrepresentable letters being '?' mark. ⚠️
  • On perl-5.8.0 or later, this module handles the UTF-8 flag: the method utf8() returns UTF-8 byte string, and the method getu() returns UTF-8 character string. Currently the method get() returns UTF-8 byte string but this behavior may be changed in the future. Methods like sjis(), jis(), utf8(), and such like return byte string. new(), set(), getcode() methods just ignore the UTF-8 flag of strings they take.

πŸ“ REQUIREMENT πŸ“

  • perl 5.10.x, 5.8.x, etc. (5.004 and later) πŸ”₯
  • (optional) C Compiler. This module supports both XS and Pure Perl. If you have no C Compilers, Unicode::Japanese will be installed as Pure Perl module. 🧰
  • (optional) Test.pm and Test::More for testing. πŸ§ͺ

No other modules are required at run time. Periodt. πŸ’…

πŸ‘‘ METHODS πŸ‘‘

πŸ’« $s = Unicode::Japanese->new($str [, $icode [, $encode]])

Create a new instance of Unicode::Japanese.

Any given parameters will be internally passed to the method set().

πŸ’« $s = unijp($str [, $icode [, $encode]])

Same as Unicode::Jananese->new(...). (yes, keep that spelling, bestie 🫢)

πŸ’« $s->set($str [, $icode [, $encode]])

  • $str: string
  • $icode: optional character encoding (default: 'utf8')
  • $encode: optional binary encoding (default: no binary encodings are assumed)

Store a string into the instance.

Possible character encodings are:

auto
utf8 ucs2 ucs4
utf16-be utf16-le utf16
utf32-be utf32-le utf32
sjis cp932 euc euc-jp jis
sjis-imode sjis-imode1 sjis-imode2
utf8-imode utf8-imode1 utf8-imode2
sjis-doti sjis-doti1
sjis-jsky sjis-jsky1 sjis-jsky2
jis-jsky  jis-jsky1  jis-jsky2
utf8-jsky utf8-jsky1 utf8-jsky2
sjis-au sjis-au1 sjis-au2
jis-au  jis-au1  jis-au2
sjis-icon-au sjis-icon-au1 sjis-icon-au2
euc-icon-au  euc-icon-au1  euc-icon-au2
jis-icon-au  jis-icon-au1  jis-icon-au2
utf8-icon-au utf8-icon-au1 utf8-icon-au2
ascii binary

(see also "SUPPORTED ENCODINGS".)

If you want the Unicode::Japanese detect the character encoding of string, you must explicitly specify 'auto' as the second argument. In that case, the given string will be passed to the method getcode() to guess the encoding.

For binary encodings, only 'base64' is currently supported. If you specify 'base64' as the third argument, the given string will be decoded using Base64 decoder.

Specify 'binary' as the second argument if you want your string to be stored without modification.

When you specify 'sjis-imode' or 'sjis-doti' as the character encoding, any occurences of '&#dddd;' (decimal character reference) in the string will be interpreted and decoded as code point of emoji, just like emoji implanted into the string in binary form.

Since encoded forms of strings in various encodings are not clearly distinctive to each other, it is not always certainly possible to detect what encoding is used for a given string.

When a given string is possibly interpreted as both Shift_JIS and UTF-8 string, this module considers such a string to be encoded in Shift_JIS. And if the encoding is not distinguishable between 'sjis-au' and 'sjis-doti', this module considers it 'sjis-au'.

πŸ’« $str = $s->get

  • $str: string (UTF-8)

Get the internal string in UTF-8.

This method currently returns a byte string (whose UTF-8 flag is turned off), but this behavior may be changed in the future.

If you absolutely want a byte string, you should use the method utf8() instead. And if you want a character string (whose UTF-8 flag is turned on), you have to use the method getu().

πŸ’« $str = $s->getu

  • $str: string (UTF-8)

Get the internal string in UTF-8.

On perl-5.8.0 or later, this method returns a character string with its UTF-8 flag turned on.

πŸ’« $code = $s->getcode($str)

  • $str: string
  • $code: name of character encoding

Detect the character encoding of given string.

Note that this method, exceptionaly, doesn't deal with the internal string of an instance.

To guess the encoding, the following algorithm is used:

(For pure perl implementation)

  1. If the string has an UTF-32 BOM, its encoding is 'utf32'.
  2. If it has an UTF-16 BOM, its encoding is 'utf16'.
  3. If it is valid for UTF-32BE, its encoding is 'utf32-be'.
  4. If it is valid for UTF-32LE, its encoding is 'utf32-le'.
  5. If it contains no ESC characters or bytes whose eighth bit is on, its encoding is 'ascii'. Every ASCII control characters (0x00-0x1F and 0x7F) except ESC (0x1B) are considered to be in the range of 'ascii'.
  6. If it contains escape sequences of ISO-2022-JP, its encoding is 'jis'.
  7. If it contains any emoji defined for J-PHONE, its encoding is 'sjis-jsky'.
  8. If it is valid for EUC-JP, its encoding is 'euc'.
  9. If it is valid for Shift_JIS, its encoding is 'sjis'.
  10. If it contains any emoji defined for au, and everything else is valid for Shift_JIS, its encoding is 'sjis-au'.
  11. If it contains any emoji defined for i-mode, and everything else is valid for Shift_JIS, its encoding is 'sjis-imode'.
  12. If it contains any emoji defined for dot-i, and everything else is valid for Shift_JIS, its encoding is 'sjis-doti'.
  13. If it is valid for UTF-8, its encoding is 'utf8'.
  14. If no conditions above are fulfilled, its encoding is 'unknown'.

(For XS implementation)

  1. If the string has an UTF-32 BOM, its encoding is 'utf32'.
  2. If it has an UTF-16 BOM, its encoding is 'utf16'.
  3. Find all possible encodings that might have been applied to the string from the following:

ascii / euc / sjis / jis / utf8 / utf32-be / utf32-le / sjis-jsky / sjis-imode / sjis-au / sjis-doti

  1. If any encodings have been found possible, this module picks out one encoding having the highest priority among them. The priority order is as follows:

utf32-be / utf32-le / ascii / jis / euc / sjis / sjis-jsky / sjis-imode / sjis-au / sjis-doti / utf8

  1. If no conditions above are fulfilled, its encoding is 'unknown'.

Pay attention to the following pitfalls in the above algorithm:

  • UTF-8 strings might be accidentally considered to be encoded in Shift_JIS.
  • UCS-2 strings (sequence of raw UCS-2 letters in big-endian; each letters has always 2 bytes) can't be detected because they look like nothing but sequences of random bytes whose length is an even number.
  • UTF-16 strings must have BOM to be detected.
  • Emoji are only be recognized if they are implanted into the string in binary form. If they are described in '&#dddd;' form, they aren't considered to be emoji.

Since the XS and pure perl implementations use different algorithms to guess encoding, they may guess differently for the same string. Especially, the pure perl implementation finds Shift_JIS strings containing ESC character (0x1B) to be actually encoded in Shift_JIS but XS implementation doesn't. This is because such strings can hardly be distinguished from 'sjis-jsky'. In addition, EUC-JP strings containing ESC character are also rejected for the same reason.

πŸ’« $code = $s->getcodelist($str)

  • $str: string
  • $code: name of character encodings

Detect the character encoding of given string.

Unlike the method getcode(), getcodelist() returns a list of possible encodings.

πŸ’– $str = $s->conv($ocode, $encode)

bestie this is the ✨main character✨ method. you give it an encoding, it hands you back the instance's internal string encoded that way, periodt.

  • $ocode β€” the character encoding you want. the possible encodings are:
    • utf8, ucs2, ucs4, utf16
    • sjis, cp932, euc, euc-jp, jis
    • sjis-imode, sjis-imode1, sjis-imode2
    • utf8-imode, utf8-imode1, utf8-imode2
    • sjis-doti, sjis-doti1
    • sjis-jsky, sjis-jsky1, sjis-jsky2
    • jis-jsky, jis-jsky1, jis-jsky2
    • utf8-jsky, utf8-jsky1, utf8-jsky2
    • sjis-au, sjis-au1, sjis-au2
    • jis-au, jis-au1, jis-au2
    • sjis-icon-au, sjis-icon-au1, sjis-icon-au2
    • euc-icon-au, euc-icon-au1, euc-icon-au2
    • jis-icon-au, jis-icon-au1, jis-icon-au2
    • utf8-icon-au, utf8-icon-au1, utf8-icon-au2
    • binary

(also go peek at "SUPPORTED ENCODINGS" below πŸ‘€)

okay so some of the mobile-phone encodings have a lil trailing digit like 'sjis-au2'. that digit is the version number of the encoding, bestie. those same encodings also have a no-digit variant like 'sjis-au', and that one is literally the same as the latest version out of all its variants. πŸŽ€

  • $encode β€” optional binary encoding
  • $str β€” string

so you call it and you get the internal string of the instance, encoded with whatever character encoding method you named.

want the result in Base64?? 🫢 just hand 'base64' over as the second argument and boom.

heads up: on perl-5.8.0 or later, the UTF-8 flag of the resulting string is turned off even if you pass 'utf8' as the first argument. no cap.

πŸ’… $s->tag2bin

reads all the decimal character references (&#dddd;) sitting in the instance and swaps them out for the actual single characters they represent. iconic behaviour honestly.

πŸ¦‹ $s->z2h

swaps zenkaku (full-width) letters in the instance for hankaku (half-width) letters.

🌸 $s->h2z

swaps hankaku (half-width) letters in the instance for zenkaku (full-width) letters.

πŸŽ€ $s->hira2kata

turns any hiragana in the instance into katakana.

✨ $s->kata2hira

turns any katakana in the instance into hiragana.


πŸ’œ $str = $s->jis

  • $str β€” byte string in ISO-2022-JP

gives you the instance's internal string encoded in ISO-2022-JP. that's it, that's the vibe.

πŸ’œ $str = $s->euc

  • $str β€” byte string in EUC-JP

same deal but the instance's internal string, encoded in EUC-JP.

πŸ’œ $str = $s->utf8

  • $str β€” byte string in UTF-8

get the internal UTF-8 string of the instance.

on perl-5.8.0 or later the UTF-8 flag of the resulting string gets turned off.

πŸ’œ $str = $s->ucs2

  • $str β€” byte string in UCS-2

you get the internal string of the instance as a sequence of raw UCS-2 letters in big-endian. and listen β€” this is not the same as UTF-16BE, because a raw UCS-2 sequence has literally no concept of a surrogate pair. like none. 🫠

πŸ’œ $str = $s->ucs4

  • $str β€” byte string in UCS-4

internal string of the instance as a sequence of raw UCS-4 letters in big-endian. practically speaking this is the same thing as UTF-32BE.

πŸ’œ $str = $s->utf16

  • $str β€” byte string in UTF-16

internal string of the instance, encoded in UTF-16 in big-endian, with no BOM prepended.

πŸ’œ $str = $s->sjis

  • $str β€” byte string in Shift_JIS

internal string of the instance encoded in Shift_JIS (which is MS-SJIS / MS-CP932).

πŸ’œ $str = $s->sjis_imode

  • $str β€” byte string in 'sjis-imode'

internal string of the instance, encoded in 'sjis-imode'.

πŸ’œ $str = $s->sjis_imode1

  • $str β€” byte string in 'sjis-imode1'

internal string of the instance, encoded in 'sjis-imode1'.

πŸ’œ $str = $s->sjis_imode2

  • $str β€” byte string in 'sjis-imode2'

internal string of the instance, encoded in 'sjis-imode2'.

πŸ’œ $str = $s->sjis_doti

  • $str β€” byte string in 'sjis-doti'

internal string of the instance, encoded in 'sjis-doti'.

πŸ’œ $str = $s->sjis_jsky

  • $str β€” byte string in 'sjis-jsky'

internal string of the instance, encoded in 'sjis-jsky'.

πŸ’œ $str = $s->sjis_jsky1

  • $str β€” byte string in 'sjis-jsky1'

internal string of the instance, encoded in 'sjis-jsky1'.

πŸ’œ $str = $s->sjis_jsky

  • $str β€” byte string in 'sjis-jsky'

internal string of the instance, encoded in 'sjis-jsky'. (yes, again β€” the docs list it twice, so we list it twice 🫢)

πŸ’œ $str = $s->sjis_icon_au

  • $str β€” byte string in 'sjis-icon-au'

internal string of the instance, encoded in 'sjis-icon-au'.


🧚 $str_arrayref = $s->strcut($len)

  • $len β€” maximum length of each chunks (measured in number of full-width characters)
  • $str_arrayref β€” reference to array of strings

chops the instance's internal string up into chunks of a given length. so satisfying ngl.

on perl-5.8.0 or later, the UTF-8 flags of each chunk get turned on.

🧚 $len = $s->strlen

  • $len β€” character width of the internal string

calculates the character width of the internal string. half-width characters count as width of one unit, and full-width characters count as width of two units. the math is mathing. πŸ’…

🧚 $s->join_csv(@values);

  • @values β€” array of strings

builds a line of CSV out of the arguments and shoves it into the instance. the resulting line has a trailing line break ("\n"), okay so don't come for me when it does. πŸ’–

🧚 @values = $s->split_csv;

  • @values β€” array of strings

parses a line of CSV sitting in the instance and returns each column. the line gets chomp()ed before it gets parsed.

if the internal string was decoded from 'binary' encoding (peep the methods new() and set()), the UTF-8 flags of the resulting array of strings are turned off. otherwise the flags are turned on.


🌈 SUPPORTED ENCODINGS

here's the whole menu, bestie:

+---------------+----+-----+-------+
|encoding       | in | out | guess |
+---------------+----+-----+-------+
|auto           : OK : --  | ----- |
+---------------+----+-----+-------+
|utf8           : OK : OK  | OK    |
|ucs2           : OK : OK  | ----- |
|ucs4           : OK : OK  | ----- |
|utf16-be       : OK : --  | ----- |
|utf16-le       : OK : --  | ----- |
|utf16          : OK : OK  | OK(#) |
|utf32-be       : OK : --  | OK    |
|utf32-le       : OK : --  | OK    |
|utf32          : OK : --  | OK(#) |
+---------------+----+-----+-------+
|sjis           : OK : OK  | OK    |
|cp932          : OK : OK  | ----- |
|euc            : OK : OK  | OK    |
|euc-jp         : OK : OK  | ----- |
|jis            : OK : OK  | OK    |
+---------------+----+-----+-------+
|sjis-imode     : OK : OK  | OK    |
|sjis-imode1    : OK : OK  | ----- |
|sjis-imode2    : OK : OK  | ----- |
|utf8-imode     : OK : OK  | ----- |
|utf8-imode1    : OK : OK  | ----- |
|utf8-imode2    : OK : OK  | ----- |
+---------------+----+-----+-------+
|sjis-doti      : OK : OK  | OK    |
|sjis-doti1     : OK : OK  | ----- |
+---------------+----+-----+-------+
|sjis-jsky      : OK : OK  | OK    |
|sjis-jsky1     : OK : OK  | ----- |
|sjis-jsky2     : OK : OK  | ----- |
|jis-jsky       : OK : OK  | ----- |
|jis-jsky1      : OK : OK  | ----- |
|jis-jsky2      : OK : OK  | ----- |
|utf8-jsky      : OK : OK  | ----- |
|utf8-jsky1     : OK : OK  | ----- |
|utf8-jsky2     : OK : OK  | ----- |
+---------------+----+-----+-------+
|sjis-au        : OK : OK  | OK    |
|sjis-au1       : OK : OK  | ----- |
|sjis-au2       : OK : OK  | ----- |
|jis-au         : OK : OK  | ----- |
|jis-au1        : OK : OK  | ----- |
|jis-au2        : OK : OK  | ----- |
|sjis-icon-au   : OK : OK  | ----- |
|sjis-icon-au1  : OK : OK  | ----- |
|sjis-icon-au2  : OK : OK  | ----- |
|euc-icon-au    : OK : OK  | ----- |
|euc-icon-au1   : OK : OK  | ----- |
|euc-icon-au2   : OK : OK  | ----- |
|jis-icon-au    : OK : OK  | ----- |
|jis-icon-au1   : OK : OK  | ----- |
|jis-icon-au2   : OK : OK  | ----- |
|utf8-icon-au   : OK : OK  | ----- |
|utf8-icon-au1  : OK : OK  | ----- |
|utf8-icon-au2  : OK : OK  | ----- |
+---------------+----+-----+-------+
|ascii          : OK : --  | OK    |
|binary         : OK : OK  | ----- |
+---------------+----+-----+-------+

and the footnote, don't skip it: (#): guessed when it has bom. πŸ‘‘

πŸ” GUESSING ORDER

the order it guesses stuff in, exactly like this:

  1. utf32 (#)
  2. utf16 (#)
  3. utf32-be
  4. utf32-le
  5. ascii
  6. jis
  7. sjis-jsky (pp)
  8. euc
  9. sjis
  10. sjis-jsky (xs)
  11. sjis-au
  12. sjis-imode
  13. sjis-doti
  14. utf8
  15. unknown

✨ DESCRIPTION OF UNICODE MAPPING ✨

so here's how transcoding between Unicode encodings and the other ones actually goes down:

πŸ’  Shift_JIS

this module uses the mapping table of MS-CP932, the one living right here:

ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT

when the module tries to convert a Unicode string to Shift_JIS, it represents most letters that aren't available in Shift_JIS as decimal character references ('&#dddd;'). but there is one exception, bestie, don't forget it: every graphic character made for mobile phones gets replaced with a '?' mark instead.

and for the variants of Shift_JIS defined for mobile phones? every unrepresentable character gets replaced with '?', unlike the plain Shift_JIS. different rules, no cap.

πŸ’  EUC-JP/ISO-2022-JP

this module does not directly convert a Unicode string from/to EUC-JP or ISO-2022-JP. instead it converts from/to Shift_JIS first, and then does the rest of the translation. so characters that aren't available in Shift_JIS can't be properly translated. so it goes. 😭

πŸ’  DoCoMo i-mode

this module maps emoji in the range of F800 - F9FF to U+0FF800 - U+0FF9FF.

πŸ’  ASTEL dot-i

this module maps emoji in the range of F000 - F4FF to U+0FF000 - U+0FF4FF.

πŸ’  J-PHONE J-SKY

the encoding method defined by J-SKY goes like this: first an escape sequence "\e\$" comes along to indicate the beginning of emoji, then the first byte of an emoji comes next, then the second bytes of at least one emoji comes next, and then "\x0f" comes last to indicate the end of emoji. and if a string contains a series of emoji whose first bytes are identical, that sequence can be compressed by cascading the second bytes of them onto the single first byte. it's giving compression queen. πŸ’…

this module treats a pair of those first and second bytes as one letter, and maps them from 4500 - 47FF to U+0FFB00 - U+0FFDFF.

when the module encodes J-SKY emoji, it performs that compression automatically. yes, for free. πŸŽ€

πŸ’  AU

this module maps AU emoji to U+0FF500 - U+0FF6FF.

πŸ’  PurePerl mode

use Unicode::Japanese qw(PurePerl);

if you want to explicitly take the pure perl implementation, pass 'PurePerl' to the argument of the use statement. that's the whole move, bestie. πŸ¦‹

πŸ› BUGS πŸ›

okay bestie, if you find a bug or you're like "girl i NEED this feature", pls report it to bug-unicode-japanese at rt.cpan.org or hit up <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Unicode-Japanese>. And the iconic part? If you file it through the web interface, any progress on your report gets automatically sent back to you. no ghosting, we love that for you πŸ’–

  • This module doesn't directly convert Unicode string from/to EUC-JP or ISO-2022-JP β€” it once converts from/to Shift_JIS and then do the rest translation. So characters which aren't available in the Shift_JIS can not be properly translated. it's giving middleman energy, and the middleman has limits 😭
  • The XS implementation of getcode() fails to detect the encoding when the given string contains \e while its encoding is EUC-JP or Shift_JIS. yep, that's a real one, don't come for me
  • Japanese.pm is composed of textual perl script and binary character conversion table β€” If you transfer it on FTP using ASCII mode, the file will collapse. ASCII mode is NOT it, bestie, please use binary mode, i am literally begging πŸ™ˆ

🫢 SUPPORT 🫢

you can find documentation for this module with the perldoc command.

perldoc Unicode::Japanese

You can find more information at:


🌸 CREDITS 🌸

Thanks very much to:

NAKAYAMA Nao

SUGIURA Tatsuki & Debian JP Project

legends, every single one of them, they ate and left no crumbs πŸ‘‘


Copyright 2001-2008 SANO Taku (SAWATARI Mikage) and YAMASHINA Hio, all rights reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.


okay that's the whole page bestie, you're officially a Unicode::Japanese girlie now πŸŽ€ this was perl v5.42.3, dated 2026-08-19, from the Unicod...apanese(3pm) page β€” go forth and convert responsibly, drink water, and don't transfer anything in ASCII mode πŸ’…βœ¨πŸ¦‹

πŸ“¦ where this page comes from