Unicod...ese::JA(3pm)

📦
version 0.51 · 6 versions
, in ubuntu stonking

💖 NAME 💖

Unicode::Japanese::JA — 日本語文字コード変換 — bestie, it's giving Japanese character code conversion, and it's gonna be your whole personality 💅✨

(You're reading the Unicode::Japanese::JA(3pm) page from User Contributed Perl Documentation. Yes, we're doing all of it. No skimming. 🫶)

🎀 概要 (SYNOPSIS) 🎀

okay so here's the vibe, the actual receipts, copy-paste ready, don't @ me:

use Unicode::Japanese;
use Unicode::Japanese qw(unijp);

# convert utf8 -> sjis

print Unicode::Japanese->new($str)->sjis;
print unijp($str)->sjis; # same as above.

# convert sjis -> utf8

print Unicode::Japanese->new($str,'sjis')->get;

# convert sjis (imode_EMOJI) -> utf8

print Unicode::Japanese->new($str,'sjis-imode')->get;

# convert zenkaku (utf8) -> hankaku (utf8)

print Unicode::Japanese->new($str)->z2h->get;

like. look at that. unijp()? iconic shorthand. ->z2h->get? chained like a queen. 👑

🦋 説明 (DESCRIPTION) 🦋

Unicode::Japanese は，日本語の文字コードの相互変換を行うモジュールです． — fr fr, this module takes Japanese character codes and swaps them back and forth for you. that's the whole job and it does it slayfully. 🌈

✨ 機能 (FEATURES) ✨

  • 💖 A Unicode::Japanese instance holds its string in UTF-8. internally. always. that's the home base.
  • 🦋 XS and non-XS are both supported, and you get to pick your fighter. the XS version is for when you need performance, the No-XS version is for when you just want things easy (literally just copy Japanese.pm and it works — no build drama, no compiler tears).
  • 🌸 Full-width ⇄ half-width conversion? supported. Katakana ⇄ hiragana conversion? also supported. we contain multitudes.
  • 🎀 Phone emoji (DoCoMo i-mode, KDDI AU, Softbank Mobile, ASTEL dot-i) get mapped into the Unicode Private Use Area, which means you can safely keep them in a DB and stuff. no corrupting your data, we don't do that here. 💀
  • 🧚 Emoji with the same image can be converted between different phone carriers. so your AU emoji can become an i-mode emoji. cross-platform friendship, literally.
  • ⭐ SJIS is treated as MS-CP932 when it's mapped against Unicode. no cap.
  • 😭 When you're mapping Unicode -> SJIS (and also EUC-JP/JIS), any character that SJIS can't express gets turned into the &#dddd; form. but — and this is the important part — emoji sitting in the Unicode Private Use Area become '?'. also, when you're converting for a phone, every character that isn't supported becomes '?'. like, all of them. be normal about it.
  • 🍓 From Perl-5.8.0 onward, this handles the utf8 flag for you. To grab a utf-8 バイト (byte) sequence, use the utf8() method. To grab a utf-8 文字 (character) sequence, use the getu() method.

right now, the get() method returns a utf-8 バイト (byte) sequence — and yeah, that could change in the future. heads up. 🙈

sjis(), jis(), utf8(), etc.. — those methods return byte sequences. and for the input of the new, set, and getcode methods? it does not care whether it's utf8-flaged or bytes. it accepts both. flexible queen.

🌸 動作に必要なもの (REQUIREMENTS) 🌸

what it needs to run, keep it honest:

  • perl 5.10.x, 5.8.x, etc. (5.004 以降 — that's 5.004 and later).
  • (optional, you're fine without it) a C compiler. this module supports both XS and Pure Perl. when there's no C compiler around, Unicode::Japanese just installs itself as a Pure Perl module. adaptable.
  • (optional, you're fine without it) Test.pm 及び Test::More — for testing.
  • and for the record: there are no modules required at runtime. none. zero. clean. ✨

👑 メソッド (METHODS) 👑

the main characters. let's meet them.

$s = Unicode::Japanese->new($str [, $icode [, $encode]])

Spins up a brand new Unicode::Japanese instance.

If you pass parameters, they get handed straight over to the "set" method.

$s = unijp($str [, $icode [, $encode]])

Same thing as Unicode::Janaese->new(...).

yes it literally says Janaese in the original and i am NOT fixing it, that's history, iykyk 😭

$s->set($str [, $icode [, $encode]])

  • $str: 文字列 — the string
  • $icode: 文字コード指定．省略可．省略時は 'utf8' — the character code. optional. if you skip it, it assumes 'utf8'
  • $encode: バイナリ符号化方式．省略可． — the binary encoding method. optional.

Sets a string into the instance. If you leave out the character code, it's treated as UTF-8.

The character codes you can use:

auto
utf8 ucs2 ucs4
utf16-be utf16-le utf16
utf32-be utf32-le utf32
sjis cp932 euc euc-jp jis
sjis-imode sjis-imode1 sjis-imode2
utf8-imode utf8-imode1 utf8-imode2
sjis-doti sjis-doti1
sjis-jsky sjis-jsky1 sjis-jsky2
jis-jsky  jis-jsky1  jis-jsky2
utf8-jsky utf8-jsky1 utf8-jsky2
sjis-au sjis-au1 sjis-au2
jis-au  jis-au1  jis-au2
sjis-icon-au sjis-icon-au1 sjis-icon-au2
euc-icon-au  euc-icon-au1  euc-icon-au2
jis-icon-au  jis-icon-au1  jis-icon-au2
utf8-icon-au utf8-icon-au1 utf8-icon-au2
ascii binary

(see also: "サポートされているエンコーディング" — "supported encodings".)

If you want the character code auto-detected, you have to specify 'auto'. When you use 'auto', the auto-detection is done by the getcode() method.

For the binary encoding method, 'base64' is the only thing you can pass. If you specify base64, the string gets base64-decoded first and then becomes the Unicode::Japanese class's string. cute little pipeline.

If you want the string you passed in to be stored exactly as-is, without being touched, specify 'binary' as the character code.

In the case of sjis-imode and sjis-doti, any &#dddd; in the string gets converted into an emoji. 🎀

the character code areas can overlap with each other, which means the auto-detection is not guaranteed to be right. foreshadowing.

For a string that can be interpreted as both sjis and utf8, it returns sjis. For a string that can be interpreted as both sjis-au and sjis-doti, it returns sjis-au.

$str = $s->get

  • $str: 文字列(UTF-8) — string (UTF-8)

Pulls the string out as UTF-8 code.

Right now it returns a バイト (byte) sequence, but it might change in the future. okay?? 😭

If you need a byte sequence, use the utf8() method. If you need a character string, use the getu() method. that's the recommendation. take it. 🫶

$str = $s->getu

  • $str: 文字列(UTF-8) — string (UTF-8)

Pulls the string out as UTF-8 code.

From Perl-5.8.0 onward, it returns it as a utf-8 string with the utf-8 flag set on it. the good version. ✨

$code = $s->getcode($str)

  • $str: 文字列 — string
  • $code: 文字コードを表す文字列 — a string representing the character code

Auto-detects the character code of the string you passed in ($str).

⚠️ important: this function is the exception to the rule — it does NOT detect the code of the string held inside the instance. like, read that twice. it's detecting the argument, bestie.

When it auto-detects, here's the algorithm it runs:

(PurePerlの場合)

  1. If there's a UTF-32 BOM, it decides utf32.
  2. If there's a UTF-16 BOM, it decides utf16.
  3. If it's a valid form as UTF-32BE, it decides utf32-be.
  4. If it's a valid form as UTF-32LE, it decides utf32-le.
  5. If there's no ESC character and no character with the 8th bit set, it decides ascii. ASCII control characters excluding ESC (0x00-0x1F and 0x7F) are considered to be within the ascii range.
  6. If a JIS escape sequence is present, it decides jis.
  7. If a J-PHONE emoji is present, it decides sjis-jsky.
  8. If it's a valid form as EUC-JP code, it decides euc.
  9. If it's a valid form as SJIS code, it decides sjis.
  10. If it's a valid form as SJIS code and au emoji, it decides sjis-au.
  11. If it's a valid form as SJIS and i-mode emoji, it decides sjis-imode.
  12. If it's a valid form as SJIS and dot-i emoji, it decides sjis-doti.
  13. If it's a valid form as UTF-8, it decides utf8.
  14. If it doesn't match any of the above, it decides unknown.

(XSの場合)

  1. If there's a UTF-32 BOM, it decides utf32.
  2. If there's a UTF-16 BOM, it decides utf16.
  3. For the following codes, it checks whether the string is valid code by using a state transition:

ascii / euc / sjis / jis / utf8 / utf32-be / utf32-le / sjis-jsky / sjis-imode / sjis-au / sjis-doti

  1. From the ones that stayed valid all the way to the end, it picks ONE using the following priority and decides that's what it is:

utf32-be / utf32-le / ascii / jis / euc / sjis / sjis-jsky / sjis-imode / sjis-au / sjis-doti / utf8

  1. If it doesn't match any of the above, it decides unknown.

Because of the algorithm above, please keep the following in mind:

  • A UTF-8 string can still end up being treated as SJIS code. 😭
  • UCS2 cannot be auto-detected.
  • UTF-16 is only auto-recognized when there's a BOM.
  • Phone emoji can only be recognized when the emoji is there directly in binary. If it's written in the &#dddd; form, auto-detection of the phone emoji does not happen.

XS and PurePerl use different detection algorithms, so they can give you different results. In particular, for an sjis string that contains an escape character, PurePerl recognizes it as sjis but XS does not. That's because it would become indistinguishable from sjis-jsky. Also, to prevent misrecognition caused by this behavior, euc-jp similarly no longer accepts escape characters.

$code = $s->getcodelist($str)

  • $str: 文字列 — string
  • $code: 文字コードを表す文字列 — a string representing the character code

Auto-detects the character code of the string you passed in ($str).

Unlike getcode, this returns a list of every acceptable character code. the whole menu, not just the pick. ✨

$str = $s->conv($ocode, $encode)

  • $ocode: 出力コード (以下から指定) — the output code (specify from the following):
utf8 ucs2 ucs4 utf16
sjis cp932 euc euc-jp jis
sjis-imode sjis-imode1 sjis-imode2
utf8-imode utf8-imode1 utf8-imode2
sjis-doti sjis-doti1
sjis-jsky sjis-jsky1 sjis-jsky2
jis-jsky  jis-jsky1  jis-jsky2
utf8-jsky utf8-jsky1 utf8-jsky2
sjis-au sjis-au1 sjis-au2
jis-au  jis-au1  jis-au2
sjis-icon-au sjis-icon-au1 sjis-icon-au2
euc-icon-au  euc-icon-au1  euc-icon-au2
jis-icon-au  jis-icon-au1  jis-icon-au2
utf8-icon-au utf8-icon-au1 utf8-icon-au2
binary

(see also: "サポートされているエンコーディング" — "supported encodings".)

For the phone-oriented character codes, the ones with a number on the end represent a bigger emoji set the bigger the number is (the newest model's emoji set, basically). The ones with no number are identical to the character code with the largest number.

  • $encode: バイナリ符号化方式．省略可． — the binary encoding method. optional.
  • $str: 文字列 — string

So for the method that converts your string into a charset you pick and then hands it back to you:

For the encoding you can literally only pass 'base64'. That is the entire menu, bestie. If you do pass base64, what comes back is a base64-encoded string. ✨

And on perl-5.8.0 and later? The output is a byte string with no utf-8 flag on it. No flag, no notes, periodt.

  • $s->tag2bin — takes every &#dddd;-looking thing chilling inside your string and swaps it out for the actual character it stands for. 🎀
  • $s->z2h — full-width ➡️ half-width. glow-down but make it useful 💅
  • $s->h2z — half-width ➡️ full-width. the reverse slay.
  • $s->hira2kata — hiragana ➡️ katakana.
  • $s->kata2hira — katakana ➡️ hiragana.
  • $str = $s->jis
    • $str — the byte string in JIS encoding format
    • hands you the string in JIS (ISO-2022-JP) code.
  • $str = $s->euc
    • $str — the byte string in euc-jp encoding format
    • hands you the string in EUC-JP code.
  • $str = $s->utf8
    • $str — the byte string in utf-8 encoding format
    • hands you the string in UTF-8 code.
    • and yeah, even on perl-5.8.0 and up, it still returns you a plain byte string. no utf-8 flag, no exceptions 🙈
  • $str = $s->ucs2
    • $str — the byte string in ucs2 encoding format
    • hands you the string in UCS2 code.
  • $str = $s->ucs4
    • $str — the byte string in ucs4 encoding format
    • hands you the string in UCS4 code.
  • $str = $s->utf16
    • $str — the byte string in ucs-16 encoding format
    • hands you the string in UTF-16 code. No BOM gets attached, and it comes back in big endian format. that's just the deal 🦋
  • $str = $s->sjis
    • $str — the byte string in sjis encoding format
    • hands you the string in SJIS (MS-CP932) code.
  • $str = $s->sjis_imode
    • $str — the byte string in sjis/imode emoji encoding format
    • hands you the string in SJIS code built for i-mode devices. it's the alias for the newest imode emoji, iykyk.
  • $str = $s->sjis_imode1
    • $str — the byte string in sjis/imode emoji encoding format
    • hands you the string in SJIS code for i-mode devices. this one contains only the basic emoji.
  • $str = $s->sjis_imode2
    • $str — the byte string in sjis/imode emoji encoding format
    • hands you the string in SJIS code for i-mode devices. this one contains the basic emoji plus the extended emoji.
  • $str = $s->sjis_doti
    • $str — the byte string in sjis/dot-i emoji encoding format
    • hands you the string in SJIS code for dot-i devices.
  • $str = $s->sjis_jsky
    • $str — the byte string in sjis/j-sky emoji encoding format
    • hands you the string in SJIS code for j-sky devices. it's the alias for the newest j-sky emoji (in VERSION 0.15, that's jsky2).
  • $str = $s->sjis_jsky1
    • $str — the byte string in sjis/j-sky emoji encoding format
    • hands you the string in SJIS code for j-sky devices. this one contains only the Page 1～3 emoji.
  • $str = $s->sjis_jsky — (yep, the page lists this exact same line twice, and I'm giving it to you exactly how it's written 💀)
    • $str — the byte string in sjis/j-sky emoji encoding format
    • hands you the string in SJIS code for j-sky devices. this one contains the Page 1～6 emoji.
  • $str = $s->sjis_icon_au
    • $str — the byte string in sjis/AU icon tag encoding format
    • hands you the string in SJIS code for AU devices.
  • $str_arrayref = $s->strcut($len)
    • $len — the number of characters to split by (full-width equivalent)
    • $str_arrayref — the string
    • splits things up into an array of strings, each one no longer than the character count (full-width) you gave in $len.
    • every element of that array is a utf-8 string that has the utf-8 flag. ✨
  • $len = $s->strlen
    • $len — the display width of the string
    • here's the tea: if you call length() on UTF-8 text, a full-width character ends up counting as length 3 per character, which is lowkey tragic. use this method instead and, just like the classic SJIS behaviour, a full-width character counts as length 2 per character. 🫶
  • $s->join_csv(@values);
    • @values — the data array
    • turns your array into a CSV string and stashes it in the instance. and yes, a newline ("\n") gets appended to the end of the string. no cap.
  • @values = $s->split_csv;
    • @values — the data array
    • treats whatever string is stored in the instance as CSV and splits it into an array. the newline ("\n") at the end of the string gets removed before the splitting happens, bestie.
    • if the input is not binary, it returns a utf-8 string. if it was binary, it returns a byte string.

✨ サポートされているエンコーディング ✨

the whole lineup, exactly as printed:

+---------------+----+-----+-------+
|encoding       | in | out | guess |
+---------------+----+-----+-------+
|auto           : OK : --  | ----- |
+---------------+----+-----+-------+
|utf8           : OK : OK  | OK    |
|ucs2           : OK : OK  | ----- |
|ucs4           : OK : OK  | ----- |
|utf16-be       : OK : --  | ----- |
|utf16-le       : OK : --  | ----- |
|utf16          : OK : OK  | OK(#) |
|utf32-be       : OK : --  | OK    |
|utf32-le       : OK : --  | OK    |
|utf32          : OK : --  | OK(#) |
+---------------+----+-----+-------+
|sjis           : OK : OK  | OK    |
|cp932          : OK : OK  | ----- |
|euc            : OK : OK  | OK    |
|euc-jp         : OK : OK  | ----- |
|jis            : OK : OK  | OK    |
+---------------+----+-----+-------+
|sjis-imode     : OK : OK  | OK    |
|sjis-imode1    : OK : OK  | ----- |
|sjis-imode2    : OK : OK  | ----- |
|utf8-imode     : OK : OK  | ----- |
|utf8-imode1    : OK : OK  | ----- |
|utf8-imode2    : OK : OK  | ----- |
+---------------+----+-----+-------+
|sjis-doti      : OK : OK  | OK    |
|sjis-doti1     : OK : OK  | ----- |
+---------------+----+-----+-------+
|sjis-jsky      : OK : OK  | OK    |
|sjis-jsky1     : OK : OK  | ----- |
|sjis-jsky2     : OK : OK  | ----- |
|jis-jsky       : OK : OK  | ----- |
|jis-jsky1      : OK : OK  | ----- |
|jis-jsky2      : OK : OK  | ----- |
|utf8-jsky      : OK : OK  | ----- |
|utf8-jsky1     : OK : OK  | ----- |
|utf8-jsky2     : OK : OK  | ----- |
+---------------+----+-----+-------+
|sjis-au        : OK : OK  | OK    |
|sjis-au1       : OK : OK  | ----- |
|sjis-au2       : OK : OK  | ----- |
|jis-au         : OK : OK  | ----- |
|jis-au1        : OK : OK  | ----- |
|jis-au2        : OK : OK  | ----- |
|sjis-icon-au   : OK : OK  | ----- |
|sjis-icon-au1  : OK : OK  | ----- |
|sjis-icon-au2  : OK : OK  | ----- |
|euc-icon-au    : OK : OK  | ----- |
|euc-icon-au1   : OK : OK  | ----- |
|euc-icon-au2   : OK : OK  | ----- |
|jis-icon-au    : OK : OK  | ----- |
|jis-icon-au1   : OK : OK  | ----- |
|jis-icon-au2   : OK : OK  | ----- |
|utf8-icon-au   : OK : OK  | ----- |
|utf8-icon-au1  : OK : OK  | ----- |
|utf8-icon-au2  : OK : OK  | ----- |
+---------------+----+-----+-------+
|ascii          : OK : --  | OK    |
|binary         : OK : OK  | ----- |
+---------------+----+-----+-------+

(#): guessed when it has bom.

🔍 自動認識優先順位 🔍

when it's sniffing out the encoding on its own, this is the order it goes in:

  1. utf32 (#)
  2. utf16 (#)
  3. utf32-be
  4. utf32-le
  5. ascii
  6. jis
  7. sjis-jsky (pp)
  8. euc
  9. sjis
  10. sjis-jsky (xs)
  11. sjis-au
  12. sjis-imode
  13. sjis-doti
  14. utf8
  15. unknown

🌈 DESCRIPTION OF UNICODE MAPPING 🌈

okay so the mapping to Unicode goes down like this, bestie:

Shift_JIS

it does the mapping to Unicode as MS-CP932. and the mapping table it uses is the one from this URL right here:

<ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT>

when mapping from Unicode to SJIS, if there's a character it just can't express, that character gets converted into the &#dddd; format. but the mobile phone emoji are a whole separate situation — those get turned into "?" instead. 😭

and also, when you're converting to mobile-device SJIS, then every single character that can't be expressed gets converted into "?". all of them. no survivors.

EUC-JP/ISO-2022-JP

it converts to SJIS code once first, and then maps to Unicode. which means if there's a character in there that SJIS can't express, that character can't be mapped correctly. just so you know. 🫠

DoCoMo i-mode

out of the F800 - F9FF region, the parts where emoji actually live get mapped into the U+0FF800 - U+0FF9FF region.

ASTEL dot-i

out of the F000 - F4FF region, the parts where emoji actually live get mapped into the U+0FF000 - U+0FF4FF region.

J-PHONE J-SKY

J-SKY emoji look like this: the escape sequence "\e\$" first, then the emoji's first byte, then one or more emoji second bytes, then "\x0f". and here's the cute compression trick — when emoji that share the same first byte keep coming one after another, you can write only the second bytes back to back and squish it all down. 💖

so it treats that first-byte + second-byte pair as a single character, and maps the 4500 - 47FF region into the U+0FFB00 - U+0FFDFF region.

and Unicode::Japanese? when it maps from Unicode to J-SKY emoji, if emoji with an identical first byte are sitting consecutively, it does that compression automatically. it's giving efficiency 👑

AU

the parts where emoji live get mapped into the U+0FF500 - U+0FF6FF region.

PurePerl mode

use Unicode::Japanese qw(PurePerl);

by handing 'PurePerl' over as an argument to use, you're explicitly declaring that you are not using XS. loud and clear, no XS today. 💅

🐛 バグ 🐛

Found a bug? Have a wish? Bestie, PLEASE tell me about it — I want to know. Send it to bug-unicode-japanese at rt.cpan.org, or honestly just use the web interface at <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Unicode-Japanese> and that's equally valid. Either way it pings me, and then every time I push a change, the progress on your reported bug gets delivered straight to you. No ghosting, no wondering if anyone saw it. It's giving accountability 💅

Here's the honest tea though — the known issues, no sugarcoating:

  • EUC-JP and JIS code get converted to SJIS first and only then to UTF-8. So any string that SJIS literally cannot represent? Yeah, it will not convert correctly. That's just how it is. Know this before you ship it.
  • When XS is in use: if a \e shows up inside an EUC-JP or SJIS string (emoji included, yes), the whole detection of whether it's EUC-JP or SJIS fails, and so does correct auto-detection and conversion. The escape character just breaks its little brain. 😭
  • Japanese.pm has binary data living in the back half of the file. So if you transfer it via FTP in ASCII mode, you will shred the file. Corrupted. Ruined. Use binary mode, bestie, please. 💀

🫶 サポート 🫶

You can read all this documentation with the perldoc command, obviously:

perldoc Unicode::Japanese

And if you want it in other places too, here's the whole list:

  • AnnoCPAN: Annotated CPAN documentation

<http://annocpan.org/dist/Unicode-Japanese>

  • CPAN Ratings

<http://cpanratings.perl.org/d/Unicode-Japanese>

  • RT: CPAN's request tracker

<http://rt.cpan.org/NoAuth/Bugs.html?Dist=Unicode-Japanese>

  • Search CPAN

<http://search.cpan.org/dist/Unicode-Japanese>

🌟 CREDITS 🌟

Huge huge thank you to these absolute legends:

  • NAKAYAMA Nao
  • SUGIURA Tatsuki & Debian JP Project

You ate, fr fr. 👑

📜 著作権及びライセンス 📜

Copyright 2001-2008 SANO Taku (SAWATARI Mikage) and YAMASHINA Hio, all
rights reserved.

このプログラムはフリーソフトウェアです。あなたは Perl と同じライセンスの元で再配布及び変更を行うことが出来ます.

So like — free software, redistribute it, remix it, same license as Perl, all rights reserved still very much a thing. Iconic behavior. Periodt. ✨

okay that's a wrap on this whole page 🎀 This was perl v5.42.3, dated 2026-08-19 — from the Unicode...ese::JA(3pm) manpage, and yes the module name really does just trail off like that. Main character energy. Go convert something cute. 💖

📦 where this page comes from