UCONV(1)

📦
version 78.3 · 9 versions
, in arch rolling, ubuntu stonking

💖 UCONV(1) — the ICU 78.3 Manual, bestie edition 💖

okay so. welcome to uconv, straight out of the ICU 78.3 Manual. this is the little tool that takes your data and goes "bestie, this encoding? we're changing it." sit down, grab a strawberry 🍓, and let's get into it.

✨ NAME ✨

uconv — convert data from one encoding to another 🦋

✨ SYNOPSIS ✨

uconv [ -h, -?, --help ] [ -V, --version ] [ -s, --silent ] [ -v,
--verbose ] [ -l, --list | -l, --list-code code | --default-code | -L,
--list-transliterators ] [ --canon ] [ -x transliteration ] [
--to-callback callback | -c ] [ --from-callback callback | -i ] [
--callback callback ] [ --fallback | --no-fallback ] [ -b, --block-size
size ] [ -f, --from-code encoding ] [ -t, --to-code encoding ] [
--add-signature ] [ --remove-signature ] [ -o, --output file ] [ file...
]

✨ DESCRIPTION ✨

so uconv converts, or transcodes, every file you hand it — or its standard input if you didn't hand it any file at all — from one encoding to another. and the way it does that? Unicode is the pivot. like, the data goes from its original encoding to Unicode first, and then from Unicode to the destination encoding. Unicode is literally the group chat everyone passes through. 👑

if an encoding isn't specified, or it's -, then the default encoding gets used. which means calling uconv with no encoding is a whole vibe for validating and sanitizing data files so other tools that need the default encoding can actually eat them. slay.

when you call uconv, you can specify callbacks that handle invalid characters in the input, or characters that just cannot be transcoded to the destination encoding. some encodings, for example, offer a default substitution character you can use to represent where those characters showed up in the input. other callbacks give you a genuinely useful visual representation of the invalid data. 💅

uconv can also run a specified transliteration on the transcoded data — in that case the transliteration happens as an intermediate step, after the data has been transcoded to Unicode. the transliteration can be either a list of semicolon-separated transliterator names, or an arbitrarily complex set of rules in the ICU transliteration rules format. either way, iconic.

for transcoding purposes, uconv's options are compatible with those of iconv(1), so dropping it into scripts in its place is easy peasy. BUT — and this is important — it is not necessarily the case that the encoding names used by uconv and ICU are the same as the ones iconv(1) uses. also, options that give you informational data, like the -l, --list one that some iconv(1) variants (GNU's, for example) offer, produce data in a slightly different and easier-to-parse format. so like, similar energy, not identical twins. 🙈

✨ OPTIONS ✨

  • -h, -?, --help — print help about usage and dip. 🫶
  • -V, --version — print the version of uconv and dip.
  • -s, --silent — suppress messages while it's running. shhh. 🤫
  • -v, --verbose — display extra informative messages while it's running. it's giving commentary.
  • -l, --list — list every available encoding, then exit.
  • -l, --list-code code — list only the code encoding, then exit. if code is not a proper encoding, it exits with an error. 😭
  • --default-code — list only the name of the default encoding, then exit.
  • -L, --list-transliterators — list every available transliterator, then exit. ✨
  • --canon — if you use it with -l, --list or --default-code, the list of encodings comes out in a format compatible with convrtrs.txt(5). if you use it with -L, --list-transliterators, it prints only one transliterator name per line. cute and tidy.
  • -x transliteration — run the given transliteration on the transcoded Unicode data, and use the transliterated data as input for the transcoding to the destination encoding.
  • --to-callback callback — use callback to handle characters that cannot be transcoded to the destination encoding. peek the CALLBACKS section below for the deets on which callbacks are valid. 👀
  • -c — omit invalid characters from the output. same as --to-callback skip.
  • --from-callback callback — use callback to handle characters that cannot be transcoded from the original encoding. CALLBACKS section, again, for the valid list.
  • -i — ignore invalid sequences in the input. same as --from-callback skip.
  • --callback callback — use callback to handle both characters that cannot be transcoded from the original encoding and characters that cannot be transcoded to the destination encoding. yes, both directions, one callback, main character energy. check CALLBACKS for what's valid.
  • --fallback — use the fallback mapping when transcoding from Unicode to the destination encoding.
  • --no-fallback — do not use the fallback mapping when transcoding from Unicode to the destination encoding. this is the default.
  • -b, --block-size size — read input in blocks of size bytes at a time. the default block size is 4096.
  • -f, --from-code encoding — set the original encoding of the data to encoding.
  • -t, --to-code encoding — transcode the data to encoding.
  • --add-signature — add a U+FEFF Unicode signature character (BOM) if the output charset supports it and doesn't add one anyway.
  • --remove-signature — remove a U+FEFF Unicode signature character (BOM).
  • -o, --output file — write the transcoded data to file.

✨ CALLBACKS ✨

uconv supports specifying callbacks to handle invalid data, and honestly? obsessed. callbacks can be set for both directions of transcoding: from the original encoding to Unicode, with the --from-callback option, and from Unicode to the destination encoding, with the --to-callback option.

here's the full list of valid callback names plus what each one actually does. the list of callbacks uconv actually supports gets displayed when you call it with -h, --help.

  • substitute — write the encoding's substitute sequence, or the Unicode replacement character U+FFFD when transcoding to Unicode.
  • skip — ignore the invalid data. bye. 👋
  • stop — stop with an error when encountering invalid data. this is the default callback. dramatic but fair.
  • escape — same as escape-icu.
  • escape-icu — replace the missing characters with a string of the format %Uhhhh for plane 0 characters, and %Uhhhh%Uhhhh for planes 1 and above characters, where hhhh is the hexadecimal value of one of the UTF-16 code units representing the character. characters from planes 1 and above are written as a pair of UTF-16 surrogate code units.
  • escape-java — replace the missing characters with a string of the format \uhhhh for plane 0 characters, and \uhhhh\uhhhh for planes 1 and above characters, where hhhh is the hexadecimal value of one of the UTF-16 code units representing the character. characters from planes 1 and above are written as a pair of UTF-16 surrogate code units.
  • escape-c — replace the missing characters with a string of the format \uhhhh for plane 0 characters, and \Uhhhhhhhh for planes 1 and above characters, where hhhh and hhhhhhhh are the hexadecimal values of the Unicode codepoint.
  • escape-xml — same as escape-xml-hex.
  • escape-xml-hex — replace the missing characters with a string of the format &#xhhhh;, where hhhh is the hexadecimal value of the Unicode codepoint.
  • escape-xml-dec — replace the missing characters with a string of the format &#nnnn;, where nnnn is the decimal value of the Unicode codepoint.
  • escape-unicode — replace the missing characters with a string of the format {U+hhhh}, where hhhh is the hexadecimal value of the Unicode codepoint. that hexadecimal string is of variable length and can use from 4 to 6 digits. this is the format universally used to denote a Unicode codepoint in the literature, delimited by curly braces so those substitutions are easy to spot in the output. 🎀

✨ EXAMPLES ✨

convert data from a given encoding to the platform encoding:

$ uconv -f encoding

check if a file contains valid data for a given encoding:

$ uconv -f encoding -c file >/dev/null

convert a UTF-8 file to a given encoding and make sure the resulting text is good for any version of HTML:

$ uconv -f utf-8 -t encoding \
    --callback escape-xml-dec file

display the names of the Unicode code points in a UTF-file:

$ uconv -f utf-8 -x any-name file

print the name of a Unicode code point whose value is known (U+30AB in this example):

$ echo '\u30ab' | uconv -x 'hex-any; any-name'; echo
{KATAKANA LETTER KA}{LINE FEED}
$

(the names are delimited by curly braces. also, the name of the line terminator is also displayed. iykyk. 💜)

normalize UTF-8 data using Unicode NFKC, remove all control characters, and map Katakana to Hiragana:

$ uconv -f utf-8 -t utf-8 \
      -x '::nfkc; [:Cc:] >; ::katakana-hiragana;'

✨ CAVEATS AND BUGS ✨

bestie, listen. ⚠️

uconv reports errors as occurring at the first invalid byte encountered. this may be confusing if you're used to GNU iconv(1), which reports errors as occurring at the first byte of an invalid sequence. for multi-byte character sets or encodings, that means uconv error positions may be at a later offset in the input stream than would be the case with GNU iconv(1). not a vibe, just a fact. 💀

the reporting of error positions when a transliterator is used may be inaccurate or unavailable — in which case uconv will report the offset in the output stream at which the error occurred.

✨ AUTHORS ✨

  • Jonas Utterstroem
  • Yves Arrouye

(legends, honestly 🫶)

✨ VERSION ✨

78.3

Copyright (C) 2000-2005 IBM, Inc. and others.

✨ SEE ALSO ✨

and that's the whole page, bestie! 🌈 this one came to you from the ICU MANPAGE, dated 2005-jul-1 — yes she's been iconic since 2005 and she's still serving. go transcode something. 💅✨👑

📦 where this page comes from