unac(3)

🎀 unac(3) — the Library Functions Manual, but make it make sense 🎀

okay bestie, so this is unac, from the Library Functions Manual. She's a C library and she's here to strip accents off your characters like she's doing their makeup removal routine. no cap. 💅

---

✨ NAME ✨

unac — remove accents from string or character 🦋

---

🌸 SYNOPSIS 🌸

#include <unac.h>

const char* unac_version();

int unac_string(const char* charset,
          const char* in, int in_length,
          char** out, int* out_length);

int unac_string_utf16(const char* in, int in_length,
          char** out, int* out_length);

/* MACRO: side effect on unaccented and length arguments */
unac_char_utf16(unsigned short c,
                unsigned short* unaccented,
                int length);

const char* unac_version()

/*
 * The level argument can be one of:
 *    UNAC_DEBUG_NONE UNAC_DEBUG_LOW UNAC_DEBUG_HIGH
 */
void unac_debug(int level)

typedef void (*unac_debug_print_t)(const char* message, void* data);
void unac_debug_callback(int level, unac_debug_print_t function, void* data)

like… look at her. she has range. 💖

---

💜 DESCRIPTION 💜

okay so unac is a C library that removes accents from characters, and the way it does it is genuinely iconic: it does not care what character set you're in. ISO-8859-15, ISO-CELTIC, KOI8-RU… literally whatever, as long as iconv(3) is able to convert it into UTF-16 (Unicode). she said "i'll take it from here." periodt. ✨

The unac_string function is given a charset (ISO-8859-15 for instance) and a string. She converts the string into UTF-16 and calls the unac_string_utf16 function to remove all accents from the UTF-16 version. Then the unaccented string gets converted back into the original charset (ISO-8859-15 for instance) and handed back to whoever called unac_string. full circle moment. 🫶

but wait, she does a little more than removing accents. every character that is made of two characters — like æ (ISO-8859-15 octal code 346) — will be expanded into two characters a and e. and if a character were made of three characters, she'd decompose it the exact same way. she's thorough like that. 👑

The conversion from and to UTF-16 is done with iconv(3). The iconv -l command will list all available charsets. Using UTF-16 as a pivot implies an overhead but ensures that accents can be removed from all characters for which there is an equivalent character in Unicode. yes bestie, that's the tradeoff. it's giving "worth it." 🔥

unac_char_utf16 is a CPP macro that returns a pointer to the unaccented equivalent of a given UTF-16 character. She is the basic building block of unac. the origin story. the blueprint. 🌈

unac_string_utf16 repeatidly applies the unac_char_utf16 macro to each character of an UTF-16 string. (that spelling is the original's, i'm not touching it, i'm just the messenger 💀)

---

⭐ FUNCTIONS ⭐

🦋 int unac_string(const char* charset, const char* in, size_t in_length, char** out, size_t* out_length)

Returns the unaccented equivalent of the string 'in' of length 'in_length' bytes. The returned string is stored in the pointer pointed by the 'out' argument and the length of the 'out' string, in bytes, is stored in the integer pointed by the 'out_length ' argument. If the '*out' pointer is not null, it must point to an area allocated by malloc(3) and the length of the array must be specified in the '*out_length' argument. Both arguments '*out' and '*out_length' will be replaced with the return values when the function returns on success. The '*out' pointer may point to a memory location that has been reallocated (using realloc(3)) by the unac_string function. There is no guarantee that '*out' is identical to the value given by the caller. The pointer provided as '*out' by the caller may not be useable when the function returns (either error or success). If the '*out' pointer is null, the unac_string function allocates a new memory block using malloc(3). It is the responsibility of the caller to deallocate the area returned in the '*out' pointer.

🚨 bestie listen. you free that memory. she mallocs, you free. that's the deal. no take-backsies. 😭

The return value of unac_string is 0 on success and -1 on error, in which case the errno variable is set to the corresponding error code. See the ERROR section below for more information. The iconv(3) manual page may also help. 💖

---

🧚 int unac_string_utf16(const char* in, int in_length, char** out, int* out_length)

Has the same effect as unac_string("UTF-16", in, in_length, out, out_length). Since the unac_string_utf16 is the backend function of the unac_string function it is more efficient because no charset conversion of the input string (from and to UTF-16) is necessary. she's the main character behind the scenes, lowkey. ✨

---

💅 unac_char_utf16(const unsigned short c, unsigned short* p, int l)

⚠️ Warning: this is a macro, each argument may be evaluated more than once. not a drill. do not pass your i++ in there, i am begging you. 🙈

Returns the unaccented equivalent of the UTF-16 character 'c' in the pointer 'p'. The length of the unsigned short array pointed by 'p' is returned in the 'l' argument.

---

🎀 const char* unac_version()

Return the version number of unac. she's introducing herself. iconic behavior. 👑

---

🌸 void unac_debug(int level)

Set the debug level of the unac library to 'level'. Possible values are: UNAC_DEBUG_NONE for no debug at all, UNAC_DEBUG_LOW for terse human readable information, UNAC_DEBUG_HIGH for very detailed information only usable when translating a few strings.

🚨 unac_debug_callback with anything but UNAC_DEBUG_NONE is not thread safe. say it with me: not. thread. safe. 💀

---

🌈 void unac_debug_callback(int level, unac_debug_print_t function, void* data)

Set the debug level and define a printing function callback. The 'level' is the same as in unac_debug. The 'function' is in charge of dealing with the debug messages, presumably to print them to the user. The 'data' is an opaque pointer that is passed along to function, should it need to manage a persistent context.

The prototype of 'function' accepts two arguments. The first is the debug message (const char*), the second is the opaque pointer given as 'data' argument to unac_debug_callback.

If 'function' is NULL, messages are printed on the standard error output using fprintf(stderr...).

🚨 unac_debug_callback with anything but UNAC_DEBUG_NONE is not thread safe. yes again. it's that serious. 😭

---

😭 ERRORS 😭

• EINVAL — the requested conversion pair is not available. For instance, when specifying the ISO-0000 charset (imaginary), it means it is not possible to convert from ISO-0000 to UTF-16. bestie ISO-0000 is not real, she made it up to prove a point. 💀

---

🍓 EXAMPLES 🍓

Convert the été string into ete. yes. the glow-down of the century. 🦋

#include <unac.h>

char* out = 0;
int out_length = 0;
if(unac_string("ISO-8859-1", "été", strlen("été"), &out, &out_length)) {
   perror("unac_string");
} else {
   printf("%.*s0, out_length, out);
   free(out);
}

---

…and that's only part 1, bestie. more parts are coming, don't you dare close the tab. 🫶

✨ IMPLEMENTATION NOTES ✨

okay so the endianness of the UTF-16 strings that unac manipulates has to be big endian, always, no exceptions, bestie. When you're using iconv(3) to translate strings, use UTF-16BE instead of UTF-16 so you know it's big endian (BE). On some systems where UTF-16BE isn't available, unac will rely on UTF-16 and just hope it's properly big endian encoded. 😭 For more info, check RFC2781 (http://www.faqs.org/rfcs/rfc2781.html: UTF-16, an encoding of ISO 10646).

The unac library uses the Unicode database to map accented letters to their unaccented equivalent. Mapping tables are generated from the UnicodeData-4.0.0.txt file (as found at http://www.unicode.org/Public/4.0-Update/) by the builder perl script. The builder script inserts these tables in the unac.h and unac.c files, replacing the existing ones. If you look for the 'Generated by builder' string in the unac.[ch] files, that lets you spot the various parts handled by the builder script. ✨

Some desirable decompositions may not be included in the UnicodeData file, such as AE. To complement the standard decompositions for the purpose of the unac library, the unaccent-local-map.perl script is used. It maps character names (such as LATIN SMALL LETTER AE) to an array of character names into which it will be decomposed. This script is used by the builder script and has precedence over decomposition rules defined in the Unicode data file. periodt.

The library data occupies 30KB where a simple minded table would occupy around 512Kbytes. The idea used to compress the tables is that many Unicode characters do not have unaccented equivalent. Instead of relying on a table mapping each Unicode character to the corresponding unaccented character, an intermediate array of pointers is created. In the drawing below, the range of UTF-16 character is not accurate but illustrates the method. The unac_data_table points to a set of unac_dataXX arrays. Each pointer covers a range of UTF-16 characters (4 in the example below). When a range of character does not contain any accented character, unac_data_table always points to the same array : unac_data0. Since there are many characters without accents, this is enough to achieve a good compression. 💅

             unac_data15                                   unac_data16
       [ NULL, NULL, NULL, e ] <----       /------> [ a, NULL, NULL, NULL ]
                                    |       |
                                    |       |
                                    ^       ^
                 |-----| |-----| |-----| |-----| |-----| |-----|
           [ ... a b c d e f g h i j k é à 0 1 2 3 4 5 6 7 8 9 A... ] unac_data_table
                 |-----| |-----| |-----| |-----| |-----| |-----|
                     v      v                       v       v
                     |      |                       |       |
                     |      |                       |       |
                     --------------------------------------/
                                       |
                                       V
                           [ NULL, NULL, NULL, NULL ]
                                    unac_data0

Beside this simple optimization, a table (unac_positions) listing the actual position of the unaccented replacement within a block (unac_dataXX) is necessary because they are not of fixed length. Some characters such as æ will be replaced by two characters a and e therefore unac_dataXX has a variable size.

The unaccented equivalent of an UTF-16 character is calculated by applying a compatibility decomposition and then stripping all characters that belong to the mark category. For a precise definition see the Unicode-4.0 normalization forms at http://www.unicode.org/unicode/reports/tr15/.

All original Unicode data files were taken from http://www.unicode.org/Public and are subject to the UCD Terms of Use.

http://www.unicode.org/Public/4.0-Update/UCD-4.0.0.html#UCD_Terms

Disclaimer

The Unicode Character Database is provided as is by Unicode, Inc. No claims are made as to fitness for any particular purpose. No warranties of any kind are expressed or implied. The recipient agrees to determine applicability of information provided. If this file has been purchased on magnetic or optical media from Unicode, Inc., the sole remedy for any claim will be exchange of defective media within 90 days of receipt.

This disclaimer is applicable for all other data files accompanying the Unicode Character Database, some of which have been compiled by the Unicode Consortium, and some of which have been supplied by other sources.

Limitations on Rights to Redistribute This Data

Recipient is granted the right to make copies in any form for internal distribution and to freely use the information supplied in the creation of products supporting the UnicodeTM Standard. The files in the Unicode Character Database can be redistributed to third parties or other organizations (whether for profit or not) as long as this notice and the disclaimer notice are retained. Information can be extracted from these files and used in documentation or programs, as long as there is an accompanying notice indicating the source.

The file Unihan.txt contains older and inconsistent Terms of Use. That language is overridden by these terms.

💀 BUGS 💀

The input string must not contain partially formed characters, there is no support for this case. Like, don't do it, bestie.

UTF-16 surrogates are not handled.

Unicode may contain bugs in the decomposition of characters. When you suspect such a bug on a given string, add a test case with the faulty string in the t_unac.in test script (you will find it in the source distribution) and run make check. It will describe, in a very verbose way, how the string was unaccented. You may then fix the UnicodeData-4.0.0.txt file and run make check again to make sure the problem is solved. Please send such fixes to the author and to the Unicode consortium. 🙈

👀 SEE ALSO 👀

• unaccent(1), iconv(3)
• http://www.unicode.org/
• http://oss.software.ibm.com/icu/
• http://www.gnu.org/manual/glibc-2.2.5/libc.html

👑 AUTHOR 👑

Loic Dachary loic@senga.org http://www.senga.org/unac/

---

and the lil footer at the bottom says local and unac(3) — that's the source/version line and the page title, and there's no date in this footer, so don't come for me. 💖