std::codecvt_utf16(3)

📦 version 2024.07.05, in nixpkgs unstable, opensuse-leap 16.0

🌸 std::codecvt_utf16 — NAME 🌸

std::codecvt_utf16 — bestie, it's literally std::codecvt_utf16, the facet that speaks UTF-16 💅

This page lives in the C++ Standard Library manual, section 3. Pull up a chair 🫶

✨ Synopsis ✨

Defined in header <codecvt>

template<
    class Elem,                                         (since C++11)
    unsigned long Maxcode = 0x10ffff,                   (deprecated in C++17)
    std::codecvt_mode Mode = (std::codecvt_mode)0 >     (removed in C++26)
class codecvt_utf16
    : public std::codecvt<Elem, char, std::mbstate_t>;
Heads up queen: this whole thing is deprecated in C++17 and removed in C++26. So like… use it if you must, but know it's on its way out 💀

Okay so std::codecvt_utf16 is a std::codecvt facet that encapsulates conversion between a UTF-16 encoded byte string and a UCS-2 or UTF-32 character string (which one depends on the type of Elem). This std::codecvt facet can be used to read and write UTF-16 files in binary mode. No cap, that's the whole gig 🔥

UCS-2 is the same encoding as UTF-16, except that it encodes scalar values in the range U+0000-U+FFFF (Basic Multilingual Plane) only (since C++23). So it's UTF-16's little sibling with a smaller range 🎀

🦋 Template Parameters 🦋

  • Elem — either char16_t, char32_t, or wchar_t
  • Maxcode — the largest value of Elem that this facet will read or write without error
  • Mode — a constant of type std::codecvt_mode

💖 Member functions 💖

  • constructor — constructs a new codecvt_utf16 facet (public member function)
  • destructor — destroys a codecvt_utf16 facet (public member function)

🎀 std::codecvt_utf16::codecvt_utf16 🎀

explicit codecvt_utf16( std::size_t refs = 0 );

Constructs a new std::codecvt_utf16 facet, passes the initial reference counter refs to the base class. Simple as that, periodt ✨

Parameters

  • refs — the number of references that link to the facet

🎀 std::codecvt_utf16::~codecvt_utf16 🎀

~codecvt_utf16();

Destroys the facet. Iconic detail: unlike the locale-managed facets, this facet's destructor is public 💅

🌈 Inherited from std::codecvt 🌈

Member types

  • intern_type → internT
  • extern_type → externT
  • state_type → stateT

Member objects

  • id (static) — type std::locale::id

Member functions

  • out — invokes do_out (public member function of std::codecvt<InternT,ExternT,StateT>)
  • in — invokes do_in (public member function of std::codecvt<InternT,ExternT,StateT>)
  • unshift — invokes do_unshift (public member function of std::codecvt<InternT,ExternT,StateT>)
  • encoding — invokes do_encoding (public member function of std::codecvt<InternT,ExternT,StateT>)
  • always_noconv — invokes do_always_noconv (public member function of std::codecvt<InternT,ExternT,StateT>)
  • length — invokes do_length (public member function of std::codecvt<InternT,ExternT,StateT>)
  • max_length — invokes do_max_length (public member function of std::codecvt<InternT,ExternT,StateT>)

🔥 Protected member functions 🔥

  • do_out [virtual] — converts a string from InternT to ExternT, such as when writing to file (virtual protected member function of std::codecvt<InternT,ExternT,StateT>)
  • do_in [virtual] — converts a string from ExternT to InternT, such as when reading from file (virtual protected member function of std::codecvt<InternT,ExternT,StateT>)
  • do_unshift [virtual] — generates the termination character sequence of ExternT characters for incomplete conversion (virtual protected member function of std::codecvt<InternT,ExternT,StateT>)
  • do_encoding [virtual] — returns the number of ExternT characters necessary to produce one InternT character, if constant (virtual protected member function of std::codecvt<InternT,ExternT,StateT>)
  • do_always_noconv [virtual] — tests if the facet encodes an identity conversion for all valid argument values (virtual protected member function of std::codecvt<InternT,ExternT,StateT>)
  • do_length [virtual] — calculates the length of the ExternT string that would be consumed by conversion into given InternT buffer (virtual protected member function of std::codecvt<InternT,ExternT,StateT>)
  • do_max_length [virtual] — returns the maximum number of ExternT characters that could be converted into a single InternT character (virtual protected member function of std::codecvt<InternT,ExternT,StateT>)

⭐ Inherited from std::codecvt_base ⭐

Member types

  • enum result { ok, partial, error, noconv }; — unscoped enumeration type

Enumeration constants

  • ok — conversion was completed with no error
  • partial — not all source characters were converted
  • error — encountered an invalid character
  • noconv — no conversion required, input and output types are the same

🙈 Notes 🙈

Okay so this is the messy part, bestie, listen up 😭 Although the standard requires that this facet works with UCS-2 when the size of Elem is 16 bits, some implementations use UTF-16 instead, making this a non-converting locale. The term "UCS-2" was deprecated and removed from ISO 10646.

And since C++23, UCS-2 does not refer to its original definition anymore. It becomes a synonym of UTF-16, but with a smaller encoding range. The way the terminology keeps shifting… 💀

🧚 Example 🧚

The following example demonstrates decoding of UTF-16le file on a system with 32-bit wchar_t. On a system with 16-bit wchar_t, decoding of the third character will fail because std::codecvt_utf16<char16_t> produces UCS-2, not UTF-16. Yeah. Read that again before you file a bug 🫶

// Run this code

#include <codecvt>
#include <cwchar>
#include <fstream>
#include <iostream>
#include <locale>
#include <string>

void prepare_file()
{
    // UTF-16le data (if host system is little-endian)
    char16_t utf16le[4] = {0x007a,          // latin small letter 'z' U+007a
                           0x6c34,          // CJK ideograph "water" U+6c34
                           0xd834, 0xdd0b}; // musical sign segno U+1d10b

    // store in a file
    std::ofstream fout("text.txt");
    fout.write(reinterpret_cast<char*>(utf16le), sizeof utf16le);
}

int main()
{
    prepare_file();
    // open as a byte stream
    std::wifstream fin("text.txt", std::ios::binary);
    // apply facet
    fin.imbue(std::locale(fin.getloc(),
        new std::codecvt_utf16<wchar_t, 0x10ffff, std::little_endian>));

    wchar_t c = 0;
    for (std::cout << std::showbase << std::hex; fin.get(c);
         std::cout << static_cast<std::wint_t>(c) << '\n');
}

Output

0x7a
0x6c34
0x1d10b

Look at that. Slayed. ✨

👑 See also 👑

Character conversions 💜

The big grid has these columns: Character conversions · locale-defined multibyte (UTF-8, GB18030) · UTF-8 · UTF-16

  • UTF-16 — mbrtoc16 / c16rtomb (with C11's DR488) · codecvt_utf8_utf16<char16_t>, codecvt_utf8_utf16<char32_t>, codecvt_utf8_utf16<wchar_t> · N/A
  • UCS-2 — c16rtomb (without C11's DR488) · codecvt_utf8<char16_t>, codecvt<char16_t,char,mbstate_t> · codecvt_utf16<char16_t>
  • UTF-32 — mbrtoc32 / c32rtomb · codecvt<char32_t,char,mbstate_t>, codecvt_utf8<char32_t> · codecvt_utf16<char32_t>
  • system wchar_t: UTF-32 (non-Windows), UCS-2 (Windows) — mbsrtowcs / wcsrtombs · use_facet<codecvt<wchar_t,char,mbstate_t>>(locale), codecvt_utf8<wchar_t> · codecvt_utf16<wchar_t>
  • codecvt — converts between character encodings, including UTF-8, UTF-16, UTF-32 (class template)
  • codecvt_mode (C++11) (deprecated in C++17) (removed in C++26) — tags to alter behavior of the standard codecvt facets (enum)
  • codecvt_utf8 (C++11) (deprecated in C++17) (removed in C++26) — converts between UTF-8 and UCS-2/UCS-4 (class template)
  • codecvt_utf8_utf16 (C++11) (deprecated in C++17) (removed in C++26) — converts between UTF-8 and UTF-16 (class template)

That's the whole page, bestie — go forth and decode responsibly 💖 This cutie came from http://cppreference.com, version 2024.06.10, and the page is std::codecvt_utf16(3). Mwah 🫶✨🎀

📦 where this page comes from