File::BOM(3pm)

📦
version 0.18 · 5 versions
, in ubuntu noble, ubuntu resolute, ubuntu stonking

🌸 File::BOM 🌸

okay bestie, so this is the manual page for File::BOM(3pm) 💖 — it's a Perl module and it's literally Utilities for handling Byte Order Marks. If you've ever opened a file and gotten a mysterious invisible gremlin at the start of it, that's a BOM, and this module is here to handle it for you. no cap.

✨ SYNOPSIS ✨

use File::BOM qw( :all )

🌈 high-level functions

# read a file with encoding from the BOM:
open_bom(FH, $file)
open_bom(FH, $file, ':utf8') # the same but with a default encoding

# get encoding too
$encoding = open_bom(FH, $file, ':utf8');

# open a potentially unseekable file:
($encoding, $spillage) = open_bom(FH, $file, ':utf8');

# change encoding of an open handle according to BOM
$encoding = defuse(*HANDLE);
($encoding, $spillage) = defuse(*HANDLE);

# Decode a string according to leading BOM:
$unicode = decode_from_bom($string_with_bom);

# Decode a string and get the encoding:
($unicode, $encoding) = decode_from_bom($string_with_bom)

🦋 PerlIO::via interface

# Read the Right Thing from a unicode file with BOM:
open(HANDLE, '<:via(File::BOM)', $filename)

# Writing little-endian UTF-16 file with BOM:
open(HANDLE, '>:encoding(UTF-16LE):via(File::BOM)', $filename)

🎀 lower-level functions

# read BOM encoding from a filehandle:
$encoding = get_encoding_from_filehandle(FH)

# Get encoding even if FH is unseekable:
($encoding, $spillage) = get_encoding_from_filehandle(FH);

# Get encoding from a known unseekable handle:
($encdoing, $spillage) = get_encoding_from_stream(FH);

# get encoding and BOM length from BOM at start of string:
($encoding, $offset) = get_encoding_from_bom($string);

💜 variables

# print a BOM for a known encoding
print FH $enc2bom{$encoding};

# get an encoding from a known BOM
$enc = $bom2enc{$bom}

💅 DESCRIPTION 💅

So this module hands you a whole lil toolkit for dealing with unicode byte order marks 🫶 — the ones that sit at the very beginning of some files and streams. secret little signature at the top of the file. iykyk.

For the full lore on what a byte order mark even is, go look here:

<http://www.unicode.org/unicode/faq/utf_bom.html#BOM>

The whole intention of File::BOM, like the actual mission statement, is that files with BOMs should be readable as seamlessly as possible, no matter what encoding was used. To make that happen it gives you several different interfaces, all shown up in the synopsis above. obsessed with the choices honestly ✨

🧚 EXPORTS 🧚

Nothing by default. you get nothing unless you ask, and that's on boundaries 💅

⭐ symbols

  • open_bom()
  • defuse()
  • decode_from_bom()
  • get_encoding_from_filehandle()
  • get_encoding_from_stream()
  • get_encoding_from_bom()
  • %bom2enc
  • %enc2bom

⭐ tags

  • :all — All of the above
  • :subs — subroutines only
  • :vars — just %bom2enc and %enc2bom

👑 VARIABLES 👑

%bom2enc

Maps Byte Order marks to their encodings. a little matchmaking service for bytes 💞

The keys of this hash are strings which represent the BOMs, and the values are their encodings, in a format which is understood by Encode.

The encodings represented in this hash are: UTF-8, UTF-16BE, UTF-16LE, UTF-32BE and UTF-32LE.

%enc2bom

A reverse-lookup hash for bom2enc, with a few aliases used in Encode, namely utf8, iso-10646-1 and UCS-2.

heads up bestie: UTF-16, UTF-32 and UCS-4 are not included in this hash. Mainly because Encode::encode automatically puts BOMs on output. See Encode::Unicode 💖

🔥 FUNCTIONS 🔥

open_bom

$encoding = open_bom(HANDLE, $filename, $default_mode)

($encoding, $spill) = open_bom(HANDLE, $filename, $default_mode)

opens HANDLE for reading on $filename, and sets the mode to whatever encoding matches the BOM chillin' inside that file. it reads the vibe and adjusts. 💅

On failure, it raises a fatal error — check the DIAGNOSTICS section for the tea on how to catch those. And it does that on purpose, so that the return value(s) stay free to be used for other stuff instead.

If the file doesn't contain a BOM, $default_mode gets used instead. So like:

open_bom(FH, 'my_file.txt', ':utf8')

That opens my_file.txt for reading in an appropriate encoding found from the BOM in that file, or as a UTF-8 file if none is found. flexible queen 🌈

In the absence of a $default_mode argument, these next 2 calls should be equivalent:

open_bom(FH, 'no_bom.txt');

open(FH, '<', 'no_bom.txt');

If an undefined value is passed as the handle, a symbol will be generated for it, exactly like open() does:

# create filehandle on the fly
$enc = open_bom(my $fh, $filename, ':utf8');
$line = <$fh>;

The filehandle will be cued up to read after the BOM. Unseekable files (e.g. fifos) will cause croaking, unless you call it in list context to catch spillage from the handle. Any spillage will be automatically decoded from the encoding, if found.

e.g.

# croak if my_socket is unseekable
open_bom(FH, 'my_socket');

# keep spillage if my_socket is unseekable
($encoding, $spillage) = open_bom(FH, 'my_socket');

# discard any spillage from open_bom
($encoding) = open_bom(FH, 'my_socket');

defuse

$enc = defuse(FH);

($enc, $spill) = defuse(FH);

FH should be a filehandle opened for reading, and it will have the relevant encoding layer pushed onto it by binmode if a BOM is found. Spillage should be Unicode, not bytes. don't mix those up babes 😭

⚠️ Any uncaptured spillage will be silently lost. If the handle is unseekable, use list context to avoid data loss. fr fr.

If no BOM is found, the mode will be unaffected. no drama, no changes 💖

decode_from_bom

$unicode_string = decode_from_bom($string, $default, $check)

($unicode_string, $encoding) = decode_from_bom($string, $default, $check)

Reads a BOM from the beginning of $string, decodes $string (minus the BOM) and returns it to you as a perl unicode string. squeaky clean ✨

if $string doesn't have a BOM, $default is used instead.

$check, if supplied, is passed to Encode::decode as the third argument.

If there's no BOM and no default, the original string is returned and encoding is ''.

See Encode

get_encoding_from_filehandle

$encoding = get_encoding_from_filehandle(HANDLE)

($encoding, $spillage) = get_encoding_from_filehandle(HANDLE)

Returns the encoding found in the given filehandle.

The handle should be opened in a non-unicode way (e.g. mode '<:bytes') so that the BOM can be read in its natural state. let the BOM be its authentic self 🦋

After calling, the handle will be set to read at a point after the BOM (or at the beginning of the file if no BOM was found).

If called in scalar context, unseekable handles cause a croak().

If called in list context, unseekable handles will be read byte-by-byte and any spillage will be returned. See get_encoding_from_stream()

get_encoding_from_stream

($encoding, $spillage) = get_encoding_from_stream(*FH);

Read a BOM from an unrewindable source. This means reading the stream one byte at a time until either a BOM is found or every possible BOM is ruled out. Any non-BOM bytes read from the handle will be returned in $spillage. main character detective work 🔍

If a BOM is found and the spillage contains a partial character (judging by the expected character width for the encoding) more bytes will be read from the handle to ensure that a complete character is returned. no half-characters here, we finish what we start 💅

Spillage is always in bytes, not characters.

This function is less efficient than get_encoding_from_filehandle, but should work just as well on a seekable handle as on an unseekable one.

get_encoding_from_bom

($encoding, $offset) = get_encoding_from_bom($string)

Returns the encoding and length in bytes of the BOM in $string.

If there is no BOM, an empty string is returned and $offset is zero.

To get the data from the string, the following should work:

use Encode;

my($encoding, $offset) = get_encoding_from_bom($string);

if ($encoding) {
    $string = decode($encoding, substr($string, $offset))
}

🌈 PerlIO::via interface 🌈

File::BOM can be used as a PerlIO::via interface. iconic behaviour honestly 👑

open(HANDLE, '<:via(File::BOM)', 'my_file.txt');

open(HANDLE, '>:encoding(UTF-16LE):via(File::BOM)', 'out_file.txt');
print "foo\n"; # BOM is written to file here

This method is less prone to errors on non-seekable files as spillage is incorporated into an internal buffer, but it doesn't give you any information about the encoding being used, or indeed whether or not a BOM was present. it just handles it and doesn't gossip about it 🫶

There are a few known problems with this interface, especially surrounding seek() and tell(), please see the BUGS section for more details about this. yeah... we'll get to that 😭

🌸 Reading

The via(File::BOM) layer must be added before the handle is read from, otherwise any BOM will be missed. If there is no BOM, no decoding will be done. order of operations babes, it matters!

Because of a limitation in PerlIO::via, read() always works on bytes, not characters. BOM decoding will still be done but output will be bytes of UTF-8.

open(BOM, '<:via(File::BOM)', $file);
$bytes_read = read(BOM, $buffer, $length);
$unicode = decode('UTF-8', $buffer, Encode::FB_QUIET);

# Now $unicode is valid unicode and $buffer contains any left-over bytes

🌸 Writing

Add the via(File::BOM) layer on top of a unicode encoding layer to print a BOM at the start of the output file. This needs to be done before any data is written. The BOM is written as part of the first print command on the handle, so if you don't print anything to the handle, you won't get a BOM. no print, no BOM, that's the deal 💖

There is a "Wide character in print" warning generated when the via(File::BOM) layer doesn't receive utf8 on writing. This glitch was resolved in perl version 5.8.7, but if your perl version is older than that, you'll need to make sure that the via(File::BOM) layer receives utf8 like this:

# This works OK
open(FH, '>:encoding(UTF-16LE):via(File::BOM):utf8', $filename)

# This generates warnings with older perls
open(FH, '>:encoding(UTF-16LE):via(File::BOM)', $filename)

🌸 Seeking

Seeking with SEEK_SET results in an offset equal to the length of any detected BOM being applied to the position parameter. Thus:

# Seek to end of BOM (not start of file!)
seek(FILE_BOM_HANDLE, 0, SEEK_SET)

🌸 Telling

In order to work correctly with seek(), tell() also returns a postion adjusted by the length of the BOM.

💖 SEE ALSO 💖

💀 DIAGNOSTICS 💀

okay bestie, so when this module is not having a good time, it throws via croak(). Here's every single one, no gatekeeping:

  • Couldn't read '<filename>': $! — so open_bom() tried to open the file you asked for and... couldn't. Rude of it, honestly. 💀
  • Couldn't set binmode of handle opened on '<filename>' to '<mode>': $! — open_bom() opened it fine but then choked trying to set the binmode on the handle. tragic.
  • No string — decode_from_bom got called on an undefined value. like babes, there's nothing there. 😭
  • Unseekable handle: $! — get_encoding_from_filehandle() or open_bom() got hit with an unseekable file or handle and was called in scalar context. the vibes were simply not seekable.
  • Couldn't read from handle: $! — _get_encoding_seekable() couldn't read the handle. and just so you know, this function gets called from get_encoding_from_filehandle(), defuse() and open_bom(), so it's lowkey everywhere. 🔥
  • Couldn't reset read position: $! — _get_encoding_seekable couldn't seek back to the position right after the BOM. it had one job.
  • Couldn't read byte: $! — get_encoding_from_stream couldn't read from the handle. this one shows up when get_encoding_from_filehandle() and open_bom() run into an unseekable handle or file. 😭

🐛 BUGS 🐛

ngl, there's some stuff you need to know about:

  • Older versions of PerlIO::via are kind of messy when it comes to writing — see above, iykyk. 💅
  • The current version of PerlIO::via has limitations around seek and tell. rn this module only supports line-wise seek and tell, that's it. so if you use read() to read partial lines, tell() will still hand you the position at the end of the last line that was read. not ideal, just the reality bestie.
  • Under windows, tell() seems to return erroneously when you're reading files with unix line endings. 💀
  • Under windows, warnings may pop up when you use the PerlIO::via interface to read UTF-16LE and UTF-32LE encoded files. this looks like a bug in the relevant encoding(...) layers, so like... not our fault, but also very much our problem. 🙈

👑 AUTHOR 👑

Matt Lawrence <mattlaw@cpan.org>

Huge thanks to Mark Fowler and Steve Purkis for the additional tests and advice — the real ones, fr fr. 🫶

Copyright 2005 Matt Lawrence, All Rights Reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. slay, go forth and remix. ✨

aaand that's the whole page, bestie! this came from perl v5.36.0, dated 2022-10-22, and the page itself is File::BOM(3pm). go read some BOMs, you absolute icon 💖🌸

📦 where this page comes from