Data::...:Format(3pm)

💖 Data::CompactReadonly::V0::Format 💖

okay so bestie, welcome to the whole entire tea on the CompactReadonly data format, version 0. This is the manpage for Data::CompactReadonly::V0::Format, and it's filed under the User Contributed Perl Documentation, so you already know it's the real deal. We're talking bytes, we're talking bit-fields, we're talking the entire blueprint of how the file is laid out. Grab a snack, we're going in 🎀

---

✨ NAME ✨

Data::CompactReadonly::V0::Format — a description of CompactReadonly data format, version 0.

---

🧾 NOTATION

So when we're reading this doc together, here's the vibe: bytes with values that are printable ASCII get shown as a single ASCII character. Everything else gets shown either in hexadecimal — like 0xAB — or in binary — like 0b01010101.

When it makes sense to, bit-fields get shown in binary, like 0b100.

Bytes are separated by spaces. Bit-fields within a single byte get separated by hyphens, like 0b11-0000-00. Slay, we love a consistent notation 😌

---

🔄 ENDIANNESS

All internal structures are big-endian. That's it, that's the whole section. Periodt. 💅

---

📁 FILE HEADER

The file header is five bytes long, bestie. Five. The first four are there purely to identify the file type:

C R O D

The fifth byte is a bit-field, and it's doing double duty (main character energy honestly): the most significant five bits encode the file format version number, and the least significant three bits encode the pointer length that's used in the file. Values from 0 to 7 correspond to pointer lengths from 1 to 8 bytes.

The five byte header is immediately followed by the root node. No gap, no small talk, straight to business 🌸

Version number 31 (0b11111-XXX) is reserved for future use. Hands off, it's not yours 🙈

---

🌳 NODES

Data is encoded in nodes, and nodes can be of several types. Those types fall into two big categories:

Scalar types

Scalars encode a simple value. That can be a number, or the NULL value. And the five integer numeric types are also available as NegativeByte, NegativeMedium and so on — they're all getting their moment ✨

• Byte — 8 bit integer
• Short — 16 bit integer
• Medium — 24 bit integer
• Long — 32 bit integer
• Huge — 64 bit integer
• Float64 — 64 bit IEEE754 double-precision
• Null
• True
• False

Collection types

Collections encode multiple values. Plural. A little squad of values, if you will 🫶

Text

Encodes a list of characters — that is, a string.

Array

Encodes a list of nodes, which can themselves be of any type.

Dictionary

Encodes a list of key-value pairs, the keys being strings or numbers and the values being nodes of any type. The keys must be stored in ASCIIbetical order.

Real talk though: while the use of numeric types is permitted for keys, it is not recommended, because you may run into problems finding floating point keys thanks to the usual floating point imprecision issues. It's giving chaos, and not the cute kind 💀

Each node is encoded as a type header occupying from 1 to 9 bytes, followed by data if necessary.

---

🏷️ NODE TYPE HEADERS

The type header consists of a type specifier followed by up to 8 bytes telling us how much data is in the node. The type specifier is a bit field. The first two bits will tell us whether the node is a collection or not:

• 0b00 — Text node
• 0b01 — Array node
• 0b10 — Dictionary node
• 0b11 — it's not a collection, it's a scalar node

Then the next four bits tell us, for scalar nodes, the type — or, for collection nodes, some of them tell us what type is used to encode the collection's length. Only Byte, Short, Medium, and Long are valid for lengths, just so you know 👑

• 0b0000 — Byte (valid as a length)
• 0b0001 — NegativeByte
• 0b0010 — Short (valid as a length)
• 0b0011 — NegativeShort
• 0b0100 — Medium (valid as a length)
• 0b0101 — NegativeMedium
• 0b0110 — Long (valid as a length)
• 0b0111 — NegativeLong
• 0b1000 — Huge
• 0b1001 — NegativeHuge
• 0b1010 — Null
• 0b1011 — Float64
• 0b1100 — True
• 0b1101 — False

Any unspecified bits or combinations of bits are reserved for future use.

Unspecified bits should be set to zero if you want your data to be compatible with future versions. Do it for your future self, bestie 🌈

---

📦 NODE DATA

NUMERIC NODES

The header is followed by the appropriate number of bytes of data.

NULL, TRUE and FALSE NODES

These are just a header. That's the whole node. Nothing more. Iconic minimalism, honestly ✨

TEXT NODES

The header is followed by the appropriate number of bytes to encode the text's length, followed by that many bytes of text. Now here's the thing — text lengths are stored in bytes, but text is actually encoded in UTF-8. So the 3 character string "北京市" is stored as the 9 bytes:

北: 0xE5 0x8C 0x97    京: 0xE4 0xBA 0xAC    市: 0xE5 0xB8 0x82

and the entire node would be the 11 bytes:

0b00-0000-00:  this is a Text node, with the length stored in a Byte
0x09:          the length of the text
0xE5 ... 0x82: nine bytes of text

ARRAY NODES

The header is followed by the appropriate number of bytes to encode the number of elements in the array, N. Zero obviously means an empty array. That is immediately followed by N pointers of the size specified in the database header. Each pointer is the location in the file of another node, which can be of any type.

DICTIONARY NODES

The header is followed by the appropriate number of bytes to encode the number of elements in the dictionary, N. Zero means an empty dictionary. That is immediately followed by N pairs of pointers of the size specified in the database header.

The first pointer in each pair must point to a Text or numeric node, which will be used as a key for looking up values. The second pointer in each pair points to the value, which can be any type of node.

The pointers to keys must list them in ASCIIbetical order.

If they are out of order, some elements may not be able to be found. No cap. Sort your keys 🚨

---

and that's the whole format, bestie — you're officially fluent in CompactReadonly v0 🦋 this page came to you straight out of the User Contributed Perl Documentation, running under perl v5.36.0, dated 2023-10-30. go forth and read some bytes responsibly, love u 💜