RFC 8949
Concise Binary Object Representation (CBOR)
RFC 8949: Concise Binary Object Representation (CBOR) ๐
- Series: Internet Engineering Task Force (IETF) ๐
- Request for Comments: 8949 โจ
- STD: 94 ๐
- Category: Standards Track ๐
- Date: December 2020 ๐ธ
- Authors: C. Bormann (Universitรคt Bremen TZI) and P. Hoffman (ICANN) ๐ซถ
- Obsoletes: 7049 (RFC 7049) ๐ฆ
- ISSN: 2070-1721 ๐
Abstract ๐ธ
Okay so CBOR (the Concise Binary Object Representation) is a data format that was designed to be super tiny in code size, pretty small in message size, and extensible without needing version negotiation ๐ญ. Those design goals are what make it different from older binary formats like ASN.1 and MessagePack, and it's giving main character energy โจ.
This document obsoletes RFC 7049 ๐ฆ. It brings editorial improvements, new details, and errata fixes, but it keeps full compatibility with the interchange format of RFC 7049. It does not create a new version of the format ๐.
Status of This Memo ๐ง
This is an Internet Standards Track document ๐.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check out Section 2 of RFC 7841 ๐.
Info about the current status of this document, any errata, and how to give feedback on it lives at https://www.rfc-editor.org/info/rfc8949 ๐ฅ.
Copyright Notice ๐
Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved ๐.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License ๐ซถ.
Table of Contents ๐ฆ
1. Introduction
1.1. Objectives
1.2. Terminology
2. CBOR Data Models
2.1. Extended Generic Data Models
2.2. Specific Data Models
3. Specification of the CBOR Encoding
3.1. Major Types
3.2. Indefinite Lengths for Some Major Types
3.2.1. The "break" Stop Code
3.2.2. Indefinite-Length Arrays and Maps
3.2.3. Indefinite-Length Byte Strings and Text Strings
3.2.4. Summary of Indefinite-Length Use of Major Types
3.3. Floating-Point Numbers and Values with No Content
3.4. Tagging of Items
3.4.1. Standard Date/Time String
3.4.2. Epoch-Based Date/Time
3.4.3. Bignums
3.4.4. Decimal Fractions and Bigfloats
3.4.5. Content Hints
3.4.5.1. Encoded CBOR Data Item
3.4.5.2. Expected Later Encoding for CBOR-to-JSON Converters
3.4.5.3. Encoded Text
3.4.6. Self-Described CBOR
4. Serialization Considerations
4.1. Preferred Serialization
4.2. Deterministically Encoded CBOR
4.2.1. Core Deterministic Encoding Requirements
4.2.2. Additional Deterministic Encoding Considerations
4.2.3. Length-First Map Key Ordering
5. Creating CBOR-Based Protocols
5.1. CBOR in Streaming Applications
5.2. Generic Encoders and Decoders
5.3. Validity of Items
5.3.1. Basic validity
5.3.2. Tag validity
5.4. Validity and Evolution
5.5. Numbers
5.6. Specifying Keys for Maps
5.6.1. Equivalence of Keys
5.7. Undefined Values
6. Converting Data between CBOR and JSON
6.1. Converting from CBOR to JSON
6.2. Converting from JSON to CBOR
7. Future Evolution of CBOR
7.1. Extension Points
7.2. Curating the Additional Information Space
8. Diagnostic Notation
8.1. Encoding Indicators
9. IANA Considerations
9.1. CBOR Simple Values Registry
9.2. CBOR Tags Registry
9.3. Media Types Registry
9.4. CoAP Content-Format Registry
9.5. Structured Syntax Suffix Registry
10. Security Considerations
11. References
11.1. Normative References
11.2. Informative References
Appendix A. Examples of Encoded CBOR Data Items
Appendix B. Jump Table for Initial Byte
Appendix C. Pseudocode
Appendix D. Half-Precision
Appendix E. Comparison of Other Binary Formats to CBOR's Design
Objectives
E.1. ASN.1 DER, BER, and PER
E.2. MessagePack
E.3. BSON
E.4. MSDTP: RFC 713
E.5. Conciseness on the Wire
Appendix F. Well-Formedness Errors and Examples
F.1. Examples of CBOR Data Items That Are Not Well-Formed
Appendix G. Changes from RFC 7049
G.1. Errata Processing and Clerical Changes
G.2. Changes in IANA Considerations
G.3. Changes in Suggestions and Other Informational Components
Acknowledgements
Authors' Addresses1. Introduction โจ
Okay so there are hundreds of standardized formats for binary representation of structured data (also known as binary serialization formats) ๐. Some are made for specific domains of information, and others are general-purpose for any data at all. In the IETF, probably the best-known general-purpose formats are ASN.1's BER and DER [ASN.1] ๐.
The format defined here follows some specific design goals that current formats don't really meet ๐. The underlying data model is an extended version of the JSON data model [RFC8259]. It's important to note that this is not a proposal to extend the grammar in RFC 8259 in general, because that would cause a major backwards incompatibility with JSON documents that are already out there in the wild ๐ญ. Instead, this document simply defines its own data model that starts from JSON ๐ง.
Appendix E lists some existing binary formats and talks about how well they do (or don't) fit the design objectives of the Concise Binary Object Representation (CBOR) ๐ .
This document obsoletes [RFC7049], bringing editorial improvements, new details, and errata fixes while keeping full compatibility with the interchange format of RFC 7049. It does not create a new version of the format ๐ฆ.
1.1. Objectives ๐
The objectives of CBOR, roughly in decreasing order of importance, are:
- The representation must be able to unambiguously encode most common data formats used in Internet standards ๐.
- It must represent a reasonable set of basic data types and structures using binary encoding. "Reasonable" here is largely shaped by what JSON can do, plus the major addition of binary byte strings ๐. The structures supported are limited to arrays and trees. Loops and lattice-style graphs are not supported ๐ โโ๏ธ.
- There is no requirement that all data formats be uniquely encoded. Like, it's totally fine if the number "7" can be encoded in multiple different ways ๐.
- The code for an encoder or decoder must be able to be compact, so it works on systems with very limited memory, processor power, and instruction sets ๐.
- An encoder and a decoder need to be implementable in a very small amount of code (for example, in class 1 constrained nodes as defined in [RFC7228]) ๐ฅ.
- The format should use contemporary machine representations of data (for example, not requiring binary-to-decimal conversion) โจ.
- Data must be able to be decoded without a schema description ๐ซถ.
- Similar to JSON, encoded data should be self-describing so that a generic decoder can be written ๐ง.
- The serialization must be reasonably compact, but data compactness is secondary to code compactness for the encoder and decoder ๐.
- "Reasonable" here is bounded by JSON as an upper bound on size, and by implementation complexity, which limits how much effort can go into achieving that compactness. Using either general compression schemes or extensive bit-fiddling violates the complexity goals ๐.
- The format must be applicable to both constrained nodes and high-volume applications ๐ธ.
- This means it must be reasonably frugal in CPU usage for both encoding and decoding. This matters both for constrained nodes and for potential use in applications with a very high volume of data ๐.
- The format must support all JSON data types for conversion to and from JSON ๐ฆ.
- It must support a reasonable level of conversion as long as the data represented is within the capabilities of JSON. It must be possible to define a unidirectional mapping towards JSON for all types of data ๐.
- The format must be extensible, and the extended data must be decodable by earlier decoders ๐.
- The format is designed for decades of use ๐.
- The format must support a form of extensibility that allows fallback, so that a decoder that does not understand an extension can still decode the message ๐ .
- The format must be able to be extended in the future by later IETF standards โจ.
1.2. Terminology โจ
okay bestie, this is the vocab section, and it matters because these words are doing heavy lifting ๐ The requirement words are "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL". They only count as requirement words when they show up in ALL CAPS like that, and then they mean what BCP 14 [RFC2119] [RFC8174] says they mean. lowercase "may" is just vibes ๐ฆ
A "byte" is the same thing as an "octet," and every multi-byte value gets written in network byte order, so the most significant byte goes first. You might also see that called big-endian ๐ธ
Here's the terminology list, so we're all on the same page:
Data item: One piece of CBOR data. A data item can hold zero, one, or more nested data items inside it. The name covers both the thing as it's written out in the wire format and the abstract idea a decoder pulls out of it. When you specifically mean the written-out version, say "encoded data item" ๐
Decoder: A process that takes a well-formed encoded CBOR data item and hands it to an application. Technically, a decoder has two parts: a parser that splits the input up using the CBOR syntax rules, and a semantic processor that gets the data into a shape the application can actually use ๐ง
Encoder: A process that makes the well-formed wire representation of a CBOR data item from whatever information the application has ๐
Data Stream: A sequence of zero or more data items that aren't wrapped up into a bigger containing item (see [RFC8742] for one example of this). The independent items in a data stream are sometimes called "top-level data items" ๐
Well-formed: A data item that follows CBOR's syntax. It reads the initial bytes, plus whatever byte strings or data items those initial bytes say should follow, exactly the way CBOR defines them, and there's no extra junk tacked on at the end. Decoders only ever return stuff from well-formed data items, by definition ๐ฅ
Valid: A data item that's well-formed AND also follows the semantic restrictions that apply to CBOR data items (Section 5.3) โจ
Expected: Okay so this one has a normal English meaning, but here it's a specific term for the extra requirements an application puts on its input, on top of basic CBOR validity ๐ Well-formed means it can be processed at all, valid means a validity-checking generic decoder is happy with it, and expected means the application itself is happy with it. Those three form a ladder of acceptability, and each rung is stricter than the one below it ๐ช
Stream decoder: A process that decodes a data stream and makes each data item in the sequence available to an application as soon as it arrives ๐
Terms and concepts for floating-point values, like Infinity, NaN (not a number), negative zero, and subnormal, are defined in [IEEE754] ๐ฆ
When this document does bit arithmetic or explains data types, it uses the notation from the C programming language [C]. There are two twists, though. ".." means a range that includes both ends, and superscripts mean exponentiation. So 2 to the power of 64 is written 2^(64). Plain-text versions of this spec can't show superscripts, so they use a surrogate notation instead. That notation isn't really optimized for this RFC, and it's honestly kind of ambiguous with C's exclusive-or operator. Exclusive-or only shows up in the appendices, and the appendices don't use exponentiation, so you just have to be a little careful when you're reading the plain-text version ๐ค
Examples and pseudocode assume two things: signed integers use two's complement, and right shifts of signed integers do sign extension. Those assumptions are also written down in Sections 6.8.1 (basic.fundamental) and 7.6.7 (expr.shift) of the 2020 version of C++ (currently a final draft, [Cplusplus20]) ๐
Just like the 0x prefix marks hexadecimal, binary numbers get a 0b prefix. You can add underscores to a number purely so it's easier to read. For example, 0b00100001 (0x21) could be written as 0b001_00001 to show how the bits are meant to be grouped, here split into a three-bit chunk and a five-bit chunk ๐ Encoded CBOR data items sometimes show up in 0x or 0b notation. Those values get read as numbers first, the way C would read them, and then they're treated as byte strings in network byte order. That includes any leading zero bytes written out in the notation ๐ธ
Words can be _italicized_ for emphasis. In plain text, that's shown by putting underscores around the word. Verbatim text, like names from a programming language, can be set in "monospace" type. In plain text, that gets approximated (kind of ambiguously) by wrapping the text in double quotes, which still keep their usual meaning too ๐
2. CBOR Data Models ๐
CBOR is super explicit about its generic data model, which defines the full set of data items that can be represented in CBOR. The basic generic data model can be extended by registering "simple values" and tags. Applications can then carve out a subset of that extended generic data model to build their own specific data models ๐
Inside any environment that can represent the items in the generic data model, you can implement generic CBOR encoders and decoders. That usually means defining extra implementation data types for items that don't already have a natural representation in that environment. Making generic encoders and decoders possible is an explicit design goal of CBOR. That said, plenty of applications will write their own application-specific encoders and/or decoders anyway ๐ซถ
In the basic (unextended) generic data model from Section 3, a data item is one of the following:
- an integer in the range
-2^(64)..2^(64)-1inclusive - a simple value, identified by a number between
0and255, but distinct from that number itself - a floating-point value, distinct from an integer, from the set representable by IEEE 754 binary64 (including non-finites) [IEEE754]
- a sequence of zero or more bytes ("byte string")
- a sequence of zero or more Unicode code points ("text string")
- a sequence of zero or more data items ("array")
- a mapping (a mathematical function) from zero or more data items ("keys"), each to a data item ("values") ("map")
- a tagged data item ("tag"), made up of a tag number (an integer in the range
0..2^(64)-1) and the tag content (a data item)
Note that integer and floating-point values are distinct in this model, even when they have the same numeric value ๐
Also note that serialization variants are invisible at the generic data model level. That includes how many bytes an encoded floating-point value takes up. It also includes how an "argument" (see Section 3) gets encoded, like an integer, the length of a text or byte string, the number of elements in an array or pairs in a map, or a tag number. Totally on purpose, btw ๐ฆ
2.1. Extended Generic Data Models โจ
This basic generic data model gets extended in this document by registering a bunch of simple values and tag numbers. Some examples:
- "false", "true", "null", and "undefined" (simple values identified by
20..23, Section 3.3) - integer and floating-point values with a bigger range and more precision than the above (tag numbers 2 to 5, Section 3.4)
- application data types, like a point in time or a date/time string defined in RFC 3339 (tag numbers 1 and 0, Section 3.4) ๐ฐ๏ธ
Other elements of the extended generic data model can be defined (and have been defined) through the IANA registries created for CBOR. Even if a generic encoder or decoder doesn't recognize one of those extensions, data items using it can still be passed to or from the application. You just represent them at the application interface using the basic generic data model, which means as generic simple values or generic tags ๐
So basically, the basic generic data model stays stable as defined in this document. The extended generic data model keeps growing as new simple values or tag numbers get registered, but it never shrinks ๐ฑ
There's a strong expectation that generic encoders and decoders can represent "false", "true", and "null" in whatever form fits their programming environment. ("undefined" is intentionally left out of that.) Supporting the data model extensions that tags create is truly optional, though, and it's a matter of implementation quality ๐
2.2. Specific Data Models โจ
A specific data model for a CBOR-based protocol usually takes a subset of the extended generic data model and gives application meaning to the data items in that subset and to their components. When you document a specific data model and describe the types of its data items, it's better to name them by their generic data model names ("negative integer", "array") rather than by aspects of their CBOR representation ("major type 1", "major type 4") ๐ง
Specific data models can also define value equivalency, including between values of different types, for the purposes of map keys and encoder freedom. For example, in the generic data model, a valid map MAY have both 0 and 0.0 as keys, and an encoder MUST NOT encode 0.0 as an integer (major type 0, Section 3.1). But if a specific data model says floating-point and integer representations of integral values are equivalent, then using both 0 and 0.0 as map keys in one map counts as duplicates. Even though they're encoded as different major types, that makes the map invalid ๐ซ Also, an encoder could turn integral-valued floats into integers or the other way around, maybe just to save encoded bytes ๐ธ
3. Specification of the CBOR Encoding โจ
okay bestie, so a CBOR data item (see Section 2) gets encoded to or decoded from a byte string carrying a well-formed encoded data item ๐ Everything is summarized in Table 7 in Appendix B, and you look it up by the initial byte ๐ง. An encoder MUST produce only well-formed encoded data items. A decoder MUST NOT return a decoded data item when it hits input that isn't a well-formed encoded CBOR data item. (Diagnostic and recovery tools can still pull some info out of a damaged item, so that's not banned, it's just not the main event ๐.)
The initial byte of every encoded data item holds two things: the major type in the high-order 3 bits (more on that in Section 3.1) and the additional information in the low-order 5 bits ๐. With a few exceptions, the additional information tells you how to load an unsigned integer "argument":
- Less than 24: The argument's value is the value of the additional information.
- 24, 25, 26, or 27: The argument's value is held in the following 1, 2, 4, or 8 bytes, respectively, in network byte order. For major type 7 and additional information value 25, 26, 27, these bytes are not used as an integer argument, but as a floating-point value (see Section 3.3). ๐ธ
- 28, 29, 30: These values are reserved for future additions to the CBOR format. In the present version of CBOR, the encoded item is not well-formed. ๐
- 31: No argument value is derived. If the major type is 0, 1, or 6, the encoded item is not well-formed. For major types 2 to 5, the item's length is indefinite, and for major type 7, the byte does not constitute a data item at all but terminates an indefinite-length item; all are described in Section 3.2. โจ
The initial byte and any additional bytes consumed to build the argument are collectively called the _head_ of the data item ๐ฆ
What the argument means depends on the major type ๐. In major type 0, the argument is the value of the data item. In major type 1, the value is computed from the argument. In major types 2 and 3, it gives the length of the string data in bytes that follows. And in major types 4 and 5, it's used to figure out how many data items are enclosed.
If the encoded bytes end before the end of a data item, that item is not well-formed. If bytes are still left over after the outermost encoded item is decoded, that encoding is not a single well-formed CBOR item. Depending on the application, the decoder may either treat the encoding as not well-formed, or just point out where the remaining bytes start to the application ๐
A CBOR decoder can be built on a jump table covering all 256 defined values for the initial byte (Table 7) ๐. A decoder in a constrained implementation can instead use the structure of the initial byte and following bytes for more compact code (see Appendix C for a rough impression of how this could look). ๐ค
3.1. Major Types
Here's the lowdown on each major type and the additional information and other bytes that go with it ๐
Major type 0: An unsigned integer in the range 0..2^(64)-1 inclusive. The value of the encoded item is the argument itself. For example, the integer 10 is denoted as the one byte 0b000_01010 (major type 0, additional information 10). The integer 500 would be 0b000_11001 (major type 0, additional information 25) followed by the two bytes 0x01f4, which is 500 in decimal. ๐ฅ
Major type 1: A negative integer in the range -2^(64)..-1 inclusive. The value of the item is -1 minus the argument. For example, the integer -500 would be 0b001_11001 (major type 1, additional information 25) followed by the two bytes 0x01f3, which is 499 in decimal. ๐
Major type 2: A byte string. The number of bytes in the string is equal to the argument. For example, a byte string whose length is 5 would have an initial byte of 0b010_00101 (major type 2, additional information 5 for the length), followed by 5 bytes of binary content. A byte string whose length is 500 would have 3 initial bytes of 0b010_11001 (major type 2, additional information 25 to indicate a two-byte length), followed by the two bytes 0x01f4 for a length of 500, followed by 500 bytes of binary content. ๐ง
Major type 3: A text string (Section 2) encoded as UTF-8 [RFC3629]. The number of bytes in the string is equal to the argument. A string containing an invalid UTF-8 sequence is well-formed but invalid (Section 1.2). This type is provided for systems that need to interpret or display human-readable text, and allows the differentiation between unstructured bytes and text that has a specified repertoire (that of Unicode) and encoding (UTF-8). In contrast to formats such as JSON, the Unicode characters in this type are never escaped. Thus, a newline character (U+000A) is always represented in a string as the byte 0x0a, and never as the bytes 0x5c6e (the characters "" and "n") nor as 0x5c7530303061 (the characters "", "u", "0", "0", "0", and "a"). ๐
Major type 3 is the "no escaping, no cap" one: a newline is just 0x0a, full stop, not a backslash situation ๐
Major type 4: An array of data items. In other formats, arrays are also called lists, sequences, or tuples (a "CBOR sequence" is something slightly different, though [RFC8742]). The argument is the number of data items in the array. Items in an array do not need to all be of the same type. For example, an array that contains 10 items of any type would have an initial byte of 0b100_01010 (major type 4, additional information 10 for the length), followed by the 10 remaining items. ๐ฐ
Major type 5: A map of pairs of data items. Maps are also called tables, dictionaries, hashes, or objects (in JSON). A map is made up of pairs of data items, each pair being a key immediately followed by a value. The argument is the number of _pairs_ of data items in the map. For example, a map that contains 9 pairs would have an initial byte of 0b101_01001 (major type 5, additional information 9 for the number of pairs), followed by the 18 remaining items. The first item is the first key, the second item is the first value, the third item is the second key, and so on. Because items in a map come in pairs, their total number is always even: a map that contains an odd number of items (no value data present after the last key data item) is not well-formed. A map that has duplicate keys may be well-formed, but it is not valid, and thus it causes indeterminate decoding; see also Section 5.6. ๐๏ธ
Major type 6: A tagged data item ("tag") whose tag number, an integer in the range 0..2^(64)-1 inclusive, is the argument, and whose enclosed data item (_tag content_) is the single encoded data item that follows the head. See Section 3.4. ๐ท๏ธ
Major type 7: Floating-point numbers and simple values, as well as the "break" stop code. See Section 3.3. ๐
These eight major types lead to a simple table showing which of the 256 possible values for the initial byte of a data item are used (Table 7). ๐งฉ
In major types 6 and 7, many of the possible values are reserved for future specification. See Section 9 for more information on these values. ๐ฎ
Table 1 summarizes the major types defined by CBOR, ignoring Section 3.2 for now. The number N in this table stands for the argument. ๐
+============+=======================+=========================+
| Major Type | Meaning | Content |
+============+=======================+=========================+
| 0 | unsigned integer N | - |
+------------+-----------------------+-------------------------+
| 1 | negative integer -1-N | - |
+------------+-----------------------+-------------------------+
| 2 | byte string | N bytes |
+------------+-----------------------+-------------------------+
| 3 | text string | N bytes (UTF-8 text) |
+------------+-----------------------+-------------------------+
| 4 | array | N data items (elements) |
+------------+-----------------------+-------------------------+
| 5 | map | 2N data items (key/ |
| | | value pairs) |
+------------+-----------------------+-------------------------+
| 6 | tag of number N | 1 data item |
+------------+-----------------------+-------------------------+
| 7 | simple/float | - |
+------------+-----------------------+-------------------------+
Table 1: Overview over the Definite-Length Use of CBOR Major
Types (N = Argument)3.2. Indefinite Lengths for Some Major Types
Four CBOR items (arrays, maps, byte strings, and text strings) can be encoded with an indefinite length using additional information value 31. This is handy if the encoding of the item needs to start before the number of items inside the array or map, or the total length of the string, is known. (Being able to start sending a data item before all of it is known is often called "streaming" within that data item. ๐)
Indefinite-length arrays and maps are dealt with differently than indefinite-length strings (byte strings and text strings). ๐คธ
3.2.1. The "break" Stop Code
The "break" stop code is encoded with major type 7 and additional information value 31 (0b111_11111). It is not itself a data item: it is just a syntactic feature to close an indefinite-length item. ๐
If the "break" stop code appears where a data item is expected, other than directly inside an indefinite-length string, array, or map (for example, directly inside a definite-length array or map), the enclosing item is not well-formed. โ
3.2.2. Indefinite-Length Arrays and Maps โจ
okay so indefinite-length arrays and maps start with their major type plus the additional information value of 31 ๐. Then you just keep adding items (for an array) or key/value pairs (for a map), as many as you want, even zero ๐ธ, and close the whole thing off with the "break" stop code (Section 3.2.1) ๐ฆ. ngl the vibe is that they look exactly like regular arrays and maps, except they start with 31 and end with "break" ๐ . That's the tea.
If the "break" stop code shows up right after a key in a map, sitting where that key's value should go, the map is not well-formed ๐. Absolutely not.
There's no rule against nesting indefinite-length arrays or maps inside each other ๐ซถ. A "break" only closes one item, so nested indefinite-length items need exactly as many "break" stop codes as there are type bytes that start an indefinite-length item. It's giving bookkeeping, bestie ๐ง.
For example, say an encoder wants to represent the abstract array [1, [2, 3], [4, 5]]. The definite-length encoding would be 0x8301820203820405:
83 -- Array of length 3
01 -- 1
82 -- Array of length 2
02 -- 2
03 -- 3
82 -- Array of length 2
04 -- 4
05 -- 5Indefinite-length encoding can be applied independently to each of the three arrays in this data item, as required, which gives you stuff like:
0x9f018202039f0405ffff
9F -- Start indefinite-length array
01 -- 1
82 -- Array of length 2
02 -- 2
03 -- 3
9F -- Start indefinite-length array
04 -- 4
05 -- 5
FF -- "break" (inner array)
FF -- "break" (outer array)0x9f01820203820405ff
9F -- Start indefinite-length array
01 -- 1
82 -- Array of length 2
02 -- 2
03 -- 3
82 -- Array of length 2
04 -- 4
05 -- 5
FF -- "break"0x83018202039f0405ff
83 -- Array of length 3
01 -- 1
82 -- Array of length 2
02 -- 2
03 -- 3
9F -- Start indefinite-length array
04 -- 4
05 -- 5
FF -- "break"0x83019f0203ff820405
83 -- Array of length 3
01 -- 1
9F -- Start indefinite-length array
02 -- 2
03 -- 3
FF -- "break"
82 -- Array of length 2
04 -- 4
05 -- 5Here's an indefinite-length map (this one happens to have two key/value pairs) ๐:
0xbf6346756ef563416d7421ff
BF -- Start indefinite-length map
63 -- First key, UTF-8 string length 3
46756e -- "Fun"
F5 -- First value, true
63 -- Second key, UTF-8 string length 3
416d74 -- "Amt"
21 -- Second value, -2
FF -- "break"3.2.3. Indefinite-Length Byte Strings and Text Strings ๐
Indefinite-length strings are a byte with the major type for byte string or text string plus the additional information value of 31 ๐. After that you get a bunch of strings of that type ("chunks") that each have definite lengths, zero or more of them ๐ฆ, and then the "break" stop code (Section 3.2.1) wraps it up. The data item is just all the chunks glued together. No chunks at all? Then it's an empty string of that type ๐ซง. Zero-length chunks are allowed too. Are they useful? Not really, but they're permitted, so no drama โจ.
If any item between the indefinite-length string indicator (0b010_11111 or 0b011_11111) and the "break" stop code is not a definite-length string of the same major type, the string is not well-formed ๐.
The design doesn't let you nest indefinite-length strings as chunks inside other indefinite-length strings ๐ โโ๏ธ. Why? Decoders would need a stack (or at least a count) of nesting levels, which is extra mess. Encoders don't need it either, since the inner string's chunks could just go straight into the outer string ๐ .
If any definite-length text string inside an indefinite-length text string is invalid, then the whole indefinite-length text string is invalid ๐ญ. Also, the UTF-8 bytes of one Unicode code point (scalar value) can't be split across chunks. A new chunk of a text string can only start at a code point boundary. Boundaries are everything, bestie ๐ธ.
Here's an example. Say you have an encoded data item made of these bytes:
0b010_11111 0b010_00100 0xaabbccdd 0b010_00011 0xeeff99 0b111_11111
5F -- Start indefinite-length byte string
44 -- Byte string of length 4
aabbccdd -- Bytes content
43 -- Byte string of length 3
eeff99 -- Bytes content
FF -- "break"After decoding, you get one byte string with seven bytes: 0xaabbccddeeff99 ๐.
3.2.4. Summary of Indefinite-Length Use of Major Types ๐
Table 2 is the cheat sheet ๐. It sums up the major types CBOR defines, as used for indefinite-length encoding (additional information set to 31).
+============+===================+==================================+
| Major Type | Meaning | Enclosed up to "break" Stop Code |
+============+===================+==================================+
| 0 | (not well- | - |
| | formed) | |
+------------+-------------------+----------------------------------+
| 1 | (not well- | - |
| | formed) | |
+------------+-------------------+----------------------------------+
| 2 | byte string | definite-length byte strings |
+------------+-------------------+----------------------------------+
| 3 | text string | definite-length text strings |
+------------+-------------------+----------------------------------+
| 4 | array | data items (elements) |
+------------+-------------------+----------------------------------+
| 5 | map | data items (key/value pairs) |
+------------+-------------------+----------------------------------+
| 6 | (not well- | - |
| | formed) | |
+------------+-------------------+----------------------------------+
| 7 | "break" stop | - |
| | code | |
+------------+-------------------+----------------------------------+
Table 2: Overview of the Indefinite-Length Use of CBOR Major
Types (Additional Information = 31)3.3. Floating-Point Numbers and Values with No Content ๐ฆ
Major type 7 is the glow-up category, and it holds two kinds of data: floating-point numbers and "simple values" that don't need any extra content ๐. Each value of the 5-bit additional information in the initial byte has its own meaning, as defined in Table 3 ๐. Like the integer major types, items of this type don't carry content data. All the info lives in the initial bytes (the head) ๐ซถ.
+=============+===================================================+
| 5-Bit Value | Semantics |
+=============+===================================================+
| 0..23 | Simple value (value 0..23) |
+-------------+---------------------------------------------------+
| 24 | Simple value (value 32..255 in following byte) |
+-------------+---------------------------------------------------+
| 25 | IEEE 754 Half-Precision Float (16 bits follow) |
+-------------+---------------------------------------------------+
| 26 | IEEE 754 Single-Precision Float (32 bits follow) |
+-------------+---------------------------------------------------+
| 27 | IEEE 754 Double-Precision Float (64 bits follow) |
+-------------+---------------------------------------------------+
| 28-30 | Reserved, not well-formed in the present document |
+-------------+---------------------------------------------------+
| 31 | "break" stop code for indefinite-length items |
| | (Section 3.2.1) |
+-------------+---------------------------------------------------+
Table 3: Values for Additional Information in Major Type 7Like every other major type, the 5-bit value 24 signals a single-byte extension: it's followed by one more byte that holds the simple value ๐. (To keep things from getting confusing, only the values 32 to 255 get used.) This keeps the structure of the initial bytes intact. As with the other major types, the length always depends on the additional information in the first byte ๐ . Table 4 lists the numeric values that are assigned and available for simple values ๐.
+=========+==============+
| Value | Semantics |
+=========+==============+
| 0..19 | (unassigned) |
+---------+--------------+
| 20 | false |
+---------+--------------+
| 21 | true |
+---------+--------------+
| 22 | null |
+---------+--------------+
| 23 | undefined |
+---------+--------------+
| 24..31 | (reserved) |
+---------+--------------+
| 32..255 | (unassigned) |
+---------+--------------+
Table 4: Simple ValuesAn encoder MUST NOT issue two-byte sequences that start with 0xf8 (major type 7, additional information 24) and continue with a byte less than 0x20 (32 decimal) ๐ซ. Those sequences are not well-formed ๐. (This means an encoder can't encode "false", "true", "null", or "undefined" as two-byte sequences, and only the one-byte versions of these are well-formed. More generally, each simple value has only a single representation variant ๐ง.)
The 5-bit values 25, 26, and 27 are for 16-bit, 32-bit, and 64-bit IEEE 754 binary floating-point values [IEEE754] โจ. These floating-point values are encoded in the additional bytes of the matching size. (See Appendix D for some info about 16-bit floating-point numbers ๐ธ.)
3.4. Tagging of Items ๐
Okay bestie, so in CBOR you can wrap a data item in a tag to give it some extra semantics ๐ท๏ธ. Every tag is uniquely identified by a tag number. The tag is major type 6, its argument (Section 3) tells you the tag number, and it holds one enclosed data item, the tag content ๐ . (If a tag needs more structure, that structure is provided by the item it encloses.) We say tag for the whole package, the tag number and the tag content together. The tag content is the item that's getting tagged. โจ
Let's say a byte string of length 12 gets marked with tag number 2, which means it's an unsigned bignum (Section 3.4.3) ๐ข. The encoded data item starts with 0b110_00010 (major type 6, additional information 2 for the tag number). Then comes the encoded tag content, 0b010_01100 (major type 2, additional information 12 for the length), followed by the 12 bytes of the bignum. It's so specific and I'm obsessed ๐ญ
In the extended generic data model, a tag number's definition explains the extra semantics it brings. Those semantics can include equivalence between a tagged item and other items, including some that can be written in the basic generic data model ๐ฆ. For example, 0xc24101 is a bignum whose tag content is the byte string with the single byte 0x01, and that's equivalent to the integer 1. The integer 1 could also be encoded as 0x01, 0x1801, or 0x190001. The tag definition may name a preferred serialization (Section 4.1) that generic encoders are recommended to use. That preferred form may favour basic generic data model representations over ones that use a tag ๐ธ
The tag definition usually spells out which nested data items are valid for that tag. Some tags restrict their content to a very specific syntactic structure, like the tags defined in this document do. Others define their content more semantically. Tags 40 and 1040 are a good example of the second kind, since they accept multiple ways to represent arrays [RFC8746] ๐
As a matter of convention, lots of tags don't accept "null" or "undefined" values as tag content ๐. Instead, the expectation is that a "null" or "undefined" value gets used in place of the entire tag. Section 3.4.2 has some extra considerations for one specific tag about how this convention works in application protocols and when mapping to platform types.
Decoders don't have to understand every tag number, and tags might not matter much in applications where the sender and the receiver both already know what each item in the data flow means ๐ง The main purpose of tags in this spec is to define common data types, like dates ๐ . A secondary purpose is to give conversion hints for when a CBOR data item needs to be translated into a different format and those hints about the content are useful. Understanding tag semantics is optional for a decoder. It can just hand both the tag number and the tag content to the application, without interpreting the extra semantics. No pressure ๐
A tag applies its semantics to the data item it encloses. Tags can nest. If tag A encloses tag B, which encloses data item C, then tag A applies to the result of applying tag B to data item C. Main character energy, literally ๐
IANA maintains a registry of tag numbers, as described in Section 9.2 ๐. Table 5 lists the tag numbers that were defined in [RFC7049], with their definitions in the rest of this section. (Tag number 35 was also defined in [RFC7049], and we'll talk about it in Section 3.4.5.3.) Lots of other tag numbers have been defined since [RFC7049] came out, so check the registry described at Section 9.2 for the full list ๐
+=======+=============+==================================+
| Tag | Data Item | Semantics |
+=======+=============+==================================+
| 0 | text string | Standard date/time string; see |
| | | Section 3.4.1 |
+-------+-------------+----------------------------------+
| 1 | integer or | Epoch-based date/time; see |
| | float | Section 3.4.2 |
+-------+-------------+----------------------------------+
| 2 | byte string | Unsigned bignum; see |
| | | Section 3.4.3 |
+-------+-------------+----------------------------------+
| 3 | byte string | Negative bignum; see |
| | | Section 3.4.3 |
+-------+-------------+----------------------------------+
| 4 | array | Decimal fraction; see |
| | | Section 3.4.4 |
+-------+-------------+----------------------------------+
| 5 | array | Bigfloat; see Section 3.4.4 |
+-------+-------------+----------------------------------+
| 21 | (any) | Expected conversion to base64url |
| | | encoding; see Section 3.4.5.2 |
+-------+-------------+----------------------------------+
| 22 | (any) | Expected conversion to base64 |
| | | encoding; see Section 3.4.5.2 |
+-------+-------------+----------------------------------+
| 23 | (any) | Expected conversion to base16 |
| | | encoding; see Section 3.4.5.2 |
+-------+-------------+----------------------------------+
| 24 | byte string | Encoded CBOR data item; see |
| | | Section 3.4.5.1 |
+-------+-------------+----------------------------------+
| 32 | text string | URI; see Section 3.4.5.3 |
+-------+-------------+----------------------------------+
| 33 | text string | base64url; see Section 3.4.5.3 |
+-------+-------------+----------------------------------+
| 34 | text string | base64; see Section 3.4.5.3 |
+-------+-------------+----------------------------------+
| 36 | text string | MIME message; see |
| | | Section 3.4.5.3 |
+-------+-------------+----------------------------------+
| 55799 | (any) | Self-described CBOR; see |
| | | Section 3.4.6 |
+-------+-------------+----------------------------------+Table 5: Tag Numbers Defined in RFC 7049
Conceptually, tags get interpreted in the generic data model, not at (de-)serialization time. But a few tags, namely tag number 25 and tag number 29 [IANA.cbor-tags] at this time, have been registered with semantics that may need processing at (de-)serialization time โ ๏ธ. The decoder has to be aware of the exact sequence in which data items are encoded into the CBOR data item, and the encoder has to be in control of that sequence. So these tags can't be built on top of an arbitrary generic CBOR encoder or decoder, because those might not reflect the serialization order of map entries at the data model level, and the other way around. That's why their implementation typically has to be built into the generic encoder or decoder itself ๐ง. Defining new tags with this property is NOT RECOMMENDED ๐ โโ๏ธ
IANA allocated the tag numbers 65535, 4294967295, and 18446744073709551615, which are binary all-ones in 16-bit, 32-bit, and 64-bit form ๐. These are a convenience for implementers who want a single-integer data structure to signal either that a specific tag is present or that it's absent. That allocation is described in Section 10 of [CBOR-TAGS]. These tags aren't meant to show up in actual CBOR data items, and implementations MAY flag such an occurrence as an error ๐ฉ
Protocols can extend the generic data model (Section 2) with data items that represent points in time, using tag numbers 0 and 1. They can also represent arbitrarily sized integers with tag numbers 2 and 3, and floating-point values of arbitrary size and precision with tag numbers 4 and 5 ๐
3.4.1. Standard Date/Time String ๐
Tag number 0 contains a text string in the standard format described by the "date-time" production in [RFC3339], as refined by Section 3.3 of [RFC4287], and it represents the point in time described there. A nested item of another type is invalid. So is a text string that doesn't match the format described in [RFC4287] ๐ โโ๏ธ
3.4.2. Epoch-Based Date/Time โฐ
Tag number 1 contains a numerical value that counts the number of seconds from 1970-01-01T00:00Z in UTC time to the point in civil time being represented ๐ฐ๏ธ
The tag content MUST be an unsigned or negative integer (major types 0 and 1) or a floating-point number (major type 7 with additional information 25, 26, or 27). Other contained types are invalid. Very strict, very iconic ๐
Nonnegative values (major type 0 and nonnegative floating-point numbers) stand for time values on or after 1970-01-01T00:00Z UTC. They're interpreted according to POSIX [TIME_T] ๐ง. (POSIX time is also known as "UNIX Epoch time".) POSIX handles leap seconds specially, and that causes a 1-second discontinuity several times per decade. Also, applications that need to express times beyond early 2106 can't leave out support for 64-bit integers in the tag content ๐
Negative values (major type 1 and negative floating-point numbers) are interpreted as the application requirements determine, because there's no universal standard for UTC count-of-seconds time before 1970-01-01T00:00Z. This is especially true for points in time that come before discontinuities in national calendars ๐๏ธ The same goes for non-finite values ๐
To show fractional seconds, you can use floating-point values within tag number 1 instead of integer values โจ. This generally requires binary64 support, since binary16 and binary32 only give nonzero fractions of seconds for a short stretch of time around early 1970. An application that needs tag number 1 support may restrict the tag content to be an integer (or a floating-point value) only ๐
Platform types for date/time may include "null" or "undefined" values, and that might also be desirable at the application protocol level ๐ซถ. Emitting tag number 1 values with non-finite tag content, like NaN for undefined date/time values or Infinity for an expiry date that isn't set, might seem like an obvious way to handle this. But using untagged "null" or "undefined" avoids non-finite values and gives a shorter encoding โจ. Application protocol designers are encouraged to think about these cases and include clear guidelines for handling them ๐ธ
3.4.3. Bignums โจ
okay so, tag numbers 2 and 3 are how protocols get bignums ๐ข, which are integers of basically any size. They extend the generic data model from Section 2 ๐ . ngl there's a twist: in the basic generic data model, a bignum is not equal to an integer from that same model, even when the numbers match ๐. But the extended generic data model from this tag definition says equivalence is based on numeric value, and preferred serialization (Section 4.1) never uses a bignum for a number that can already be written as a basic integer (more on that below) ๐ซถ.
Bignums are encoded as a byte string data item, which gets read as an unsigned integer n in network byte order ๐. Contained items of other types are invalid ๐ซ. For tag number 2, the value of the bignum is n. For tag number 3, the value is -1 - n ๐. The preferred serialization of the byte string leaves out any leading zeroes ๐งน. (So for n = 0, the preferred serialization is the empty byte string, but see below ๐.) Decoders that understand these tags MUST be able to decode bignums that do have leading zeroes ๐ช. The preferred serialization of an integer that can be represented using major type 0 or 1 is to encode it that way instead of as a bignum. That means the empty string never shows up in a bignum when you're using preferred serialization ๐. A non-preferred bignum for a number that could be a basic integer isn't meant to carry application semantics. Same deal as using a longer basic integer than needed, like 0x1800 for 0x00 ๐คทโโ๏ธ.
For example, the number 18446744073709551616 (2^(64)) is written as 0b110_00010 (major type 6, tag number 2), followed by 0b010_01001 (major type 2, length 9), followed by 0x010000000000000000 (one byte 0x01 and eight bytes 0x00). In hexadecimal:
C2 -- Tag 2
49 -- Byte string of length 9
010000000000000000 -- Bytes content3.4.4. Decimal Fractions and Bigfloats ๐ฆ
okay this one's giving math glow-up ๐. Protocols using tag number 4 extend the generic data model with data items for arbitrary-length decimal fractions of the form m*(10^(e)). Protocols using tag number 5 extend the generic data model with arbitrary-length binary fractions of the form m*(2^(e)). Just like with bignums, values of different types are not equal in the generic data model ๐
โโ๏ธ.
Decimal fractions pair an integer mantissa with a base-10 scaling factor ๐. They're most useful when an app needs the exact representation of a decimal like 1.1, because lots of decimal fractions have no exact representation in binary floating-point ๐ญ.
"Bigfloats" pair an integer mantissa with a base-2 scaling factor ๐งฎ. They're binary floating-point values that can go past the range or precision of the three IEEE 754 formats CBOR supports (Section 3.3) ๐. Constrained apps that want some basic binary floating-point ability, without needing to support IEEE 754, can use them too ๐ซถ.
A decimal fraction or a bigfloat is a tagged array holding exactly two integers: an exponent e and a mantissa m ๐. Decimal fractions (tag number 4) use base-10 exponents, so the value is m*(10^(e)). Bigfloats (tag number 5) use base-2 exponents, so the value is m*(2^(e)). The exponent e MUST be represented in an integer of major type 0 or 1, while the mantissa can also be a bignum (Section 3.4.3) ๐. Contained items with other structures are invalid ๐ซ.
An example of a decimal fraction is the number 273.15, written as 0b110_00100 (major type 6 for the tag, additional information 4 for the tag number), followed by 0b100_00010 (major type 4 for the array, additional information 2 for the length of the array), followed by 0b001_00001 (major type 1 for the first integer, additional information 1 for the value of -2), followed by 0b000_11001 (major type 0 for the second integer, additional information 25 for a two-byte value), followed by 0b0110101010110011 (27315 in two bytes). In hexadecimal:
C4 -- Tag 4
82 -- Array of length 2
21 -- -2
19 6ab3 -- 27315An example of a bigfloat is the number 1.5, written as 0b110_00101 (major type 6 for the tag, additional information 5 for the tag number), followed by 0b100_00010 (major type 4 for the array, additional information 2 for the length of the array), followed by 0b001_00000 (major type 1 for the first integer, additional information 0 for the value of -1), followed by 0b000_00011 (major type 0 for the second integer, additional information 3 for the value of 3). In hexadecimal:
C5 -- Tag 5
82 -- Array of length 2
20 -- -1
03 -- 3Decimal fractions and bigfloats have no way to represent Infinity, -Infinity, or NaN ๐. If you need those in place of a decimal fraction or bigfloat, the IEEE 754 half-precision representations from Section 3.3 can be used ๐ซถ.
3.4.5. Content Hints ๐
the tags in this section are content hints that generic CBOR processors might use ๐. These content hints don't extend the generic data model, so nothing about the core data model changes here ๐ธ.
3.4.5.1. Encoded CBOR Data Item
sometimes it's super useful to carry an embedded CBOR data item that isn't meant to be decoded right away, when the outer item gets decoded ๐ฆ. Tag number 24 (CBOR data item) tags the embedded byte string as a single data item encoded in CBOR format ๐ง. Contained items that aren't byte strings are invalid ๐ซ. A contained byte string is valid if it encodes a well-formed CBOR data item. Checking that the decoded CBOR item is valid isn't required for tag validity, though a generic decoder could offer it as a special option ๐ .
3.4.5.2. Expected Later Encoding for CBOR-to-JSON Converters
tag numbers 21 to 23 are a heads-up that a byte string might need a specific encoding when it's working with a text-based format ๐ฆ. These tags are handy when the encoder knows the byte string data is probably going to get converted to a particular JSON-based usage later on. That usage says some strings get encoded as base64, base64url, and so on ๐. The encoder uses byte strings instead of doing the encoding itself, to keep the message size down, keep the encoder's code size down, or both ๐ง . The encoder doesn't know if the converter will be generic, so it's basically saying "this is what I think is the right way to convert binary strings to JSON" ๐.
The tagged data item can be a byte string or any other data item. If it's any other data item, the tag applies to all of the byte string data items inside it, except for the ones inside a nested data item that's tagged with an expected conversion ๐ซถ.
These three tag numbers suggest conversions to three of the base data encodings defined in [RFC4648] ๐. Tag number 21 suggests conversion to base64url encoding (Section 5 of [RFC4648]) where padding is not used (see Section 3.2 of [RFC4648]). That means all trailing equals signs ("=") get removed from the encoded string ๐งผ. Tag number 22 suggests conversion to classical base64 encoding (Section 4 of [RFC4648]) with padding as defined in RFC 4648 โจ. For both base64url and base64, the padding bits are set to zero (see Section 3.5 of [RFC4648]), and the conversion is performed on the contents of the byte string. That means no line breaks, whitespace, or other extra characters get added ๐. Tag number 23 suggests conversion to base16 (hex) encoding with uppercase alphabetics (see Section 8 of [RFC4648]) ๐ . Note that for all three tag numbers, the encoding of the empty byte string is the empty text string ๐ซง.
3.4.5.3. Encoded Text
some text strings hold data in formats that are widely used on the Internet. Sometimes the decoder can validate those formats and present them to the application in a proper form ๐. There are tags for some of these formats ๐.
- Tag number 32 is for URIs, as defined in [RFC3986]. If the text string doesn't match the
URI-referenceproduction, the string is invalid ๐. - Tag numbers 33 and 34 are for base64url- and base64-encoded text strings, respectively, as defined in [RFC4648]. If any of the following apply, the string is invalid ๐จ:
- the encoded text string contains non-alphabet characters, or only 1 alphabet character in the last block of 4 (where alphabet is defined by Section 5 of [RFC4648] for tag number 33 and Section 4 of [RFC4648] for tag number 34), or
- the padding bits in a 2- or 3-character block are not 0, or
- the base64 encoding has the wrong number of padding characters, or
- the base64url encoding has padding characters.
- Tag number 36 is for MIME messages (including all headers), as defined in [RFC2045]. A text string that isn't a valid MIME message is invalid. (For this tag, validity checking may be particularly onerous for a generic decoder and so might not be offered. Note that many MIME messages are general binary data and therefore can't be represented in a text string. [IANA.cbor-tags] lists a registration for tag number 257 that is similar to tag number 36 but uses a byte string as its tag content.) ๐ง
Note that tag numbers 33 and 34 differ from 21 and 22 in that the data is transported in base-encoded form for the former, and in raw byte string form for the latter ๐ง.
[RFC7049] also defined a tag number 35 for regular expressions in Perl Compatible Regular Expressions (PCRE/PCRE2) form [PCRE] or in JavaScript regular expression syntax [ECMA262]. The state of the art in these regex specs has moved on and keeps moving, so this specification doesn't try to update the references ๐. Instead, this tag stays available (as registered in [RFC7049]) for apps that specify the particular regex variant they use out-of-band. That might mean limiting usage to a defined common subset of both PCRE and ECMA262 ๐. This specification clarifies tag validity beyond [RFC7049]. Because the tag was defined so openly in [RFC7049], any contained string value needs to be valid at the CBOR tag level. It may still not be "expected" at the application level ๐.
3.4.6. Self-Described CBOR ๐ง
Okay so, in most apps it's super obvious that CBOR is what's being used ๐ Like a protocol might spell it out, or there's a media type saying "this is CBOR, bestie." But sometimes there's no context at all, like when CBOR data is chilling in a file with zero labels. In that case the data itself needs a little distinguishing vibe โจ
That's where tag number 55799 comes in ๐
It's meant for the very start of a stored encoded CBOR data item, as an application specifies. And here's the iconic part: it doesn't add any special semantics! The tag content it wraps means exactly the same thing as the content would mean on its own ๐ซถ
The head of this tag serializes as 0xd9d9f7, and that byte pattern doesn't seem to be used as a marker by any commonly used file types. Even better, 0xd9d9f7 can't be a valid start of a Unicode text in any Unicode encoding when a valid CBOR data item follows it ๐ญ๐ฆ
Here's a real-life example. A decoder might handle both CBOR and JSON, and it would need a way to tell the two formats apart mechanically ๐ค An easy assist from the encoder side is to wrap the entire CBOR item in tag number 55799, because that serialization will never show up at the beginning of a JSON text. Main character energy, no cap ๐
4. Serialization Considerations ๐ธ
4.1. Preferred Serialization
For some values at the data model level, CBOR gives you multiple ways to serialize them ๐ For a lot of apps, it's really nice when an encoder always picks a preferred serialization (the "preferred encoding"). BUT, and this is important, this specification doesn't force that burden onto the encoder or the decoder ๐
Some constrained decoders are a bit limited in what they can handle. Say only integers below 1_000_000_000 (one billion) show up in an app. The decoder is allowed to skip the code for 64-bit arguments in integers ๐ A "preferred encoder," which always uses the preferred serialization, plays nicely with that decoder for all the numbers that can actually occur in that app. In general, a preferred encoder is way more universally interoperable (and less wasteful) than one that always uses 64-bit integers, for example ๐
Constrained encoders can be a little limited too, so they might not support every representation variant. Then they won't always emit the preferred serialization, and they're called a "variant encoder" ๐ฆ For instance, a constrained encoder could always use the 32-bit variant for an integer it encodes, even when a shorter form exists. This assumes the app doesn't need integers that can only be represented with the 64-bit variant. A decoder that doesn't rely on getting only preferred serializations is a "variation-tolerant decoder," and it can be said to be more universally interoperable ๐ It might still optimize for the case of receiving preferred serializations, though! Full CBOR decoder implementations are variation tolerant by definition. The difference only matters when a constrained decoder meets a variant encoder โจ
The preferred serialization always uses the shortest form of representing the argument (Section 3). It also uses the shortest floating-point encoding that preserves the value being encoded ๐ฅ
For a floating-point value, the preferred serialization is the shortest floating-point encoding that keeps its value. For example, 0xf94580 is the number 5.5, and 0xfa45ad9c00 is the number 5555.5 ๐ For NaN values, a shorter encoding is preferred if zero-padding the shorter significand towards the right rebuilds the original NaN value. (For lots of apps, the single NaN encoding 0xf97e00 will be enough ๐ซถ)
Definite-length encoding is preferred whenever the length is known at the time the serialization of the item starts ๐ง
4.2. Deterministically Encoded CBOR
Some protocols want encoders to only emit CBOR in one particular deterministic format. Those protocols might also have decoders check that their input is in that format ๐ They're totally free to define what "deterministic format" means to them, and what encoders and decoders should do about it. This section just sets out a group of restrictions that can be the base for such a format, so it's like a starter kit ๐
4.2.1. Core Deterministic Encoding Requirements
A CBOR encoding satisfies the "core deterministic encoding requirements" if it meets all of the following restrictions ๐
- Preferred serialization MUST be used. In particular, this means that arguments (see Section 3) for integers, lengths in major types 2 through 5, and tags MUST be as short as possible. For instance:
- 0 to 23 and -1 to -24 MUST be expressed in the same byte as the major type;
- 24 to 255 and -25 to -256 MUST be expressed only with an additional
uint8_t; - 256 to 65535 and -257 to -65536 MUST be expressed only with an additional
uint16_t; - 65536 to 4294967295 and -65537 to -4294967296 MUST be expressed only with an additional
uint32_t.
Floating-point values also MUST use the shortest form that preserves the value. For example, 1.5 is encoded as 0xf93e00 (binary16) and 1000000.5 as 0xfa49742408 (binary32) ๐ One way to do this is to start with every float as a 64-bit float, then do a test conversion to a 32-bit float. If the numeric value stays the same, use the shorter form and repeat the process with a test conversion to a 16-bit float. This also works to select the 16-bit float for positive and negative Infinity as well โจ
- Indefinite-length items MUST NOT appear. They can be encoded as definite-length items instead ๐ โโ๏ธ
- The keys in every map MUST be sorted in the bytewise lexicographic order of their deterministic encodings. For example, these keys are sorted correctly ๐ง
10, encoded as0x0a.100, encoded as0x1864.-1, encoded as0x20."z", encoded as0x617a."aa", encoded as0x626161.[100], encoded as0x811864.[-1], encoded as0x8120.false, encoded as0xf4.
๐ง Implementation note: because CBOR encoding is self-delimiting, no two well-formed CBOR encoded data items can have one be a prefix of the other. So when you compare the bytewise lexicographic order of deterministic encodings of different map keys, the comparison always finds a byte that differs between the keys before reaching the end of either key ๐ค
4.2.2. Additional Deterministic Encoding Considerations
CBOR tags bring some extra things to think about for deterministic encoding ๐ Say a CBOR-based protocol gives the same semantics for a tag being present or absent. For example, it allows both tag 1 data items and raw numbers in a date/time position, and treats the raw numbers as if they were tagged. Then the deterministic format wouldn't allow the tag to be present, because of the "shortest form" principle ๐ Another example: a protocol might let encoders pick between a text string or, using Section 3.4.5.3, tag number 32 containing a text string, to represent a URL. That protocol's deterministic encoding needs to either require the tag or require that it's absent. It can't allow either one ๐
In a protocol that does require tags in certain places to get specific semantics, the tag needs to show up in the deterministic format too. Deterministic encoding considerations also apply to the content of tags ๐ซถ
If a protocol has a field that can express integers with an absolute value of 2^(64) or larger using tag numbers 2 or 3 (Section 3.4.3), its deterministic encoding needs to say whether smaller integers are expressed using these tags, or using major types 0 and 1. Preferred serialization uses the latter choice, so that's the recommended one ๐
Protocols that include floating-point values (whether they're basic floating-point values from Section 3.3, values using tags, or both) may need extra requirements for their deterministic encodings. Some examples are:
- Even though IEEE floating-point values can represent both positive and negative zero as distinct values, the application might not tell them apart. It could decide to represent all zero values with a positive sign, which disallows negative zero ๐ (The application may also want to limit the precision of floating-point values so that it never needs to represent 64-bit, or even 32-bit, floating-point values.)
- If a protocol has a field that can express floating-point values, and its data model says integer and floating-point values are interchangeable, the deterministic encoding needs to specify how the integer 1.0 gets encoded. The options are
0x01(unsigned integer),0xf93c00(binary16),0xfa3f800000(binary32), or0xfb3ff0000000000000(binary64). Here are some example rules:- Encode integral values that fit in 64 bits as values from major types 0 and 1, and other values as the preferred (smallest of 16-, 32-, or 64-bit) floating-point representation that accurately represents the value,
- Encode all values as the preferred floating-point representation that accurately represents the value, even for integral values, or
- Encode all values as 64-bit floating-point representations.
Rule 1 straddles the boundary between integers and floating-point values, and Rule 3 doesn't use preferred serialization. So Rule 2 may be a good choice in many cases ๐
- If NaN is an allowed value, and there's no intent to support NaN payloads or signaling NaNs, the protocol needs to pick a single representation, typically
0xf97e00. If that simple choice isn't possible, NaN handling will need specific attention ๐ญ - Subnormal numbers (nonzero numbers with the lowest possible exponent of a given IEEE 754 number format) may be flushed to zero outputs, or treated as zero inputs, in some floating-point implementations. A protocol's deterministic encoding may want to specifically accommodate those implementations. But that puts an onus on other implementations, by excluding subnormal numbers from interchange and interchanging zero instead ๐
- The same number can be represented by different decimal fractions, by different bigfloats, and by different forms under other tags that might be defined to express numeric values. Depending on the implementation, it may not always be practical to figure out whether any of these forms (or forms in the basic generic data model) are equivalent. An application protocol that offers choices like this for how numbers are represented needs to be explicit about how the formats for deterministic encoding get chosen ๐ฆ
4.2.3. Length-First Map Key Ordering ๐
Okay so, the core deterministic encoding requirements (Section 4.2.1) sort map keys in a different order than the one Section 3.9 of [RFC7049] suggested. That one was called "Canonical CBOR" ๐. If your protocol needs to match the [RFC7049] order, you can define it using this spec's "length-first core deterministic encoding requirements" instead โจ:
A CBOR encoding satisfies the "length-first core deterministic encoding requirements" if it satisfies the core deterministic encoding requirements except that the keys in every map MUST be sorted such that:
- If two keys have different lengths, the shorter one sorts earlier;
- If two keys have the same length, the one with the lower value in (bytewise) lexical order sorts earlier.
For example, under the length-first core deterministic encoding requirements, these keys are sorted correctly ๐ง:
- 10, encoded as
0x0a. - -1, encoded as
0x20. - false, encoded as
0xf4. - 100, encoded as
0x1864. - "z", encoded as
0x617a. - [-1], encoded as
0x8120. - "aa", encoded as
0x626161. - [100], encoded as
0x811864.
Although [RFC7049] used the term "Canonical CBOR" for its deterministic encoding rules, this document avoids it ๐. "Canonicalization" tends to get tied to specific uses only. The terms are basically interchangeable, though! The core set of requirements here could also be called "Canonical CBOR," and the length-first version could be called "Old Canonical CBOR" ๐.
5. Creating CBOR-Based Protocols ๐
Data formats like CBOR are often used where there's no format negotiation, and that's kind of the whole point โจ. A specific design goal of CBOR is that it doesn't need any included or assumed schema. A decoder can take a CBOR item and decode it with zero other knowledge ๐คฏ.
Of course, in real life, the encoder and the decoder will share a view of what should be in a CBOR data item ๐ซถ. For example, an agreed-to format might be "the item is an array whose first value is a UTF-8 string, second value is an integer, and subsequent values are zero or more floating-point numbers" or "the item is a map that has byte strings for keys and contains a pair whose key is 0xab01".
CBOR-based protocols MUST specify how their decoders handle invalid and other unexpected data. CBOR-based protocols MAY specify that they treat arbitrary valid data as unexpected. Encoders for CBOR-based protocols MUST produce only valid items. In other words, the protocol can't be designed to make use of invalid items ๐ โโ๏ธ. An encoder can be capable of encoding as many or as few types of values as the protocol needs, and a decoder can understand as many or as few types as the protocols using it need. This lack of restrictions is what lets CBOR work in extremely constrained environments ๐ธ.
The rest of this section covers some things to think about when creating CBOR-based protocols. With a few exceptions, it's advisory only, and it explicitly leaves out any BCP 14 [RFC2119] [RFC8174] language other than words that could be read as "MAY" in the BCP 14 sense. The exceptions are there to help CBOR-based protocols interoperate while still working with a wide variety of generic and application-specific encoders and decoders ๐.
5.1. CBOR in Streaming Applications ๐
In a streaming application, a data stream can be a sequence of CBOR data items concatenated back-to-back. In that setup, the decoder immediately starts decoding a new data item if data shows up after the end of the previous one ๐ฆ.
Not all of the bytes making up a data item will necessarily be available to the decoder right away. Some decoders buffer additional data until a complete data item can be handed to the application. Other decoders can pass partial information about a top-level data item to the application, like nested data items that could already be decoded, or even parts of a byte string that haven't fully arrived yet ๐. An application like this MUST also have a matching streaming security mechanism, so the desired protection is available for incremental data given to the application ๐.
Some applications and protocols won't want indefinite-length encoding, and that's valid ๐ . Indefinite-length encoding lets an encoder skip marshalling all the data up front for counting. The tradeoff is that the decoder has to allocate more and more memory while it waits for the end of the item ๐ญ. That might be fine for some apps but not others.
5.2. Generic Encoders and Decoders ๐งฉ
A generic CBOR decoder can decode all well-formed encoded CBOR data items and present them to an application. See Appendix C. (The diagnostic notation, Section 8, can be used to show well-formed CBOR values to humans ๐ฉโ๐ป.)
Generic CBOR encoders give the application an interface that lets it specify any well-formed value to be encoded as a CBOR data item. That includes simple values and tags the encoder doesn't know about ๐.
CBOR tries really hard to minimize these cases, but not all well-formed CBOR data is valid ๐ฌ. For example, the encoded text string "0x62c0ae" doesn't contain valid UTF-8, because [RFC3629] requires always using the shortest form, so it isn't a valid CBOR item. Also, specific tags can have semantic constraints that get violated. For instance, a bignum tag could enclose another tag, or tag number 0 could contain a byte string, or a text string whose contents don't match the "date-time" production of [RFC3339] ๐. Generic encoders and decoders don't have to make unnatural choices in their application interface just to process invalid data ๐
โโ๏ธ. Generic encoders and decoders are expected to forward simple values and tags even if their specific codepoints weren't registered yet when the encoder or decoder was written (Section 5.4) ๐.
5.3. Validity of Items ๐
A well-formed but invalid CBOR data item (Section 1.2) makes it tricky to interpret the data inside it in the CBOR data model ๐ค. A CBOR-based protocol could be specified in layers, where the lower layers don't process the semantics of some of the CBOR data they forward. Those layers can't notice validity errors in data they don't process, so they MUST forward that data as-is. The first layer that does process the semantics of an invalid CBOR item MUST pick one of two choices:
- Replace the problematic item with an error marker and continue with the next item, or
- Issue an error and stop processing altogether.
A CBOR-based protocol MUST specify which of these options its decoders take for each kind of invalid item they might run into ๐.
These problems can show up at the basic validity level of CBOR, or in the context of tags (tag validity).
5.3.1. Basic validity ๐งฑ
Two kinds of validity errors can happen in the basic generic data model:
- Duplicate keys in a map: Generic decoders (Section 5.2) hand data to applications using the native CBOR data model. That model includes maps (key-value mappings with unique keys), not multimaps (key-value mappings where multiple entries can share the same key). So a generic decoder that gets a CBOR map item with duplicate keys will either decode to a map with only one instance of that key, or stop processing altogether. A "streaming decoder," on the other hand, may not even be able to notice ๐. See Section 5.6 for more on keys in maps.
- Invalid UTF-8 string: A decoder might or might not want to check that the bytes in a UTF-8 string (major type 3) are actually valid UTF-8, and react appropriately if they aren't ๐คทโโ๏ธ.
5.3.2. Tag validity ๐ท๏ธ
Two more kinds of validity errors come from adding tags to the basic generic data model:
- Inadmissible type for tag content: Tag numbers (Section 3.4) specify what type of data item is supposed to be used as their tag content. For example, the tag numbers for unsigned or negative bignums are supposed to go on byte strings. A decoder that turns the tagged item into a native representation (a native big integer, in this example) is expected to check the type of the item being tagged. Even decoders without such native representations in their environment may check the tags they know about and react appropriately ๐.
- Inadmissible value for tag content: The type of data item might be admissible for a tag's content, while the specific value isn't. For example, "yesterday" isn't acceptable as the content of tag 0, even though it's properly a text string ๐ . A decoder that normally turns such tags into equivalent platform types might present the tag to the application the same way it would present a tag with an unknown tag number (Section 5.4) ๐โโ๏ธ.
5.4. Validity and Evolution ๐ฆ
A decoder with validity checking will put in the effort to reliably detect data items with validity errors ๐ช. For example, such a decoder needs an API that reports an error (and doesn't return data) for any CBOR data item containing one of the validity errors listed in the previous subsection ๐ซ.
The set of tags in the "Concise Binary Object Representation (CBOR) Tags" registry (Section 9.2), and the set of simple values in the "Concise Binary Object Representation (CBOR) Simple Values" registry (Section 9.1), can grow at any time, beyond what a generic decoder understands ๐ฑ. When a validity-checking decoder runs into one of these unrecognized cases, it can do one of two things:
- It can report an error (and not return data). Note that treating this case as an error can cause ossification, so it's not encouraged ๐ โโ๏ธ. This error isn't a validity error, per se. This kind of error is more likely to come from a decoder that would be doing validity checking if this were a known case.
- It can emit the unknown item (type, value, and, for tags, the decoded tagged data item) to the application calling the decoder. Then it gives the application an indication that the decoder didn't recognize that tag number or simple value ๐ซง.
The second approach is also fine for decoders that don't support validity checking ๐. It gives forward compatibility with newly registered tags and simple values, without needing to update the encoder at the same time as the calling application โจ. (For this, the decoder's API needs a way to mark unknown items, so the calling application can handle them in a way that makes sense for the program ๐ง .)
Since some of the processing needed for validity checking can be pretty costly (especially duplicate detection for maps), support for validity checking is not a requirement placed on all CBOR decoders ๐ธ.
Some encoders rely on their applications to provide input data in a way that makes the encoder produce valid CBOR ๐ค. A generic encoder may also want a validity-checking mode, where it reliably limits its output to valid CBOR, no matter whether its application is actually providing API-conformant data ๐ก๏ธ.
5.5. Numbers ๐ขโจ
okay bestie, so CBOR-based protocols should take into account that different language environments have different rules about the range and precision of numbers they can represent ๐ธ. For example, basic JavaScript treats every number as a floating-point value, which means you can silently lose precision when decoding integers with more than 53 significant bits ๐. Another fun one: CBOR keeps the sign bit for its integer representation in the major type, so it gets one extra bit for signed numbers of a certain length (like -2^(64)..2^(64)-1 for 1+8-byte integers) compared to the usual platform signed integer of the same length (-2^(63)..2^(63)-1 for 8-byte int64_t) ๐งฎ. So a protocol that uses numbers should define what it expects from decoders and receiving applications when it comes to the nontrivial ones ๐.
A CBOR-based protocol that includes floating-point numbers can restrict which of the three formats (half-precision, single-precision, and double-precision) are to be supported ๐ฆ. For an integer-only application, a protocol may want to completely exclude the use of floating-point values ๐ โโ๏ธ.
A CBOR-based protocol designed for compactness may want to exclude specific integer encodings that are longer than necessary for the application, such as to save the need to implement 64-bit integers ๐. There is an expectation that encoders will use the most compact integer representation that can represent a given value ๐
. However, a compact application that does not require deterministic encoding should accept values that use a longer-than-needed encoding (such as encoding "0" as 0b000_11001 followed by two bytes of 0x00) as long as the application can decode an integer of the given size ๐ง. Similar considerations apply to floating-point values; decoding both preferred serializations and longer-than-needed ones is recommended ๐.
CBOR-based protocols for constrained applications that provide a choice between representing a specific number as an integer and as a decimal fraction or bigfloat (such as when the exponent is small and nonnegative) might express a quality-of-implementation expectation that the integer representation is used directly โจ.
5.6. Specifying Keys for Maps ๐๏ธ๐
the encoding and decoding applications need to agree on what types of keys are going to be used in maps ๐ค. In applications that need to interwork with JSON-based applications, conversion is simplified by limiting keys to text strings only ๐งต. Otherwise, there has to be a specified mapping from the other CBOR types to text strings, and this often leads to implementation errors ๐ญ. In applications where keys are numeric in nature, and numeric ordering of keys is important to the application, directly using the numbers for the keys is useful ๐ข.
If multiple types of keys are to be used, consideration should be given to how these types would be represented in the specific programming environments that are to be used ๐. For example, in JavaScript Maps [ECMA262], a key of integer 1 cannot be distinguished from a key of floating-point 1.0 ๐. This means that, if integer keys are used, the protocol needs to avoid the use of floating-point keys the values of which happen to be integer numbers in the same map ๐ซ.
Decoders that deliver data items nested within a CBOR data item immediately on decoding them ("streaming decoders") often do not keep the state that is necessary to ascertain uniqueness of a key in a map ๐. Similarly, an encoder that can start encoding data items before the enclosing data item is completely available ("streaming encoder") may want to reduce its overhead significantly by relying on its data source to maintain uniqueness ๐ฆ.
A CBOR-based protocol MUST define what to do when a receiving application sees multiple identical keys in a map ๐จ. The resulting rule in the protocol MUST respect the CBOR data model: it cannot prescribe a specific handling of the entries with the identical keys, except that it might have a rule that having identical keys in a map indicates a malformed map and that the decoder has to stop with an error ๐. When processing maps that exhibit entries with duplicate keys, a generic decoder might do one of the following:
- Not accept maps with duplicate keys (that is, enforce validity for maps, see also Section 5.4). These generic decoders are universally useful. An application may still need to perform its own duplicate checking based on application rules (for instance, if the application equates integers and floating-point values in map key positions for specific maps) ๐ง.
- Pass all map entries to the application, including ones with duplicate keys. This requires that the application handle (check against) duplicate keys, even if the application rules are identical to the generic data model rules ๐ซถ.
- Lose some entries with duplicate keys, e.g., deliver only the final (or first) entry out of the entries with the same key. With such a generic decoder, applications may get different results for a specific key on different runs, and with different generic decoders, which value is returned is based on generic decoder implementation and the actual order of keys in the map. In particular, applications cannot validate key uniqueness on their own as they do not necessarily see all entries; they may not be able to use such a generic decoder if they need to validate key uniqueness. These generic decoders can only be used in situations where the data source and transfer always provide valid maps; this is not possible if the data source and transfer can be attacked ๐โ ๏ธ.
Generic decoders need to document which of these three approaches they implement ๐.
The CBOR data model for maps does not allow ascribing semantics to the order of the key/value pairs in the map representation ๐ธ. Thus, a CBOR-based protocol MUST NOT specify that changing the key/value pair order in a map changes the semantics, except to specify that some orders are disallowed, for example, where they would not meet the requirements of a deterministic encoding (Section 4.2) ๐ . (Any secondary effects of map ordering such as on timing, cache usage, and other potential side channels are not considered part of the semantics but may be enough reason on their own for a protocol to require a deterministic encoding format ๐.)
Applications for constrained devices should consider using small integers as keys if they have maps with a small number of frequently used keys ๐. For instance, a set of 24 or fewer keys can be encoded in a single byte as unsigned integers, up to 48 if negative integers are also used ๐ง . Less frequently occurring keys can then use integers with longer encodings ๐.
5.6.1. Equivalence of Keys ๐งฉ
The specific data model that applies to a CBOR data item is used to determine whether keys occurring in maps are duplicates or distinct ๐.
At the generic data model level, numerically equivalent integer and floating-point values are distinct from each other, as they are from the various big numbers (Tags 2 to 5) ๐ โโ๏ธ. Similarly, text strings are distinct from byte strings, even if composed of the same bytes ๐งต. A tagged value is distinct from an untagged value or from a value tagged with a different tag number ๐ท๏ธ.
Within each of these groups, numeric values are distinct unless they are numerically equal (specifically, -0.0 is equal to 0.0) ๐ข. For the purpose of map key equivalence, NaN values are equivalent if they have the same significand after zero-extending both significands at the right to 64 bits ๐.
Both byte strings and text strings are compared byte by byte, arrays are compared element by element, and are equal if they have the same number of bytes/elements and the same values at the same positions ๐ซง. Two maps are equal if they have the same set of pairs regardless of their order; pairs are equal if both the key and value are equal ๐.
Tagged values are equal if both the tag number and the tag content are equal ๐ท๏ธ. (Note that a generic decoder that provides processing for a specific tag may not be able to distinguish some semantically equivalent values, e.g., if leading zeroes occur in the content of tag 2 or tag 3 (Section 3.4.3).) Simple values are equal if they simply have the same value ๐. Nothing else is equal in the generic data model; a simple value 2 is not equivalent to an integer 2, and an array is never equivalent to a map ๐
โโ๏ธ.
As discussed in Section 2.2, specific data models can make values equivalent for the purpose of comparing map keys that are distinct in the generic data model ๐ฆ. Note that this implies that a generic decoder may deliver a decoded map to an application that needs to be checked for duplicate map keys by that application (alternatively, the decoder may provide a programming interface to perform this service for the application) ๐ . Specific data models are not able to distinguish values for map keys that are equal for this purpose at the generic data model level ๐ซถ.
5.7. Undefined Values ๐ซฅ
In some CBOR-based protocols, the simple value (Section 3.3) of "undefined" might be used by an encoder as a substitute for a data item with an encoding problem, in order to allow the rest of the enclosing data items to be encoded without harm ๐ฉนโจ.
6. Converting Data between CBOR and JSON ๐๐ซ
This section gives non-normative advice about converting between CBOR and JSON ๐. Implementations of converters MAY use whichever advice here they want ๐.
It is worth noting that a JSON text is a sequence of characters, not an encoded sequence of bytes, while a CBOR data item consists of bytes, not characters ๐ธ๐ง .
6.1. Converting from CBOR to JSON ๐
okay so most CBOR types have a direct JSON buddy ๐ but a few don't, and if you're building a CBOR-to-JSON converter you have to decide what happens to those oddballs โจ this part is non-normative advice, and it handles them by swapping each one for a single substitute value, like a JSON null ๐ง
- An integer (major type 0 or 1) becomes a JSON number. ๐ข
- A byte string (major type 2) that is not embedded in a tag that specifies a proposed encoding is encoded in base64url without padding and becomes a JSON string. ๐ฆ
- A UTF-8 string (major type 3) becomes a JSON string. Note that JSON requires escaping certain characters ([RFC8259], Section 7): quotation mark (
U+0022), reverse solidus (U+005C), and the "C0 control characters" (U+0000throughU+001F). All other characters are copied unchanged into the JSON UTF-8 string. ๐ - An array (major type 4) becomes a JSON array. ๐
- A map (major type 5) becomes a JSON object. This is possible directly only if all keys are UTF-8 strings. A converter might also convert other keys into UTF-8 strings (such as by converting integers into strings containing their decimal representation); however, doing so introduces a danger of key collision. Note also that, if tags on UTF-8 strings are ignored as proposed below, this will cause a key collision if the tags are different but the strings are the same. this is the kind of collision that sneaks up on you ๐ญ
- False (major type 7, additional information 20) becomes a JSON false. โ
- True (major type 7, additional information 21) becomes a JSON true. โ
- Null (major type 7, additional information 22) becomes a JSON null. ๐ซง
- A floating-point value (major type 7, additional information 25 through 27) becomes a JSON number if it is finite (that is, it can be represented in a JSON number); if the value is non-finite (NaN, or positive or negative Infinity), it is represented by the substitute value. ๐ธ
- Any other simple value (major type 7, any additional information value not yet discussed) is represented by the substitute value. ๐
- A bignum (major type 6, tag number 2 or 3) is represented by encoding its byte string in base64url without padding and becomes a JSON string. For tag number 3 (negative bignum), a
~(ASCII tilde) is inserted before the base-encoded value. (The conversion to a binary blob instead of a number is to prevent a likely numeric overflow for the JSON decoder.) ๐ - A byte string with an encoding hint (major type 6, tag number 21 through 23) is encoded as described by the hint and becomes a JSON string. โจ
- For all other tags (major type 6, any other tag number), the tag content is represented as a JSON value; the tag number is ignored. ๐งโโ๏ธ
- Indefinite-length items are made definite before conversion. ๐ง
A CBOR-to-JSON converter may want to keep to the JSON profile I-JSON [RFC7493], to maximize interoperability and increase confidence that the JSON output can be processed with predictable results. For example, this has implications on the range of integers that can be represented reliably, as well as on the top-level items that may be supported by older JSON implementations. basically, I-JSON is the "play nice with everyone" mode ๐ซถ
6.2. Converting from JSON to CBOR ๐ฆ
All JSON values, once decoded, directly map into one or more CBOR values. As with any kind of CBOR generation, decisions have to be made with respect to number representation. In a suggested conversion:
- JSON numbers without fractional parts (integer numbers) are represented as integers (major types 0 and 1, possibly major type 6, tag number 2 and 3), choosing the shortest form; integers longer than an implementation-defined threshold may instead be represented as floating-point values. The default range that is represented as integer is
-2^(53)+1..2^(53)-1(fully exploiting the range for exact integers in the binary64 representation often used for decoding JSON [RFC7493]). A CBOR-based protocol, or a generic converter implementation, may choose-2^(32)..2^(32)-1or-2^(64)..2^(64)-1(fully using the integer ranges available in CBOR withuint32_toruint64_t, respectively) or even-2^(31)..2^(31)-1or-2^(63)..2^(63)-1(using popular ranges for two's complement signed integers). (If the JSON was generated from a JavaScript implementation, its precision is already limited to 53 bits maximum.) ๐ขโจ - Numbers with fractional parts are represented as floating-point values, performing the decimal-to-binary conversion based on the precision provided by IEEE 754 binary64. The mathematical value of the JSON number is converted to binary64 using the roundTiesToEven procedure in Section 4.3.1 of [IEEE754]. Then, when encoding in CBOR, the preferred serialization uses the shortest floating-point representation exactly representing this conversion result; for instance,
1.5is represented in a 16-bit floating-point value (not all implementations will be capable of efficiently finding the minimum form, though). Instead of using the default binary64 precision, there may be an implementation-defined limit to the precision of the conversion that will affect the precision of the represented values. Decimal representation should only be used on the CBOR side if that is specified in a protocol. ๐
CBOR has been designed to generally provide a more compact encoding than JSON. One implementation strategy that might come to mind is to perform a JSON-to-CBOR encoding in place in a single buffer. This strategy would need to carefully consider a number of pathological cases, such as that some strings represented with no or very few escapes and longer (or much longer) than 255 bytes may expand when encoded as UTF-8 strings in CBOR. Similarly, a few of the binary floating-point representations might cause expansion from some short decimal representations (1.1, 1e9) in JSON. This may be hard to get right, and any ensuing vulnerabilities may be exploited by an attacker. ngl this is a security warning, so don't skip it ๐๐ฅ
7. Future Evolution of CBOR ๐
okay, successful protocols evolve over time ๐ฆ new ideas show up, platforms improve, related protocols grow, and new requirements keep rolling in. So making evolution easy is a big design concern for any protocol โจ
For protocols that will use CBOR, CBOR gives you some handy mechanisms to help them evolve. Best practices for this are already well known, especially from the JSON format development of JSON-based protocols. So those best practices are outside the scope of this specification. ๐
BUT making CBOR itself evolve-friendly is totally in scope ๐ CBOR is meant to be a stable base for CBOR-based protocols and also be able to change over time. A successful protocol may live for decades, so CBOR needs to be designed for decades of use and evolution. This section gives some guidance on how CBOR can evolve. It's necessarily more subjective than other parts of this document, and it's necessarily incomplete, lest it turn into a textbook on protocol development. ๐
7.1. Extension Points ๐งฉ
In a protocol design, opportunities for evolution are often included in the form of extension points. For example, there may be a codepoint space that is not fully allocated from the outset, and the protocol is designed to tolerate and embrace implementations that start using more codepoints than initially allocated. ๐ธ
Sizing the codepoint space may be difficult because the range required may be hard to predict. Protocol designs should attempt to make the codepoint space large enough so that it can slowly be filled over the intended lifetime of the protocol. ๐
CBOR has three major extension points:
- the "simple" space (values in major type 7): Of the 24 efficient (and 224 slightly less efficient) values, only a small number have been allocated. Implementations receiving an unknown simple data item may easily be able to process it as such, given that the structure of the value is indeed simple. The IANA registry in Section 9.1 is the appropriate way to address the extensibility of this codepoint space. ๐
- the "tag" space (values in major type 6): The total codepoint space is abundant; only a tiny part of it has been allocated. However, not all of these codepoints are equally efficient: the first 24 only consume a single (
1+0) byte, and half of them have already been allocated. The next 232 values only consume two (1+1) bytes, with nearly a quarter already allocated. These subspaces need some curation to last for a few more decades. Implementations receiving an unknown tag number can choose to process just the enclosed tag content or, preferably, to process the tag as an unknown tag number wrapping the tag content. The IANA registry in Section 9.2 is the appropriate way to address the extensibility of this codepoint space. ๐ท๏ธ - the "additional information" space: An implementation receiving an unknown additional information value has no way to continue decoding, so allocating codepoints in this space is a major step beyond just exercising an extension point. There are also very few codepoints left. See also Section 7.2. this one is the tightest real estate in the whole spec ๐
7.2. Curating the Additional Information Space ๐
The human mind is sometimes drawn to filling in little perceived gaps to make something neat. We expect the remaining gaps in the codepoint space for the additional information values to be an attractor for new ideas, just because they are there. ๐ง โจ
The present specification does not manage the additional information codepoint space by an IANA registry. Instead, allocations out of this space can only be done by updating this specification. ๐
For an additional information value of n >= 24, the size of the additional data typically is 2^(n-24) bytes. Therefore, additional information values 28 and 29 should be viewed as candidates for 128-bit and 256-bit quantities, in case a need arises to add them to the protocol. Additional information value 30 is then the only additional information value available for general allocation, and there should be a very good reason for allocating it before assigning it through an update of the present specification. so yeah, 30 is basically the last good seat at the table ๐ช๐
8. Diagnostic Notation โจ
okay so CBOR is a binary interchange format ๐ค, which means the actual data is always sent in binary. but humans still need to read it when they're debugging, so this section gives us a simple, human-readable diagnostic notation ๐ซถ. all real interchange still happens in binary, just so we're clear ๐
this truly is a diagnostic format, not something you parse ๐ so there's no formal grammar (no ABNF) for it here. if you need a text format for CBOR data items in config files, YAML [YAML] might be more your vibe ๐
the notation is loosely based on JSON as defined in RFC 8259, with some extras added where needed ๐ธ it borrows JSON's syntax for numbers (integer and floating-point), true, false, null, UTF-8 strings, arrays, and maps. (in JSON these are called objects, but the diagnostic notation lets any data item sit in the key position, which is very main character of it ๐) undefined is written undefined, like in JavaScript ๐ง the non-finite floating-point numbers Infinity, -Infinity, and NaN are written exactly as I just wrote them. JavaScript does this too, even though JSON won't allow them ๐
a tag is written as its tag number, followed by the tag content in parentheses โจ for example, a date in the format from RFC 3339 (ISO 8601) could be notated like this ๐ฆ
0("2013-03-21T20:04:00Z")or the same moment as relative time, like this ๐
1(1363896240)byte strings get notated in one of the base encodings, with no padding, wrapped in single quotes and prefixed by h for base16, b32 for base32, h32 for base32hex, or b64 for base64 or base64url ๐ the encodings don't overlap, so the string stays unambiguous, which is honestly so satisfying ๐ญ for example, the byte string 0x12345678 could be written as h'12345678', b32'CI2FM6A', or b64'EjRWeA'.
unassigned simple values get written as simple() with the right integer inside the parentheses ๐ for example, simple(42) means major type 7, value 42.
a bunch of useful extras for this notation are in Appendix G of [RFC8610], "Extended Diagnostic Notation" (EDN) ๐ฅ and this notation could be extended in a separate document to cover NaN payloads, which aren't covered here ๐ซถ
8.1. Encoding Indicators
sometimes you want to show which of several alternative representations was actually used ๐ like, a data item written as 1.5 by a diagnostic decoder might have been encoded as a half-, single-, or double-precision float ๐คฏ
the convention is that anything starting with an underscore, plus all the following characters that are alphanumeric or underscores, is an encoding indicator ๐ธ anyone who doesn't care about this info can just ignore it. examples are _ or _3. and encoding indicators are always optional, so no pressure โจ
a single underscore can go right after the opening brace of a map or the opening bracket of an array, to show the item was represented in indefinite-length format ๐ง for example, [_ 1, 2] has an indicator saying an indefinite-length representation was used for the data item [1, 2].
an underscore followed by a decimal digit n means the preceding item (or, for arrays and maps, the item starting with the preceding bracket or brace) was encoded with an additional information value of 24+n ๐ so 1.5_1 is a half-precision floating-point number, while 1.5_3 is encoded as double precision. this indicator doesn't show up in Appendix A. (and yes, the plain _ is basically a shortcut for the full form _7, which just isn't used ๐)
the detailed chunk structure of indefinite-length byte and text strings can be notated like this: (_ h'0123', h'4567') and (_ "foo", "bar") ๐ but an indefinite-length string with no chunks inside can't be written as (_ ), because that would be ambiguous. it could mean either a byte string (0x5fff) or a text string (0x7fff), and that's a no-go ๐ญ so the basic forms ''_ and ""_ are used for that case instead. they're reserved for the no-chunks case only. they aren't shortcuts for the (allowed, but not really useful) encodings with only empty chunks, which have to be notated as (_ ''), (_ ""), and so on, so the chunk structure stays intact ๐
9. IANA Considerations ๐
okay now the IANA bit ๐๏ธ IANA has made two registries for new CBOR values ๐ the registries are separate, not under some umbrella registry, and they follow the rules in [RFC8126] ๐ธ IANA also assigned a new media type, an associated CoAP Content-Format entry, and a structured syntax suffix. we're eating well here ๐
9.1. CBOR Simple Values Registry
IANA made the "Concise Binary Object Representation (CBOR) Simple Values" registry at [IANA.cbor-simple-values] โจ the initial values are shown in Table 4.
new entries in the range 0 to 19 are assigned by Standards Action [RFC8126]. it's suggested that IANA start allocating at the number 16, so the lower numbers stay free for contiguous blocks (if any) ๐ง
new entries in the range 32 to 255 are assigned by Specification Required ๐ฆ
9.2. CBOR Tags Registry
IANA made the "Concise Binary Object Representation (CBOR) Tags" registry at [IANA.cbor-tags] ๐ the tags that were defined in [RFC7049] are described in detail in Section 3.4, and other tags have been defined since then ๐
new entries in the range 0 to 23 ("1+0") are assigned by Standards Action โจ new entries in the ranges 24 to 255 ("1+1") and 256 to 32767 (lower half of "1+2") are assigned by Specification Required ๐ new entries in the range 32768 to 18446744073709551615 (upper half of "1+2", "1+4", and "1+8") are assigned by First Come First Served ๐ฅ
the template for registration requests is:
- Data item
- Semantics (short form)
and for First Come First Served requests, you also include:
- Point of contact
- Description of semantics (URL). this description is optional, since the URL can point to something like an Internet-Draft or a web page ๐
heads up bestie โ ๏ธ if you're applying in the First Come First Served range and you pick a tag number that can't be represented in 32 bits (so anything bigger than 4294967295), you should be aware that it could reduce interoperability with implementations that don't support 64-bit numbers ๐
9.3. Media Types Registry
the Internet media type [RFC6838] ("MIME type") for a single encoded CBOR data item is "application/cbor", as defined in the "Media Types" registry [IANA.media-types] ๐ here's the whole entry, verbatim ๐ซถ
Type name: application
Subtype name: cbor
Required parameters: n/a
Optional parameters: n/a
Encoding considerations: Binary
Security considerations: See Section 10 of RFC 8949.
Interoperability considerations: n/a
Published specification: RFC 8949
Applications that use this media type: Many
Additional information:
Magic number(s): n/a
File extension(s): .cbor
Macintosh file type code(s): n/a
Person & email address to contact for further information: IETF CBOR
Working Group (cbor@ietf.org) or IETF Applications and Real-Time
Area (art@ietf.org)
Intended usage: COMMON
Restrictions on usage: none
Author: IETF CBOR Working Group (cbor@ietf.org)
Change controller: The IESG (iesg@ietf.org)9.4. CoAP Content-Format Registry
the CoAP Content-Format for CBOR got registered in the "CoAP Content-Formats" subregistry, which lives inside the "Constrained RESTful Environments (CoRE) Parameters" registry [IANA.core-parameters] ๐ธ here's the entry:
Media Type: application/cbor
Encoding: -
ID: 60
Reference: RFC 89499.5. Structured Syntax Suffix Registry
the structured syntax suffix [RFC6838] for media types based on a single encoded CBOR data item is +cbor, which IANA registered in the "Structured Syntax Suffixes" registry [IANA.structured-suffix] โจ the full registration:
Name: Concise Binary Object Representation (CBOR)
+suffix: +cbor
References: RFC 8949
Encoding Considerations: CBOR is a binary format.
Interoperability Considerations: n/a
Fragment Identifier Considerations: The syntax and semantics of
fragment identifiers specified for +cbor SHOULD be as specified
for "application/cbor". (At publication of RFC 8949, there is no
fragment identification syntax defined for "application/cbor".)
The syntax and semantics for fragment identifiers for a specific
"xxx/yyy+cbor" SHOULD be processed as follows:
* For cases defined in +cbor, where the fragment identifier
resolves per the +cbor rules, then process as specified in
+cbor.
* For cases defined in +cbor, where the fragment identifier does
not resolve per the +cbor rules, then process as specified in
"xxx/yyy+cbor".
* For cases not defined in +cbor, then process as specified in
"xxx/yyy+cbor".
Security Considerations: See Section 10 of RFC 8949.
Contact: IETF CBOR Working Group (cbor@ietf.org) or IETF
Applications and Real-Time Area (art@ietf.org)
Author/Change Controller: IETFthe fragment identifier rules are the spicy part ๐ถ๏ธ the +cbor suffix says fragment identifiers SHOULD follow the "application/cbor" rules. then, for a specific "xxx/yyy+cbor" type, the processing goes like this: if the fragment resolves per the +cbor rules, you follow +cbor. if it doesn't resolve that way, you fall back to "xxx/yyy+cbor" ๐ซถ
10. Security Considerations โจ
okay bestie, this section is SO important and I'm not gonna lie, it's giving "parsers are where the drama starts" ๐๐ฅ. A network-facing app can have vulnerabilities in how it handles incoming data. Complex parsers are a super well-known source of these, and they can let someone remotely crash a node or even run arbitrary code on it ๐ญ. CBOR tries to cut down on those chances by keeping parser complexity low and by giving basically every encodable value a meaning where it can ๐ซถ.
Okay so, CBOR decoders are often the first thing touching unvalidated input, which means they need to be fully ready for hostile input ๐ก๏ธ. That input might be built to corrupt, overrun, or take over the system decoding it. A decoder has to assume all input could be hostile, even if a firewall already checked it, it came over a secure channel like TLS, it's encrypted or signed, or it came from some source that's presumed trusted. No exceptions, periodt ๐ .
Section 4.1 shows examples of interoperability limits when a constrained CBOR decoder gets input from an encoder that uses a non-preferred serialization. If one data item gets consumed by both a constrained decoder and a full decoder, that can open up security issues an attacker can exploit if they can inject or manipulate content ๐.
As the document covers throughout, lots of values can be "equivalent" in some situations and "not equivalent" in others ๐. For example, the number "one" can be written as an integer or as a bignum. A system interpreting CBOR input might accept either form, or might reject one or both. That accept-or-reject choice can have security implications for the program using the interpreted input ๐.
Hostile input can be crafted to overrun buffers, overflow or underflow integer math, or mess with decoding in other ways ๐ฅ. CBOR data items might claim lengths or sizes that are intentionally super huge or super tiny. Resource exhaustion attacks might try to lure a decoder into allocating really big data items (strings, arrays, maps, or even arbitrary precision numbers), or exhaust the stack by setting up deeply nested items ๐งจ. Decoders need proper resource management to mitigate all of this. (Items with very large sizes can also try to exploit integer overflow vulnerabilities, so keep an eye out ๐.)
By definition, a CBOR decoder ONLY accepts well-formed CBOR, and that's the first step to being robust ๐ฆ. Input that isn't well-formed gets no further processing from the point where the problem was detected. If possible, any data decoded up to that point should have no impact on the application using the decoder โจ.
Beyond checking well-formedness, a CBOR decoder might also do validity checks on the CBOR data. Or it can leave those checks to the application using it. Whichever choice it makes needs to be clearly documented in the decoder ๐. On top of CBOR-level validity, the application also needs to make sure the input lines up with the application protocol that's serialized in CBOR ๐.
The input check itself uses resources. That's usually linear in the size of the input, which means an attacker has to spend resources comparable to what the defender spends on validation ๐. However, an attacker might craft inputs that take a target decoder longer to process than it took the attacker to produce them ๐ฑ. Processing arbitrary-precision numbers may go beyond linear effort ๐ข. Also, some hash-table implementations decoders use to build in-memory maps can be attacked to spend quadratic effort, unless a secret key (see Section 7 of [SIPHASH_LNCS], also [SIPHASH_OPEN]) or some other mitigation is used ๐. Superlinear efforts like these can let an attacker exhaust resources at or before the input validator, so they need to be avoided in a CBOR decoder implementation ๐ซ. Note that tag number definitions and their implementations can add security considerations of this kind, and those should be discussed in the security considerations of the tag number definition ๐ท๏ธ.
CBOR encoders don't receive input directly from the network, so they aren't directly attackable in the same way decoders are ๐ธ. Still, encoders often have an API that takes input from another level of the implementation, and they can be attacked through that API ๐ธ๏ธ. The design and implementation of that API should assume the caller's behavior may be based on hostile input or on coding mistakes. It should check inputs for buffer overruns, overflow and underflow of integer arithmetic, and other errors aimed at disrupting the encoder ๐ ๏ธ.
Protocols should be defined so that multiple possible interpretations reliably collapse into a single interpretation ๐งฉ. For example, an attacker could use invalid input like duplicate keys in maps, or exploit different precision when processing numbers, to make one application base its decisions on a different interpretation than the one a second application uses ๐. To keep interpretation consistent, encoder and decoder implementations should provide a validity-checking mode of operation (Section 5.4) ๐. But note that a generic decoder can't know every requirement an application puts on its input data, so it doesn't relieve the application from doing its own input checking ๐ โโ๏ธ. Also, since the set of defined tag numbers keeps evolving, an application may use a tag number that a generic decoder doesn't yet support for validity checking. So generic decoders need to document which tag numbers they support and what validity checking they provide for those, as well as for basic CBOR (UTF-8 checking, duplicate map key checking) ๐.
Section 3.4.3 notes that using the non-preferred choice of a bignum representation instead of a basic integer to encode a number is NOT intended to carry application semantics ๐. But it CAN have those semantics if an application receiving CBOR data is using a decoder in the basic generic data model. That disparity causes a security issue if the two sets of semantics differ ๐จ. So applications using CBOR need to specify the data model they're using for each use of CBOR data ๐ฏ.
Converting CBOR data to other formats is super common ๐. In lots of cases CBOR has more expressive types than other formats, and that's especially true for the common conversion to JSON. Losing type information can cause security issues for systems processing the less-expressive data ๐ฌ.
Section 6.2 describes a possibly common usage scenario of converting between CBOR and JSON that could allow an attack if the attacker knows the application is doing that conversion ๐ต๏ธโโ๏ธ.
Security considerations for base16 and base64 from [RFC4648], and for UTF-8 from [RFC3629], are relevant to CBOR too ๐.
11. References ๐
11.1. Normative References
- [C]: International Organization for Standardization, "Information technology - Programming languages - C", Fourth Edition, ISO/IEC 9899:2018, June 2018, <https://www.iso.org/standard/74528.html>.
- [Cplusplus20]: International Organization for Standardization, "Programming languages - C++", Sixth Edition, ISO/IEC DIS 14882, ISO/IEC ISO/IEC JTC1 SC22 WG21 N 4860, March 2020, <https://isocpp.org/files/papers/N4860.pdf>.
- [IEEE754]: IEEE, "IEEE Standard for Floating-Point Arithmetic", IEEE Std 754-2019, DOI 10.1109/IEEESTD.2019.8766229, <https://ieeexplore.ieee.org/document/8766229>.
- [RFC2045]: Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, DOI 10.17487/RFC2045, November 1996, <https://www.rfc-editor.org/info/rfc2045>.
- [RFC2119]: Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.
- [RFC3339]: Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002, <https://www.rfc-editor.org/info/rfc3339>.
- [RFC3629]: Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, DOI 10.17487/RFC3629, November 2003, <https://www.rfc-editor.org/info/rfc3629>.
- [RFC3986]: Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", STD 66, RFC 3986, DOI 10.17487/RFC3986, January 2005, <https://www.rfc-editor.org/info/rfc3986>.
- [RFC4287]: Nottingham, M., Ed. and R. Sayre, Ed., "The Atom Syndication Format", RFC 4287, DOI 10.17487/RFC4287, December 2005, <https://www.rfc-editor.org/info/rfc4287>.
- [RFC4648]: Josefsson, S., "The Base16, Base32, and Base64 Data Encodings", RFC 4648, DOI 10.17487/RFC4648, October 2006, <https://www.rfc-editor.org/info/rfc4648>.
- [RFC8126]: Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, <https://www.rfc-editor.org/info/rfc8126>.
- [RFC8174]: Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.
- [TIME_T]: The Open Group, "The Open Group Base Specifications", Section 4.16, 'Seconds Since the Epoch', Issue 7, 2018 Edition, IEEE Std 1003.1, 2018, <https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_16>.
11.2. Informative References ๐โจ
okay so this is the extra-credit reading list, bestie ๐ง these aren't required to implement the spec, but they give you the backstory, the drafts, the errata, and the outside specs CBOR leans on ๐
[ASN.1] International Telecommunication Union, "Information
Technology - ASN.1 encoding rules: Specification of Basic
Encoding Rules (BER), Canonical Encoding Rules (CER) and
Distinguished Encoding Rules (DER)", ITU-T Recommendation
X.690, 2015,
<https://www.itu.int/rec/T-REC-X.690-201508-I/en>.
[BSON] Various, "BSON - Binary JSON", <http://bsonspec.org/>.
[CBOR-TAGS]
Bormann, C., "Notable CBOR Tags", Work in Progress,
Internet-Draft, draft-bormann-cbor-notable-tags-02, 25
June 2020, <https://tools.ietf.org/html/draft-bormann-
cbor-notable-tags-02>.
[ECMA262] Ecma International, "ECMAScript 2020 Language
Specification", Standard ECMA-262, 11th Edition, June
2020, <https://www.ecma-
international.org/publications/standards/Ecma-262.htm>.
[Err3764] RFC Errata, Erratum ID 3764, RFC 7049,
<https://www.rfc-editor.org/errata/eid3764>.
[Err3770] RFC Errata, Erratum ID 3770, RFC 7049,
<https://www.rfc-editor.org/errata/eid3770>.
[Err4294] RFC Errata, Erratum ID 4294, RFC 7049,
<https://www.rfc-editor.org/errata/eid4294>.
[Err4409] RFC Errata, Erratum ID 4409, RFC 7049,
<https://www.rfc-editor.org/errata/eid4409>.
[Err4963] RFC Errata, Erratum ID 4963, RFC 7049,
<https://www.rfc-editor.org/errata/eid4963>.
[Err4964] RFC Errata, Erratum ID 4964, RFC 7049,
<https://www.rfc-editor.org/errata/eid4964>.
[Err5434] RFC Errata, Erratum ID 5434, RFC 7049,
<https://www.rfc-editor.org/errata/eid5434>.
[Err5763] RFC Errata, Erratum ID 5763, RFC 7049,
<https://www.rfc-editor.org/errata/eid5763>.
[Err5917] RFC Errata, Erratum ID 5917, RFC 7049,
<https://www.rfc-editor.org/errata/eid5917>.
[IANA.cbor-simple-values]
IANA, "Concise Binary Object Representation (CBOR) Simple
Values",
<https://www.iana.org/assignments/cbor-simple-values>.
[IANA.cbor-tags]
IANA, "Concise Binary Object Representation (CBOR) Tags",
<https://www.iana.org/assignments/cbor-tags>.
[IANA.core-parameters]
IANA, "Constrained RESTful Environments (CoRE)
Parameters",
<https://www.iana.org/assignments/core-parameters>.
[IANA.media-types]
IANA, "Media Types",
<https://www.iana.org/assignments/media-types>.
[IANA.structured-suffix]
IANA, "Structured Syntax Suffixes",
<https://www.iana.org/assignments/media-type-structured-
suffix>.
[MessagePack]
Furuhashi, S., "MessagePack", <https://msgpack.org/>.
[PCRE] Hazel, P., "PCRE - Perl Compatible Regular Expressions",
<https://www.pcre.org/>.
[RFC0713] Haverty, J., "MSDTP-Message Services Data Transmission
Protocol", RFC 713, DOI 10.17487/RFC0713, April 1976,
<https://www.rfc-editor.org/info/rfc713>.
[RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type
Specifications and Registration Procedures", BCP 13,
RFC 6838, DOI 10.17487/RFC6838, January 2013,
<https://www.rfc-editor.org/info/rfc6838>.
[RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object
Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049,
October 2013, <https://www.rfc-editor.org/info/rfc7049>.
[RFC7228] Bormann, C., Ersue, M., and A. Keranen, "Terminology for
Constrained-Node Networks", RFC 7228,
DOI 10.17487/RFC7228, May 2014,
<https://www.rfc-editor.org/info/rfc7228>.
[RFC7493] Bray, T., Ed., "The I-JSON Message Format", RFC 7493,
DOI 10.17487/RFC7493, March 2015,
<https://www.rfc-editor.org/info/rfc7493>.
[RFC7991] Hoffman, P., "The "xml2rfc" Version 3 Vocabulary",
RFC 7991, DOI 10.17487/RFC7991, December 2016,
<https://www.rfc-editor.org/info/rfc7991>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
Interchange Format", STD 90, RFC 8259,
DOI 10.17487/RFC8259, December 2017,
<https://www.rfc-editor.org/info/rfc8259>.
[RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data
Definition Language (CDDL): A Notational Convention to
Express Concise Binary Object Representation (CBOR) and
JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610,
June 2019, <https://www.rfc-editor.org/info/rfc8610>.
[RFC8618] Dickinson, J., Hague, J., Dickinson, S., Manderson, T.,
and J. Bond, "Compacted-DNS (C-DNS): A Format for DNS
Packet Capture", RFC 8618, DOI 10.17487/RFC8618, September
2019, <https://www.rfc-editor.org/info/rfc8618>.
[RFC8742] Bormann, C., "Concise Binary Object Representation (CBOR)
Sequences", RFC 8742, DOI 10.17487/RFC8742, February 2020,
<https://www.rfc-editor.org/info/rfc8742>.
[RFC8746] Bormann, C., Ed., "Concise Binary Object Representation
(CBOR) Tags for Typed Arrays", RFC 8746,
DOI 10.17487/RFC8746, February 2020,
<https://www.rfc-editor.org/info/rfc8746>.
[SIPHASH_LNCS]
Aumasson, J. and D. Bernstein, "SipHash: A Fast Short-
Input PRF", Progress in Cryptology - INDOCRYPT 2012, pp.
489-508, DOI 10.1007/978-3-642-34931-7_28, 2012,
<https://doi.org/10.1007/978-3-642-34931-7_28>.
[SIPHASH_OPEN]
Aumasson, J. and D.J. Bernstein, "SipHash: a fast short-
input PRF", <https://www.aumasson.jp/siphash/siphash.pdf>.
[YAML] Ben-Kiki, O., Evans, C., and I.d. Net, "YAML Ain't Markup
Language (YAML[TM]) Version 1.2", 3rd Edition, October
2009, <https://www.yaml.org/spec/1.2/spec.html>.Appendix A. Examples of Encoded CBOR Data Items ๐
okay this appendix is the cheat sheet, and it's giving receipts ๐งพ๐ there's a table pairing diagnostic notation (the human-readable stuff, on the left) with the actual CBOR bytes in hexadecimal (on the right) ๐ฆ
ngl the weird-looking strings are the best part. "\u00fc" is just one way to write a UTF-8 string holding the single character U+00FC (LATIN SMALL LETTER U WITH DIAERESIS, "รผ") ๐ธ "\u6c34" is a UTF-8 string with one character, U+6C34 (CJK UNIFIED IDEOGRAPH-6C34, "ๆฐด"), which often means "water" ๐ง and "\ud800\udd51" is a UTF-8 string with one character, U+10151 (GREEK ACROPHONIC ATTIC FIFTY STATERS, "๐
") ๐
highkey, all of these single-character strings could also be written as native UTF-8 in diagnostic notation, just not if an ASCII-only specification is required ๐
and for bignums, the diagnostic notation shows the intended numeric value as a plain decimal number, like 18446744073709551616, instead of the tagged byte string, like 2(h'010000000000000000') โจ that's way more iconic and so much easier to read ๐ฅ
okay so the example table keeps going and it's giving pure main character energy ๐โจ every row is a little CBOR snack, and the encodings are the important part ๐
+==============================+====================================+
|Diagnostic | Encoded |
+==============================+====================================+
|0 | 0x00 |
+------------------------------+------------------------------------+
|1 | 0x01 |
+------------------------------+------------------------------------+
|10 | 0x0a |
+------------------------------+------------------------------------+
|23 | 0x17 |
+------------------------------+------------------------------------+
|24 | 0x1818 |
+------------------------------+------------------------------------+
|25 | 0x1819 |
+------------------------------+------------------------------------+
|100 | 0x1864 |
+------------------------------+------------------------------------+
|1000 | 0x1903e8 |
+------------------------------+------------------------------------+
|1000000 | 0x1a000f4240 |
+------------------------------+------------------------------------+
|1000000000000 | 0x1b000000e8d4a51000 |
+------------------------------+------------------------------------+
|18446744073709551615 | 0x1bffffffffffffffff |
+------------------------------+------------------------------------+
|18446744073709551616 | 0xc249010000000000000000 |
+------------------------------+------------------------------------+
|-18446744073709551616 | 0x3bffffffffffffffff |
+------------------------------+------------------------------------+
|-18446744073709551617 | 0xc349010000000000000000 |
+------------------------------+------------------------------------+
|-1 | 0x20 |
+------------------------------+------------------------------------+
|-10 | 0x29 |
+------------------------------+------------------------------------+
|-100 | 0x3863 |
+------------------------------+------------------------------------+
|-1000 | 0x3903e7 |
+------------------------------+------------------------------------+
|0.0 | 0xf90000 |
+------------------------------+------------------------------------+
|-0.0 | 0xf98000 |
+------------------------------+------------------------------------+
|1.0 | 0xf93c00 |
+------------------------------+------------------------------------+
|1.1 | 0xfb3ff199999999999a |
+------------------------------+------------------------------------+
|1.5 | 0xf93e00 |
+------------------------------+------------------------------------+
|65504.0 | 0xf97bff |
+------------------------------+------------------------------------+
|100000.0 | 0xfa47c35000 |
+------------------------------+------------------------------------+
|3.4028234663852886e+38 | 0xfa7f7fffff |
+------------------------------+------------------------------------+
|1.0e+300 | 0xfb7e37e43c8800759c |
+------------------------------+------------------------------------+
|5.960464477539063e-8 | 0xf90001 |
+------------------------------+------------------------------------+
|0.00006103515625 | 0xf90400 |
+------------------------------+------------------------------------+
|-4.0 | 0xf9c400 |
+------------------------------+------------------------------------+
|-4.1 | 0xfbc010666666666666 |
+------------------------------+------------------------------------+
|Infinity | 0xf97c00 |
+------------------------------+------------------------------------+
|NaN | 0xf97e00 |
+------------------------------+------------------------------------+
|-Infinity | 0xf9fc00 |
+------------------------------+------------------------------------+
|Infinity | 0xfa7f800000 |
+------------------------------+------------------------------------+
|NaN | 0xfa7fc00000 |
+------------------------------+------------------------------------+
|-Infinity | 0xfaff800000 |
+------------------------------+------------------------------------+
|Infinity | 0xfb7ff0000000000000 |
+------------------------------+------------------------------------+
|NaN | 0xfb7ff8000000000000 |
+------------------------------+------------------------------------+
|-Infinity | 0xfbfff0000000000000 |
+------------------------------+------------------------------------+
|false | 0xf4 |
+------------------------------+------------------------------------+
|true | 0xf5 |
+------------------------------+------------------------------------+
|null | 0xf6 |
+------------------------------+------------------------------------+
|undefined | 0xf7 |
+------------------------------+------------------------------------+
|simple(16) | 0xf0 |
+------------------------------+------------------------------------+
|simple(255) | 0xf8ff |
+------------------------------+------------------------------------+
|0("2013-03-21T20:04:00Z") | 0xc074323031332d30332d32315432303a |
| | 30343a30305a |
+------------------------------+------------------------------------+
|1(1363896240) | 0xc11a514b67b0 |
+------------------------------+------------------------------------+
|1(1363896240.5) | 0xc1fb41d452d9ec200000 |
+------------------------------+------------------------------------+
|23(h'01020304') | 0xd74401020304 |
+------------------------------+------------------------------------+
|24(h'6449455446') | 0xd818456449455446 |
+------------------------------+------------------------------------+
|32("http://www.example.com") | 0xd82076687474703a2f2f7777772e6578 |
| | 616d706c652e636f6d |
+------------------------------+------------------------------------+
|h'' | 0x40 |
+------------------------------+------------------------------------+
|h'01020304' | 0x4401020304 |
+------------------------------+------------------------------------+
|"" | 0x60 |
+------------------------------+------------------------------------+
|"a" | 0x6161 |
+------------------------------+------------------------------------+
|"IETF" | 0x6449455446 |
+------------------------------+------------------------------------+
|"\"\\" | 0x62225c |
+------------------------------+------------------------------------+
|"\u00fc" | 0x62c3bc |
+------------------------------+------------------------------------+
|"\u6c34" | 0x63e6b0b4 |
+------------------------------+------------------------------------+
|"\ud800\udd51" | 0x64f0908591 |
+------------------------------+------------------------------------+
|[] | 0x80 |
+------------------------------+------------------------------------+
|[1, 2, 3] | 0x83010203 |
+------------------------------+------------------------------------+
|[1, [2, 3], [4, 5]] | 0x8301820203820405 |
+------------------------------+------------------------------------+
|[1, 2, 3, 4, 5, 6, 7, 8, 9, | 0x98190102030405060708090a0b0c0d0e |
|10, 11, 12, 13, 14, 15, 16, | 0f101112131415161718181819 |
|17, 18, 19, 20, 21, 22, 23, | |
|24, 25] | |
+------------------------------+------------------------------------+
|{} | 0xa0 |
+------------------------------+------------------------------------+
|{1: 2, 3: 4} | 0xa201020304 |
+------------------------------+------------------------------------+
|{"a": 1, "b": [2, 3]} | 0xa26161016162820203 |
+------------------------------+------------------------------------+
|["a", {"b": "c"}] | 0x826161a161626163 |
+------------------------------+------------------------------------+
|{"a": "A", "b": "B", "c": "C",| 0xa5616161416162614261636143616461 |
|"d": "D", "e": "E"} | 4461656145 |
+------------------------------+------------------------------------+
|(_ h'0102', h'030405') | 0x5f42010243030405ff |
+------------------------------+------------------------------------+
|(_ "strea", "ming") | 0x7f657374726561646d696e67ff |
+------------------------------+------------------------------------+
|[_ ] | 0x9fff |
+------------------------------+------------------------------------+
|[_ 1, [2, 3], [_ 4, 5]] | 0x9f018202039f0405ffff |
+------------------------------+------------------------------------+
|[_ 1, [2, 3], [4, 5]] | 0x9f01820203820405ff |
+------------------------------+------------------------------------+
|[1, [2, 3], [_ 4, 5]] | 0x83018202039f0405ff |
+------------------------------+------------------------------------+
|[1, [_ 2, 3], [4, 5]] | 0x83019f0203ff820405 |
+------------------------------+------------------------------------+
|[_ 1, 2, 3, 4, 5, 6, 7, 8, 9, | 0x9f0102030405060708090a0b0c0d0e0f | |10, 11, 12, 13, 14, 15, 16, | 101112131415161718181819ff |
|17, 18, 19, 20, 21, 22, 23, | |
|24, 25] | |
+------------------------------+------------------------------------+
|{_ "a": 1, "b": [_ 2, 3]} | 0xbf61610161629f0203ffff |
+------------------------------+------------------------------------+
|["a", {_ "b": "c"}] | 0x826161bf61626163ff |
+------------------------------+------------------------------------+
|{_ "Fun": true, "Amt": -2} | 0xbf6346756ef563416d7421ff |
+------------------------------+------------------------------------+Table 6: Examples of Encoded CBOR Data Items ๐
Appendix B. Jump Table for Initial Byte โจ
Okay bestie, quick heads up before we dive in ๐: this jump table is not the full list ๐. It skips the initial bytes that are reserved for future extension, and it only shows a selection of the ones that can be used for optional features. Also, all unsigned integers in here are in network byte order ๐ธ
+============+================================================+
| Byte | Structure/Semantics |
+============+================================================+
| 0x00..0x17 | unsigned integer 0x00..0x17 (0..23) |
+------------+------------------------------------------------+
| 0x18 | unsigned integer (one-byte uint8_t follows) |
+------------+------------------------------------------------+
| 0x19 | unsigned integer (two-byte uint16_t follows) |
+------------+------------------------------------------------+
| 0x1a | unsigned integer (four-byte uint32_t follows) |
+------------+------------------------------------------------+
| 0x1b | unsigned integer (eight-byte uint64_t follows) |
+------------+------------------------------------------------+
| 0x20..0x37 | negative integer -1-0x00..-1-0x17 (-1..-24) |
+------------+------------------------------------------------+
| 0x38 | negative integer -1-n (one-byte uint8_t for n |
| | follows) |
+------------+------------------------------------------------+
| 0x39 | negative integer -1-n (two-byte uint16_t for n |
| | follows) |
+------------+------------------------------------------------+
| 0x3a | negative integer -1-n (four-byte uint32_t for |
| | n follows) |
+------------+------------------------------------------------+
| 0x3b | negative integer -1-n (eight-byte uint64_t for |
| | n follows) |
+------------+------------------------------------------------+
| 0x40..0x57 | byte string (0x00..0x17 bytes follow) |
+------------+------------------------------------------------+
| 0x58 | byte string (one-byte uint8_t for n, and then |
| | n bytes follow) |
+------------+------------------------------------------------+
| 0x59 | byte string (two-byte uint16_t for n, and then |
| | n bytes follow) |
+------------+------------------------------------------------+
| 0x5a | byte string (four-byte uint32_t for n, and |
| | then n bytes follow) |
+------------+------------------------------------------------+
| 0x5b | byte string (eight-byte uint64_t for n, and |
| | then n bytes follow) |
+------------+------------------------------------------------+
| 0x5f | byte string, byte strings follow, terminated |
| | by "break" |
+------------+------------------------------------------------+
| 0x60..0x77 | UTF-8 string (0x00..0x17 bytes follow) |
+------------+------------------------------------------------+
| 0x78 | UTF-8 string (one-byte uint8_t for n, and then |
| | n bytes follow) |
+------------+------------------------------------------------+
| 0x79 | UTF-8 string (two-byte uint16_t for n, and |
| | then n bytes follow) |
+------------+------------------------------------------------+
| 0x7a | UTF-8 string (four-byte uint32_t for n, and |
| | then n bytes follow) |
+------------+------------------------------------------------+
| 0x7b | UTF-8 string (eight-byte uint64_t for n, and |
| | then n bytes follow) |
+------------+------------------------------------------------+
| 0x7f | UTF-8 string, UTF-8 strings follow, terminated |
| | by "break" |
+------------+------------------------------------------------+
| 0x80..0x97 | array (0x00..0x17 data items follow) |
+------------+------------------------------------------------+
| 0x98 | array (one-byte uint8_t for n, and then n data |
| | items follow) |
+------------+------------------------------------------------+
| 0x99 | array (two-byte uint16_t for n, and then n |
| | data items follow) |
+------------+------------------------------------------------+
| 0x9a | array (four-byte uint32_t for n, and then n |
| | data items follow) |
+------------+------------------------------------------------+
| 0x9b | array (eight-byte uint64_t for n, and then n |
| | data items follow) |
+------------+------------------------------------------------+
| 0x9f | array, data items follow, terminated by |
| | "break" |
+------------+------------------------------------------------+
| 0xa0..0xb7 | map (0x00..0x17 pairs of data items follow) |
+------------+------------------------------------------------+
| 0xb8 | map (one-byte uint8_t for n, and then n pairs |
| | of data items follow) |
+------------+------------------------------------------------+
| 0xb9 | map (two-byte uint16_t for n, and then n pairs |
| | of data items follow) |
+------------+------------------------------------------------+
| 0xba | map (four-byte uint32_t for n, and then n |
| | pairs of data items follow) |
+------------+------------------------------------------------+
| 0xbb | map (eight-byte uint64_t for n, and then n |
| | pairs of data items follow) |
+------------+------------------------------------------------+
| 0xbf | map, pairs of data items follow, terminated by |
| | "break" |
+------------+------------------------------------------------+
| 0xc0 | text-based date/time (data item follows; see |
| | Section 3.4.1) |
+------------+------------------------------------------------+
| 0xc1 | epoch-based date/time (data item follows; see |
| | Section 3.4.2) |
+------------+------------------------------------------------+
| 0xc2 | unsigned bignum (data item "byte string" |
| | follows) |
+------------+------------------------------------------------+
| 0xc3 | negative bignum (data item "byte string" |
| | follows) |
+------------+------------------------------------------------+
| 0xc4 | decimal Fraction (data item "array" follows; |
| | see Section 3.4.4) |
+------------+------------------------------------------------+
| 0xc5 | bigfloat (data item "array" follows; see |
| | Section 3.4.4) |
+------------+------------------------------------------------+
| 0xc6..0xd4 | (tag) |
+------------+------------------------------------------------+
| 0xd5..0xd7 | expected conversion (data item follows; see |
| | Section 3.4.5.2) |
+------------+------------------------------------------------+
| 0xd8..0xdb | (more tags; 1/2/4/8 bytes of tag number and |
| | then a data item follow) |
+------------+------------------------------------------------+
| 0xe0..0xf3 | (simple value) |
+------------+------------------------------------------------+
| 0xf4 | false |
+------------+------------------------------------------------+
| 0xf5 | true |
+------------+------------------------------------------------+
| 0xf6 | null |
+------------+------------------------------------------------+
| 0xf7 | undefined |
+------------+------------------------------------------------+
| 0xf8 | (simple value, one byte follows) |
+------------+------------------------------------------------+
| 0xf9 | half-precision float (two-byte IEEE 754) |
+------------+------------------------------------------------+
| 0xfa | single-precision float (four-byte IEEE 754) |
+------------+------------------------------------------------+
| 0xfb | double-precision float (eight-byte IEEE 754) |
+------------+------------------------------------------------+
| 0xff | "break" stop code |
+------------+------------------------------------------------+Table 7: Jump Table for Initial Byte ๐
Appendix C. Pseudocode โจ
okay so the well-formedness of a CBOR item can be checked with the pseudocode in Figure 1 ๐ the data is well-formed if and only if:
- the pseudocode does not "fail";
- after execution of the pseudocode, no bytes are left in the input (except in streaming applications).
the pseudocode has a few prerequisites, and they're honestly simple ๐
- take(n) reads n bytes from the input data and returns them as a byte string. if n bytes are no longer available, take(n) fails.
- uint() converts a byte string into an unsigned integer by interpreting the byte string in network byte order.
- arithmetic works as in C.
- all variables are unsigned integers of sufficient range.
note that "well_formed" returns the major type for well-formed definite-length items, but 99 for an indefinite-length item (or -1 for a "break" stop code, only if "breakable" is set) ๐ธ this is used in "well_formed_indefinite" to make sure indefinite-length strings only contain definite-length strings as chunks ๐ซถ
well_formed(breakable = false) {
// process initial bytes
ib = uint(take(1));
mt = ib >> 5;
val = ai = ib & 0x1f;
switch (ai) {
case 24: val = uint(take(1)); break;
case 25: val = uint(take(2)); break;
case 26: val = uint(take(4)); break;
case 27: val = uint(take(8)); break;
case 28: case 29: case 30: fail();
case 31:
return well_formed_indefinite(mt, breakable);
}
// process content
switch (mt) {
// case 0, 1, 7 do not have content; just use val
case 2: case 3: take(val); break; // bytes/UTF-8
case 4: for (i = 0; i < val; i++) well_formed(); break;
case 5: for (i = 0; i < val*2; i++) well_formed(); break;
case 6: well_formed(); break; // 1 embedded data item
case 7: if (ai == 24 && val < 32) fail(); // bad simple
}
return mt; // definite-length data item
}
well_formed_indefinite(mt, breakable) {
switch (mt) {
case 2: case 3:
while ((it = well_formed(true)) != -1)
if (it != mt) // need definite-length chunk
fail(); // of same type
break;
case 4: while (well_formed(true) != -1); break;
case 5: while (well_formed(true) != -1) well_formed(); break;
case 7:
if (breakable)
return -1; // signal break out
else fail(); // no enclosing indefinite
default: fail(); // wrong mt
}
return 99; // indefinite-length data item
}Figure 1: Pseudocode for Well-Formedness Check
the rest of a complete CBOR decoder is mostly about presenting the decoded data to the application in an appropriate form ๐
major types 0 and 1 are designed so that they can be encoded in C from a signed integer without an if-then-else for positive/negative ๐ง it works because (-1-n), the transformation for major type 1, is the same as ~n (bitwise complement) in C unsigned arithmetic. ~n can then be written as (-1)^n for the negative case, while 0^n leaves n unchanged for nonnegative. the sign of a number can be turned into -1 for negative and 0 for nonnegative (0 or positive) by arithmetic-shifting the number by one bit less than its bit length (for example, by 63 for 64-bit numbers) ๐ญ
void encode_sint(int64_t n) {
uint64t ui = n >> 63; // extend sign to whole length
unsigned mt = ui & 0x20; // extract (shifted) major type
ui ^= n; // complement negatives
if (ui < 24)
*p++ = mt + ui;
else if (ui < 256) {
*p++ = mt + 24;
*p++ = ui;
} else
...Figure 2: Pseudocode for Encoding a Signed Integer
see Section 1.2 for some specific assumptions about the profile of the C language used in these pieces of code ๐
Appendix D. Half-Precision ๐ฆ
half-precision floating-point numbers were only added to IEEE 754 in 2008 [IEEE754], so today's programming platforms often still have limited support for them ๐ฅบ the good news is it's very easy to include at least decoding support even without it. an example of a small decoder for half-precision floating-point numbers in C is shown in Figure 3. a similar program for Python is in Figure 4; this code assumes that the 2-byte value has already been decoded as an (unsigned short) integer in network byte order (as would be done by the pseudocode in Appendix C) ๐ฅ
#include <math.h>
double decode_half(unsigned char *halfp) {
unsigned half = (halfp[0] << 8) + halfp[1];
unsigned exp = (half >> 10) & 0x1f;
unsigned mant = half & 0x3ff;
double val;
if (exp == 0) val = ldexp(mant, -24);
else if (exp != 31) val = ldexp(mant + 1024, exp - 25);
else val = mant == 0 ? INFINITY : NAN;
return half & 0x8000 ? -val : val;
}Figure 3: C Code for a Half-Precision Decoder
import struct
from math import ldexp
def decode_single(single):
return struct.unpack("!f", struct.pack("!I", single))[0]
def decode_half(half):
valu = (half & 0x7fff) << 13 | (half & 0x8000) << 16
if ((half & 0x7c00) != 0x7c00):
return ldexp(decode_single(valu), 112)
return decode_single(valu | 0x7f800000)Figure 4: Python Code for a Half-Precision Decoder
Appendix E. Comparison of Other Binary Formats to CBOR's Design Objectives ๐
the CBOR proposal follows a history of binary formats that is as long as the history of computers themselves ๐ฐ๏ธ different formats had different objectives. most of the time, those objectives were never stated, although you can sometimes guess them from the context where the format was first used. some formats were meant to be universally usable, but history has proven that no binary format meets the needs of all protocols and applications ๐
CBOR is different from many of these because it started with a set of objectives and tried to meet just those โจ this section compares a few of the dozens of formats against CBOR's objectives so you can decide whether CBOR or a different format is right for your protocol or application ๐
the discussion here is not a criticism of any format. to the best of our knowledge, no format before CBOR was meant to cover CBOR's objectives in the priority we've assigned them ๐ซถ here's a quick recap of the objectives from Section 1.1:
- unambiguous encoding of most common data formats from Internet standards
- code compactness for encoder or decoder
- no schema description needed
- reasonably compact serialization
- applicability to constrained and unconstrained applications
- good JSON conversion
- extensibility
a discussion of CBOR and other formats with respect to a different set of design objectives is provided in Section 5 and Appendix C of [RFC8618] ๐
E.1. ASN.1 DER, BER, and PER ๐
[ASN.1] has many serializations. in the IETF, DER and BER are the most common. the serialized output isn't particularly compact for many items, and the code needed to decode numeric items can get complex on a constrained device ๐
few (if any) IETF protocols have adopted one of the several variants of Packed Encoding Rules (PER). there could be many reasons, but one that gets mentioned a lot is that PER uses the schema even to parse the surface structure of the data item, which requires significant tool support ๐ฉ there are also different versions of the ASN.1 schema language in use, which has hampered adoption too.
E.2. MessagePack ๐ง
[MessagePack] is a concise, widely implemented, counted binary serialization format. it's similar to CBOR in many ways, although somewhat less regular ๐ while its data model can represent JSON data, MessagePack has also been used in lots of remote procedure call (RPC) applications and for long-term storage of data ๐
MessagePack has been essentially stable since it was first published around 2011, and it has not yet had a transition ๐ฅน its evolution is held back by the need to keep complete backwards compatibility with existing stored data, while only a few bytecodes are still available for extension. over the years the MessagePack user community has repeatedly asked to separate binary and text strings in the encoding. that recently led to an extension proposal that would leave MessagePack's "raw" data ambiguous between its binary and text uses. the extension mechanism for MessagePack is still unclear ๐คทโโ๏ธ
E.3. BSON ๐ธ
[BSON] is a data format developed for storing JSON-like maps (JSON objects) in the MongoDB database. its major distinguishing feature is the capability for in-place update, which prevents a compact representation ๐ฌ BSON uses a counted representation except for map keys, which are null-byte terminated. BSON can be used to represent JSON-like objects on the wire, but its specification is dominated by the requirements of the database application and has gotten somewhat baroque ๐ซ how BSON extensions will be implemented remains unclear.
E.4. MSDTP: RFC 713 ๐ฆ
Message Services Data Transmission (MSDTP) is a very early example of a compact message format. it's described in [RFC0713], written in 1976 ๐ฐ๏ธ it's included here for its historical value, not because it was ever widely used ๐ฅฐ
E.5. Conciseness on the Wire ๐
CBOR's design objective of code compactness for encoders and decoders is a higher priority than its objective of conciseness on the wire, but honestly, lots of people focus on wire size ๐ Table 8 shows some encoding examples for the simple nested array [1, [2, 3]]. where the encoding supports some form of indefinite-length encoding, [_ 1, [2, 3]] (indefinite length on the outer array) is shown too โจ
+=============+============================+================+
| Format | [1, [2, 3]] | [_ 1, [2, 3]] |
+=============+============================+================+
| RFC 713 | c2 05 81 c2 02 82 83 | |
+-------------+----------------------------+----------------+
| ASN.1 BER | 30 0b 02 01 01 30 06 02 01 | 30 80 02 01 01 |
| | 02 02 01 03 | 30 06 02 01 02 |
| | | 02 01 03 00 00 |
+-------------+----------------------------+----------------+
| MessagePack | 92 01 92 02 03 | |
+-------------+----------------------------+----------------+
| BSON | 22 00 00 00 10 30 00 01 00 | |
| | 00 00 04 31 00 13 00 00 00 | |
| | 10 30 00 02 00 00 00 10 31 | |
| | 00 03 00 00 00 00 00 | |
+-------------+----------------------------+----------------+
| CBOR | 82 01 82 02 03 | 9f 01 82 02 03 |
| | | ff |
+-------------+----------------------------+----------------+Table 8: Examples for Different Levels of Conciseness
Appendix F. Well-Formedness Errors and Examples โจ
okay so there are three basic kinds of well-formedness errors that can pop up when you're decoding a CBOR data item ๐ง
- Too much data: there are input bytes left over that nobody consumed ๐ฆ. This only counts as an error if the application assumed the input would be exactly one data item. If the app is using CBOR's self-delimiting nature to allow extra data after the item, like in CBOR sequences [RFC8742], the decoder can just say which part of the input it didn't use. Totally fine, no drama ๐
- Too little data: the input would need more bytes tacked onto the end to make a complete CBOR data item ๐ฅบ. This might mean the input got cut off, and it's also a super common result when someone tries to decode random data as CBOR ๐. For some apps it isn't actually an error, though, because they might not have all the data yet and can wait for more bytes โณ. Some of those apps have a cap on how much extra data can show up, and in that case the decoder can say the item can't be finished within that limit ๐ โโ๏ธ
- Syntax error: the input doesn't match the CBOR encoding rules, and adding or removing stuff at the end won't fix it ๐ซ
In Appendix C, the first kind is handled in the first paragraph and bullet list (the one requiring "no bytes are left") ๐. The second kind is handled in the second paragraph and bullet list (failing "if n bytes are no longer available") ๐. The third kind gets identified in the pseudocode by specific calls to fail(), in this order:
- a reserved value is used for additional information (28, 29, 30)
- major type 7, additional information 24, value < 32 (incorrect)
- incorrect substructure of indefinite-length byte string or text string (may only contain definite-length strings of the same major type)
- "break" stop code (major type 7, additional information 31) occurs in a value position of a map or except at a position directly in an indefinite-length item where also another enclosed data item could occur
- additional information 31 used with major type 0, 1, or 6
F.1. Examples of CBOR Data Items That Are Not Well-Formed
this part shows a few examples of CBOR data items that are not well-formed ๐ Each example is a sequence of bytes shown in hexadecimal, and when there are several examples in a list they're separated by commas ๐
examples for well-formedness error kind 1 (too much data) are super easy to make, you just add data to a well-formed encoded CBOR data item โ
the same goes for error kind 2 (too little data), you can make those by truncating a well-formed encoded CBOR data item โ๏ธ In test suites it can be really useful to test with incomplete data items that would need a huge amount of extra bytes to finish, like starting the encoding of a string of a very large size ๐งช
a premature end of the input can happen in a head, or inside the enclosed data ๐งบ That enclosed data might be bare strings or enclosed data items that are either counted or should have been ended by a "break" stop code ๐
End of input in a head:
18, 19, 1a, 1b, 19 01, 1a 01 02, 1b 01 02 03 04 05 06 07, 38, 58, 78, 98, 9a 01 ff 00, b8, d8, f8, f9 00, fa 00 00, fb 00 00 00Definite-length strings with short data:
41, 61, 5a ff ff ff ff 00, 5b ff ff ff ff ff ff ff ff 01 02 03, 7a ff ff ff ff 00, 7b 7f ff ff ff ff ff ff ff 01 02 03Definite-length maps and arrays not closed with enough items:
81, 81 81 81 81 81 81 81 81 81, 82 00, a1, a2 01 02, a1 00, a2 00 00 00Tag number not followed by tag content:
c0Indefinite-length strings not closed by a "break" stop code:
5f 41 00, 7f 61 00Indefinite-length maps and arrays not closed by a "break" stop code:
9f, 9f 01 02, bf, bf 01 02 01 02, 81 9f, 9f 80 00, 9f 9f 9f 9f 9f ff ff ff ff, 9f 81 9f 81 9f 9f ff ff ffNext up, a few examples for the five subkinds of well-formedness error kind 3 (syntax error) ๐งฉ
Subkind 1:
Reserved additional information values:
1c, 1d, 1e, 3c, 3d, 3e, 5c, 5d, 5e, 7c, 7d, 7e, 9c, 9d, 9e, bc, bd, be, dc, dd, de, fc, fd, fe,Subkind 2:
Reserved two-byte encodings of simple values:
f8 00, f8 01, f8 18, f8 1fSubkind 3:
Indefinite-length string chunks not of the correct type:
5f 00 ff, 5f 21 ff, 5f 61 00 ff, 5f 80 ff, 5f a0 ff, 5f c0 00 ff, 5f e0 ff, 7f 41 00 ffIndefinite-length string chunks not definite length:
5f 5f 41 00 ff ff, 7f 7f 61 00 ff ffSubkind 4:
Break occurring on its own outside of an indefinite-length item:
ffBreak occurring in a definite-length array or map or a tag:
81 ff, 82 00 ff, a1 ff, a1 ff 00, a1 00 ff, a2 00 00 ff, 9f 81 ff, 9f 82 9f 81 9f 9f ff ff ff ffBreak in an indefinite-length map that would lead to an odd number of items (break in a value position):
bf 00 ff, bf 00 00 00 ffSubkind 5:
Major type 0, 1, 6 with additional information 31:
1f, 3f, dfAppendix G. Changes from RFC 7049 ๐ธ
as talked about in the introduction, this document formally obsoletes RFC 7049 while keeping full compatibility with the interchange format from RFC 7049 ๐ค It brings editorial improvements, added detail, and fixed errata โจ But it does not create a new version of the format, so no new format drama ๐
G.1. Errata Processing and Clerical Changes
the two verified errata on RFC 7049, [Err3764] and [Err3770], were about two encoding examples in the text, and both got corrected ๐ง (Section 3.4.3: "29" -> "49", Section 5.5: "0b000_11101" -> "0b000_11001").
RFC 7049 also had an example using the numeric value 24 for a simple value [Err5917]. That's not well-formed, so the example got removed ๐๏ธ
errata report 5763 [Err5763] pointed out an error in the wording of the definition of tags ๐ท๏ธ That got sorted out during a rewrite of Section 3.4.
errata report 5434 [Err5434] said the Universal Binary JSON (UBJSON) example in Appendix E no longer matched the version of UBJSON current when the report was submitted ๐ญ Turns out the UBJSON specification had completely changed since 2013, so that example was removed.
other errata reports [Err4409] [Err4963] [Err4964] complained that the map key sorting rules for canonical encoding were onerous ๐ฉ That led to a reconsideration of the canonical encoding suggestions, which were replaced by the deterministic encoding suggestions (described below) ๐
an editorial suggestion in errata report 4294 [Err4294] also got implemented: improved symmetry by adding "Second value" to a comment on the last example in Section 3.2.2 ๐
other clerical changes include:
- the use of new xml2rfc functionality [RFC7991];
- more explanation of the notation used;
- the update of references, e.g., from RFC 4627 to [RFC8259], from CNN-TERMS to [RFC7228], and from the 5.1 edition to the 11th edition of [ECMA262]; the addition of a reference to [IEEE754] and importation of required definitions; the addition of references to [C] and [Cplusplus20]; and the addition of a reference to [RFC8618] that further illustrates the discussion in Appendix E;
- in the discussion of diagnostic notation (Section 8), the "Extended Diagnostic Notation" (EDN) defined in [RFC8610] is now mentioned, the gap in representing NaN payloads is now highlighted, and an explanation of representing indefinite-length strings with no chunks has been added (Section 8.1);
- the addition of this appendix.
G.2. Changes in IANA Considerations
the IANA considerations got a general update (clerical changes, e.g., now pointing to the CBOR Working Group as the author of the specification) ๐ References to the respective IANA registries were added to the informative references too.
in the "Concise Binary Object Representation (CBOR) Tags" registry [IANA.cbor-tags], tags in the space from 256 to 32767 (lower half of "1+2") are no longer assigned by First Come First Served. This range is now Specification Required ๐
G.3. Changes in Suggestions and Other Informational Components ๐
Okay bestie, so while the working group was busy with the errata reports, they also got to learn from nearly seven years of CBOR being used in a bunch of different apps ๐ฆ. That led to a whole round of editorial glow-ups ๐ . They added tables for illustration, and they leaned into some topics while quieting down others โจ.
A big addition is Section 2 ๐ง , which covers the CBOR data model and the small variations that show up when CBOR gets processed. The new terms for those variations (basic generic, extended generic, and specific) make other parts of the document way more concise. They also help make it clear what implementations and the extensibility features of the format are expected to do ๐.
RFC 7049 came from the JSON ecosystem, so it picked up the JSON number system, which was inherited from JavaScript at the time ๐. JSON doesn't have separate integers and floating-point values (and in the format, the floats are decimal). CBOR gives you binary representations of numbers, and those really do differ between integers and floats ๐ข. Implementation and real-world use showed that the line between these two number domains needed to be drawn more clearly in the document, so the language that suggested an integer could seamlessly stand in for a floating-point value got removed ๐. There's also a new suggestion for converting JSON to CBOR when handling these types, based on I-JSON [RFC7493], and a specific rounding mechanism is recommended for it ๐.
For a single value in the data model, CBOR often gives you several encoding options ๐. So there's a new Section 4 that introduces the term "preferred serialization" (Section 4.1) and defines it for different kinds of data items ๐. Building on that, the section explains how a CBOR-based protocol can define "deterministic encoding" (Section 4.2). This avoids the terms "canonical" and "canonicalization" from RFC 7049 ๐. The suggestion of "Core Deterministic Encoding Requirements" (Section 4.2.1) gives protocols a way to get generic support for their own encoding requirements ๐ง. This document also makes deterministic encoding way easier to implement by simplifying the map ordering from RFC 7049 into a plain lexicographic ordering of encoded keys ๐ซถ. The older approach isn't gone, though. It's kept as an alternative, now called "length-first map key ordering" (Section 4.2.3) ๐ธ.
The terminology for well-formed and valid data got sharpened and is used way more strictly ๐ฅ. Terms like "syntax error", "decoding error", and "strict mode" were dropped everywhere outside of examples ๐. There's also now an explicit third layer of requirements that an application has on its input data, beyond CBOR-level validity. Well-formed means it can be processed at all, valid means it's been checked by a validity-checking generic decoder, and expected input means it's been checked by the application itself. Together they form a hierarchy of layers of acceptability ๐ช.
The handling of non-well-formed simple values got clarified, both in text and in pseudocode ๐งฉ. Appendix F was added to talk about well-formedness errors and give examples of them ๐ญ. The pseudocode was also updated to be more portable, and some portability considerations were added ๐.
Validity got sharpened in two areas โญ. Map validity, which covers handling duplicate keys, is clearer now, and the document explains the domain of applicability for certain implementation choices ๐. While the terminology for tags, tag numbers, and tag content got streamlined, there's new discussion on tag validity, and the restrictions on tag content got clarified, both in general and specifically for tag 1 ๐ .
Section 3.4 got an implementation note, plus a note for future tag definitions, about defining tags whose semantics depend on serialization order ๐ฆ.
Tag 35 is not defined by this document. The registration based on the definition in RFC 7049 stays right where it is ๐.
Section 3 also introduced terminology for "argument" and "head", which makes the rest of the discussion much simpler ๐.
Last but not least, the security considerations (Section 10) got mostly rewritten and significantly expanded ๐จ. In multiple other places, the document is now way more explicit that a decoder cannot simply condone well-formedness errors. No shortcuts, bestie ๐ โโ๏ธ๐.
Acknowledgements โจ
CBOR was inspired by MessagePack ๐ซถ. MessagePack was developed and promoted by Sadayuki Furuhashi ("frsyuki"). This reference to MessagePack is solely for attribution. CBOR is not meant to be a version of MessagePack, or a replacement for it, since it has different design goals and requirements ๐.
The need for functionality beyond the original MessagePack specification became obvious to lots of people at about the same time, around 2012 ๐. BinaryPack is a minor derivation of MessagePack that Eric Zhang developed for the binaryjs project. Tim Caswell made a similar but different extension for his msgpack-js and msgpack-js-browser projects. Plenty of people also contributed to the discussion about extending MessagePack to separate text string representation from byte string representation ๐ง.
The encoding of the additional information in CBOR was inspired by the encoding of length information that Klaus Hartke designed for CoAP ๐ธ.
This document also incorporates suggestions from many people, notably Dan Frost, James Manger, Jeffrey Yasskin, Joe Hildebrand, Keith Moore, Laurence Lundblade, Matthew Lepinski, Michael Richardson, Nico Williams, Peter Occil, Phillip Hallam-Baker, Ray Polk, Stuart Cheshire, Tim Bray, Tony Finch, Tony Hansen, and Yaron Sheffer ๐๐. Benjamin Kaduk provided an extensive review during IESG processing. รric Vyncke, Erik Kline, Robert Wilton, and Roman Danyliw provided further IESG comments, which included an IoT directorate review by Eve Schooler ๐.
Authors' Addresses ๐
Carsten Bormann
Universitรคt Bremen TZI
Postfach 330440
D-28359 Bremen
Germany
Phone: +49-421-218-63921
Email: cabo@tzi.org
Paul Hoffman
ICANN
Email: paul.hoffman@icann.org๐ฆ where this page comes from
- ietf: rfc8949 2020-12
- ๐ girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms