RFC 8746

Concise Binary Object Representation (CBOR) Tags for Typed Arrays

📅 published February 2020, from ietf

RFC 8746: CBOR Tags for Typed Arrays 💖

Okay bestie, let's get into it. RFC 8746 is titled Concise Binary Object Representation (CBOR) Tags for Typed Arrays 🎀. It was written by C. Bormann, Ed. from Universität Bremen TZI and published in February 2020. It's on the Standards Track (so it's a real, official Internet standard 👑), and its ISSN is 2070-1721. The title block doesn't list anything it obsoletes or updates, so it's a standalone document ✨.

Abstract ✨

So CBOR is the Concise Binary Object Representation, defined in RFC 7049 🦋. Its design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. Literally built different 💅.

This doc uses that extensibility to define a bunch of CBOR tags for typed arrays of numeric data 🔥. It also adds more tags for multi-dimensional and homogeneous arrays 🌈. It's meant to be the reference doc for the IANA registration of the CBOR tags it defines 📋.

Status of This Memo 🌸

This is an Internet Standards Track document. It's a product of the Internet Engineering Task Force (IETF), which means it represents the consensus of the IETF community 🫶. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check out Section 2 of RFC 7841 💜.

Info about the current status of this doc, any errata, and how to give feedback is at https://www.rfc-editor.org/info/rfc8746 🧚.

Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved 🦋.

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication. Please actually read these, because they describe your rights and restrictions for this document 🙈. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License ⚠️.

Table of Contents 🎀

1.  Introduction
  1.1.  Terminology
2.  Typed Arrays
  2.1.  Types of Numbers
3.  Additional Array Tags
  3.1.  Multi-dimensional Array
    3.1.1.  Row-Major Order
    3.1.2.  Column-Major Order
  3.2.  Homogeneous Array
4.  Discussion
5.  CDDL Typenames
6.  IANA Considerations
7.  Security Considerations
8.  References
  8.1.  Normative References
  8.2.  Informative References
Acknowledgements
Contributors
Author's Address

1. Introduction ✨

Okay so CBOR (that's [RFC7049]) lets you interchange structured data without needing a pre-agreed schema 🌸. [RFC7049] defines a basic set of data types, plus a tagging mechanism that lets you extend the set of data types through an IANA registry 🧚.

Recently, a simple form of typed arrays of numeric data has been getting a LOT of love from the Web graphics community [TypedArray] and from the JavaScript spec (see Section 22.2 (https://www.ecma-international.org/ecma-262/10.0/index.html#sec-typedarray-objects) of [ECMA-ES10]) 💜. Corresponding implementations [ArrayBuffer] are showing up too 🔥.

Typed arrays can carry serious amounts of data 📦, so people want to send them over CBOR without having to slowly convert every single number in the array 😭. Encoding a type for each element of an array would also waste space, so this helps with that too 🍓.

This document defines a bunch of interrelated CBOR tags that cover these typed arrays, plus extra tags for multi-dimensional and homogeneous arrays 🦋. It's the reference doc for the IANA registration of the tags it defines 📋.

Now, an app that generates CBOR with these tags has a lot of freedom 🌈. It can pick variants (like endianness, embedded type signed vs. unsigned, and number of bits per element) 🫧. It can also decide whether to use a tag from this spec at all, instead of plain, more basic CBOR 🙃. Unlike representation variants of single CBOR numbers, there is no representation that could be identified as "preferred" 🤷‍♀️. So if a CBOR-based protocol wants deterministic encoding while using these tags, the protocol itself has to define which encoding variants get used in each individual case 📜.

1.1. Terminology 💅

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this doc are to be interpreted as described in BCP 14 [RFC2119] [RFC8174], but only when they appear in all capitals, exactly as shown here 🎀. Lowercase versions are just regular words, bestie 🙈.

"Byte" means the now-customary synonym for "octet" 🫶. When bit arithmetic gets explained, this doc uses the familiar notation from the programming language C [C] (including C++14's 0bnnn binary literals [CPlusPlus]) 🧠. The only exception is the operator "**", which means exponentiation 🔢.

"Array" is used in a general sense here unless it gets more specific 🌸. A "classical CBOR array" is an array represented with CBOR major type 4 📚. A "homogeneous array" is an array where all the elements are the same type 💖. (The term doesn't care whether that type is a representation type or an application data model type 🫧.)

"Big endian" and "little endian" mean most significant byte first (MSB first) and least significant byte first (LSB first) representations of integers, respectively ✨.

2. Typed Arrays 🦋

Typed arrays are homogeneous arrays of numbers, and every number is encoded in one single form of binary representation 🌈. You concatenate those representations and encode the result as a single CBOR byte string (major type 2) 🧵. Then it gets wrapped in a single tag that says what type and encoding all the numbers in the byte string have 🎁.

2.1. Types of Numbers 🔥

There are three classes of numbers we care about: unsigned integers (uint), signed integers (two's complement, sint), and IEEE 754 binary floating point numbers (which are always signed) 👑. Each of these classes has multiple representation lengths that are actually in use 📏:

              +-----------+--------+--------+-----------+
              | Length ll | uint   | sint   | float     |
              +===========+========+========+===========+
              | 0         | uint8  | sint8  | binary16  |
              +-----------+--------+--------+-----------+
              | 1         | uint16 | sint16 | binary32  |
              +-----------+--------+--------+-----------+
              | 2         | uint32 | sint32 | binary64  |
              +-----------+--------+--------+-----------+
              | 3         | uint64 | sint64 | binary128 |
              +-----------+--------+--------+-----------+

                         Table 1: Length Values

Here, sintN means a signed integer of exactly N bits (like sint16), and uintN means an unsigned integer of exactly N bits (like uint32) 🧮. The name binaryN means the number form of the same name defined in IEEE 754 [IEEE754] 🤓.

Here's the big reason for all this: one goal of these tags is to directly ship the ArrayBuffers underlying the Typed Arrays without re-encoding them 🚀. Those buffers might be big-endian (network byte order) or little-endian, so we need tags for both variants 💕.

All in all, that's 24 variants 🌟. In the tag, we have to express the choice between integer and float, the signedness (for integers), the endianness, and one of the four length values 🎀.

To keep implementation simple, a range of tags is allocated so you can pull all of this info straight from the bits of the tag: tag values from 64 to 87 🧚‍♀️.

The value is split into bit fields: 0b010, f, s, e, and ll, as detailed in Table 2 🦋.

  +-------+-------------------------------------------------------+
  | Field | Use                                                   |
  +=======+=======================================================+
  | 0b010 | the constant bits 0, 1, 0                             |
  +-------+-------------------------------------------------------+
  | f     | 0 for integer, 1 for float                            |
  +-------+-------------------------------------------------------+
  | s     | 0 for float or unsigned integer, 1 for signed integer |
  +-------+-------------------------------------------------------+
  | e     | 0 for big endian, 1 for little endian                 |
  +-------+-------------------------------------------------------+
  | ll    | A number for the length (Table 1).                    |
  +-------+-------------------------------------------------------+

         Table 2: Bit Fields in the Low 8 Bits of the Tag

To get the number of bytes in each array element, you compute "2**(f + ll)" (or "1 << (f + ll)" in a typical programming language) 🧠. (Notice that 0f and ll are the two least significant bits, respectively, of each 4-bit nibble in the byte 🙈.)

In the CBOR representation, the total number of elements isn't written out explicitly 🤷‍♀️. It's implied from the length of the byte string and the length of each representation 🧩. You can compute it from the length, in bytes, of the byte string holding the array's representation by inverting the previous formula: "bytelength >> (f + ll)" 🔄.

For the uint8/sint8 values, the endianness is redundant 🙃. So only the tag for the big-endian variant is used and assigned as such 💖. The tag that would signify the little-endian variant of sint8 MUST NOT be used, and its tag number is marked as reserved 🚫.

As a special case, the tag that would signify the little-endian variant of uint8 is instead assigned to signify that the numbers in the array use clamped conversion from integers 🎀. This is described in more detail in Section 7.1.11 (http://www.ecma-international.org/ecma-262/6.0/#sec-touint8clamp) of the ES10 JavaScript specification ("ToUint8Clamp") [ECMA-ES10] 📖. The assumption is that a program-internal representation of this array, after decoding, would be marked this way for further processing, which gives you "roundtripping" of JavaScript-typed arrays through CBOR 🔁.

IEEE 754 binary floating numbers are always signed 👑. So for the float variants (where "f" == 1), there's no need to tell signed and unsigned apart, and the "s" bit is always zero 🌸. The tag numbers where "s" would be one (tag values 88 to 95) stay free for other specifications to use 🌈.

3. Additional Array Tags 🧜‍♀️

This spec defines three additional array tags 💅. The Multi-dimensional Array tags can be combined with classical CBOR arrays as well as with Typed Arrays, so you can build multi-dimensional arrays with constant numbers of elements in the sub-arrays 📐. The Homogeneous Array tag lets an app flag a classical CBOR array as a homogeneous array, even when a Typed Array doesn't apply 🫶.

3.1. Multi-dimensional Array 🌈

okay so a multi-dimensional array is basically a tagged array that holds two one-dimensional arrays 💅 The first one lists the dimensions (outer ones first, inner ones last) and the second one holds the actual contents. And if that second array is itself tagged as a Typed Array, the element type of the whole thing is the same as that Typed Array's type 🫶

This spec defines two tags: one for elements in row-major order and one for column-major order [RowColMajor] 🦋

3.1.1. Row-Major Order

Tag: 40 ✨

Data Item: Array (major type 4) of two arrays: one array (major type 4) of dimensions, which are unsigned integers distinct from zero; and one array (any one of a CBOR array of major type 4, a Typed Array, or a Homogeneous Array) of elements.

the data in the second array is a run of consecutive values where the last dimension is the contiguous one (row-major order) 🍓

Figure 1 shows a 2D array declared in C, then how it looks in CBOR using both the multi-dimensional array tag and the typed array tag. this is the glow-up 🤩

uint16_t a[2][3] = {
  {2, 4, 8},   /* row 0 */
  {4, 16, 256},
};

<Tag 40> # multi-dimensional array tag
   82       # array(2)
     82      # array(2)
       02     # unsigned(2) 1st Dimension
       03     # unsigned(3) 2nd Dimension
     <Tag 65> # uint16 array
       4c     # byte string(12)
         0002 # unsigned(2)
         0004 # unsigned(4)
         0008 # unsigned(8)
         0004 # unsigned(4)
         0010 # unsigned(16)
         0100 # unsigned(256)

Figure 1: Multi-dimensional Array in C and CBOR

Figure 2 is the same 2D array but with the multi-dimensional array tag paired with a basic CBOR array instead. with the small numbers picked for this example, it just happens to be shorter 🙈

<Tag 40> # multi-dimensional array tag
   82       # array(2)
     82      # array(2)
       02     # unsigned(2) 1st Dimension
       03     # unsigned(3) 2nd Dimension
     86     # array(6)
       02      # unsigned(2)
       04      # unsigned(4)
       08      # unsigned(8)
       04      # unsigned(4)
       10      # unsigned(16)
       19 0100 # unsigned(256)

Figure 2: Multi-dimensional Array Using Basic CBOR Array

3.1.2. Column-Major Order

the multi-dimensional arrays in the previous sub-subsection (3.1.1) are in "row major" order, and that's the preferred order for this spec 👑 the column-major version is the analogous one, and it lives under tag 1040 💜 Figure 3 shows it.

Tag: 1040 🧚

Data Item: The same as tag 40, except the data in the second array consists of consecutive values where the first dimension is considered contiguous (column-major order).

<Tag 1040> # multi-dimensional array tag, column-major order
   82       # array(2)
     82      # array(2)
       02     # unsigned(2) 1st Dimension
       03     # unsigned(3) 2nd Dimension
     86     # array(6)
       02      # unsigned(2)
       04      # unsigned(4)
       04      # unsigned(4)
       10      # unsigned(16)
       08      # unsigned(8)
       19 0100 # unsigned(256)

Figure 3: Multi-dimensional Array Using Basic CBOR Array, Column-Major Order

3.2. Homogeneous Array 🦄

Tag: 41 💖

Data Item: Array (major type 4)

this tag marks a classic CBOR array (a one-dimensional array) as a homogeneous array, meaning every element is the same application model data type 🎀 The element type gets decided by the type of the first element.

this is useful for application data models that give homogeneous arrays special meaning. also, in some cases, strongly typed languages can build native homogeneous arrays of a specific type instead of plain ordered lists while decoding ✨ which CBOR data items count as the same application type is up to the application.

Figure 4 is an example of a homogeneous array of booleans in C++ [CPlusPlus] and CBOR 🌸

bool boolArray[2] = { true, false };

<Tag 41>  # Homogeneous Array Tag
   82           #array(2)
      F5        # true
      F4        # false

Figure 4: Homogeneous Array in C++ and CBOR

Figure 5 levels it up with a more complex structure 🧠

typedef struct {
  bool active;
  int value;
} foo;
foo myArray[2] = { {true, 3}, {true, -4} };

<Tag 41>
    82  # array(2)
       82  #  array(2)
             F5  # true
             03  # 3
       82 # array(2)
             F5  # true
             23  # -4

Figure 5: Homogeneous Array in C++ and CBOR

4. Discussion 💬

support for both little- and big-endian representation might feel a bit off for CBOR, since CBOR is otherwise fully big endian 🙃 but it fits the whole point of typed arrays, which is not having to convert every single element ✨

this spec uses a pretty big chunk of the single-byte tag space 👀 that's justified by how widely typed arrays get used in data interchange 🌈

a column-major variant of the multi-dimensional array might seem unnecessary to some and handy to others 🦋 it's cheap to define the extra tag, so it's there when you actually need it. putting it in a different number space makes the preference for row-major obvious 👑

applying the Homogeneous Array tag to a Typed Array would usually be redundant, so this spec doesn't provide it 💅

5. CDDL Typenames 🎀

For use with CDDL [RFC8610], the typenames in Figure 6 are recommended 🫶

ta-uint8 = #6.64(bstr)
ta-uint16be = #6.65(bstr)
ta-uint32be = #6.66(bstr)
ta-uint64be = #6.67(bstr)
ta-uint8-clamped = #6.68(bstr)
ta-uint16le = #6.69(bstr)
ta-uint32le = #6.70(bstr)
ta-uint64le = #6.71(bstr)
ta-sint8 = #6.72(bstr)
ta-sint16be = #6.73(bstr)
ta-sint32be = #6.74(bstr)
ta-sint64be = #6.75(bstr)
; reserved: #6.76(bstr)
ta-sint16le = #6.77(bstr)
ta-sint32le = #6.78(bstr)
ta-sint64le = #6.79(bstr)
ta-float16be = #6.80(bstr)
ta-float32be = #6.81(bstr)
ta-float64be = #6.82(bstr)
ta-float128be = #6.83(bstr)
ta-float16le = #6.84(bstr)
ta-float32le = #6.85(bstr)
ta-float64le = #6.86(bstr)
ta-float128le = #6.87(bstr)
homogeneous<array> = #6.41(array)
multi-dim<dim, array> = #6.40([dim, array])
multi-dim-column-major<dim, array> = #6.1040([dim, array])

Figure 6: Recommended Typenames for CDDL

6. IANA Considerations 🎀

Okay so IANA has allocated the tags in Table 3, and this document is the specification reference for them 💅 (the reserved value is saved for a future revision of typed array tags, so no touching 🙅‍♀️).

The allocations came from the "specification required" space (24..255), with one exception: 1040 came from the "first come first served" space (256..) ✨

    +------+----------------------+----------------------------+
    |  Tag | Data Item            | Semantics                  |
    +======+======================+============================+
    |   40 | array of two arrays* | Multi-dimensional Array,   |
    |      |                      | row-major order            |
    +------+----------------------+----------------------------+
    |   41 | array                | Homogeneous Array          |
    +------+----------------------+----------------------------+
    |   64 | byte string          | uint8 Typed Array          |
    +------+----------------------+----------------------------+
    |   65 | byte string          | uint16, big endian, Typed  |
    |      |                      | Array                      |
    +------+----------------------+----------------------------+
    |   66 | byte string          | uint32, big endian, Typed  |
    |      |                      | Array                      |
    +------+----------------------+----------------------------+
    |   67 | byte string          | uint64, big endian, Typed  |
    |      |                      | Array                      |
    +------+----------------------+----------------------------+
    |   68 | byte string          | uint8 Typed Array, clamped |
    |      |                      | arithmetic                 |
    +------+----------------------+----------------------------+
    |   69 | byte string          | uint16, little endian,     |
    |      |                      | Typed Array                |
    +------+----------------------+----------------------------+
    |   70 | byte string          | uint32, little endian,     |
    |      |                      | Typed Array                |
    +------+----------------------+----------------------------+
    |   71 | byte string          | uint64, little endian,     |
    |      |                      | Typed Array                |
    +------+----------------------+----------------------------+
    |   72 | byte string          | sint8 Typed Array          |
    +------+----------------------+----------------------------+
    |   73 | byte string          | sint16, big endian, Typed  |
    |      |                      | Array                      |
    +------+----------------------+----------------------------+
    |   74 | byte string          | sint32, big endian, Typed  |
    |      |                      | Array                      |
    +------+----------------------+----------------------------+
    |   75 | byte string          | sint64, big endian, Typed  |
    |      |                      | Array                      |
    +------+----------------------+----------------------------+
    |   76 | byte string          | (reserved)                 |
    +------+----------------------+----------------------------+
    |   77 | byte string          | sint16, little endian,     |
    |      |                      | Typed Array                |
    +------+----------------------+----------------------------+
    |   78 | byte string          | sint32, little endian,     |
    |      |                      | Typed Array                |
    +------+----------------------+----------------------------+
    |   79 | byte string          | sint64, little endian,     |
    |      |                      | Typed Array                |
    +------+----------------------+----------------------------+
    |   80 | byte string          | IEEE 754 binary16, big     |
    |      |                      | endian, Typed Array        |
    +------+----------------------+----------------------------+
    |   81 | byte string          | IEEE 754 binary32, big     |
    |      |                      | endian, Typed Array        |
    +------+----------------------+----------------------------+
    |   82 | byte string          | IEEE 754 binary64, big     |
    |      |                      | endian, Typed Array        |
    +------+----------------------+----------------------------+
    |   83 | byte string          | IEEE 754 binary128, big    |
    |      |                      | endian, Typed Array        |
    +------+----------------------+----------------------------+
    |   84 | byte string          | IEEE 754 binary16, little  |
    |      |                      | endian, Typed Array        |
    +------+----------------------+----------------------------+
    |   85 | byte string          | IEEE 754 binary32, little  |
    |      |                      | endian, Typed Array        |
    +------+----------------------+----------------------------+
    |   86 | byte string          | IEEE 754 binary64, little  |
    |      |                      | endian, Typed Array        |
    +------+----------------------+----------------------------+
    |   87 | byte string          | IEEE 754 binary128, little |
    |      |                      | endian, Typed Array        |
    +------+----------------------+----------------------------+
    | 1040 | array of two arrays* | Multi-dimensional Array,   |
    |      |                      | column-major order         |
    +------+----------------------+----------------------------+

                         Table 3: Values for Tags

   *40 or 1040 data item: The second element of the outer array in the
   data item is a native CBOR array (major type 4) or Typed Array (one
   of tag 64..87)

7. Security Considerations 🔐

Big safety talk, bestie 🫶 The security considerations of [RFC7049] apply, and you need to give special attention to the second paragraph of Section 8 of [RFC7049] 👀

The tag for homogeneous arrays makes a promise about its tagged data item, and a maliciously constructed CBOR input can then choose to ignore that promise 😭 So the decoder has to make sure it is not driven into an undefined state by array elements that do not fulfill the promise, and that it still continues to fulfill its API contract in this case too. No skipping this part, periodt 💀

As with all formats used for data interchange, an attacker may have control over the shape of the data delivered as input to the application. That means the application needs to validate that shape before it builds further processing on top of it 🚨 Typed arrays add one extra twist: an attacker might swap a Uint8ClampedArray in where the application expects a Uint8Array, or the other way around. That can lead to very different (and unexpected) processing semantics for the in-memory data structures that get built 🙈 So applications that could be affected will need to be careful about making this distinction in their input validation. Main character energy is validating everything 👑

8. References 📚

8.1. Normative References

8.2. Informative References

Acknowledgements 🙌

Jim Schaad gave helpful comments and reminded everyone that column-major order is still in use 🌸 Jeffrey Yaskin helped improve the definition of homogeneous arrays 💜 IANA helped correct an error in a previous draft version 🧚 Francesca Palombini acted as Shepherd, and Alexey Melnikov was the responsible Area Director 👑 Elwyn Davies, as Gen-ART reviewer, and IESG members Martin Vigoureux, Adam Roach, Roman Danyliw, and Benjamin Kaduk helped find further improvements to the text. Thanks also to the other reviewers, you're all iconic ✨

Contributors 🦋

The initial draft version of this specification was written by Johnathan Roatch <roatch@gmail.com>. Many thanks for getting this ball rolling 🎀

Glenn Engel suggested the tags for multi-dimensional arrays and homogeneous arrays 🍓

Author's Address 💌

Carsten Bormann (editor)
Universität Bremen TZI
Postfach 330440
D-28359 Bremen
Germany

Phone: +49-421-218-63921
Email: cabo@tzi.org
📦 where this page comes from