RFC 7373

Textual Representation of IP Flow Information Export (IPFIX) Abstract Data Types

📅 published September 2014, from ietf

RFC 7373: Textual Representation of IP Flow Information Export (IPFIX) Abstract Data Types 🎀

Okay bestie, here's the header rundown ✨

  • Number and category: RFC 7373, Standards Track 💅
  • Date: September 2014
  • Author: B. Trammell, ETH Zurich
  • ISSN: 2070-1721
  • Obsoletes and updates: the header doesn't list any, so no drama there 🫶

Abstract ✨

So this doc gives you UTF-8 ways to write down IPFIX abstract data types (ADTs). The goal is that Information Elements can be used with protocols that run on text, so everyone's on the same page, no cap 💖

Status of This Memo 📜

This is an Internet Standards Track document 🔥

It's a product of the Internet Engineering Task Force (IETF), so it counts as the community's consensus. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want the deets on Internet Standards, check Section 2 of RFC 5741 🦋

For the current status, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc7373 🌸

Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. Obviously 🙈

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stood on the publication date. Please read those carefully, because they spell out what you can and can't do with this doc 😭

If you pull Code Components out of this document, they have to include the Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions. They come with no warranty, as described in the Simplified BSD License 💅

Table of Contents 🗺️

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
3.  Identifying Information Elements  . . . . . . . . . . . . . .   3
4.  Data Type Encodings . . . . . . . . . . . . . . . . . . . . .   3
  4.1.  octetArray  . . . . . . . . . . . . . . . . . . . . . . .   4
  4.2.  unsigned8, unsigned16, unsigned32, and unsigned64 . . . .   4
  4.3.  signed8, signed16, signed32, and signed64 . . . . . . . .   5
  4.4.  float32 and float64 . . . . . . . . . . . . . . . . . . .   6
  4.5.  boolean . . . . . . . . . . . . . . . . . . . . . . . . .   7
  4.6.  macAddress  . . . . . . . . . . . . . . . . . . . . . . .   7
  4.7.  string  . . . . . . . . . . . . . . . . . . . . . . . . .   7
  4.8.  The dateTime ADTs . . . . . . . . . . . . . . . . . . . .   8
  4.9.  ipv4Address . . . . . . . . . . . . . . . . . . . . . . .   8
  4.10. ipv6Address . . . . . . . . . . . . . . . . . . . . . . .   9
  4.11. basicList, subTemplateList, and subTemplateMultiList  . .   9
5.  Security Considerations . . . . . . . . . . . . . . . . . . .   9
6.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  10
  6.1.  Normative References  . . . . . . . . . . . . . . . . . .  10
  6.2.  Informative References  . . . . . . . . . . . . . . . . .  11
Appendix A.  Example  . . . . . . . . . . . . . . . . . . . . . .  13
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . .  14
Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .

1. Introduction ✨

The IPFIX Information Model [RFC7012] gives you a set of abstract data types (ADTs) for the IANA "IPFIX Information Elements" registry [IANA-IPFIX]. That registry is packed with Information Elements that describe network entities and traffic, plus ADTs for those elements 🧚

The IPFIX Protocol Specification [RFC7011] then defines a big-endian binary encoding for those ADTs, which works for the IPFIX protocol 💖

But present and future operations and management protocols and apps might use textual encodings and generic framing and structure, like JSON [RFC7159] or XML. A canonical textual encoding for the IPFIX ADTs would let those apps use the same Information Elements and interoperate with IPFIX apps at the Information Element definition level. Iconic behaviour 😭

Quick heads up: templating and other data description mechanisms are app specific, so they're out of scope here. This doc only defines how Information Elements are identified and how their values are represented 🙈

In most cases where a textual representation is used, the tradeoff is clear: you pick human readability and manipulability over compactness. That assumption is baked into the standard representations of IPFIX ADTs 🌈

2. Terminology 💅

Capitalized terms defined in the IPFIX Protocol Specification [RFC7011] and the IPFIX Information Model [RFC7012] mean what those docs say. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. This doc also defines its own terminology, as follows 🫶

Enclosing Context

A textual representation of Information Element values is applied to use the IPFIX Information Model within some existing textual format (e.g., XML [W3C-XML] and JSON [RFC7159]). That outer format is called the Enclosing Context in this document. Enclosing Contexts define escaping and quoting rules for represented values. ✨

3. Identifying Information Elements 🦋

The "IPFIX Information Elements" registry [IANA-IPFIX] lists a bunch of Information Elements, each with a numeric Information Element identifier and a human-friendly name. Those identifiers are made for the IPFIX protocol, so they don't mean much once you're working with textual representations 🙈

So apps that use textual representations identify Information Elements by name instead. Appendix A has examples of this in action 🍓

4. Data Type Encodings 🌸

Each subsection here defines a textual encoding for one of the abstract data types in [RFC7012]. This section uses ABNF, including the Core Rules in Appendix B of [RFC5234], to describe how textual representations of IPFIX ADTs are formatted 💜

If future documents update [RFC7012] to add new ADTs to the IPFIX Information Model, and those ADTs are generally useful, this document will also need an update so it can define textual encodings for them. Stay tuned 👀

4.1. octetArray 🍓

If the Enclosing Context defines a representation for binary objects, that representation SHOULD be used.

Otherwise, since the whole point of textual representations is human readability over compactness, Information Elements with the octetArray data type are written as pairs of hexadecimal digits, one pair per byte. The pairs go in the order the bytes would appear on the wire if the octetArray were encoded directly in IPFIX per [RFC7011]. Whitespace can go between any pair of digits to help humans read the string, but it's not required 🫶

In ABNF:

hex-octet = 2HEXDIG

octetarray = hex-octet *([WSP] hex-octet)

4.2. unsigned8, unsigned16, unsigned32, and unsigned64 🔥

If the Enclosing Context defines a representation for unsigned integers, that representation SHOULD be used.

There's a special case where the unsigned Information Element has identifier semantics and points to a set of codepoints, either in an external registry, a sub-registry, or straight in the Information Element's description. In that case, the name or short description of the codepoint as a string MAY be used to make it more readable ✨

Otherwise, values of unsigned integer type can be written as unprefixed base-10 (decimal) strings, base-16 (hexadecimal) strings prefixed with 0x, or base-2 (binary) strings prefixed with 0b. In ABNF:

unsigned = 1*DIGIT / "0x" 1*HEXDIG / "0b" 1*BIT

Leading zeroes are allowed in any representation, and they don't mean base-8 (octal). Binary representation is meant for Information Elements with flag semantics, but you can use it anywhere 💖

The encoded value MUST be in range for the corresponding abstract data type or Information Element. Values that are out of range get clipped to the implicit range of the Information Element as defined by the abstract data type, or to the explicit range of the Information Element if one is defined. Min and max values for the abstract data types are in Table 1 below 👑

          +------------+---------+----------------------+
          |       type | minimum |              maximum |
          +------------+---------+----------------------+
          |  unsigned8 |       0 |                  255 |
          | unsigned16 |       0 |                65535 |
          | unsigned32 |       0 |           4294967295 |
          | unsigned64 |       0 | 18446744073709551615 |
          +------------+---------+----------------------+

   Table 1: Ranges for Unsigned Abstract Data Types (in Decimal)

4.3. signed8, signed16, signed32, and signed64 🦋

If the Enclosing Context defines a representation for signed integers, that representation SHOULD be used.

Otherwise, values of signed integer types are written as optionally prefixed base-10 (decimal) strings. In ABNF:

sign = "+" / "-"

signed = [sign] 1*DIGIT

If the sign is left out, it's assumed to be positive. Leading zeroes are allowed and don't mean base-8 (octal) encoding. The -0 representation is explicitly allowed and equals zero 🧚

The encoded value MUST be in range for the corresponding abstract data type or Information Element. Values that are out of range are to be interpreted as clipped to the implicit range of the Information Element as defined by the abstract data type, or to the explicit range of the Information Element if one is defined. Min and max values for the abstract data types are in Table 2 below 💅

        +----------+----------------------+----------------------+
        |     type |              minimum |              maximum |
        +----------+----------------------+----------------------+
        |  signed8 |                 -128 |                 +127 |
        | signed16 |               -32768 |               +32767 |
        | signed32 |          -2147483648 |          +2147483647 |
        | signed64 | -9223372036854775808 | +9223372036854775807 |
        +----------+----------------------+----------------------+

        Table 2: Ranges for Signed Abstract Data Types (in Decimal)

To be continued in part 2 🎀💖

4.4. float32 and float64 ✨

Okay so if the Enclosing Context already has a way to write floating-point numbers, bestie, you SHOULD use that one. 💖

Otherwise, float32 and float64 values get written as floating-point decimal numbers 🔢. You can put an optional sign in front, and an optional base-10 exponent at the end, like in [IEEE.754.2008]. The special strings "NaN", "+inf", and "-inf" mean "not a number", "positive infinity", and "negative infinity" 🌈.

In ABNF:

sign = "+" / "-"

exponent = "e" [sign] 1*3DIGIT

right-decimal = "." 1*DIGIT

mantissa = 1*DIGIT [right-decimal]

num = [sign] mantissa [exponent]

naninf = "NaN" / (sign "inf")

float = num / naninf

The value you get is ( mantissa * 10 ^ exponent ) 🧮. No sign? It's positive. No exponent? It's zero. Leading zeroes are allowed in the mantissa and/or the exponent 🫶. Values MUST be within range for single- or double-precision numbers as defined in [IEEE.754.2008], and finite values that fall outside that range get clamped to be within it, so it's giving tidy 💅. You only ever need three digits for an exponent in this encoding, because of those ranges, and you're not allowed to use more.

Here's the tea ☕: this representation is built for humans to read, so writers MAY trade away some precision to make a value nicer to look at. The catch is the reader might not get back the exact bit pattern. That means decoders MUST NOT assume the represented values can be compared for exact equality 🙈.

4.5. boolean 🦋

If the Enclosing Context has its own representation for booleans, you SHOULD use that. 💜

If not, a true boolean is the literal string "true" and a false boolean is the literal string "false". In ABNF:

boolean-true = "true"

boolean-false = "false"

boolean = boolean-true / boolean-false

4.6. macAddress 🍓

Media Access Control (MAC) addresses are written as IEEE 802 MAC-48 addresses. That means hexadecimal bytes, most significant byte first, separated by colons 🧚. In ABNF:

hex-octet = 2HEXDIG

macaddress = hex-octet 5( ":" hex-octet )

4.7. string 📝

String Information Elements are just Unicode strings. When they show up in Data Sets in IPFIX Messages [RFC7011], they're encoded as UTF-8 ✨. Because of that, they get written directly, using the Unicode encoding rules and the quoting and escaping rules of the Enclosing Context.

If the Enclosing Context can't natively handle Unicode characters, you MUST use its escaping facility for any characters it can't represent 🙅‍♀️. Also, strings that contain characters reserved in the Enclosing Context (think control characters, markup characters, and quotes) MUST be escaped or quoted following the Enclosing Context's rules 🎀.

It's assumed the Enclosing Context has enough limits on Unicode use to stop unsafe use of nonprinting and control characters 🛡️. Mixed-direction strings are a no-go right now, because there's no agreed way to process and display them safely, so they should be avoided with this encoding 🙃. Also, this document adds no extra rules for normalizing Unicode strings, so be careful when you compare strings. Comparing raw bytes isn't enough to tell whether two strings are equivalent 😭. For more on the weird results and risks you can run into when comparing Unicode strings, check out [RFC6885] and [PRECIS].

4.8. The dateTime ADTs ⏰

Timestamp abstract data types are generally written like in [RFC3339], with two big differences 💅. First, every IPFIX timestamp is in UTC, so the text versions are in UTC too. That means time zone offsets are neither required nor supported. Second, there are four timestamp abstract data types, and they're sorted by how precise they can be. dateTimeSeconds leaves out fractional seconds, dateTimeMilliseconds has milliseconds, and so on up the ladder 🌸.

In ABNF, taken from [RFC3339] and modified as follows:

date-fullyear   = 4DIGIT
date-month      = 2DIGIT  ; 01-12
date-mday       = 2DIGIT  ; 01-28, 01-29, 01-30, 01-31
time-hour       = 2DIGIT  ; 00-23
time-minute     = 2DIGIT  ; 00-59
time-second     = 2DIGIT  ; 00-58, 00-59, 00-60
time-msec       = "." 3DIGIT
time-usec       = "." 6DIGIT
time-nsec       = "." 9DIGIT
full-date       = date-fullyear "-" date-month "-" date-mday
integer-time    = time-hour ":" time-minute ":" time-second

datetimeseconds      = full-date "T" integer-time
datetimemilliseconds = full-date "T" integer-time "." time-msec
datetimemicroseconds = full-date "T" integer-time "." time-usec
datetimenanoseconds  = full-date "T" integer-time "." time-nsec

4.9. ipv4Address 🌐

IP version 4 addresses are written in dotted-quad format, most significant byte first, the same way they'd look in a Uniform Resource Identifier [RFC3986] 🦋. The ABNF for an IPv4 address comes from [RFC3986], and it's copied down below:

dec-octet   = DIGIT                 ; 0-9
            / %x31-39 DIGIT         ; 10-99
            / "1" 2DIGIT            ; 100-199
            / "2" %x30-34 DIGIT     ; 200-249
            / "25" %x30-35          ; 250-255

ipv4address = dec-octet 3( "." dec-octet )

4.10. ipv6Address 💜

IP version 6 addresses are written as in Section 2.2 of [RFC4291], as updated by Section 4 of [RFC5952] 🌈. The ABNF for an IPv6 address comes from [RFC3986], and it's reproduced below. It uses the ipv4address production from the previous section ✨.

ls32        = ( h16 ":" h16 ) / ipv4address
            ; least significant 32 bits of address
h16         = 1*4HEXDIG
            ; 16 bits of address represented in hexadecimal
            ; zeroes to be suppressed as in RFC 5952

ipv6address =                            6( h16 ":" ) ls32
            /                       "::" 5( h16 ":" ) ls32
            / [               h16 ] "::" 4( h16 ":" ) ls32
            / [     h16 ":"   h16 ] "::" 3( h16 ":" ) ls32
            / [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
            / [ *3( h16 ":" ) h16 ] "::"    h16 ":"   ls32
            / [ *4( h16 ":" ) h16 ] "::"              ls32
            / [ *5( h16 ":" ) h16 ] "::"              h16
            / [ *6( h16 ":" ) h16 ] "::"

4.11. basicList, subTemplateList, and subTemplateMultiList 🧩

These abstract data types are defined for IPFIX Structured Data [RFC6313]. They aren't real data types themselves. Instead, they're a way to show complex structure in IPFIX below the template level 🏗️. It's assumed that protocols using textual Information Element representation will bring their own structure. So Information Elements of these data types MUST NOT be used in textual representations. 🚫

5. Security Considerations 🔐

The security considerations for the IPFIX protocol [RFC7011] apply. 💀

Implementations of decoders of Information Element values using these representations must take care to correctly handle invalid input. That said, the encodings here aren't special in that respect 🫶.

The encoding in this document, and any representations built on top of it, are specifically not meant for storing data 📦. But storing data in the same format it's exchanged in is super common, and text is really easy to index and search. That makes it much easier to accidentally index or search privacy-sensitive data 😱. So when you store data using this encoding that came from measuring network traffic, the privacy considerations in Section 11.8 of [RFC7011] are especially important to follow 🙏.

When you use representations based on this encoding to send or store network traffic data, think about leaving out especially privacy-sensitive values by not representing the columns or keys that hold them. That's the black-marker anonymization idea discussed in Section 4 of [RFC6235] 🖍️. Other anonymization techniques described in [RFC6235] may also help in these situations.

The encodings for every abstract data type other than 'string' are designed to be representable in the US-ASCII character set. So they should be unproblematic for all Enclosing Contexts 🌸. The 'string' abstract data type, though, may run into trouble with ill-formed UTF-8 strings, as discussed in Section 6.1.6 of [RFC7011]. See [UTF8-EXPLOIT] for background ⚠️.

6. References 📚

6.1. Normative References

6.2. Informative References 📚

okay so these are the informative refs, which means they're the extra reading that gives context. the spec isn't blocked on them, but you'll want them if you're going deeper 🤓

  • [IANA-IPFIX] IANA, "IPFIX Information Elements", http://www.iana.org/assignments/ipfix/. 🌸
  • [IEEE.754.2008] Institute of Electrical and Electronics Engineers, "IEEE Standard for Floating-Point Arithmetic", IEEE Standard 754, August 2008. 🔢
  • [PRECIS] Saint-Andre, P. and M. Blanchet, "PRECIS Framework: Preparation and Comparison of Internationalized Strings in Application Protocols", Work in Progress, draft-ietf-precis-framework-18, September 2014. 🦋
  • [RFC6235] Boschi, E. and B. Trammell, "IP Flow Anonymization Support", RFC 6235, May 2011, http://www.rfc-editor.org/info/rfc6235. 🎀
  • [RFC6313] Claise, B., Dhandapani, G., Aitken, P., and S. Yates, "Export of Structured Data in IP Flow Information Export (IPFIX)", RFC 6313, July 2011, http://www.rfc-editor.org/info/rfc6313. 🧩
  • [RFC6885] Blanchet, M. and A. Sullivan, "Stringprep Revision and Problem Statement for the Preparation and Comparison of Internationalized Strings (PRECIS)", RFC 6885, March 2013, http://www.rfc-editor.org/info/rfc6885. 💜
  • [RFC7012] Claise, B. and B. Trammell, "Information Model for IP Flow Information Export (IPFIX)", RFC 7012, September 2013, http://www.rfc-editor.org/info/rfc7012. ✨
  • [RFC7013] Trammell, B. and B. Claise, "Guidelines for Authors and Reviewers of IP Flow Information Export (IPFIX) Information Elements", BCP 184, RFC 7013, September 2013, http://www.rfc-editor.org/info/rfc7013. 👑
  • [RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data Interchange Format", RFC 7159, March 2014, http://www.rfc-editor.org/info/rfc7159. 🍓
  • [UTF8-EXPLOIT] Davis, M. and M. Suignard, "Unicode Technical Report #36: Unicode Security Considerations", The Unicode Consortium, November 2012. 🔐
  • [W3C-XML] Bray, T., Paoli, J., Sperberg-McQueen, C., Maler, E., and F. Yergeau, "Extensible Markup Language (XML) 1.0 (Fifth Edition)", W3C Recommendation REC-xml, November 2008. 🌈

Appendix A. Example 🧁

okay this is the fun part!! we're gonna look at an IPFIX Template and a Data Record that follows that Template, and then see how that record would look in JSON using the stuff in this document 🥰

quick heads up: this is not a recommended way to represent anything. it's just an illustration of the encodings in this document. the quoting and formatting are JSON-specific, so don't take them as the one true format 💅

Figure 1 shows the Template written in Information Element Specifier (IESpec) format, which is defined in Section 10.1 of [RFC7013]. Figure 2 shows a matching JSON object for a record built from that template, in the text format this document describes. ✨

     flowStartMilliseconds(152)<dateTimeMilliseconds>[8]
     flowEndMilliseconds(153)<dateTimeMilliseconds>[8]
     octetDeltaCount(1)<unsigned64>[4]
     packetDeltaCount(2)<unsigned64>[4]
     sourceIPv6Address(27)<ipv6Address>[16]{key}
     destinationIPv6Address(28)<ipv6Address>[16]{key}
     sourceTransportPort(7)<unsigned16>[2]{key}
     destinationTransportPort(11)<unsigned16>[2]{key}
     protocolIdentifier(4)<unsigned8>[1]{key}
     tcpControlBits(6)<unsigned16>[2]
     flowEndReason(136)<unsigned8>[1]

          Figure 1: Sample Flow Template in IESpec Format
       {
           "flowStartMilliseconds": "2012-11-05T18:31:01.135",
           "flowEndMilliseconds": "2012-11-05T18:31:02.880",
           "octetDeltaCount": 195383,
           "packetDeltaCount": 88,
           "sourceIPv6Address": "2001:db8:c:1337::2",
           "destinationIPv6Address": "2001:db8:c:1337::3",
           "sourceTransportPort": 80,
           "destinationTransportPort": 32991,
           "protocolIdentifier": "tcp",
           "tcpControlBits": 19,
           "flowEndReason": 3
       }

           Figure 2: JSON Object Containing Sample Flow

Acknowledgments 🫶

huge thanks to Paul Aitken, Benoit Claise, Andrew Feren, Juergen Quittek, David Black, and the IESG for their reviews and comments 💖 thanks also to Dave Thaler and Stephan Neuhaus for the chats that made the floating-point representation section better 🔥. this work was materially supported by the European Union Seventh Framework Programme under grant agreement 318627 mPlane 🌸

Author's Address 💌

Brian Trammell Swiss Federal Institute of Technology Zurich Gloriastrasse 35 8092 Zurich Switzerland

Phone: +41 44 632 70 13 EMail: ietf@trammell.ch

📦 where this page comes from