RFC 6313
Export of Structured Data in IP Flow Information Export (IPFIX)
RFC 6313: Export of Structured Data in IP Flow Information Export (IPFIX) 💖
okay bestie, this one's a whole vibe 🎀 It's an Internet Standards Track document from the Internet Engineering Task Force (IETF), dated July 2011, and it's written by B. Claise, G. Dhandapani, P. Aitken and S. Yates from Cisco Systems, Inc. It's labelled ISSN 2070-1721 (the number for the series it lives in) ✨ It updates RFC 5102, which is the IPFIX information model, so you'll want that one open next to this. The title block doesn't list anything this document obsoletes, so nothing gets retired here 🌸
Abstract ✨
Okay so this doc is an extension to the IPFIX protocol from RFC 5101 and to the IPFIX information model from RFC 5102 🦋 It lets data records hold hierarchical structured data and lists (sequences) of Information Elements, which is a big deal because it means you can build complex structures like variable-length lists 👑 It also lets you describe how Templates are nested inside each other (the hierarchical containment relationships, iykyk), and it gives you the semantics for how all those list elements relate to one another inside a single structured data record 💅
Status of This Memo ✨
This is an Internet Standards Track document 🎀
It's a product of the IETF, which means it represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) 🫶 If you want more on Internet Standards, check out Section 2 of RFC 5741 📚
Info about the current status of this document, any errata, and how to send feedback is over at http://www.rfc-editor.org/info/rfc6313 🔥
Copyright Notice 🌈
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. 💜
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date of publication 🦋 Please actually read those docs, bestie, because they spell out your rights and restrictions for this document. Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License ⭐
Table of Contents 🧚
1. Overview ........................................................5
1.1. IPFIX Documents Overview ...................................5
1.2. Relationship between IPFIX and PSAMP .......................6
2. Introduction ....................................................6
2.1. The IPFIX Track ............................................7
2.2. The IPFIX Limitations ......................................8
2.3. Structured Data Use Cases ..................................8
2.4. Specifications Summary ....................................11
3. Terminology ....................................................11
3.1. New Terminology ...........................................12
3.2. Conventions Used in This Document .........................12
4. Linkage with the IPFIX Information Model .......................12
4.1. New Abstract Data Types ...................................12
4.1.1. basicList ..........................................12
4.1.2. subTemplateList ....................................12
4.1.3. subTemplateMultiList ...............................12
4.2. New Data Type Semantic ....................................13
4.2.1. List ...............................................13
4.3. New Information Elements ..................................13
4.3.1. basicList ..........................................13
4.3.2. subTemplateList ....................................13
4.3.3. subTemplateMultiList ...............................13
4.4. New Structured Data Type Semantics ........................13
4.4.1. undefined ..........................................14
4.4.2. noneOf .............................................14
4.4.3. exactlyOneOf .......................................14
4.4.4. oneOrMoreOf ........................................15
4.4.5. allOf ..............................................16
4.4.6. ordered ............................................16
4.5. Encoding of IPFIX Data Types ..............................16
4.5.1. basicList ..........................................17
4.5.2. subTemplateList ....................................19
4.5.3. subTemplateMultiList ...............................21
5. Structured Data Format .........................................25
5.1. Length Encoding Considerations ............................25
5.2. Recursive Structured Data .................................26
5.3. Structured Data Information Elements Applicability
in Options Template Sets ..................................26
5.4. Usage Guidelines for Equivalent Data Representations ......27
5.5. Padding ...................................................29
5.6. Semantic ..................................................29
6. Template Management ............................................33
7. The Collecting Process's Side ..................................33
8. Defining New Information Elements Based on the New
Abstract Data Types ............................................34
9. Structured Data Encoding Examples ..............................34
9.1. Encoding a Multicast Data Record with basicList ...........35
9.2. Encoding a Load-Balanced Data Record with a basicList .....37
9.3. Encoding subTemplateList ..................................38
9.4. Encoding subTemplateMultiList .............................41
9.5. Encoding an Options Template Set Using Structured Data ....46
10. Relationship with the Other IPFIX Documents ...................51
10.1. Relationship with Reducing Redundancy ....................51
10.1.1. Encoding Structured Data Element Using
Common Properties .................................51
10.1.2. Encoding Common Properties Elements with
Structured Data Information Element ...............51
10.2. Relationship with Guidelines for IPFIX Testing ...........53
10.3. Relationship with IPFIX Mediation Function ...............54
11. IANA Considerations ...........................................54
11.1. New Abstract Data Types ..................................54
11.1.1. basicList .........................................54
11.1.2. subTemplateList ...................................54
11.1.3. subTemplateMultiList ..............................55
11.2. New Data Type Semantics ..................................55
11.2.1. list ..............................................55
11.3. New Information Elements .................................55
11.3.1. basicList .........................................55
11.3.2. subTemplateList ...................................56
11.3.3. subTemplateMultiList ..............................56
11.4. New Structured Data Semantics ............................56
11.4.1. undefined .........................................56
11.4.2. noneOf ............................................57
11.4.3. exactlyOneOf ......................................57
11.4.4. oneOrMoreOf .......................................57
11.4.5. allOf .............................................57
11.4.6. ordered ...........................................58
12. Security Considerations .......................................58
13. References ....................................................58
13.1. Normative References .....................................58
13.2. Informative References ...................................58
14. Acknowledgements ..............................................59
Appendix A. Additions to XML Specification of IPFIX
Information Elements and Abstract Data Types ..........60
Appendix B. Encoding IPS Alert Using Structured Data
Information Elements ..................................651. Overview ✨
1.1. IPFIX Documents Overview 💖
Okay so, the IPFIX protocol [RFC5101] gives network admins access to IP Flow information 🫶
The architecture for exporting measured IP Flow information out of an IPFIX Exporting Process to a Collecting Process is defined in the IPFIX architecture [RFC5470], per the requirements defined in RFC 3917 [RFC3917].
The IPFIX architecture [RFC5470] specifies how IPFIX Data Records and Templates are carried via a congestion-aware transport protocol from IPFIX Exporting Processes to IPFIX Collecting Processes. 🚛
IPFIX has a formal description of IPFIX Information Elements, their name, type, and additional semantic information, as specified in the IPFIX information model [RFC5102]. The info model is literally the source of truth 📚
To gain a level of confidence in the IPFIX implementation, probe the conformity and robustness, and allow interoperability, the guidelines for IPFIX testing [RFC5471] present a list of tests for implementers of compliant Exporting Processes and Collecting Processes. Testing is very much giving main character energy 🧪
The Bidirectional Flow Export [RFC5103] specifies a method for exporting bidirectional flow (biflow) information using the IP Flow Information Export (IPFIX) protocol, representing each biflow using a single Flow Record. 🦋
"Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports" [RFC5473] specifies a bandwidth-saving method for exporting Flow or packet information. It separates information common to several Flow Records from information specific to an individual Flow Record, so common Flow information is exported only once. Lowkey genius 🧠✨
1.2. Relationship between IPFIX and PSAMP 🌸
The specification in this document applies to the IPFIX protocol specifications [RFC5101]. All specifications from [RFC5101] apply unless specified otherwise in this document.
The Packet Sampling (PSAMP) protocol [RFC5476] specifies the export of packet information from a PSAMP Exporting Process to a PSAMP Collecting Process. Like IPFIX, PSAMP has a formal description of its information elements, their name, type, and additional semantic information. The PSAMP information model is defined in [RFC5477]. Same energy, different vibe 💅
As the PSAMP protocol specifications [RFC5476] are based on the IPFIX protocol specifications, the specifications in this document are also valid for the PSAMP protocol. So it all connects 🫶
Indeed, the major difference between IPFIX and PSAMP is that the IPFIX protocol exports Flow Records while the PSAMP protocol exports Packet Reports. From a pure export point of view, IPFIX will not distinguish a Flow Record composed of several packets aggregated together from a Flow Record composed of a single packet. So the PSAMP export can be seen as a special IPFIX Flow Record containing information about a single packet. It's giving "one packet, one main character" 🧚
2. Introduction 🎀
While collecting the interface counters every five minutes has proven to be useful in the past, more and more granular information is required from network elements for a series of applications: performance assurance, capacity planning, security, billing, or simply monitoring. However, the amount of information has become so large that, when dealing with highly granular information such as Flow information, a push mechanism (as opposed to a pull mechanism, such as Simple Network Management Protocol (SNMP)) is the only solution for routers whose primary function is to route packets. Indeed, polling short-lived Flows via SNMP is not an option: high-end routers can support hundreds of thousands of Flows simultaneously. 😭 Furthermore, in order to reduce the export bandwidth requirements, the network elements have to integrate mediation functions to aggregate the collected information, both in space (typically, from different linecards or different Exporters) and in time.
Typically, it would be beneficial if access routers could export Flow Records, composed of the counters before and after an optimization mechanism on the egress interface, instead of exporting two Flow Records with identical tuple information. Why send duplicates when you can just send the good stuff? 💖
In terms of aggregation in time, let us imagine that, for performance assurance, the network management application must receive the performance metrics associated with a specific Flow, every millisecond. Since the performance metrics will be constantly changing, there is a new dimension to the Flow definition: we are not dealing anymore with a single Flow lasting a few seconds or a few minutes, but with a multitude of one millisecond sub-flows for which the performance metrics are reported. Sub-flows everywhere, it's so chaotic and I'm obsessed 🙈⏱️
Which current protocol is suitable for these requirements: push mechanism, highly granular information, and huge number of similar records? IPFIX, as specified in RFC 5101 would give part of the solution. IPFIX is the answer, bestie 👑
2.1. The IPFIX Track 🦋
The IPFIX working group has specified a protocol to export Flow information [RFC5101]. This protocol is designed to export information about IP traffic Flows and related measurement data, where a Flow is defined by a set of key attributes (e.g., source and destination IP address, source and destination port). Think of the key attributes as the Flow's ID card 🪪
The IPFIX protocol specification [RFC5101] specifies that traffic measurements for Flows are exported using a TLV (type, length, value) format. The information is exported using a Template Record that is sent once to export the {type, length} pairs that define the data format for the Information Elements in a Flow. The Data Records specify values for each Flow. So the Template is the blueprint and the Data Records are the actual content, iykyk 📐💜
Based on the requirements for IP Flow Information Export (IPFIX) [RFC3917], the IPFIX protocol has been optimized to export Flow-related information. However, thanks to its Template mechanism, the IPFIX protocol can export any type of information, as long as the relevant Information Element is specified in the IPFIX information model [RFC5102], registered with IANA [IANA-IPFIX], or specified as an enterprise-specific Information Element. For each Information Element, the IPFIX information model [RFC5102] defines a numeric identifier, an abstract data type, an encoding mechanism for the data type, and any semantic constraints. Only basic, single-valued data types, e.g., numbers, strings, and network addresses, are currently supported. The templates are so versatile, it's iconic 🌈✨
2.2. The IPFIX Limitations
okay so, the IPFIX protocol spec [RFC5101] has a big gap 🥲 It does NOT let you encode hierarchical structured data or arbitrary-length lists (sequences) of Information Elements as fields inside a Template Record. As it's currently specced, a Data Record is a "flat" list of single-valued attributes. But literally every data modeler wants to build complex hierarchies with multiple occurrences, like 0..* cardinality for each Information Element in the tree. It's giving "I need more" 💅
A classic example is the MPLS label stack entries model. An early NetFlow implementation used two Information Elements to represent one MPLS label stack entry: a "label stack entry position" followed by a "label stack value". But several drawbacks showed up 💀 First, the Information Elements in the Template Record had to be forced into an order where the position always came before the value. Problem is, some encoding optimizations depend on permuting the Information Element order. Second, a whole new layer of semantic intelligence, which isn't described in the information model, had to be hard-coded into the Collecting Process: the label value at position "X" in the stack is carried in the "label stack value" Information Element following by a "label stack entry position" Information Element containing the value "X". Honestly, that's too much 😭 So this model was abandoned.
The solution chosen in the IPFIX information model [RFC5102] is a long series of Information Elements: mplsTopLabelStackSection, mplsLabelStackSection2, mplsLabelStackSection3, mplsLabelStackSection4, mplsLabelStackSection5, mplsLabelStackSection6, mplsLabelStackSection7, mplsLabelStackSection8, mplsLabelStackSection9, and mplsLabelStackSection10. Ngl, this removes any ambiguity, but it overloads the IPFIX information model with repetitive info 🙃 And if mplsLabelStackSection11 is ever needed, IANA [IANA-IPFIX] won't be able to slot the new Information Element in next to the others in the registry, which could cause some confusion 🌀
2.3. Structured Data Use Cases
Clearly, the MPLS label stack entries problem is best solved with a real structured data type made of ("label stack entry position", "label stack value") pairs, which can be repeated multiple times in Flow Records. That's the most efficient option from an information model point of view, no cap 🎀
More examples fall into the same category: how to encode the list of output interfaces in a multicast Flow, how to encode the list of BGP Autonomous Systems (AS) in a BGP Flow, how to encode the BGP communities in a BGP Flow, etc. ✨
The one-way delay passive measurement, which is described in the IPFIX applicability [RFC5472], is another example that would benefit from a structured data encoding 🦋 Assuming synchronized clocks, the Collector can deduce the one-way delay between two Observation Points from the following two Information Elements, collected from two different Observation Points:
In practice, this means many pairs of (observationTimeMicroseconds, digestHashValue) have to be exported for each Observation Point, even if Hash-Based Filtering [RFC5475] is used. On top of that, if the goal is to understand the one-way delay per application type, the 5-tuple (source IP address, destination IP address, protocol, source port, destination port) would need to be added to every Flow Record. Instead of exporting that repetitive 5-tuple as part of every single Flow Record, a Flow Record built on a structured data type like the one below would save a lot of bandwidth 🫶
5-tuple
{ observationTimeMicroseconds 1, digestHashValue 1 }
{ observationTimeMicroseconds 2, digestHashValue 2 }
{ observationTimeMicroseconds 3, digestHashValue 3 }
{ ... , ... }As a last example, here's a more complex case of hierarchical structured data encoding 👑 Picture an IPS (Intrusion Prevention System) alert data structure with multiple participants, where each participant contains multiple attackers and multiple targets, and each target might be made of multiple applications, like this:
alert
signatureId
protocolIdentifier
riskRating
participant 1
attacker 1
sourceIPv4Address
applicationId
...
attacker N
sourceIPv4Address
applicationId
target 1
destinationIPv4Address
applicationId 1
...
applicationId n
...
target N
destinationIPv4Address
applicationId 1
...
applicationId n
participant 2
...To export this in IPFIX, the data would have to be flattened (so the hierarchical relationships get lost) and a new IPFIX Template created for each alert, based on the number of applicationId elements in each target, the number of targets and attackers in each participant, and the number of participants in each alert. Literally a template for every alert?? 💀 Each Template would be unique to its alert, and a huge amount of CPU, memory, and export bandwidth would get wasted creating, exporting, maintaining, and withdrawing the Templates. See Appendix B for a specific example related to this case study 🧚
2.4. Specifications Summary
This document specifies an IPFIX extension to support hierarchical structured data and variable-length lists. It does this by defining three new Information Elements and three corresponding new abstract data types called basicList, subTemplateList, and subTemplateMultiList. These are defined in Sections 4.1 and 4.3 🌈
The three Structured Data Information Elements carry some semantic information so the Collecting Process can understand the relationship between the different list elements. That semantic is there to express the relationship among the multiple top-level list elements. For example, if a list is made of the elements (A,B,C), the semantic expresses the relationship among A, B, and C, regardless of whether A, B, and C are individual elements or a list of elements 💖
It's important to note that while the Information Elements and abstract data types defined in the IPFIX information model [RFC5102] represent single values, these new abstract data types are structural in nature. They mainly contain references to other Information Elements and to Templates. By referencing other Information Elements and Templates from an Information Element's data content, it's possible to define complex data structures like variable-length lists and to specify hierarchical containment relationships between Templates. So this document prefers the more generic "Data Record" term over the "Flow Record" term 🍓
This document specifies three new abstract data types, which are basic building blocks for representing structured data. However, this document doesn't comment on all possible combinations of basicList, subTemplateList, and subTemplateMultiList. It doesn't limit the possible combinations either 🦋
3. Terminology
IPFIX-specific terminology used in this document is defined in Section 2 of the IPFIX protocol specification [RFC5101] and Section 3 of the PSAMP protocol specification [RFC5476]. As in [RFC5101], these IPFIX-specific terms have the first letter of a word capitalized when used in this document ✨
3.1. New Terminology
Structured Data Information Element
One of the Information Elements supporting structured data, i.e., the basicList, subTemplateList, or subTemplateMultiList Information Elements specified in Section 4.3.
3.2. Conventions Used in This Document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119] 📚
4. Linkage with the IPFIX Information Model
As in the IPFIX protocol specification [RFC5101], the new Information Elements specified in Section 4.3 MUST be sent in canonical format in network-byte order (also known as the big-endian byte ordering) 🔥
4.1. New Abstract Data Types
This document specifies three new abstract data types, as described below 💜
4.1.1. basicList
The type "basicList" represents a list of zero or more instances of any Information Element, primarily used for single-valued data types. Examples include a list of port numbers, a list of interface indexes, a list of AS in a BGP AS-PATH, etc. 🌸
4.1.2. subTemplateList
The type "subTemplateList" represents a list of zero or more instances of a structured data type, where the data type of each list element is the same and corresponds with a single Template Record. Examples include a structured data type made of multiple pairs of ("MPLS label stack entry position", "MPLS label stack value"), a structured data type made of performance metrics, and a structured data type made of multiple pairs of IP address, etc. 🎀
4.1.3. subTemplateMultiList
The type "subTemplateMultiList" represents a list of zero or more instances of a structured data type, where the data type of each list element can be different and corresponds with different Template definitions. Examples include a structured data type made of multiple access-list entries, where entries can be composed of different criteria types 🧚
4.2. New Data Type Semantic
This document specifies a new data type semantic, in addition to the ones specified in Section 3.2 of the IPFIX information model [RFC5102], as described below 🌈
4.2.1. List
A list represents an arbitrary-length sequence of zero or more structured data Information Elements, either composed of regular Information Elements or composed of data conforming to a Template Record 🦋
4.3. New Information Elements
This document specifies three new Information Elements, as described below 💅
4.3.1. basicList
A basicList specifies a generic Information Element with a basicList abstract data type as defined in Section 4.1.1 and list semantics as defined in Section 4.2.1. Examples include a list of port numbers, a list of interface indexes, etc. 🍓
4.3.2. subTemplateList
A subTemplateList specifies a generic Information Element with a subTemplateList abstract data type as defined in Section 4.1.2 and list semantics as defined in Section 4.2.1 🫶
4.3.3. subTemplateMultiList
A subTemplateMultiList specifies a generic Information Element with a subTemplateMultiList abstract data type as defined in Section 4.1.3 and list semantics as defined in Section 4.2.1 ✨
4.4. New Structured Data Type Semantics
Structured data type semantics are provided to express the relationship among multiple list elements in a Structured Data Information Element. These structured data type semantics require a new IPFIX subregistry, as specified in the "IANA Considerations" section. The semantics are specified in the following subsections 👑
4.4.1. undefined
okay so the undefined structured data type semantic is basically the "idk, you figure it out" setting 🤷♀️ it says the semantic of the list elements is not specified, and if a semantic does exist, it's up to the Collecting Process to draw its own conclusions 🔍 this is the default value, so it gets used whenever no other structured data type semantic applies 💅
for example, say a mediator wants to translate IPFIX [RFC5101] into structured data export using the specs in this document 🦋 the mediator doesn't know what the semantic is. it could only guess, because the IPFIX specifications [RFC5101] don't contain any semantic at all 😭 so the mediator should use the undefined semantic. honestly this is the safe choice when you're not sure!
4.4.2. noneOf
the noneOf structured data type semantic says that none of the elements are actual properties of the Data Record 🙅♀️ it's giving "this is NOT me" energy 💀
for example, a mediator wants to tell a Collector that a specific Flow is suspicious 👀 but it already checked and this Flow does not belong to attack type 1, attack type 2, or attack type 3. so this Flow might need some further inspection 🔎 in that case, the mediator reports the Flow Record with a basicList composed of (attack type 1, attack type 2, attack type 3) and the respective structured data type semantic of "noneOf". iykyk, that's a lot of "not" 😂
another example: a router is watching some specific BGP AS-PATHs and reports if a Flow belongs to any of them 🛰️ if the router wants to export that a Flow does NOT belong to any of the monitored BGP AS-PATHs, it reports a Data Record with a basicList composed of (BGP AS-PATH 1, BGP AS-PATH 2, BGP AS-PATH 3) and the respective structured data type semantic of "noneOf" ✨
4.4.3. exactlyOneOf
the exactlyOneOf structured data type semantic says that only a single element from the structured data is an actual property of the Data Record 🎯 this is basically a logical XOR operation, so it's a very picky, one-and-only vibe 💜
for example, if a Flow record has a basicList of outgoing interfaces with the "exactlyOneOf" semantic, then the reported Flow only egressed from a single interface, even though the Flow Record lists all the possible outgoing interfaces 🌈 this is a typical example of per destination load-balancing. the Flow picked one lane and that's it 🛣️
another example is a mediator that has to aggregate Data Records from different Observation Points and report one aggregated Observation Point 🧩 the catch is that the different Observation Points can be specified by different Information Element types depending on the Exporter. for example:
- Exporter1 Observation Point is characterized by the exporterIPv4Address, so a specific Exporter can be represented.
- Exporter2 Observation Point is characterized by the exporterIPv4Address and a basicList of ingressInterface, so the Exporting Process can express that the observations were made on a series of input interfaces.
- Exporter3 Observation Point is characterized by the exporterIPv4Address and a specific lineCardId, so the Exporting Process can express that the observation was made on a specific linecard.
if the mediator models those three types of Observation Points with the three Template Records below 🧁
Template Record 1: exporterIPv4Address
Template Record 2: exporterIPv4Address, basicList of
ingressInterface
Template Record 3: exporterIPv4Address, lineCardIdthen it can represent the aggregated Observation Point with a subTemplateMultiList and the semantic "exactlyOneOf" 🫶 the aggregated Observation Point is modeled with Data Records corresponding to either Template Record 1, Template Record 2, or Template Record 3, but not more than one of these. this means the Flow was observed at exactly one of the Observation Points reported. one spot, no duplicates, periodt 💅
4.4.4. oneOrMoreOf
the oneOrMoreOf structured data type semantic says that one or more elements from the list in the structured data are actual properties of the Data Record 🌸 this is equivalent to a logical OR operation, so it's a "at least one, maybe more, no pressure" kind of semantic 🙈
consider an example where a mediator has to report an aggregated Flow (for example, by aggregating IP addresses from IP prefixes), with an aggregated Observation Point 🍓 the different Observation Points can be specified by different Information Element types, as described in Section 4.4.2.
if the mediator models the three different types of Observation Points with the three Template Records below 🧚
Template Record 1: exporterIPv4Address
Template Record 2: exporterIPv4Address, basicList of
ingressInterface
Template Record 3: exporterIPv4Address, lineCardIdthen it can represent the aggregated Observation Point with a subTemplateMultiList and the semantic "oneOrMoreOf" ✨ the aggregated Observation Point is modeled with Data Records corresponding to either Template Record 1, Template Record 2, or Template Record 3. this implies that the Flow was observed on at least one of the Observation Points reported, and potentially on multiple Observation Points. main character Flow, multiple stages 🎀
4.4.5. allOf
the allOf structured data type semantic says that all of the list elements from the structured data are actual properties of the Data Record 👑 everybody's invited!
for example, if a Record has a basicList of outgoing interfaces with the "allOf" semantic, then the observed Flow is typically a multicast Flow, where each packet in the Flow has been replicated to each outgoing interface in the basicList 🦢 it's giving copy-paste, but make it networking 😭
4.4.6. ordered
the ordered structured data type semantic says that the elements from the list in the structured data are ordered 📋 the order actually matters here, bestie!
for example, an Exporter might want to export the AS10 AS20 AS30 AS40 BGP AS-PATH 🛤️ in that case, the Exporter reports a basicList composed of (AS10, AS20, AS30, AS40) and the respective structured data type semantic of "ordered" 🧠 lined up, in order, no skipping 💜
4.5. Encoding of IPFIX Data Types
the following subsections define how the abstract data types from Section 4.1 get encoded 🎀 these data types may be encoded using either fixed- or variable-length Information Elements, as discussed in Section 5.1 ✨ like in the IPFIX specifications [RFC5101], all lengths are specified in octets 🔢 numbers, but make them cute 🦋
4.5.1. basicList 🎀
okay bestie, the basicList Information Element from Section 4.3.1 is a list holding zero or more instances of an Information Element 🦋. Here's how it gets encoded:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Semantic |0| Field ID | Element... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...Length | basicList Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 1: basicList Encoding
Semantic 💖
The Semantic field tells you how the different Information Element values inside this Structured Data Information Element are related to each other. Go check IANA's "IPFIX Structured Data Types Semantics" registry for the details.
Field ID 🌸
Field ID is the Information Element identifier of whatever Information Element(s) live in the list.
Element Length ✨
Per Section 7 of [RFC5101], the Element Length field is the length, in octets, of each list element named by Field ID. Or it holds the value 0xFFFF if the length is encoded as a variable-length Information Element at the start of the basicList Content. 🫶
Effectively, Element Length is part of the header, so even for a zero-element list, it MUST NOT be omitted. 💅
basicList Content 🌈
A Collecting Process keeps decoding list elements from the basicList Content until no further data remains. No field count is included, but you can totally derive one when the Information Element gets decoded 🧚
Quick heads up: in the diagram above, Field ID has the Enterprise bit (the most significant bit) set to 0. If the Enterprise bit is set to 1 instead, a four-byte Enterprise Number MUST be encoded immediately after the Element Length, as shown below. See the "Field Specifier Format" section in the IPFIX protocol [RFC5101] for additional information. 👑
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Semantic |1| Field ID | Element... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...Length | Enterprise Number ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | basicList Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 2: basicList Encoding with Enterprise Number
Also, note that if a basicList has zero elements, the encoded data still contains the Semantic field, Field ID, the Element Length field, and the four-byte Enterprise Number (if present). The basicList Content is just empty in that case 🙈
If the basicList is a variable-length Information Element in less than 255 octets, it MAY be encoded with the Length field per Section 7 of [RFC5101], as shown in Figure 3. However, the three-byte length encoding, as shown in Figure 4, is RECOMMENDED (see Section 5.1) 💜
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (< 255)| Semantic |0| Field ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Element Length | basicList Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3: Variable-Length basicList Encoding (Length < 255 Octets)
If the basicList is a variable-length Information Element in 255 or more octets, it MUST be encoded with the Length field per Section 7 of [RFC5101], as follows: 🔥
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | Length (0 to 65535) | Semantic |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| Field ID | Element Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| basicList Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4: Variable-Length basicList Encoding (Length 0 to 65535 Octets)
4.5.2. subTemplateList 🦋
The subTemplateList Information Element represents a list of zero or more Data Records that all correspond to one specific Template. 💅 The Template Record referenced by a subTemplateList Information Element can itself contain other subTemplateList Information Elements. Those Template Record references are part of the Information Element's content in the Data Record, so you can build seriously complex hierarchical data structures ✨ Here's how a subTemplateList Information Element is encoded within a Data Record:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Semantic | Template ID | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| subTemplateList Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 5: subTemplateList Encoding
Semantic 🌸
The Semantic field indicates the relationship among the different Data Records within this Structured Data Information Element.
Template ID 💖
The Template ID field contains the ID of the Template used to encode and decode the subTemplateList Content.
subTemplateList Content 🧚
subTemplateList Content is made up of zero or more instances of Data Records, all corresponding to the Template ID in the Template ID field. A Collecting Process decodes the subTemplateList Content until no further data remains. A record count is not included, but it can be derived when the subTemplateList gets decoded. Encoding and decoding are done recursively if the specified Template itself contains Structured Data Information Elements, as described here. 🌈
Note that if a subTemplateList has zero elements, the encoded data contains only the Semantic field and the Template ID field. The subTemplateList Content is empty 🙈
If the subTemplateList is a variable-length Information Element in less than 255 octets, it MAY be encoded with the Length field per Section 7 of [RFC5101], as shown in Figure 6. However, the three-byte length encoding, as shown in Figure 7, is RECOMMENDED (see Section 5.1). 💜
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (< 255)| Semantic | Template ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| subTemplateList Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6: Variable-Length subTemplateList Encoding (Length < 255 Octets)
If the subTemplateList is a variable-length Information Element in 255 or more octets, it MUST be encoded with the Length field per Section 7 of [RFC5101], as follows: 🔥
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | Length (0 to 65535) | Semantic |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID | subTemplateList Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7: Variable-Length subTemplateList Encoding (Length 0 to 65535 Octets)
4.5.3. subTemplateMultiList 👑
Okay so, each element in a subTemplateList Information Element corresponds to a single Template. Sometimes it's really useful for a list to hold elements that correspond to different Templates 🌟 To make that work, each top-level element in a subTemplateMultiList Information Element carries a Template ID, a Length, and zero or more Data Records corresponding to that Template ID. The following diagram shows how a subTemplateMultiList Information Element is encoded within a Data Record. Note that the encoding following the Semantic field is consistent with the Set Header specified in [RFC5101]. 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Semantic | Template ID X |Data Records...|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... Length X | Data Record X.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record X.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record X.L Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Template ID Y |Data Records...|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... Length Y | Data Record Y.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record Y.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record Y.M Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Template ID Z |Data Records...|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... Length Z | Data Record Z.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record Z.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record Z.N Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+
Figure 8: subTemplateMultiList EncodingSemantic 🎀
Okay so the Semantic field is basically the big-picture vibe 💅 It tells you the top-level relationship among the whole series of Data Records, across all the different Template Records, inside this Structured Data Information Element. Main character energy, fr fr ✨
Template ID 🌸
Unlike subTemplateList, every element in a subTemplateMultiList carries its own Template ID, and that ID is what tells you how the Data Records right after it are encoded. Each element gets to be its own little star 🧚⭐
Data Records Length 💜
This is the total length of the Data Records encoding for the Template ID we just talked about. It counts the two bytes for the Template ID and the two bytes for the Data Records Length field itself, so don't forget to count those 🙈
Data Record X.M 🍓
The Data Record X.M is the Mth Data Record of Template Record X. A Collecting Process decodes those Data Records according to Template Record X until no data is left, using the Data Records Length X. Then further Template IDs and Data Records can be decoded according to the overall subTemplateMultiList length, so it's giving layers 🎀 A record count isn't included, but you can derive it when the Element Content gets decoded. And if the specified Template has Structured Data Information Elements of its own, encoding and decoding happen recursively, as described here. It's a whole inception situation 🌈
In the exceptional case of zero instances in the subTemplateMultiList, no data gets encoded. You only get the Semantic field and the Template ID field(s), and the Data Record Length field is set to zero. Quiet, but still valid 🦋
If the subTemplateMultiList is encoded as a variable-length Information Element in less than 255 octets, it MAY be encoded with the Length field per Section 7 of [RFC5101] as shown in Figure 9. However, the three-byte length encoding, as shown in Figure 10, is RECOMMENDED (see Section 5.1). Hot take: go with the three-byte one 💖
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (< 255)| Semantic | Template ID X |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Records Length X | Data Record X.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record X.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record X.L Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Template ID Y |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Records Length Y | Data Record Y.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record Y.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record Y.M Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Template ID Z |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Records Length Z | Data Record Z.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record Z.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | Data Record Z.N Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 9: Variable-Length subTemplateMultiList Encoding
(Length < 255 Octets)If the subTemplateMultiList is encoded as a variable-length Information Element in 255 or more octets, it MUST be encoded with the Length field per Section 7 of [RFC5101] as follows. No wiggle room here, bestie 🫶
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | Length (0 to 65535) | Semantic |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID X | Data Records Length X |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record X.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record X.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record X.L Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID Y | Data Records Length Y |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record Y.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record Y.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record Y.M Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID Z | Data Records Length Z |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record Z.1 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record Z.2 Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data Record Z.N Content ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 10: Variable-Length subTemplateMultiList Encoding
(Length 0 to 65535 Octets)5. Structured Data Format ✨
5.1. Length Encoding Considerations ✨
okay so the new Structured Data Information Elements are basically lists that can hold complex, nested, and repeated data 🌸 that's a lot of main character energy, bestie 💅
if the encoding has a fixed length (like, it has the same number of fixed-length elements, or every permutation of the list always produces the same total length), then the element length can go in the matching Template Record 🎀
but when the data is variable-length, hierarchical, or repeated with different element counts, where the number and length of elements change from record to record, we RECOMMEND that the Information Elements use the variable-length encoding from Section 7 of [RFC5101], with the length sitting before the Structured Data Information Element encoding 🦋 literally so iconic
the Exporting Process might not know the exact encoding size ahead of time, since the data is complex and repeated 🙈 so it can write the available data starting at a fixed offset and fill in the final length afterwards. that's why the three-byte length encoding is RECOMMENDED for variable-length Information Elements in all Template Records containing a Structured Data Information Element, even when the encoded length could be under 255 bytes, because the starting offset is already known 💖
and one super important rule 🚨 an Exporting Process MUST take care to not exceed the maximum allowed IPFIX message length of 65535 bytes as specified in [RFC5101]. no cap, don't break the message size limit 😭
5.2. Recursive Structured Data 🌈
it's totally possible to define recursive relationships between IPFIX structured data instances, like when you're representing a tree structure 🧚 the simplest version is probably a basicList where each element is itself a basicList, or a subTemplateList where one of the fields in the referenced Template is itself a subTemplateList pointing back to the same Template. iykyk 💜
also, the Exporting Process MUST take care when encoding recursively-defined structured data not to exceed the maximum allowed length of an IPFIX Message (as noted in Length Encoding Considerations). so yeah, the nesting can get wild, but the size limit doesn't care about your vibes 💀
5.3. Structured Data Information Elements Applicability in Options Template Sets ✨
Structured Data Information Elements MAY be used in Options Template Sets 🎀
let's look at a cute example. imagine a mediation function that has to aggregate Data Records from multiple Observation Point types:
Router 1, (interface 1)
Router 2, (linecard A)
Router 3, (linecard B)
Router 4, (linecard C, interface 2)to encode the PSAMP Selection Sequence Report Interpretation [RFC5476], the mediation function has to express this combo of Observation Points as one single new Observation Point 👑 remember from [RFC5476] that the PSAMP Selection Sequence Report Interpretation has these fields:
Scope: selectionSequenceId
Non-Scope: one Information Element mapping the Observation Point
selectorId (one or more)without structured data, there's literally no way to express that complex aggregated Observation Point as "one Information Element mapping the Observation Point" 😭 but structured data types make it totally doable. check out Section 9.5. for an encoding example of this case study 🦋
now about scope in the Options Template Record: the IPFIX specification [RFC5101] says that "the IPFIX protocol doesn't prevent the use of any Information Elements for scope". so a Structured Data Information Element MAY be used as scope in an Options Template Set 💅
building on that, the mediation function could export a name for this aggregated Observation Point too 🫶
Scope: Aggregated Observation Point (structured data)
Non-Scope: a new Information Element containing the name5.4. Usage Guidelines for Equivalent Data Representations 🍓
okay so basicList, subTemplateList, and subTemplateMultiList are all lists, which means there's often more than one way to represent what's basically the same data structure 🌸 sometimes one option has an edge over the other (like it's more compact, or uses fewer resources), and that's the one that's preferred over the alternate representation ✨
a subTemplateList can represent the same simple list of single-valued Information Elements as a basicList, as long as the referenced Template only has one single-valued Information Element. the subTemplateList encoding is two bytes more compact, BUT it needs a brand-new Template for each Information Element 😭 a basicList doesn't need any extra Template, so basicList is RECOMMENDED in this case 💖
a subTemplateMultiList with one Element can represent the contents of a subTemplateList too, but it carries two extra bytes (Element Length) 🙈 it's also potentially useful for a Collecting Process to know ahead of time that a subTemplateList directly means the list element types are consistent. so the subTemplateList Information Element is therefore RECOMMENDED in this case 🎀
the Semantic field in a subTemplateMultiList shows the top-level relationship among the series of Data Records from the different Template Records, inside this Structured Data Information Element 👑 if you need a semantic to describe the relationship among the Data Records for a single Template ID inside the subTemplateMultiList, then use an encoding based on a basicList of subTemplateLists instead; see Section 5.6 for more 🧚 or, if you need a semantic for ALL Data Records in a subTemplateMultiList (no matter which Template Record they came from), an encoding based on a subTemplateMultiList with one Data Record for a single Template ID can work 🦋
the referenced Information Elements in these Structured Data Information Elements can come from the IPFIX information model [RFC5102], the PSAMP information model [RFC5477], any Information Element in the IANA IPFIX registry [IANA-IPFIX], or enterprise-specific Information Elements 🌈
if a Template Record has a subTemplateList as its only field, you should consider a Set encoding from the IPFIX protocol specifications [RFC5101], unless:
- a relationship among multiple list elements has to be exported. in that case, the semantic from the IPFIX Structured Data Information Element can convey this relationship 💜
- the Exporting Process wants to convey the number of elements in the list, even for the special cases of zero or one element. honestly the empty list can't be represented with the IPFIX protocol specifications [RFC5101] at all 💀 for a single element list, the Template Record from the IPFIX protocol specification [RFC5101] could be used. BUT on top of the Template Record with the subTemplateList for multiple list elements, this extra Template would add some management work for both the Exporting Process and the Collecting Process, which might have to correlate info from two Template Records 😭
similarly, if a Template Record has a subTemplateMultiList as its only field, you should consider an IPFIX Message as described in the IPFIX protocol specification [RFC5101], unless:
- a relationship among top-level list elements has to be exported. then the semantic from the IPFIX Structured Data Information Element can convey this relationship ✨
- the Exporting Process wants to convey the number of Data Records for every Template in the subTemplateMultiList 🫶
5.5. Padding 🎀
the Exporting Process MAY insert some padding octets into structured data field values in a Data Record by including the 'paddingOctets' Information Element as described in [RFC5101], Section 3.3.1. 🌸 the paddingOctets Information Element can be included in a Template Record that's referenced by a structured data Information Element for this purpose 💖
5.6. Semantic ✨
okay so, the semantics of a Data Record are explicitly undefined once it gets to the Collecting Process (or anything past it) 🙈, UNLESS the data is sent with this extension and comes with explicit structured data type semantic info 💅. The Exporter is not the one who checks whether the semantic representation of a Data Record is valid 🫶.
if you want fancier meanings, you can stack up the Semantic Data Information Elements from this document 🧩 that's literally the whole vibe ✨
For example, exporting the AS10 AS20 AS30 AS40 {AS50,AS60} BGP AS-PATH gets reported as a basicList of two elements, and each element is its own basicList of BGP AS. The top-level structured data type semantic is "ordered". The first element holds a basicList of (AS10,AS20,AS30,AS40) with the semantic "ordered", and the second holds a basicList of (AS50, AS60) with the semantic "exactlyOneOf". A high-level Data Record diagram looks like this 👑
BGP AS-PATH = (basicList, ordered,
(basicList, ordered, AS10,AS20,AS30,AS40),
(basicList, exactlyOneOf, AS50, AS60)
)If you need a semantic that describes the relationship between the different Data Records that share one Template ID inside a subTemplateMultiList, then you should use an encoding based on a basicList of subTemplateLists. Case study time 🍓
Case study 1:
In this one, an Exporter watching security attacks has to export a list of security events made of attackers and targets. For the sake of the example, let's say the Collector can tell the attacker (shown with source fields) apart from the target (shown with destination fields). Imagine attackers A1 or A2 may attack targets T1 and T2 🎯
The first try uses a subTemplateMultiList made of two Template Records, one for the attacker and one for the target, and each one holds an IP address and a port.
Attacker Template Record = (src IP address, src port)
Target Template Record = (dst IP address, dst port)A high-level Data Record diagram looks like this:
Alert = (subTemplateMultiList, allOf,
(Attacker Template Record, A1, A2),
(Target Template Record, T1, T2)
)Here's the tea ☕ the Collecting Process can only conclude that the list of attackers (A1, A2) and the list of targets (T1, T2) are both there, but it has no idea how attackers and targets relate to each other. The Exporting Process would have to explicitly call out that relationship, because the top-level semantic offered by the subTemplateMultiList just isn't enough 💀
The only proper encoding for that semantic (that is, attacker A1 or A2 may attack target T1 and T2) uses a basicList of subTemplateLists, and it looks like this:
Attacker Template Record = (src IP address, src port)
Target Template Record = (dst IP address, dst port)
Alert = (basicList, allOf,
(subTemplateList, exactlyOneOf, attacker A1, A2)
(subTemplateList, allOf, target T1, T2)
)Case study 2:
This time the Exporter is still watching attacks, but now it needs to export a list of attackers and targets. Same deal: the Collector can tell attackers (source fields) from targets (destination fields). Imagine attacker A1 or A2 is hitting target T1, while attacker A3 is hitting targets T2 and T3 🦋
The first try uses a subTemplateMultiList holding Data Records for two Template Records, one for the attacker and one for the target, and each one has an IP address and a port.
Attacker Template Record = (src IP address, src port)
Target Template Record = (dst IP address, dst port)A high-level Data Record diagram looks like this:
Alert = (subTemplateMultiList, allOf,
(Attacker Template Record, A1, A2, A3),
(Target Template Record, T1, T2, T3)
)Again the Collecting Process can only conclude that the list of attackers (A1, A2, A3) and the list of targets (T1, T2, T3) are present, without knowing how they relate 🙃
The second try could use a Data Record definition made of the following:
Alert = (subTemplateMultiList, allOf,
(Attacker Template Record, A1, A2),
(Target Template Record, T1),
(Attacker Template Record, A3),
(Target Template Record, T2, T3)
)With that version, the Collecting Process can infer that the alert is made of the list of attackers (A1, A2), target (T1), attacker (A3), and list of targets (T2, T3). From the order the attackers and targets show up in, the Collector might be able to guess that some relationship exists among (A1, A2, T1) and (A2, T1, T2), but it can't figure out what that relationship actually is. So the Exporting Process needs to explicitly define the relationship between the attackers and targets, because the top-level semantic of the subTemplateMultiList is not enough 😭
The only proper encoding for that semantic (that is, attacker A1 or A2 attacks target T1, and attacker A3 attacks targets T2 and T3) uses a basicList of subTemplateLists and looks like this:
Participant P1 =
(basicList, allOf,
(subTemplateList, exactlyOneOf, attacker A1, A2)
(subTemplateList, undefined, target T1)
)
Participant P2 =
(basicList, allOf,
(subTemplateList, undefined, attacker A3,
(subTemplateList, allOf, targets T2, T3)
)The security alert is then represented as a subTemplateList of participants:
Alert =
(subTemplateList, allOf, Participant P1, Participant P2)Quick note 📝 when a Structured Data Information Element only has a single element, the Semantic field isn't really useful, since its whole job is describing the relationship between multiple elements. allOf, exactlyOneOf, or OneOrMoreOf would all give the same result semantically. So for a single element in a Structured Data Information Element, the default "undefined" semantic SHOULD be used 🌸
6. Template Management 🎀
This section adds a few more specific Template management and Template Withdrawal Message rules, compared to the IPFIX protocol specification [RFC5101].
First up, Template ID uniqueness is unchanged from [RFC5101]. The uniqueness is local to the Transport Session and Observation Domain that generated the Template ID. In other words, the Set ID used to export the Template Record has no effect on Template ID uniqueness 🧠
[RFC5101] says that "if an Information Element is required more than once in a Template, the different occurrences of this Information Element SHOULD follow the logical order of their treatments by the Metering Process". This rule MAY be ignored within Structured Data Information Elements 🦄
As [RFC5101] specifies, Templates that aren't used anymore SHOULD be deleted. Deleting a Template means it MUST NOT be used within subTemplateList and subTemplateMultiList anymore. Before reusing a Template ID, the Template MUST be deleted. To delete an allocated Template, you withdraw it with a Template Withdrawal Message 🗑️✨
7. The Collecting Process's Side 💖
This section adds some more specific specifications for the Collection Process, compared to Section 9 in the IPFIX protocol [RFC5101].
Okay, the big plot twist here: unlike the IPFIX specification in [RFC5101], IPFIX Messages with IPFIX Structured Data Information Elements change the IPFIX concept from the Collector's point of view. The data types live in the Data Records instead of the Template Records 🤯 For example, a basicList Information Element in a Template Record doesn't say what the list element data type is. That info is in the Data Record. And with a subTemplateMultiList, the Collecting Process must refer to the included Template Records right in the middle of decoding the Data Record.
As described in [RFC5101], a Collecting Process MUST note the Information Element identifier of any Information Element it does not understand, and it MAY discard that Information Element from the Flow Record. So a Collection Process that doesn't support the extension in this document can ignore the Structured Data Information Elements in a Data Record. It can also ignore the Data Records that contain these new Structured Data Information Elements, while still processing the other Data Records 🙌
If the structured data contains the "undefined" structured data type semantic, the Collecting Process MAY attempt to draw its own conclusion about the semantic in the Data Record 🔍
8. Defining New Information Elements Based on the New Abstract Data Types 🧚
This document specifies three new abstract data types: basicList, subTemplateList, and subTemplateMultiList. As specified in [RFC5102], defining new IPFIX Information Elements uses the Template specified in Section 2.1 of [RFC5102]. That Template mentions existing and future data types: "One of the types listed in Section 3.1 of this document or in a future extension of the information model". So you can define new Information Elements based on the three new abstract data types 🌈
The authors expect both enterprise-specific and IANA Information Elements based on the IPFIX structured data types to show up. For example, bgpPathList, bgpSequenceList, and bgpSetList have the abstract types and semantics basicList/ordered, basicList/ordered, and basicList/exactlyOneOf respectively, and those would define the complete semantic of the list. This specification doesn't define any new Information Elements beyond the ones in Section 4.3 💅
9. Structured Data Encoding Examples 🍓
The following examples are made only to illustrate how the extensions proposed in this document get encoded 💜
9.1. Encoding a Multicast Data Record with basicList ✨
okay so bestie, let's encode a multicast Data Record 🦋 It holds this data:
---------------------------------------------------------------
Ingress If | Source IP | Destination IP | Egress Interfaces
---------------------------------------------------------------
9 192.0.2.201 233.252.0.1 1, 4, 8
---------------------------------------------------------------First up is the Template Record for these multicast Flows, with Template ID 256 💖
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 256 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| ingressInterface = 10 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceIPv4Address = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| DestinationIPv4Address = 12 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| basicList = 291 | Field Length = 0xFFFF |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 11: Encoding basicList, Template Record
The list of outgoing interfaces is a basicList with semantic allOf 🌸 The Length of the list is chosen to be encoded in three bytes, even though it may be less than 255 octets, so don't be surprised by that little detail 🙈
Now the Data Set looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 256 | Length = 36 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ingressInterface = 9 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceIPv4Address = 192.0.2.201 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DestinationIPv4Address = 233.252.0.1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | List Length = 17 | semantic=allOf|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egressInterface FieldId = 14 |egressInterface Field Length=4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egressInterface value 1 = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egressInterface value 2 = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egressInterface value 3 = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 12: Encoding basicList, Data Record, Semantic allOf
In the example above, the basicList holds fixed-length elements. To show how variable-length elements get encoded, here's the same example with variable-length interface names in the basicList instead 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 256 | Length = 44 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ingressInterface = 9 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceIPv4Address = 192.0.2.201 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DestinationIPv4Address = 233.252.0.1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | List Length = 25 | semantic=allOf|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| InterfaceName FieldId = 82 | InterfaceName Field Len=0xFFFF|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length = 5 | 'F' | 'E' | '0' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| '/' | '0' | Length = 7 | 'F' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'E' | '1' | '0' | '/' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| '1' | '0' | Length = 5 | 'F' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'E' | '2' | '/' | '2' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 13: Encoding basicList, Data Record with Variable-Length Elements, Semantic allOf
9.2. Encoding a Load-Balanced Data Record with a basicList 💜
Next up, a load-balanced Data Record with the same data:
---------------------------------------------------------------
Ingress If | Source IP | Destination IP | Egress Interfaces
---------------------------------------------------------------
9 192.0.2.201 233.252.0.1 1, 4, 8
---------------------------------------------------------------So the Data Record egressed from either interface 1, 4, or 8 ✨ The Data Set is represented as follows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 256 | Length = 36 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ingressInterface = 9 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceIPv4Address = 192.0.2.201 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DestinationIPv4Address = 233.252.0.1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | List Length = 17 |sem=exactlyOne |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egressInterface FieldId = 14 |egressInterface Field Length=4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egressInterface value 1 = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egressInterface value 2 = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egressInterface value 3 = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Note: sem=exactlyOne represents semantic=exactlyOneOf
Figure 14: Encoding basicList, Data Record, Semantic exactlyOneOf
9.3. Encoding subTemplateList ✨
Okay so, remember Section 2.2? 🦋 To passively compute the one-way delay across the network, you need multiple pairs of (observationTimeMicroseconds, digestHashValue) collected from two different Observation Points. Not just one pair, a whole list. 💅 The cute way to export that is an optimized Data Record with a 5-tuple plus the list:
5-tuple
{ observationTimeMicroseconds 1, digestHashValue 1 }
{ observationTimeMicroseconds 2, digestHashValue 2 }
{ observationTimeMicroseconds 3, digestHashValue 3 }
{ ... , ... }A subTemplateList is the best fit for exporting that list of (observationTimeMicroseconds, digestHashValue) 🎀. For illustration, the list has 5 elements, but in real life it could be way more. 🌸
------------------------------------------------------------------
srcIP | dstIP | src | dst |proto| one-way delay
| | Port | Port | | metrics
------------------------------------------------------------------
192.0.2.1 192.0.2.105 1025 80 6 Time1, 0x0x91230613
Time2, 0x0x91230650
Time3, 0x0x91230725
Time4, 0x0x91230844
Time5, 0x0x91230978
------------------------------------------------------------------Next up, the Template for the one-way delay metrics 🧚 Here it is:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 16 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 257 | Field Count = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| observationTimeMicroSec=324 | Field Length = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| digestHashValue = 326 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 15: Encoding subTemplateList, Template for One-Way Delay
MetricsNow the Template Record for the Optimized Data Record. This is the one that's giving main character energy 👑
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 32 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 258 | Field Count = 6 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceIPv4Address = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| destinationIPv4Address = 12 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceTransportPort = 7 | Field Length = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| destinationTransportPort= 11| Field Length = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| protocolIdentifier = 4 | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| subTemplateList = 292 | Field Length = 0xFFFF |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 16: Encoding subTemplateList, Template RecordThe list of (observationTimeMicroseconds, digestHashValue) gets exported as a subTemplateList with semantic allOf. 🫶 The Length of the subTemplateList is chosen to be encoded in three bytes, even though it may be less than 255 octets. Yes, it's a little extra, but it's a choice, so respect 🙈
And here's the Data Record itself 😭🔥
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 258 | Length = 83 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceIPv4Address = 192.0.2.1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| destinationIPv4Address = 192.0.2.105 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceTransportPort = 1025 | destinationTransportPort = 80 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Protocol = 6 | 255 | one-way metrics list len = 63 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| semantic=allOf| TemplateID = 257 | TimeValue1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 2-5 of TimeValue1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 6-8 of TimeValue1 |digestHashVal1=|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 0x0x91230613 | TimeValue2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 2-5 of TimeValue2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 6-8 of TimeValue2 |digestHashVal2=|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 0x0x91230650 | TimeValue3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 2-5 of TimeValue3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 6-8 of TimeValue3 |digestHashVal3=|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 0x0x91230725 | TimeValue4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 2-5 of TimeValue4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 6-8 of TimeValue4 |digestHashVal4=|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 0x0x91230844 | TimeValue5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 2-5 of TimeValue5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... octets 6-8 of TimeValue5 |digestHashVal5=|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 0x0x91230978 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 17: Encoding subTemplateList, Data Set9.4. Encoding subTemplateMultiList 🎀
okay so this one is giving main character energy 💅 As explained in Section 4.5.3, a subTemplateMultiList is used to export a list of mixed-type content where each top-level element corresponds to a different Template Record. It's basically a list where every item can have its own shape 🦋
To show how it works, we're going to use a Data Record with these attributes:
5-tuple (Flow Keys), octetCount, packetCount
attributes for filtering
selectorId,
selectorAlgorithm
attributes for sampling
selectorId,
selectorAlgorithm,
samplingPacketInterval,
samplingPacketSpaceThis example shows that the Selector Report Interpretation [RFC5476] can be encoded with the subTemplateMultiList 🧚 To be specific, it covers the Property Match Filtering Selector Report Interpretation [RFC5476] used for filtering, and the Systemic Count-Based Sampling described in Section 6.5.2.1 of [RFC5476]. Some traffic gets filtered based on the match properties that were configured, some gets sampled, some gets both, and some gets neither 😭 Very dramatic, very real.
A subTemplateMultiList is the best fit for exporting this kind of variable data 💜 We define one Template for the filtering attributes and another for the sampling attributes. A Data Record can contain data for either Template, for both of them, or for neither one. Chaos, but organized chaos ✨
Here's an example where the Data Record includes both the filtering and sampling attributes.
Key attributes of the Data Record:
------------------------------------------------------------------
srcIP | dstIP | src | dst | proto | octetCount | packet
| | Port | Port | | | Count
------------------------------------------------------------------
2001:DB8::1 2001:DB8::2 1025 80 6 108000 120
------------------------------------------------------------------Filtering attributes:
-------------------------------------------
selectorId | selectorAlgorithm
-------------------------------------------
100 5 (Property Match Filtering)
-------------------------------------------Sampling attributes:
For Systemic Count-Based Sampling as defined in Section 6.5.2.1 of [RFC5476], the required algorithm-specific Information Elements are:
samplingPacketInterval: number of packets selected in a row samplingPacketSpace: number of packets between selections
Here's an example of a simple 1-out-of-100 systematic count-based Selector definition, where the samplingPacketInterval is 1 and the samplingPacketSpace is 99. Iconic math 🔥
--------------------------------------------------------------
selectorId | selectorAlgorithm | sampling | sampling
| | Packet | Packet
| | Interval | Space
--------------------------------------------------------------
15 1 (Count-Based Sampling) 1 99
--------------------------------------------------------------To represent the Data Record, we define these Template Records:
Template for filtering attributes: 259
Template for sampling attributes: 260
Template for Flow Record: 261
Flow record (261)
| (sourceIPv6Address)
| (destinationIPv6Address)
| (sourceTransportPort)
| (destinationTransportPort)
| (protocolIdentifier)
| (octetTotalCount)
| (packetTotalCount)
|
+------ filtering attributes (259)
| (selectorId)
| (selectorAlgorithm)
|
+------ sampling attributes (260)
| (selectorId)
| (selectorAlgorithm)
| (samplingPacketInterval)
| (samplingPacketSpace)The Template Record for filtering attributes looks like this 👑
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 259 | Field Count = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| selectorId = 302 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| selectorAlgorithm = 304 | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 18: Encoding subTemplateMultiList, Template for Filtering
AttributesThe Template for sampling attributes is defined as follows 🌸
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 24 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 260 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| selectorId = 302 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| selectorAlgorithm = 304 | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| samplingPacketInterval = 305| Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| samplingPacketSpace = 306 | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 19: Encoding subTemplateMultiList, Template for Sampling
AttributesQuick note 🙈 selectorAlgorithm is defined as unsigned16, and samplingPacketInterval and samplingPacketSpace are defined as unsigned32. Here they're squished down to 1 octet each, which is allowed by Reduced Size Encoding in Section 6.2 of the IPFIX protocol specifications [RFC5101]. Smart and efficient 🧠✨
The Template for the Flow Record is shown below 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 40 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 261 | Field Count = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceIPv6Address = 27 | Field Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| destinationIPv6Address = 28 | Field Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceTransportPort = 7 | Field Length = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| destinationTransportPort=11 | Field Length = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| protocolIdentifier = 4 | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| octetTotalCount = 85 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| packetTotalCount = 86 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| subTemplateMultiList = 293 | Field Length = 0XFFFF |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 20: Encoding subTemplateMultiList, Template for Flow RecordA subTemplateMultiList with semantic allOf is used to export the filtering and sampling attributes 💖 The Length field of the subTemplateMultiList is encoded in three bytes, even though it may be less than 255 octets. Extra? Yes. Valid? Also yes 💅
Here's how the Data Record gets encoded 🌈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 261 | Length = 73 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceIPv6Address = ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 2001:DB8::1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| destinationIPv6Address = ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 2001:DB8::2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceTransportPort = 1025 | destinationTransportPort = 80 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| protocol = 6 | octetTotalCount = 108000 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | packetTotalCount = 120 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | 255 | Attributes List Length = 21 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|semantic=allOf | Filtering Template ID = 259 |Filtering Attr |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...Length = 9 | selectorId = ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 100 |selectorAlg = 5| Sampling Template ID = 260 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sampling Attributes Length=11 | selectorId = ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 15 |selectorAlg = 1| Interval = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Space = 99 |
+-+-+-+-+-+-+-+-+
Figure 21: Encoding subTemplateMultiList, Data Set9.5. Encoding an Options Template Set Using Structured Data
okay so remember Section 5.3? 💖 Picture a mediation function that has to combine Data Records from different Observation Points, and it's giving chaos 😭
Observation Point 1 has one or more interfaces. Observation Points 2 and 3 have one or more linecards. Observation Point 4 has one or more interfaces and one or more linecards. Without structured data, you'd need a whole Template for every possible combo just to read each Observation Point's data. Ngl, that's a lot. 💀 With structured data, one basicList holds the interfaces and another basicList holds the linecards, and suddenly it's clean 🧚
To keep things simple, each Observation Point below has the Router's IP address plus an <interface>, a <linecard>, or a <linecard and interface>. This can totally be extended to a list of interfaces and a list of linecards using basicLists, like we just went over.
Observation Point 1: Router 1, (interface 1)
Observation Point 2: Router 2, (linecard A)
Observation Point 3: Router 3, (linecard B)
Observation Point 4: Router 4, (linecard C, interface 2)The mediation function wants to express all of this as one single Observation Point 🎀 so it can encode the PSAMP Selection Sequence Report Interpretation (SSRI). Quick recap from [RFC5476]: the PSAMP SSRI is made of a Scope part and a Non-Scope part.
Scope: selectionSequenceId
Non-Scope: one Information Element mapping the
Observation Point
selectorId (one or more)Here's how the Observation Point we just described can be encoded in a PSAMP SSRI 🦋
Selection Sequence 7 (Filter->Sampling):
Observation Point: subTemplateMultiList.
Router 1 (IP address = 192.0.2.11), (interface 1)
Router 2 (IP address = 192.0.2.12), (linecard A)
Router 3 (IP address = 192.0.2.13), (linecard B)
Router 4 (IP address = 192.0.2.14), (linecard C, interface 2)
selectorId: 5 (Filter, match IPv4SourceAddress 192.0.2.1)
selectorId: 10 (Sampler, Random 1 out-of ten)These Templates represent the PSAMP SSRI, so get ready to meet them 🌸
Template for representing PSAMP SSRI: 262
Template for representing interface: 263
Template for representing linecard: 264
Template for representing linecard and interface: 265And here's the tree version, so you can see how the Templates fit together 👑
PSAMP SSRI (262)
| (SelectionSequenceId)
|
+--- Observation Point 1 (263)
| (exporterIPv4Address)
| (Interface Id)
|
+--- Observation Point 2 and 3 (264)
| (exporterIPv4Address)
| (linecard)
|
+--- Observation Point 4 (265)
| (exporterIPv4Address)
| (linecard)
| (Interface Id)
|
| (selectorId 1)
| (selectorId 2)The example could honestly get even better with a basicList of selectorId if there are lots of Selector IDs to report. Future-you will thank me 🫶
Figure 22: PSAMP SSRI to Be Encoded
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 26 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 262 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field Count = 1 |0| selectionSequenceId = 301 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope 1 Length = 4 |0| subTemplateMultiList = 293 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 0xFFFF |0| selectorId = 302 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| selectorId = 302 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 23: Options Template Record for PSAMP SSRI Using
subTemplateMultiListA subTemplateMultiList with semantic allOf is used to hold the list of Observation Points. Slay 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 263 | Field Count = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| exporterIPv4Address = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| ingressInterface = 10 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 24: PSAMP SSRI, Template Record for interface
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 264 | Field Count = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| exporterIPv4Address = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| lineCardId = 141 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 25: PSAMP SSRI, Template Record for linecard
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 20 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 265 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| exporterIPv4Address = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| lineCardId = 141 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| ingressInterface = 10 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 26: PSAMP SSRI, Template Record for linecard and interfaceNow for the PSAMP SSRI Data Set itself 🧚♀️
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 262 | Length = 68 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| selectionSequenceId = 7 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | Observation Point List Len=49 |semantic=allOf |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OP1 Template ID = 263 | OP1 Length = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Router 1 exporterIPv4Address = 192.0.2.11 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OP1 ingressInterface = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OP2&OP3 Template ID = 264 | OP2 & OP3 Length = 20 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Router 2 exporterIPv4Address = 192.0.2.12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OP2 lineCardId = A |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Router 3 exporterIPv4Address = 192.0.2.13 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OP3 lineCardId = B |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OP4 Template ID = 265 | OP4 Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Router 4 exporterIPv4Address = 192.0.2.14 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OP4 lineCardId = C |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OP4 ingressInterface = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| selectorId = 5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| selectorId = 10 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 27: Example of a PSAMP SSRI Data Record, Encoded Using a
subTemplateMultiListHeads up 👀 this Data Record has multiple instances of Template 264, one for Observation Point 2 (Router2, linecard A) and one for Observation Point 3 (Router3, linecard B). If one Observation Point had both linecard A and linecard B instead, a basicList would be the move for listing those linecards.
10. Relationship with the Other IPFIX Documents
10.1. Relationship with Reducing Redundancy
"Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports" [RFC5473] is all about saving bandwidth when you export Flow or packet information with the IP Flow Information Export (IPFIX) protocol 💜
It defines the commonPropertiesID Information Element for exporting Common Properties.
10.1.1. Encoding Structured Data Element Using Common Properties
When Structured Data Information Elements contain repeated elements, those elements may be swapped out for a commonPropertiesID Information Element, as specified in [RFC5473]. The elements that get replaced may include the basicList, subTemplateList, and subTemplateMultiList Information Elements 🌈
This technique might help with reducing the bandwidth needed for the export. However, a detailed analysis of the gain hasn't been done yet, so check out Section 8.3 of [RFC5473] for more considerations 🦋
10.1.2. Encoding Common Properties Elements with Structured Data Information Element 🎀
okay so the Structured Data Information Element MAY be used to define a list of commonPropertiesID, and it works as a replacement for the stuff in [RFC5473] 💅✨
and bestie, the example in Figures 1 and 2 of [RFC5473] can totally be encoded with the specs in this doc. no cap 🧚
+----------------+-------------+---------------------------+
| sourceAddressA | sourcePortA | <Flow1 information> |
+----------------+-------------+---------------------------+
| sourceAddressA | sourcePortA | <Flow2 information> |
+----------------+-------------+---------------------------+
| sourceAddressA | sourcePortA | <Flow3 information> |
+----------------+-------------+---------------------------+
| sourceAddressA | sourcePortA | <Flow4 information> |
+----------------+-------------+---------------------------+
| ... | ... | ... |
+----------------+-------------+---------------------------+Figure 28: Common and Specific Properties Exported Together [RFC5473]
+------------------------+-----------------+-------------+
| index for properties A | sourceAddressA | sourcePortA |
+------------------------+-----------------+-------------+
| ... | ... | ... |
+------------------------+-----------------+-------------+
+------------------------+---------------------------+
| index for properties A | <Flow1 information> |
+------------------------+---------------------------+
| index for properties A | <Flow2 information> |
+------------------------+---------------------------+
| index for properties A | <Flow3 information> |
+------------------------+---------------------------+
| index for properties A | <Flow4 information> |
+------------------------+---------------------------+Figure 29: Common and Specific Properties Exported Separately According to [RFC5473]
+----------------+-------------+---------------------------+
| sourceAddressA | sourcePortA | <Flow1 information> |
+----------------+-------------+---------------------------+
| <Flow2 information> |
+---------------------------+
| <Flow3 information> |
+---------------------------+
| <Flow4 information> |
+---------------------------+
| ... |
+---------------------------+Figure 30: Common and Specific Properties Exported with Structured Data Information Element
so for Figure 28, the <Flow information> could be encoded with a basicList if it's a single Information Element, with a subTemplateList if it's a Template Record, or with a subTemplateMultiList if it's made up of different Template Records. pick your fighter 🦋
using Structured Data Information Elements instead of the techniques from "Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports" [RFC5473] is giving main character energy 👑 because only one Template Record gets defined. so the Collector's life gets easier with Template management and combining Template/Options Template Records 🥰
BUT, and this is important bestie 🫶, swapping in Structured Data Information Elements for the techniques in "Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports" only works for the simplified cases. the "Multiple Data Reduction" (Section 7.1 [RFC5473]) might be too complex to encode with Structured Data Information Elements 😭
10.2. Relationship with Guidelines for IPFIX Testing 🧪
[RFC5471] has a list of tests for implementers of IP Flow Information Export (IPFIX) compliant Exporting Processes and Collecting Processes 🍓
ngl, [RFC5471] doesn't define any structured data element specific tests. BUT the Structured Data Information Elements can still be used in lots of the [RFC5471] tests 💜
the [RFC5471] series of tests could be super useful because the document says every Information Element type should be tested. however, not every case from this doc is tested in [RFC5471] 💀
these sections are especially noteworthy:
- 3.2.1. Transmission of Template with Fixed-Size Information Elements
- each data type should be used in at least one test. The new data types specified in Section 4.1 should be included in this test.
- 3.2.2. Transmission of Template with Variable-Length Information Elements
- this test should be expanded to include Data Records containing variable length basicList, subTemplateList, and subTemplateMultiList Information Elements.
- 3.3.1. Enterprise-Specific Information Elements
- this test should include the export of basicList, subTemplateList, and subTemplateMultiList Information Elements containing Enterprise-specific Information Elements, e.g., see the example in Figure 2.
- 3.3.3. Multiple Instances of the Same Information Element in One Template
- this test should verify that multiple instances of the basicList, subTemplateList, and subTemplateMultiList Information Elements are accepted.
- 3.5. Stress/Load Tests
- since the structured data types defined here allow modeling of complex data structures, they may be useful for stress testing both Exporting Processes and Collecting Processes.
10.3. Relationship with IPFIX Mediation Function 🌸
the Structured Data Information Elements would be seriously beneficial for exporting aggregated Data Records in the mediation function, like we saw with the aggregated Observation Point example in Section 5.3 🔥
11. IANA Considerations 📋
this doc specifies several new IPFIX abstract data types, a new IPFIX Data Type Semantic, and several new Information Elements 🌈
there are two new IPFIX registries, and the existing IPFIX Information Element registry has been updated (details below) ✨
11.1. New Abstract Data Types 🧚
Section 4.1 of this document specifies several new IPFIX abstract data types. Per Section 6 of the IPFIX information model [RFC5102], new abstract data types can be added to the IPFIX information model in the IPFIX Information Element Data Types registry 💖
these abstract data types have been added to the IPFIX Information Element Data Types registry:
11.1.1. basicList
The type "basicList" represents a list of any Information Element used for single-valued data types. 🍓
11.1.2. subTemplateList
The type "subTemplateList" represents a list of a structured data type, where the data type of each list element is the same and corresponds with a single Template Record. 🦋
11.1.3. subTemplateMultiList
The type "subTemplateMultiList" represents a list of structured data types, where the data types of the list elements can be different and correspond with different Template definitions. 💅
11.2. New Data Type Semantics 🎀
Section 4.2 of this document specifies a new IPFIX Data Type Semantic. Per Section 3.2 of the IPFIX information model [RFC5102], new data type semantics can be added to the IPFIX information model. So the IANA IPFIX informationElementSemantics registry [IANA-IPFIX], which contains all the data type semantics from Section 3.2 of [RFC5102], has been augmented with the "list" value below 🫶
11.2.1. list
A list is a structured data type, being composed of a sequence of elements, e.g., Information Element, Template Record. 🌟
11.3. New Information Elements 💜
Section 4.3 of this document specifies several new Information Elements that have been created in the IPFIX Information Element registry [IANA-IPFIX] 🔥
these new Information Elements have been added to the IPFIX Information Element registry:
11.3.1. basicList
Name: basicList
Description:
Specifies a generic Information Element with a basicList abstract
data type. Examples include a list of port numbers, and a list of
interface indexes.
Abstract Data Type: basicList
Data Type Semantics: list
ElementId: 291
Status: current11.3.2. subTemplateList
Name: subTemplateList
Description:
Specifies a generic Information Element with a subTemplateList
abstract data type.
Abstract Data Type: subTemplateList
Data Type Semantics: list
ElementId: 292
Status: current11.3.3. subTemplateMultiList
Name: subTemplateMultiList
Description:
Specifies a generic Information Element with a
subTemplateMultiList abstract data type.
Abstract Data Type: subTemplateMultiList
Data Type Semantics: list
ElementId: 293
Status: current11.4. New Structured Data Semantics 🧁
Section 4.4 of this document specifies a series of new IPFIX structured data type semantics, which are expressed as an 8-bit value. This means a new "IPFIX Structured Data Types Semantics" IPFIX subregistry [IANA-IPFIX] has to be created 🙈
entries may be added to this subregistry subject to a Standards Action [RFC5226]. Initially, this registry includes all the structured data type semantics listed below ✨
11.4.1. undefined
Name: undefined
Description: The "undefined" structured data type semantic specifies
that the semantic of list elements is not specified and that, if a
semantic exists, then it is up to the Collecting Process to draw its
own conclusions. The "undefined" structured data type semantic is
the default structured data type semantic.
Value: 0xFF
Reference: RFC 631311.4.2. noneOf
Name: noneOf
Description: The "noneOf" structured data type semantic specifies
that none of the elements are actual properties of the Data Record.
Value: 0x00
Reference: RFC 631311.4.3. exactlyOneOf
Name: exactlyOneOf
Description: The "exactlyOneOf" structured data type semantic
specifies that only a single element from the structured data is an
actual property of the Data Record. This is equivalent to a logical
XOR operation.
Value: 0x01
Reference: RFC 631311.4.4. oneOrMoreOf
Name: oneOrMoreOf
Description: The "oneOrMoreOf" structured data type semantic
specifies that one or more elements from the list in the structured
data are actual properties of the Data Record. This is equivalent to
a logical OR operation.
Value: 0x02
Reference: RFC 631311.4.5. allOf
Name: allOf
Description: The "allOf" structured data type semantic specifies that
all of the list elements from the structured data are actual
properties of the Data Record.
Value: 0x03
Reference: RFC 631311.4.6. ordered
Name: ordered Description: The "ordered" structured data type
semantic specifies that elements from the list in the structured data
are ordered.
Value: 0x04
Reference: RFC 631312. Security Considerations 🔐
okay bestie, the security part is SERIOUS 😭 adding complex data types makes the Collector way more complicated to build, and more complicated code is where sneaky bugs like buffer overflows love to hide 💀 this adds extra risk if Datagram Transport Layer Security (DTLS) isn't being used, or if the Observation Point and the Collector live in different trust domains 🫣 otherwise, the same security considerations from the IPFIX protocol [RFC5101] and the IPFIX information model [RFC5102] still apply 🔒✨
13. References 📚
13.1. Normative References
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 💜
- [RFC5101] Claise, B., Ed., "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information", RFC 5101, January 2008. 🦋
- [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, "Information Model for IP Flow Information Export", RFC 5102, January 2008. 🌸
- [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008. 🎀
13.2. Informative References
- [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander, "Requirements for IP Flow Information Export (IPFIX)", RFC 3917, October 2004. 🧚
- [RFC5103] Trammell, B. and E. Boschi, "Bidirectional Flow Export Using IP Flow Information Export (IPFIX)", RFC 5103, January 2008. 🔥
- [RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, "Architecture for IP Flow Information Export", RFC 5470, March 2009. ⭐
- [RFC5471] Schmoll, C., Aitken, P., and B. Claise, "Guidelines for IP Flow Information Export (IPFIX) Testing", RFC 5471, March 2009. 🍓
- [RFC5472] Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP Flow Information Export (IPFIX) Applicability", RFC 5472, March 2009. 🌈
- [RFC5473] Boschi, E., Mark, L., and B. Claise, "Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports", RFC 5473, March 2009. 💅
- [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F. Raspall, "Sampling and Filtering Techniques for IP Packet Selection", RFC 5475, March 2009. 🫶
- [RFC5476] Claise, B., Ed., Johnson, A., and J. Quittek, "Packet Sampling (PSAMP) Protocol Specifications", RFC 5476, March 2009. ✨
- [RFC5477] Dietz, T., Claise, B., Aitken, P., Dressler, F., and G. Carle, "Information Model for Packet Sampling Exports", RFC 5477, March 2009. 👑
- [IANA-IPFIX] IANA, "IP Flow Information Export (IPFIX) Entities",
<http://www.iana.org/>. 🌸
14. Acknowledgements 🫶
the authors wanna give a big thank-you to Zhipu Jin, Nagaraj Varadharajan, Brian Trammel, Atsushi Kobayashi, and Rahul Patel for their feedback 💖 and also to Gerhard Muenz for proofreading the whole doc 🙈 the real MVPs fr fr ✨
Appendix A. Additions to XML Specification of IPFIX Information Elements and Abstract Data Types 🧚
okay so this appendix is basically a bunch of extra stuff for the machine-readable description of the IPFIX information model, which is written in XML in Appendices A and B in [RFC5102] 🤓 quick heads up: this appendix is informational, but the text in Section 4 (which gets generated from this appendix) is the normative one, so that's the one that really counts 💅
the new field definitions below get tacked onto the IPFIX information model in Appendix A of [RFC5102] 🦋
<field name="basicList"
dataType="basicList"
group="structured-data"
dataTypeSemantics="List"
elementId="291" applicability="all" status="current">
<description>
<paragraph>
Represents a list of zero or more instances of
any Information Element, primarily used for
single-valued data types. Examples include a list of port
numbers, list of interface indexes, and a list of AS in a
BGP AS-PATH.
</paragraph>
</description>
</field>
<field name="subTemplateList"
dataType="subTemplateList"
group="structured-data"
dataTypeSemantics="List"
elementId="292" applicability="all" status="current">
<description>
<paragraph>
Represents a list of zero or more instances of a
structured data type, where the data type of each list
element is the same and corresponds with a single
Template Record. Examples include a structured data type
composed of multiple pairs of ("MPLS label stack entry
position", "MPLS label stack value"), a structured data
type composed of performance metrics, and a structured data
type composed of multiple pairs of IP address.
</paragraph>
</description>
</field>
<field name="subTemplateMultiList"
dataType="subTemplateMultiList"
group="structured-data"
dataTypeSemantics="List"
elementId="293" applicability="all" status="current">
<description>
<paragraph>
Represents a list of zero or more instances of
structured data types, where the data type of each list
element can be different and corresponds with
different Template definitions. Examples include, a
structured data type composed of multiple access-list
entries, where entries can be composed of different
criteria types.
</paragraph>
</description>
</field>next up, the structured data type semantic definitions, which also get appended to the IPFIX information model in Appendix A of [RFC5102] ✨ these are the rules for how the items in a structured list relate to each other, and they're honestly so satisfying 🥰
<structuredDataTypeSemantics>
<structuredDataTypeSemantic name="undefined" value="255">
<description>
<paragraph>
The "undefined" structured data type semantic specifies
that the semantic of list elements is not specified and
that, if a semantic exists, then it is up to the
Collecting Process to draw its own conclusions. The
"undefined" structured data type semantic is the default
structured data type semantic.
</paragraph>
</description>
</structuredDataTypeSemantic>
<structuredDataTypeSemantic name="noneOf" value="0">
<description>
<paragraph>
The "noneOf" structured data type semantic specifies
that none of the elements are actual properties of the
Data Record.
</paragraph>
</description>
</structuredDataTypeSemantic>
<structuredDataTypeSemantic name="exactlyOneOf" value="1">
<description>
<paragraph>
The "exactlyOneOf" structured data type semantic
specifies that only a single element from the structured
data is an actual property of the Data Record. This is
equivalent to a logical XOR operation.
</paragraph>
</description>
</structuredDataTypeSemantic>
<structuredDataTypeSemantic name="oneOrMoreOf" value="2">
<description>
<paragraph>
The "oneOrMoreOf" structured data type semantic
specifies that one or more elements from the list in the
structured data are actual properties of the Data
Record. This is equivalent to a logical OR operation.
</paragraph>
</description>
</structuredDataTypeSemantic>
<structuredDataTypeSemantic name="allOf" value="3">
<description>
<paragraph>
The "allOf" structured data type semantic specifies that
all of the list elements from the structured data are
actual properties of the Data Record.
</paragraph>
</description>
</structuredDataTypeSemantic>
<structuredDataTypeSemantic name="ordered" value="4">
<description>
<paragraph>
The "ordered" structured data type semantic specifies
that elements from the list in the structured data are
ordered.
</paragraph>
</description>
</structuredDataTypeSemantic>
</structuredDataTypeSemantics>last bit of the appendix 🎀 the schema definitions below get appended to the abstract data types from Appendix B of [RFC5102] 💜 this schema and its namespace are registered by IANA at http://www.iana.org/assignments/xml-registry/schema/ipfix.xsd, so it's official official 🍓
<simpleType name="dataType">
<restriction base="string">
<enumeration value="basicList">
<annotation>
<documentation>
Represents a list of zero or more instances of
any Information Element, primarily used for
single-valued data types. Examples include a list of port
numbers, a list of interface indexes, and a list of AS in a
BGP AS-PATH.
</documentation>
</annotation>
</enumeration>
<enumeration value="subTemplateList">
<annotation>
<documentation>
Represents a list of zero or more instances of a
structured data type, where the data type of each list
element is the same and corresponds with a single
Template Record. Examples include a structured data type
composed of multiple pairs of ("MPLS label stack entry
position", "MPLS label stack value"), a structured
data type composed of performance metrics, and a
structured data type composed of multiple pairs of IP
address.
</documentation>
</annotation>
</enumeration>
<enumeration value="subTemplateMultiList">
<annotation>
<documentation>
Represents a list of zero or more instances of
structured data types, where the data type of each
list element can be different and corresponds with
different Template definitions. An example is a
structured data type composed of multiple
access-list entries, where entries can be
composed of different criteria types.
</documentation>
</annotation>
</enumeration>
</restriction>
</simpleType>
<simpleType name="dataTypeSemantics">
<restriction base="string">
<enumeration value="List">
<annotation>
<documentation>
Represents an arbitrary-length sequence of structured
data elements, either composed of regular Information
Elements or composed of data conforming to a Template
Record.
</documentation>
</annotation>
</enumeration>
</restriction>
</simpleType>
<complexType name="structuredDataTypeSemantics">
<sequence>
<element name="structuredDataTypeSemantic"
minOccurs="1" maxOccurs="unbounded">
<complexType>
<sequence>
<element name="description" type="text"/>
</sequence>
<attribute name="name" type="string" use="required"/>
<attribute name="value" type="unsignedByte" use="required"/>
</complexType>
</element>
</sequence>
</complexType>
<element name="structuredDataTypeSemantics"
type="structuredDataTypeSemantics">
<annotation>
<documentation>
Structured data type semantics express the relationship
among multiple list elements in a structured data
Information Element.
</documentation>
</annotation>
</element>that's the whole appendix, bestie! 🌈✨
Appendix B. Encoding IPS Alert Using Structured Data Information Elements ✨
okay so this appendix is a full-on example of how to pack complex, multi-level data into Structured Data Information Elements 🦋 We're building an IPS alert, and it shows how a basicList of subTemplateLists can carry meaning at several levels of the hierarchy at once. iconic, honestly 💅
An IPS alert has three mandatory attributes: signatureId, protocolIdentifier, and riskRating 🔥 It can also hold zero or more participants, and each participant can hold zero or more attackers and zero or more targets. An attacker carries sourceIPv4Address and applicationId, and a target carries destinationIPv4Address and applicationId 🎀
Quick heads-up: the signatureId and riskRating Information Element fields are made up just for these examples, so their Field IDs are shown as N/A 🙈 signatureId helps uniquely identify the IPS signature that set off the alert. riskRating is the potential risk of the traffic that triggered the alert, on a scale of 0-100, where 100 is the most serious 💀
Now let's look at case study 2 of Section 5.6 🌸 The IPS alert holds participants encoded as a subTemplateList with semantic allOf. Each participant uses a basicList of subTemplateLists for its attackers and targets. To keep things simple, the alert has two participants, P1 and P2. In participant P1, attacker A1 or A2 attacks target T1. In participant P2, attacker A3 attacks targets T2 and T3 🧚
Participant P1:
(basicList, allOf,
(subTemplateList, exactlyOneOf, attacker A1, A2)
(subTemplateList, undefined, target T1)
)Participant P2:
(basicList, allOf,
(subTemplateList, undefined, attacker A3,
(subTemplateList, allOf, targets T2, T3)
)Alert :
(subTemplateList, allOf, Participant P1, Participant P2)Here's the whole alert laid out as a table 🍓
------------------------------------------------------------------
| | | participant
sigId |protocol| risk | attacker | target
| Id | Rating | IP | appId | IP | appId
------------------------------------------------------------------
1003 17 10 192.0.2.3 103 192.0.2.103 3001
192.0.2.4 104
192.0.2.5 105 192.0.2.104 4001
192.0.2.105 5001
------------------------------------------------------------------Participant P1 contains:
Attacker A1: (IP, appId)=(192.0.2.3, 103)
Attacker A2: (IP, appId)=(192.0.2.4, 104)
Target T1: (IP, appId)= (192.0.2.103, 3001)Participant P2 contains:
Attacker A3: (IP, appId) = (192.0.2.5, 105)
Target T2: (IP, appId)= (192.0.2.104, 4001)
Target T3: (IP, appId)= (192.0.2.105, 5001)To represent an alert, we define these four Templates 💜
- Template for target (268)
- Template for attacker (269)
- Template for participant (270)
- Template for alert (271)
Picture the nesting like this 🌈
alert (271)
| (signatureId)
| (protocolIdentifier)
| (riskRating)
|
+------- participant (270)
|
+------- attacker (269)
| (sourceIPv4Address)
| (applicationId)
|
+------- target (268)
| (destinationIPv4Address)
| (applicationId)Attackers always have a single applicationId, but targets usually have several 👀 For simplicity, this example only shows one applicationId in the target.
Here's the Template Record for target, Template ID 268 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 16 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 268 | Field Count = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| destinationIPv4Address = 12 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| applicationId = 95 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 31: Encoding IPS Alert, Template for Target
Template Record for attacker, Template ID 269 🌸
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 16 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 269 | Field Count = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceIPv4Address = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| applicationId = 95 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 32: Encoding IPS Alert, Template for Attacker
Template Record for participant, Template ID 270 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 12 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 270 | Field Count = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| basicList = 291 | Field Length = 0xFFFF |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 33: Encoding IPS Alert, Template for Participant
So the participant Template Record has one basicList Information Element, which is a list of subTemplateLists of attackers and targets 🫶
Template Record for IPS alert, Template ID 271 🌈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 24 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 271 | Field Count = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| signatureId = N/A | Field Length = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| protocolIdentifier = 4 | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| riskRating = N/A | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| subTemplateList = 292 | Field Length = 0xFFFF |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 34: Encoding IPS Alert, Template for IPS Alert
The subTemplateList in the alert Template Record holds a list of participants 🍓
The Length of basicList and subTemplateList is encoded in three bytes, even though they may be less than 255 octets 🎀
Last but not least, here's the Data Set 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 271 | Length = 102 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| signatureId = 1003 | protocolId=17 | riskRating=10 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 |participant List Length = 91 |semantic=allOf |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| participant Template ID = 270 | 255 | P1 List Len = |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 41 | semantic=allOf| P1 List Field ID = 292 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P1 List Field ID Len = 0xFFFF | 255 |P1 attacker ...|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| List Len = 19 |sem=exactlyOne | P1 attacker Template ID = 269 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P1 attacker A1 sourceIPv4Address = 192.0.2.3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P1 attacker A1 applicationId = 103 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P1 attacker A2 sourceIPv4Address = 192.0.2.4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P1 attacker A2 applicationId = 104 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | P1 target List Len = 11 | sem=undefined |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P1 target Template ID = 268 | P1 target T1 destinationIPv4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... Address = 192.0.2.103 |P1 target T1 applicationId =...|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 3001 | 255 | P2 List Len = |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 41 | semantic=allOf| P2 List Field ID = 292 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P2 List Field ID Len = 0xFFFF | 255 |P2 attacker ...|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| List Len = 11 | sem=undefined | P2 attacker Template ID = 269 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P2 attacker A3 sourceIPv4Address = 192.0.2.5 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P2 attacker A3 applicationId = 105 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 255 | P2 target List Len = 19 |semantic=allOf |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| P2 target Template ID = 268 | P2 target T2 destinationIPv4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... Address = 192.0.2.104 |P2 target T2 applicationId =...|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 4001 | P2 target T3 destinationIPv4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... Address = 192.0.2.105 |P2 target T3 applicationId =...|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... 5001 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Note: sem=exactlyOne represents semantic=exactlyOneOf
Figure 35: Encoding IPS Alert, Data Set
Authors' Addresses ✨
okay bestie, here are the four Cisco people who wrote this whole thing 🫶 they're literally the reason IPFIX got its glow-up, so here's where to find them 💖
Benoit Claise 🦋 is in Belgium:
Benoit Claise
Cisco Systems, Inc.
De Kleetlaan 6a b1
Diegem 1813
Belgium
Phone: +32 2 704 5622
EMail: bclaise@cisco.comGowri Dhandapani 🌸 is in the United States, in Herndon, Virginia:
Gowri Dhandapani
Cisco Systems, Inc.
13615 Dulles Technology Drive
Herndon, Virginia 20171
United States
Phone: +1 408 853 0480
EMail: gowri@cisco.comPaul Aitken 🧚 is in the United Kingdom, in Edinburgh:
Paul Aitken
Cisco Systems, Inc.
96 Commercial Quay
Commercial Street
Edinburgh, EH6 6LX
United Kingdom
Phone: +44 131 561 3616
EMail: paitken@cisco.comAnd last but not least, Stan Yates 👑 is in the United States too, over in Research Triangle Park, North Carolina:
Stan Yates
Cisco Systems, Inc.
7100-8 Kit Creek Road
PO Box 14987
Research Triangle Park, North Carolina 27709-4987
United States
Phone: +1 919 392 8044
EMail: syates@cisco.comthat's the whole crew, periodt 💅🌈
📦 where this page comes from
- ietf: rfc6313 2011-07
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms