RFC 5655
Specification of the IP Flow Information Export (IPFIX) File Format
RFC 5655 🎀
Okay bestie, this is RFC 5655, and it's called "Specification of the IP Flow Information Export (IPFIX) File Format" ✨. It's a Standards Track document from the Network Working Group, so it's a real protocol spec 💅. It's dated October 2009. Five authors wrote it: B. Trammell and E. Boschi (Hitachi Europe), L. Mark (Fraunhofer IFAM), T. Zseby (Fraunhofer FOKUS), and A. Wagner (ETH Zurich) 🫶. The title block doesn't list any RFCs that it obsoletes or updates, so nothing gets replaced or changed by it 🦋.
Abstract ✨
Okay so this doc is all about a file format for storing flow data, based on the IPFIX protocol 🧚. It lays out a list of requirements for flat-file, binary flow data file formats, and then it specifies the IPFIX File format that meets those requirements, built on IPFIX Messages. The whole goal is interoperability and reusability, so a huge range of flow storage, processing, and analysis tools can all work together with it 🌈.
Status of This Memo 🌸
This document specifies an Internet standards track protocol for the Internet community, and it's asking for discussion and suggestions for improvements 💬. For the standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1) 📚. Distribution of this memo is unlimited 🥰.
Copyright Notice 💜
Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the BSD License ⚠️.
Table of Contents 🧾
1. Introduction ....................................................4
1.1. IPFIX Documents Overview ...................................4
2. Terminology .....................................................5
3. Design Overview .................................................6
4. Motivation ......................................................7
5. Requirements ...................................................10
5.1. Record Format Flexibility .................................10
5.2. Self-Description ..........................................10
5.3. Data Compression ..........................................11
5.4. Indexing and Searching ....................................11
5.5. Error Recovery ............................................12
5.6. Authentication, Confidentiality, and Integrity ............12
5.7. Anonymization and Obfuscation .............................13
5.8. Session Auditability and Replayability ....................13
5.9. Performance Characteristics ...............................14
6. Applicability ..................................................14
6.1. Storage of IPFIX-Collected Flow Data ......................14
6.2. Storage of NetFlow-V9-Collected Flow Data .................15
6.3. Testing IPFIX Collecting Processes ........................15
6.4. IPFIX Device Diagnostics ..................................16
7. Detailed File Format Specification .............................16
7.1. File Reader Specification .................................16
7.2. File Writer Specification .................................17
7.3. Specific File Writer Use Cases ............................18
7.3.1. Collocating a File Writer with a Collecting
Process ............................................18
7.3.2. Collocating a File Writer with a Metering Process ..19
7.3.3. Using IPFIX Files for Archival Storage .............20
7.3.4. Using IPFIX Files as Documents .....................20
7.3.5. Using IPFIX Files for Testing ......................21
7.3.6. Writing IPFIX Files for Device Diagnostics .........22
7.3.7. IPFIX File Manipulation ............................22
7.4. Media Type of IPFIX Files .................................22
8. File Format Metadata Specification .............................22
8.1. Recommended Options Templates for IPFIX Files .............22
8.1.1. Message Checksum Options Template ..................23
8.1.2. File Time Window Options Template ..................23
8.1.3. Export Session Details Options Template ............24
8.1.4. Message Details Options Template ...................26
8.2. Recommended Information Elements for IPFIX Files ..........29
8.2.1. collectionTimeMilliseconds .........................29
8.2.2. collectorCertificate ...............................29
8.2.3. exporterCertificate ................................29
8.2.4. exportSctpStreamId .................................30
8.2.5. maxExportSeconds ...................................30
8.2.6. maxFlowEndMicroseconds .............................30
8.2.7. maxFlowEndMilliseconds .............................31
8.2.8. maxFlowEndNanoseconds ..............................31
8.2.9. maxFlowEndSeconds ..................................32
8.2.10. messageMD5Checksum ................................32
8.2.11. messageScope ......................................32
8.2.12. minExportSeconds ..................................33
8.2.13. minFlowStartMicroseconds ..........................33
8.2.14. minFlowStartMilliseconds ..........................34
8.2.15. minFlowStartNanoseconds ...........................34
8.2.16. minFlowStartSeconds ...............................34
8.2.17. opaqueOctets ......................................35
8.2.18. sessionScope ......................................35
9. Signing and Encryption of IPFIX Files ..........................36
9.1. CMS Detached Signatures ...................................36
9.1.1. ContentInfo ........................................37
9.1.2. SignedData .........................................38
9.1.3. SignerInfo .........................................38
9.1.4. EncapsulatedContentInfo ............................39
9.2. Encryption Error Resilience ...............................39
10. Compression of IPFIX Files ....................................39
10.1. Supported Compression Formats ............................40
10.2. Compression Recognition at the File Reader ...............40
10.3. Compression Error Resilience .............................40
11. Recommended File Integration Strategies .......................41
11.1. Encapsulation of Non-IPFIX Data in IPFIX Files ...........41
11.2. Encapsulation of IPFIX Files within Other File Formats ...42
12. Security Considerations .......................................42
12.1. Relationship between IPFIX File and Transport
Encryption ...............................................43
12.2. End-to-End Assertions for IPFIX Files ....................43
12.3. Recommendations for Strength of Cryptography for
IPFIX Files ..............................................44
13. IANA Considerations ...........................................44
14. Acknowledgements ..............................................46
15. References ....................................................47
15.1. Normative References .....................................47
15.2. Informative References ...................................48
Appendix A. Example IPFIX File ...................................49
A.1. Example Options Templates .................................50
A.2. Example Supplemental Options Data .........................52
A.3. Example Message Checksum ..................................54
A.4. File Example Data Set .....................................55
A.5. Complete File Example .....................................55
Appendix B. Applicability of IPFIX Files to NetFlow V9 Flow
Storage ..............................................57
B.1. Comparing NetFlow V9 to IPFIX .............................57
B.1.1. Message Header Format .................................57
B.1.2. Set Header Format .....................................58
B.1.3. Template Format .......................................59
B.1.4. Information Model .....................................59
B.1.5. Template Management ...................................59
B.1.6. Transport .............................................59
B.2. A Method for Transforming NetFlow V9 Messages to IPFIX ....60
B.3. NetFlow V9 Transformation Example .........................611. Introduction 🦋
Okay so this document specifies a file format based on IPFIX 📂, and the whole point is making it interoperable and reusable across a wild variety of flow storage, processing, and analysis tools 🛠️. It starts with an overview of the IPFIX File format, plus a quick summary of how IPFIX Files work in Section 3 👀. The detailed spec lives in Section 7, which also covers general specifications for IPFIX File Readers and IPFIX File Writers, along with specific recommendations for common situations where they get used 💅. The format uses the IPFIX Options mechanism for extra file metadata, so it doesn't need any protocol extensions and implementations don't have to do much work to adapt 🙌. The detailed definition of the Options Templates used for storage metadata is in Section 8, and Appendix A has a full example IPFIX File 🎀.
Here's the main advantage of file-based storage: files can be easily encapsulated inside each other and inside other data storage and transmission formats 🧚. The IPFIX File format takes advantage of that to provide encryption (Section 9) and compression (Section 10) 🔐. Section 11 gives specific recommendations for integrating IPFIX File data with other formats 🌸.
The IPFIX File format was designed to fit a wide variety of flow storage situations, and Section 4 explains the motivation behind creating it 💖. The document also outlines the set of requirements the format is meant to meet in Section 5, and in Section 6 it looks at how well the format applies to various specific application areas 🔥. These sections are there to give background on how IPFIX Files were developed, so read them if you want the backstory ✨.
1.1. IPFIX Documents Overview 📚
okay so the IPFIX protocol has a whole family of docs, and bestie, you need to know who does what 💅 "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information" [RFC5101] and its associated documents define the IPFIX protocol, which gives network engineers and administrators access to IP traffic flow information 🦋
"Architecture for IP Flow Information Export" [RFC5470] defines the architecture for exporting measured IP flow information from an IPFIX Exporting Process to an IPFIX Collecting Process, plus the basic terminology for the parts of that architecture. It follows the requirements set out in "Requirements for IP Flow Information Export" [RFC3917]. Then [RFC5101] covers the details of how IPFIX Data Records and Templates get carried over a congestion-aware transport protocol from an IPFIX Exporting Process to an IPFIX Collecting Process 🚀
"Information Model for IP Flow Information Export" [RFC5102] describes the Information Elements IPFIX uses, including how they're named, numbered, and how their data types are encoded 🧚
"IP Flow Information Export (IPFIX) Applicability" [RFC5472] covers the different applications of the IPFIX protocol and how they use the exported information. It also ties the IPFIX architecture to other measurement architectures and frameworks 🌸
And then, lowkey important: "Exporting Type Information for IP Flow Information Export (IPFIX) Information Elements" [RFC5610] specifies a way to encode Information Model properties inside an IPFIX Message stream ✨
So this document references [RFC5101] and [RFC5470] for terminology. It defines IPFIX File Writer and IPFIX File Reader in terms of the IPFIX Exporting Process and IPFIX Collecting Process definitions from [RFC5101]. It also extends the IPFIX Information Model from [RFC5102] with new Information Elements for IPFIX File metadata. And it uses the method from [RFC5610] so IPFIX Files containing enterprise-specific Information Elements can describe themselves 🎀💖
2. Terminology 📖
okay this section is where the vocab gets locked in, so pay attention 🙈 It defines terms related to the IPFIX File format. Any terms this document uses that are defined in the "Terminology" section of [RFC5101] are to be interpreted the way they're defined there, so you don't need to memorize two versions 💜
IPFIX File: An IPFIX File is a serialized stream of IPFIX Messages. That stream can live on a filesystem or travel using any technique people normally use for files 🗂️ Any IPFIX Message stream that would count as valid when sent over one of the specified IPFIX transports (Stream Control Transmission Protocol (SCTP), TCP, or UDP) as defined in [RFC5101] is considered an IPFIX File. This document adds on top of that with recommendations for building IPFIX Files that meet the requirements in Section 5 ✨
IPFIX File Reader: An IPFIX File Reader is a process that reads IPFIX Files from a filesystem. It operates as an IPFIX Collecting Process as specified in [RFC5101], except as modified by this document 👑
IPFIX File Writer: An IPFIX File Writer is a process that writes IPFIX Files to a filesystem. It operates as an IPFIX Exporting Process as specified in [RFC5101], except as modified by this document 🌈
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. These are the capitalized requirement words, and they carry their RFC 2119 meaning here 🫶
3. Design Overview 🧩
okay the big idea is simple: an IPFIX File is just a data stream holding one or more IPFIX Messages, serialized to some filesystem 💅 Any set of valid IPFIX Messages can be serialized into an IPFIX File, but the specification also includes guidelines meant to make it easier to store and retrieve flow data in the IPFIX File format 🦋
IPFIX Files contain only IPFIX Messages. Any file metadata, like checksums or export session details, is stored using Options within the IPFIX Message. This design is completely compatible with the IPFIX protocol on the wire, which is honestly so iconic 🌸 Here's a schematic of a typical IPFIX File:
+=======================================+
| IPFIX File |
| +===================================+ |
| | IPFIX Message | |
| | +-------------------------------+ | |
| | | IPFIX Message Header | | |
| | +-------------------------------+ | |
| | +-------------------------------+ | |
| | | Options Template Set | | |
| | | Options Template Record | | |
| | | . . . | | |
| | +-------------------------------+ | |
| | +-------------------------------+ | |
| | | Template Set | | |
| | | Template Record | | |
| | | . . . | | |
| | +-------------------------------+ | |
| +===================================+ |
| | IPFIX Message | |
| | +-------------------------------+ | |
| | | IPFIX Message Header | | |
| | +-------------------------------+ | |
| | +-------------------------------+ | |
| | | Data Set | | |
| | | Data Record | | |
| | | . . . | | |
| | +-------------------------------+ | |
| | +-------------------------------+ | |
| | | Data Set | | |
| | | Data Record | | |
| | | . . . | | |
| | +-------------------------------+ | |
| | . . . | |
| +===================================+ |
| . . . |
+=======================================+
Figure 1: Typical File Structure4. Motivation ✨
okay bestie, flow data gets stored as files across a whole continuum of time scales 🕰️. The tools that analyze flow data and make analysis products usually treat files as a convenient unit of work with an ephemeral lifetime 💅. A set of flows for a security investigation might sit in one file for the whole investigation, then get passed around between incident handlers over email or inside an external incident handling workflow app 🔍. Internet measurement research is another big one: flow sets get published as files, kinda like libpcap [pcap] packet trace files, so everyone has common datasets for repeatable research 🧪. Those files can live for months or years! 😭 And operational flow measurement systems need long-term archival storage too, either as the primary flow data repository or as a backing tier for online storage in a relational database management system (RDBMS) 🗄️.
The huge variety of flow data uses, plus all the different storage approaches out there right now, shows we need a standard approach to flow storage that works across that whole continuum of time scales 🌈. A storage format based on flat files would best meet all those different storage requirements ✨. Tons of work has gone into structured storage via RDBMS, but relational database systems aren't a good basis for format standardization. Their internal data structures are generally private to one implementation and can change for internal reasons 🙅. There are also a wide variety of operations available on flat files, and external tools and standards can be leveraged to meet file-based flow storage requirements 🛠️. Flow data isn't very semantically complicated, but it's managed in really high volume, so an RDBMS-based flow storage system wouldn't benefit much from relational database technology 💀.
The simplest way to make a new file format is to serialize some internal data model to disk, with either textual or binary representation of data elements, plus some framing strategy for delimiting fields and records 📝. "Ad hoc" file formats like this have several important disadvantages 😤. They impose the semantics of the data model they came from onto the file format, so they're difficult to extend, describe, and standardize 🙃.
Honestly, one de facto standard for storing flow data is one of these ad hoc formats 👀. A common method of storing data collected via Cisco NetFlow is to serialize a stream of raw NetFlow datagrams into files. These NetFlow PDU files are a collection of header-prefixed blocks (corresponding to the datagrams as received on the wire) containing fixed-length binary flow records 📦. NetFlow V5, V7, and V8 data can be mixed within one file, because the header on each datagram defines the NetFlow version of the records following it 🔀. The NetFlow PDU file format has all the disadvantages of an ad hoc format and isn't extensible to data models other than Cisco NetFlow's, but it's at least reasonably well understood because it's everywhere 🫶.
Over the past decade, XML has emerged as a new "universal" representation format for structured data 🌐. It's meant to be human readable, and that's a big reason for its rapid adoption 💖. But XML has limited usefulness for representing network flow data 🙃. Flow data has a simple, repetitive, non-hierarchical structure that doesn't benefit much from XML. An XML representation of flow data would basically be an essentially flat list of the attributes and their values for each flow record 📋.
The XML approach to data encoding is very heavyweight compared to binary flow encoding 🏋️. XML's use of start- and end-tags, plus plaintext encoding of the actual values, leads to significant inefficiency in encoding size 😮💨. Typical network traffic datasets can contain millions or billions of flows per hour of traffic represented 🤯, so any increase in storage size per record can have a dramatic impact on flow data storage and transfer sizes 📈. Data compression algorithms can partially remove the redundancy XML introduces, but they add their own overhead too 🫠.
A further problem is that XML processing tools require a full XML parser 🧐. XML parsers are fully general, which makes them complex, resource-intensive, and relatively slow, introducing significant processing time overhead for large network-flow datasets 🐌. Binary flow data encodings are way simpler. Their parsers only need to parse a very small header and then have complete knowledge of all the following fields for that flow, so these can be read in a very efficient linear fashion ⚡.
This leads us to propose the IPFIX Message format as the basis for a new flow data file format 🎯. The IPFIX Working Group, in defining the IPFIX protocol, already defined an information model and data formatting rules for representing flow data 📚. Especially at shorter time scales, when a file is a unit of data interchange, the filesystem can be viewed as simply another IPFIX Message transport between processes 🚚. This format is especially well suited to representing flow data, since it was designed specifically for flow data export. It's easily extensible (unlike ad hoc serialization) and compact (unlike XML) 💅. Plus, IPFIX is an IETF Standards-Track protocol for the export and collection of flow data. Using a common format for storage and analysis at the collection side lets implementors use substantially the same information model and data formatting implementation for transport as well as storage 🤝✨.
5. Requirements
In this section, we outline a proposed set of requirements [SAINT2007] for any persistent storage format for flow data 📜. First and foremost, a flow data file format should support storage across the continuum of time scales important to flow storage applications ⏳. Each requirement below is broadly applicable to flow storage applications, though each may be more important at certain time scales. For each one, we first identify the requirement, then explain how the IPFIX Message format addresses it, or briefly outline the changes that must be made so an IPFIX-based file format can meet it 🧩.
5.1. Record Format Flexibility
Because different network flow attribute measurement systems collect a wide variety of flow attributes, the ideal flow storage format won't impose a single data model or a specific record type on the flows it stores 🎨. The file format must be flexible and extensible. That means it must support the definition of multiple record types within the file itself, and it must be able to support new field types for data within the records in a graceful way 🌷.
IPFIX provides record format flexibility through the use of Templates to describe each Data Record, through the use of an IANA Registry to define its Information Elements, and through the use of enterprise-specific Information Elements 🏢✨.
5.2. Self-Description
Archived data may be read at a time in the future, when any external reference to the meaning of the data may be lost 🕰️. So the ideal flow storage format should be self-describing. That means a process reading flow data from storage should be able to properly interpret the stored flows without reference to anything other than standard sources (e.g., the standards document describing the file format) and the stored flow data itself 🔎.
The IPFIX Message format is partially self-describing. IPFIX Templates containing only IANA-assigned Information Elements can be completely interpreted according to the IPFIX Information Model without additional external data ✅.
However, Templates containing private information elements lack detailed type and semantic information 😬. A Collecting Process receiving Data Records described by a Template containing enterprise-specific Information Elements it does not understand can only treat the data contained within those Information Elements as octet arrays 🥲.
To be fully self-describing, enterprise-specific Information Elements must be additionally described via IPFIX Options according to the Information Element Type Options Template defined in [RFC5610] 💫.
5.3. Data Compression
Regardless of the representation format, flow data describing traffic on real networks tends to be highly compressible 🗜️. Compression tends to improve the scalability of flow collection systems, by reducing the disk storage and I/O bandwidth requirement for a given workload 📉. The ideal flow storage format should support applications that wish to leverage this fact by supporting compression of stored data 💜.
The IPFIX Message format has no support for data compression, as the IPFIX protocol was designed for speed and simplicity of export 🏎️. Of course, any flat file is readily compressible using a wide variety of external data compression tools, formats, and algorithms. Therefore, this requirement can be met via encapsulation in one of these formats 📦. Section 10 specifies an encapsulation based on bzip2 or gzip, to maximize interoperability 🌍.
A few simple optimizations can be made by File Writers to increase the integrity and usability of compressed IPFIX data. These are outlined in Section 10.3 ✨.
5.4. Indexing and Searching
Binary, record-stream-oriented file formats natively support only one form of searching: sequential scan in file order 🔍. By choosing the order of records in a file carefully (e.g., by flow end time), a file can be indexed by a single key 🔑.
Beyond this, properly addressing indexing is an application-specific problem, as it inherently involves trade-offs between storage complexity and retrieval speed. Requirements vary widely based on time scales and the types of queries used from site to site ⚖️. However, a generic standard flow storage format may provide limited direct support for indexing and searching 🙂.
The ideal flow storage format will support a limited table of contents facility noting that the records in a file contain data relating only to certain keys or values of keys, in order to keep multi-file search implementations from having to scan a file for data it does not contain 📖.
The IPFIX Message format has no direct support for indexing 😅. However, the technique described in "Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports" [RFC5473] can be used to describe the contents of a file in a limited way 🧠. Additionally, as flow data is often sorted and divided by time, the start and end time of the flows in a file may be declared using the File Time Window Options Template defined in Section 8.1.2 ⏰.
5.5. Error Recovery 🛟
okay so when you're storing flow data for archival purposes, the big worry is that hardware or software faults sneak in and corrupt the data over time 😭 the ideal flow storage format will support detecting and correcting encoding-level errors in the data, so nothing gets quietly ruined. 🫶
ngl, more advanced error correction is best handled at a layer below this document. the storage industry has this covered already (think Redundant Array of Independent Disks (RAID) and friends 💅). by specifying a flow storage format based on files, we get to borrow those features and tick this requirement off without doing extra work. ✨
that said, the ideal flow storage format will also be resilient against errors on its own, with an internal way to detect them and isolate them to as few data records as possible. main character energy 👑
this requirement also interacts with the choice of data compression or encryption algorithm. for example, block compression algorithms can keep an error contained to a single compression block. stream compressors, on the other hand, may fail to resynchronize after a single bit error, which invalidates the entire message stream. 💀
the IPFIX Message format does not support data integrity assurance. it's assumed that advanced error correction will be provided externally. compression and encryption, if used, give some allowance for detecting errors, if not correcting them. 🦋 and for simple error detection when there's no compression or encryption, checksums may be attached to messages via IPFIX Options, following the Message Checksum Options Template defined in Section 8.1.1. 🎀
5.6. Authentication, Confidentiality, and Integrity 🔐
archival storage of flow data may also need assurance that no unauthorized entity can read or modify the stored data. cryptography can handle this, providing integrity and confidentiality through signing and encryption. 💜
just like error correction, this problem has been solved well at a layer below this document. we can lean on the fact that existing cryptographic technologies work really well on data stored in files, so this requirement is covered too. 🌸
beyond support for using Transport Layer Security (TLS) over TCP, or Datagram Transport Layer Security (DTLS) over SCTP or UDP (both give transient authentication and confidentiality), the IPFIX protocol doesn't support this requirement directly. the IETF has specified the Cryptographic Message Syntax (CMS) [RFC3852] for creating detached signatures for integrity and authentication, and Section 9 specifies a CMS-based method for signing IPFIX Files. 🫧
confidentiality protection is assumed to be handled by methods outside this specification. you can use one of the many technologies for encrypting files to meet specific application and process needs. however, notes on improving the archival integrity of encrypted IPFIX Files are given in Section 9.2. 🔒✨
5.7. Anonymization and Obfuscation 🫥
to protect the privacy of individuals and organizations at the endpoints of the communications represented by flow records, it's often necessary to obfuscate or anonymize stored and exported flow data. 🙈 the ideal flow storage format will provide a notation saying that a given information element on a given record type represents anonymized data rather than real data. 🧚
the IPFIX protocol currently has no support for anonymization notation. anonymization was one of the requirements given for IPFIX in [RFC3917], and it's worth noting that. the requirement was qualified with 'MAY' and not 'MUST' in the requirements document, and it hasn't been specified in the current IPFIX protocol. why? because anonymization algorithms are still an open area of research, and there aren't any standardized methods for it yet. 🔬
no support is defined in [RFC5101] or in this IPFIX-based File format for anonymization, since anonymization notation is still open work for the IPFIX Working Group. 🌈
5.8. Session Auditability and Replayability 🕵️♀️
certain archival flow storage use cases need the storage of collection infrastructure details alongside the data itself. these details include information about how and when data was received, and where it was received from. they're useful for auditing, and also for replaying received data for testing purposes. 🍓
the IPFIX protocol doesn't have direct support for auditability and replayability. still, the IPFIX Information Model does define various Information Elements that are required to represent collection infrastructure details. these details may be stored in IPFIX Files using the Export Session Details Options Template defined in Section 8.1.3, and the Message Details Options Template defined in Section 8.1.4. 💖
5.9. Performance Characteristics 🏎️
the ideal standard flow storage format won't have a significant negative impact on the performance of the application generating or processing the flow data stored in it. this is a non-functional requirement, but it's important, because a standard that implies a big performance penalty is unlikely to be widely implemented and adopted. 😤
an examination of the IPFIX protocol suggests that implementations of it aren't particularly prone to slowness. honestly, a template-based data representation is easier to optimize for common cases than representations that embed structural information directly in the data stream (XML being the classic example). 🧠 however, a full analysis of how using IPFIX Messages as a basis for flow data storage affects read/write performance will need more implementation experience and performance measurement. 📊
6. Applicability 📍
this section describes the specific applicability of IPFIX Files to various use cases. IPFIX Files are especially useful in a flow collection and processing infrastructure that uses IPFIX for flow export. we explore the applicability and give guidelines for using IPFIX Files to store flow data collected by IPFIX Collecting Processes and NetFlow V9 collectors, to test IPFIX Collecting Processes, and to run diagnostics on IPFIX Devices. 🗺️
6.1. Storage of IPFIX-Collected Flow Data 💾
IPFIX Files can naturally be used to store flow data collected by an IPFIX Collecting Process. and honestly? this was one of the primary initial motivations behind the file format described in this document 🥹. using IPFIX Files this way gives you a single, standard, well-understood encoding for flow data on disk and on the wire. it also lets IPFIX implementations use substantially the same code for flow export and flow storage. 🙌
storing single Transport Sessions in IPFIX Files is especially important for network measurement research. it lets experiments be repeated, because it provides a format for storing and exchanging IPFIX flow trace data, much like the libpcap [pcap] format is used for experiments on packet trace data. 🧪✨
6.2. Storage of NetFlow-V9-Collected Flow Data 🌐
the IPFIX protocol is based on the Cisco NetFlow Services, Version 9 (NetFlow V9) protocol [RFC3954], but the two have diverged since work on IPFIX began. still, since the NetFlow V9 information model is a compatible subset of the IPFIX Information Model, you can use IPFIX Files to store collected NetFlow V9 flow data. 🦋 this can be really useful in multi-vendor, multi-protocol collection infrastructures that use both NetFlow V9 and IPFIX to export flow data. 💅
the applicability of IPFIX Files to this use case is outlined in Appendix B. 📖
6.3. Testing IPFIX Collecting Processes 🧪
IPFIX Files can be used to store IPFIX Messages for testing IPFIX Collecting Processes, and a variety of test cases can be stored in them. first, IPFIX data collected in real network environments and stored in an IPFIX File can be used as input to check the behavior of new or extended IPFIX Collector implementations. 🔍 IPFIX Files can also validate how a given IPFIX Collecting Process operates in a new environment. in other words, you can test it with recorded IPFIX data from the target network before installing the Collecting Process in that network. 🛡️
the IPFIX File format can also store artificial, non-compliant reference messages for specific Collecting Process test cases. examples include sets of IPFIX records with undefined Information Elements, Data Records described by missing Templates, or incorrectly framed Messages or Data Sets. 🙃 representative error handling test cases are defined in [RFC5471]. 🫠
fast replay of IPFIX Messages stored in a file can also be used for stress and load tests. think a high rate of incoming Data Records, or large Templates with high Information Element counts, as described in [RFC5471]. 🏋️♀️ provisioning and using a set of reference files for testing makes performance tests simpler and makes test results more comparable. 🎀
6.4. IPFIX Device Diagnostics 🩺
an IPFIX File can be used to store any collection of flows, so the format can also be used to dump and store various types of flow data for IPFIX Device diagnostics. for example, you might dump the open flow cache of a Metering Process, or the flow backlog of an Exporting or Collecting Process at the time of a process reset or crash. 💀📦 in those error-recovery situations, file-based storage is preferable to remote transmission. 🌸
7. Detailed File Format Specification 📜
any valid serialized IPFIX Message stream MUST be accepted by a File Reader as a valid IPFIX File. this way, the filesystem is simply treated as another IPFIX transport alongside SCTP, TCP, and UDP. it's a potentially high-latency transport, though, since the File Reader and File Writer don't necessarily run at the same time. ⏳✨
this section specifies the detailed actions of File Readers and File Writers when handling IPFIX Files. it also specifies the actions of File Writers in specific use cases. unless otherwise specified here, IPFIX File Writers MUST behave as IPFIX Exporting Processes, and IPFIX File Readers MUST behave as IPFIX Collecting Processes, where appropriate. 🫶💖
7.1. File Reader Specification ✨
okay so first things first, an IPFIX File Reader MUST act as an IPFIX Collecting Process as specified in [RFC5101], except as modified by this document 🎀
An IPFIX File Reader MUST accept as valid any serialized IPFIX Message stream that would be considered valid by one or more of the other defined IPFIX transport layers. Basically this means the union of IPFIX Template management features supported by SCTP, TCP, and UDP MUST be supported in IPFIX Files. File Readers MUST:
- accept IPFIX Messages containing Template Sets, Options Template Sets, and Data Sets within the same message, just like IPFIX over TCP or UDP 💅
- accept Template Sets that define Templates already defined within the File, which can happen when Templates get retransmitted with IPFIX over UDP as described in Section 10.3.6 of [RFC5101] 🔁
- resolve any conflict between a resent definition and a previous definition by assuming that the new Template replaces the old, which is consistent with Template expiration and ID reuse when using UDP at the IPFIX transport protocol 🦋
- accept Template Withdrawals as described in Section 8 of [RFC5101], provided that the Template to be withdrawn is defined, as is the case with IPFIX over TCP and SCTP 🫶
Okay now the filesystem-as-transport vibe 🧚. In the general case, an IPFIX File SHOULD be treated as containing a single Transport Session as defined by [RFC5101]. BUT some applications may benefit from treating a collection of IPFIX Files as one single Transport Session, so see especially Section 7.3.3 below 👀 A File Reader MAY be configurable to treat a collection of Files as a single Transport Session. However, a File Reader MUST NOT treat a single IPFIX File as containing multiple Transport Sessions. Never ever 🙅♀️
If an IPFIX File uses the technique described in [RFC5473] AND all of the non-Options Templates in the File contain the commonPropertiesId Information Element, a File Reader MAY assume the set of commonPropertiesId definitions provides a complete table of contents for the File for searching purposes. That's a whole mood for searching 🔍✨
7.2. File Writer Specification 🖊️
An IPFIX File Writer MUST act as an IPFIX Exporting Process as specified in [RFC5101], except as modified by this document. This section has the specs for IPFIX File Writers in all situations, and the specs and recommendations for specific File Writer use cases live in Section 7.3 below 🌸
File Writers SHOULD store the Templates and Options required to decode the data within the File itself, unless modified by the requirements of a specific use case in a subsection of Section 7.3. This way, a single IPFIX File generally contains a single notional Transport Session as defined by [RFC5101] 💜
File Writers SHOULD emit each Template Set or Options Template Set to appear in the File before any Data Set described by the Templates within that Set. The point is so the File Reader can decode every Data Set without waiting around to process later Templates or Options Templates ⏳
File Writers SHOULD emit Data Records described by Options Templates to appear in the File before any Data Records that depend on the scopes defined by those options 🧾
File Writers SHOULD use Template Withdrawals to withdraw Templates if Template IDs need to be reused. Template Withdrawals SHOULD NOT be used unless it is necessary to reuse Template IDs. So only reach for them when you actually need to 🙅♀️🍓
File Writers SHOULD write IPFIX Messages within an IPFIX File in ascending Export Time order 📈
File Writers MAY write Data Records to an IPFIX File in any order. However, File Writers that write flow records to an IPFIX File in flowStartTime or flowEndTime order SHOULD be consistent in this ordering within each File. Consistency is the whole vibe 🎀
7.3. Specific File Writer Use Cases 🧁
The specifications in this section apply to specific situations. Each section below extends or modifies the base File Writer specification in Section 7.2. There's stuff about collocating a File Writer with IPFIX Collecting Processes and Metering Processes, plus specific guidelines for using IPFIX Files for archival storage or as documents. It also covers using IPFIX Files in the testing and diagnostics of IPFIX Devices 🔬
7.3.1. Collocating a File Writer with a Collecting Process
When collocating a File Writer with an IPFIX Collecting Process for archival storage of collected data in IPFIX Files as described in Section 6.1, the following recommendations may improve the usefulness of the stored data 📦
The simplest way for a File Writer to store the data collected in a single Transport Session is to just write the incoming IPFIX Messages to an IPFIX File as they are collected. Lowkey this has a few drawbacks 😬 First, if the original Exporting Process did not conform to the recommendations in Section 7.2 with respect to Template and Data Record ordering, the written file can be difficult to use later. In this case, File Writers MAY reorder records as received in order to ensure that Templates appear before the Data Records they describe 🔄
A File Writer collocated with a Collecting Process that starts writing data from a running Transport Session SHOULD write all the Templates currently active within that Transport Session before writing any Data Records described by them 🏁
Also, the resulting IPFIX Files will lack information about the IPFIX Transport Session used to export them, such as the network addresses of the Exporting and Collecting Processes and the protocols used to transport them. In this case, if information about the Transport Session is required, the File Writer SHOULD store a single IPFIX Transport Session in an IPFIX File and SHOULD record information about the Transport Session using the Export Session Details Options Template described in Section 8.1.3 🗂️
Additional per-Message information MAY be recorded by the File Writer using the Message Details Options Template described in Section 8.1.4. Per-Message information includes the time at which each IPFIX Message was received at the Collecting Process, and it can be used to resend IPFIX Messages while keeping the original measurement plane traffic profile 🕰️✨
When collocating a File Writer with a Collecting Process, the Export Time of each Message SHOULD be the Export Time of the Message received by the Collecting Process containing the first Data Record in the Message. Also, File Writers storing IPFIX data collected from an IPFIX Collecting Process using SCTP as the transport protocol SHOULD interleave messages from multiple streams in order to preserve Export Time order. They SHOULD also reorder the written messages as necessary so that each Template Set or Options Template Set appears in the File before any Data Set described by the Templates within that Set. Template reordering MUST preserve the sequence of Template Sets with Template Withdrawals in order to ensure consistency of Templates. Ordering is serious business, bestie 🧩
Note that when adding additional information to IPFIX Messages received from Collecting Processes (for example, Message Checksum Options or Message Detail Options), the File Writer SHOULD extend the length of the Message for the additional data if possible. Otherwise, the Message SHOULD be split into two approximately equal-size Messages aligned on a Data Set or Template Set boundary from the original Message if possible. If that's not possible either, the Message SHOULD be split into two approximately equal-size Messages aligned on a Data Record boundary. Note that the Maximum Segment Size (MSS) or MTU of most network links (1500-9000 for common Ethernets) is smaller than the maximum IPFIX Message size (65536) within an IPFIX File, so it is expected that message length extension will suffice in most circumstances 📏💖
A File Writer collocated with a Collecting Process SHOULD NOT sign a File as specified in Section 9.1 unless the Transport Session over which the data was exported was protected via TLS or DTLS, and the Collecting Process positively identified the Exporting Process by its certificate. See Section 12.2 for more information on this issue. Signing is not a casual flex 🔐🚨
7.3.2. Collocating a File Writer with a Metering Process
Note that File Writers may also be collocated directly with IPFIX Metering Processes, for writing measured information straight to disk without intermediate IPFIX Exporting or Collecting Processes. This setup may be particularly useful when providing data to an analysis environment with an IPFIX-File-based workflow, when testing Metering Processes during development, or when the authentication of a Metering Process is important 🧪
When collocating a File Writer with a Metering Process, note that Information Elements associated with Exporting or Collecting Processes are meaningless, and they SHOULD NOT appear in the Export Session Details Options Template described in Section 8.1.3 or the Message Details Options Template described in Section 8.1.4. Not them 🙅♀️
When collocating a File Writer with a Metering Process, the Export Time of each Message SHOULD be the time at which the first Data Record in the Message was received from the Metering Process ⏱️
Note that collocating a File Writer with a Metering Process is the only way to provide positive authentication of a Metering Process through signatures as in Section 9.1. See Section 12.2 for more information on this issue. Main character energy for the Metering Process 👑
7.3.3. Using IPFIX Files for Archival Storage
While in the general case File Writers should store one Transport Session per IPFIX File, some applications storing large collections of data over long periods of time may benefit from the ability to treat a collection of IPFIX Files as a single Transport Session. A File Writer MAY be configurable to write data from a single Transport Session into multiple IPFIX Files. However, File Writers supporting such a configuration option MUST provide a configuration option to support one-file-per-session behavior for interoperability purposes. Interop is non-negotiable, periodt 🤝🌈
File Writers using IPFIX Files for archival storage SHOULD support compression as in Section 10. Store it smaller, slay 🗜️💅
7.3.4. Using IPFIX Files as Documents ✨
okay so when IPFIX Files are used as documents 📄 (like storing a set of flows for a query, an investigation, or some other shared context, or publishing traffic datasets for network research), each File MUST be readable as a single Transport Session 💖. It has to be self-contained, aside from any detached signature from Section 9.1, and it can't point at metadata stored in separate Files. That's how we keep things interoperable, periodt 💅
When File Writers make Files for documents, they MAY emit the special Data Records described by Options Templates before any other Data Records in the File. They go in this order so File Readers can inspect and use the document easily 🧚
- Time Window records described by the File Time Window Options Template as defined in Section 8.1.2 below;
- Information Element Type Records as described in [RFC5610];
- commonPropertiesId definitions as described in [RFC5473];
- Export Session details records described by the Export Session Details Options Template as defined in Section 8.1.3 below.
The Export Time of each Message within a File used as a document SHOULD be the time at which the File Writer wrote the Message ⏰
If an IPFIX File used as a document uses the technique described in [RFC5473] AND all of the non-Options Templates in the File contain the commonPropertiesId Information Element, a File Reader MAY assume the set of commonPropertiesId definitions provides a complete table of contents for the File for searching purposes. Main character energy 🎀
7.3.5. Using IPFIX Files for Testing 🧪
IPFIX Files can be used for testing IPFIX Collecting Processes in two ways. First, they can store specific flow data for regression and stress testing of Collectors. There are no special considerations for IPFIX Files used this way, so that's the easy one 🌸
Second, IPFIX Files are useful for storing reference messages that do not comply to the IPFIX protocol, which lets you test the error handling and recovery behavior of Collectors. Of course, IPFIX Files intended for this application necessarily MAY violate any of the specifications in this document or in [RFC5101], and such Files MUST NOT be transmitted to Collecting Processes or given as input to File Readers not under test. Do NOT mix these up with your real data 🙈
Note that an extremely simple IPFIX Exporting Process can be crafted for testing by reading an IPFIX File and transmitting it directly to a Collecting Process. Similarly, an extremely simple Collecting Process can be crafted for testing by accepting connections and/or IPFIX Messages from Exporting Processes and writing the session's message stream to an IPFIX File. So cute and so simple 🦋
7.3.6. Writing IPFIX Files for Device Diagnostics 🔧
IPFIX Files can help debug devices that use flow data as internal state, since they work as a common format for representing flow tables. In these situations, the opaqueOctets information element can store additional non-IPFIX encoded, non-flow information (e.g., stack backtraces, process state, etc.) inside the IPFIX File, as in Section 11.1. The IPFIX flow table information can also be embedded in a larger proprietary diagnostic format using delimiters, as in Section 11.2 💜
7.3.7. IPFIX File Manipulation 🛠️
For many applications, it may prove useful for implementations to provide functionality for manipulating IPFIX Files. For example, they might select data from a File, change the Templates used within a File, or split or join data in Files. Any such utility should take special care to make sure its output remains a valid IPFIX File, especially with respect to Templates and Options, which are scoped to Transport Sessions. Not the vibe of a messy output 🚫
Any operation that splits one File into multiple Files SHOULD write all necessary Templates and Options to each resulting File, and make sure the written Options are valid for each resulting File (e.g., the Time Window Options Template in Section 8.1.2). Any operation that joins multiple Files into a single File should do the same, and also make sure Template IDs do not collide, through the use of different Observation Domain IDs or Template ID rewriting. Combining operations may also want to keep any desired ordering of flow records. Ordering matters, bestie 🧩
7.4. Media Type of IPFIX Files 📁
The media type for IPFIX Files is application/ipfix. The registration information [RFC4288] for this media type is given in the IANA Considerations section 💅
8. File Format Metadata Specification 🎀
This section defines the Options Templates used for IPFIX File metadata, along with the Information Elements they require 🫶
8.1. Recommended Options Templates for IPFIX Files 💖
The following Options Templates let IPFIX Message streams meet the requirements outlined above, without any extension to the message format or protocol. They are defined in terms of existing Information Elements from [RFC5102], the Information Elements from [RFC5610], and the Information Elements defined in Section 8.2. IPFIX File Readers and Writers SHOULD support these Options Templates as defined below 🌈
In addition, IPFIX File Readers and Writers SHOULD support the Options Templates defined in [RFC5610] so they can self-describe enterprise-specific Information Elements ✨
8.1.1. Message Checksum Options Template 🧮
The Message Checksum Options Template specifies the structure of a Data Record for attaching an MD5 message checksum to an IPFIX Message. An MD5 message checksum as described MAY be used if data integrity is important to the application but file signing is not available or desired. The described Data Record MUST appear only once per IPFIX Message, but MAY appear anywhere within the Message 🔐
This Options Template SHOULD contain the following Information Elements:
+--------------------+----------------------------------------------+
| IE | Description |
+--------------------+----------------------------------------------+
| messageScope | A marker denoting this Option applies to the |
| [scope] | whole IPFIX Message; content is ignored. |
| | This Information Element MUST be defined as |
| | a Scope Field. |
| messageMD5Checksum | The MD5 checksum of the containing IPFIX |
| | Message. |
+--------------------+----------------------------------------------+8.1.2. File Time Window Options Template 🕰️
The File Time Window Options Template specifies the structure of a Data Record for attaching a time window to an IPFIX File. This Data Record is called a time window record. A time window record defines the earliest flow start time and the latest flow end time of the flow records within a File. One and only one time window record MAY appear within an IPFIX File if the time window information is available. A File Writer MUST NOT write more than one time window record to an IPFIX File. A File Writer that writes a time window record to a File MUST NOT write any Flow with a start time before the beginning of the window or an end time after the end of the window to that File 🪟
This Options Template SHOULD contain the following Information Elements:
+---------------+---------------------------------------------------+
| IE | Description |
+---------------+---------------------------------------------------+
| sessionScope | A marker denoting this Option applies to the |
| [scope] | whole IPFIX Transport Session (i.e., the IPFIX |
| | File in the common case); content is ignored. |
| | This Information Element MUST be defined as a |
| | Scope Field. |
| minFlowStart* | Exactly one of minFlowStartSeconds, |
| | minFlowStartMilliseconds, |
| | minFlowStartMicroseconds, or |
| | minFlowStartNanoseconds SHOULD match the |
| | precision of the accompanying maxFlowEnd* |
| | Information Element. The start time of the |
| | earliest flow in the Transport Session (i.e., |
| | File). |
| maxFlowEnd* | Exactly one of maxFlowEndSeconds, |
| | maxFlowEndMilliseconds, maxFlowEndMicroseconds, |
| | or maxFlowEndNanoseconds SHOULD match the |
| | precision of the accompanying minFlowStart* |
| | Information Element. The end time of the latest |
| | flow in the Transport Session (i.e., File). |
+---------------+---------------------------------------------------+8.1.3. Export Session Details Options Template ✨
Okay bestie, this is the Export Session Details Options Template 🎀. It's the blueprint for a Data Record that describes one whole IPFIX Transport Session 🦋. The whole point is to store one complete Transport Session inside one IPFIX File 📁. Since it's meant to be the single source of truth for that session, the Data Record SHOULD appear only once in a given IPFIX File 💅. Lowkey iconic ngl.
This Options Template SHOULD contain at least the following Information Elements 🌸, and each one comes with its own applicability rules, so read the table carefully 🙈:
+----------------------------+--------------------------------------+
| IE | Description |
+----------------------------+--------------------------------------+
| sessionScope [scope] | A marker denoting this Option |
| | applies to the whole IPFIX Transport |
| | Session (i.e., the IPFIX File in the |
| | common case); content is ignored. |
| | This Information Element MUST be |
| | defined as a Scope Field. |
| exporterIPv4Address | IPv4 address of the IPFIX Exporting |
| | Process from which the Messages in |
| | this Transport Session were |
| | received. Present only for |
| | Exporting Processes with an IPv4 |
| | interface. For multi-homed SCTP |
| | associations, this SHOULD be the |
| | primary path endpoint address of the |
| | Exporting Process. |
| exporterIPv6Address | IPv6 address of the IPFIX Exporting |
| | Process from which the Messages in |
| | this Transport Session were |
| | received. Present only for |
| | Exporting Processes with an IPv6 |
| | interface. For multi-homed SCTP |
| | associations, this SHOULD be the |
| | primary path endpoint address of the |
| | Exporting Process. |
| exporterTransportPort | The source port from which the |
| | Messages in this Transport Session |
| | were received. |
| exporterCertificate | The certificate used by the IPFIX |
| | Exporting Process from which the |
| | Messages in this Transport Session |
| | were received. Present only for |
| | Transport Sessions protected by TLS |
| | or DTLS. |
| collectorIPv4Address | IPv4 address of the IPFIX Collecting |
| | Process that received the Messages |
| | in this Transport Session. Present |
| | only for Collecting Processes with |
| | an IPv4 interface. For multi-homed |
| | SCTP associations, this SHOULD be |
| | the primary path endpoint address of |
| | the Collecting Process. |
| collectorIPv6Address | IPv6 address of the IPFIX Collecting |
| | Process that received the Messages |
| | in this Transport Session. Present |
| | only for Collecting Processes with |
| | an IPv6 interface. For multi-homed |
| | SCTP associations, this SHOULD be |
| | the primary path endpoint address of |
| | the Collecting Process. |
| collectorTransportPort | The destination port on which the |
| | Messages in this Transport Session |
| | were received. |
| collectorTransportProtocol | The IP Protocol Identifier of the |
| | transport protocol used to transport |
| | Messages within this Transport |
| | Session. |
| collectorProtocolVersion | The version of the export protocol |
| | used to transport Messages within |
| | this Transport Session. Applicable |
| | only in mixed NetFlow V9-IPFIX |
| | collection environments when storing |
| | NetFlow V9 data in IPFIX Messages, |
| | as in Appendix B. |
| collectorCertificate | The certificate used by the IPFIX |
| | Collecting Process that received the |
| | Messages in this Transport Session. |
| | Present only for Transport Sessions |
| | protected by TLS or DTLS. |
| minExportSeconds | The Export Time of the first Message |
| | in the Transport Session. |
| maxExportSeconds | The Export Time of the last Message |
| | in the Transport Session. |
+----------------------------+--------------------------------------+Quick vibe check on the key words: the sessionScope IE MUST be defined as a Scope Field 👑, and the multi-homed SCTP primary path addresses SHOULD be used for the IPv4 and IPv6 address fields 💜. Everything else is either present only in certain situations or just there for context 🧚.
8.1.4. Message Details Options Template ✨
okay so this one is the Message Details Options Template 🎀 and its whole job is to describe a Data Record that gets attached to an IPFIX Message 🦋 to record the extra export details: when the message was received, and info about the export and collection infrastructure that moved it along 🚚. bestie, it also lets you store the export session metadata, as long as you're using the Export Session Details Options Template too, so you can keep info from multiple Transport Sessions in the same IPFIX File 📁✨
this Options Template SHOULD contain at least the following Information Elements 💅, subject to applicability for each one (the table tells you when each one applies). and ngl, there's a special case that's giving main character energy 👑: when you're pairing this with the Export Session Details Options Template and storing one complete IPFIX Transport Session in an IPFIX File, this Options Template SHOULD contain only the messageScope and collectionTimeMilliseconds Information Elements 🌸, plus exportSctpStreamId for Messages sent over SCTP 🧚
+----------------------------+--------------------------------------+
| IE | Description |
+----------------------------+--------------------------------------+
| messageScope [scope] | A marker denoting this Option |
| | applies to the whole IPFIX message; |
| | content is ignored. This |
| | Information Element MUST be defined |
| | as a Scope Field. |
| collectionTimeMilliseconds | The absolute time at which this |
| | Message was received by the IPFIX |
| | Collecting Process. |
| exporterIPv4Address | IPv4 address of the IPFIX Exporting |
| | Process from which this Message was |
| | received. Present only for |
| | Exporting Processes with an IPv4 |
| | interface, and if this information |
| | is not available via the Export |
| | Session Details Options Template. |
| | For multi-homed SCTP associations, |
| | this SHOULD be the primary path |
| | endpoint address of the Exporting |
| | Process. |
| exporterIPv6Address | IPv6 address of the IPFIX Exporting |
| | Process from which this Message was |
| | received. Present only for |
| | Exporting Processes with an IPv6 |
| | interface and if this information is |
| | not available via the Export Session |
| | Details Options Template. For |
| | multi-homed SCTP associations, this |
| | SHOULD be the primary path endpoint |
| | address of the Exporting Process. |
| exporterTransportPort | The source port from which this |
| | Message was received. Present only |
| | if this information is not available |
| | via the Export Session Details |
| | Options Template. |
| exporterCertificate | The certificate used by the IPFIX |
| | Exporting Process from which this |
| | Message was received. Present only |
| | for Transport Sessions protected by |
| | TLS or DTLS. |
| collectorIPv4Address | IPv4 address of the IPFIX Collecting |
| | Process that received this Message. |
| | Present only for Collecting |
| | Processes with an IPv4 interface, |
| | and if this information is not |
| | available via the Export Session |
| | Details Options Template. For |
| | multi-homed SCTP associations, this |
| | SHOULD be the primary path endpoint |
| | address of the Collecting Process. |
| collectorIPv6Address | IPv6 address of the IPFIX Collecting |
| | Process that received this Message. |
| | Present only for Collecting |
| | Processes with an IPv6 interface, |
| | and if this information is not |
| | available via the Export Session |
| | Details Options Template. For |
| | multi-homed SCTP associations, this |
| | SHOULD be the primary path endpoint |
| | address of the Collecting Process. |
| collectorTransportPort | The destination port on which this |
| | Message was received. Present only |
| | if this information is not available |
| | via the Export Session Details |
| | Options Template. |
| collectorTransportProtocol | The IP Protocol Identifier of the |
| | transport protocol used to transport |
| | this Message. Present only if this |
| | information is not available via the |
| | Export Session Details Options |
| | Template. |
| collectorProtocolVersion | The version of the export protocol |
| | used to transport this Message. |
| | Present only if necessary and if |
| | this information is not available |
| | via the Export Session Details |
| | Options Template. |
| collectorCertificate | The certificate used by the IPFIX |
| | Collecting Process that received |
| | this Message. Present only for |
| | Transport Sessions protected by TLS |
| | or DTLS. |
| exportSctpStreamId | The SCTP stream used to transport |
| | this Message. Present only if the |
| | Message was transported via SCTP. |
+----------------------------+--------------------------------------+8.2. Recommended Information Elements for IPFIX Files 🌈
okay so these are the Information Elements that Section 8.1 uses 🎯 so IPFIX Message streams can meet the requirements from above without extending the protocol 💖. IPFIX File Readers and Writers SHOULD support these, as defined below 🫶
also, IPFIX File Readers and Writers SHOULD support the Information Elements defined in [RFC5610] 📚 so the file can be fully self-describing, which is literally so iconic 😭✨
8.2.1. collectionTimeMilliseconds 🕰️
Description: the absolute timestamp at which the data within the scope containing this Information Element was received by a Collecting Process 💅. this IE SHOULD be bound to its containing IPFIX Message via IPFIX Options and the messageScope Information Element, as defined below 🎀
Abstract Data Type: dateTimeMilliseconds ElementId: 258 Status: current
8.2.2. collectorCertificate 🔐
Description: the full X.509 certificate, encoded in ASN.1 DER format, used by the Collector when IPFIX Messages were transmitted using TLS or DTLS 🔒. this IE SHOULD be bound to its containing IPFIX Transport Session via an options record and the sessionScope Information Element, or to its containing IPFIX Message via an options record and the messageScope Information Element 🦋
Abstract Data Type: octetArray ElementId: 274 Status: current
8.2.3. exporterCertificate 🔐
Description: the full X.509 certificate, encoded in ASN.1 DER format, used by the Collector when IPFIX Messages were transmitted using TLS or DTLS 🌸. this IE SHOULD be bound to its containing IPFIX Transport Session via an options record and the sessionScope Information Element, or to its containing IPFIX Message via an options record and the messageScope Information Element 💜
Abstract Data Type: octetArray ElementId: 275 Status: current
8.2.4. exportSctpStreamId 🧚
Description: the value of the SCTP Stream Identifier used by the Exporting Process for exporting IPFIX Message data 🍓. this is carried in the Stream Identifier field of the header of the SCTP DATA chunk containing the IPFIX Message(s) ✨
Abstract Data Type: unsigned16 Data Type Semantics: identifier ElementId: 259 Status: current
8.2.5. maxExportSeconds ⏱️
Description: the absolute Export Time of the latest IPFIX Message within the scope containing this Information Element 💫. this IE SHOULD be bound to its containing IPFIX Transport Session via IPFIX Options and the sessionScope Information Element 🎀
Abstract Data Type: dateTimeSeconds ElementId: 260 Status: current Units: seconds
8.2.6. maxFlowEndMicroseconds 🌟
Description: the latest absolute timestamp of the last packet within any Flow within the scope containing this Information Element, rounded up to the microsecond if necessary 🦄. this IE SHOULD be bound to its containing IPFIX Transport Session via IPFIX Options and the sessionScope Information Element. this IE SHOULD be used only in Transport Sessions containing Flow Records with microsecond-precision (or better) timestamp Information Elements 🙈
Abstract Data Type: dateTimeMicroseconds ElementId: 268 Status: current Units: microseconds
8.2.7. maxFlowEndMilliseconds ⭐
Description: the latest absolute timestamp of the last packet within any Flow within the scope containing this Information Element, rounded up to the millisecond if necessary 🌈. this IE SHOULD be bound to its containing IPFIX Transport Session via IPFIX Options and the sessionScope Information Element. this IE SHOULD be used only in Transport Sessions containing Flow Records with millisecond-precision (or better) timestamp Information Elements 💅
Abstract Data Type: dateTimeMilliseconds ElementId: 269 Status: current Units: milliseconds
8.2.8. maxFlowEndNanoseconds 🌈
Description: okay so this one is the latest absolute timestamp of the last packet in any Flow within the scope that contains this Information Element ⏰. This Information Element SHOULD be bound to its containing IPFIX Transport Session via IPFIX Options and the sessionScope Information Element. It SHOULD be used only in Transport Sessions containing Flow Records with nanosecond-precision timestamp Information Elements 💅
Abstract Data Type: dateTimeNanoseconds
ElementId: 270
Status: current
Units: nanoseconds
8.2.9. maxFlowEndSeconds 🎀
Description: this is the latest absolute timestamp of the last packet in any Flow within the scope containing this Information Element, rounded up to the second if necessary 🫶. This Information Element SHOULD be bound to its containing IPFIX Transport Session via IPFIX Options and the sessionScope Information Element.
Abstract Data Type: dateTimeSeconds
ElementId: 261
Status: current
Units: seconds
8.2.10. messageMD5Checksum 🔥
Description: this one is the MD5 checksum of the IPFIX Message containing this record 🧾. This Information Element SHOULD be bound to its containing IPFIX Message via an options record and the messageScope Information Element, as defined below, and SHOULD appear only once in a given IPFIX Message. Here's how you calculate it, bestie: first buffer the containing IPFIX Message, setting the value of this Information Element to all zeroes. Then calculate the MD5 checksum of the resulting buffer as defined in [RFC1321], place the resulting value in this Information Element, and export the buffered message 💖.
This Information Element is intended as a simple checksum only. So collision resistance and algorithm agility are not required, and MD5 is an appropriate message digest 🌸.
Abstract Data Type: octetArray (16 bytes)
ElementId: 262
Status: current
Reference: RFC 1321, The MD5 Message-Digest Algorithm [RFC1321]
8.2.11. messageScope 🦋
Description: when this Information Element shows up as scope in an Options Template, it means the options described by the Template apply to the IPFIX Message that contains them 📬. It's defined for general purpose message scoping of options, and it was proposed specifically so checksum and collection information can be attached to a message via IPFIX Options ✨. The value of this Information Element MUST be written as 0 by the File Writer or Exporting Process. The value of this Information Element MUST be ignored by the File Reader or the Collecting Process 🙈.
Abstract Data Type: unsigned8
ElementId: 263
Status: current
8.2.12. minExportSeconds 🍓
Description: this is the absolute Export Time of the earliest IPFIX Message within the scope containing this Information Element ⏳. This Information Element SHOULD be bound to its containing IPFIX Transport Session via an options record and the sessionScope Information Element.
Abstract Data Type: dateTimeSeconds
ElementId: 264
Status: current
Units: seconds
8.2.13. minFlowStartMicroseconds 🧚
Description: the earliest absolute timestamp of the first packet within any Flow within the scope containing this Information Element, rounded down to the microsecond if necessary 🪄. This Information Element SHOULD be bound to its containing IPFIX Transport Session via an options record and the sessionScope Information Element. It SHOULD be used only in Transport Sessions containing Flow Records with microsecond-precision (or better) timestamp Information Elements 💜.
Abstract Data Type: dateTimeMicroseconds
ElementId: 271
Status: current
Units: microseconds
8.2.14. minFlowStartMilliseconds 🌷
Description: the earliest absolute timestamp of the first packet within any Flow within the scope containing this Information Element, rounded down to the millisecond if necessary 🕐. This Information Element SHOULD be bound to its containing IPFIX Transport Session via an options record and the sessionScope Information Element. It SHOULD be used only in Transport Sessions containing Flow Records with millisecond-precision (or better) timestamp Information Elements 🫧.
Abstract Data Type: dateTimeMilliseconds
ElementId: 272
Status: current
Units: milliseconds
8.2.15. minFlowStartNanoseconds 💫
Description: the earliest absolute timestamp of the first packet within any Flow within the scope containing this Information Element 🌙. This Information Element SHOULD be bound to its containing IPFIX Transport Session via an options record and the sessionScope Information Element. It SHOULD be used only in Transport Sessions containing Flow Records with nanosecond-precision timestamp Information Elements 🩷.
Abstract Data Type: dateTimeNanoseconds
ElementId: 273
Status: current
Units: nanoseconds
8.2.16. minFlowStartSeconds 👑
Description: the earliest absolute timestamp of the first packet within any Flow within the scope containing this Information Element, rounded down to the second if necessary ⭐. This Information Element SHOULD be bound to its containing IPFIX Transport Session via an options record and the sessionScope Information Element.
Abstract Data Type: dateTimeSeconds
ElementId: 265
Status: current
Units: seconds
8.2.17. opaqueOctets 🍄
Description: this Information Element is used to wrap non-IPFIX data inside an IPFIX Message stream 🎁. The point is to let a non-IPFIX data processor store a data stream inline within an IPFIX File. A Collecting Process or File Writer MUST NOT try to interpret this binary data 🚫. It's different from paddingOctets because its contents are meaningful in some non-IPFIX context, while the contents of paddingOctets MUST be 0x00 and are only there for Information Element alignment.
Abstract Data Type: octetArray
ElementId: 266
Status: current
8.2.18. sessionScope 🦢
Description: when this Information Element shows up as scope in an Options Template, the options described by the Template apply to the IPFIX Transport Session that contains them 💌. Here's the tea: all options are already implicitly scoped to Transport Session and Observation Domain, so this Information Element is equivalent to a "null" scope 🫠. It's defined for general purpose session scoping of options, and it was proposed specifically so time window and collection information can be attached to an IPFIX File via IPFIX Options. The value of this Information Element MUST be written as 0 by the File Writer or Exporting Process. The value of this Information Element MUST be ignored by the File Reader or the Collecting Process 💖.
Abstract Data Type: unsigned8
ElementId: 267
Status: current
9. Signing and Encryption of IPFIX Files 🔐
In order to keep IPFIX Files intact and prove who the IPFIX File Writers are, File Writers and File Readers SHOULD provide an interoperable and easily implemented method for signing IPFIX Files and verifying those signatures 🫶. This section lays out the method via CMS detached signatures.
Note that while CMS specifies an encapsulation format that can be used for encryption as well as signing, no method is specified for encapsulation for confidentiality protection. It's assumed that application-specific or process-specific requirements outweigh the need for interoperability for encrypted files 🙈.
9.1. CMS Detached Signatures 🦋
The Cryptographic Message Syntax (CMS) [RFC3852] defines an encapsulation syntax for data protection. It's used to digitally sign, authenticate, or encrypt arbitrary message content 📝. CMS can also create detached signatures, where the signature is stored in a separate file. This arrangement maximizes interoperability, because File Readers that aren't aware of CMS detached signatures and have no requirement for them can just ignore them. The content of the IPFIX File itself is unchanged by the signature 🌈.
The detached signature file for an IPFIX File SHOULD be stored, transported, or otherwise made available (e.g., by FTP or HTTP) alongside the signed IPFIX File. It uses the same filename as the IPFIX File, except the file extension ".p7s" is added to the end, conforming to the naming convention in [RFC3851] 🎀.
Within the detached signature, the CMS ContentInfo type MUST always be present. It MUST encapsulate the CMS SignedData content type, which in turn MUST NOT encapsulate the signed IPFIX File content 🚫. The CMS detached signature is summarized as follows:
ContentInfo {
contentType id-signedData, -- (1.2.840.113549.1.7.2)
content SignedData
}
SignedData {
version CMSVersion, -- Always set to 3
digestAlgorithms DigestAlgorithmIdentifiers,
encapContentInfo EncapsulatedContentInfo,
certificates CertificateSet, -- File Writer certificate(s)
crls CertificateRevocationLists, -- Optional
signerInfos SET OF SignerInfo -- Only one signer
}
SignerInfo {
version CMSVersion, -- Always set to 3
sid SignerIdentifier,
digestAlgorithm DigestAlgorithmIdentifier,
signedAttrs SignedAttributes,
signatureAlgorithm SignatureAlgorithmIdentifier,
signature SignatureValue,
unsignedAttrs UnsignedAttributes
}
EncapsulatedContentInfo {
eContentType id-data, -- (1.2.840.113549.1.7.1)
eContent OCTET STRING -- Always absent
}The details of the contents of each CMS encapsulation are in the subsections below 💅.
9.1.1. ContentInfo 💜
[RFC3852] requires the outer-most encapsulation to be ContentInfo. The fields of ContentInfo are as follows:
- contentType: the type of the associated content. For the detached signature file, the encapsulated type is always SignedData, so the id-signedData (1.2.840.113549.1.7.2) object identifier MUST be present in this field ✨.
- content: a SignedData content, detailed in Section 9.1.2 🫧.
9.1.2. SignedData ✨
Okay so SignedData is the content type that holds the signature of the IPFIX File, plus everything a validator needs to check it. Here are its fields, bestie 💅
version: MUST be 3. 🎀digestAlgorithms: a collection of one-way hash function identifiers. It MUST contain the identifier used by the File Writer to generate the digital signature. 🔥encapContentInfo: the signed content, including a content type identifier. Since a detached signature is being created, it does not encapsulate the IPFIX File. The EncapsulatedContentInfo is detailed in Section 9.1.4. 🦋certificates: a collection of certificates. It SHOULD include the X.509 certificate needed to validate the digital signature file. Certification Authority (CA) and File Writer certificates MUST conform to the certificate profile specified in [RFC5280]. 💜crls: an optional collection of certificate revocation lists (CRLs). It SHOULD NOT contain any CRLs; any CRLs that are present MUST conform to the certificate profile specified in [RFC5280]. 🌸signerInfos: a collection of per-signer information; this identifies the File Writer. More than one SignerInfo MAY appear to facilitate transitions between keys or algorithms. The SignerInfo type is detailed in Section 9.1.3. ✨
9.1.3. SignerInfo
The SignerInfo type is the one that identifies the File Writer, and its fields are as follows 🧚
version: MUST be 3.sid: identifies the File Writer's public key. This identifier MUST match the value included in thesubjectKeyIdentifiercertificate extension on the File Writer's X.509 certificate. 🔑digestAlgorithm: identifies the one-way hash function and associated parameters used to generate the signature.signedAttrs: an optional set of attributes that are signed along with the content.digestAlgorithm: identifies the digital signature algorithm and associated parameters used to generate the signature.signature: the digital signature of the associated file.unsignedAttrs: an optional set of attributes that are not signed.
9.1.4. EncapsulatedContentInfo
The EncapsulatedContentInfo structure contains a content type identifier. Since a detached signature is being created, it does not encapsulate the IPFIX File. Its fields are as follows 🍓
eContentType: an object identifier that uniquely specifies the content type. The content type associated with IPFIX File MUST beid-data(1.2.840.113549.1.7.1). 🎀eContent: an optional field containing the signed content. Since this is a detached signature,eContentMUST be absent. 🙈
9.2. Encryption Error Resilience
Heads up, bestie: single bit errors in the encrypted data stream can result in larger errors in the decrypted stream, depending on the encryption scheme used. 😭
In applications (e.g., archival storage) where error resilience is very important, File Writers SHOULD use an encryption scheme that can resynchronize after bit errors. A common example is a block cipher in CBC (Cipher Block Chaining) mode. In this case, File Writers MAY also use the Message Checksum Options Template to attach a checksum to each IPFIX Message in the IPFIX File, so that errors in the decrypted data can be recognized. 🛡️💖
10. Compression of IPFIX Files 🗜️
Network traffic measurement data is generally highly compressible, and it's giving main character energy 👑 IPFIX Templates tend to pack more information into each record by not requiring the export of irrelevant or non-present fields. The technique described in [RFC5473] also cuts down on the export of redundant information. Even with all of that, generalized compression can decrease storage requirements significantly, so IPFIX File Writers and File Readers SHOULD support compression as described in this section. ✨
10.1. Supported Compression Formats
IPFIX Files support two compression encapsulation formats: bzip2 [bzip2] and gzip [RFC1952]. bzip2 gives better compression than gzip and, as a block compression algorithm, better error recovery characteristics, but it's slower at compressing. gzip is potentially the better choice when compression time is an issue. These two algorithms and encapsulation formats were chosen for ubiquity and ease of implementation, which is so practical 🧚
IPFIX File Readers and Writers supporting compression MUST support bzip2, and SHOULD support gzip. 💅
10.2. Compression Recognition at the File Reader
bzip2, gzip, and uncompressed IPFIX Files each have distinct magic numbers. IPFIX File Readers SHOULD use these magic numbers to figure out what compression, if any, is in use for an IPFIX File, and then invoke the proper decompression. 🔍
- bzip2 files are identified by the initial three-octet string
0x42, 0x5A, 0x68("BZh"). - gzip files are identified by the initial two-octet string
0x1F, 0x8B. - IPFIX Files are identified by the initial two-octet string
0x00, 0x0A. These are the version bytes of the first IPFIX Message header in the File. 🦋
10.3. Compression Error Resilience
Compression at the file level, like encryption, is not particularly resilient to errors. In the worst case, a single bit error in a stream-compressed file could result in the loss of the entire file. 😱
Since block compression algorithms that support the identification and isolation of blocks containing errors limit the impact of errors on the recoverability of compressed data, the use of bzip2 in applications where error resilience is important is RECOMMENDED. 🌈
Since the block boundary of a block-compressed IPFIX File may fall in the middle of an IPFIX Message, resynchronizing an IPFIX Message stream after a compression error needs some care. The beginning of an IPFIX Message can be identified by its header signature (the Version field of the Message Header, 0x00 0x0A, followed by a 16-bit Message Length). But simply searching for the first occurrence of the Version field is insufficient, since these two bytes may occur in valid IPFIX Template or Data Sets. 🙃
Therefore, we specify the following algorithm for File Readers to resynchronize an IPFIX Message Stream after skipping a compressed block containing errors: 🧠
- Search after the error for the first occurrence of the octet string
0x00, 0x0A(the IPFIX Message Header Version field). - Treat this field as the beginning of a candidate IPFIX Message. Read the two bytes following the Version field as a Message Length, and seek to that offset from the beginning of the candidate IPFIX Message.
- If the first two octets after the candidate IPFIX Message are
0x00, 0x0A(i.e., the IPFIX Message Header Version field of the next message in the stream), or if the end-of-file is reached precisely at the end of the candidate IPFIX Message, presume that the candidate IPFIX Message is valid, and begin reading the IPFIX File from the start of the candidate IPFIX Message. - If not, or if the seek reaches end-of-file or another block containing errors before finding the end of the candidate message, go back to step 1, starting the search two bytes from the start of the candidate IPFIX Message.
⚠️ The algorithm above will improperly identify a non-message as a message approximately 1 in 2^32 times, assuming random IPFIX data. It may be expanded to consider multiple candidate IPFIX Messages in order to increase reliability.
In applications (e.g., archival storage) where error resilience is very important, File Writers SHOULD use block compression algorithms, and MAY attempt to align IPFIX Messages within compression blocks to ease resynchronization after errors. File Readers SHOULD use the resynchronization algorithm above to minimize data loss due to compression errors. 💾
11. Recommended File Integration Strategies 🧩
This section describes methods for integrating IPFIX File data with other file formats. 🎀
11.1. Encapsulation of Non-IPFIX Data in IPFIX Files
At times, it may be useful to export or store non-IPFIX data inline in an IPFIX File or Message stream. To do this cleanly, this data MUST be encapsulated into IPFIX Messages so that an IPFIX File Reader or Collecting Process can handle it without any need to interpret it. At the same time, this data MUST NOT be changed during transmission or storage. The opaqueOctets Information Element, as defined in Section 8.2.17, is provided for this encapsulation. 📦
Processing the encapsulated non-IPFIX data is left to a separate processing mechanism that can identify encapsulated non-IPFIX data in an IPFIX Message Stream. That mechanism doesn't need any other IPFIX handling capability, except the ability to skip over all IPFIX Messages that do not encapsulate non-IPFIX data. 🙌
The Message Checksum Options Template, described in Section 8.1.1, may be used as a uniform mechanism to identify errors within encapsulated data. 🔎
Note that this mechanism can only encapsulate data objects up to 65,515 octets in length. If the space available in one IPFIX Message is not enough for the amount of data to be encapsulated, then the data must be broken into smaller segments that are encapsulated into consecutive IPFIX Messages. Any additional structuring or semantics of the raw data is outside the scope of IPFIX and must be implemented within the encapsulated binary data itself. Furthermore, an IPFIX File Reader cannot assume that the raw encapsulated data has any specific format. 🚧
11.2. Encapsulation of IPFIX Files within Other File Formats
Consequently, it may also be useful to reverse the encapsulation. That is, to export or store IPFIX data inline within a non-IPFIX File or data stream. This makes sense when the other file format is not compatible with the encapsulation described above in Section 11.1. Generally speaking, the encapsulation here will be specific to the format of the containing file. For example, IPFIX Files may be embedded in XML elements using hex or Base64 encoding, or in raw binary files using start and end delimiters or some form of run-length encoding. 🌀
As there are as many potential encapsulations here as there are potential file formats, the specifics of each are out of scope for this specification. 🫶
12. Security Considerations 🔐
The Security Considerations section of [RFC5101], on which the IPFIX File format is based, is largely concerned with the proper application of TLS and DTLS to ensure confidentiality and integrity when exporting IPFIX Messages. By analogy, this document specifies the use of CMS [RFC3852] detached signatures to provide equivalent integrity protection to TLS and DTLS in Section 9.1. 🛡️ However, aside from merely applying CMS for signatures, there are several security issues that must be considered in certain circumstances. These are covered in the subsections below. 🚨
12.1. Relationship between IPFIX File and Transport Encryption 🔐
okay so the protocol that moves the info you're storing in this format has to be just as careful about keeping it safe. integrity and confidentiality are non-negotiable, bestie. 🫶 [RFC5101] covers those details. The key bit: IPFIX Files holding data from an IPFIX Collecting Process that used TLS or DTLS for transport security SHOULD be signed as in Section 9.1 and SHOULD be encrypted out of band. Storing that flow data unencrypted may be a potential breach of confidentiality, so yikes 😭. Flip it around, and if the data is sensitive enough to need encryption at rest and you later send it over IPFIX, it SHOULD be sent using TLS or DTLS for transport security 💜.
12.2. End-to-End Assertions for IPFIX Files 🕵️♀️
ngl this one is a bit of a plot twist. TLS and CMS can both sign an IPFIX Transport Session or an IPFIX File, but there is no method for protecting data integrity end-to-end when a Collecting Process is collocated with a File Writer. 💀 The Exporting Process to Collecting Process channel is signed by the Exporting Process key and protected via TLS and DTLS, while the File is signed by the File Writer key and protected via CMS. The identity of the Exporting Process isn't asserted in the file, and the records may be modified between the Collecting Process and the File Writer. Main character drama 🎭.
There are two potential ways to handle this. The first is by fiat, and it's only appropriate when the application lets the Collecting-Process-to-File-Writer channel be trusted 🌸. In that case, the File Writer's signature is an implicit assertion that the channel to the Exporting Process was protected, that the Exporting Process's signature was verified, and that the data wasn't changed after collection. For this to work, a File Writer collocated with a Collecting Process SHOULD NOT sign a File as specified in Section 9.1 unless the Transport Session over which the data was exported was protected via TLS or DTLS, and the Collecting Process positively identified the Exporting Process by its certificate. The File Writer SHOULD include the Exporting Process and Collecting Process certificates within the File using the Export Session Detail Options Template in Section 8.1.3 or the Message Detail Options Template in Section 8.1.4 so they can be verified later 🧚.
If the Collecting Process and/or File Writer can't be trusted, you can get close to end-to-end integrity by collocating the File Writer with the Metering Process and removing the IPFIX protocol completely from the chain 🦋. Here, the File Writer's signature is an implicit assertion that the Metering Process is identified and isn't tampering with the information as observed on the wire.
Verifying these trust relationships is out of scope for this document and should be considered on a per-implementation basis 🎀.
12.3. Recommendations for Strength of Cryptography for IPFIX Files 🔑
so when you encrypt files for archival storage, the cryptographic strength depends on how long the archived data is expected to be kept 🍓. Long-term storage may need cryptographic protection to be reapplied, so you periodically resign and reencrypt files with stronger keys. In that case, it is recommended that the existing signed and/or encypted data be encapsulated within newer, stronger protection. See [RFC4810] for a discussion of this issue 📚.
13. IANA Considerations 🌈
this document specifies the creation of several new IPFIX Information Elements in the IPFIX Information Element registry located at http://www.iana.org, as defined in Section 8.2 above. IANA has assigned the following Information Element numbers for their respective Information Elements:
258 collectionTimeMilliseconds
274 collectorCertificate
275 exporterCertificate
259 exportSctpStreamId
260 maxExportSeconds
268 maxFlowEndMicroseconds
269 maxFlowEndMilliseconds
270 maxFlowEndNanoseconds
261 maxFlowEndSeconds
262 messageMD5Checksum
263 messageScope
264 minExportSeconds
271 minFlowStartMicroseconds
272 minFlowStartMilliseconds
273 minFlowStartNanoseconds
265 minFlowStartSeconds
266 opaqueOctets
267 sessionScopeIANA has created the media type application/ipfix for IPFIX data, registered with the following information 💖:
Type name: application
Subtype name: ipfix
Required parameters: none
Optional parameters: none
Encoding considerations: IPFIX Files are binary, and therefore must
be encoded in non-binary contexts.
Security considerations: See the Security Considerations
(Section 12) of RFC 5655, and the Security Considerations of
[RFC5101].
Interoperability considerations: See the "Detailed Specification"
(Section 7) of RFC 5655. The format is designed to be broadly
interoperable, as any valid stream of IPFIX Messages over any
transport specified in [RFC5101] MUST be recognizable as a valid
IPFIX File.
Published specification: RFC 5655, especially Section 7, and
[RFC5101].
Applications that use this media type: Various IPFIX
implementations (see [RFC5153]) support the construction of IPFIX
File Readers and Writers.
Additional information:
Magic number(s): None, although the first two bytes of any IPFIX
File are the first two bytes of a message header, the Version
field, which as of [RFC5101] are always 10 in network byte
order: 0x00, 0x0A.
File extension(s): .ipfix
Macintosh file type code(s): none
Person & email address to contact for further information: Brian
Trammell <brian.trammell@hitachi-eu.com> for the authors of RFC
5655; Nevil Brownlee <n.brownlee@auckland.ac.nz> for the IPFIX
Working Group.
Intended usage: LIMITED USE
Restrictions on usage: none
Change controller: Brian Trammell <brian.trammell@hitachi-eu.com>
for the authors of RFC 5655; Nevil Brownlee
<n.brownlee@auckland.ac.nz> for the IPFIX Working Group.14. Acknowledgements 🫶
thank you to Maurizio Molina, Tom Kosnar, and Andreas Kind for technical help with the requirements for a standard flow storage format ✨. Thanks to Benoit Claise, Paul Aitken, Andrew Johnson, Gerhard Muenz, and Nevil Brownlee for their reviews and feedback 💅. Thanks to Pasi Eronen for pointing out [RFC5485], and to Russ Housley for writing it. It specifies a detached signature format, and Section 9.1 is largely drawn from it 🌟. Thanks also to the PRISM project for supporting this work 🌸.
15. References 📚
15.1. Normative References
[RFC5101] Claise, B., "Specification of the IP Flow Information
Export (IPFIX) Protocol for the Exchange of IP Traffic
Flow Information", RFC 5101, January 2008.
[RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
Meyer, "Information Model for IP Flow Information
Export", RFC 5102, January 2008.
[RFC5610] Boschi, E., Trammell, B., Mark, L., and T. Zseby,
"Exporting Type Information for IP Flow Information
Export (IPFIX) Information Elements", RFC 5610,
July 2009.
[RFC1321] Rivest, R., "The MD5 Message-Digest Algorithm",
RFC 1321, April 1992.
[RFC1952] Deutsch, P., Gailly, J-L., Adler, M., Deutsch, L., and
G. Randers-Pehrson, "GZIP file format specification
version 4.3", RFC 1952, May 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3852] Housley, R., "Cryptographic Message Syntax (CMS)",
RFC 3852, July 2004.
[RFC4810] Wallace, C., Pordesch, U., and R. Brandner, "Long-Term
Archive Service Requirements", RFC 4810, March 2007.
[RFC5280] Cooper, D., Santesson, S., Farrell, S., Boeyen, S.,
Housley, R., and W. Polk, "Internet X.509 Public Key
Infrastructure Certificate and Certificate Revocation
List (CRL) Profile", RFC 5280, May 2008.
[bzip2] Seward, J., "bzip2 (http://www.bzip.org/)", March 2008.15.2. Informative References
[RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander,
"Requirements for IP Flow Information Export (IPFIX)",
RFC 3917, October 2004.
[RFC3954] Claise, B., "Cisco Systems NetFlow Services Export
Version 9", RFC 3954, October 2004.
[RFC5153] Boschi, E., Mark, L., Quittek, J., Stiemerling, M., and
P. Aitken, "IP Flow Information Export (IPFIX)
Implementation Guidelines", RFC 5153, April 2008.
[RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek,
"Architecture for IP Flow Information Export", RFC 5470,
March 2009.
[RFC5471] Schmoll, C., Aitken, P., and B. Claise, "Guidelines for
IP Flow Information Export (IPFIX) Testing", RFC 5471,
March 2009.
[RFC5472] Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP
Flow Information Export (IPFIX) Applicability",
RFC 5472, March 2009.
[RFC5473] Boschi, E., Mark, L., and B. Claise, "Reducing
Redundancy in IP Flow Information Export (IPFIX) and
Packet Sampling (PSAMP) Reports", RFC 5473, March 2009.
[SAINT2007] Trammell, B., Boschi, E., Mark, L., and T. Zseby,
"Requirements for a standardized flow storage solution",
in Proceedings of the SAINT 2007 workshop on Internet
Measurement Technology, Hiroshima, Japan, January 2007.
[RFC3851] Ramsdell, B., "Secure/Multipurpose Internet Mail
Extensions (S/MIME) Version 3.1 Message Specification",
RFC 3851, July 2004.
[RFC4288] Freed, N. and J. Klensin, "Media Type Specifications and
Registration Procedures", BCP 13, RFC 4288,
December 2005.
[RFC5485] Housley, R., "Digital Signatures on Internet-Draft
Documents", RFC 5485, March 2009.
[pcap] "libpcap (http://www.tcpdump.org/)", October 2008.Appendix A. Example IPFIX File 🎀
okay so this appendix is my absolute fave because we get to walk through a whole example IPFIX File, like a real one, and see every feature in action 🦋 this File holds flow records that all follow a single Template, and it also has a File Time Window record that says when the data starts and ends ⏰ plus an Export Session Details record that spills the tea on the collection infrastructure 🕵️♀️ and here's the bestie part: every single Message has a Message Checksum record too, because the File might get encrypted from the outside and/or stored as an archive 🔐 literally so careful, periodt 💅 the whole layout is laid out in Figure 2 below ✨
+=================================================+
| IPFIX Message seq. 0 |
| +---------------------------------------------+ |
| | Template Set (ID 2) 1 rec | |
| | Data Tmpl. ID 256 | |
| +---------------------------------------------+ |
| | Options Template Set (ID 3) 3 recs | |
| | File Time Window Opt. Tmpl. ID 257 | |
| | Message Checksum Opt. Tmpl. ID 259 | |
| | Export Session Details Opt. Tmpl. ID 258 | |
| +---------------------------------------------+ |
| | Data Set (ID 259) [Message Checksum] 1 rec | |
| +---------------------------------------------+ |
+=================================================+
| IPFIX Message seq. 1 |
| +---------------------------------------------+ |
| | Data Set (ID 257) [File Time Window] 1 rec | |
| +---------------------------------------------+ |
| | Data Set (ID 258) [Export Session] 1 rec | |
| +---------------------------------------------+ |
| | Data Set (ID 259) [Message Checksum] 1 rec | |
| +---------------------------------------------+ |
+=================================================+
| IPFIX Message seq. 4 |
| +---------------------------------------------+ |
| | Data Set (ID 256) 50 recs | |
| | contains flow data | |
| +---------------------------------------------+ |
| | Data Set (ID 259) [Message Checksum] 1 rec | |
| +---------------------------------------------+ |
+=================================================+
| IPFIX Message seq. 55 |
| . . . |
Figure 2: File Example Structureokay the Template for the data records is giving main character energy 🌸 it has a flow start timestamp, an IPv4 5-tuple, and packet and octet total counts 📦 the Template Set that defines all of that is in Figure 3:
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 40 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 256 | Field Count = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| flowStartSeconds = 150 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceIPv4Address = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| dest.IPv4Address = 12 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceTransportPort = 7 | Field Length = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| dest.TransportPort = 11 | Field Length = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| protocolIdentifier = 4 | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| octetTotalCount = 85 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| packetTotalCount = 86 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: File Example Data TemplateA.1. Example Options Templates
so next up is an Options Template Set holding every Options Template you need to read the File 📖 that means the File Time Window Options Template (from Section 8.1.2 above), the Export Session Details Options Template (from Section 8.1.3 above), and the Message Checksum Options Template (from Section 8.1.1 above) 🧩 this Options Template Set shows up in Figure 4 and Figure 5 below ✨
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 80 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 257 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field Count = 1 |0| sessionScope = 267 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 1 |0| minFlowStartSeconds = 265 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| maxFlowEndSeconds = 261 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 | Template ID = 259 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Count = 2 | Scope Field Count = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| messageScope = 263 | Field Length = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| messageMD5Checksum = 262 | Field Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4: File Example Options Templates (Time Window and Checksum) 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 258 | Field Count = 9 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Scope Field Count = 1 |0| sessionScope = 267 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 1 |0| exporterIPv4Address = 130 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| collectorIPv4Address = 211 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| exporterTransportPort = 217 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 |0| col.TransportPort = 216 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 2 |0| col.TransportProtocol = 215 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 1 |0| col.ProtocolVersion = 214 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 1 |0| minExportSeconds = 264 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 |0| maxExportSeconds = 260 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Field Length = 4 | set padding (2 octets) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5: File Example Options Templates, Continued (Session Details)A.2. Example Supplemental Options Data
after the Templates come the supplemental IPFIX Options records, which describe what's inside the File and what type of data it is 🍓 first up is the File Time Window record: it says the File holds data from 9 October 2007, between 00:01:13 and 23:56:27 UTC 🕐 it looks like Figure 6:
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 257 | Length = 13 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sessionScope | minFlowStartSeconds
| 0 | 2007-10-09 00:01:13 UTC . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| maxFlowEndSeconds
. . . | 2007-10-09 23:56:27 UTC . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
. . . |
+-+-+-+-+-+-+-+-+
Figure 6: File Example Time Windownext is the Export Session Details record, which is where the collection drama lives 🕵️♀️ this one says the session in the File was sent over SCTP from an Exporter at 192.0.2.30 port 32769 to a Collector at 192.0.2.40 port 4739 🦋 it holds messages that were exported between 00:01:57 and 23:57:12 UTC on 9 October 2007 ⭐ and in its Data Set it's represented like Figure 7:
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 258 | Length = 27 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sessionScope | exporterIPv4Address
| 0 | 192.0.2.30 . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| collectorIPv4Address
. . . | 192.0.2.31 . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| exporterTransportPort | cTPort
. . . | 32769 | 4739 . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| cTProtocol | cPVersion |
. . . | 132 | 10 | . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
minExportSeconds |
. . . 2007-10-09 00:01:57 UTC | . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
maxExportSeconds |
. . . 2007-10-09 23:57:12 UTC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7: File Example Export Session DetailsA.3. Example Message Checksum
Okay so every IPFIX Message inside the File gets wrapped up with its own Message Checksum record 🎀 Its layout inside its Data Set is in Figure 8 below ✨
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 259 | Length = 24 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| messageScope | |
| 0 | |
+-+-+-+-+-+-+-+-+ |
| messageMD5Checksum |
| (16-byte MD5 checksum of options message) |
| |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | set padding (3 octets) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 8: File Example Message Checksum
A.4. File Example Data Set
After the Templates and supplemental Options information comes the actual data, the main event 💅 The first record of an example Data Set is shown with its message and set headers in Figure 9:
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version = 10 | Length = 1296 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Export Time = 2007-10-09 00:01:57 UTC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Observation Domain ID = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 256 | Length = 1254 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flowStartSeconds |
| 2007-10-09 00:01:13 UTC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceIPv4Address |
| 192.0.2.2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| destinationIPv4Address |
| 192.0.2.3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceTransportPort | destinationTransportPort |
| 32770 | 80 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| protocolId | totalOctetCount
| 6 | 18000 . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| totalPacketCount
. . . | 65 . . .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (49 more records)
. . . |
+-+-+-+-+-+-+-+-+Figure 9: File Example Data Set
A.5. Complete File Example
Okay, time to bring it all together 🦋 Adding the message headers where they belong, a hex dump of the first 317 bytes of the example File we built up above looks like Figure 10 below, with the annotations included.
0:|00 0A 00 A0 47 0A B6 E5 00 00 00 00 00 00 00 01
[^ first message header (length 160 bytes) -->
16:|00 02 00 28 01 00 00 08 00 96 00 04 00 08 00 04
[^ data template set -->
32: 00 0C 00 04 00 07 00 02 00 0B 00 02 00 04 00 01
48: 00 55 00 04 00 56 00 04|00 03 00 50 01 01 00 03
[^ opt template set -->
64: 00 01 01 0B 00 01 01 09 00 04 01 05 00 04 01 03
80: 00 02 00 01 01 07 00 01 01 06 00 10 01 02 00 09
96: 00 01 01 0B 00 01 00 82 00 04 00 D3 00 04 00 D9
112: 00 02 00 D8 00 02 00 D7 00 01 00 D0 00 01 01 08
128: 00 04 01 04 00 04 00 00|01 03 00 18 00 73 F1 12
[^ checksum record -->
144: D6 C7 58 BE 44 E6 60 06 4E 78 74 AE 7D 00 00 00
176:|00 0A 00 50 47 0A B6 E5 00 00 00 01 00 00 00 01
[^ second message header (length 80 bytes) -->
192:|01 01 00 0E 00 47 0A B6 B9 47 0C 07 1B 00|01 02
[^ time window rec -> [ session detail rec ^ -->
208: 00 1C 00 C0 00 02 1E 0C 00 02 1F 80 01 12 83 84
224: 0A 47 0A B6 E5 47 0C 07 48 00|01 03 00 18 00 3E
[ message checksum record ^ -->
240: 2B 37 08 CE B2 0E 30 11 32 12 4A 5F E3 AD DB 00
256:|00 0A 05 10 47 0A B6 E5 00 00 00 06 00 00 00 01
[^ third message header (length 1296 bytes) -->
272:|01 00 04 E6|47 0A B6 B9 C0 00 02 02 C0 00 02 03
[^ set hdr ][^ first data rec -->
288: 80 02 00 50 06 00 00 46 50 00 00 00 41Figure 10: File Example Hex Dump
Appendix B. Applicability of IPFIX Files to NetFlow V9 Flow Storage ✨
Here's the tea 🍓 The IPFIX Message format is nearly a superset of the NetFlow V9 packet format, so IPFIX Files can be used to store NetFlow V9 data pretty easily. This appendix describes a method for doing so. The differences between the two protocols are outlined in Appendix B.1 below. A simple, lightweight, message-for-message translation method for transforming V9 Packets into IPFIX Messages for storage within IPFIX Files is described in Appendix B.2. An example of this translation method is given in Appendix B.3.
B.1. Comparing NetFlow V9 to IPFIX
With a few caveats, the IPFIX protocol is a superset of the NetFlow V9 protocol 🌈 It evolved from NetFlow V9 mostly by adding features, bringing it into line with the IPFIX Requirements and the needs of stakeholders in the IPFIX Working Group. This appendix walks through the differences between the two protocols. It is informative only, and it's presented as an exploration of the two protocols to motivate using IPFIX Files to store V9-collected flow data.
B.1.1. Message Header Format
Both NetFlow V9 and IPFIX use streams of messages, each prefixed by a message header, though the message header differs quite a lot between the two 👑 Note that in NetFlow V9 terminology these messages are called packets, and messages must be delimited by datagram boundaries. IPFIX does not have this constraint. The header formats are detailed below:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version Number | Count |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sysUpTime |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UNIX Secs |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 11: NetFlow V9 Packet Header Format
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version Number | Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Export Time |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Observation Domain ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 12: IPFIX Message Header Format
Here's how the fields line up, bestie 🫶
- Version Number: The IPFIX Version Number MUST be 10, while the NetFlow V9 Version Number MUST be 9.
- Length vs. Count: The Count field in the NetFlow V9 packet header counts records in the message (including Data and Template Records), while the Length field in the IPFIX Message Header counts octets in the message. Note that this implies that NetFlow V9 collectors must rely on datagram boundaries or some other external delimiter; otherwise, they must completely consume a message before finding its end. 😭
- System Uptime: System uptime in milliseconds is exported in the NetFlow V9 packet header. This field is not present in the IPFIX Message Header, and must be exported using an IPFIX Option if required.
- Export Time: Aside from being called UNIX Secs in the NetFlow V9 packet header specification, the export time in seconds since 1 January 1970 at 0000 UTC appears in both NetFlow V9 and IPFIX message headers. ⭐
- Sequence Number: The NetFlow V9 Sequence Number counts packets, while the IPFIX Sequence Number counts records in Data Sets. Both are scoped to Observation Domain.
- Observation Domain ID: Similarly, the NetFlow V9 sourceID has become the IPFIX Observation Domain ID. 🧚
B.1.2. Set Header Format
Set headers are identical between NetFlow V9 and IPFIX. Each Set (FlowSet in NetFlow V9 terminology) is prefixed by a 4-byte set header containing the Set ID and the length of the set in octets. Easy win 💖
Note that the special Set IDs are different between IPFIX and NetFlow V9. IPFIX Template Sets are identified by Set ID 2, while NetFlow V9 Template FlowSets are identified by Set ID 0. Similarly, IPFIX Options Template Sets are identified by Set ID 3, while NetFlow V9 Options Template FlowSets are identified by Set ID 1.
Both protocols reserve Set IDs 0-255, and use Set IDs 256-65535 for Data Sets (or FlowSets, in NetFlow V9 terminology). 🌸
B.1.3. Template Format
Template FlowSets in NetFlow V9 support a subset of the functionality found in IPFIX templates 🥺 Specifically, NetFlow V9 has no support for vendor-specific Information Elements the way IPFIX does, so there is no enterprise bit or facility for associating a private enterprise number with an information element. NetFlow V9 also does not support variable-length fields.
Options Template FlowSets in NetFlow V9 are similar to Options Template Sets in IPFIX, subject to the same caveats. 💅
B.1.4. Information Model
The NetFlow V9 field type definitions are a compatible subset of the IPFIX Information Model, and they have evolved in concert with it 🔥 IPFIX Information Element identifiers in the range 1-127 are defined by the IPFIX Information Model [RFC5102] to be compatible with the corresponding NetFlow V9 field types.
B.1.5. Template Management ✨
okay so NetFlow V9 has no concept of a Transport Session like IPFIX does 🫶 because NetFlow V9 was designed with a connectionless transport in mind 🦋. That means Template IDs are scoped to an Exporting Process lifetime (i.e., an Exporting Process instance between restarts) 💅. There is no facility in NetFlow V9, unlike IPFIX, for Template withdrawal or Template ID reuse 💀. Template retransmission at the Exporter works the same way as in UDP-based IPFIX Exporting Processes ✨.
B.1.6. Transport 🎀
In practice, even though NetFlow V9 is designed to be transport-independent, it is only ever transported over UDP 🌸. There is no facility like the one in IPFIX for full connection-oriented transport without datagram boundaries 🥺, and that's because NetFlow V9 uses a record count field instead of a message length field in the packet header 🧚. There is also no support in NetFlow V9 for transport layer security via TLS or DTLS 😭.
B.2. A Method for Transforming NetFlow V9 Messages to IPFIX 🦋
this appendix is giving us a method for transforming NetFlow V9 Packets into IPFIX Messages 💖, which is how you can store NetFlow V9 data in IPFIX Files 📁. A process that transforms NetFlow V9 Packets into IPFIX Messages must handle the fact that NetFlow V9 Packets and IPFIX Messages are framed differently 🖼️, that sequence numbering works differently 🔢, and that the NetFlow V9 field type definitions are only compatible with the IPFIX Information Model below Information Element identifier 128 🙈. main character energy, fr fr 👑.
For each incoming NetFlow V9 packet, the transformation process must:
- Verify that the Version field of the packet header is 9.
- Verify that the Sequence Number field of the packet header is valid.
- Scan the packet to:
- Verify that it contains no Templates with field types outside the range 1-127;
- Verify that it contains no FlowSets with Set IDs between 2 and 255 inclusive;
- Verify that it contains the number of records in FlowSets, Template FlowSets, and Options Template FlowSets declared in the Count field of the packet header; and
- Count the number of records in Data FlowSets for calculating the IPFIX Sequence Number.
- Calculate a Sequence Number for each IPFIX Observation Domain by storing the last Sequence Number sent for each Observation Domain plus the count of records in Data FlowSets in the previous step, to be sent as the Sequence Number for the IPFIX Message following this one within that Observation Domain 🧮.
- Generate a new IPFIX Message Header with:
- a Version field of 10;
- a Length field with the number of octets in the IPFIX Message, generally available by subtracting 4 from the length of the NetFlow V9 packet as returned from the transport layer (accounting for the difference in message header lengths);
- the Sequence Number calculated for this message by the Sequence Number calculation step; and
- Export Time and Observation Domain ID taken from the UNIX secs and Source ID fields of the NetFlow V9 packet header, respectively.
- Copy each FlowSet from the Netflow V9 packet to the IPFIX Message after the header. Replace Set ID 0 with Set ID 2 for Template Sets, and Set ID 1 with Set ID 3 for Options Template Sets 🔁.
⚠️ Note that this process loses system uptime information 🥲. If such information is required, the transformation process will have to export that information using IPFIX Options. This may require a more sophisticated transformation process structure 💅.
B.3. NetFlow V9 Transformation Example 🍓
the next two figures show a single NetFlow V9 packet with templates and the corresponding IPFIX Message 🎁. It's exporting a single flow record representing 60,303 octets sent from 192.0.2.2 to 192.0.2.3 💌. This would be the third packet exported in Observation Domain 33 from the NetFlow V9 exporter, containing records starting with the 12th record (packet and record sequence numbers count from 0) 🧁.
The ** symbol in the IPFIX example shows those fields that required modification from the NetFlow V9 packet by the transformation process 👀.
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version = 9 | Count = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Uptime = 3750405 ms (1:02:30.405) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Export Time = 1171557627 epoch sec (2007-02-15 16:40:27) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number = 2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Observation Domain ID = 33 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 0 | Set Length = 20 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 256 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPV4_SRC_ADDR = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPV4_DST_ADDR = 12 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IN_BYTES = 1 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 256 | Set Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPV4_SRC_ADDR |
| 192.0.2.2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPV4_DST_ADDR |
| 192.0.2.3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IN_BYTES |
| 60303 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 13: Example NetFlow V9 Packet 📸
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ** Version = 10 | ** Length = 52 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Export Time = 1171557627 epoch sec (2007-02-15 16:40:27) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ** Sequence Number = 11 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Observation Domain ID = 33 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ** Set ID = 2 | Set Length = 20 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 256 | Field Count = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| sourceIPv4Address = 8 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| destinationIPv4Address = 12 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| octetDeltaCount = 1 | Field Length = 4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 256 | Set Length = 16 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| sourceIPv4Address |
| 192.0.2.2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| destinationIPv4Address |
| 192.0.2.3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| octetDeltaCount |
| 60303 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 14: Corresponding Example IPFIX Message 🦋
Authors' Addresses 💜
Brian Trammell
Hitachi Europe
c/o ETH Zurich
Gloriastrasse 35
8092 Zurich
Switzerland
Phone: +41 44 632 70 13
EMail: brian.trammell@hitachi-eu.com
Elisa Boschi
Hitachi Europe
c/o ETH Zurich
Gloriastrasse 35
8092 Zurich
Switzerland
Phone: +41 44 632 70 57
EMail: elisa.boschi@hitachi-eu.com
Lutz Mark
Fraunhofer IFAM
Wiener Str. 12
28359 Bremen
Germany
Phone: +49 421 2246206
EMail: lutz.mark@ifam.fraunhofer.de
Tanja Zseby
Fraunhofer Institute for Open Communication Systems
Kaiserin-Augusta-Allee 31
10589 Berlin
Germany
Phone: +49 30 3463 7153
EMail: tanja.zseby@fokus.fraunhofer.de
Arno Wagner
ETH Zurich
Gloriastrasse 35
8092 Zurich
Switzerland
Phone: +41 44 632 70 04
EMail: arno@wagner.namethat's the whole RFC, bestie 🌈✨🫶
📦 where this page comes from
- ietf: rfc5655 2009-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms