RFC 5473

Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports

πŸ“… published March 2009, from ietf

Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports πŸ¦‹

Network Working Group, Informational, March 2009. ✨

Authors: E. Boschi (Hitachi Europe), L. Mark (Fraunhofer IFAM), and B. Claise (Cisco Systems, Inc.) πŸ’œ

The title block doesn't list any documents this one obsoletes or updates. πŸ€·β€β™€οΈ


Status of This Memo πŸ“œ

okay so this memo is just giving information to the Internet community πŸ’… It is NOT a standard, not even a little bit. Anyone can read it and it can be shared freely with no limits 🌸

Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved πŸŽ€

This document is covered by BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, as they stood on the publication date. The IETF Trust's license info lives at http://trustee.ietf.org/license-info. Read those carefully, because they spell out your rights and restrictions for this doc 🫢

Some of the material in here came from IETF Documents or IETF Contributions that were published or made publicly available before November 10, 2008. The people who own the copyright on some of that older material may not have given the IETF Trust permission to allow changes to it outside the IETF Standards Process. Unless you get a proper license from those copyright holders, this document may not be modified outside the IETF Standards Process, and you may not make derivative works of it outside that process. The one exceptions are formatting it for publication as an RFC and translating it into languages other than English 🌈

Abstract πŸ“

This document describes a bandwidth-saving method for exporting Flow or packet information using the IP Flow Information eXport (IPFIX) protocol. Since the Packet Sampling (PSAMP) protocol is built on IPFIX, everything here applies to PSAMP exports too πŸ’–

The trick is splitting information that's shared by several Flow Records away from the information that belongs to just one Flow Record. The common Flow information gets exported only once, in a Data Record defined by an Options Template. Then the rest of each Flow's specific information links back to that common info through a unique identifier ✨


Table of Contents πŸ“š

   1. Introduction ....................................................3
      1.1. IPFIX Documents Overview ...................................3
      1.2. PSAMP Documents Overview ...................................4
   2. Terminology .....................................................4
      2.1. Terminology Summary Table ..................................5
      2.2. IPFIX Flows versus PSAMP Packets ...........................5
   3. Specifications for Bandwidth-Saving Information Export ..........5
      3.1. Problem Statement and High-Level Solution ..................6
      3.2. Data Reduction Technique ...................................7
   4. Transport Protocol Choice .......................................8
      4.1. PR-SCTP ....................................................9
      4.2. UDP ........................................................9
      4.3. TCP ........................................................9
   5. commonPropertiesID Management ...................................9
   6. The Collecting Process Side ....................................10
      6.1. UDP .......................................................11
      6.2. TCP .......................................................12
   7. Advanced Techniques ............................................12
      7.1. Multiple Data Reduction ...................................12
      7.2. Cascading Common Properties ...............................15
   8. Export and Evaluation Considerations ...........................15
      8.1. Transport Protocol Choice .................................16
      8.2. Reduced Size Encoding .....................................16
      8.3. Efficiency Gain ...........................................16
   9. Security Considerations ........................................17
   10. Acknowledgments ...............................................17
   11. References ....................................................17
      11.1. Normative References .....................................17
      11.2. Informative References ...................................18
   Appendix A. Examples ..............................................19
      A.1. Per-Flow Data Reduction ...................................19
      A.2. Per-Packet Data Reduction .................................23
      A.3. Common Properties Withdrawal Message ......................26

1. Introduction ✨

The IPFIX working group designed a protocol for exporting IP Flow information [RFC5101]. It's built to export info about IP traffic Flows and related measurement data πŸ“Š A Flow is defined by a set of key attributes, like source and destination IP address, source and destination port, and so on.

The really cool part is the template mechanism. Thanks to it, IPFIX can export any kind of information, as long as the Information Element is specified in the IPFIX information model [RFC5101] or registered with IANA 🧚

Here's the tea πŸ«– Flow Records with common properties export the same fields in every single Data Record, no matter what the field contents are. Those common properties can be values shared by a bunch of Flows or packets, or values that stay the same over time. Just so we're clear, common properties are NOT the list of Flow Keys that define a Flow. They can still contain some of the Flow Keys though πŸ’… Cutting all that repeated data out of the export stream can save a huge amount of transferred data 🀯

This document is about exporting those invariant or common properties just once. The Flow-specific properties go out in regular Data Records as usual. Unique common properties identifiers are what link the Data Records back to the common attributes πŸ”—

This method works for IPFIX Flow information and for PSAMP per-packet information. It doesn't require any changes to the IPFIX or PSAMP protocol specifications, which is lowkey the best part πŸ™Œ

1.1. IPFIX Documents Overview πŸ“–

The IPFIX protocol [RFC5101] gives network admins access to IP Flow information. The architecture for exporting measured IP Flow information from an IPFIX exporting process to a collecting process is defined in the IPFIX Architecture [RFC5470], following the requirements in RFC 3917 [RFC3917]. The IPFIX Architecture [RFC5470] spells out how IPFIX Data Records and templates get carried from IPFIX exporting processes to IPFIX collecting processes over a congestion-aware transport protocol πŸš€

IPFIX also has a formal description of its Information Elements, covering their names, types, and extra semantic info. That lives in the IPFIX information model [RFC5102] 🧠 Last but not least, the IPFIX applicability statement [RFC5472] explains what kinds of applications can use the IPFIX protocol and how they can use the info it provides. It also shows how the IPFIX framework fits in with other architectures and frameworks 🌸

1.2. PSAMP Documents Overview 🌈

The document "A Framework for Packet Selection and Reporting" [RFC5474] describes the PSAMP framework. It's how network elements pick subsets of packets using statistical and other methods, then export a stream of reports about the selected packets to a collector πŸ“¦

The packet selection techniques PSAMP supports (sampling, filtering, and hashing) are covered in "Sampling and Filtering Techniques for IP Packet Selection" [RFC5475] 🎯 The PSAMP protocol [RFC5476] specifies how packet info gets exported from a PSAMP exporting process to a PSAMP collecting process πŸ’Œ

Just like IPFIX, PSAMP has a formal description of its Information Elements, their names, types, and additional semantic information. The PSAMP information model is defined in [RFC5477] πŸ€“ And finally, [PSAMP-MIB] describes the PSAMP Management Information Base 🧁

2. Terminology πŸŽ€

IPFIX-specific terminology used here is defined in Section 2 of the IPFIX protocol specification [RFC5101] and Section 3 of the PSAMP protocol specification [RFC5476]. Like in [RFC5101] and [RFC5476], these IPFIX-specific terms get a capitalized first letter when they're used in this document πŸ’–

This document also defines these new terms:

commonPropertiesID: The commonPropertiesID is an identifier for a set of common properties. It's locally unique per Observation Domain and Transport Session. Usually this Information Element is used to link to info reported in separate Data Records. Check the IPFIX information model [RFC5102] for the Information Element definition πŸͺ„

Common Properties: Common Properties are a collection of one or more attributes shared by a set of different Flow Records. Each set of Common Properties can be uniquely identified with a commonPropertiesID πŸ‘‘

Specific Properties: Specific Properties are a collection of one or more attributes reported in a Flow Record that are not included in the Common Properties defined for that Flow Record πŸ¦‹

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Those are the official BCP 14 vibes, so this is where the capitals matter πŸ™ˆ

2.1. Terminology Summary Table πŸ“

Here's the table that sums up the set and record terms. Each row is a kind of Set, and each column says what's inside it πŸ‘€

   +------------------+---------------------------------------------+
   |                  |                 Contents                    |
   |                  +--------------------+------------------------+
   |       Set        |      Template      |         Record         |
   +------------------+--------------------+------------------------+
   |     Data Set     |          /         |     Data Record(s)     |
   +------------------+--------------------+------------------------+
   |   Template Set   | Template Record(s) |           /            |
   +------------------+--------------------+------------------------+
   | Options Template | Options Template   |           /            |
   |       Set        | Record(s)          |                        |
   +------------------+--------------------+------------------------+

                         Terminology Summary Table

A Data Set is made up of Data Record(s). No Template Record is included in it. A Template Record or an Options Template Record is what defines the Data Record 🧩

A Template Set contains only Template Record(s) πŸ’•

An Options Template Set contains only Options Template Record(s) ✨

2.2. IPFIX Flows versus PSAMP Packets ✨

okay so the big difference between IPFIX and PSAMP is literally in the name of what they export πŸ’… IPFIX exports Flow Records, while PSAMP exports Packet Records. As [RFC5476] explains, that's the core split. From a pure export point of view, IPFIX can't tell whether a Flow Record was built from a bunch of packets squished together or from just one packet πŸ™ˆ So a PSAMP export is basically a special IPFIX Flow Record that's all about one single packet. iykyk, it's giving "one is just the other, zoomed in" πŸ”

To keep things clear in this doc, "Flow Record" is the generic umbrella term. It means either an IPFIX Flow Record or a PSAMP Packet Record, depending on how it's defined. When it matters, though, we'll call out which one we mean πŸŽ€

3. Specifications for Bandwidth-Saving Information Export πŸ¦‹

bestie, so a lot of Flow Records share the same set of Common Properties πŸ‘‘ The problem is that if you repeat that info on every single Flow Record, you're wasting so much space it's not even funny 😭 This document is here to fix that redundancy with a method for cutting it down ✨

PSAMP is used to export per-packet information, which means each observed packet gets sent inside its own IPFIX Flow Record. That's just a special case of a Flow Record with exactly one packet in it 🧚 So the method in this document works for per-packet data reduction too. The examples given are export of One-Way Delay (OWD) measurements (see Appendix) and trajectory sampling, among others 🌈

3.1. Problem Statement and High-Level Solution πŸ’–

picture this: you've got a set of properties "A", like a common sourceAddressA and sourcePortA, that are the same on every Flow Record you export 🌸 Figure 1 shows the messy version, where that info gets repeated over and over with classic IPFIX Flow Records. Total bandwidth waste, no cap πŸ’€

   +----------------+-------------+---------------------------+
   | sourceAddressA | sourcePortA |     <Flow1 information>   |
   +----------------+-------------+---------------------------+
   | sourceAddressA | sourcePortA |     <Flow2 information>   |
   +----------------+-------------+---------------------------+
   | sourceAddressA | sourcePortA |     <Flow3 information>   |
   +----------------+-------------+---------------------------+
   | sourceAddressA | sourcePortA |     <Flow4 information>   |
   +----------------+-------------+---------------------------+
   |      ...       |     ...     |            ...            |
   +----------------+-------------+---------------------------+

        Figure 1: Common and Specific Properties Exported Together

Figure 2 is the glow-up πŸ¦‹ Here, the Common Properties get split away from the Specific Properties for each Flow Record. The Common Properties only get exported once, inside a specific Data Record defined by an Options Template. Each Flow Record then carries a pointer to the Common Properties A, plus its own flow-specific info. To keep the relationship between the two sets straight, we add indices (here, the index for properties A). These indices are unique across all Common Properties entries within an Observation Domain, and they act like a "key" that identifies the "rows" of the Common Properties table. The Specific Properties reference those rows by using the right value for the Common Properties identifier πŸ”‘

   +------------------------+-----------------+-------------+
   | index for properties A | sourceAddressA  | sourcePortA |
   +------------------------+-----------------+-------------+
   |          ...           |      ...        |     ...     |
   +------------------------+-----------------+-------------+


   +------------------------+---------------------------+
   | index for properties A |     <Flow1 information>   |
   +------------------------+---------------------------+
   | index for properties A |     <Flow2 information>   |
   +------------------------+---------------------------+
   | index for properties A |     <Flow3 information>   |
   +------------------------+---------------------------+
   | index for properties A |     <Flow4 information>   |
   +------------------------+---------------------------+

       Figure 2: Common and Specific Properties Exported Separately

Exporting the Common Properties just once means less bandwidth needed on the path between the Exporter and the Collector πŸŽ‰ Slay.

3.2. Data Reduction Technique 🧠

so IPFIX [RFC5101] is Template based, which means Templates describe how data should be exported. They lay out the data fields, along with each field's type and meaning πŸ“ IPFIX has two kinds of Templates: the Template Record and the Options Template Record. The big difference is that an Options Template Record has the concept of scope, which says how far the Data Record applies. Scope only exists in Options Template Records, and it gives context for the Information Elements reported in the Data Records 🌍 You need the Template Records and Options Template Records to decode the Data Records, because if you only look at a Data Record by itself, there's no way to tell whether it was defined by a Template Record or an Options Template Record 😡 To export info more efficiently, this spec groups Flow Records by their common properties. We define Common Properties as a collection of attributes shared by a bunch of different Flow Records πŸ’œ

An implementation using this spec MUST follow the IPFIX transport protocol specifications defined in the IPFIX protocol [RFC5101]. No skipping that, bestie 🫢

As Figure 3 shows, the info gets split into two parts, and each part goes in a different Data Record. Common Properties MUST be exported via Data Records defined by an Options Template Record. Just like Template Records, they MUST be sent only once per SCTP association or TCP connection, and if SCTP is the transport protocol, they MUST be sent reliably via SCTP. These properties hold values that are common to several Flow Records, like the IP source and destination address 🌊 The Common Properties Data Records MUST be sent prior to the corresponding Specific Properties Data Records. The Data Records reporting Specific Properties MUST be associated with the Data Records reporting the Common Properties, using a unique identifier for the Common Properties: the commonPropertiesID Information Element [RFC5102] πŸ†” The commonPropertiesID Information Element MUST be included in the scope of the Options Template Record, and it MUST also be included in the associated Template Record. Okay, that's the whole vibe πŸ’…

   +---------------------------+     +---------------------+
   | Common Properties         |     | Specific Properties |  Template
   | Options Template Record   |     | Template Record     |  Definition
   |                           |     |                     |
   | scope: commonPropertiesID |     | commonPropertiesID  |
   | Common Properties         |     | Specific Properties |
   +------------+--------------+     +----------+----------+
   .............|...............................|.......................
                |                               |
   +------------v-------------+      +----------v----------+
   | Common Properties        |      | Specific Properties |+  Exported
   | Data Record              |------> Data Records        ||  Data
   +--------------------------+      +---------------------+|  Records
                                      +---------------------+

          Figure 3: Template Record and Data Record Dependencies

Here's the tea: from the IPFIX protocol's point of view, per-Flow and per-packet data reduction are exactly the same. The only difference might be the naming, since the Specific Properties could be called packet Specific Properties in the figure above πŸ¦‹

4. Transport Protocol Choice πŸš€

this doc follows the IPFIX transport protocol specs defined in the IPFIX protocol [RFC5101]. But depending on which transport protocol you pick, there are some extra rules to follow πŸ“‹ If Partial Reliable Stream Control Transmission Protocol (PR-SCTP) [RFC3758] is selected as the IPFIX protocol, the PR-SCTP subsection specifications MUST be respected. If UDP is selected as the IPFIX protocol, the UDP subsection specifications MUST be respected. If TCP is selected as the IPFIX protocol, the TCP subsection specifications MUST be respected. Check your transport, then follow its rules πŸ’–

4.1. PR-SCTP 🌸

The active Common Properties MUST be sent after the SCTP association establishment and before the corresponding Specific Properties Data Records. If the SCTP association gets re-established, all active Common Properties MUST be resent before the corresponding Specific Properties Data Records πŸ”

The Common Properties Data Records MUST be sent reliably 🫑

4.2. UDP 🌈

Common Properties Data Records MUST be resent on a regular basis. The periodicity MUST be configurable, and the default value for how often Common Properties get sent (the refresh timeout) is 10 minutes ⏰

The Exporting Process SHOULD transmit the Common Properties definition in advance of any Data Record that uses those Common Properties. That helps make sure the Collector already has the definition before the first associated Data Record shows up πŸ“¬

If a commonPropertiesID is no longer used, the Exporting Process stops resending the related Common Properties Data Record. The old commonPropertiesID MUST NOT be used until its lifetime (see Section 6.1) has expired βŒ›

4.3. TCP πŸ“

Common Properties MUST be sent after the TCP connection establishment and before the corresponding Specific Properties Data Records. If the TCP connection gets re-established, all active Common Properties MUST be resent before the corresponding Specific Properties Data Records πŸ’œ

5. commonPropertiesID Management ✨

okay so the commonPropertiesID is basically a name tag for a bundle of common properties πŸŽ€ it's locally unique per Observation Domain and Transport Session, and the Exporting Process MUST manage the commonPropertiesID allocations for its Observation Domains and Transport Session πŸ’… different Observation Domains from the same Exporter MAY use the same commonPropertiesID value to refer to different sets of Common Properties, so no drama there πŸ¦‹

the commonPropertiesID values MAY be assigned sequentially, but it is NOT REQUIRED 🌸 particular commonPropertiesID ranges or values MAY have explicit meanings for the IPFIX Device. for example, commonPropertiesID values may be assigned based on the result of a hash function, etc. ✨

using a 64-bit commonPropertiesID Information Element lets you export 2**64 active sets of Common Properties, per Observation Domain and per Transport Session 😭 that is a LOT of main character energy πŸ‘‘

commonPropertiesIDs that aren't used anymore SHOULD be withdrawn 🧹 the Common Properties Withdrawal message is a Data Record defined by an Options Template with only one scope field, which is the commonPropertiesID (type 137 [RFC5102]), and zero non-scope fields. it's giving minimal πŸ“

   0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Set ID = 3            |      Length = 14 octets       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Template ID N           |       Field Count = 1         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Scope Field count = 1    |0|  commonPropertiesID = 137   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Scope 1 Field Length = 8     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 4: Common Properties Withdrawal Message

⚠️ if you're using UDP as your transport protocol, the Common Properties Withdrawal messages MUST NOT be used πŸ™…β€β™€οΈ it's inefficient because UDP is unreliable, so don't even go there πŸ’€

6. The Collecting Process Side 🧚

this section is all about the Collecting Process when you're using SCTP and PR-SCTP as the transport protocol πŸ’œ if you need changes for TCP or UDP, those are in the subsections, so keep scrolling bestie 🫢

the Collecting Process MUST store the commonPropertiesID information for the whole duration of the association, so it can actually read the matching Data Records that show up in later Data Sets πŸ“¦

the Collecting Process can either store Data Records as they arrive, without rebuilding the original Flow Record, or it can rebuild the initial Flow Record. storing them as-is might need less storage on the Collector side πŸ—„οΈ rebuilding is more complex and time-consuming, because real-time record processing needs way more resources πŸ₯΅

if the Collecting Process gets the Specific Properties Data Record before its Common Properties Data Record, the Collecting Process SHOULD store the Specific Properties Data Record and wait for the Common Properties Data Record to be retransmitted or to arrive out of order ⏳ patience is a whole vibe 🌈

commonPropertiesIDs are unique per SCTP association and per Observation Domain πŸ”‘ if the Collecting Process receives an Options Template Record whose scope has a commonPropertiesID that was already received and hasn't been withdrawn (so, a commonPropertiesID from the same Exporter Observation Domain on that SCTP association), the Collecting Process MUST shut down the association 🚨 that's a hard no, periodt πŸ’…

when an SCTP association is closed, the Collecting Process MUST discard all commonPropertiesIDs received over that association and stop decoding the IPFIX Messages that use them 🧹 clean slate, iykyk ✨

if a Collecting Process receives a Common Properties Withdrawal message, it MUST delete the matching Common Properties tied to that specific SCTP association and specific Observation Domain, and stop interpreting Data Records that refer to them πŸ—‘οΈ any Data Records pointing at withdrawn Common Properties MUST be ignored, and SHOULD be logged by the Collecting Process πŸ“

if the Collecting Process gets a Common Properties Withdrawal message for Common Properties it never received on this SCTP association, it MUST reset the SCTP association and discard the IPFIX Message πŸ™ˆ it SHOULD log the error the same way it does for malformed IPFIX Messages πŸ“‹ not the vibe πŸ’€

6.1. UDP πŸ¦‹

the Collecting Process MUST associate a lifetime with each Common Property received via UDP πŸ•°οΈ Common Properties that the Exporting Process doesn't refresh within that lifetime get expired at the Collecting Process πŸ‘‹

if the Common Properties aren't refreshed before that lifetime runs out, the Collecting Process MUST discard the matching definition of the commonPropertiesID and any current and future associated Data Records 😱 in this case, an alarm MUST be logged 🚨

the Collecting Process MUST NOT decode any more Data Records associated with expired Common Properties 🚫 if a Common Property gets refreshed with a definition that's different from the previous one, the Collecting Process SHOULD log a warning and replace the old Common Property with the new one ⚠️ the Common Property lifetime at the Collecting Process MUST be at least 3 times higher than the refresh timeout of the Template used to export the Common Property definition, as configured on the Exporting Process ⏱️ that's the rule, no cap 🫑

the Collecting Process SHOULD accept Data Records even if it doesn't have the Common Properties needed to decode them yet πŸ“¬ if those Common Properties haven't arrived when the Data Records show up, the Collecting Process SHOULD store the Data Records for a short period and decode them once the Common Properties definitions come in 🧠 that short period MUST be lower than the lifetime of definitions associated with identifiers considered unique within the UDP session ⏳ timing is everything, bestie πŸ’–

6.2. TCP 🌸

when the TCP connection is reset, either gracefully or abnormally, the Collecting Processes MUST delete all commonPropertiesID values and the associated Common Properties data for that connection 🧽 total wipe πŸ’…

if a Collection Process receives a Common Properties Withdrawal message, the Collection Process MUST expire the related Common Properties data πŸ‘‹


7. Advanced Techniques ✨

7.1. Multiple Data Reduction ✨

okay so a Flow Record can point at one or more Common Properties sets, and using multiple sets can make exports way more efficient πŸ¦‹ when you spot more than one set of properties that don't overlap, that's your moment to shine.

ngl this part is so important: when you have multiple Common Properties in one Data Record, the different sets MUST be disjoint (i.e., MUST NOT have Information Elements in common) to avoid potential collisions. no overlapping allowed, bestie πŸ™…β€β™€οΈ

Let's say we have properties "A", like sourceAddressA and sourcePortA, and another set "B", like destinationAddressB and destinationPortB. Figure 5 shows how classic IPFIX export keeps repeating that same info across a bunch of Flow Records πŸ’€

   +--------+--------+---------+---------+---------------------+
   |srcAddrA|srcPortA|destAddrB|destPortB| <Flow1 information> |
   +--------+--------+---------+---------+---------------------+
   |srcAddrA|srcPortA|destAddrC|destPortC| <Flow2 information> |
   +--------+--------+---------+---------+---------------------+
   |srcAddrD|srcPortD|destAddrB|destPortB| <Flow3 information> |
   +--------+--------+---------+---------+---------------------+
   |srcAddrD|srcPortD|destAddrC|destPortC| <Flow4 information> |
   +--------+--------+---------+---------+---------------------+
   |   ...  |   ...  |   ...   |   ...   |        ...          |
   +--------+--------+---------+---------+---------------------+

        Figure 5: Common and Specific Properties Exported Together

Besides A and B, other sets of Properties might be repeated too (like Properties C and D in the figure above) πŸ‘€

Here's the glow-up 🌸 we can split the Common Properties into properties A (sourceAddressA and sourcePortA), properties D (sourceAddressD and sourcePortD), properties B (destinationAddressB and destinationPortB), and properties C (destinationAddressC and destinationPortC). These four records can then be expanded into four combinations of Data Records, so we cut down on redundancy without needing to define four complete sets of Common Properties (see the figure below). The more Common Properties sets you define, the more combinations you get to play with 🧚

   +-------------------+-----------------+-------------+
   | index for prop. A | sourceAddressA  | sourcePortA |
   +-------------------+-----------------+-------------+
   | index for prop. D | sourceAddressD  | sourcePortD |
   +-------------------+-----------------+-------------+

   +-------------------+---------------------+------------------+
   | index for prop. B | destinationAddressB | destinationPortB |
   +-------------------+---------------------+------------------+
   | index for prop. C | destinationAddressC | destinationPortC |
   +-------------------+---------------------+------------------+

   +------------------+------------------+-----------------------+
   |index for prop. A |index for prop. B |  <Flow1 information>  |
   +------------------+------------------+-----------------------+
   |index for prop. A |index for prop. C |  <Flow2 information>  |
   +------------------+------------------+-----------------------+
   |index for prop. D |index for prop. B |  <Flow3 information>  |
   +------------------+------------------+-----------------------+
   |index for prop. D |index for prop. C |  <Flow4 information>  |
   +------------------+------------------+-----------------------+

     Figure 6: Multiple Common (above) and Specific Properties (below)
                            Exported Separately

The win with multiple Common Properties is that you still hit the bandwidth goal and keep the number of indices super low πŸŽ€ Adding an extra index to every record wouldn't save any bandwidth in the Figure 5 case, and it's generally the less efficient move, periodt πŸ’…

If a set of Flow Records shares multiple sets of Common Properties, multiple commonPropertiesID instances MAY be used to make the export even more efficient, as shown in Figure 7 ✨

   +--------------------------- +     +---------------------+
   | Common Properties          |     | Specific Properties | Template
   | Options Template Record    |     | Template Record     | Definition
   |                            |     |                     |
   | Scope: commonPropertiesID1 |     | commonPropertiesID1 |
   | Scope: commonPropertiesID2 |     | commonPropertiesID2 |
   | Common Properties          |     | Specific Properties |
   +------------+---------------+     +---------+-----------+
   .............|...............................|.......................
                |                               |
   +------------v-------------+      +----------v----------+
   | Common Properties        |      | Specific Properties |+  Exported
   | Data Record              |------> Data Records        ||  Data
   +------------------------- +      +---------------------+|  Records
                                      +---------------------+

                     Figure 7: Multiple Data Reduction

7.2. Cascading Common Properties πŸ”—

okay the rule here is giving strict 🚫 An Exporting Process MUST NOT export any set of Common Properties that contains references to itself, either directly or through other cascaded Common Properties, in its own definition (aka a circular definition). Circular definitions are a no-go, full stop πŸ’€

When the Collecting Process gets Common Properties that reference other Common Properties, it MUST resolve those references 🧩 If the Common Properties aren't available when the Data Records arrive, the Collecting Process SHOULD store the Data Records for a short period of time and decode them once the Common Properties show up.

If the Collecting Process couldn't decode a cascading Common Properties definition because the referenced Common Properties weren't available before that short period of time ran out, then the Collecting Process SHOULD log the error πŸ“

And if the Collecting Process couldn't decode a cascading Common Properties definition because it detected a circular definition, then the Collecting Process SHOULD log the error too πŸ™ˆ

Information Element ordering MUST be preserved when creating and expanding Common Properties. no shuffling the order, sis 🫢

8. Export and Evaluation Considerations πŸ“Š

The whole point of this method is to reduce how much measurement data has to be transferred from the Exporter to the Collector πŸ¦‹ The efficiency can vary, as we'll talk about in this section. Also, there might be less storage needed on the Collector side if it stores the Data Records as they arrive, without rebuilding the original Flow Record. big brain energy 🧠

BUT (and it's a big but) this method asks for extra resources on both ends 😭 The Exporter has to manage the Common Properties information and assign commonPropertiesID values. The Collector has to process records described by two templates instead of just one. And there's extra work when post-processing the measurement data, because you have to correlate Flow Records with the Common Properties information. it's a trade-off, babe πŸ’…

8.1. Transport Protocol Choice 🚚

The proposed method works best with a reliable transport protocol for transferring the Common Properties. So PR-SCTP with full reliability or TCP is recommended for sending IPFIX Messages that contain Common Properties 🌈 Heads up: UDP is less efficient for Common Properties, because they have to be resent regularly 😬

8.2. Reduced Size Encoding πŸ“

Transferring the commonPropertiesIDs comes with some overhead, and it might even increase the amount of exported data if the commonPropertiesID field isn't shorter than the fields it replaces πŸ’€

In cases where the range of the commonPropertiesID can be restricted, it is RECOMMENDED to apply reduced size encoding to the commonPropertiesID to get an extra bandwidth gain πŸ“

8.3. Efficiency Gain πŸ“ˆ

okay so the goal is to reduce bandwidth, but the efficiency can be limited 🫠 The gain depends on how much redundant info is in the Flows, and it's directly proportional to how often the defined commonPropertiesID values get reused. The theoretical limit is when every Data Record uses a single commonPropertiesID. in other words, the more you reuse a commonPropertiesID value, the better the gain πŸ”₯ The Exporting Process can check the direct gain for the Flow Records it's about to export, but it can't predict whether future Flow Records will contain the info specified by the active commonPropertiesID values πŸ‘€ So this spec is way more efficient for specific apps where filtering is involved, like one-way delay or trajectory sampling ✨

Plot twist: this technique can actually increase bandwidth usage under certain conditions 😱 If you count the overhead of exporting the commonPropertiesID values, and those values never get used in future Data Records, the technique ends up increasing export bandwidth. A classic example is assigning Common Properties based on past observed traffic and just hoping future Flows have the same characteristics πŸ™ˆ not it, hope isn't a strategy lol

The efficiency gain also depends on the difference between the length of the replaced fields and the length of the commonPropertiesID. The shorter the commonPropertiesID is (compared to the total length of the Common Properties fields), the bigger the gain 🧚

The example in Appendix A.2 below uses IPFIX to export measurement data for each received packet. In that case, for a Flow of 1000 packets, the amount of data can be decreased by more than 26 percent πŸŽ‰

9. Security Considerations πŸ”

The same security considerations as for the IPFIX protocol [RFC5101] apply. so yes, the IPFIX security story carries over, and you should take it seriously πŸ›‘οΈπŸ’œ

10. Acknowledgments 🫢

The authors would like to thank Guido Pohl for initiating this work and for his contribution to early versions of this document. Thanks also to Andrew Johnson, Gehrard Muenz, Brian Trammell, and Paul Aitken for their comments and feedback. such a supportive crew 🌸

11. References πŸ“š

11.1. Normative References

  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [RFC5101] Claise, B., "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information", RFC 5101, January 2008.
  • [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, "Information Model for IP Flow Information Export", RFC 5102, January 2008.
  • [RFC5476] Claise, B., Ed., "Packet Sampling (PSAMP) Protocol Specifications", RFC 5476, March 2009.

11.2. Informative References

   [PSAMP-MIB]  Dietz, T., Ed. and B. Claise, "Definitions of Managed
                Objects for Packet Sampling", Work in Progress,
                June 2006.

   [RFC3550]    Schulzrinne, H., Casner, S., Frederick, R., and V.
                Jacobson, "RTP: A Transport Protocol for Real-Time
                Applications", STD 64, RFC 3550, July 2003.

   [RFC3758]    Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P.
                Conrad, "Stream Control Transmission Protocol (SCTP)
                Partial Reliability Extension", RFC 3758, May 2004.

   [RFC3917]    Quittek, J., Zseby, T., Claise, B., and S. Zander,
                "Requirements for IP Flow Information Export (IPFIX)",
                RFC 3917, October 2004.

   [RFC4960]    Stewart, R., "Stream Control Transmission Protocol",
                RFC 4960, September 2007.

   [RFC5470]    Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek,
                "Architecture for IP Flow Information Export", RFC 5470,
                March 2009.

   [RFC5472]    Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP
                Flow Information Export (IPFIX) Applicability",
                RFC 5472, March 2009.

   [RFC5474]    Duffield, N., Ed., "A Framework for Packet Selection and
                Reporting", RFC 5474, March 2009.

   [RFC5475]    Zseby, T., Molina, M., Duffield, N., Niccolini, S., and
                F. Raspall, "Sampling and Filtering Techniques for IP
                Packet Selection", RFC 5475, March 2009.

   [RFC5477]    Dietz, T., Claise, B., Aitken, P., Dressler, F., and G.
                Carle, "Information Model for Packet Sampling Exports",
                RFC 5477, March 2009.

Appendix A. Examples ✨

A.1. Per-Flow Data Reduction πŸ¦‹

Okay so in this section we're going to see how Flow information can be exported super efficiently using the method from this document πŸ’– Let's say we need to keep sending data about two IPv6 Flows, periodically, bestie 🫢

In this example, we're reporting the following info:

   Flow|        dstIPv6Address                 | dst- |nPkts|nBytes
       |                                       | Port |     |
   ----------------------------------------------------------------
    A  |2001:DB8:80AD:5800:0058:0800:2023:1D71 |  80  | 30  |  6000
       |                                       |      |     |
    A  |2001:DB8:80AD:5800:0058:0800:2023:1D71 |  80  | 50  |  9500
       |                                       |      |     |
    B  |2001:DB8:80AD:5800:0058:00AA:00B7:AF2B | 1932 | 60  |  8000
       |                                       |      |     |
    A  |2001:DB8:80AD:5800:0058:0800:2023:1D71 |  80  | 40  |  6500
       |                                       |      |     |
    A  |2001:DB8:80AD:5800:0058:0800:2023:1D71 |  80  | 60  |  9500
       |                                       |      |     |
    B  |2001:DB8:80AD:5800:0058:00AA:00B7:AF2B | 1932 | 54  |  7600

Figure 8: Flow Information Example πŸ“Έ

The Common Properties here are the destination IPv6 address and the destination port 🌸 We start by defining an Options Template with these Information Elements:

  • Scope: commonPropertiesID in [RFC5102], with a type of 137 and a length of 8 octets. πŸŽ€
  • The destination IPv6 address: destinationIPv6Address in [RFC5102], with a type of 28 and a length of 16 octets. 🧚
  • The destination port: destinationTransportPort in [RFC5102], with a type of 11, and a length of 2 octets. πŸ’œ

Figure 9 shows the Options Template defining the Common Properties with commonPropertiesID as scope πŸ‘‘

     0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Set ID = 3            |      Length = 24 octets       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Template ID = 257       |       Field Count = 3         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Scope Field count = 1    |0|  commonPropertiesID = 137   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Scope 1 Field Length = 8     |0|  destinationIPv6Address = 28|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Field Length = 16        |0|destinationTransportPort = 11|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Field Length = 2         |        (Padding)              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 9: Common Properties Options Template πŸ¦‹

The Specific Properties Template is everything not already in the Options Templates, so basically the Flow-specific stuff πŸ’… In this case that's the number of packets and the number of bytes being reported. It also includes the commonPropertiesID. In Data Records, that field's value holds one of the unique indices from the Option Records exported before ✨ It has these Information Elements (see also Figure 10):

  • commonPropertiesID with a length of 8 octets.
  • The number of packets of the Flow: inPacketDeltaCount in [RFC5102], with a length of 4 octets.
  • The number of octets of the Flow: inOctetDeltaCount in [RFC5102], with a length of 4 octets.
     0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Set ID = 2            |      Length = 20 octets       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Template ID = 258       |       Field Count = 3         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0|  commonPropertiesID = 137   |       Field Length = 8        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0|    inPacketDeltaCount = 2   |       Field Length = 4        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0|    inOctetDeltaCount = 1    |       Field Length = 4        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 10: Specific Properties Template 🌈

Looking at the data from the start of this example, these two Data Records get exported πŸ“

   Common-      |           dstAddress                    | dst-
   PropertiesID |                                         | Port
   -------------+-----------------------------------------+-------
       101      | 2001:DB8:80AD:5800:0058:0800:2023:1D71  |  80
                |                                         |
       102      | 2001:DB8:80AD:5800:0058:00AA:00B7:AF2B  | 1932

Figure 11 🫢

The Data Records reporting the Common Properties will look like this:

   0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Set ID = 257          |      Length = 60 octets       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   +-                             101                             -+
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   +-                                                             -+
   |                                                               |
   +-           2001:DB8:80AD:5800:0058:0800:2023:1D71            -+
   |                                                               |
   +-                                                             -+
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |            80                 |                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                             -+
   |                              102                              |
   +-                             -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                               |                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                              -+
   |                                                               |
   +-                                                             -+
   |            2001:DB8:80AD:5800:0058:00AA:00B7:AF2B             |
   +-                                                             -+
   |                                                               |
   +-                             -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                               |          1932                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 12: Data Records reporting Common Properties πŸ’–

The Data Records will then be:

commonPropertiesID  |  inPacketDeltaCount  | inOctetDeltaCount
---------------------------------------------------------------
        101         |          30          |       6000
        101         |          50          |       9500
        102         |          60          |       8000
        101         |          40          |       6500
        101         |          60          |       9500
        102         |          54          |       7600

Figure 13 πŸ”₯

Figure 14 shows the first Data Record from that table, and it's giving clean 🧚

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Set ID = 258         |          Length = 16          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   +-                             101                             -+
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |               30              |             6000              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 14: Data Record reporting Common Properties ✨

A.2. Per-Packet Data Reduction ⏱️

okay so the example for per-packet data reduction is One-Way Delay (OWD) measurement 🫢 the exact same packet has to be spotted at BOTH the source and the destination of the path being measured. you get the OWD by subtracting the time it was seen at the two endpoints, and that only works if their clocks are synchronized ⏰

bestie, OWD is measured for an application that has a Service Level Agreement (SLA) attached, so you end up watching a bunch of packets with Specific Properties πŸ’… to match the identical packet at both Observation Points, you need a set of properties to be observed at both ends. think: all packets from one specific source and destination IP address, with one specific Diffserv codepoint (DSCP) value, going to one specific destination transport port πŸŽ€

so the source and destination each export a series of Flow Records with two kinds of info: common info that's the same for every packet, and unique info for each packet, so you can make a unique identifier for each one passing the Observation Point (like a hash value over the invariant fields of the packet) πŸ”₯ the two endpoints can each apply the redundancy technique from this document on their own, totally independently, to save bandwidth on their own Flow Record exports ✨

the Templates needed to export this kind of measurement data are in the figures below. Figure 15 shows the Options Template holding the Flow info, using the commonPropertiesID as scope πŸ¦‹ the Common Properties Template exports these Information Elements:

  • The source IPv4 Address: sourceIPv4Address in [RFC5102], with a type of 8 and a length of 4 octets.
  • The destination IPv4 Address: destinationIPv4Address in [RFC5102], with a type of 12 and a length of 4 octets.
  • The Class of Service field: ClassOfServiceIPv4 in [RFC5102], with a type of 5 and a length of 1 octet.
  • The Protocol Identifier: protocolIdentifier in [RFC5102], with a type of 4 and a length of 1 octet.
  • The source port: sourceTransportPort in [RFC5102], with a type of 7 and a length of 2 octets.
  • The destination port: destinationTransportPort in [RFC5102], with a type of 11 and a length of 2 octets.

The commonPropertiesID Information Element is used as the Scope Field 🌸

 0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Set ID = 3            |      Length = 40 octets       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Template ID = 256       |       Field Count = 7         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Scope Field count = 1    |0|  commonPropertiesID = 137   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Scope 1 Field Length = 4     |0|    sourceIPv4Address = 8    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Field Length = 4         |0| destinationIPv4Address = 12 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Field Length = 4         |0|  classOfServiceIPv4 = 5     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Field Length = 1         |0|  protocolIdentifier = 4     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Field Length = 1         |0|  transportSourcePort = 7    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Field Length = 2         |0|transportDestinationPort = 11|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Field Length = 2         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 15: Example Flow Properties Template πŸ’œ

for passive OWD measurement, the Packet Properties Template or Specific Properties Template has at least the timestamp and packet ID 🧚 it also has a commonPropertiesID field so the packet can be linked to its Flow πŸ”—

Figure 16 shows the template with the packet properties. in this example we export these Information Elements:

  • commonPropertiesID. In this case, reduced size encoding is used, and the Information Element is declared with a length of 4 octets instead of 8.
  • The packet timestamp: observationTimeMilliseconds in the PSAMP information model [RFC5477], with a type of 323 and a length of 8 octets.
  • digestHashValue in the PSAMP information model [RFC5477], with a type of 326 and a length of 8 octets.
  • The packet length: ipTotalLength in the IPFIX information model [RFC5102], with a type of 224 and a length of 8 octets.
  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |         Set ID = 2            |      Length = 36 octets       |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |       Template ID = 257       |       Field Count = 4         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |0|  commonPropertiesID = 137   |       Field Length = 4        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |0| observationTimeMillis.= 323 |       Field Length = 8        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |0|    digestHashValue = 326    |       Field Length = 8        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |0|    ipTotalLength = 224      |       Field Length = 8        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 16: Example Packet Properties Template πŸ’œ

at the collection point, the packet records from the two measurement points get gathered and correlated using the packet ID πŸ¦‹ the resulting delay Data Records are exported in a similar way to the packet data. OWD data is linked to Flow info through the commonPropertiesID field. the OWD properties hold the Packet Pair ID (the packet ID of the two contributing packet records), the timestamp of the packet passing the reference monitor point so you can rebuild a time series, the calculated delay value, and the commonPropertiesID πŸ“ˆ

in this example, using IPFIX to export the measurement data for each received packet, 38 bytes have to be transferred (sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2, destinationTransportPort=2, observationTimeMilliseconds=8, digestHashValue=8, ipTotalLength=8). Without considering the IPFIX protocol overhead, a Flow of 1000 packets produces 38000 bytes of measurement data πŸ“

using the proposed optimization, each packet produces an export of only 28 bytes (observationTimeMilliseconds=8, digestHashValue=8, ipTotalLength=8, commonPropertiesID=4). The export of the Flow information produces 18 bytes (sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1, sourceTransportPort=2, destinationTransportPort=2, commonPropertiesID=4). For a Flow of 1000 packets, this sums to 28018 bytes. That's a decrease of more than 26 percent! πŸŽ‰πŸ‘‘

A.3. Common Properties Withdrawal Message πŸ—‘οΈ

this section shows an example commonPropertiesID Withdrawal message πŸ’… Figure 17 depicts the Options Template Record with the commonPropertiesID as unique scope field, and no non-scope fields.

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |         Set ID = 3            |      Length = 14 octets       |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |       Template ID 259         |       Field Count = 1         |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |      Scope Field count = 1    |0|  commonPropertiesID 137     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |  Scope 1 Field Length = 8     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 17: Example Common Properties Withdrawal Template πŸ’œ

Figure 18 shows the Option Data Record withdrawing commonPropertiesID N:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |         Set ID = 259          |      Length = 12 octets       |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                                                               |
 +-                              N                              -+
 |                                                               |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 18: Record Withdrawing commonPropertiesID N πŸ’œ


Authors' Addresses πŸ–ŠοΈ

Elisa Boschi
Hitachi Europe
c/o ETH Zurich
Gloriastrasse 35
8092 Zurich
Switzerland

Phone: +41 44 6327057
EMail: elisa.boschi@hitachi-eu.com


Lutz Mark
Fraunhofer IFAM
Wiener Str. 12
28359 Bremen
Germany

Phone: +49 421 2246206
EMail: lutz.mark@ifam.fraunhofer.de


Benoit Claise
Cisco Systems, Inc.
De Kleetlaan 6a b1
Diegem  1813
Belgium

Phone: +32 2 704 5622
EMail: bclaise@cisco.com

That's the whole RFC, bestie! 🌈✨ Elisa, Lutz and Benoit slayed this one πŸ‘‘πŸ’–

πŸ“¦ where this page comes from