RFC 6526
IP Flow Information Export (IPFIX) Per Stream Control Transmission Protocol (SCTP) Stream
IP Flow Information Export (IPFIX) Per Stream Control Transmission Protocol (SCTP) Stream 💖
okay so this is RFC 6526 🎀 It's an IETF document, published March 2012, and it's a Standards Track doc (ISSN 2070-1721). The authors are B. Claise, P. Aitken, and A. Johnson from Cisco Systems, Inc., plus G. Muenz from TU Muenchen. The title block doesn't list anything it obsoletes or updates, so it's not replacing or amending anything. 🦋
Abstract ✨
bestie, this doc is an extension to the specs in RFC 5101, IP Flow Information Export (IPFIX), for when you're using the Partial Reliability extension of SCTP (PR-SCTP, Partial Reliability Stream Control Transmission Protocol). 💅
when the Exporting Process and the Collecting Process BOTH implement it, you get a whole bunch of perks: you can calculate Data Record losses for PR-SCTP per Template, export Template Withdrawal Messages immediately, reuse Template IDs right away within an SCTP stream, lower the chance of losing Data Records, and put less demand on the Collecting Process. 🌈 If only one side implements it (just the Collecting Process or just the Exporting Process), you just get normal IPFIX behavior without the extra benefits. 🥲
Status of This Memo 🌸
this is an Internet Standards Track document 💖 It's a product of the Internet Engineering Task Force (IETF), it represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication. If you want more on Internet Standards, check Section 2 of RFC 5741.
info about the current status of this doc, any errata, and how to give feedback lives at http://www.rfc-editor.org/info/rfc6526 ✨
Copyright Notice 🎀
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. 🫶
this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. please read these carefully because they spell out your rights and restrictions for this doc 📜
Code Components pulled out of this doc must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License. ⚠️
Table of Contents 🧚
1. Introduction ....................................................3
1.1. Relationship with IPFIX and PSAMP ..........................4
1.2. Applicability ..............................................5
1.3. Limitations ................................................5
2. Terminology .....................................................6
2.1. Conventions Used in This Document ..........................6
2.2. IPFIX Documents Overview ...................................6
2.3. PSAMP Documents Overview ...................................7
3. IPFIX Protocol Specifications: Limitations and Improvements .....7
3.1. Data Record Loss Calculated Per Template ...................7
3.1.1. IPFIX Protocol Specifications: Limitation ...........7
3.1.2. IPFIX Export Per SCTP Stream: Advantage .............8
3.2. Immediate Template Withdrawal and Reuse ....................8
3.2.1. IPFIX Protocol Specifications: Limitation ...........8
3.2.2. IPFIX Export Per SCTP Stream: Advantages ............9
3.3. Requirement for Data Set Buffering .........................9
3.3.1. IPFIX Protocol Specifications: Limitation ...........9
3.3.2. IPFIX Export Per SCTP Stream: Advantages ...........10
4. Specifications .................................................10
4.1. New Information Element ...................................10
4.2. Template Management .......................................11
4.3. SCTP ......................................................12
4.4. Template Withdrawal Message ...............................13
4.5. The Collecting Process's Side .............................14
4.5.1. SCTP ...............................................14
4.5.2. Enabling the Per-SCTP-Stream Extension .............14
4.5.3. Disabling the Per-SCTP-Stream Extension ............15
4.5.4. Calculating Data Record Loss Per Template ..........16
5. Resource Impact ................................................16
6. Examples .......................................................17
7. IANA Considerations ............................................20
8. Security Considerations ........................................21
9. References .....................................................21
9.1. Normative References ......................................21
9.2. Informative References ....................................21
10. Acknowledgments ...............................................221. Introduction ✨
the IPFIX protocol [RFC5101] has the goal of exporting Flow information 🦋 It's designed to export info about IP traffic Flows and related measurement data, where a Flow is defined by a set of key attributes (e.g., source and destination IP address, source and destination port, etc.). BUT thanks to its Template mechanism, IPFIX can export any type of information, as long as the relevant Information Element is specified in the IPFIX information model [RFC5102], registered with IANA [IANA], or specified as an enterprise-specific Information Element. 💜
so IPFIX [RFC5101] says traffic measurements for Flows get exported in a TLV (Type, Length, Value) format 🍓 The info goes out in a Template Record, which is sent once to export the {Type, Length} pairs that define the data format for the Information Elements in a Flow. After that, the Data Records specify values for each Flow. ✨
IPFIX [RFC5101] is honestly so flexible 🔥 It foresees multiple SCTP streams per association, it allows Data Sets, Template Sets, and/or Options Template Sets on any SCTP stream, it offers full and partially reliable export of Data Sets, and it specifies both ordered and out-of-order delivery of Data Sets. But because of bandwidth restrictions, packet losses in the network, and resource constraints on the Exporter and Collector (e.g., limited buffer sizes), it's not always possible to export all Data Sets in a reliable way. 😭
this doc specifies a method for exporting a Template Record and its associated Data Sets in a single SCTP stream, limiting each Template ID to a single SCTP stream if possible, and imposing in-order transmission. 🫶
this method has several advantages over IPFIX export as specified in [RFC5101]: it can calculate Data Record losses for PR-SCTP per Template, it allows immediate export of Template Withdrawal Messages, it allows immediate reuse of Template IDs within an SCTP stream, it reduces the likelihood of Data Record loss, and it reduces demands on the Collecting Process. 👑
1.1. Relationship with IPFIX and PSAMP 🌷
the specs in this doc apply to the IPFIX protocol specifications [RFC5101]. BUT they only apply to the SCTP transport protocol [RFC4960] option of IPFIX (see Section 10 of [RFC5101]), specifically if the Partial Reliability extension [RFC3758] is used. All specs from [RFC5101] apply, unless this document says otherwise. 💅
as the Packet Sampling (PSAMP) protocol specifications [RFC5476] are based on the IPFIX protocol specifications, the specs in this doc are also valid for the PSAMP protocol 🧚
1.2. Applicability 🌈
the specs in this doc are for cases where application requirements include knowing how many Data Records of a certain type (i.e., from a certain Template) were lost. A typical example is a router exporting billing records, where the Exporting Process can't afford to export all the Flow Records reliably because it has limited resources to buffer a large number of them. This can happen when Data Sets get generated at a higher rate at the Exporter than can be transferred to the Collector, because of bandwidth limitations in the network or slow reception at the Collector. 😩
to be more precise, the applicability is the case where multiple Templates are active at the same time within a single SCTP Transport Session, and calculating the Data Record loss for a particular Template is required. Honestly, with the current IPFIX specs [RFC5101], if an IPFIX Message is lost (UDP or SCTP partially reliable), it's not possible to tell which Template(s) the lost Data Records belonged to. 🙈
Exporting Processes following this doc will interoperate with existing Collecting Processes that comply with [RFC5101]. No changes are required at the Collecting Process to receive data from an Exporting Process compliant with this method. However, Collecting Processes may add support for the per-stream export specified in this doc to get all the benefits of the approach. Since the specs here mandate in-order transmission of (Options) Templates and associated Data Records, late arrival of (Options) Templates at the Collecting Process is avoided. That means there are no Data Records that need to be dropped or buffered. 🦋
1.3. Limitations 🥀
when multiple Templates are required, this method requires multiple SCTP streams in the association between the Exporting Process and Collecting Process, ideally one stream per Template. To properly handle the transmission of additional Templates during the Transport Session, additional SCTP streams are sometimes required. These SCTP streams can only be added within the existing SCTP association if the specs in [RFC6525] are supported. ⚠️
2. Terminology 📖
IPFIX-specific terminology used in this doc is defined in Section 2 of [RFC5101]. As in [RFC5101], these IPFIX-specific terms have the first letter of a word capitalized when used in this document. ✨
note that, in this doc, "(Options) Template" is used to refer to Templates AND Options Templates. Unless it says otherwise, "Template" alone refers to Templates exclusive of Options Templates. 🎀
Template Reuse Delay
the time the Exporting Process needs to wait after sending the last Data Set described by a given Template, before sending a Template Withdrawal Message for that Template. A suitable default value is 5 seconds, as specified in [RFC5101]. ⏳
2.1. Conventions Used in This Document 💜
okay the key words here are "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL". In this document they are to be interpreted as described in RFC 2119 [RFC2119]. 🫶
2.2. IPFIX Documents Overview ✨
okay so the IPFIX protocol [RFC5101] is the thing that gives network admins access to Flow information 👀🫶
the architecture for exporting measured Flow info from an IPFIX Exporting Process to a Collecting Process lives in the IPFIX architecture doc [RFC5470], and it follows the requirements in [RFC3917] 💅
that architecture spells out how IPFIX Data Records and Templates get carried over a congestion-aware transport protocol, from Exporting Processes to Collecting Processes 🚛✨
IPFIX also has a formal description of its Information Elements, covering their names, their types, and extra semantic info. that's all in the IPFIX information model [RFC5102] 🧠
and last but not least, the IPFIX applicability statement [RFC5472] tells you what kinds of apps can use IPFIX and how they can use the info it gives them. it also shows how the IPFIX framework fits in with other architectures and frameworks. main character energy 🎀
2.3. PSAMP Documents Overview 🌸
so the doc "A Framework for Packet Selection and Reporting" [RFC5474] is where you find the Packet Sampling (PSAMP) framework. it lets network elements pick out subsets of packets using statistical and other methods, then export a stream of reports about those picked packets to a Collector 📦💖
the packet selection techniques PSAMP supports (sampling, filtering, and hashing) are covered in "Sampling and Filtering Techniques for IP Packet Selection" [RFC5475] 🔍
the PSAMP protocol [RFC5476] handles exporting packet info from a PSAMP Exporting Process to a PSAMP Collecting Process. just like IPFIX, PSAMP has a formal description of its Information Elements, their names, their types, and extra semantic info. the PSAMP information model is defined in [RFC5477] 🧚
3. IPFIX Protocol Specifications: Limitations and Improvements 🦋
bestie, this section is the tea ☕ it covers three specific topics: "Data Record Loss Calculated Per Template", "Immediate Template Withdrawal and Reuse", and "Requirement for Data Set Buffering". for each one, it shows where the IPFIX protocol specs fall short, and then how the method in this document fixes it 🙌
3.1. Data Record Loss Calculated Per Template 💀
3.1.1. IPFIX Protocol Specifications: Limitation
so [RFC3917] has a section called "Requirements for IP Flow Information Export" (that's Section 6.3.2 of it) and it talks about data transfer reliability. it says:
Loss of flow records during the data transfer from the Exporting Process to the Collecting Process must be indicated at the Collecting Process.
the problem is that sometimes it really matters to know how many Data Records of a certain type got lost (like for billing, for example 💸), and IPFIX doesn't usually give you that info 😭
a Collecting Process can spot out-of-sequence, dropped, or duplicate IPFIX Messages by tracking the Sequence Number [RFC5101]. but here's the thing: the Sequence Number field in the IPFIX Message header goes up based on the number of IPFIX Data Records within the SCTP stream. so loss only gets detected per stream 🫠
the IPFIX protocol specs [RFC5101] say that Data Records defined by any Template can be sent on any SCTP stream. so if there's more than one Template in the whole SCTP association, there's literally no way to know which Template a lost Data Record belonged to. and this is true no matter what convention the Exporting Process uses to send Data Records on different SCTP streams, because the protocol makes no guarantees. not me being shocked 🙈
there's a workaround that [RFC5101] allows: use only one Template Record per SCTP Transport Session. the catch is that when you need multiple Template Records, you end up multiplying the number of SCTP Transport Sessions 🥴
3.1.2. IPFIX Export Per SCTP Stream: Advantage
okay this is the glow-up ✨. with the specs in this document, any lost Data Records are guaranteed to be associated only with the Templates defined on that SCTP stream.
so if you define only one Template per SCTP stream, any loss is tied to that single Template. basically, you export each Template and its corresponding Data Records on their own SCTP stream, separate from other Templates and Data Records. then the loss for each specific Template can be figured out from the Sequence Number field in the IPFIX Message headers. iconic, honestly 👑
3.2. Immediate Template Withdrawal and Reuse 🎀
3.2.1. IPFIX Protocol Specifications: Limitation
a Collecting Process needs to have received the Template Record linked to the Data Records before it can decode the info in them. [RFC5101] says:
The Exporting Process SHOULD transmit the Template Set and Options Template Set in advance of any Data Sets that use that (Options) Template ID, to help ensure that the Collecting Process has the Template Record before receiving the first Data Record.
and because the Collecting Process can't decode Data Records without the matching Template Record, those Data Records might get thrown out, as [RFC5101] explains:
The Collecting Process normally receives Template Records from the Exporting Process before receiving Data Records. The Data Records are then decoded and stored by the Collector. If the Template Records have not been received at the time Data Records are received, the Collecting Process MAY store the Data Records for a short period of time and decode them after the Template Records are received.
3.2.2. IPFIX Export Per SCTP Stream: Advantages
ok so if you export each Template Record and its matching Data Records within a single SCTP stream, and you enforce in-order transmission, the Template Record always shows up before the Data Records it goes with. so there's no risk of the Collecting Process tossing Data Records while it waits for the Template Record. no drama 🫶
also, when you reuse a Template ID within an SCTP stream, the Template Withdrawal Message is guaranteed to arrive before the new definition of the Template. that means the Template Record can be sent right after the Template Withdrawal Message. in other words, the Template Reuse Delay restriction (5 seconds by default, as specified in [RFC5101]) doesn't need to be applied to Template ID reuse within the same SCTP stream. the wait is gone, babe ⏩💅
another perk of the new specs in this document is less load on the Collecting Process. it doesn't have to store Data Records while it waits for the Template Record, because the transmission order is always guaranteed. so you get extra reliability for your Data Records without extra burden on the Collector 🧚✨
3.3. Requirement for Data Set Buffering 🍓
3.3.1. IPFIX Protocol Specifications: Limitation
the [RFC5101] specs are pretty flexible about which SCTP stream(s) the Template Set, Options Template Set, and corresponding Data Sets get exported on. that means the (Options) Template Record might go out on a different SCTP stream from its Data Records. this can cause Data Record loss at the Collecting Process, because ordered transmission across SCTP streams isn't guaranteed 😵💫
for example, a Template Record could get held up waiting on reliable transmission on one SCTP stream, while its Data Records get sent right away on another. and because of different levels of SCTP stream congestion, even if both the Template Record and the Data Records are sent reliably, the Data Records on a different stream might still arrive before the Template Record. chaotic 💀
3.3.2. IPFIX Export Per SCTP Stream: Advantages
so if you export each Template Record and all its corresponding Data Records within a single SCTP stream, and you impose in-order transmission, the ordered-transmission-across-streams problem just... doesn't happen 🙌
when all the matching Data Records are on the same ordered SCTP stream as the Template Record, each stream becomes independent and self-contained. the only interaction between streams is the one between the Options Template and its Data Records sent on different streams. this has a bunch of good consequences: order preservation that doesn't block unrelated data, and less load on the Collecting Process. since Template Records are guaranteed to arrive before their Data Records, the Collector doesn't need to buffer Data Sets that belong to Templates it's still missing. unbothered and efficient 💜
4. Specifications 💖
this section lays out Exporting Process and Collecting Process behavior that's different from [RFC5101]. the goal is to get all the benefits of per-stream export 🌈
fair warning though: Exporting Processes that follow these specs will interoperate with [RFC5101]-compliant Collecting Processes. but Collecting Processes will have to follow extra non-interoperable specs to get the full benefits of the technique. these new specs add to the ones in [RFC5101], and they're written using the key words defined in [RFC2119] 📜
4.1. New Information Element 🧁
here's the new Information Element, dataRecordsReliability, in its full glory:
dataRecordsReliability
Description:
The export reliability of Data Records, within this SCTP
stream, for the element(s) in the Options Template scope. A
typical example of an element for which the export reliability
will be reported is the Template ID, as specified in the Data
Records Reliability Options Template. A value of 'True' means
that the Exporting Process MUST send any Data Records
associated with the element(s) reliably within this SCTP
stream. A value of 'False' means that the Exporting Process
MAY send any Data Records associated with the element(s)
unreliably within this SCTP stream.
Abstract Data Type: boolean
Data Type Semantics: identifier
ElementId: 276
Status: currentso, the TL;DR: a value of 'True' means the Exporting Process has to send the Data Records for those elements reliably within this SCTP stream. a value of 'False' means it's allowed to send them unreliably 🫣🌸
and per Section 6.1.5 of [RFC5101], the boolean data type gets encoded as a single octet. the value 1 means True, and the value 2 means False. simple and iconic 🦋✨
4.2. Template Management 💅✨
okay bestie, to actually use per-stream export, Exporting Processes MUST follow this section on top of Section 8, "Template Management", of [RFC5101]. 🎀
As [RFC5101] already says, Template Sets and Options Template Sets MUST be sent reliably. 🫶
Any Data Sets tied to a Template Record MUST be sent on the same SCTP stream that the Template Record went out on. 🧚
The Data Records Reliability Options Template is how the Exporting Process tells the Collecting Process which Templates live on each SCTP stream, and whether each batch of Data Records will be sent reliably or unreliably. 💜 After defining a Template ID, and before sending any associated Data Records on an SCTP stream, the Exporting Process MUST tell the Collecting Process whether those Data Records will be sent reliably or unreliably within that stream. 📣 It does this by sending a Data Record defined by the Data Records Reliability Options Template, for the Template the Data Records belong to. If it skips that step, the Collecting Process MUST disable this extension for the SCTP association. 😭 The one exception is the Data Records for the Data Records Reliability Options Template itself. They don't need an explicit notification, because they MUST always be sent reliably. 🔒
The Data Records Reliability Options Template MUST contain these Information Elements:
Scope: Template ID
Non-scope: dataRecordsReliabilityAfter sending a value of 'True' for the dataRecordsReliability Element, the Exporting Process MUST send any Data Records tied to the currently defined Template ID reliably within this SCTP stream. After sending a value of 'False', the Exporting Process MAY send Data Records for that Template ID unreliably within this SCTP stream. 🦋
Heads up: if the Exporting Process wants to change the Data Records Reliability value (reliable to unreliable, or unreliable to reliable) for Data Records on an SCTP stream, the Template MUST be withdrawn, and a new Template MUST be used. No shortcuts here, bestie! 🙅♀️
The Data Records Reliability Options Template MAY contain other non-scope Information Elements associated with the (Options) Template. 🌸
When an Options Template (including the Data Records Reliability Options Template) and its associated Data Records are sent in the same SCTP stream, the first associated Data Record can follow the Options Template immediately. ⚡ When they're sent in different SCTP streams, the Exporting Process SHOULD transmit the Options Template in advance of any Data Sets that use it. That way the Collector has the Options Template Record before the first associated Data Record shows up. 🍓
It is RECOMMENDED that the Exporting Process only sends a single Template and its corresponding Data Sets within a single SCTP stream, so the potential Data Record loss for that Template can be calculated. 📊 The Exporting Process MAY group related (Options) Templates and their associated Data Records within a single SCTP stream, so loss statistics are calculated for the group of Templates being sent unreliably within that stream. 🧩 This is a good fit when the Templates only differ slightly (for example, leaving out unavailable fields for export efficiency). It may also be necessary if the SCTP association doesn't support enough SCTP streams to give each Template its own stream. 🫠
If an SCTP stream has a mix of Data Records defined by Template Records and by Options Template Records, the Data Records defined by the Options Template Records SHOULD be sent reliably. That way the Collecting Process won't think any loss is tied to the Options Data Records. 🙈
4.3. SCTP 🌊
To take advantage of per-stream export, Exporting Processes MUST manage SCTP streams according to this section, in addition to Section 10.2.4.3, "Stream", of [RFC5101]. 💖
PR-SCTP [RFC3758] MUST be implemented by all compliant implementations. 👑
All IPFIX Messages in an SCTP stream MUST be sent in order. 🔢
As [RFC5101] describes, depending on the requirements of the application, the Exporting Process may send Data Sets with full or partial reliability. ✨
If the Exporting Process needs to export a new Template Record but there are no more free SCTP streams available, it SHOULD attempt to increase the number of outbound SCTP streams it can send to, per [RFC6525]. 📈 Alternatively, the Exporting Process MAY add the Template Set and Data Records to an existing SCTP stream, but that dilutes the granularity of any Data Record loss attribution. 🫤 Another option, which may result in the loss of Flow Records (for example, due to lack of buffering on the Exporter), is to restart the SCTP association with an increased number of SCTP streams. 🔄
4.4. Template Withdrawal Message 🗑️
To take advantage of per-stream export, Exporting Processes MUST send Template Withdrawal Messages according to this section, in addition to Section 8, "Template Management", of [RFC5101]. 💅
As [RFC5101] specifies, Templates that are no longer in use SHOULD be deleted. 🧹 Before reusing a Template ID, the Template MUST be deleted. To delete an allocated Template, it gets withdrawn using a Template Withdrawal Message. 📬
The Template Withdrawal Message MUST be sent on the same SCTP stream as the associated Template Record. 🛤️
The Template Withdrawal Message MUST be sent reliably, using SCTP-ordered delivery per [RFC5101]. 🔐 Since all IPFIX Messages are sent in order within an SCTP stream (per the specifications in this document), the IPFIX Message containing the Template Withdrawal Message will not arrive at the Collecting Process before any associated and previously sent Data Record. 🪄 So no Data Records will be lost due to delayed arrival at the Collecting Process. 🙌
The Template ID from a withdrawn Template MAY be reused on the same SCTP stream immediately after the Template Withdrawal Message is sent. This case is equivalent to using a Template Reuse Delay value of 0. ⏩
After reusing the Template ID, the Exporting Process MUST send a Data Record associated with the Data Records Reliability Options Template, to specify the reliability level of the Data Records associated with the new Template. 🎯
If the Template ID is going to be reused on a different SCTP stream, the new Template Record MUST NOT be sent before the Template Reuse Delay interval. ⏳
A Template Withdrawal Message that withdraws all Templates for the Observation Domain ID specified in the IPFIX Message header MUST NOT be used. 🚫
Multiple Template IDs MAY be withdrawn with a single Template Withdrawal Message, under the condition that all the Template IDs in that message are used on the same SCTP stream as the Template Withdrawal Message. 🧺
4.5. The Collecting Process's Side 🧑💻
Collecting Processes must operate in a fashion slightly contrary to [RFC5101] in order to realize the full benefits of per-stream export. 🙃 However, this section contains a mechanism that lets per-stream-capable Collecting Processes selectively enable per-stream export. That keeps them interoperable with Exporting Processes that don't implement per-stream export. 🤝
4.5.1. SCTP 🔌
As [RFC5101] specifies, the Collecting Process SHOULD listen for a new association request from the Exporting Process. The Exporting Process will request a number of SCTP streams to use for export. 📡
A Collecting Process SHOULD support the procedure for the addition of an SCTP stream specified in [RFC6525]. ➕
4.5.2. Enabling the Per-SCTP-Stream Extension 🔛
In IPFIX, there's no explicit notification of the Exporting Process's capabilities. 🤷♀️ There's also no return channel for the Collecting Process to communicate its own capabilities. 📭
When the Exporting Process is sending according to the per-SCTP-stream extension, the first Data Record received by the Collecting Process will be associated with the Data Records Reliability Options Template. ✅ In that case, the Collecting Process enables the extension for this Transport Session. Otherwise, the Collecting Process MUST NOT enable the extension for this Transport Session. 🚫
The Collecting Process MUST accept other non-scope Information Elements in the Data Records Reliability Options Template. 🫶
4.5.3. Disabling the Per-SCTP-Stream Extension 🔴
Nothing stops an implementation that doesn't meet the specification of the per-SCTP-stream extension from sending a Template that looks like a Data Records Reliability Options Template. 👀 So a Collecting Process MUST detect if the Exporting Process fails to meet the specification fully. 🕵️♀️ If any of the conditions below is met, the Exporting Process isn't using the per-SCTP-stream extension properly. The Collecting Process MUST then log an error message and disable this extension for the SCTP association. 🚨
- A Data Record is received before the appropriate Data Record associated with the Data Records Reliability Options Template has been received on the same SCTP stream (see Section 4.2). Note: Data Records associated with the Data Records Reliability Options Template are an exception to this rule.
- A Data Record associated with a Data Records Reliability Options Template is received on an SCTP stream for a (non-Options) Template that was defined on a different SCTP stream.
- A second Data Record associated with the Data Records Reliability Options Template is received for the same (Options) Template.
- A Data Record or a Template Withdrawal Message is associated with a Template that was defined on a different SCTP stream.
- Loss of Data Records is detected within a stream where a Data Record associated with the Data Records Reliability Options Template indicating unreliable transmission for any Template has not been received.
- A message is received with the SCTP
U(nordered)flag set to1(i.e., the message was sent unordered), even if it is processed in order. 🫨
4.5.4. Calculating Data Record Loss Per Template ✨
okay so you know the Sequence Number field in the IPFIX Message header? it goes up by the number of IPFIX Data Records in each message 🧚 As specified in [RFC5101], that's how a Collecting Process can spot out-of-sequence, dropped, or duplicate IPFIX Messages by tracking it. it's giving detective energy 🕵️♀️💖
When one or more sequential IPFIX Messages are considered lost, the number of lost Data Records is the Sequence Number of the first IPFIX Message Header after the lost packets (S2) minus the Sequence Number of the first lost IPFIX Message (S1). To find S1, you take the Sequence Number of the last IPFIX Message before the lost sequence (S0) and add the number of Data Records in that message (N0). the math is lowkey so clean 🫶
S1 = S0 + N0
loss = (S2 - S1) (mod(2^32))
= (S2 - (S0 + N0)) (mod(2^32))the mod(2^32) part matters bestie! 💅 The Sequence Number can wrap around while you're inside a run of lost IPFIX Messages, so modulo 2^32 arithmetic is required. If less than 2^32 Data Records are lost in a row (which can be assumed in practice), the equation gives you the exact number of lost Data Records. no cap, that's the whole point ✨
fun fact: using an unsigned32 type for the loss takes care of the mod(2^32) operation automatically 🎀
Since this Sequence Number is incremented per SCTP stream, you can calculate the loss of Data Records sent in that SCTP stream when you're doing partially reliable export 🦋 That loss can be attributed to the Data Records sent for the (Options) Template(s) whose records are being sent unreliably within that SCTP stream. main character energy for the Template that's actually getting dropped 👑
5. Resource Impact 🌸
okay so adding the new SCTP streams does need a message exchange, but it's way more lightweight to set up extra SCTP streams than to set up a whole new SCTP association 🫶 Adding SCTP stream(s) to an existing SCTP association only costs 16-24 more bytes, and that's allocated in the SCTP association a single time. Setting up a new SCTP association takes more overhead, so streams win 💖
For throughput, these specifications discourage multiplexing Templates and Data Records of different Template IDs, and that may lead to a slightly larger IPFIX Message overhead 🧁
If the Data Record rate is low for a specific Template (and hence a specific SCTP stream), the Exporting Process might not be able to fill the IPFIX Messages with Data Records associated with other Templates. in that situation there's potential overhead from the extra IPFIX Message headers and SCTP chunk headers 😭
Last thing: the processing overhead on the Exporter. a lot of state information has to be stored when a large number of SCTP streams are used within one SCTP association 💀 However, no comparison of the performance impact of multiple streams within an SCTP association versus opening the same number of independent SCTP associations is available. so we can't say which one is faster, lowkey a gap in the research 🙈
6. Examples 🧚
Figure 1 shows an example where SCTP stream 10 carries a Template Record with Template ID 257 transmitted with full reliability (FR), together with associated Data Records transmitted with partial reliability (PR). The Data Records Reliability Options Template with Template ID 256 is transmitted with full reliability. Its corresponding Data Set contains one Data Record. ✨
Record 1:
o Scope: Template ID = 257
o Non-scope: dataRecordsReliability = False +--------+ +---------+ +--------+
| | | | | |
stream 10 ----| Data | . . . | Data |---| Data |---...
| 257 | | 257 | | 256 |
| PR| | PR| | FR|
+--------+ +---------+ +--------+
+----------+ +-------------+
| | | Reliability |
| | | Options |
...---| Template |-------| Template |------>
| 257 | | 256 |
| FR| | FR|
+----------+ +-------------+
Figure 1Note that Template 257 will always be processed before the Data Records by the Collecting Process, because all IPFIX Messages are sent in order within an SCTP stream. the Collecting Process's job gets so much easier because of this 🙌 Furthermore, the Data Record loss for Template 257 can easily be calculated by the Collecting Process. 💅
If an Options Template is necessary to understand the content of a Data Record (i.e., the scope in the Options Template Record is an Information Element contained in the Data Record or associated with the Data Record), the Options Template Record should be sent in the same SCTP stream, as displayed in Figure 2. 🎀
+--------+ +--------+ +--------+
| | | | | |
stream 20 ----| Data |...| Data |-----| Data |--- ...
| 260 | | 260 | | 259 |
| PR| | PR| | FR|
+--------+ +--------+ +--------+
+--------+ +----------+
| | | |
...---| Data |-------| Template |---...
| 258 | | 260 |
| FR| | FR|
+--------+ +----------+
+----------+ +-------------+
| Options | | Reliability |
| Template | | Options |
...---| |-------| Template |------>
| 259 | | 258 |
| FR| | FR|
+----------+ +-------------+
Figure 2Figure 2 shows an example where SCTP stream 20 carries the following:
- a Data Records Reliability Options Template with Template ID 258, transmitted with full reliability.
- an Options Template Record with Template ID 259, transmitted with full reliability. This Options Template Record contains additional information related to the subsequent Data Records based on Template ID 260. Typical examples are the Common Properties information [RFC5473] or the Selector Report Interpretation [RFC5476].
- a Template Record with Template ID 260, transmitted with full reliability.
- a Data Set specified by the Reliability Options Template with Template ID 258, transmitted with full reliability.
The Data Set contains three Data Records:
Record 1, Record 2 and Record 3:
Record 1:
o Scope: Template ID = 258
o Non-scope: dataRecordsReliability = True
Record 2:
o Scope: Template ID = 259
o Non-scope: dataRecordsReliability = True
Record 3:
o Scope: Template ID = 260
o Non-scope: dataRecordsReliability = FalseThese Data Records tell the Collecting Process that the Data Records for Template IDs 258 and 259 are sent reliably, while the Data Records for Template ID 260 are not. Note that the first Data Record associated with the Data Record Reliability Options Template (Template ID 258) is not required and can be omitted. 🧠
- a Data Record specified by Template ID 259, transmitted with full reliability.
- a Data Record specified by Template ID 260, transmitted with partial reliability.
If the Collecting Process observes some Data Record loss using the Sequence Number, the loss can only stem from the Data Records associated with Template ID 260, as these are the only Data Records not exported reliably. So calculating loss per Template ID 260 is possible 🎯
Note that Options Templates 258, 259, and 260 will always arrive before their associated Data Records, respectively, because all IPFIX Messages must be sent in order within an SCTP stream. 💜
Figure 3 shows an example where SCTP stream 30 carries a Template Record with Template ID 262 transmitted with full reliability, an associated Data Record transmitted with full reliability, and a Template Withdrawal Message, followed by a redefinition of Template ID 262, and finally the Data Record associated with the new Template transmitted with partial reliability. The Template Withdrawal Message and the new definition of Template ID 262 are sent immediately, without waiting for the Template Reuse Delay interval. this part is so dramatic, the Template just gets a glow-up 🦋✨
+--------+ +----------+ +----------+
| | |Data | | |
stream 30 ... ---| Data |...| 261 |-----| Template |---
| 262 | |tmpID: 262| | 262 |
| PR| |dRR: False| | FR|
+--------+ +----------+ +----------+
+----------+ +--------+ +----------+
| Template | | | | Data |
...| Withdraw |-----| Data |-------| 261 |---...
| 262 | | 262 | |tmpID: 262|
| FR| | FR| |dRR: True|
+----------+ +--------+ +----------+
+----------+ +-------------+
| | | Reliability |
| Template | | Options |
...---| |-------| Template |------>
| 262 | | 261 |
| FR| | FR|
+----------+ +-------------+
dRR: Data Records Reliability
Figure 3The second Data Record associated with the Data Records Reliability Options Template shows that the Data Records associated with the newly specified Template ID 262 will be sent unreliably. 🙈
7. IANA Considerations 🌈
According to the process defined in [RFC5102], IANA has allocated the dataRecordsReliability Information Element (defined in Section 4.1) in the "IPFIX Information Elements" registry [IANA]. ✨
8. Security Considerations 🔐
The same security considerations as for the IPFIX protocol [RFC5101] apply. 💀
9. References 📚
9.1. Normative References 📚
Okay bestie, these are the must-reads 💅 Every one of them is needed to understand the spec, so no skipping!
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. Conrad, "Stream Control Transmission Protocol (SCTP) Partial Reliability Extension", RFC 3758, May 2004.
- [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", RFC 4960, September 2007.
- [RFC5101] Claise, B., Ed., "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information", RFC 5101, January 2008.
- [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, "Information Model for IP Flow Information Export", RFC 5102, January 2008.
- [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F. Raspall, "Sampling and Filtering Techniques for IP Packet Selection", RFC 5475, March 2009.
- [RFC6525] Stewart, R., Tuexen, M., and P. Lei, "Stream Control Transmission Protocol (SCTP) Stream Reconfiguration", RFC 6525, February 2012.
9.2. Informative References 🧚
These ones are the extra context, super helpful but not strictly required 🌸
- [IANA] IPFIX Information Elements Registry, <http://www.iana.org/assignments/ipfix>.
- [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander, "Requirements for IP Flow Information Export (IPFIX)", RFC 3917, October 2004.
- [RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, "Architecture for IP Flow Information Export", RFC 5470, March 2009.
- [RFC5472] Zseby, T., Boschi, E., Brownlee, N., and B. Claise, "IP Flow Information Export (IPFIX) Applicability", RFC 5472, March 2009.
- [RFC5473] Boschi, E., Mark, L., and B. Claise, "Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports", RFC 5473, March 2009.
- [RFC5474] Duffield, N., Ed., Chiou, D., Claise, B., Greenberg, A., Grossglauser, M., and J. Rexford, "A Framework for Packet Selection and Reporting", RFC 5474, March 2009.
- [RFC5476] Claise, B., Ed., Johnson, A., and J. Quittek, "Packet Sampling (PSAMP) Protocol Specifications", RFC 5476, March 2009.
- [RFC5477] Dietz, T., Claise, B., Aitken, P., Dressler, F., and G. Carle, "Information Model for Packet Sampling Exports", RFC 5477, March 2009.
10. Acknowledgments 🫶
The authors want to give a huge shoutout to Brian Trammell for his expert feedback and his nonstop effort to make the specs better ✨ Elisa Boschi gets thanked for her thorough reading 📖 Randall Stewart, Peter Lei, and Michael Tuexen get props for their SCTP feedback and expertise 👑 And Tobias Limmer, thank you for being part of the squad 💜
Authors' Addresses 🎀
Benoit Claise
Benoit Claise
Cisco Systems, Inc.
De Kleetlaan 6a b1
Diegem 1813
Belgium
Phone: +32 2 704 5622
EMail: bclaise@cisco.comPaul Aitken
Cisco Systems, Inc.
96 Commercial Quay
Commercial Street
Edinburgh, EH6 6LX, United Kingdom
Phone: +44 131 561 3616
EMail: paitken@cisco.comAndrew Johnson
Cisco Systems, Inc.
96 Commercial Quay
Commercial Street
Edinburgh, EH6 6LX, United Kingdom
Phone: +44 131 561 3641
EMail: andrjohn@cisco.comGerhard Muenz
Technische Universitaet Muenchen
Department of Informatics - I8
Boltzmannstr. 3
Garching D-85748
DE
EMail: muenz@net.in.tum.de
URI: http://www.net.in.tum.de/~muenzThat's the whole RFC, bestie! 🌈💖 The squad that wrote it is seriously iconic 😭🫶
📦 where this page comes from
- ietf: rfc6526 2012-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms