ipfixcol2-fds-output(7)

💖 ipfixcol2-fds-output(7) — the FDS output plugin, bestie edition 💖

okay so first things first: this is a page from the IPFIXcol collector manual, and the plugin we're talking about is ipfixcol2-fds-output. It's giving Flow Data Storage, and it's an output plugin. Buckle up, babes. 🎀

---

✨ Name ✨

ipfixcol2-fds-output — Flow Data Storage (output plugin), periodt. 💅

---

🌸 Description 🌸

So this plugin takes IPFIX Data Records, converts them, and stores them in the FDS file format. And the file format is based on IPFIX, which is highkey the moment because it's a super-effective way to do long-term storage. It keeps complete flow records — and yes bestie, that means all the Enterprise-specific fields, biflow, literally everything — along with the identification of the flow exporters who exported those records. Obsessed. 🫶

All the data goes into flat files, and those files get automatically rotated and renamed every N minutes (by default, 5 minutes). Cute, right? 🦋

⚠️ Warning: The plugin is still under development and some incompatible changes in file format may be introduced! Not me warning you so you don't get jumpscared later. 😭

---

🧚 Example configuration 🧚

<output>
    <name>FDS output</name>
    <plugin>fds</plugin>
    <params>
        <storagePath>/tmp/ipfixcol2/fds/</storagePath>
        <compression>none</compression>
        <dumpInterval>
            <timeWindow>300</timeWindow>
            <align>yes</align>
        </dumpInterval>
        <outputSelection>
            <element>iana:octetDeltaCount</element>
            <element>iana@reverse:octetDeltaCount@reverse</element>
            <element>iana:packetDeltaCount</element>
            <element>iana@reverse:packetDeltaCount@reverse</element>
            <element>iana:destinationIPv4Address</element>
            <element>iana:destinationIPv6Address</element>
            <element>iana:sourceIPv4Address</element>
            <element>iana:sourceIPv6Address</element>
            <element>iana:destinationTransportPort</element>
            <element>iana:sourceTransportPort</element>
            <element>iana:protocolIdentifier</element>
            <element>cesnet:quicSNI</element>
            <element>cesnet:TLSSNI</element>
        </outputSelection>
    </params>
</output>

---

👑 Parameters 👑

• storagePath — this is where you tell the plugin where to keep your data files. 💾 It's the storage directory, and bestie, the path must already exist on your system — otherwise literally no files get stored. (Tragic. 💀) Every file lands in there based on the configuration, following this template: <storagePath>/YYYY/MM/DD/flows.<ts>.fds, where YYYY/MM/DD means year/month/day and <ts> is a UTC timestamp in the format YYMMDDhhmmss.
• compression — data compression helps significantly reduce the size of your output files. Slay. 🎀 Here are the algorithms available:
  • none — compression disabled [default] 🙈
  • lz4 — LZ4 compression (very fast, slightly worse compression ratio) ⚡
  • zstd — ZSTD compression (slightly slower, good compression ratio) 🌈
• dumpInterval — the configuration for output file rotation. 🔄
  • timeWindow — specifies the time interval in seconds to rotate files, i.e. close the current file and create a new one. [default: 300]
  • align — aligns file rotation with the next N minute interval. For example, if it's enabled and the window size is 5 minutes long, files will be created at 0, 5, 10, etc. [values: yes/no, default: yes]
• outputSelection — lets you select only a subset of fields to be saved in the output. You provide a list of IPFIX elements, and any fields of an element that are not part of this list get skipped and won't be included in the resulting FDS output file. 👀 Note that when [default: all fields]
  • element — the identifier of the IPFIX element, in the form SCOPE_NAME:ELEMENT_NAME, for example iana:sourceIPv4Address. The SCOPE_NAME: prefix can be omitted in case of "iana:". 📝 Note: reverse elements use the following naming pattern: iana@reverse:sourceIPv4Address@reverse
• asyncIO — allows the use of asynchronous I/O for writing to the file. Normally, when parts of the file are being written, the process is blocked on synchronous I/O and just waits for the operation to complete (boring, ngl). But asynchronous I/O lets the plugin simultaneously write to the file and process flow records, which significantly improves overall performance. 💅 (Note: a pool of service threads shared among instances of the FDS plugin might be created). [values: true/false, default: true]

---

💜 Author 💜

Lukáš Huták (lukas.hutak@cesnet.cz)

---

🌷 Copyright 🌷

Copyright © 2019 CESNET, z.s.p.o.

---

okay and that's the whole page, bestie!! 🫶 this is version 2.0, dated 2019-07-01, and it's the ipfixcol2-fds-output(7) page from the IPFIXcol collector manual. go forth and store those flows, you absolute icon. 💖✨🎀🦋🌸