ipfixcol2-json-output(7)

💖 NAME 💖

okay bestie, welcome to the manual page for ipfixcol2-json-output, section 7, from the IPFIXcol collector. It's giving output plugin energy. ipfixcol2-json-output — JSON (output plugin). That's the whole name, periodt. 💅

🌸 DESCRIPTION 🌸

okay so first things first: this is a special version of the JSON output plugin for Apache Kafka. The plugin is literally a copy of the original plugin, so it offers the same formatting options, however, it provides only Kafka output. No cap.

Peep the "Kafka notes" section below for more information. 👀

The plugin converts IPFIX flow records into JSON format and sends them to Apache Kafka. Some elements (timestamps, IP/MAC addresses, TCP flags, etc.) can be converted into human readable format. Others are represented as plain numbers. So some fields get the glow-up, others stay raw digits. It's a vibe. ✨

Each flow record is converted individually and sent to outputs where at least one output method must be configured. All fields in an IPFIX record are formatted as: <field name>:<value>, where the field name represents a name of a corresponding Information Element as "<Enterprise Name>:<Information Element Name>" and value is a number, string, boolean or null (if conversion from IPFIX to JSON fails or it is a zero-length field). The Enterprise Name is a name of an organization that manages the field, such as IANA, CISCO, etc. iykyk.

If the field name and type is unknown and ignoreUnknown option is disabled, the field name use format 'enXX:idYY' where XX is an Enterprise Number and YY is an Information Element ID. See notes for instructions on how to add missing definitions of Information Elements. Important little detail, bestie!

Flow records with structured data types (basicList, subTemplateList and subTemplateMultiList) are supported. See the particular section below for information about the formatting of these types. Those are the fancy structured girlies. 🦋

✨ EXAMPLE OUTPUT ✨

The following output is for illustration purpose only. Available IPFIX fields depend on a structure of records that your exporter sends. See documentation of the exporter, if you want to change available IPFIX fields. So don't come for me if your fields look different — it's all about your exporter's structure, periodt. 💖

{
    "@type": "ipfix.entry",
    "iana:octetDeltaCount": 3568720,
    "iana:packetDeltaCount": 2383,
    "iana:flowStartMilliseconds": "2018-05-11T19:44:29.006Z",
    "iana:flowEndMilliseconds": "2018-05-11T19:44:30.364Z",
    "iana:ingressInterface": 1,
    "iana:ipVersion": 4,
    "iana:sourceIPv4Address": "192.168.0.100",
    "iana:destinationIPv4Address": "192.168.0.101",
    "iana:ipClassOfService": 0,
    "iana:ipTTL": 62,
    "iana:protocolIdentifier": "TCP",
    "iana:tcpControlBits": ".AP.S.",
    "iana:sourceTransportPort": 47200,
    "iana:destinationTransportPort": 20810,
    "iana:egressInterface": 0,
    "iana:samplingInterval": 0,
    "iana:samplingAlgorithm": 0
}

The plugin can produce multiple record types, which are always distinguished by "@type" key:

• ipfix.entry — Flow record captured by the exporter as shown in the example above. This record is always converted and it is the most common type. Main character record, fr.
• ipfix.optionsEntry — Record containing additional information provided by the exporter to the collector. Usually it is used for information about the exporter configuration (such as packet sampling) and flow export statistics. Periodicity and content of the record always depends on the particular exporter. By default, record conversion is disabled, see ignoreOptions parameter in the configuration section. Lowkey extra, but useful.
• ipfix.template and ipfix.optionsTemplate — Records describing structure of flow records (i.e. ipfix.entry) and options records (i.e. ipfix.optionsEntry), respectively. These types are usually useful only for exporter analytics. Periodicity of these records usually depends on the exporter and used transport protocol. For example, for connection over UDP it corresponds to "template refresh interval" configured on the exporter. However, for TCP and SCTP sessions, the template records are usually sent only once immediately after session start. By default, template record conversion is disabled, see templateInfo parameter in the configuration section. Exporter analytics girlies, this one's for you.

💅 EXAMPLE CONFIGURATION 💅

Below you see a complex configuration with all available output options enabled. Don't forget to remove (or comment) outputs that you don't want to use! Seriously, don't leave outputs you don't want — it's giving messy. 😭

<output>
    <name>JSON output</name>
    <plugin>json-kafka</plugin>
    <params>
        <tcpFlags>formatted</tcpFlags>
        <timestamp>formatted</timestamp>
        <protocol>formatted</protocol>
        <ignoreUnknown>true</ignoreUnknown>
        <ignoreOptions>true</ignoreOptions>
        <nonPrintableChar>true</nonPrintableChar>
        <octetArrayAsUint>true</octetArrayAsUint>
        <numericNames>false</numericNames>
        <splitBiflow>false</splitBiflow>
        <detailedInfo>false</detailedInfo>
        <templateInfo>false</templateInfo>

        <outputs>
            <kafka>
                <name>Send to Kafka</name>
                <brokers>127.0.0.1</brokers>
                <topic>ipfix</topic>
                <blocking>false</blocking>
                <partition>unassigned</partition>

                <!-- Zero or more additional properties -->
                <property>
                    <key>compression.codec</key>
                    <value>lz4</value>
                </property>
            </kafka>
        </outputs>
    </params>
</output>

👑 PARAMETERS 👑

okay bestie, this is where the actual knobs are. buckle up 💅

✨ Formatting parameters ✨

These control how the pretty JSON gets written out. Pick your fighter:

• tcpFlags — do you want your TCP flags as the cute common textual form (like ".A..S.") or as a plain number (raw)? [values: formatted/raw, default: formatted]
• timestamp — want that ISO 8601 textual glow-up (every timestamp in UTC, milliseconds included, e.g. "2018-01-22T09:29:57.828Z"), or a unix timestamp (again, all timestamps in milliseconds)? [values: formatted/unix, default: formatted]
• protocol — formatted style means you get "TCP" instead of a boring 6; raw means you get the number, no glamour. [values: formatted/raw, default: formatted]
• ignoreUnknown — skip unknown Information Elements, aka record fields with a name and data type nobody recognizes 🙈. If you turn this off, the data of unknown elements gets formatted as unsigned integer or hexadecimal values. Wanna know more? go peek at the octetArrayAsUint option. [values: true/false, default: true]
• ignoreOptions — skip non-flow records that are just reporting metadata about IPFIX Exporting and Metering Processes (i.e. records described by Options Templates). [values: true/false, default: true]
• nonPrintableChar — ignore non-printable characters (newline, tab, control characters, etc.) inside IPFIX strings. Turn it off and those characters get escaped on output instead. [values: true/false, default: true]
• octetArrayAsUint — converts each IPFIX field with octetArray type (and yes, that includes IPFIX fields with unknown definitions) as an unsigned integer if the field is 8 bytes or smaller. Anything bigger than that limit always gets converted to a string holding the hexadecimal value, which is typically in network byte order (e.g. "0x71E1"). Keep in mind there can be octetArray fields with variable length that might get interpreted differently depending on their size, so don't come for me if the vibe shifts 😭. If you disable this, octetArray fields are never interpreted as unsigned integers. [values: true/false, default: true]
• numericNames — use only the short identification of Information Elements (i.e. "enXX:idYY"). If you enable it, the short version is used even when the field's definition is known. This one can help you build shorter JSON records with key identifiers that are independent of the internal configuration. [values: true/false, default: false]
• splitBiflow — got Biflow records? split each one into two unidirectional flow records. Non-biflow records are untouched, they're living their best life. [values: true/false, default: false]
• detailedInfo — sprinkle extra info about the IPFIX message each record belongs to (export time, Observation Domain ID, IP address of the exporter, etc.) into the output. Those extra fields all start with the "ipfix:" prefix. [values: true/false, default: false]
• templateInfo — convert Template and Options Template records. Scroll down to the particular section below for how those records get formatted. [values: true/false, default: false]

---

🌈 Output types 🌈

At least one output must be configured, no skipping. Multiple kafka outputs can run at the same time — but only if the outputs are not in collision with each other, so play nice ✨

kafka — send data to Kafka, aka Kafka producer 💖

• name — identification name of the output. Used only for readability.
• brokers — initial list of brokers, given as a CSV list of broker "host" or "host:port".
• topic — Kafka topic to produce to.
• partition — partition number to produce to. If the value is "unassigned", then the default random distribution is used. [default: "unassigned"]
• brokerVersion — older broker versions (before 0.10.0) give a client no way to query for supported protocol features, which makes it impossible for the client to know what features it may use. As a workaround, a user may set this property to the expected broker version and the client will automatically adjust its feature set. [default: <empty>]
• blocking — enable blocking on produce. If it's disabled and a cluster is down or can't retrieve records fast enough, some flow records may be dropped 💀. On the other hand, if it's enabled, no records are dropped. However, if the cluster is slow or not accessible at all, the plugin waits (i.e. blocks) until data are send. This can significantly slow down or block(!) the whole collector and other output plugins. [true/false, default: false]
• performanceTuning — by default, the connection provided by librdkafka is not optimized for the high throughput you need for transporting JSON records. This option adds optional library parameters, which reduces messaging overhead and significantly improves throughput. In particular, Kafka message capacity is increased and maximal buffering interval is prolonged. These options can be overwritten by user defined properties. [true/false, default: true]
• property — additional configuration properties of the librdkafka library as key/value pairs. Multiple <property> parameters, which can improve performance, can be defined. See the project website for the full list of supported options. Keep in mind that some options might not be available in all versions of the library. 👑

---

📝 NOTES 📝

If one or more Information Element definitions are missing, you can easily add them. All definitions are provided by the libfds <https://github.com/CESNET/libfds/> library. See the project website for help. 🫶

If a flow record contains multiple occurrences of the same Information Element, their values are stored into a single name/value pair as a JSON array. The order of the values in the array corresponds to their order in the flow record. no cap.

For higher performance, it is advisable to use non-formatted conversion of IPFIX data types. In that case you should prefer, for example, timestamps as numbers over ISO 8601 strings, and numeric identifiers of fields, since they are usually shorter. ✨

🦋 STRUCTURED DATA TYPES 🦋

Flow records can be extended with structured data types (as described in RFC6313). Each of these types gets formatted as a JSON object with a "@type" field, which helps distinguish its formatting. Moreover, as the standard describes, the semantic of the list is also included. iconic tbh.

Converted basicList contains "fieldID" with the Information Element identifier of zero or more Information Element(s) contained in the list. All values are stored as a JSON array in the "data" field.

{
    "example:blField": {
        "@type": "basicList",
        "semantic": "anyOf",
        "fieldID": "iana:octetDeltaCount",
        "data": [23, 34, 23]
    }
}

Converted subTemplateList contains only an additional "data" field with an array of zero or more JSON objects. As all nested JSON objects are described by the same IPFIX Template, it's guaranteed that their structure is also the same. The "semantic" field indicates the relationship among the different JSON objects.

{
    "example:stlField": {
        "@type": "subTemplateList",
        "semantic": "allOf",
        "data": [
            {"keyA.1": "value1", "keyA.2": "value2"},
            {"keyB.1": "value1", "keyB.2": "value2"},
        ]
    }
}

Converted subTemplateMultiList is similar to the previous type, however, sub-records can be even more nested. The "data" field contains a JSON array with zero or more nested JSON arrays. Each nested array contains zero or more JSON objects, and it's guaranteed that JSON objects in the same array have the same structure. The "semantic" field indicates the top-level relationship among the nested arrays.

{
    "example:stmlField": {
        "@type": "subTemplateMultiList",
        "semantic": "allOf",
        "data" : [
            [
                {"keyA.1": "value1", "keyA.2": "value2"},
                {"keyB.1": "value1", "keyB.2": "value2"},
            ],
            [
                {"idA.1": "something", "idB.1": 123}
            ]
        ]
    }
}

Keep in mind that all these structures can be nested in each other. it's giving inception, honestly 🌀

🎀 Template and Options Template records 🎀

okay so listen bestie, these records only get converted if you've got the <templateInfo> option turned on. no option, no templates, that's the deal. 💅

A Template record is literally just a blueprint — it describes the structure of flow records — and its "@type" is "ipfix.template". A converted Template record carries the "ipfix:templateId" field, which is unique to the Transport Session and the Observation Domain (iykyk), plus "ipfix:fields", which is an array of JSON objects spelling out the fields of flow records.

{
    "@type": "ipfix.template",
    "ipfix:templateId": 49171,
    "ipfix:fields": [
        {
            "ipfix:elementId": 16399,
            "ipfix:enterpriseId": 6871,
            "ipfix:fieldLength": 1
        }, {
            "ipfix:elementId": 184,
            "ipfix:enterpriseId": 29305,
            "ipfix:fieldLength": 4
        }, {
            ...
        }
    ]
}

An Options Template record describes the structure of additional info for the collector — think sampling configuration, export statistics, etc. 🌸 The description gets converted into a record whose "@type" equals "ipfix.optionsEntry". A converted Options Template record is basically the same vibe as the previous type, except it also has an "ipfix:scopeCount" field, which tells you the number of scope fields in the record. A scope count of N means the first N field specifiers are scope fields. noted. ✨

{
    "@type": "ipfix.optionsTemplate",
        "ipfix:templateId": 53252,
        "ipfix:scopeCount": 1,
        "ipfix:fields": [
            {
                    "ipfix:elementId": 130,
                    "ipfix:enterpriseId": 0,
                    "ipfix:fieldLength": 4
            }, {
                    "ipfix:elementId": 103,
                    "ipfix:enterpriseId": 6871,
                    "ipfix:fieldLength": 4
            }, {
            ...
        ]
    }
}

---

🦋 Kafka notes 🦋

okay this part is important so pls pay attention. 👀

This plugin depends on librdkafka, and librdkafka unfortunately redefines some common symbols (like thrd_create) when it's built without support for C11 threads. Usually that's chill, not a problem at all — BUT the expected return codes of thread functions differ from the common glibc/musl implementation. 💀

Here's the whole saga: the JSON plugin and the librdkafka library get loaded after the collector starts, so the common version of the symbols is what actually gets used, and the redefined ones just get ignored. Then later, when librdkafka tries to create its internal processing threads, it fails — because thrd_create() returned an unexpected return code. not the moment bestie wanted. 😭

You can resolve this by using the RTLD_DEEPBIND flag of the dlopen() function when loading the plugin. However — and this is a big however — that flag is a GNU extension, so it's not supported by all C libraries, and it can also break some other standard functions (like C++ iostream). so, tradeoffs. 🙈

Affected versions of the library are available in the official package repositories of multiple distributions. Which is why we made this special version of the JSON output that instructs the collector core to use the RTLD_DEEPBIND flag. slay, I guess. 👑

This issue should be fixed since librdkafka v1.4.2. For more information, see librdkafka issue #2681 on Github.

---

👑 Author 👑

Lukáš Huták (lukas.hutak@cesnet.cz), Petr Velan (petr.velan@cesnet.cz)

💖 Copyright 💖

Copyright © 2020 CESNET, z.s.p.o.

---

and that's the whole page, bestie!! 🫶 this one's from version 2.2, dated 2020-07-10, and it lives at ipfixcol2-json-output(7) in section 7 of the manual. go forth and export ur flows, you got this. 🌈✨🎀