ipfixcol2-json-output(7)

hey bestie 💖 welcome to the manual page for ipfixcol2-json-output(7) from the IPFIXcol collector. It’s the JSON output plugin, and she’s about to make your flow records cute and readable. ✨

🌸 NAME 🌸

ipfixcol2-json-output - JSON (output plugin) 🎀

🦋 DESCRIPTION 🦋

So basically, this plugin takes IPFIX flow records, converts them into JSON format, and sends them to remote destinations over the network, stores them into files, or prints them on standard output. Some elements (timestamps, IP/MAC addresses, TCP flags, etc.) can be converted into human readable format. Others are represented as plain numbers.

Each flow record is converted individually and sent to outputs, and here’s the rule: at least one output method must be configured. No outputs, no vibes. 💀

⚠️ bestie, you need at least one output method configured. That’s not optional.

All fields in an IPFIX record are formatted as <field name>:<value>, where the field name represents a name of a corresponding Information Element as "<Enterprise Name>:<Information Element Name>" and value is a number, string, boolean or null (if conversion from IPFIX to JSON fails or it is a zero-length field). The Enterprise Name is a name of an organization that manages the field, such as IANA, CISCO, etc.

If the field name and type is unknown and ignoreUnknown option is disabled, the field name use format enXX:idYY where XX is an Enterprise Number and YY is an Information Element ID. See notes for instructions on how to add missing definitions of Information Elements.

Flow records with structured data types (basicList, subTemplateList and subTemplateMultiList) are supported. See the particular section below for information about the formatting of these types.

📦 EXAMPLE OUTPUT 📦

The following output is for illustration purpose only. Available IPFIX fields depend on a structure of records that your exporter sends. See documentation of the exporter, if you want to change available IPFIX fields.

{
    "@type": "ipfix.entry",
    "iana:octetDeltaCount": 3568720,
    "iana:packetDeltaCount": 2383,
    "iana:flowStartMilliseconds": "2018-05-11T19:44:29.006Z",
    "iana:flowEndMilliseconds": "2018-05-11T19:44:30.364Z",
    "iana:ingressInterface": 1,
    "iana:ipVersion": 4,
    "iana:sourceIPv4Address": "192.168.0.100",
    "iana:destinationIPv4Address": "192.168.0.101",
    "iana:ipClassOfService": 0,
    "iana:ipTTL": 62,
    "iana:protocolIdentifier": "TCP",
    "iana:tcpControlBits": ".AP.S.",
    "iana:sourceTransportPort": 47200,
    "iana:destinationTransportPort": 20810,
    "iana:egressInterface": 0,
    "iana:samplingInterval": 0,
    "iana:samplingAlgorithm": 0
}

The plugin can produce multiple record types, which are always distinguished by "@type" key:

• ipfix.entry — Flow record captured by the exporter as shown in the example above. This record is always converted and it is the most common type.
• ipfix.optionsEntry — Record containing additional information provided by the exporter to the collector. Usually it is used for information about the exporter configuration (such as packet sampling) and flow export statistics. Periodicity and content of the record always depends on the particular exporter. By default, record conversion is disabled, see ignoreOptions parameter in the configuration section.
• ipfix.template and ipfix.optionsTemplate — Records describing structure of flow records (i.e. ipfix.entry) and options records (i.e. ipfix.optionsEntry), respectively. These types are usually useful only for exporter analytics. Periodicity of these records usually depends on the exporter and used transport protocol. For example, for connection over UDP it corresponds to "template refresh interval" configured on the exporter. However, for TCP and SCTP sessions, the template records are usually sent only once immediately after session start. By default, template record conversion is disabled, see templateInfo parameter in the configuration section.

🎀 EXAMPLE CONFIGURATION 🎀

Below you see a complex configuration with all available output options enabled. Don’t forget to remove (or comment) outputs that you don’t want to use!

<output>
    <name>JSON output</name>
    <plugin>json</plugin>
    <params>
        <tcpFlags>formatted</tcpFlags>
        <timestamp>formatted</timestamp>
        <protocol>formatted</protocol>
        <ignoreUnknown>true</ignoreUnknown>
        <ignoreOptions>true</ignoreOptions>
        <nonPrintableChar>true</nonPrintableChar>
        <octetArrayAsUint>true</octetArrayAsUint>
        <numericNames>false</numericNames>
        <splitBiflow>false</splitBiflow>
        <detailedInfo>false</detailedInfo>
        <templateInfo>false</templateInfo>

        <outputs>
            <!-- Choose one or more of the following outputs -->
            <server>
                <name>Local server</name>
                <port>8000</port>
                <blocking>no</blocking>
            </server>

            <send>
                <name>Send to my server</name>
                <ip>127.0.0.1</ip>
                <port>8000</port>
                <protocol>tcp</protocol>
                <blocking>no</blocking>
            </send>

            <print>
                <name>Printer to standard output</name>
            </print>

            <file>
                <name>Store to files</name>
                <path>/tmp/ipfixcol/flow/%Y/%m/%d/</path>
                <prefix>json.</prefix>
                <timeWindow>300</timeWindow>
                <timeAlignment>yes</timeAlignment>
                <compression>none</compression>
            </file>

            <kafka>
                <name>Send to Kafka</name>
                <brokers>127.0.0.1</brokers>
                <topic>ipfix</topic>
                <blocking>false</blocking>
                <partition>unassigned</partition>

                <!-- Zero or more additional properties -->
                <property>
                    <key>compression.codec</key>
                    <value>lz4</value>
                </property>
            </kafka>
        </outputs>
    </params>
</output>

okay bestie, buckle up — this is the part where we get into the knobs and dials 💅 every single one of them, no skipping, we don't do that here.

✨ PARAMETERS ✨

🎀 Formatting parameters 💖

these control how your stuff gets cooked. pick your vibe, but pick wisely, iykyk:

• tcpFlags — turns TCP flags into the cute human-readable format (something like .A..S. 😌) or just spits out a plain number (raw). [values: formatted/raw, default: formatted]
• timestamp — converts timestamps into ISO 8601 text (everything in UTC and milliseconds, like "2018-01-22T09:29:57.828Z" ✨) OR into a unix timestamp (and yes, all timestamps in milliseconds). [values: formatted/unix, default: formatted]
• protocol — turns the protocol identification into the formatted style (like bestie, instead of 6 it literally writes "TCP") or into a plain number. [values: formatted/raw, default: formatted]
• ignoreUnknown — skip those unknown Information Elements (aka record fields where the name and data type are a mystery 🙈). If you turn this OFF, the data of unknown elements get formatted as unsigned integer or hexadecimal values. wanna know more? go peek at the octetArrayAsUint option. [values: true/false, default: true]
• ignoreOptions — skip the non-flow records that report metadata about IPFIX Exporting and Metering Processes (basically records described by Options Templates). [values: true/false, default: true]
• nonPrintableChar — ignore non-printable characters (newline, tab, control characters, etc.) living inside IPFIX strings. If you disable this, those characters get escaped on output instead. [values: true/false, default: true]
• octetArrayAsUint — converts each IPFIX field with octetArray type (including IPFIX fields with unknown definitions) as an unsigned integer, but only if the field size is less than or equal to 8 bytes. Fields bigger than that limit are always converted as a string showing the hexadecimal value, which is typically in network byte order (e.g. "0x71E1"). ⚠️ keep in mind that octetArray fields with variable length might get interpreted differently depending on their size. If you disable it, octetArray fields are never interpreted as unsigned integers, periodt. [values: true/false, default: true]
• numericNames — use only the short identification of Information Elements (like enXX:idYY). If enabled, the short version gets used even when the field's definition is actually known. this option is lowkey a lifesaver for making shorter JSON records with key identifiers that don't depend on the internal configuration 💅 [values: true/false, default: false]
• splitBiflow — when you've got Biflow records, split that record into two unidirectional flow records 🦋. Non-biflow records are literally unaffected. [values: true/false, default: false]
• detailedInfo — adds extra info about the IPFIX message (like export time, Observation Domain ID, IP address of the exporter, etc.) that each record belongs to. those extra fields start with the ipfix: prefix. [values: true/false, default: false]
• templateInfo — convert Template and Options Template records. for the deets on how these records get formatted, scroll down to the particular section below. [values: true/false, default: false]

---

🌈 Output types 👑

At least one of the following output must be configured. Multiple server/send/file/kafka outputs can be used at the same time — only if the outputs are not in collision with each other, bestie.

🖥️ server

TCP (push) server gives you data on a local port. Converted records get automatically send to all clients connected to that port. wanna test the server? use, for example, the ncat(1) utility: ncat <server ip> <port> 🌸

• name — Identification name of the output. Used only for readability.
• port — Local port number of the server.
• blocking — Enable blocking on TCP sockets (true/false). If blocking mode is disabled and a client can't retrieve records fast enough, some flow records may be dropped (only individual clients are affected, so like, not the whole party). On the other hand, if blocking mode is enabled, no records are dropped. However — if at least one client is slow, the plugin waits (i.e. blocks) until data are send. this can significantly slow down the whole collector AND other output plugins, because processing records is suspended. and in the worst-case scenario? if the client isn't responding at all, the whole collector is blocked! 💀 Therefore, it's usually preferred (and much safer) to disable blocking.

📨 send

Send records over the network to a client. If the destination isn't reachable or the client is disconnected, the plugin drops all records and tries to reconnect every 5 seconds. and just like with server, you can verify functionality using the ncat(1) utility: ncat -lk <local ip> <local port> ✨

• name — Identification name of the output. Used only for readability.
• ip — IP address or domain name of the client
• port — Remote port number
• protocol — Transport protocol: TCP or UDP (this field is case insensitive, so don't stress the caps 🫶)
• blocking — Enable blocking on a socket (true/false). See the description of this property at the server output.

📁 file

Store data to files.

• name — Identification name of the output. Used only for readability.
• path — The path specifies storage directory for data collected by the plugin. Format specifiers for day, month, etc. are supported, so you can create a suitable directory hierarchy. See strftime for conversion specification. (Note: UTC time 🕐)
• prefix — Specifies name prefix for output files.
• timeWindow — Specifies the time interval in seconds to rotate files [minimum 60, default 300]
• timeAlignment — Align file rotation with next N minute interval (yes/no).
• compression — Data compression helps to significantly reduce size of output files. Following compression algorithms are available:
  • none — Compression disabled [default]
  • gzip — GZIP compression

🐘 kafka

Send data to Kafka i.e. Kafka producer.

⚠️ Warning: Library librdkafka < 1.5.0 contains a bug which might prevent the plugin from starting. If so, use json-kafka output plugin which implements workaround for this issue. 😭

• name — Identification name of the output. Used only for readability.
• brokers — Initial list of brokers as a CSV list of broker host or host:port.
• topic — Kafka topic to produce to.
• partition — Partition number to produce to. If the value is "unassigned", then the default random distribution is used. [default: "unassigned"]
• brokerVersion — Older broker versions (before 0.10.0) provide no way for a client to query for supported protocol features, making it impossible for the client to know what features it may use. As a workaround, a user may set this property to the expected broker version and the client will automatically adjust its feature set. [default: <empty>]
• blocking — Enable blocking on produce. If disabled and a cluster is down or not able to retrieve records fast enough, some flow records may be dropped. On the other hand, if enabled, no records are dropped. However — if the cluster is slow or not accessible at all, the plugin waits (i.e. blocks) until data are send. This can significantly slow down or block(!) the whole collector and other output plugins [true/false, default: false]
• performanceTuning — By default, the connection provided by librdkafka is not optimized for the high throughput required for transport of JSON records. This option adds optional library parameters, which reduces messaging overhead and significantly improves throughput 🔥. In particular, Kafka message capacity is increased and maximal buffering interval is prolonged. These options can be overwritten by user defined properties. [true/false, default: true]
• property — Additional configuration properties of the librdkafka library as key/value pairs. Multiple <property> parameters, which can improve performance, can be defined. See the project website for the full list of supported options. Keep on mind that some options might not be available in all versions of the library.

🖨️ print

Write data on standard output.

• name — Identification name of the output. Used only for readability.

---

📝 NOTES 📝

If one or more Information Element definitions are missing, you can easily add them 🫶. All definitions are provided by the libfds <https://github.com/CESNET/libfds/> library. See the project website for help.

If a flow record contains multiple occurrences of the same Information Element, their values are stored into a single name/value pair as a JSON array. Order of the values in the array corresponds to their order in the flow record. ✨

For higher performance, it is advisable to use non-formatted conversion of IPFIX data types. In that case, you should prefer, for example, timestamps as numbers over ISO 8601 strings, and numeric identifiers of fields as they are usually shorted. 💅

🌈 Structured data types 🌈

So here's the tea, bestie: flow records can get extended with structured data types (the ones described in RFC6313). Every one of these types gets formatted as a JSON object carrying a "@type" field, and that field is literally the thing that tells you which formatting you're looking at. Also — and the standard is very clear on this — the semantic of the list is included too. No gatekeeping here. 💅

Say hi to basicList 👑 — when converted, it contains "fieldID" with the Information Element identifier of zero or more Information Element(s) that live inside the list. All the values get stored as a JSON array in the "data" field. Look at her:

{
    "example:blField": {
        "@type": "basicList",
        "semantic": "anyOf",
        "fieldID": "iana:octetDeltaCount",
        "data": [23, 34, 23]
    }
}

Next up, subTemplateList 🦋 — the converted version contains only one extra "data" field, and that's an array of zero or more JSON objects. Since every nested JSON object is described by the same IPFIX Template, it's guaranteed that their structure is the same too. Iconic. And the "semantic" field tells you the relationship among those different JSON objects:

{
    "example:stlField": {
        "@type": "subTemplateList",
        "semantic": "allOf",
        "data": [
            {"keyA.1": "value1", "keyA.2": "value2"},
            {"keyB.1": "value1", "keyB.2": "value2"},
        ]
    }
}

Now subTemplateMultiList ✨ — she's giving similar to the previous type, except the sub-records can go even more nested. The "data" field contains a JSON array with zero or more nested JSON arrays. Each one of those nested arrays contains zero or more JSON objects, and again it's guaranteed that JSON objects sitting in the same array have the same structure. The "semantic" field describes the top-level relationship among the nested arrays:

{
    "example:stmlField": {
        "@type": "subTemplateMultiList",
        "semantic": "allOf",
        "data" : [
            [
                {"keyA.1": "value1", "keyA.2": "value2"},
                {"keyB.1": "value1", "keyB.2": "value2"},
            ],
            [
                {"idA.1": "something", "idB.1": 123}
            ]
        ]
    }
}

⚠️ Keep in mind that all of these structures can be nested inside each other. Yes, really. Inception vibes. 🌀

---

💖 Template and Options Template records 💖

These records get converted only if the <templateInfo> option is enabled. That's the whole vibe check, no exceptions.

A Template record describes the structure of flow records, and its "@type" is "ipfix.template". The converted Template record contains an "ipfix:templateId" field — which is unique to the Transport Session and Observation Domain — plus "ipfix:fields", an array of JSON objects specifying the fields of flow records:

{
    "@type": "ipfix.template",
    "ipfix:templateId": 49171,
    "ipfix:fields": [
        {
            "ipfix:elementId": 16399,
            "ipfix:enterpriseId": 6871,
            "ipfix:fieldLength": 1
        }, {
            "ipfix:elementId": 184,
            "ipfix:enterpriseId": 29305,
            "ipfix:fieldLength": 4
        }, {
            ...
        }
    ]
}

An Options Template record describes the structure of additional information for the collector — think sampling configuration, export statistics, that kind of thing. That description gets converted to a record whose "@type" equals "ipfix.optionsEntry". The converted Options Template record is basically the previous type's twin, except it carries one extra "ipfix:scopeCount" field, which specifies the number of scope fields in the record. And hello, important detail: a scope count of N specifies that the first N field specifiers are scope fields. 🫶

{
    "@type": "ipfix.optionsTemplate",
        "ipfix:templateId": 53252,
        "ipfix:scopeCount": 1,
        "ipfix:fields": [
            {
                    "ipfix:elementId": 130,
                    "ipfix:enterpriseId": 0,
                    "ipfix:fieldLength": 4
            }, {
                    "ipfix:elementId": 103,
                    "ipfix:enterpriseId": 6871,
                    "ipfix:fieldLength": 4
            }, {
            ...
        ]
    ]
}

---

🎀 Author 🎀

• Lukáš Huták (lukas.hutak@cesnet.cz)
• Petr Velan (petr.velan@cesnet.cz)

⭐ Copyright ⭐

Copyright © 2019 CESNET, z.s.p.o.

---

That's the whole page, bestie, and honestly? She ate. 💅 This was version 2.1 from 2018-09-20, aka the ipfixcol2-json-output(7) era — periodt. Go forth and parse responsibly ✨🦋🌸