COMPACTOR(1)

📦 version 1.2.3, in nixpkgs unstable

🎀 COMPACTOR(1) 🎀

ok so this is the manual page for compactor, from the DNS-STATS corner of the universe ✨ welcome, bestie, pull up a chair 💖

✨ NAME ✨

compactor — grab DNS traffic, pull out the query and response messages, and write all that query/response pair info to a file 🫶

💅 SYNOPSIS 💅

compactor ['OPTIONS']... ['FILE'...]

🦋 DESCRIPTION 🦋

So here's the vibe: compactor reads DNS traffic, extracts the query and response messages, and matches them up into query/response pairs. Info about those pairs gets written to the output file. If you want, you can also write other output files containing all the packets, or the packets that didn't get used in the matching process. These output files are in PCAP format, no cap.

If you specify any input files, compactor reads from each input file in turn. Those input files must be in PCAP format. And good news: reading an input file needs no special privileges at all, we love an accessible queen 👑

If you don't specify any input files but do specify a network interface device, compactor will capture packets from that interface until it gets interrupted by a SIGINT signal (like from control-C) or a SIGTERM signal (like the ones kill(1) generates). 💀

If a SIGHUP signal shows up during network interface capture, compactor will stop the current collection, re-read the configuration file, and restart data collection with the updated settings. Important detail though: only configuration items that were previously unspecified or were taken from the configuration file can be updated — items you put on the command line cannot be changed. And during the restart there's a brief interval where packets are literally not being recorded 🙈 so like, be aware.

Reading packets from a network interface may require that you have special privileges; go read the pcap(3PCAP) man page for the details, iykyk.

compactor will throw an error if no input file or network interface device is specified. Yeah, it needs something to chew on.

Also, compactor may be built with support for DNSTAP. In that case it supports either reading data from DNSTAP capture files, or capturing traffic directly from a name server over a Unix socket 🧚

⭐ OPTIONS ⭐

Options to compactor come in two flavors: command options and configuration options.

🌸 compactor Command options 🌸

Command options are the ones you give on the command line, bestie.

  • -h, --help — prints a usage message that briefly summarises these command-line options, then dips. ✌️
  • -v, --version — prints the version number of compactor to the standard output stream, then exits.
  • -r, --report-info [arg] — report info (config and statistics summary) on exit. arg may be true or 1 to enable promiscuous mode, false or 0 to disable promiscuous mode. If arg is omitted, it defaults to true.
  • -D, --relaxed-mode [arg] — warn (instead of error) if unrecognized command line or config file options are found. arg may be true or 1 to enable promiscuous mode, false or 0 to disable promiscuous mode. If arg is omitted, it defaults to true.
  • -l, --list-interfaces — lists all network interfaces from which DNS traffic may be captured 👀
  • -c, --configfile [arg] — read configuration from file arg. If not specified, the default configuration file /nix/store/d6lld5x1pyrklnxiw4agakzl3zhxk6nb-compactor-1.2.3/etc/dns-stats-compactor/compactor.conf is read if present.
  • --excludesfile [arg] — read excludes from file arg. If not specified, the default configuration file /nix/store/d6lld5x1pyrklnxiw4agakzl3zhxk6nb-compactor-1.2.3/etc/dns-stats-compactor/excluded_fields.conf is read if present.
  • --debug-dns — prints a summary of each DNS packet to standard output after decoding. Little peek behind the curtain 🔍
  • --debug-qr — prints a summary of each query/response pair to standard output after matching query and response. Couples content 💞

💜 Configuration options 💜

Configuration options can go on the command line, or they can live in a configuration file. And if the same option appears in both a configuration file and the command line? The command line value wins, periodt.

A configuration file is a plain text file with lines in the form option=value. A # character introduces a comment that spans until the end of the line.

Heads up: in a configuration file you must use the long form of the option name. So for example, to set the snap length to 64, the configuration file entry must read snaplen=64. s=64 will not be accepted, don't even try it 🙅‍♀️

Also, the excludesfile option cannot be specified in the configuration file. That's just the rules, bestie.

🌈 Capture from network 🌈

  • -i, --interface arg — capture traffic from network interface arg. This argument may be given multiple times so you can capture for several interfaces in parallel, multitasking icon 💅
  • -S, --server-address-hint arg — arg is a single IPv4 or IPv6 address for the server. These hints are optional, they do not affect capture in any way, but they're stored in C-DNS as a potential aide to post-capture analysis. If capture from one or more interfaces is specified, then a sanity check is performed to make sure the address is present on a system interface; if it's not, an error is logged, but collection proceeds anyway (she's forgiving ✨). This argument may be given multiple times.
  • -f, --filter arg — discard packets not matching the filter expression arg. The filter syntax is described in pcap-filter(7). Packets are discarded at the point of capture, so discarded packets do not appear in any ignored PCAP output file. Gone means gone 💀
  • --dns-port arg — traffic to or from port arg is DNS traffic and should be recorded. Traffic to other ports is ignored. The default is 53.
  • -s, --snaplen arg — capture up to arg bytes per packet. The default is 65535.
  • -p, --promiscuous-mode [arg] — put the interface into promiscuous mode. arg may be true or 1 to enable promiscuous mode, false or 0 to disable promiscuous mode. If arg is omitted, it defaults to true. Promiscuous mode is disabled by default.
  • -a, --vlan-id arg — ID of the VLAN to be captured if you're on a 802.1Q network. The argument may be given multiple times to capture from several VLANs. If no vlan-id argument is given, traffic is captured from all VLANs 🦋
  • -L, --log-network-stats-period arg — every arg seconds, log basic statistics on packet collection to the system log. The default value of 0 disables this logging.
  • -F, --log-file-handling — log detailed file processing when files are rotated and compressed. This makes debugging file processing issues easier and lets you measure C-DNS file compression times 🎀
  • --sampling-threshold arg — a threshold for the percentage of traffic dropped on the internal channels, above which sampling will be enabled (if sampling-rate is greater than 0). The default value is 10.
  • --sampling-rate arg — the rate (1 in arg packets) to be applied when sampling mode is enabled (this is an experimental feature, so like, handle with care 🧪). The rate is applied for sampling-time seconds and then sampling is disabled. After that, depending on the traffic rate, sampling may be enabled again if the drops rise above the sampling-threshold. The default value of 0 disables this option.
  • --sampling-time arg — the period of time to apply sampling mode for. To avoid accidentally setting a low value that could result in instability, this must be at least 10s. The default value is 100.

🧚 Capture from DNSTAP 🧚

If compactor has been built with DNSTAP enabled, these options control capture from DNSTAP 👑

  • --dnstap [arg] — when reading input data from file, treat the data as DNSTAP data and not PCAP data if arg is true or 1. The default is false.
  • --dnstap-socket arg — capture DNSTAP traffic from Unix socket arg.
  • --dnstap-socket-owner arg — if specified, compactor will try to set the owner of the DNSTAP socket to arg.
  • --dnstap-socket-group arg — if specified, compactor will try to set the group of the DNSTAP socket to arg.
  • --dnstap-socket-write arg — if specified, compactor will try to set write permissions on the DNSTAP socket. If arg is owner, the socket owner will have write permission. If arg is group, the socket owner and group will have write permission. If arg is all, all users will have write permission. No other values of arg are permitted — those are the only three, okay? 🚫

🍓 Outputs 🍓

  • -o, --output PATTERN — use PATTERN as the template for the file path for the C-DNS output files. If no output pattern is given, no output is written.
  • -z, --gzip-output [arg] — compress data in the C-DNS output files using gzip(1) format. arg may be true or 1 to enable compression, false or 0 to disable compression. If arg is omitted, it defaults to true. If compression is enabled, an extension .gz is added to the output filename.
  • -y, --gzip-level [arg] — compression level to use when producing gzip(1) C-DNS output. arg must be a single digit 0 to 9. If not specified, the default level is 6.
  • -x, --xz-output [arg] — compress data in the C-DNS output files using xz(1) format. arg may be true or 1 to enable compression, false or 0 to disable compression. If arg is omitted, it defaults to true. If compression is enabled, an extension .xz is added to the output filename.
  • -u, --xz-preset [arg] — compression preset level to use when producing xz(1) C-DNS output. arg must be a single digit 0 to 9. If not specified, the default level is 6.
  • --max-compression-threads [arg] — maximum number of threads to use when compressing. Compression uses one thread per output file, so this argument gives the number of output files that can be compressed simultaneously. arg must be 1 or more. If not specified, the default number of threads is 2.
  • -w, --raw-pcap PATTERN — use PATTERN as the template for a file path for output of all packets captured via network capture, written to file in PCAP format. If no pattern is given, no raw packet output is written. This option is currently ignored when capturing via DNSTAP.
  • -m, --ignored-pcap PATTERN — use PATTERN as the template for a file path for output of all packets captured via network capture that were not to the configured DNS ports, or were not validly formed DNS packets. If no pattern is given, no ignored packet output is written. This option is currently ignored when capturing via DNSTAP.
  • -Z, --gzip-pcap [arg] — compress data in the PCAP output files using gzip(1) format. arg may be true or 1 to enable compression, false or 0 to disable compression. If arg is omitted, it defaults to true. If compression is enabled, an extension .gz is added to the output filename.
  • -Y, --gzip-level-pcap [arg] — compression level to use when producing gzip(1) PCAP output. arg must be a single digit 0 to 9. If not specified, the default level is 6.
  • -X, --xz-pcap [arg] — bestie, this one squishes your PCAP output files using xz(1) compression. 💅 arg can be true or 1 to turn compression on, or false or 0 to turn it off. If you just yeet it in with no arg at all, it defaults to true. Heads up though — when compression is enabled, an extension .xz gets added to the output filename. 🎀
  • -U, --xz-preset-pcap [arg] — the compression preset level to use when producing xz(1) C-DNS output. arg has to be a single digit from 0 to 9 (no vibes-based guessing, okay?). If you don't specify it, the default level is 6. ✨
  • -t, --rotation-period SECONDS — how often all your output file path patterns get re-examined. 👀 If the file path has changed after this period (say, because it contains a date/time element that ticked over), the existing output file gets closed and a shiny new one is opened using the new pattern expansion. If the file path hasn't changed, the pattern is re-examined every second until it does change. Note that file path patterns with no date/time elements in them will not trigger file rotation through this mechanism, so using a default file pattern similar to the one in the default configuration file is recommended (fr, do that). The default period is 300 seconds. This can be combined with maximum output file size rotation, and in that case rotation happens when either condition is met. 🔥
  • -n, --include SECTIONS — tells it which optional sections should be included in the main output. You can give this argument multiple times. By default, none of these optional sections are included. ⚠️ This option is deprecated and will be removed in a future release — it's been replaced by the excludesfile option. See the user guide for more info, iykyk.
  • -G, --ignore-opcode OPCODE — DNS messages with this OPCODE should NOT be included in the main output (the DNS message is just discarded, bye 💀). You can give this argument multiple times.
  • -E, --accept-opcode OPCODE — DNS messages with this OPCODE should be included in the main output. Careful though: if you use this, messages with OPCODEs you didn't list here get discarded. You can give this argument multiple times.
  • -g, --ignore-rr-type TYPE — records of this RR type TYPE should NOT be included in DNS messages when they're written to the main output. Important detail: the first question is ALWAYS included, but a TYPE nominated using the RR type name (in upper case) will be omitted from any other question and from any Answer, Authority or Additional section. You can give this argument multiple times. 🌸
  • -e, --accept-rr-type TYPE — records of this RR type TYPE should be included in DNS messages when they're written to the main output. So if you use this, RR types you didn't specify get omitted from DNS messages. Again: the first question is ALWAYS included, but only a TYPE nominated using the RR type name (in upper case) will be included in any other question and in any Answer, Authority or Additional section. You can give this argument multiple times.
  • --max-block-items arg — sets the max number of query/response items or address event items included in a single output C-DNS block. arg must be a positive integer. Default max size is 5000. 📦
  • --max-output-size arg — sets a maximum size for the uncompressed output before an output file rotation is triggered. arg must be a positive integer, and can optionally be followed by one of these multiplicative suffixes: k=1024, K=1000, m=1024*1024, M=1000*1000, and similarly for g and t. When a file rotation triggers, the remaining block and the file postlude get written, so the final file size will exceed this setting by a small margin (just a lil overshoot, no biggie). The default value is 0, which means there's no maximum size. This can be combined with a rotation period, and then rotation happens when either condition is met.
  • --client-address-prefix-ipv4 arg — sets the prefix size (the number of address bits stored) for IPv4 client addresses. The client address is the address of the sender of a query or the recipient of a response. arg must be a positive integer less than or equal to 32. The default is 32, so the whole entire IPv4 address gets stored. 🫶
  • --client-address-prefix-ipv6 arg — same deal but for IPv6 client addresses. arg must be a positive integer less than or equal to 128. Default is 128, so the whole entire IPv6 address is stored.
  • --server-address-prefix-ipv4 arg — the prefix size (number of address bits stored) for IPv4 server addresses. Server address = the address of the recipient of a query or the sender of a response. arg must be a positive integer less than or equal to 32. Default is 32, full IPv4 address stored. ⭐
  • --server-address-prefix-ipv6 arg — the prefix size for IPv6 server addresses, where the server address is the address of the recipient of a query or the sender of a response. arg must be a positive integer less than or equal to 128. Default is 128, full IPv6 address stored.

💅 Query/response matching 💅

  • -q, --query-timeout SECONDS — if no response is found for a query after SECONDS, the query times out. Default timeout is 5 seconds. Fractional values of SECONDS down to milliseconds are allowed, so yes you can get that granular. 😌
  • -k, --skew-timeout MICROSECONDS — okay so because the network stack is a chaotic little gremlin, it's possible for responses to be reported before the matching query, even though the query has an earlier timestamp than the response. A response isn't considered to be missing a query until after MICROSECONDS. The default timeout is 10 microseconds. 🦋

🗂️ OUTPUT FILE PATTERNS 🗂️

The paths used for all types of file output are described with output patterns. An output pattern can include expansions introduced by a % character in the filename part of the path — that's the last component, so no, you can't have expansions in a directory name, sorry bestie. %% adds a single % character to the output path.

Two categories of expansions exist: time expansions and configuration expansions.

Time expansions are % followed by a single letter, and they expand to a time component determined by the letter. For the full list of available letters, see strftime(3). Time expansions must be included in the output file pattern for filename-based rotation to happen via the rotation-period option. ⏰

Configuration expansions look like %{name} and substitute the value of the configuration item named. The config items you can substitute are interface, rotate-period, snaplen, query-timeout, skew-timeout, and promiscuous-mode. interface substitutes the names of all configured interfaces separated by - . The first network interface can be substituted as interface1 , a second network interface (if configured) can be substituted as interface2, and so on. Similarly, vlan-id substitutes all configured VLAN IDs separated by -. The first VLAN ID can be substituted as vlan1, vland-id2 substitutes the second VLAN ID if configured, and so on. 🌈

🙈 EXCLUDED FIELDS 🙈

To minimise the size of output files, compactor can omit fields you don't need. The fields to be omitted are specified by an exclude file, which you can point at with the excludesfile option. If you don't give one, the default configuration file

/nix/store/d6lld5x1pyrklnxiw4agakzl3zhxk6nb-compactor-1.2.3/etc/dns-stats-compactor/excluded_fields.conf

is read if present.

The fields you can omit are divided into different categories. By default all fields are included in the output. To omit a field, just name it in the exclude file under the appropriate category section. 🎀

Category [ip-header]

  • time-offset — time offset of a query/response record from the start of capture. Used to construct the query timestamp.
  • response-delay — the time difference between query and response. Used to reconstruct the response timestamp.
  • client-address — the IP address of the DNS client.
  • client-port — the port used by the DNS client.
  • client-hoplimit — the IPv4 TTL or IPv6 hoplimit of the query.
  • server-address — the IP address of the DNS server.
  • server-port — the port used by the DNS server.
  • qr-transport-flags — details on the query and response transport — IP version, UDP/TCP, etc.

Category [dns-header]

  • transaction-id — the DNS transaction ID of the query/response.
  • query-opcode — the DNS OPCODE of the query.
  • query-rcode — the DNS RCODE of the query.
  • dns-flags — the DNS flags of the query and response.
  • response-rcode — the DNS RCODE of the response.
  • query-qdcount — the QDCOUNT of the query.
  • query-ancount — the ANCOUNT of the query.
  • query-arcount — the ARCOUNT of the query.
  • query-nscount — the NSCOUNT of the query.

Category [dns-payload]

  • query-name — the QNAME of the first Question section of the query.
  • query-class-type — the TYPE and CLASS of the query.
  • rr-ttl — the TTL in any Resource Record (RR) in the response.
  • rr-rdata — the RDATA in any Resource Record (RR) in the response.
  • query-udp-size — Query EDNS sender's maximum acceptable UDP message size.
  • query-edns-version — Query EDNS version.
  • query-opt-data — Data part of Query OPT.
  • query-question-sections — Second and subsequent Question QRRs in query.
  • query-answer-sections — Answer RRs in query.
  • query-authority-sections — Authority RRs in query.
  • query-additional-sections — Additional RRs in query.
  • response-answer-sections — Answer RRs in response.
  • response-authority-sections — Authority RRs in response.
  • response-additional-sections — Additional RRs in response.

Category [dns-meta-data]

  • query-size — size of query DNS message.
  • response-size — size of response DNS message.

Category [storage-meta-data]

  • address-events — Address events. 📍

✨ OUTPUT FILE SECTIONS ✨

okay so bestie, the main output of DNS Query/Response pair info can optionally include full details on sections of the DNS messages 💅 like, yes, you can get the whole section-by-section tea if you want it 🫖✨

The sections to include are specified with the include option.

🚨 deprecated alert: this option is deprecated, and will be removed in a future release. It is replaced by the excludesfile option. See the user guide for more information.

include can only be specified if no excludesfile is read. In that case, no section details are included except for those sections specified using the include option 👀

The include option can be given multiple times. The sections that can be included are:

  • query-questions — Include second and subsequent QUESTION sections from queries. The first QUESTION section is always recorded.
  • query-answers — Include ANSWERS data from queries.
  • query-authority — Include AUTHORITY data from queries.
  • query-additional — Include ADDITIONAL data from queries.
  • query-all — Include all sections from queries.
  • response-questions — Include second and subsequent QUESTION sections from responses. The first QUESTION section is always recorded.
  • response-answers — Include ANSWERS data from responses.
  • response-authority — Include AUTHORITY data from responses.
  • response-additional — Include ADDITIONAL data from responses.
  • response-all — Include all sections from queries.
  • all — Include all sections from both queries and responses.
👀 and yes, response-all really says “from queries” in the original wording — I’m preserving that exactly, no autocorrect energy here bestie 💀

💥 EXIT STATUS 💥

The exit status is 1 if any error occurred. A successful run ends with an exit status of 0 💖

🌈 RESOURCES 🌈

https://github.com/dns-stats/compactor/wiki

Copyright 2016-2019 Internet Corporation for Assigned Names and Numbers.

Free use of this software is granted under the terms of the Mozilla Public Licence, version 2.0. See the source for full details.

👑 AUTHOR 👑

Sinodun Internet Technologies

okay that’s the whole vibe check bestie 🫶 — signed by DNS-STATS, dated 1980-01-01, and officially wrapped as COMPACTOR(1) 💅✨

📦 where this page comes from