RFC 8618
Compacted-DNS (C-DNS): A Format for DNS Packet Capture
Part 1 of 17 🎀
Compacted-DNS (C-DNS): A Format for DNS Packet Capture 💖
Okay bestie, first the receipts 🧾✨ This is RFC 8618, a Standards Track document from the IETF (Internet Engineering Task Force) published in September 2019 🔥 Its ISSN is 2070-1721. J. Dickinson and J. Hague wrote it, alongside S. Dickinson (Sinodun IT), T. Manderson (ICANN), and J. Bond (Wikimedia Foundation, Inc.) 🫶 The header doesn't list any RFCs it obsoletes or updates, so it's not replacing or amending anything else 🤷
Internet Engineering Task Force (IETF) J. Dickinson
Request for Comments: 8618 J. Hague
Category: Standards Track S. Dickinson
ISSN: 2070-1721 Sinodun IT
T. Manderson
ICANN
J. Bond
Wikimedia Foundation, Inc.
September 2019Abstract
Okay so the vibe here is data storage glow-up ✨ This doc describes a way to represent collections of DNS messages. It's built for storing and sending huge packet captures of DNS traffic efficiently 💅 The goal is to shrink those capture files as much as possible while still keeping the full DNS message contents and the most useful transport metadata 🦋 It's meant to help people build DNS traffic-monitoring apps 🌸
Status of This Memo
This is an Internet Standards Track document 🔥
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841 💖
Info about the current status of this document, any errata, and how to send feedback on it can be found at https://www.rfc-editor.org/info/rfc8618 🧚
Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved. 🎀
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document 🙈 Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License 💜
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Data Collection Use Cases . . . . . . . . . . . . . . . . . . 5
4. Design Considerations . . . . . . . . . . . . . . . . . . . . 8
5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 10
6. C-DNS Format Conceptual Overview . . . . . . . . . . . . . . 10
6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 14
6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 14
6.2.1. Optional Data Items . . . . . . . . . . . . . . . . . 15
6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 16
6.2.3. Storage Flags . . . . . . . . . . . . . . . . . . . . 17
6.2.4. IP Address Storage . . . . . . . . . . . . . . . . . 17
7. C-DNS Format Detailed Description . . . . . . . . . . . . . . 18
7.1. Map Quantities and Indexes . . . . . . . . . . . . . . . 18
7.2. Tabular Representation . . . . . . . . . . . . . . . . . 18
7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 19
7.3.1. "FilePreamble" . . . . . . . . . . . . . . . . . . . 20
7.3.1.1. "BlockParameters" . . . . . . . . . . . . . . . . 20
7.3.1.1.1. "StorageParameters" . . . . . . . . . . . . . 21
7.3.1.1.1.1. "StorageHints" . . . . . . . . . . . . . 22
7.3.1.1.2. "CollectionParameters" . . . . . . . . . . . 24
7.3.2. "Block" . . . . . . . . . . . . . . . . . . . . . . . 25
7.3.2.1. "BlockPreamble" . . . . . . . . . . . . . . . . . 26
7.3.2.2. "BlockStatistics" . . . . . . . . . . . . . . . . 27
7.3.2.3. "BlockTables" . . . . . . . . . . . . . . . . . . 28
7.3.2.3.1. "ClassType" . . . . . . . . . . . . . . . . . 29
7.3.2.3.2. "QueryResponseSignature" . . . . . . . . . . 30
7.3.2.3.3. "Question" . . . . . . . . . . . . . . . . . 33
7.3.2.3.4. "RR" . . . . . . . . . . . . . . . . . . . . 34
7.3.2.3.5. "MalformedMessageData" . . . . . . . . . . . 34
7.3.2.4. "QueryResponse" . . . . . . . . . . . . . . . . . 35
7.3.2.4.1. "ResponseProcessingData" . . . . . . . . . . 36
7.3.2.4.2. "QueryResponseExtended" . . . . . . . . . . . 37
7.3.2.5. "AddressEventCount" . . . . . . . . . . . . . . . 38
7.3.2.6. "MalformedMessage" . . . . . . . . . . . . . . . 39
8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 39
9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 40
9.1. Name Compression . . . . . . . . . . . . . . . . . . . . 42
10. Data Collection . . . . . . . . . . . . . . . . . . . . . . . 42
10.1. Matching Algorithm . . . . . . . . . . . . . . . . . . . 43
10.2. Message Identifiers . . . . . . . . . . . . . . . . . . 45
10.2.1. Primary ID (Required) . . . . . . . . . . . . . . . 45
10.2.2. Secondary ID (Optional) . . . . . . . . . . . . . . 46
10.3. Algorithm Parameters . . . . . . . . . . . . . . . . . . 46
10.4. Algorithm Requirements . . . . . . . . . . . . . . . . . 46
10.5. Algorithm Limitations . . . . . . . . . . . . . . . . . 47
10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 47
10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 47
10.8. Post-Processing . . . . . . . . . . . . . . . . . . . . 47
11. Implementation Guidance . . . . . . . . . . . . . . . . . . . 47
11.1. Optional Data . . . . . . . . . . . . . . . . . . . . . 48
11.2. Trailing Bytes . . . . . . . . . . . . . . . . . . . . . 48
11.3. Limiting Collection of RDATA . . . . . . . . . . . . . . 49
11.4. Timestamps . . . . . . . . . . . . . . . . . . . . . . . 49
12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 49
12.1. Transport Types . . . . . . . . . . . . . . . . . . . . 49
12.2. Data Storage Flags . . . . . . . . . . . . . . . . . . . 50
12.3. Response-Processing Flags . . . . . . . . . . . . . . . 51
12.4. AddressEvent Types . . . . . . . . . . . . . . . . . . . 51
13. Security Considerations . . . . . . . . . . . . . . . . . . . 52
14. Privacy Considerations . . . . . . . . . . . . . . . . . . . 52
15. References . . . . . . . . . . . . . . . . . . . . . . . . . 53
15.1. Normative References . . . . . . . . . . . . . . . . . . 53
15.2. Informative References . . . . . . . . . . . . . . . . . 55
Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 58
Appendix B. DNS Name Compression Example . . . . . . . . . . . . 69
B.1. NSD Compression Algorithm . . . . . . . . . . . . . . . . 70
B.2. Knot Authoritative Compression Algorithm . . . . . . . . 70
B.3. Observed Differences . . . . . . . . . . . . . . . . . . 71
Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 71
C.1. Comparison with Full PCAP Files . . . . . . . . . . . . . 74
C.2. Simple versus Block Coding . . . . . . . . . . . . . . . 74
C.3. Binary versus Text Formats . . . . . . . . . . . . . . . 75
C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 75
C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 75
C.6. Block Size Choice . . . . . . . . . . . . . . . . . . . . 76
Appendix D. Data Fields for Traffic Regeneration . . . . . . . . 77
D.1. Recommended Fields for Traffic Regeneration . . . . . . . 77
D.2. Issues with Small Data Captures . . . . . . . . . . . . . 77
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 78
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 791. Introduction ✨
Server operators have needed a way to collect DNS Queries and Responses on authoritative and recursive name servers for monitoring and analysis, and that need has been around for a long time 👑 The data gets used for lots of things, like traffic monitoring, looking into network attacks, and "day in the life" (DITL) [ditl] analysis 📊
There are already a bunch of tools that help collect DNS traffic data, like the DNS Statistics Collector (DSC) [dsc], packetq [packetq], dnscap [dnscap], and dnstap [dnstap] 🛠️ But here's the tea: there's no standard exchange format for large DNS packet captures 😭 People usually use the PCAP ("packet capture") [pcap] format or the PCAP Next Generation (PCAP-NG) [pcapng] format for packet captures. The problem is those formats can hold a ton of extra info that doesn't really matter for DNS traffic analysis, so the capture files get bigger than they need to be 🙈 On top of that, these tools and formats usually don't have a filter mechanism to record only certain fields at capture time. That means you end up doing post-processing for anonymization or pseudonymization to protect user privacy 🫠
Text-based formats for describing DNS packets exist too (see [dnsxml] and [RFC8427]), but that work is mostly focused on making convenient representations of single messages, so it's a different vibe entirely 💅
Many DNS operators may receive hundreds of thousands of Queries per second on a single name server instance 🔥 So a way to shrink the storage and transmission size of the collected data is highly desirable, because it also cuts down on upload overhead 🦋
The format in this document, C-DNS (Compacted-DNS), is all about capturing and storing large packet capture files of DNS traffic 🎀 It's got these goals:
- Minimize the file size for storage and transmission.
- Minimize the overhead of producing the packet capture file and the cost of any further (general-purpose) compression of the file.
This document contains:
- A chat about some common use cases where DNS data gets collected, see Section 3.
- A breakdown of the major design considerations behind an efficient data representation for collections of DNS messages, see Section 4.
- An explanation of why the Concise Binary Object Representation (CBOR) [RFC7049] was picked for this format, see Section 5.
- A conceptual overview of the C-DNS format, see Section 6.
- The definition of the C-DNS format for collections of DNS messages, see Section 7.
- Notes on converting C-DNS data to PCAP format, see Section 9.
- Some high-level implementation considerations for apps designed to produce C-DNS, see Section 10.
2. Terminology ✨
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. Okay, so these capital-letter words are the official BCP 14 vibe check, and they only count when they're written in ALL CAPS. Lowercase "must" is just... a normal word, bestie 💅
"Packet" refers to an individual IPv4 or IPv6 packet. Typically, packets are UDP datagrams, but such packets may also be part of a TCP data stream. "Message", unless otherwise qualified, refers to a DNS payload extracted from a UDP datagram or a TCP data stream. So a packet is one IP packet 📦, and a message is the DNS payload you pull out of it. Iykyk 🦋
The parts of DNS messages are named as they are in [RFC1035]. Specifically, the DNS message has five sections: Header, Question, Answer, Authority, and Additional. Five sections, no more, no less. It's giving very organized energy 🎀
3. Data Collection Use Cases 🌸
From a purely server operator perspective, collecting full packet captures of all packets going into or out of a name server provides the most comprehensive picture of network activity. However, there are several design choices or other limitations that are common to many DNS installations and operators. Full captures are the dream, but real life is messy 😭
- DNS servers are hosted in a variety of situations:
- Self-hosted servers
- Third-party hosting (including multiple third parties)
- Third-party hardware (including multiple third parties)
- Data is collected under different conditions:
- On well-provisioned servers running in a steady state
- On heavily loaded servers
- On virtualized servers
- On servers that are under DoS attack
- On servers that are unwitting intermediaries in DoS attacks
- Traffic can be collected via a variety of mechanisms:
- Within the name server implementation itself
- On the same hardware as the name server itself
- Using a network tap on an adjacent host to listen to DNS traffic
- Using port mirroring to listen from another host
- The capabilities of data collection (and upload) networks vary:
- Out-of-band networks with the same capacity as the in-band network
- Out-of-band networks with less capacity than the in-band network
- Everything being on the in-band network
Thus, there is a wide range of use cases, from very limited data collection environments (third-party hardware, servers that are under attack, packet capture on the name server itself and no out-of-band network) to "limitless" environments (self-hosted, well-provisioned servers, using a network tap or port mirroring with out-of-band networks with the same capacity as the in-band network). In the former case, it is infeasible to reliably collect full packet captures, especially if the server is under attack. In the latter case, collection of full packet captures may be reasonable. So the range goes from "barely any room to breathe" 💀 all the way to "limitless" 👑, and the answer depends on which end you're at.
As a result of these restrictions, the C-DNS data format is designed with the most limited use case in mind, such that:
- Data collection will occur on the same hardware as the name server itself
- Collected data will be stored on the same hardware as the name server itself, at least temporarily
- Collected data being returned to some central analysis system will use the same network interface as the DNS Queries and Responses
- There can be multiple third-party servers involved
Because of these considerations, a major factor in the design of the format is minimal storage size of the capture files. So the format is built around the tightest scenario, and storage size is the big priority 🧚
Another significant consideration for any application that records DNS traffic is that the running of the name server software and the transmission of DNS Queries and Responses are the most important jobs of a name server; capturing data is not. Any data collection system co-located with the name server needs to be intelligent enough to carefully manage its CPU, disk, memory, and network utilization. This leads to designing a format that requires a relatively low overhead to produce and minimizes the requirement for further potentially costly compression. Translation: the name server is the main character 🎀, and the capture tool has to know its place and not hog resources ✨
However, it is also essential that interoperability with less restricted infrastructure is maintained. In particular, it is highly desirable that the collection format should facilitate the re-creation of common formats (such as PCAP) that are as close to the original as is realistic, given the restrictions above. So even though it's built for the tight scenario, it still needs to play nicely with the fancier setups 🫶
4. Design Considerations 💖
This section presents some of the major design considerations used in the development of the C-DNS format. Let's get into the big decisions 👀
- The basic unit of data is a combined DNS Query and the associated Response (a "Query/Response (Q/R) data item"). The same structure will be used for unmatched Queries and Responses. Queries without Responses will be captured omitting the Response data. Responses without Queries will be captured omitting the Query data (but using the Question section from the Response, if present, as an identifying QNAME).
- Rationale: A Query and the associated Response represent the basic level of a client's interaction with the server. Also, combining the Query and Response into one item often reduces storage requirements due to commonality in the data of the two messages. Pairing them up is just smarter 🧠
In the context of generating a C-DNS file, it is assumed that only those DNS payloads that can be parsed to produce a well-formed DNS message are stored in the structured Query/Response data items of the C-DNS format and that all other messages will (optionally) be recorded as separate malformed messages. Parsing a well-formed message means, at a minimum, the following:
- The packet has a well-formed 12-byte DNS Header with a recognized OPCODE.
- The section counts are consistent with the section contents.
- All of the Resource Records (RRs) can be fully parsed.
- All top-level fields in each Query/Response data item will be optional.
- Rationale: Different operators will have different requirements for data to be available for analysis. Operators with minimal requirements should not have to pay the cost of recording full data, though this will limit the ability to perform certain kinds of data analysis and also to reconstruct packet captures. For example, omitting the RRs from a Response will reduce the C-DNS file size; in principle, Responses can be synthesized if there is enough context. Operators may have different policies for collecting user data and can choose to omit or anonymize certain fields at capture time, e.g., client address. Everybody gets to pick their own vibe here 🌈
- Multiple Query/Response data items will be collected into blocks in the format. Common data in a block will be abstracted and referenced from individual Query/Response data items by indexing. The maximum number of Query/Response data items in a block will be configurable.
- Rationale: This blocking and indexing action provides a significant reduction in the volume of file data generated. Although this introduces complexity, it provides compression of the data that makes use of knowledge of the DNS message structure. Smart packing, no cap 📦
- It is anticipated that the files produced can be subject to further compression using general-purpose compression tools. Measurements show that blocking significantly reduces the CPU required to perform such strong compression. See Appendix C.2.
- Examples of commonality between DNS messages are that in most cases the QUESTION RR is the same in the Query and Response and that there is a finite set of Query "signatures" (based on a subset of attributes). For many authoritative servers, there is very likely to be a finite set of Responses that are generated, of which a large number are NXDOMAIN. Basically the same questions get asked over and over, lowkey predictable 😭
- Traffic metadata can optionally be included in each block. Specifically, counts of some types of non-DNS packets (e.g., ICMP, TCP resets) sent to the server may be of interest. Extra context, if you want it ✨
- The wire-format content of malformed DNS messages may optionally be recorded.
- Rationale: Any structured capture format that does not capture the DNS payload byte for byte will be limited to some extent in that it cannot represent malformed DNS messages. Only those messages that can be fully parsed and transformed into the structured format can be fully represented. Note, however, that this can result in rather misleading statistics. For example, a malformed Query that cannot be represented in the C-DNS format will lead to the (well-formed) DNS Response with error code FORMERR appearing as "unmatched". Therefore, it can greatly aid downstream analysis to have the wire format of the malformed DNS messages available directly in the C-DNS file. Okay this one is the plot twist 😱 a malformed query can make a perfectly fine response look "unmatched" and that's so misleading 💀
5. Choice of CBOR 🦋
This document presents a detailed format description for C-DNS. The format uses CBOR [RFC7049].
The choice of CBOR was made taking a number of factors into account.
- CBOR is a binary representation and thus is economical in storage space.
- Other binary representations were investigated, and whilst all had attractive features, none had a significant advantage over CBOR. See Appendix C for some discussion of this.
- CBOR is an IETF specification and is familiar to IETF participants. It is based on the now-common ideas of lists and objects and thus requires very little familiarization for those in the wider industry.
- CBOR is a simple format and can easily be implemented from scratch if necessary. Formats that are more complex require library support, which may present problems on unusual platforms.
- CBOR can also be easily converted to text formats such as JSON [RFC8259] for debugging and other human inspection requirements.
- CBOR data schemas can be described using the Concise Data Definition Language (CDDL) [RFC8610].
Basically CBOR is the storage-friendly, well-known pick that everybody already kinda knows, and it's easy to turn into JSON when you need to peek inside 👑
6. C-DNS Format Conceptual Overview ✨
okay so this part is where the C-DNS format finally gets its little diagram moment 🎀 the figures are purely schematic, so they're just here to show the big-picture shape of everything. the detailed CBOR stuff and every single element gets the full treatment in Section 7 💅
Figure 1 is the top-level view: the file header, then the data blocks. notice how the Query/Response data items, the Address/Event Count data items, and the Malformed Message data items all link out to the various Block Tables 🦋
+-------+
+ C-DNS |
+-------+--------------------------+
| File Type Identifier |
+----------------------------------+
| File Preamble |
| +--------------------------------+
| | Format Version |
| +--------------------------------+
| | Block Parameters |
+-+--------------------------------+
| Block |
| +--------------------------------+
| | Block Preamble |
| +--------------------------------+
| | Block Statistics |
| +--------------------------------+
| | Block Tables |
| +--------------------------------+
| | Query/Response data items |
| +--------------------------------+
| | Address/Event Count data items |
| +--------------------------------+
| | Malformed Message data items |
+-+--------------------------------+
| Block |
| +--------------------------------+
| | Block Preamble |
| +--------------------------------+
| | Block Statistics |
| +--------------------------------+
| | Block Tables |
| +--------------------------------+
| | Query/Response data items |
| +--------------------------------+
| | Address/Event Count data items |
| +--------------------------------+
| | Malformed Message data items |
+-+--------------------------------+
| Further Blocks... |
+----------------------------------+Figure 1: The C-DNS Format
Figure 2 zooms in on the good stuff 🔥 it shows the relationships inside each Block, mainly between the Query/Response data item and the relevant Block Tables. some fields got left out so it doesn't melt your brain 🧠 (it's still a lot, bestie, and that's fine 😭)
+----------------+
| Query/Response |
+-------------------------+
| Time Offset |
+-------------------------+ +------------------+
| Client Address |---------+->| IP Address array |
+-------------------------+ | +------------------+
| Client Port | |
+-------------------------+ | +------------------+
| Transaction ID | +---)->| Name/RDATA array |<--------+
+-------------------------+ | | +------------------+ |
| Query Signature |--+ | | |
+-------------------------+ | | | +-----------------+ |
| Client Hoplimit (q) | +--)---)->| Query Signature | |
+-------------------------+ | | +-----------------+-------+ |
| Response Delay (r) | | +--| Server Address | |
+-------------------------+ | +-------------------------+ |
| Query Name |--+--+ | Server Port | |
+-------------------------+ | +-------------------------+ |
| Query Size (q) | | | Transport Flags | |
+-------------------------+ | +-------------------------+ |
| Response Size (r) | | | QR Type | |
+-------------------------+ | +-------------------------+ |
| Response Processing (r) | | | QR Signature Flags | |
| +-----------------------+ | +-------------------------+ |
| | Bailiwick |--+ | Query OPCODE (q) | |
| +-----------------------+ +-------------------------+ |
| | Flags | | QR DNS Flags | |
+-+-----------------------+ +-------------------------+ |
| Extra Query Info (q) | | Query RCODE (q) | |
| +-----------------------+ +-------------------------+ |
| | Question |--+---+ +--+-Query Class/Type (q) | |
| +-----------------------+ | | +-------------------------+ |
| | Answer |--+ | | | Query QDCOUNT (q) | |
| +-----------------------+ | | | +-------------------------+ |
| | Authority |--+ | | | Query ANCOUNT (q) | |
| +-----------------------+ | | | +-------------------------+ |
| | Additional |--+ | | | Query NSCOUNT (q) | |
+-+-----------------------+ | | | +-------------------------+ |
| Extra Response Info (r) | |-+ | | | Query ARCOUNT (q) | |
| +-----------------------+ | | | | +-------------------------+ |
| | Answer |--+ | | | | Query EDNS version (q) | |
| +-----------------------+ | | | | +-------------------------+ |
| | Authority |--+ | | | | Query EDNS UDP Size (q) | |
| +-----------------------+ | | | | +-------------------------+ |
| | Additional |--+ | | | | Query OPT RDATA (q) |--+
+-+-----------------------+ | | | +-------------------------+ |
| | | | Response RCODE (r) | |
| | | +-------------------------+ |
+ -----------------------------+ | +----------+ |
| | | |
| + -----------------------------+ | |
| | +---------------+ +----------+ | |
| +->| Question List |->| Question | | |
| | array | | array | | |
| +---------------+ +----------+--+ | |
| | Name |--+-----)--------------------+
| +-------------+ | | +------------+
| | Class/Type |--)---+-+->| Class/Type |
| +-------------+ | | | array |
| | | +------------+--+
| | | | CLASS |
| +---------------+ +----------+ | | +---------------+
+--->| RR List array |->| RR array | | | | TYPE |
+---------+-----+ +----------+--+ | | +---------------+
| Name |--+ |
+-------------+ |
| Class/Type |------+
+-------------+Figure 2: The Query/Response Data Item and Subsidiary Tables
okay, so in Figure 2, anything marked (q) only shows up when a Query/Response has a Query 🙈 and anything marked (r) only shows up when a Response is present ✨
A C-DNS file starts with a file header. that header holds a File Type Identifier and a File Preamble 🌸 The File Preamble has the file Format Version, plus an array of Block Parameters items (the stuff in there includes the Collection and Storage Parameters used for one or more Blocks) 💜
After the file header comes a series of Blocks 🧚
Each Block is made of a Block Preamble item, some Block Statistics for the traffic stored in that Block, and then a bunch of arrays of common data that are collectively called the Block Tables. after that comes an array of the Query/Response data items, which detail the Queries and Responses stored in the Block. that array gets followed by the Address/Event Count data items (per-client counts of particular IP events) and then the Malformed Message data items (the malformed messages stored in the Block) 🍓
what size is best for a Block depends on the DNS data, bestie 🤷♀️ but sample data from a root server showed that Block sizes up to 10,000 Query/Response data items work really well ⭐ See Appendix C.6 for more details.
this design leans hard on data commonality and block-based storage to keep the C-DNS file size small 🦋 the trade-off is that C-DNS cannot be streamed below the level of a Block 😭
6.1. Block Parameters
the Block Parameters items aren't shown in the diagrams, but we're going to talk about them here so you have the context 💖
An array of Block Parameters items is stored in the File Preamble, with a minimum of one item at index 0 🎀 each Block Parameters item is a collection of Storage and Collection Parameters that apply to any given Block. why an array? so you can do stuff like merging C-DNS files from different sources, obviously 🫶 The Block Preamble item then holds an optional index for the Block Parameters item that applies to that Block. if it's not there, the index defaults to 0. so basically, you get a global Block Parameters item, and each Block can override it 👑
6.2. Storage Parameters
the Block Parameters item includes a Storage Parameters item, and this is where the info about the specific data fields stored in the C-DNS file lives 💅
These parameters include:
- The sub-second timing resolution used by the data.
- Information (hints) on which optional data are omitted. See Section 6.2.1.
- Recorded OPCODES [opcodes] and RR TYPEs [rrtypes]. See Section 6.2.2.
- Flags indicating, for example, whether the data is sampled or anonymized. See Sections 6.2.3 and 14.
- Client and server IPv4 and IPv6 address prefixes. See Section 6.2.4.
6.2.1. Optional Data Items 🎀
Okay so, to let implementations store exactly what they need in the most space-efficient way possible 💅, every field in the following arrays is optional:
- Query/Response
- Query Signature
- Malformed Messages
Basically, an implementation can skip any data item it doesn't need for its use case, as long as it follows the IP address storage restrictions in Section 6.2.4 🙈. Implementations may also be configured to not record all RRs, or to only record messages with certain OPCODES ✨.
BUT (and this is a big but 😭) a consumer of a C-DNS file ends up facing two problems:
- How can it quickly tell if a file definitely doesn't contain the data items it needs for a task (like rebuilding DNS traffic or doing a specific kind of data analysis)?
- How can it tell whether a data item is missing because it was (1) explicitly not recorded or (2) not available/present?
For example, capturing C-DNS data from inside a name server implementation makes it unlikely that the Client Hoplimit can be recorded. Or, if there's no Query ARCOUNT recorded and no Query OPT RDATA [RFC6891] recorded, is that because no Query had an OPT RR, or because that data wasn't stored? 🤔
So the Storage Parameters item also contains a Storage Hints item 🍓. It says which items the encoder of the file leaves out of the stored data, which means those items will never be present. (This approach is used because a flag saying which items were included for collection wouldn't guarantee the item was there. It would only mean it might be 🙃.) A decoder can then check these flags to quickly figure out whether the input data isn't rich enough for what it needs.
One scenario where this matters a lot is regenerating traffic. You can collect such a small set of data items that a decoder can't tell whether a given Query/Response data item came from just a Query, just a Response, or a Query/Response pair. That makes it impossible to rebuild DNS traffic, even if sensible defaults are provided for the missing data items. More on this in Section 9 🦋.
6.2.2. Optional RRs and OPCODEs 🧚
Also included in the Storage Parameters item are explicit arrays listing the RR TYPEs and the OPCODEs to be recorded. These arrays clear up any confusion about whether, for example, messages with certain OPCODEs are missing because (1) those OPCODEs never happened or (2) the implementation isn't set up to record them 💜.
For OPCODEs, a message has to be fully parsable, which means the OPCODE must be known to the collecting implementation. Any message with an OPCODE the collector doesn't know can't be validated as correctly formed, so it must be treated as malformed. Messages with OPCODES that the recording application knows about but that aren't listed in the Storage Parameters item get discarded by that recording application during C-DNS capture, whether they're malformed or not 🗑️.
For RRs, each record in a message must be fully parsable, including parsing the record RDATA. Otherwise the message can't be validated as correctly formed. Any RR with an RR TYPE the collecting implementation doesn't know can't be validated either, so it must be treated as malformed 🙈.
Once a message is correctly parsed, an implementation is free to record only a subset of the RRs present 💖.
6.2.3. Storage Flags 🌸
The Storage Parameters item contains flags that can be used to indicate if:
- the data is anonymized,
- the data is produced from sample data, or
- names in the data have been normalized (converted to uniform case).
The Storage Parameters item also has optional fields with details of the sampling method and the anonymization method that were used. It is RECOMMENDED that these fields contain URIs [RFC3986] pointing to resources that describe the methods used. Check out Section 14 for more on anonymization and normalization 🔥.
6.2.4. IP Address Storage 🌈
The format can store either full IP addresses or just IP prefixes. The Storage Parameters item has fields that show whether only IP prefixes were stored 👀.
If the IP address prefixes are absent, then full addresses are stored. In this case, the IP version can be directly inferred from the stored address length. The fields qr-transport-flags in QueryResponseSignature, ae-transport-flags in AddressEventCount, and mm-transport-flags in MalformedMessageData (which contain the IP version bit) are optional 🫶.
If IP address prefixes are given, only the prefix bits of addresses are stored. Here, to work out the IP version, the fields qr-transport-flags in QueryResponseSignature, ae-transport-flags in AddressEventCount, and mm-transport-flags in MalformedMessageData MUST be present. See Sections 7.3.2.3.2 and 7.3.2.3.5 🎀.
Here's an example of storing only IP prefixes. If a client IPv6 prefix of 48 is specified, a client address of 2001:db8:85a3::8a2e:370:7334 will be stored as 0x20010db885a3, which cuts down on address storage space requirements 💅. Similarly, if a client IPv4 prefix of 16 is specified, a client address of 192.0.2.1 will be stored as 0xc000 (192.0) ✨.
7. C-DNS Format Detailed Description 💜
The CDDL definition for the C-DNS format is given in Appendix A 📜.
7.1. Map Quantities and Indexes 🦋
All map keys are integers with values specified in the CDDL. String keys would significantly bloat the file size 😭.
All key values specified are positive integers under 24, so their CBOR representation is a single byte. Positive integer values that aren't currently used as keys in a map are reserved for use in future standard extensions 🔮.
Implementations may choose to add additional implementation-specific entries to any map. Negative integer map keys are reserved for these values. Key values from -1 to -24 also have a single-byte CBOR representation, so these implementation-specific extensions don't have any space efficiency disadvantage 🙌.
An item described as an index is the index of the data item in the referenced array. Indexes are 0-based 🔢.
7.2. Tabular Representation 🍓
The following sections present the C-DNS specification in tabular format, with a detailed description of each item 📊.
In all quantities that contain bit flags, bit 0 indicates the least significant bit, i.e., flag "n" in quantity "q" is on if (q & (1 << n)) != 0 🧠.
For readability, all type and field names defined in the CDDL definition are shown in double quotes. Type names are by convention camel case (e.g., "BlockTables"), and field names are lowercase with hyphens (e.g., "block-tables") 💖.
For brevity, the following conventions are used in the tables:
- The column M marks whether items in a map are mandatory.
- X - Mandatory items.
- C - Conditionally mandatory items. Such items are usually optional but may be mandatory in some configurations.
- If the column is empty, the item is optional.
- The column T gives the CBOR datatype of the item.
- U - Unsigned integer.
- I - Signed integer (i.e., either a CBOR unsigned integer or a CBOR negative integer).
- B - Boolean.
- S - Byte string.
- T - Text string.
- M - Map.
- A - Array.
In the case of maps and arrays, more information on the type of each value, including the CDDL definition name if applicable, is given in the description 🌸.
7.3. "File" 🎁
A C-DNS file has an outer structure "File", an array that contains the following:
+---------------+---+---+-------------------------------------------+
| Field | M | T | Description |
+---------------+---+---+-------------------------------------------+
| file-type-id | X | T | String "C-DNS" identifying the file type. |
| | | | |
| file-preamble | X | M | Version and parameter information for the |
| | | | whole file. Map of type "FilePreamble"; |
| | | | see Section 7.3.1. |
| | | | |
| file-blocks | X | A | Array of items of type "Block"; see |
| | | | Section 7.3.2. The array may be empty if |
| | | | the file contains no data. |
+---------------+---+---+-------------------------------------------+7.3.1. "FilePreamble" 🧁
Information about data in the file. It's a map containing the following 🗂️:
+----------------------+---+---+------------------------------------+
| Field | M | T | Description |
+----------------------+---+---+------------------------------------+
| major-format-version | X | U | Unsigned integer "1". The major |
| | | | version of the format used in the |
| | | | file. See Section 8. |
| | | | |
| minor-format-version | X | U | Unsigned integer "0". The minor |
| | | | version of the format used in the |
| | | | file. See Section 8. |
| | | | |
| private-version | | U | Version indicator available for |
| | | | private use by implementations. |
| | | | |
| block-parameters | X | A | Array of items of type |
| | | | "BlockParameters". See Section |
| | | | 7.3.1.1. The array must contain |
| | | | at least one entry. (The |
| | | | "block-parameters-index" item in |
| | | | each "BlockPreamble" indicates |
| | | | which array entry applies to that |
| | | | "Block".) |
+----------------------+---+---+------------------------------------+7.3.1.1. "BlockParameters" 🎀
okay so BlockParameters is the map that holds the storage and collection settings that apply to one or more "Block" items 🧚 it has two fields, and yes, they're both giving important 💅
+-----------------------+---+---+-----------------------------------+
| Field | M | T | Description |
+-----------------------+---+---+-----------------------------------+
| storage-parameters | X | M | Parameters relating to data |
| | | | storage in a "Block" item. Map |
| | | | of type "StorageParameters"; see |
| | | | Section 7.3.1.1.1. |
| | | | |
| collection-parameters | | M | Parameters relating to collection |
| | | | of the data in a "Block" item. |
| | | | Map of type |
| | | | "CollectionParameters"; see |
| | | | Section 7.3.1.1.2. |
+-----------------------+---+---+-----------------------------------+7.3.1.1.1. "StorageParameters" 💜
this one is all about how the data is actually stored inside those "Block" items 📦 it's a map, and it's packed with knobs to turn 🎛️
+------------------+---+---+----------------------------------------+
| Field | M | T | Description |
+------------------+---+---+----------------------------------------+
| ticks-per-second | X | U | Sub-second timing is recorded in |
| | | | ticks. This specifies the number of |
| | | | ticks in a second. |
| | | | |
| max-block-items | X | U | The maximum number of items stored in |
| | | | any of the arrays in a "Block" item |
| | | | (Q/R, Address/Event Count, or |
| | | | Malformed Message data items). An |
| | | | indication to a decoder of the |
| | | | resources needed to process the file. |
| | | | |
| storage-hints | X | M | Collection of hints as to which fields |
| | | | are omitted in the arrays that have |
| | | | optional fields. Map of type |
| | | | "StorageHints". See Section |
| | | | 7.3.1.1.1.1. |
| | | | |
| opcodes | X | A | Array of OPCODES [opcodes] (unsigned |
| | | | integers, each in the range 0 to 15 |
| | | | inclusive) recorded by the collecting |
| | | | implementation. See Section 6.2.2. |
| | | | |
| rr-types | X | A | Array of RR TYPEs [rrtypes] (unsigned |
| | | | integers, each in the range 0 to 65535 |
| | | | inclusive) recorded by the collecting |
| | | | implementation. See Section 6.2.2. |
| | | | |
| storage-flags | | U | Bit flags indicating attributes of |
| | | | stored data. |
| | | | Bit 0. 1 if the data has been |
| | | | anonymized. |
| | | | Bit 1. 1 if the data is sampled data. |
| | | | Bit 2. 1 if the names have been |
| | | | normalized (converted to uniform |
| | | | case). |
| | | | |
| client-address | | U | IPv4 client address prefix length, in |
| -prefix-ipv4 | | | the range 1 to 32 inclusive. If |
| | | | specified, only the address prefix |
| | | | bits are stored. |
| | | | |
| client-address | | U | IPv6 client address prefix length, in |
| -prefix-ipv6 | | | the range 1 to 128 inclusive. If |
| | | | specified, only the address prefix |
| | | | bits are stored. |
| | | | |
| server-address | | U | IPv4 server address prefix length, in |
| -prefix-ipv4 | | | the range 1 to 32 inclusive. If |
| | | | specified, only the address prefix |
| | | | bits are stored. |
| | | | |
| server-address | | U | IPv6 server address prefix length, in |
| -prefix-ipv6 | | | the range 1 to 128 inclusive. If |
| | | | specified, only the address prefix |
| | | | bits are stored. |
| | | | |
| sampling-method | | T | Information on the sampling method |
| | | | used. See Section 6.2.3. |
| | | | |
| anonymization | | T | Information on the anonymization |
| -method | | | method used. See Section 6.2.3. |
+------------------+---+---+----------------------------------------+7.3.1.1.1.1. "StorageHints" 🙈
think of StorageHints as a little cheat sheet 📝 it tells you which fields the collecting implementation leaves out of the maps that have optional fields
the precedence here goes top-down, bestie 👑 if a map contains another map and the containing map has a hint, that hint overrides whatever the contained map says, and the contained map just gets omitted. no cap, read that twice 😭
it's a map too, with these fields:
+------------------+---+---+----------------------------------------+
| Field | M | T | Description |
+------------------+---+---+----------------------------------------+
| query-response | X | U | Hints indicating which "QueryResponse" |
| -hints | | | fields are omitted; see Section |
| | | | 7.3.2.4. If a bit is unset, the field |
| | | | is omitted from the capture. |
| | | | Bit 0. time-offset |
| | | | Bit 1. client-address-index |
| | | | Bit 2. client-port |
| | | | Bit 3. transaction-id |
| | | | Bit 4. qr-signature-index |
| | | | Bit 5. client-hoplimit |
| | | | Bit 6. response-delay |
| | | | Bit 7. query-name-index |
| | | | Bit 8. query-size |
| | | | Bit 9. response-size |
| | | | Bit 10. response-processing-data |
| | | | Bit 11. query-question-sections |
| | | | Bit 12. query-answer-sections |
| | | | Bit 13. query-authority-sections |
| | | | Bit 14. query-additional-sections |
| | | | Bit 15. response-answer-sections |
| | | | Bit 16. response-authority-sections |
| | | | Bit 17. response-additional-sections |
| | | | |
| query-response | X | U | Hints indicating which |
| -signature-hints | | | "QueryResponseSignature" fields are |
| | | | omitted; see Section 7.3.2.3.2. If a |
| | | | bit is unset, the field is omitted |
| | | | from the capture. |
| | | | Bit 0. server-address-index |
| | | | Bit 1. server-port |
| | | | Bit 2. qr-transport-flags |
| | | | Bit 3. qr-type |
| | | | Bit 4. qr-sig-flags |
| | | | Bit 5. query-opcode |
| | | | Bit 6. qr-dns-flags |
| | | | Bit 7. query-rcode |
| | | | Bit 8. query-classtype-index |
| | | | Bit 9. query-qdcount |
| | | | Bit 10. query-ancount |
| | | | Bit 11. query-nscount |
| | | | Bit 12. query-arcount |
| | | | Bit 13. query-edns-version |
| | | | Bit 14. query-udp-size |
| | | | Bit 15. query-opt-rdata-index |
| | | | Bit 16. response-rcode |
| | | | |
| rr-hints | X | U | Hints indicating which optional "RR" |
| | | | fields are omitted; see Section |
| | | | 7.3.2.3.4. If a bit is unset, the |
| | | | field is omitted from the capture. |
| | | | Bit 0. ttl |
| | | | Bit 1. rdata-index |
| other-data-hints | X | U | Hints indicating which other datatypes |
| | | | are omitted. If a bit is unset, the |
| | | | datatype is omitted from the capture. |
| | | | Bit 0. malformed-messages |
| | | | Bit 1. address-event-counts |
+------------------+---+---+----------------------------------------+7.3.1.1.2. "CollectionParameters" 🕵️♀️✨
Okay bestie, this one is all about the collection backstory 🎀. These parameters describe how the collecting setup was configured, but they only apply to some collection environments, not all of them 💅. They're informational only, basically metadata for downstream analyzers. They can help explain why certain data is in the capture or missing from it, but you can't always check them against the data that was actually captured 🙈.
These parameters have no default, so if they don't show up, you literally can't infer anything about their value 💀. Here's the tea:
A map containing the following items:
+------------------+---+---+----------------------------------------+
| Field | M | T | Description |
+------------------+---+---+----------------------------------------+
| query-timeout | | U | To be matched with a Query, a Response |
| | | | must arrive within this number of |
| | | | milliseconds. |
| | | | |
| skew-timeout | | U | The network stack may report a |
| | | | Response before the corresponding |
| | | | Query. A Response is not considered |
| | | | to be missing a Query until after this |
| | | | many microseconds. |
| | | | |
| snaplen | | U | Collect up to this many bytes per |
| | | | packet. |
| | | | |
| promisc | | B | "true" if promiscuous mode |
| | | | [pcap-options] was enabled on the |
| | | | interface, "false" otherwise. |
| | | | |
| interfaces | | A | Array of identifiers (of type text |
| | | | string) of the interfaces used for |
| | | | collection. |
| | | | |
| server-addresses | | A | Array of server collection IP |
| | | | addresses (of type byte string). |
| | | | Metadata for downstream analyzers; |
| | | | does not affect collection. |
| | | | |
| vlan-ids | | A | Array of identifiers (of type unsigned |
| | | | integer, each in the range 1 to 4094 |
| | | | inclusive) of VLANs [IEEE802.1Q] |
| | | | selected for collection. VLAN IDs are |
| | | | unique only within an administrative |
| | | | domain. |
| | | | |
| filter | | T | Filter for input, in "tcpdump" |
| | | | [pcap-filter] style. |
| | | | |
| generator-id | | T | Implementation-specific human-readable |
| | | | string identifying the collection |
| | | | method. |
| | | | |
| host-id | | T | String identifying the collecting |
| | | | host. |
+------------------+---+---+----------------------------------------+7.3.2. "Block" 🧺💖
A "Block" is a container for data that shares the same collection and storage parameters. It's a map with the following fields:
+--------------------+---+---+--------------------------------------+
| Field | M | T | Description |
+--------------------+---+---+--------------------------------------+
| block-preamble | X | M | Overall information for the "Block" |
| | | | item. Map of type "BlockPreamble"; |
| | | | see Section 7.3.2.1. |
| | | | |
| block-statistics | | M | Statistics about the "Block" item. |
| | | | Map of type "BlockStatistics"; see |
| | | | Section 7.3.2.2. |
| | | | |
| block-tables | | M | The arrays containing data |
| | | | referenced by individual |
| | | | "QueryResponse" or |
| | | | "MalformedMessage" items. Map of |
| | | | type "BlockTables"; see Section |
| | | | 7.3.2.3. |
| | | | |
| query-responses | | A | Details of individual C-DNS Q/R data |
| | | | items. Array of items of type |
| | | | "QueryResponse"; see Section |
| | | | 7.3.2.4. If present, the array must |
| | | | not be empty. |
| | | | |
| address-event | | A | Per-client counts of ICMP messages |
| -counts | | | and TCP resets. Array of items of |
| | | | type "AddressEventCount"; see |
| | | | Section 7.3.2.5. If present, the |
| | | | array must not be empty. |
| | | | |
| malformed-messages | | A | Details of malformed DNS messages. |
| | | | Array of items of type |
| | | | "MalformedMessage"; see Section |
| | | | 7.3.2.6. If present, the array must |
| | | | not be empty. |
+--------------------+---+---+--------------------------------------+7.3.2.1. "BlockPreamble" 🌸
This is the overall info for a "Block" item, so think of it as the cover page 📖. It's a map with the following fields:
+------------------+---+---+----------------------------------------+
| Field | M | T | Description |
+------------------+---+---+----------------------------------------+
| earliest-time | C | A | A timestamp (two unsigned integers, of |
| | | | type "Timestamp") for the earliest |
| | | | record in the "Block" item. The first |
| | | | integer is the number of seconds since |
| | | | the POSIX epoch [posix-time] |
| | | | ("time_t"), excluding leap seconds. |
| | | | The second integer is the number of |
| | | | ticks (see Section 7.3.1.1.1) since |
| | | | the start of the second. This field |
| | | | is mandatory unless all block items |
| | | | containing a time offset from the |
| | | | start of the Block also omit that time |
| | | | offset. |
| | | | |
| block-parameters | | U | The index of the item in the |
| -index | | | "block-parameters" array (in the |
| | | | "file-preamble" item) applicable to |
| | | | this block. If not present, index 0 |
| | | | is used. See Section 7.3.1. |
+------------------+---+---+----------------------------------------+7.3.2.2. "BlockStatistics" 📊✨
These are the basic stats about a "Block" item, so you can see at a glance how much happened inside it 👀. It's a map with the following fields:
+---------------------+---+---+-------------------------------------+
| Field | M | T | Description |
+---------------------+---+---+-------------------------------------+
| processed-messages | | U | Total number of well-formed DNS |
| | | | messages processed from the input |
| | | | traffic stream during collection of |
| | | | data in this "Block" item. |
| | | | |
| qr-data-items | | U | Total number of Q/R data items in |
| | | | this "Block" item. |
| | | | |
| unmatched-queries | | U | Number of unmatched Queries in this |
| | | | "Block" item. |
| | | | |
| unmatched-responses | | U | Number of unmatched Responses in |
| | | | this "Block" item. |
| | | | |
| discarded-opcode | | U | Number of DNS messages processed |
| | | | from the input traffic stream |
| | | | during collection of data in this |
| | | | "Block" item but not recorded |
| | | | because their OPCODE is not in the |
| | | | list to be collected. |
| | | | |
| malformed-items | | U | Number of malformed messages |
| | | | processed from the input traffic |
| | | | stream during collection of data in |
| | | | this "Block" item. |
+---------------------+---+---+-------------------------------------+7.3.2.3. "BlockTables" ✨
okay so this is the bit where the Block gets its own little filing cabinet 🗄️🎀 it's a map of arrays, and every array holds stuff that the individual "QueryResponse" or "MalformedMessage" items in this "Block" point back to. each element is an array that, if it shows up at all, MUST NOT be empty 💅
quick vibe check on the wiring: an item in the "qlist" array is basically a set of indexes pointing into the "qrr" array 🔗. so if "qlist" is there, "qrr" MUST also be there. same deal for "rrlist" and "rr" 🦋
here's the full menu of items in the map 👇
+-------------------+---+---+---------------------------------------+
| Field | M | T | Description |
+-------------------+---+---+---------------------------------------+
| ip-address | | A | Array of IP addresses, in network |
| | | | byte order (of type byte string). If |
| | | | client or server address prefixes are |
| | | | set, only the address prefix bits are |
| | | | stored. Each string is therefore up |
| | | | to 4 bytes long for an IPv4 address, |
| | | | or up to 16 bytes long for an IPv6 |
| | | | address. See Section 7.3.1.1.1. |
| | | | |
| classtype | | A | Array of RR CLASS and TYPE |
| | | | information. Type is "ClassType". |
| | | | See Section 7.3.2.3.1. |
| | | | |
| name-rdata | | A | Array where each entry is the |
| | | | contents of a single NAME or RDATA in |
| | | | wire format (of type byte string). |
| | | | Note that NAMEs, and labels within |
| | | | RDATA contents, are full domain names |
| | | | or labels; no name compression (per |
| | | | [RFC1035]) is used on the individual |
| | | | names/labels within the format. |
| | | | |
| qr-sig | | A | Array of Q/R data item signatures. |
| | | | Type is "QueryResponseSignature". |
| | | | See Section 7.3.2.3.2. |
| | | | |
| qlist | | A | Array of type "QuestionList". A |
| | | | "QuestionList" is an array of |
| | | | unsigned integers, indexes to |
| | | | "Question" items in the "qrr" array. |
| | | | |
| qrr | | A | Array of type "Question". Each entry |
| | | | is the contents of a single Question, |
| | | | where a Question is the second or |
| | | | subsequent Question in a Query. See |
| | | | Section 7.3.2.3.3. |
| | | | |
| rrlist | | A | Array of type "RRList". An "RRList" |
| | | | is an array of unsigned integers, |
| | | | indexes to "RR" items in the "rr" |
| | | | array. |
| | | | |
| rr | | A | Array of type "RR". Each entry is |
| | | | the contents of a single RR. See |
| | | | Section 7.3.2.3.4. |
| | | | |
| malformed-message | | A | Array of the contents of malformed |
| -data | | | messages. Array of type |
| | | | "MalformedMessageData". See Section |
| | | | 7.3.2.3.5. |
+-------------------+---+---+---------------------------------------+7.3.2.3.1. "ClassType" 🎀
okay this one is the RR CLASS and TYPE info, packaged up as a little map 🌸 it has these two fields:
+-------+---+---+--------------------------+
| Field | M | T | Description |
+-------+---+---+--------------------------+
| type | X | U | TYPE value [rrtypes]. |
| | | | |
| class | X | U | CLASS value [rrclasses]. |
+-------+---+---+--------------------------+so yeah, "type" is the TYPE value from [rrtypes], and "class" is the CLASS value from [rrclasses] 💜 both are unsigned integers, and both are mandatory, no skipping 🙈
7.3.2.3.2. "QueryResponseSignature" ✨
okay so this one is the QueryResponseSignature, and it's giving "stuff that keeps repeating, so let's put it in one drawer" 💅 Lots of Q/R data items share the same bits, so this map holds the shared details once. Here's the full lineup 🎀
+--------------------+---+---+--------------------------------------+
| Field | M | T | Description |
+--------------------+---+---+--------------------------------------+
| server-address | | U | The index in the "ip-address" array |
| -index | | | of the server IP address. See |
| | | | Section 7.3.2.3. |
| | | | |
| server-port | | U | The server port. |
| | | | |
| qr-transport-flags | C | U | Bit flags describing the transport |
| | | | used to service the Query. Same |
| | | | definition as "mm-transport-flags" |
| | | | in Section 7.3.2.3.5, with an |
| | | | additional indicator for trailing |
| | | | bytes. See Appendix A. |
| | | | Bit 0. IP version. 0 if IPv4, 1 if |
| | | | IPv6. See Section 6.2.4. |
| | | | Bits 1-4. Transport. 4-bit |
| | | | unsigned value where |
| | | | 0 = UDP [RFC1035] |
| | | | 1 = TCP [RFC1035] |
| | | | 2 = TLS [RFC7858] |
| | | | 3 = DTLS [RFC8094] |
| | | | 4 = HTTPS [RFC8484] |
| | | | 15 = Non-standard transport (see |
| | | | below) |
| | | | Values 5-14 are reserved for future |
| | | | use. |
| | | | Bit 5. 1 if trailing bytes in Query |
| | | | packet. See Section 11.2. |
| | | | |
| qr-type | | U | Type of Query/Response transaction |
| | | | based on the definitions in the |
| | | | dnstap schema [dnstap-schema]. |
| | | | 0 = Stub. A transaction between a |
| | | | stub resolver and a DNS server from |
| | | | the perspective of the stub |
| | | | resolver. |
| | | | 1 = Client. A transaction between a |
| | | | client and a DNS server (a proxy or |
| | | | full recursive resolver) from the |
| | | | perspective of the DNS server. |
| | | | 2 = Resolver. A transaction between |
| | | | a recursive resolver and an |
| | | | authoritative server from the |
| | | | perspective of the recursive |
| | | | resolver. |
| | | | 3 = Authoritative. A transaction |
| | | | between a recursive resolver and an |
| | | | authoritative server from the |
| | | | perspective of the authoritative |
| | | | server. |
| | | | 4 = Forwarder. A transaction |
| | | | between a downstream forwarder and |
| | | | an upstream DNS server (a recursive |
| | | | resolver) from the perspective of |
| | | | the downstream forwarder. |
| | | | 5 = Tool. A transaction between a |
| | | | DNS software tool and a DNS server, |
| | | | from the perspective of the tool. |
| | | | |
| qr-sig-flags | | U | Bit flags explicitly indicating |
| | | | attributes of the message pair |
| | | | represented by this Q/R data item |
| | | | (not all attributes may be recorded |
| | | | or deducible). |
| | | | Bit 0. 1 if a Query was present. |
| | | | Bit 1. 1 if a Response was present. |
| | | | Bit 2. 1 if a Query was present and |
| | | | it had an OPT RR. |
| | | | Bit 3. 1 if a Response was present |
| | | | and it had an OPT RR. |
| | | | Bit 4. 1 if a Query was present but |
| | | | had no Question. |
| | | | Bit 5. 1 if a Response was present |
| | | | but had no Question (only one |
| | | | query-name-index is stored per Q/R |
| | | | data item). |
| | | | |
| query-opcode | | U | Query OPCODE. |
| | | | |
| qr-dns-flags | | U | Bit flags with values from the Query |
| | | | and Response DNS flags. Flag values |
| | | | are 0 if the Query or Response is |
| | | | not present. |
| | | | Bit 0. Query Checking Disabled |
| | | | (CD). |
| | | | Bit 1. Query Authenticated Data |
| | | | (AD). |
| | | | Bit 2. Query reserved (Z). |
| | | | Bit 3. Query Recursion Available |
| | | | (RA). |
| | | | Bit 4. Query Recursion Desired |
| | | | (RD). |
| | | | Bit 5. Query TrunCation (TC). |
| | | | Bit 6. Query Authoritative Answer |
| | | | (AA). |
| | | | Bit 7. Query DNSSEC answer OK (DO). |
| | | | Bit 8. Response Checking Disabled |
| | | | (CD). |
| | | | Bit 9. Response Authenticated Data |
| | | | (AD). |
| | | | Bit 10. Response reserved (Z). |
| | | | Bit 11. Response Recursion |
| | | | Available (RA). |
| | | | Bit 12. Response Recursion Desired |
| | | | (RD). |
| | | | Bit 13. Response TrunCation (TC). |
| | | | Bit 14. Response Authoritative |
| | | | Answer (AA). |
| | | | |
| query-rcode | | U | Query RCODE. If the Query contains |
| | | | an OPT RR [RFC6891], this value |
| | | | incorporates any EXTENDED-RCODE |
| | | | value [rcodes]. |
| | | | |
| query-classtype | | U | The index in the "classtype" array |
| -index | | | of the CLASS and TYPE of the first |
| | | | Question. See Section 7.3.2.3. |
| | | | |
| query-qdcount | | U | The QDCOUNT in the Query, or |
| | | | Response if no Query present. |
| | | | |
| query-ancount | | U | Query ANCOUNT. |
| | | | |
| query-nscount | | U | Query NSCOUNT. |
| | | | |
| query-arcount | | U | Query ARCOUNT. |
| | | | |
| query-edns-version | | U | The Query EDNS version. ("EDNS" |
| | | | stands for Extension Mechanisms for |
| | | | DNS.) |
| | | | |
| query-udp-size | | U | The Query EDNS sender's UDP payload |
| | | | size. |
| | | | |
| query-opt-rdata | | U | The index in the "name-rdata" array |
| -index | | | of the OPT RDATA. See Section |
| | | | 7.3.2.3. |
| | | | |
| response-rcode | | U | Response RCODE. If the Response |
| | | | contains an OPT RR [RFC6891], this |
| | | | value incorporates any EXTENDED- |
| | | | RCODE value [rcodes]. |
+--------------------+---+---+--------------------------------------+Last bit of this one 🌸 C-DNS version 1.0 is able to handle transport values for the DNS transports that are defined in IETF Standards Track documents as of when this was written ✨ There are loads of non-standard ways to send DNS messages over all sorts of transports and protocols, but those are outside what this document covers 🙈 For now you can stash them under the value 15 (Non-standard transport), or you can use the negative integer map keys to make your own mappings. And honestly? Non-standard transports might get their own extension in the future 🦋
7.3.2.3.3. "Question" 🎀
This is where we get into the nitty-gritty of each Question in a Question section 🧚. It's a map with these fields:
+-----------------+---+---+-----------------------------------------+
| Field | M | T | Description |
+-----------------+---+---+-----------------------------------------+
| name-index | X | U | The index in the "name-rdata" array of |
| | | | the QNAME. See Section 7.3.2.3. |
| | | | |
| classtype-index | X | U | The index in the "classtype" array of |
| | | | the CLASS and TYPE of the Question. |
| | | | See Section 7.3.2.3. |
+-----------------+---+---+-----------------------------------------+7.3.2.3.4. "RR" 💜
Okay, now the RR entries, which live in the RR sections 💅. Same vibe: it's a map, and each field points back into the arrays we already set up in Section 7.3.2.3.
+-----------------+---+---+-----------------------------------------+
| Field | M | T | Description |
+-----------------+---+---+-----------------------------------------+
| name-index | X | U | The index in the "name-rdata" array of |
| | | | the NAME. See Section 7.3.2.3. |
| | | | |
| classtype-index | X | U | The index in the "classtype" array of |
| | | | the CLASS and TYPE of the RR. See |
| | | | Section 7.3.2.3. |
| | | | |
| ttl | | U | The RR Time to Live. |
| | | | |
| rdata-index | | U | The index in the "name-rdata" array of |
| | | | the RR RDATA. See Section 7.3.2.3. |
+-----------------+---+---+-----------------------------------------+7.3.2.3.5. "MalformedMessageData" 🙈
Okay this one is for the messes 😭 It holds details on malformed DNS messages that get stored in this "Block" item. Still a map, of course 🍓
+--------------------+---+---+--------------------------------------+
| Field | M | T | Description |
+--------------------+---+---+--------------------------------------+
| server-address | | U | The index in the "ip-address" array |
| -index | | | of the server IP address. See |
| | | | Section 7.3.2.3. |
| | | | |
| server-port | | U | The server port. |
| | | | |
| mm-transport-flags | C | U | Bit flags describing the transport |
| | | | used to service the Query. See |
| | | | Section 6.2.4. |
| | | | Bits 1-4. Transport. 4-bit |
| | | | unsigned value where |
| | | | 0 = UDP [RFC1035] |
| | | | 1 = TCP [RFC1035] |
| | | | 2 = TLS [RFC7858] |
| | | | 3 = DTLS [RFC8094] |
| | | | 4 = HTTPS [RFC8484] |
| | | | 15 = Non-standard transport |
| | | | Values 5-14 are reserved for future |
| | | | use. |
| | | | |
| mm-payload | | S | The payload (raw bytes) of the DNS |
| | | | message. |
+--------------------+---+---+--------------------------------------+7.3.2.4. "QueryResponse" 🦋
This is the big one 👑 It holds details on each individual Q/R data item.
Quick heads-up: there's no requirement that the entries in the query-responses array show up in strict chronological order. So don't go assuming they're lined up by time 💀
Here's the map of items:
+----------------------+---+---+------------------------------------+
| Field | M | T | Description |
+----------------------+---+---+------------------------------------+
| time-offset | | U | Q/R timestamp as an offset in |
| | | | ticks (see Section 7.3.1.1.1) from |
| | | | "earliest-time". The timestamp is |
| | | | the timestamp of the Query, or the |
| | | | Response if there is no Query. |
| | | | |
| client-address-index | | U | The index in the "ip-address" |
| | | | array of the client IP address. |
| | | | See Section 7.3.2.3. |
| | | | |
| client-port | | U | The client port. |
| | | | |
| transaction-id | | U | DNS transaction identifier. |
| | | | |
| qr-signature-index | | U | The index in the "qr-sig" array of |
| | | | the "QueryResponseSignature" item. |
| | | | See Section 7.3.2.3. |
| | | | |
| client-hoplimit | | U | The IPv4 TTL or IPv6 Hoplimit from |
| | | | the Query packet. |
| | | | |
| response-delay | | I | The time difference between Query |
| | | | and Response, in ticks. See |
| | | | Section 7.3.1.1.1. Only present |
| | | | if there is a Query and a |
| | | | Response. The delay can be |
| | | | negative if the network |
| | | | stack/capture library returns |
| | | | packets out of order. |
| | | | |
| query-name-index | | U | The index in the "name-rdata" |
| | | | array of the item containing the |
| | | | QNAME for the first Question. See |
| | | | Section 7.3.2.3. |
| | | | |
| query-size | | U | DNS Query message size (see |
| | | | below). |
| | | | |
| response-size | | U | DNS Response message size (see |
| | | | below). |
| | | | |
| response-processing | | M | Data on Response processing. Map |
| -data | | | of type "ResponseProcessingData". |
| | | | See Section 7.3.2.4.1. |
| | | | |
| query-extended | | M | Extended Query data. Map of type |
| | | | "QueryResponseExtended". See |
| | | | Section 7.3.2.4.2. |
| | | | |
| response-extended | | M | Extended Response data. Map of |
| | | | type "QueryResponseExtended". See |
| | | | Section 7.3.2.4.2. |
+----------------------+---+---+------------------------------------+The query-size and response-size fields hold the DNS message size ✨ For UDP, that's the size of the UDP payload that carried the DNS message. For TCP, it's the size of the DNS message as given in the two-byte message length header. Trailing bytes in UDP Queries show up a lot in traffic to authoritative servers 👀 and this value lets you figure out how many trailing bytes were actually there.
7.3.2.4.1. "ResponseProcessingData" 🌈
This section is all about the server processing that made the Response 🔥 It's a map with these fields:
+------------------+---+---+----------------------------------------+
| Field | M | T | Description |
+------------------+---+---+----------------------------------------+
| bailiwick-index | | U | The index in the "name-rdata" array of |
| | | | the owner name for the Response |
| | | | bailiwick. See Section 7.3.2.3. |
| | | | |
| processing-flags | | U | Flags relating to Response processing. |
| | | | Bit 0. 1 if the Response came from |
| | | | cache. |
+------------------+---+---+----------------------------------------+7.3.2.4.2. "QueryResponseExtended" ⭐
This is the extra-details pack for a Q/R data item 🎀
Each item in the map only shows up if collecting the relevant details has been configured. So if you didn't turn it on, it's just not there, no drama 🫶
It's a map with these items:
+------------------+---+---+----------------------------------------+
| Field | M | T | Description |
+------------------+---+---+----------------------------------------+
| question-index | | U | The index in the "qlist" array of the |
| | | | entry listing any second and |
| | | | subsequent Questions in the Question |
| | | | section for the Query or Response. |
| | | | See Section 7.3.2.3. |
| | | | |
| answer-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Answer RR sections |
| | | | for the Query or Response. See |
| | | | Section 7.3.2.3. |
| | | | |
| authority-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Authority RR |
| | | | sections for the Query or Response. |
| | | | See Section 7.3.2.3. |
| | | | |
| additional-index | | U | The index in the "rrlist" array of the |
| | | | entry listing the Additional RR |
| | | | sections for the Query or Response. |
| | | | See Section 7.3.2.3. Note that Query |
| | | | OPT RR data can optionally be stored |
| | | | in the QuerySignature. |
+------------------+---+---+----------------------------------------+7.3.2.5. "AddressEventCount" 💖
okay so this one is all about counting IP-related events that hit traffic from individual client addresses 🦋 it's basically a map, and every key in it is a little piece of the story 💅
+--------------------+---+---+--------------------------------------+
| Field | M | T | Description |
+--------------------+---+---+--------------------------------------+
| ae-type | X | U | The type of event. The following |
| | | | event types are currently defined: |
| | | | 0. TCP reset. |
| | | | 1. ICMP time exceeded. |
| | | | 2. ICMP destination unreachable. |
| | | | 3. ICMPv6 time exceeded. |
| | | | 4. ICMPv6 destination unreachable. |
| | | | 5. ICMPv6 packet too big. |
| | | | |
| ae-code | | U | A code relating to the event. For |
| | | | ICMP or ICMPv6 events, this MUST be |
| | | | the ICMP [RFC792] or ICMPv6 |
| | | | [RFC4443] code. For other events, |
| | | | the contents are undefined. |
| | | | |
| ae-transport-flags | C | U | Bit flags describing the transport |
| | | | used to service the event. See |
| | | | Section 6.2.4. |
| | | | Bit 0. IP version. 0 if IPv4, 1 if |
| | | | IPv6. |
| | | | Bits 1-4. Transport. 4-bit |
| | | | unsigned value where |
| | | | 0 = UDP [RFC1035] |
| | | | 1 = TCP [RFC1035] |
| | | | 2 = TLS [RFC7858] |
| | | | 3 = DTLS [RFC8094] |
| | | | 4 = HTTPS [RFC8484] |
| | | | 15 = Non-standard transport |
| | | | Values 5-14 are reserved for future |
| | | | use. |
| | | | |
| ae-address-index | X | U | The index in the "ip-address" array |
| | | | of the client address. See Section |
| | | | 7.3.2.3. |
| | | | |
| ae-count | X | U | The number of occurrences of this |
| | | | event during the Block collection |
| | | | period. |
+--------------------+---+---+--------------------------------------+7.3.2.6. "MalformedMessage" 🎀
this one is for the Malformed Message data items, and it's also a map 🧚 check out the fields below, because each one tells you where to find the bad message and its data ✨
+----------------------+---+---+------------------------------------+
| Field | M | T | Description |
+----------------------+---+---+------------------------------------+
| time-offset | | U | Message timestamp as an offset in |
| | | | ticks (see Section 7.3.1.1.1) from |
| | | | "earliest-time". |
| | | | |
| client-address-index | | U | The index in the "ip-address" |
| | | | array of the client IP address. |
| | | | See Section 7.3.2.3. |
| | | | |
| client-port | | U | The client port. |
| | | | |
| message-data-index | | U | The index in the "malformed- |
| | | | message-data" array of the message |
| | | | data for this message. See |
| | | | Section 7.3.2.3. |
+----------------------+---+---+------------------------------------+8. Versioning 🌈
the C-DNS File Preamble has a file Format Version in it, and that version is made of a major number and a minor number 🔥 both of them are required fields. this document is all about version 1.0 of the C-DNS spec, and this section explains how the version numbers are meant to be used later on 👑
it's worth knowing that version 1.0 has tons of optional fields, so consumers of version 1.0 should be inherently robust when they parse files with variable data content 🫶
within one major version, a new minor version MUST be a strict superset of the previous minor version, with no semantic changes to existing fields 💜 new keys MAY be added to existing maps, and new maps MAY be added too ✨ a consumer that can read a particular major.minor version MUST also be able to read all the previous minor versions of that same major version 🦋 it SHOULD also be able to parse all the later minor versions too, just ignoring any keys or maps it doesn't recognize 💅
a new major version means the format has changed in ways that aren't backwards compatible with earlier major versions 💀 a consumer that can only read one particular major version (greater than 1) is neither required nor expected to read a previous major version 🙈
9. C-DNS to PCAP 🌸
it's usually possible to rebuild PCAP files from the C-DNS format, but it's lossy, so you don't get everything back 😭 this section goes over some of the issues with rebuilding both the DNS payload and the full packet stream 🍓
the rebuild of well-formed DNS messages depends on two factors:
- whether or not a particular subset of the optional fields got captured in the C-DNS file. specifically, the data fields needed to rebuild a valid IP header and DNS payload for both Query and Response (see Appendix D.1). obviously, if not all of those data fields were captured, the rebuild is likely to be imperfect, even if reasonable defaults are provided for it 🥺
- whether or not at least one field got captured that clearly identifies the Query/Response data item as just a Query, just a Response, or a Query/Response pair. the
qr-sig-flagsdefined in Section 7.3.2.3.2 is exactly that kind of field, but this field is optional. for more details, see Appendix D.2 ✨
it's worth saying again that just having hints showing certain data fields were not omitted doesn't guarantee those fields were actually captured 🫣 so the ability to rebuild PCAP data (without defaults) can, in principle, vary for each record captured in a C-DNS file, and between Blocks that have different hints 💫
even if every section of the Response was captured, you can't rebuild the DNS Response payload exactly, because some DNS names in the message on the wire may have been compressed 😤 Section 9.1 talks about this in more detail 🎀
some transport information isn't captured in the C-DNS format at all 💔 for example, these aspects of the original packet stream can't be rebuilt from the C-DNS format:
- IP fragmentation
- TCP stream information:
- Multiple DNS messages may have been sent in a single TCP segment
- A DNS payload may have been split across multiple TCP segments
- Multiple DNS messages may have been sent on a single TCP session
- TLS session information:
- DNS-over-HTTPS [RFC8484] message details:
- Whether the message used POST or GET
- HTTPS Headers
- Malformed DNS messages if the wire format is not recorded
- Any non-DNS messages that were in the original packet stream, e.g., ICMP
simple assumptions can be made when rebuilding: fragmented and DNS-over-TCP messages can be rebuilt into single packets, and a single TCP session can be constructed for each TCP packet 🧚
and if malformed messages and non-DNS packets were captured separately, they can be merged with the packet captures rebuilt from C-DNS to make a more complete packet stream 🦋
9.1. Name Compression 💅
all the names stored in the C-DNS format are full domain names, and no name compression (per [RFC1035]) is used on the individual names inside the format 💖 so when you rebuild a packet, name compression must be used in order to reproduce the on-the-wire representation of the packet 🔥
name compression per [RFC1035] works by swapping trailing sections of a name for a reference back to where those sections showed up earlier in the message 🌈 not all name server software uses the same algorithm when it compresses domain names within the Responses 👑 some try for maximum recompression even though it costs runtime resources, some use heuristics to balance compression and speed, and others use different rules for what counts as a valid compression target 🙈
this means Responses to the same Query from different name server software can match in terms of DNS payload content (header, counts, RRs with name compression removed) without matching byte for byte on the wire 🫶
so it isn't possible to guarantee the DNS Response payload gets rebuilt byte for byte from C-DNS data ✨ it can at least, in principle, be rebuilt with the correct payload length (since the original Response length is captured), as long as there's enough knowledge of the commonly implemented name compression algorithms 💜 for example, a simple approach would be to try each algorithm in turn to see if it reproduces the original length, and stop at the first match 🦋 this wouldn't guarantee the correct algorithm was used, because it's possible to match the length while still not matching the on-the-wire bytes 😭 but without further information added to the C-DNS data, this is the best that can be achieved 🥹
Appendix B presents an example of two different compression algorithms used by well-known name server software 🎀
10. Data Collection ✨
okay so bestie, this section is NOT normative 🙈 it's a proposed algorithm for taking a captured stream of DNS Queries and Responses and spitting out a stream of Q/R data items, matching Queries with Responses whenever it can. it's giving "sorting out the group chat so every reply sits under the message it's replying to" 💅
For the purposes of this discussion, it is assumed that the input has been preprocessed such that:
- All IP fragmentation reassembly, TCP stream reassembly, and so on, have already been performed.
- Each message is associated with transport metadata required to generate the Primary ID (see Section 10.2.1).
- Each message has a well-formed DNS Header of 12 bytes, and (if present) the first Question in the Question section can be parsed to generate the Secondary ID (see below). As noted earlier, this requirement can result in a malformed Query being removed in the preprocessing stage, but the correctly formed Response with RCODE of FORMERR being present.
DNS messages are processed in the order they are delivered to the implementation.
ngl this next bit is my favourite 🦋 packet capture libraries do NOT always hand over packets in strict chronological order. this can happen on multi-core platforms, where packets hitting a network device get processed by different cores 🌈 on systems where people have spotted this, the timestamps are still consistent: a Query always has a timestamp before its Response. but the order the packets show up in the capture stream isn't strictly chronological, so a Response can show up in the stream before the Query that provoked it 😭 for this discussion, this non-chronological delivery is called "skew" ✨
because of skew, Response packets can arrive for matching before the corresponding Query. so to avoid making up fake Responses with no Query, and fake Queries with no Response, the matching algorithm MUST take the possibility of skew into account.
10.1. Matching Algorithm
here's the schematic for the matching algorithm, shown in Figure 3 🎀 it takes individual DNS Query or Response messages as input and outputs matched Q/R data items. the numbers in the figure point to the matching operations in Table 1. the nitty-gritty details (queues, timers, identifiers) come in the sections after this 💖
.----------------------.
| Process next message |<------------------+
`----------------------' |
| |
+------------------------------+ |
| Generate message identifiers | |
+------------------------------+ |
| |
Response | Query |
+--------------< >---------------+ |
| | |
+--------------------+ +--------------------+ |
| Find earliest QR | | Create QR item (2) | |
| item in OFIFO (1) | +--------------------+ |
+--------------------+ | |
| +---------------+ |
Match | No match | Append new QR | |
+--------< >------+ | item to OFIFO | |
| | +---------------+ |
+-----------+ +--------+ | |
| Update QR | | Add to | +-------------------+ |
| item (3) | | RFIFO | | Find earliest QR | |
+-----------+ +--------+ | item in RFIFO (1) | |
| | +-------------------+ |
+-----------------+ | |
| | |
| +----------------+ Match | No match |
| | Remove R |-------< >-----+ |
| | from RFIFO (3) | | |
| +----------------+ | |
| | | |
+--------------+-----------------------+ |
| |
+----------------------------------------------+ |
| Update all timed-out (QT) OFIFO QR items (4) | |
+----------------------------------------------+ |
| |
+--------------------------------+ |
| Remove all timed-out (ST) R | |
| from RFIFO, create QR item (5) | |
+--------------------------------+ |
____________________|_______________________ |
/ / |
/ Remove all consecutive done entries from /-------+
/ front of OFIFO for further processing /
/____________________________________________/
OFIFO = output FIFO containing Q/R data items (Section 10.6)
RFIFO = Response FIFO containing unmatched Response items
(Section 10.6)
QT = Query Timeout (Section 10.3)
ST = Skew Timeout (Section 10.3)
Figure 3: Query/Response Matching Algorithm +-----------+-------------------------------------------+
| Reference | Operation |
+-----------+-------------------------------------------+
| (1) | Find earliest QR item in FIFO where: |
| | * QR.done = false |
| | * QR.Q.PrimaryID == R.PrimaryID |
| | and, if both QR.Q and R have SecondaryID: |
| | * QR.Q.SecondaryID == R.SecondaryID |
| | |
| (2) | Set: |
| | QR.Q := Q |
| | QR.R := nil |
| | QR.done := false |
| | |
| (3) | Set: |
| | QR.R := R |
| | QR.done := true |
| | |
| (4) | Set: |
| | QR.done := true |
| | |
| (5) | Set: |
| | QR.Q := nil |
| | QR.R := R |
| | QR.done := true |
+-----------+-------------------------------------------+
Table 1: Operations Used in the Matching Algorithm10.2. Message Identifiers
10.2.1. Primary ID (Required) 🔑
every message gets a Primary ID 💅 it's made up of:
- Source IP Address
- Destination IP Address
- Source Port
- Destination Port
- Transport
- DNS Message ID
10.2.2. Secondary ID (Optional) 🌸
if it's there, the first Question in the Question section gets used as a Secondary ID for each message. heads up though 🙈 some well-formed DNS Queries have a QDCOUNT of 0, and some Responses can have a QDCOUNT of 0 too (like Responses with RCODE=FORMERR or NOTIMP). in that case the Secondary ID isn't used in matching.
10.3. Algorithm Parameters
- Query Timeout (QT). a Query arrives with timestamp t1. if no Response matching that Query has shown up before other input arrives timestamped later than (t1 + QT), a Q/R data item with only a Query gets recorded. the QT value is typically on the order of 5 seconds ⏰
- Skew Timeout (ST). a Response arrives with timestamp t2. if a Response hasn't been matched by a Query before input arrives timestamped later than (t2 + ST), a Q/R data item with only a Response gets recorded. the ST value is typically a few microseconds ⚡
10.4. Algorithm Requirements
the algorithm is designed to handle this input data:
- Multiple Queries with the same Primary ID (but different Secondary ID) arriving before any Responses for these Queries are seen.
- Multiple Queries with the same Primary and Secondary ID arriving before any Responses for these Queries are seen.
- Queries for which no later Response can be found within the specified timeout.
- Responses for which no previous Query can be found within the specified timeout.
10.5. Algorithm Limitations
okay real talk 😭 for cases 1 and 2 above, it is NOT possible to unambiguously match Queries with Responses. this algorithm chooses to match to the earliest Query with the correct Primary and Secondary ID. 🫶
10.6. Workspace
the algorithm uses two FIFO queues 🧚
- OFIFO: an output FIFO containing Q/R data items in chronological order.
- RFIFO: a FIFO holding Responses without a matching Query, in order of arrival.
10.7. Output
the output is a list of Q/R data items. both the Query and Response elements are optional in these items, so Q/R data items have one of three types of content:
- A matched pair of Query and Response messages
- A Query message with no Response
- A Response message with no Query
the timestamp of a list item is that of the Query for cases 1 and 2, and that of the Response for case 3. ✨
10.8. Post-Processing
when a capture ends, all items in the RFIFO get timed out immediately, generating Response-only entries in the OFIFO. these and all other remaining entries in the OFIFO should be treated as timed-out Queries. 🌈
11. Implementation Guidance 💜
this document makes no specific recommendations about "Canonical CBOR" (see Section 3.9 of [RFC7049]), BUT the following guidance may be useful to implementers 🍓
adhering to the first two rules given in Section 3.9 of [RFC7049] will minimize file sizes 🎀
adhering to the last two rules given in Section 3.9 of [RFC7049] for all maps and arrays would unacceptably constrain implementations 🙅♀️ for example, in real-time data collection in constrained environments, outputting Block Tables after Q/R data items and allowing indefinite-length maps and arrays could reduce memory requirements 🧠
it is RECOMMENDED that implementations that have fundamental restrictions on what data fields they can collect SHOULD always store hints with the bits unset for those fields, i.e., they unambiguously indicate that those data fields will be omitted from captured C-DNS. 🌸
11.1. Optional Data 🌸
okay so when you're decoding C-DNS data, some of the items you need for whatever function you're running might be missing 😭 consumers should consider providing configurable default values to use in place of the missing values in their output. that way your output doesn't just have holes in it 🫶
11.2. Trailing Bytes 🎀
a DNS Query message in a UDP or TCP payload can be followed by some extra (spurious) bytes that are not stored in C-DNS 👀
when DNS traffic goes over TCP, every message gets a two-byte length field in front of it, and that length gives the message length without counting the two-byte length field itself. in this context, trailing bytes can show up in two situations, and they have different results ✨
- the number of bytes you consume by fully parsing the message is less than the number given in the length field (i.e., the length field is incorrect and too large). the leftover bytes count as trailing bytes, kind of like with UDP, and get recorded that way. if this is the only thing that happens, you can still process a packet with multiple DNS messages where one or more of them have trailing bytes 💅
- there are surplus bytes between the end of a well-formed message and the start of the length field for the next message. here the first surplus byte gets treated as the first byte of the next length field, and parsing continues from there. that almost certainly makes the next message and any later ones in the packet count as malformed 💀 and this won't generate a trailing-bytes record for the well-formed message that got processed.
11.3. Limiting Collection of RDATA 🍓
implementations should consider providing a configurable maximum RDATA size for captures. this is so you don't run into memory issues when you hit huge zone transfer records 😩
11.4. Timestamps 🕰️
the preamble to each block has a timestamp of the earliest record in the Block. as described in Section 7.3.2.1, the timestamp is an array of two unsigned integers. the first one is a POSIX "time_t" [posix-time]. consumers of C-DNS need to know this, because it leaves out leap seconds, which can cause small oddities in the data. for example, it can throw off your Query throughput calculations bestie, keep that in mind 🫠
12. IANA Considerations 🦋
IANA has created a registry "C-DNS DNS Capture Format" that holds the subregistries defined in Sections 12.1 to 12.4 inclusive.
in every case, new entries can be added to the subregistries through Expert Review as defined in [RFC8126]. experts are expected to use their own expert judgment, and they should think about these general guidelines on top of any guidelines that are specific to a subregistry 🧚
- there should be a real and compelling use for any new value.
- values assigned should be carefully chosen to minimize storage requirements for common cases.
12.1. Transport Types 🚚
IANA has created a registry "C-DNS Transports" of C-DNS transport type identifiers. the main point of this registry is to give unique identifiers to all the transports used for DNS Queries.
this note is included in the registry: "In version 1.0 of C-DNS [RFC8618], there is a field to identify the type of DNS transport. This field is 4 bits in size." the initial contents are below. see Sections 7.3.2.3.2, 7.3.2.3.5, and 7.3.2.5 of this document 📖
+------------+------------------------+-----------+
| Identifier | Name | Reference |
+------------+------------------------+-----------+
| 0 | UDP | RFC 8618 |
| 1 | TCP | RFC 8618 |
| 2 | TLS | RFC 8618 |
| 3 | DTLS | RFC 8618 |
| 4 | HTTPS | RFC 8618 |
| 5-14 | Unassigned | |
| 15 | Non-standard transport | RFC 8618 |
+------------+------------------------+-----------+expert reviewers should keep this in mind: is the requested DNS transport described by a Standards Track RFC? ✨
12.2. Data Storage Flags 🏷️
IANA has created a registry "C-DNS Storage Flags" of C-DNS data storage flags. its main purpose is to give hints on how the stored data should be processed 💜
this note is included in the registry: "In version 1.0 of C-DNS [RFC8618], there is a field describing attributes of the data recorded. The field is a CBOR [RFC7049] unsigned integer holding bit flags." the initial contents are below. see Section 7.3.1.1.1 of this document 🫧
+------+------------------+-----------------------------+-----------+
| Bit | Name | Description | Reference |
+------+------------------+-----------------------------+-----------+
| 0 | anonymized-data | The data has been | RFC 8618 |
| | | anonymized. | |
| | | | |
| 1 | sampled-data | The data is sampled data. | RFC 8618 |
| | | | |
| 2 | normalized-names | Names in the data have been | RFC 8618 |
| | | normalized. | |
| | | | |
| 3-63 | Unassigned | | |
+------+------------------+-----------------------------+-----------+12.3. Response-Processing Flags 🌈
IANA has created a registry "C-DNS Response Flags" of C-DNS response-processing flags. its main purpose is to give hints about how a particular Response was generated 🔍
this note is included in the registry: "In version 1.0 of C-DNS [RFC8618], there is a field describing attributes of the Responses recorded. The field is a CBOR [RFC7049] unsigned integer holding bit flags." the initial contents are below. see Section 7.3.2.4.1 of this document 🍭
+------+------------+-------------------------------+-----------+
| Bit | Name | Description | Reference |
+------+------------+-------------------------------+-----------+
| 0 | from-cache | The Response came from cache. | RFC 8618 |
| 1-63 | Unassigned | | |
+------+------------+-------------------------------+-----------+12.4. AddressEvent Types 🎈
IANA has created a registry "C-DNS Address Event Types" of C-DNS AddressEvent types. its main purpose is to give unique identifiers to the different types of C-DNS address events, and to specify what goes in the optional companion field "ae-code" for each type 🪄
this note is included in the registry: "In version 1.0 of C-DNS [RFC8618], there is a field identifying types of the events related to client addresses. This field is a CBOR [RFC7049] unsigned integer. There is a related optional field "ae-code", which, if present, holds an additional CBOR unsigned integer giving additional information specific to the event type." the initial contents are below. see Section 7.3.2.5 of this document 💌
+------------------------+---------------+--------------+-----------+
| Identifier | Event Type | ae-code | Reference |
| | | Contents | |
+------------------------+---------------+--------------+-----------+
| 0 | TCP reset | None | RFC 8618 |
| | | | |
| 1 | ICMP time | ICMP code | RFC 8618 |
| | exceeded | [icmpcodes] | |
| | | | |
| 2 | ICMP | ICMP code | RFC 8618 |
| | destination | [icmpcodes] | |
| | unreachable | | |
| | | | |
| 3 | ICMPv6 time | ICMPv6 code | RFC 8618 |
| | exceeded | [icmp6codes] | |
| | | | |
| 4 | ICMPv6 | ICMPv6 code | RFC 8618 |
| | destination | [icmp6codes] | |
| | unreachable | | |
| | | | |
| 5 | ICMPv6 packet | ICMPv6 code | RFC 8618 |
| | too big | [icmp6codes] | |
| | | | |
| 6-18446744073709551615 | Unassigned | | |
+------------------------+---------------+--------------+-----------+expert reviewers should keep this in mind: "ae-code" contents must be defined for a type or, if that isn't appropriate, specified as "None". a specification of "None" needs less storage, so it's the preferred option 🌟
13. Security Considerations 🔐
any control interface MUST perform authentication and encryption. no exceptions, bestie 🚨
any data upload MUST be authenticated and encrypted. 🔒
14. Privacy Considerations 🫣
storing DNS traffic in PCAP and other formats is something operators have been doing for ages, and it's everywhere 🌍 section 2.5 of [DNS-Priv-Cons] looks at the risks to Internet users when DNS traffic data is stored on servers (recursive resolvers, authoritative servers, and rogue servers).
section 5.2 of [DNS-Priv-Svc] describes ways to reduce those risks for data stored on recursive resolvers (though it could extend to authoritative servers too). these include data-handling practices and ways to minimize data, pseudonymize IP addresses, and anonymize them. appendix C of [DNS-Priv-Svc] analyzes seven published anonymization processes 🧐 on top of that, the ICANN Root Server System Advisory Committee (RSSAC) recently published [RSSAC04] ("Recommendations on Anonymization Processes for Source IP Addresses Submitted for Future Analysis").
the analyses above use full data capture (for example, using PCAP) as the baseline for privacy, so this format specification adds no new user privacy issues beyond the ones that come with full data capture (which are quite severe 😱). what it does do is give you mechanisms to selectively record only certain fields at capture time, to protect user privacy better and to clearly show that data is sampled, anonymized, or both. it also has flags that show whether data normalization was done. normalization helps user privacy by making it harder to fingerprint individuals 🕵️♀️ the trade-off is that you might lose some ability to spot attack traffic through Query name signatures.
operators should think carefully about their operational requirements and privacy policies, and they SHOULD capture the minimum user data they need to meet those needs, right at the source 🫶✨
15. References 📚
15.1. Normative References 📚
okay so these are the references the spec leans on hard 💅 you'll want these open in another tab bestie 🫶
- [pcap-filter] tcpdump.org, "Manpage of PCAP-FILTER", November 2017,
https://www.tcpdump.org/manpages/pcap-filter.7.html. - [pcap-options] tcpdump.org, "Manpage of PCAP", July 2018,
https://www.tcpdump.org/manpages/pcap.3pcap.html. - [posix-time] The Open Group, "IEEE Standard for Information Technology--Portable Operating System Interface (POSIX(R)) Base Specifications, Issue 7", IEEE Standard 1003.1-2017, Section 4.16, DOI 10.1109/IEEESTD.2018.8277153.
- [RFC792] Postel, J., "Internet Control Message Protocol", STD 5, RFC 792, DOI 10.17487/RFC0792, September 1981,
https://www.rfc-editor.org/info/rfc792. - [RFC1035] Mockapetris, P., "Domain names - implementation and specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, November 1987,
https://www.rfc-editor.org/info/rfc1035. - [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997,
https://www.rfc-editor.org/info/rfc2119. - [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", STD 66, RFC 3986, DOI 10.17487/RFC3986, January 2005,
https://www.rfc-editor.org/info/rfc3986. - [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification", STD 89, RFC 4443, DOI 10.17487/RFC4443, March 2006,
https://www.rfc-editor.org/info/rfc4443. - [RFC6891] Damas, J., Graff, M., and P. Vixie, "Extension Mechanisms for DNS (EDNS(0))", STD 75, RFC 6891, DOI 10.17487/RFC6891, April 2013,
https://www.rfc-editor.org/info/rfc6891. - [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, October 2013,
https://www.rfc-editor.org/info/rfc7049. - [RFC7858] Hu, Z., Zhu, L., Heidemann, J., Mankin, A., Wessels, D., and P. Hoffman, "Specification for DNS over Transport Layer Security (TLS)", RFC 7858, DOI 10.17487/RFC7858, May 2016,
https://www.rfc-editor.org/info/rfc7858. - [RFC8094] Reddy, T., Wing, D., and P. Patil, "DNS over Datagram Transport Layer Security (DTLS)", RFC 8094, DOI 10.17487/RFC8094, February 2017,
https://www.rfc-editor.org/info/rfc8094. - [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017,
https://www.rfc-editor.org/info/rfc8126. - [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017,
https://www.rfc-editor.org/info/rfc8174. - [RFC8484] Hoffman, P. and P. McManus, "DNS Queries over HTTPS (DoH)", RFC 8484, DOI 10.17487/RFC8484, October 2018,
https://www.rfc-editor.org/info/rfc8484. - [RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610, June 2019,
https://www.rfc-editor.org/info/rfc8610.
15.2. Informative References 🔍
these ones are the extra reading 🧚 the tools, the schemas, the drafts, all the good stuff that helps you actually see the whole picture ✨
- [Avro] The Apache Software Foundation, "Apache Avro(TM)", 2019,
https://avro.apache.org/. - [ditl] DNS-OARC, "DITL", 2018,
https://www.dns-oarc.net/oarc/data/ditl. - [DNS-Priv-Cons] Bortzmeyer, S. and S. Dickinson, "DNS Privacy Considerations", Work in Progress, draft-ietf-dprive-rfc7626-bis-00, July 2019.
- [DNS-Priv-Svc] Dickinson, S., Overeinder, B., van Rijswijk-Deij, R., and A. Mankin, "Recommendations for DNS Privacy Service Operators", Work in Progress, draft-ietf-dprive-bcp-op-03, July 2019.
- [dnscap] DNS-OARC, "DNSCAP", 2018,
https://www.dns-oarc.net/tools/dnscap. - [dnstap] "dnstap", 2016,
https://dnstap.info/. - [dnstap-schema] "dnstap schema", commit d860ec1, November 2016,
https://github.com/dnstap/dnstap.pb/blob/master/dnstap.proto. - [dnsxml] Daley, J., Ed., Morris, S., and J. Dickinson, "dnsxml - A standard XML representation of DNS data", Work in Progress, draft-daley-dnsxml-00, July 2013.
- [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016,
https://www.dns-oarc.net/tools/dsc. - [gzip] "gzip",
https://www.gzip.org/. - [icmp6codes] IANA, "ICMPv6 "Code" Fields",
https://www.iana.org/assignments/icmpv6-parameters/. - [icmpcodes] IANA, "Code Fields",
https://www.iana.org/assignments/icmp-parameters/. - [IEEE802.1Q] IEEE, "IEEE Standard for Local and Metropolitan Area Networks--Bridges and Bridged Networks", IEEE Standard 802.1Q.
- [Knot] "Knot DNS",
https://www.knot-dns.cz/. - [lz4] "LZ4",
https://lz4.github.io/lz4/. - [mmark] Gieben, M., "mmark", commit de69698, May 2019,
https://github.com/mmarkdown/mmark. - [NSD] NLnet Labs, "NSD", 2019,
https://www.nlnetlabs.nl/projects/nsd/about/. - [opcodes] IANA, "DNS OpCodes",
https://www.iana.org/assignments/dns-parameters/. - [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", commit c9b2e89, February 2019,
https://github.com/DNS-OARC/PacketQ. - [pcap] "PCAP", 2019,
https://www.tcpdump.org/. - [pcapng] "pcapng: PCAP next generation file format specification", commit 3c35b6a, March 2019,
https://github.com/pcapng/pcapng. - [Protocol-Buffers] Google LLC, "Protocol Buffers",
https://developers.google.com/protocol-buffers/. - [rcodes] IANA, "DNS RCODEs",
https://www.iana.org/assignments/dns-parameters/. - [RFC5077] Salowey, J., Zhou, H., Eronen, P., and H. Tschofenig, "Transport Layer Security (TLS) Session Resumption without Server-Side State", RFC 5077, DOI 10.17487/RFC5077, January 2008,
https://www.rfc-editor.org/info/rfc5077. - [RFC7413] Cheng, Y., Chu, J., Radhakrishnan, S., and A. Jain, "TCP Fast Open", RFC 7413, DOI 10.17487/RFC7413, December 2014,
https://www.rfc-editor.org/info/rfc7413. - [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, December 2017,
https://www.rfc-editor.org/info/rfc8259. - [RFC8427] Hoffman, P., "Representing DNS Messages in JSON", RFC 8427, DOI 10.17487/RFC8427, July 2018,
https://www.rfc-editor.org/info/rfc8427. - [rrclasses] IANA, "DNS CLASSes",
https://www.iana.org/assignments/dns-parameters/. - [rrtypes] IANA, "Resource Record (RR) TYPEs",
https://www.iana.org/assignments/dns-parameters/. - [RSSAC04] ICANN, "Recommendations on Anonymization Processes for Source IP Addresses Submitted for Future Analysis", August 2018,
https://www.icann.org/en/system/files/files/rssac-040-07aug18-en.pdf. - [snappy] "snappy",
https://google.github.io/snappy/. - [snzip] "Snzip, a compression/decompression tool based on snappy", commit 809c6f2, October 2018,
https://github.com/kubo/snzip. - [xz] "XZ Utils",
https://tukaani.org/xz/. - [zstd] "Zstandard - Real-time data compression algorithm",
https://facebook.github.io/zstd/.
Appendix A. CDDL 🧩
okay this appendix is the formal CDDL [RFC8610] spec for C-DNS, so it's basically the blueprint for the file format 🎀
ngl CDDL can't do ranges for bitfields 😭 so when those values matter they get written as a group, but the group is commented out so the tooling doesn't start yelling that it's unused 🙈
; CDDL specification of the file format for C-DNS,
; which describes a collection of DNS messages and
; traffic metadata.
;
; The overall structure of a file.
;
File = [
file-type-id : "C-DNS",
file-preamble : FilePreamble,
file-blocks : [* Block],
]
;
; The File Preamble.
;
FilePreamble = {
major-format-version => 1,
minor-format-version => 0,
? private-version => uint,
block-parameters => [+ BlockParameters],
}
major-format-version = 0
minor-format-version = 1
private-version = 2
block-parameters = 3
BlockParameters = {
storage-parameters => StorageParameters,
? collection-parameters => CollectionParameters,
}
storage-parameters = 0
collection-parameters = 1
IPv6PrefixLength = 1..128
IPv4PrefixLength = 1..32
OpcodeRange = 0..15
RRTypeRange = 0..65535
StorageParameters = {
ticks-per-second => uint,
max-block-items => uint,
storage-hints => StorageHints,
opcodes => [+ OpcodeRange],
rr-types => [+ RRTypeRange],
? storage-flags => StorageFlags,
? client-address-prefix-ipv4 => IPv4PrefixLength,
? client-address-prefix-ipv6 => IPv6PrefixLength,
? server-address-prefix-ipv4 => IPv4PrefixLength,
? server-address-prefix-ipv6 => IPv6PrefixLength,
? sampling-method => tstr,
? anonymization-method => tstr,
}
ticks-per-second = 0
max-block-items = 1
storage-hints = 2
opcodes = 3
rr-types = 4
storage-flags = 5
client-address-prefix-ipv4 = 6
client-address-prefix-ipv6 = 7
server-address-prefix-ipv4 = 8
server-address-prefix-ipv6 = 9
sampling-method = 10
anonymization-method = 11
; A hint indicates whether the collection method will always omit
; the item from the file.
StorageHints = {
query-response-hints => QueryResponseHints,
query-response-signature-hints =>
QueryResponseSignatureHints,
rr-hints => RRHints,
other-data-hints => OtherDataHints,
}
query-response-hints = 0
query-response-signature-hints = 1
rr-hints = 2
other-data-hints = 3okay so the CDDL keeps going, and this bit is where the query/response hints, the signature hints, and the data layout all get their little numbered keys bestie 💅🎀
QueryResponseHintValues = &(
time-offset : 0,
client-address-index : 1,
client-port : 2,
transaction-id : 3,
qr-signature-index : 4,
client-hoplimit : 5,
response-delay : 6,
query-name-index : 7,
query-size : 8,
response-size : 9,
response-processing-data : 10,
query-question-sections : 11, ; Second & subsequent
; Questions
query-answer-sections : 12,
query-authority-sections : 13,
query-additional-sections : 14,
response-answer-sections : 15,
response-authority-sections : 16,
response-additional-sections : 17,
)
QueryResponseHints = uint .bits QueryResponseHintValues
QueryResponseSignatureHintValues = &(
server-address-index : 0,
server-port : 1,
qr-transport-flags : 2,
qr-type : 3,
qr-sig-flags : 4,
query-opcode : 5,
qr-dns-flags : 6,
query-rcode : 7,
query-classtype-index : 8,
query-qdcount : 9,
query-ancount : 10,
query-nscount : 11,
query-arcount : 12,
query-edns-version : 13,
query-udp-size : 14,
query-opt-rdata-index : 15,
response-rcode : 16,
)
QueryResponseSignatureHints =
uint .bits QueryResponseSignatureHintValues
RRHintValues = &(
ttl : 0,
rdata-index : 1,
)
RRHints = uint .bits RRHintValues
OtherDataHintValues = &(
malformed-messages : 0,
address-event-counts : 1,
)
OtherDataHints = uint .bits OtherDataHintValues
StorageFlagValues = &(
anonymized-data : 0,
sampled-data : 1,
normalized-names : 2,
)
StorageFlags = uint .bits StorageFlagValues
; Metadata about data collection
VLANIdRange = 1..4094
CollectionParameters = {
? query-timeout => uint, ; Milliseconds
? skew-timeout => uint, ; Microseconds
? snaplen => uint,
? promisc => bool,
? interfaces => [+ tstr],
? server-addresses => [+ IPAddress],
? vlan-ids => [+ VLANIdRange],
? filter => tstr,
? generator-id => tstr,
? host-id => tstr,
}
query-timeout = 0
skew-timeout = 1
snaplen = 2
promisc = 3
interfaces = 4
server-addresses = 5
vlan-ids = 6
filter = 7
generator-id = 8
host-id = 9
;
; Data in the file is stored in Blocks.
;
Block = {
block-preamble => BlockPreamble,
? block-statistics => BlockStatistics, ; Much of this
; could be derived
? block-tables => BlockTables,
? query-responses => [+ QueryResponse],
? address-event-counts => [+ AddressEventCount],
? malformed-messages => [+ MalformedMessage],
}
block-preamble = 0
block-statistics = 1
block-tables = 2
query-responses = 3
address-event-counts = 4
malformed-messages = 5
;
; The (mandatory) preamble to a Block.
;
BlockPreamble = {
? earliest-time => Timestamp,
? block-parameters-index => uint .default 0,
}
earliest-time = 0
block-parameters-index = 1
; Ticks are sub-second intervals. The number of ticks in a second is
; file/block metadata. Signed and unsigned tick types are defined.
ticks = int
uticks = uint
Timestamp = [
timestamp-secs : uint, ; POSIX time
timestamp-ticks : uticks,
]
;
; Statistics about the Block contents.
;
BlockStatistics = {
? processed-messages => uint,
? qr-data-items => uint,
? unmatched-queries => uint,
? unmatched-responses => uint,
? discarded-opcode => uint,
? malformed-items => uint,
}
processed-messages = 0
qr-data-items = 1
unmatched-queries = 2
unmatched-responses = 3
discarded-opcode = 4
malformed-items = 5
;
; Tables of common data referenced from records in a Block.
;
BlockTables = {
? ip-address => [+ IPAddress],
? classtype => [+ ClassType],
? name-rdata => [+ bstr], ; Holds both names
; and RDATA
? qr-sig => [+ QueryResponseSignature],
? QuestionTables,
? RRTables,
? malformed-message-data => [+ MalformedMessageData],
}
ip-address = 0
classtype = 1
name-rdata = 2
qr-sig = 3
qlist = 4
qrr = 5
rrlist = 6
rr = 7
malformed-message-data = 8
IPv4Address = bstr .size (0..4)
IPv6Address = bstr .size (0..16)
IPAddress = IPv4Address / IPv6Address
ClassType = {
type => uint,
class => uint,
}
type = 0
class = 1
QueryResponseSignature = {
? server-address-index => uint,
? server-port => uint,
? qr-transport-flags => QueryResponseTransportFlags,
? qr-type => QueryResponseType,
? qr-sig-flags => QueryResponseFlags,
? query-opcode => uint,
? qr-dns-flags => DNSFlags,
? query-rcode => uint,
? query-classtype-index => uint,
? query-qdcount => uint,
? query-ancount => uint,
? query-nscount => uint,
? query-arcount => uint,
? query-edns-version => uint,
? query-udp-size => uint,
? query-opt-rdata-index => uint,
? response-rcode => uint,
}
server-address-index = 0
server-port = 1
qr-transport-flags = 2
qr-type = 3
qr-sig-flags = 4
query-opcode = 5
qr-dns-flags = 6
query-rcode = 7
query-classtype-index = 8
query-qdcount = 9
query-ancount = 10
query-nscount = 11
query-arcount = 12
query-edns-version = 13
query-udp-size = 14
query-opt-rdata-index = 15
response-rcode = 16
; Transport gives the values that may appear in bits 1..4 of
; TransportFlags. There is currently no way to express this in
; CDDL, so Transport is unused. To avoid confusion when used
; with CDDL tools, it is commented out.
;
; Transport = &(
; udp : 0,
; tcp : 1,
; tls : 2,
; dtls : 3,
; https : 4,
; non-standard : 15,
; )
TransportFlagValues = &(
ip-version : 0, ; 0=IPv4, 1=IPv6
) / (1..4)
TransportFlags = uint .bits TransportFlagValues
QueryResponseTransportFlagValues = &(
query-trailingdata : 5,
) / TransportFlagValues
QueryResponseTransportFlags =
uint .bits QueryResponseTransportFlagValues
QueryResponseType = &(
stub : 0,
client : 1,
resolver : 2,
auth : 3,
forwarder : 4,
tool : 5,
)
QueryResponseFlagValues = &(
has-query : 0,
has-response : 1,
query-has-opt : 2,
response-has-opt : 3,
query-has-no-question : 4,
response-has-no-question: 5,
)
QueryResponseFlags = uint .bits QueryResponseFlagValues
DNSFlagValues = &(
query-cd : 0,
query-ad : 1,
query-z : 2,
query-ra : 3,
query-rd : 4,
query-tc : 5,
query-aa : 6,
query-do : 7,
response-cd: 8,
response-ad: 9,
response-z : 10,
response-ra: 11,
response-rd: 12,
response-tc: 13,
response-aa: 14,
)
DNSFlags = uint .bits DNSFlagValues
QuestionTables = (
qlist => [+ QuestionList],
qrr => [+ Question]
)
QuestionList = [+ uint] ; Index of Question
Question = { ; Second and subsequent Questions
name-index => uint, ; Index to a name in the
; name-rdata table
classtype-index => uint,
}
name-index = 0
classtype-index = 1
RRTables = (
rrlist => [+ RRList],
rr => [+ RR]
)
RRList = [+ uint] ; Index of RR
RR = {
name-index => uint, ; Index to a name in the
; name-rdata table
classtype-index => uint,
? ttl => uint,
? rdata-index => uint, ; Index to RDATA in the
; name-rdata table
}
; Other map key values already defined above.
ttl = 2
rdata-index = 3
MalformedMessageData = {
? server-address-index => uint,
? server-port => uint,
? mm-transport-flags => TransportFlags,
? mm-payload => bstr,
}
; Other map key values already defined above.
mm-transport-flags = 2
mm-payload = 3
;
; A single Query/Response data item.
;
QueryResponse = {
? time-offset => uticks, ; Time offset from
; start of Block
? client-address-index => uint,
? client-port => uint,
? transaction-id => uint,
? qr-signature-index => uint,
? client-hoplimit => uint,
? response-delay => ticks,
? query-name-index => uint,
? query-size => uint, ; DNS size of Query
? response-size => uint, ; DNS size of Response
? response-processing-data => ResponseProcessingData,
? query-extended => QueryResponseExtended,
? response-extended => QueryResponseExtended,
}
time-offset = 0
client-address-index = 1
client-port = 2
transaction-id = 3
qr-signature-index = 4
client-hoplimit = 5
response-delay = 6
query-name-index = 7
query-size = 8
response-size = 9
response-processing-data = 10
query-extended = 11
response-extended = 12
ResponseProcessingData = {
? bailiwick-index => uint,
? processing-flags => ResponseProcessingFlags,
}
bailiwick-index = 0
processing-flags = 1
ResponseProcessingFlagValues = &(
from-cache : 0,
)
ResponseProcessingFlags = uint .bits ResponseProcessingFlagValues QueryResponseExtended = {
? question-index => uint, ; Index of QuestionList
? answer-index => uint, ; Index of RRList
? authority-index => uint,
? additional-index => uint,
}
question-index = 0
answer-index = 1
authority-index = 2
additional-index = 3
;
; Address event data.
;
AddressEventCount = {
ae-type => &AddressEventType,
? ae-code => uint,
ae-address-index => uint,
? ae-transport-flags => TransportFlags,
ae-count => uint,
}
ae-type = 0
ae-code = 1
ae-address-index = 2
ae-transport-flags = 3
ae-count = 4
AddressEventType = (
tcp-reset : 0,
icmp-time-exceeded : 1,
icmp-dest-unreachable : 2,
icmpv6-time-exceeded : 3,
icmpv6-dest-unreachable: 4,
icmpv6-packet-too-big : 5,
)
;
; Malformed messages.
;
MalformedMessage = {
? time-offset => uticks, ; Time offset from
; start of Block
? client-address-index => uint,
? client-port => uint,
? message-data-index => uint,
}
; Other map key values already defined above.
message-data-index = 3Appendix B. DNS Name Compression Example 🎀
okay so this appendix is all about how DNS names get squished down when they go into a packet 🧚 the basic algorithm follows the guidance in [RFC1035], and it's honestly so clean. you collect each name, plus the offset in the packet where it starts, while the packet is being built. every time a new name gets added, it gets offered up to each of the names already collected, in the order they were collected, starting with the very first one. if (1) some labels at the end of the name can be swapped for a reference back to part (or all) of an earlier name AND (2) the part that's still uncompressed is shorter than the best compression found so far, then that earlier name becomes the compression target for the new name. it's giving tidy little math bestie 💅
the tables below walk through the process one name at a time 🦋 in the example packet, the first name added is foo.example, and it can't be compressed because there's nothing before it to point at.
+---+-------------+--------------+--------------------+
| N | Name | Uncompressed | Compression Target |
+---+-------------+--------------+--------------------+
| 1 | foo.example | foo.example | None |
+---+-------------+--------------+--------------------+the next name is bar.example. it gets matched against foo.example, and the example part can be used as a compression target. that leaves bar as the uncompressed part 🌸
+---+-------------+--------------+-----------------------+
| N | Name | Uncompressed | Compression Target |
+---+-------------+--------------+-----------------------+
| 1 | foo.example | foo.example | None |
| 2 | bar.example | bar | 1 + offset to example |
+---+-------------+--------------+-----------------------+the third name is www.bar.example, and this one is where it gets spicy 🔥 it gets matched against foo.example first, and again the example part is recorded as a possible target, with www.bar still uncompressed. then it gets matched against the second name, bar.example, which can be a target too. the uncompressed part is just www there, which is a better compression, so that's the one that gets adopted. main character energy ✨
+---+-----------------+--------------+-----------------------+
| N | Name | Uncompressed | Compression Target |
+---+-----------------+--------------+-----------------------+
| 1 | foo.example | foo.example | None |
| 2 | bar.example | bar | 1 + offset to example |
| 3 | www.bar.example | www | 2 |
+---+-----------------+--------------+-----------------------+also there's a little optimization 🍓 if a name is already perfectly compressed, meaning nothing in it is left uncompressed, then no more names get considered for compression. why check more when you already hit the best possible result? 🙈
B.1. NSD Compression Algorithm
using that basic algorithm, the packet lengths of Responses from the Name Server Daemon (NSD) [NSD] can be matched almost exactly 💖 at the time of writing, a tiny number (<.01%) of the rebuilt packets had the wrong lengths, which is honestly iconic accuracy.
B.2. Knot Authoritative Compression Algorithm
the Knot Authoritative name server [Knot] compresses differently 👑 that's on purpose, because it's tuned to balance runtime speed against how much compression it actually gains. the short version (skipping the complications) is that Knot Authoritative only considers the QNAME and the names in the immediately preceding RR section in an RRSET as compression targets.
you can build a set of smart heuristics, described below, to mimic that. it's not perfect, but the output is nearly as good a match as NSD's, just not quite 🥹 here are the heuristics:
- A match is only perfect if the name is completely compressed AND the TYPE of the section in which the name occurs matches the TYPE of the name used as the compression target.
- If the name occurs in RDATA:
- If the compression target name is in a Query, then only the first RR in an RRSET can use that name as a compression target.
- The compression target name MUST be in RDATA.
- The name section TYPE must match the compression target name section TYPE.
- The compression target name MUST be in the immediately preceding RR in the RRSET.
using this algorithm, less than 0.1% of the rebuilt packets had the wrong lengths 🫶
B.3. Observed Differences
in sample traffic collected on a root name server, around 2-4% of Responses generated by Knot had different packet lengths than the ones produced by NSD 😭 so yeah, the heuristics aren't a perfect copy, but they're really close, and that's the tea ☕
Appendix C. Comparison of Binary Formats ✨
okay bestie, before we picked a format, we compared a bunch of binary serialization options 🧚 and lined them all up against JSON too, because why not 💅
- Apache Avro [Avro]. Data is stored according to a predefined schema, and the schema always rides along inside the data file 📦. That means the data can be stored untagged for a smaller size, and an Avro library can write and read it 🫶
- At the time of writing, Avro libraries exist for C, C++, C#, Java, Python, Ruby, and PHP. Optionally, tools exist for C++, Java, and C# to generate encoding and decoding code 🔥
- Google Protocol Buffers [Protocol-Buffers]. Data is stored according to a predefined schema, and a generator uses that schema to make encoding and decoding code 🛠️ Data can be stored untagged for a smaller size too. The schema is NOT stored with the data though, so unlike Avro, it can't be read with a generic library 🙈
- You have to generate code for a specific data schema before you can read and write data with it. At the time of writing, Google's code generator can handle encoding and decoding for C++, Go, Java, Python, Ruby, C#, Objective-C, JavaScript, and PHP 💜
- CBOR [RFC7049]. This one is comparable to JSON but in a binary form 🦋 It doesn't use a predefined schema, so the data is always stored tagged. Still, CBOR data schemas can be described using CDDL [RFC8610], and there are tools that check whether data files match the schema 🌸
- CBOR is simple to understand and simple to implement 🍓 At the time of writing, the CBOR website lists implementations for 16 languages ⭐
Avro and Protocol Buffers both allow untagged data, but because they lean on the data schema to pull that off, their implementation is way more complex than CBOR 😭 Using Avro or Protocol Buffers in an unsupported environment would take noticeably more development effort than CBOR would.
We wrote a test program that reads input from a PCAP file and writes output using one of two basic structures 🧪 Either it's a simple structure, where each Query/Response pair lives in a single record entry, or it's the C-DNS block structure.
Next, we squished those output files with a bunch of common general-purpose lossless compression tools to see how compressible each format was 🗜️ The tools we used were:
- snzip [snzip]. A command-line compression tool based on the Google Snappy library [snappy].
- lz4 [lz4]. The command-line compression tool from the reference C LZ4 implementation.
- gzip [gzip]. The ubiquitous GNU zip tool.
- zstd [zstd]. Compression using the Zstandard algorithm.
- xz [xz]. A popular tool that's known for high compression.
In every case, the compression tools ran with their default settings.
Note that this document does NOT mandate compression, or any particular compression scheme 🙅 It does, though, anticipate that in practice the output data will get general-purpose compression, so that's worth keeping in mind 🫶
"test.pcap" was the main event 🎀 It's a 662 MB capture of sample data from a root instance. The table below shows the formatted size and the size after compression (abbreviated to Comp. in the table headers), along with the task Resident Set Size (RSS) and the user time the compression took. File sizes are in MB, RSS is in kB, and user time is in seconds 💅
+-------------+-----------+-------+------------+-------+-----------+
| Format | File Size | Comp. | Comp. Size | RSS | User Time |
+-------------+-----------+-------+------------+-------+-----------+
| PCAP | 661.87 | snzip | 212.48 | 2696 | 1.26 |
| | | lz4 | 181.58 | 6336 | 1.35 |
| | | gzip | 153.46 | 1428 | 18.20 |
| | | zstd | 87.07 | 3544 | 4.27 |
| | | xz | 49.09 | 97416 | 160.79 |
| | | | | | |
| JSON simple | 4113.92 | snzip | 603.78 | 2656 | 5.72 |
| | | lz4 | 386.42 | 5636 | 5.25 |
| | | gzip | 271.11 | 1492 | 73.00 |
| | | zstd | 133.43 | 3284 | 8.68 |
| | | xz | 51.98 | 97412 | 600.74 |
| | | | | | |
| Avro simple | 640.45 | snzip | 148.98 | 2656 | 0.90 |
| | | lz4 | 111.92 | 5828 | 0.99 |
| | | gzip | 103.07 | 1540 | 11.52 |
| | | zstd | 49.08 | 3524 | 2.50 |
| | | xz | 22.87 | 97308 | 90.34 |
| | | | | | |
| CBOR simple | 764.82 | snzip | 164.57 | 2664 | 1.11 |
| | | lz4 | 120.98 | 5892 | 1.13 |
| | | gzip | 110.61 | 1428 | 12.88 |
| | | zstd | 54.14 | 3224 | 2.77 |
| | | xz | 23.43 | 97276 | 111.48 |
| | | | | | |
| PBuf simple | 749.51 | snzip | 167.16 | 2660 | 1.08 |
| | | lz4 | 123.09 | 5824 | 1.14 |
| | | gzip | 112.05 | 1424 | 12.75 |
| | | zstd | 53.39 | 3388 | 2.76 |
| | | xz | 23.99 | 97348 | 106.47 |
| | | | | | |
| JSON block | 519.77 | snzip | 106.12 | 2812 | 0.93 |
| | | lz4 | 104.34 | 6080 | 0.97 |
| | | gzip | 57.97 | 1604 | 12.70 |
| | | zstd | 61.51 | 3396 | 3.45 |
| | | xz | 27.67 | 97524 | 169.10 |
| | | | | | |
| Avro block | 60.45 | snzip | 48.38 | 2688 | 0.20 |
| | | lz4 | 48.78 | 8540 | 0.22 |
| | | gzip | 39.62 | 1576 | 2.92 |
| | | zstd | 29.63 | 3612 | 1.25 |
| | | xz | 18.28 | 97564 | 25.81 |
| | | | | | |
| CBOR block | 75.25 | snzip | 53.27 | 2684 | 0.24 |
| | | lz4 | 51.88 | 8008 | 0.28 |
| | | gzip | 41.17 | 1548 | 4.36 |
| | | zstd | 30.61 | 3476 | 1.48 |
| | | xz | 18.15 | 97556 | 38.78 |
| | | | | | |
| PBuf block | 67.98 | snzip | 51.10 | 2636 | 0.24 |
| | | lz4 | 52.39 | 8304 | 0.24 |
| | | gzip | 40.19 | 1520 | 3.63 |
| | | zstd | 31.61 | 3576 | 1.40 |
| | | xz | 17.94 | 97440 | 33.99 |
+-------------+-----------+-------+------------+-------+-----------+The above results are discussed in the following sections 👑
C.1. Comparison with Full PCAP Files
okay first big question: does moving away from PCAP actually get us anything significant? 🤔
The simple binary formats are usually bigger than PCAP, even though they leave out stuff like Ethernet Media Access Control (MAC) addresses 🙃 BUT they need less CPU to compress than PCAP does, and the compressed files end up smaller than compressed PCAP. Slay 💖
C.2. Simple versus Block Coding
the whole point of block coding is data deduplication on Query/Response records within a block 🧹 The simple and block formats shown above store EXACTLY the same information for each Query/Response record. That info is parsed out of the DNS traffic in the input PCAP file, and in every case each field has an identifier and typed field data 🏷️
The dedup on the block formats gives an order-of-magnitude reduction in file size compared to the simple formats 😭 The compression tools can find and exploit a lot of that duplication too, but since the dedup process knows about DNS traffic, the block formats keep a size advantage. That advantage shrinks as stronger compression gets applied, which, again, is what you'd expect 📉 Even with the strongest compression, though, the block-formatted data is still around 75% of the size of the simple format, and its compression needs roughly a third of the CPU time 🔥
C.3. Binary versus Text Formats
text formats have a lot going for them, especially for ad hoc data inspection and extraction 🧐 So it felt worth doing a direct comparison, implementing JSON versions of the simple and block formats 🎀
Focusing on the JSON block format, the files are a significant fraction of an order of magnitude larger than the binary formats 😬 After compression, it follows from that starting point: the stronger compression gives files that are 150% of the size of similarly compressed binary format files, and they take over 4x more CPU to compress 💀
C.4. Performance
back to the block formats, all three make format files that are close to an order of magnitude smaller than the original "test.pcap" file 🦋 CBOR gives the largest files, and Avro gives the smallest, 20% smaller than CBOR ✨
Once compression is in the picture, though, the size gap narrows. At medium compression (with gzip), the size difference is 4%. With strong compression (with xz), it drops to 2%, with Avro the largest and Protocol Buffers the smallest, although CBOR and Protocol Buffers need slightly more compression CPU 🫶
The measurements above don't include the CPU needed to generate the format files. Measurements show that writing Avro takes 10% more CPU than CBOR or Protocol Buffers 😤 So it looks like Avro's compression CPU advantage is probably canceled out by its bigger CPU cost when writing 🙈
C.5. Conclusions
the assessments above lead us to a binary format file using blocking 🎉
Like we said earlier, this document anticipates that output data will be compressed 🗜️ There's no compelling case for one binary serialization format in terms of final file size or machine resources used, so the choice has to come down mostly to other factors 💅 CBOR was therefore chosen as the binary serialization format for the reasons listed in Section 5 👑
C.6. Block Size Choice
okay so bestie, if we're using a CBOR format with blocking, we have to ask what a sensible default is for the maximum number of Query/Response pairs in a block 🧠✨ There are two parts to this:
- What happens to performance when you change the block size in the format file? 🏎️
- What happens to the size of the format file before and after compression? 📦
The first table looks at performance. It shows how a C++ program converting "test.pcap" to C-DNS behaves at each block size. File sizes are in MB, RSS is in kB, and user time is in seconds 💅
+------------+-----------+--------+-----------+
| Block Size | File Size | RSS | User Time |
+------------+-----------+--------+-----------+
| 1,000 | 133.46 | 612.27 | 15.25 |
| 5,000 | 89.85 | 676.82 | 14.99 |
| 10,000 | 76.87 | 752.40 | 14.53 |
| 20,000 | 67.86 | 750.75 | 14.49 |
| 40,000 | 61.88 | 736.30 | 14.29 |
| 80,000 | 58.08 | 694.16 | 14.28 |
| 160,000 | 55.94 | 733.84 | 14.44 |
| 320,000 | 54.41 | 799.20 | 13.97 |
+------------+-----------+--------+-----------+So bigger blocks nudge max RSS up a little 📈, but there's no real effect on CPU. If anything, it gets a tiny bit lower 🦋
The next table shows how increasing block size changes the output file size for each compression option 🎀
+------------+--------+-------+-------+-------+-------+-------+
| Block Size | None | snzip | lz4 | gzip | zstd | xz |
+------------+--------+-------+-------+-------+-------+-------+
| 1,000 | 133.46 | 90.52 | 90.03 | 74.65 | 44.78 | 25.63 |
| 5,000 | 89.85 | 59.69 | 59.43 | 46.99 | 37.33 | 22.34 |
| 10,000 | 76.87 | 50.39 | 50.28 | 38.94 | 33.62 | 21.09 |
| 20,000 | 67.86 | 43.91 | 43.90 | 33.24 | 32.62 | 20.16 |
| 40,000 | 61.88 | 39.63 | 39.69 | 29.44 | 28.72 | 19.52 |
| 80,000 | 58.08 | 36.93 | 37.01 | 27.05 | 26.25 | 19.00 |
| 160,000 | 55.94 | 35.10 | 35.06 | 25.44 | 24.56 | 19.63 |
| 320,000 | 54.41 | 33.87 | 33.74 | 24.36 | 23.44 | 18.66 |
+------------+--------+-------+-------+-------+-------+-------+Honestly, the best default is going to depend on the compression you're using, your traffic characteristics, how often the output file rolls over, and so on 🫶 There's definitely room to tune it. With a strong compression scheme, though, block sizes over 10,000 Query/Response pairs look like they'd only give limited improvements 💜
Appendix D. Data Fields for Traffic Regeneration
D.1. Recommended Fields for Traffic Regeneration
This section lists the data fields you'd need to capture to get the fullest PCAP traffic reconstruction for well-formed DNS messages that C-DNS can actually support 🎀
- All data fields in the
QueryResponsetype exceptresponse-processing-data. - All data fields in the
QueryResponseSignaturetype exceptqr-type. - All data fields in the
RR TYPE.
D.2. Issues with Small Data Captures
Now the other end of the spectrum, which is a sneaky corner case 👀 If you decide to capture a smaller data set than the one recommended above, things get weird. The list below is a subset of those fields. If you only capture these, you can't even do a minimal traffic reconstruction well, because there isn't enough info to tell whether a Query/Response data item holds just a Query, just a Response, or a full Query/Response pair 😭
- The following data fields from the
QueryResponsetype:time-offsetclient-address-indexclient-porttransaction-idquery-name-index
- The following data fields from the
QueryResponseSignaturetype:server-address-indexserver-portqr-transport-flagsquery-classtype-index
Here's the fix 🦋 If you also capture qr-sig-flags, you'll have enough to do a minimal traffic reconstruction (assuming suitable defaults are provided for the remaining fields). Capturing response-delay, query-opcode, and response-rcode as well means you won't have to lean on defaults that could be misleading for those values, and you should end up with a PCAP that shows the basics of the real traffic flow ✨
Acknowledgements
The authors want to thank CZ.NIC, especially Tomas Gavenciak, for a lot of useful chats about binary formats, compression, and packet matching 🫶 Thanks also to Jan Vcelak and Wouter Wijngaards for talking through name compression with them, and to Paul Hoffman for a detailed review of this document and the C-DNS CDDL 💖
Thanks also to Robert Edmonds, Jerry Lundstrom, Richard Gibson, Stephane Bortzmeyer, and many other members of DNSOP for reviewing it 🌸
Also, huge thanks to Miek Gieben for [mmark] 🌈
Authors' Addresses
John Dickinson
Sinodun IT
Magdalen Centre
Oxford Science Park
Oxford OX4 4GA
United Kingdom
Email: jad@sinodun.com
Jim Hague
Sinodun IT
Magdalen Centre
Oxford Science Park
Oxford OX4 4GA
United Kingdom
Email: jim@sinodun.com
Sara Dickinson
Sinodun IT
Magdalen Centre
Oxford Science Park
Oxford OX4 4GA
United Kingdom
Email: sara@sinodun.com
Terry Manderson
ICANN
12025 Waterfront Drive
Suite 300
Los Angeles, CA 90094-2536
United States of America
Email: terry.manderson@icann.org
John Bond
Wikimedia Foundation, Inc.
1 Montgomery Street
Suite 1600
San Francisco, CA 94104
United States of America
Email: ietf-wikimedia@johnbond.org📦 where this page comes from
- ietf: rfc8618 2019-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms