RFC 8483

Yeti DNS Testbed

📅 published October 2018, from ietf

Title Block 🎀 (RFC 8483)

Independent Submission 💖 Filed by L. Song, Ed., D. Liu (Beijing Internet Institute), P. Vixie (TISF), A. Kato (Keio/WIDE), and S. Kerr ✨

Category: Informational 🦋 · ISSN: 2070-1721 · Dated October 2018 🌸

This one doesn't obsolete or update anything, so it's a totally fresh start, bestie 💅

Yeti DNS Testbed 🧚

Abstract 💜

Okay so, Yeti DNS is an experimental, non-production root server testbed 🍓. It gives people a playground where they can run technical and operational experiments without any risk to the real production root server infrastructure 🛡️✨. This doc is only here to share the technical and operational experience of setting up a system that is similar to, but different from, the Root Server system (the one the Internet's Domain Name System is designed and built on) 🌈.

Status of This Memo 🌷

Lowkey important: this document is not an Internet Standards Track specification. It's published purely for informational purposes 📚.

It's a contribution to the RFC Series, independent of any other RFC stream 💅. The RFC Editor picked to publish it at their discretion and makes no statement about whether it's worth implementing or deploying 🙈. Documents the RFC Editor approves aren't candidates for any level of Internet Standard, so check out Section 2 of RFC 7841 if you wanna know more 🔥.

Want the current status, errata, or a way to send feedback? Head to https://www.rfc-editor.org/info/rfc8483 🫶.

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved 💖.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they were in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document 🎀.

Table of Contents 🦋

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
2.  Requirements Notation and Conventions . . . . . . . . . . . .   5
3.  Areas of Study  . . . . . . . . . . . . . . . . . . . . . . .   5
  3.1.  Implementation of a Testbed like the Root Server System .   5
  3.2.  Yeti-Root Zone Distribution . . . . . . . . . . . . . . .   5
  3.3.  Yeti-Root Server Names and Addressing . . . . . . . . . .   5
  3.4.  IPv6-Only Yeti-Root Servers . . . . . . . . . . . . . . .   6
  3.5.  DNSSEC in the Yeti-Root Zone  . . . . . . . . . . . . . .   6
4.  Yeti DNS Testbed Infrastructure . . . . . . . . . . . . . . .   7
  4.1.  Root Zone Retrieval . . . . . . . . . . . . . . . . . . .   8
  4.2.  Transformation of Root Zone to Yeti-Root Zone . . . . . .   9
    4.2.1.  ZSK and KSK Key Sets Shared between DMs . . . . . . .  10
    4.2.2.  Unique ZSK per DM; No Shared KSK  . . . . . . . . . .  10
    4.2.3.  Preserving Root Zone NSEC Chain and ZSK RRSIGs  . . .  11
  4.3.  Yeti-Root Zone Distribution . . . . . . . . . . . . . . .  12
  4.4.  Synchronization of Service Metadata . . . . . . . . . . .  12
  4.5.  Yeti-Root Server Naming Scheme  . . . . . . . . . . . . .  13
  4.6.  Yeti-Root Servers . . . . . . . . . . . . . . . . . . . .  14
  4.7.  Experimental Traffic  . . . . . . . . . . . . . . . . . .  16
  4.8.  Traffic Capture and Analysis  . . . . . . . . . . . . . .  16
5.  Operational Experience with the Yeti DNS Testbed  . . . . . .  17
  5.1.  Viability of IPv6-Only Operation  . . . . . . . . . . . .  17
    5.1.1.  IPv6 Fragmentation  . . . . . . . . . . . . . . . . .  18
    5.1.2.  Serving IPv4-Only End-Users . . . . . . . . . . . . .  19
  5.2.  Zone Distribution . . . . . . . . . . . . . . . . . . . .  19
    5.2.1.  Zone Transfers  . . . . . . . . . . . . . . . . . . .  19
    5.2.2.  Delays in Yeti-Root Zone Distribution . . . . . . . .  20
    5.2.3.  Mixed RRSIGs from Different DM ZSKs . . . . . . . . .  21
  5.3.  DNSSEC KSK Rollover . . . . . . . . . . . . . . . . . . .  22
    5.3.1.  Failure-Case KSK Rollover . . . . . . . . . . . . . .  22
    5.3.2.  KSK Rollover vs. BIND9 Views  . . . . . . . . . . . .  22
    5.3.3.  Large Responses during KSK Rollover . . . . . . . . .  23
  5.4.  Capture of Large DNS Response . . . . . . . . . . . . . .  24
  5.5.  Automated Maintenance of the Hints File . . . . . . . . .  24
  5.6.  Root Label Compression in Knot DNS Server . . . . . . . .  25
6.  Conclusions . . . . . . . . . . . . . . . . . . . . . . . . .  26
7.  Security Considerations . . . . . . . . . . . . . . . . . . .  28
8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  28
9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  29
  9.1.  Normative References  . . . . . . . . . . . . . . . . . .  29
  9.2.  Informative References  . . . . . . . . . . . . . . . . .  29
Appendix A.  Yeti-Root Hints File . . . . . . . . . . . . . . . .  33
Appendix B.  Yeti-Root Server Priming Response  . . . . . . . . .  34
Appendix C.  Active IPv6 Prefixes in Yeti DNS Testbed . . . . . .  36
Appendix D.  Tools Developed for Yeti DNS Testbed . . . . . . . .  36
Appendix E.  Controversy  . . . . . . . . . . . . . . . . . . . .  37
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . .  38
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  39

1. Introduction 🌸

1. Okay, the vibe check 💅

The Domain Name System (DNS), as originally specified in [RFC1034] and [RFC1035], has proved to be an enduring and important platform that almost every end-user of the Internet relies on 💖. Even though it's been around forever, extensions, new implementations, and refinements to DNS operations keep popping up, both inside and outside the IETF ✨.

The Root Server system in particular has been on a major glow-up 🌈. Think wide-scale anycast deployment, mitigating unwanted traffic on a global scale, widespread deployment of Response Rate Limiting [RRL], the introduction of IPv6 transport, the deployment of DNSSEC, changes in DNSSEC key sizes, and preparations to roll the root zone's Key Signing Key (KSK) and corresponding trust anchor 🔥. These projects caused tremendous qualitative operational change and needed seriously impressive caution and study before implementation 🙈. They happened alongside the quantitative expansion of delegations for new TLDs (see https://newgtlds.icann.org/) 🦋.

Parts of the operational structure of the Root Server system have been described in documents like [TNO2009], [ISC-TN-2003-1], [RSSAC001], and [RFC7720] 📚. Taken together, those references give enough insight into how the whole system runs that it's easy to imagine structural changes to its infrastructure and wonder what the operational consequences would be 🤔✨.

2. The origin story 🧚

The Yeti DNS Project was conceived in May 2015 🎀. Its goal was a non-production testbed open to anyone in the technical community who wanted to propose or run experiments that answer questions like these 💜. Coordination was provided by BII, TISF, and the WIDE Project 🌷. So, to be clear, Yeti DNS is independently coordinated and is not affiliated with the IETF, ICANN, IANA, or any Root Server Operator 🙅‍♀️. The objectives of the Yeti Project were set by the participants, based on experiments they thought would provide valuable information 🍓.

Many volunteers collaborated to build a distributed testbed that, at the time of writing, includes 25 Yeti root servers run by 16 operators 🌈. It handles experimental traffic from individual volunteers, universities, DNS vendors, and distributed measurement networks 🫶.

3. How it stays in sync 🦋

By design, the Yeti testbed serves the root zone published by the IANA, with only those structural modifications needed to make it work usefully in the Yeti testbed instead of the production Root Server system 💅. Specifically, no delegation for any top-level zone is changed, added, or removed from the IANA-published root zone to build the root zone the Yeti testbed serves, and changes in the root zone show up in the testbed in near real-time ⚡. In this document, for clarity, the zone derived from the IANA-published root zone is called the Yeti-Root zone 💖.

The Yeti DNS testbed does a similar job to the Root Server system, since both serve similar zones: the Yeti-Root zone and the IANA-published root zone 🎀. The difference is that the Yeti DNS testbed only serves clients that are explicitly configured to take part in the experiment, while the Root Server system serves the whole Internet 🙈. Because the Yeti end-users and systems are known and their operations are well-coordinated with the Yeti project, it has been possible to deploy structural changes in the testbed with effective measurement and analysis 🔥. That's something that's hard, or simply impractical, in the production Root Server system 😭.

4. What this doc covers 🌷

This document describes the motivation behind the Yeti project, describes the Yeti testbed infrastructure, and shares the technical and operational experiences of some Yeti testbed users 🧚. It does not address the policies under which the Root Server system is operated, and it makes no proposal for changing any aspect of its implementation or operation 🫶.

2. Requirements Notation and Conventions 📝

Throughout the document, any mention of "Root" with an uppercase "R" and no other prefix refers to the "IANA Root" systems used in the production Internet 💜. Anything that's specifically about the Yeti infrastructure gets a "Yeti" prefix, like "Yeti-Root zone", "Yeti DNS", and so on 🌸.

3. Areas of Study 🔍

This section gives some examples of topics the developers of the Yeti DNS testbed considered important 🍓. As noted in Section 1, Yeti DNS is independently coordinated and is not affiliated with the IETF, ICANN, IANA, or any Root Server Operator 🙅‍♀️. So the topics and areas for study were picked by (and for) the Yeti project's proponents, to address their own concerns, and in the hope that the information and tools would be of wider interest 🦋.

Each example below comes with indicative questions 💖.

3.1. Implementation of a Testbed like the Root Server System 🏗️

  • How can a testbed be built and deployed on the Internet so that useful public participation happens without any risk of disrupting the Root Server system? 🛡️
  • How can representative traffic be introduced into such a testbed, so that the impact of specific differences between the testbed and the Root Server system can be observed? 👀

3.2. Yeti-Root Zone Distribution 🎀

  • What are the scaling properties of Yeti-Root zone distribution as the number of Yeti-Root servers, Yeti-Root server instances, or intermediate distribution points grows? 📈✨

Part 1 of 9. More parts coming, bestie, so stay tuned 🫶💖

3.3. Yeti-Root Server Names and Addressing ✨

3.3. Yeti-Root Server Names and Addressing 🎀

Okay bestie, let's talk naming! 💅 These are the open questions the testbed folks still want answered:

  • What naming schemes other than ones closely analogous to using ROOT-SERVERS.NET in the production root zone are practical, and what are their respective advantages and disadvantages? 🌸
  • What are the risks and benefits of signing the zone that contains the names of the Yeti-Root servers? 🔐
  • What automatic mechanisms might be useful to improve the rate at which clients of Yeti-Root servers are able to react to a Yeti-Root server renumbering event? 🦋

3.4. IPv6-Only Yeti-Root Servers 🌈

Next up, the IPv6-only servers, and the questions are giving "is this a problem or a vibe?" 🫶

  • Are there negative operational effects in the use of IPv6-only Yeti-Root servers, compared to the use of servers that are dual-stack? 🤔
  • What effect does the IPv6 fragmentation model have on the operation of Yeti-Root servers, compared with that of IPv4? 🧚

3.5. DNSSEC in the Yeti-Root Zone 🔥

Ohhh, DNSSEC. Iconic, complicated, and absolutely needs a lot of testing. 👑

  • Is it practical to sign the Yeti-Root zone using multiple, independently operated DNSSEC signers and multiple corresponding Zone Signing Keys (ZSKs)? 💜
  • To what extent is [RFC5011] ("Automated Updates of DNS Security (DNSSEC) Trust Anchors") supported by resolvers? 🍓
  • Does the KSK Rollover plan designed and in the process of being implemented by ICANN work as expected on the Yeti testbed? 🌸
  • What is the operational impact of using much larger RSA key sizes in the ZSKs used in a root? 💀
  • What are the operational consequences of choosing DNSSEC algorithms other than RSA to sign a root? ✨

4. Yeti DNS Testbed Infrastructure 🧚

Okay so the whole point of the testbed is this: DNS queries from stub resolvers, going through recursive resolvers, get delivered to Yeti-Root servers, and the responses the Yeti-Root servers generate get sent back the same way. Main character energy! 🎀 Check out Figure 1 below. 💖

    ,----------.        ,-----------.        ,------------.
    |   stub   +------> | recursive +------> | Yeti-Root  |
    | resolver | <------+ resolver  | <------+ nameserver |
    `----------'        `-----------'        `------------'
       ^                   ^                    ^
       |  appropriate      |  Yeti-Root hints;  |  Yeti-Root zone
       `- resolver         `- Yeti-Root trust   `- with DNSKEY RRset
          configured          anchor               signed by
                                                   Yeti-Root KSK

               Figure 1: High-Level Testbed Components

To use the Yeti DNS testbed, a recursive resolver MUST be configured to use the Yeti-Root servers. 🫶 That configuration is a list of names and addresses for the Yeti-Root servers (people often call this a "hints file") that replaces the matching hints used for the production Root Server system (Appendix A). If resolvers are configured to validate DNSSEC, then they also need a DNSSEC trust anchor that matches a KSK used in the Yeti DNS Project, instead of the normal trust anchor set used for the Root Zone. 🔐

Since the Yeti root(s) are signed with Yeti keys, not the ones used by the IANA Root, the resolver trust anchors need to change to match. The Yeti-Root hints file also needs corresponding changes (Appendix A). Any validator using the production Root Server system's root zone trust anchor set would properly reject those changes as bogus. 🙈

Stub resolvers join the Yeti DNS testbed just by using recursive resolvers that are configured as described above. ✨

The data flow from IANA to stub resolvers through the Yeti testbed is shown in Figure 2, and the sections that follow go into more detail. 🦋

                          ,----------------.
                     ,-- / IANA Root Zone / ---.
                     |  `----------------'     |
                     |            |            |
                     |            |            |       Root Zone
 ,--------------.    ,---V---.    ,---V---.    ,---V---.
 | Yeti Traffic |    | BII   |    | WIDE  |    | TISF  |
 | Collection   |    |  DM   |    |  DM   |    |  DM   |
 `----+----+----'    `---+---'    `---+---'    `---+---'
      |    |       ,-----'    ,-------'            `----.
      |    |       |          |                         |  Yeti-Root
      ^    ^       |          |                         |     Zone
      |    |   ,---V---.  ,---V---.                 ,---V---.
      |    `---+ Yeti  |  | Yeti  |  . . . . . . .  | Yeti  |
      |        | Root  |  | Root  |                 | Root  |
      |        `---+---'  `---+---'                 `---+---'
      |            |          |                         |    DNS
      |            |          |                         |  Response
      |         ,--V----------V-------------------------V--.
      `---------+              Yeti Resolvers              |
                `--------------------+---------------------'
                                     |                       DNS
                                     |                     Response
                ,--------------------V---------------------.
                |            Yeti Stub Resolvers           |
                `------------------------------------------'

 The three coordinators of the Yeti DNS testbed:
    BII : Beijing Internet Institute
    WIDE: Widely Integrated Distributed Environment Project
    TISF: A collaborative engineering and security project by Paul Vixie

                        Figure 2: Testbed Data Flow

Heads up: the roots are NOT bound to Distribution Masters (DMs). 😭 DMs update their zone on a schedule described in Section 4.1. Each DM that updates the latest zone can notify all roots, so the zone transfer can happen between any DM and any root. 🌈

4.1. Root Zone Retrieval 🍓

The Yeti-Root zone gets distributed inside the Yeti DNS testbed through a set of internal master servers called Distribution Masters (DMs). 💅 These server elements send the Yeti-Root zone out to all Yeti-Root servers. How the Yeti DMs build the Yeti-Root zone for distribution is explained below. 🧚

Since Yeti DNS DMs don't receive DNS NOTIFY [RFC1996] messages from the Root Server system, they use a polling approach to figure out when new revisions of the root zone are available from the production Root Server system. Each Yeti DM MUST request the Root Zone SOA record from a Root server that permits unauthenticated zone transfers of the root zone, and it performs a zone transfer from that server if the retrieved value of SOA.SERIAL is greater than that of the last retrieved zone. 🔥

At the time of writing, unauthenticated zone transfers of the Root Zone are available directly from B-Root, C-Root, F-Root, G-Root, K-Root, and L-Root; two servers XFR.CJR.DNS.ICANN.ORG and XFR.LAX.DNS.ICANN.ORG; and via FTP from sites maintained by the Root Zone Maintainer and the IANA Functions Operator. The Yeti DNS testbed retrieves the Root Zone using zone transfers from F-Root. The schedule on which F-Root is polled by each Yeti DM is as follows: 🦋

              +-------------+-----------------------+
              | DM Operator | Time                  |
              +-------------+-----------------------+
              | BII         | UTC hour + 00 minutes |
              | WIDE        | UTC hour + 20 minutes |
              | TISF        | UTC hour + 40 minutes |
              +-------------+-----------------------+

The Yeti DNS testbed uses multiple DMs, and each one acts autonomously and equivalently to its siblings. 👑 Any single DM can distribute new revisions of the Yeti-Root zone, and it's also responsible for signing the RRsets that change during the transformation of the Root Zone into the Yeti-Root zone described in Section 4.2. This multiple DM model is meant to provide a basic structure for the idea of shared zone control proposed in [ITI2014]. 🌸

4.2. Transformation of Root Zone to Yeti-Root Zone 💖

Two distinct approaches have been deployed in the Yeti DNS testbed, separately, to transform the Root Zone into the Yeti-Root zone. 🎀 At a high level they're equivalent: both replace a minimal set of information in the root zone with matching data for the Yeti DNS testbed. The mechanisms that execute the transforms are different, though. The approaches are discussed in Sections 4.2.1 and 4.2.2. ✨

A third approach has also been proposed, but it hasn't been implemented yet. The motivations and changes that approach would imply are described in Section 4.2.3. 🔮

4.2.1. ZSK and KSK Key Sets Shared between DMs 🔑

This approach was the first one implemented. 🥇 It features entirely autonomous operation of each DM, but it also requires the secret key material (the private key in each of the Yeti-Root KSK and ZSK key pairs) to be distributed and maintained on each DM in a coordinated way. 🫶

The Root Zone is transformed as follows to produce the Yeti-Root zone. This transformation is carried out autonomously on each Yeti DNS Project DM. Each DM carries an authentic copy of the current set of Yeti KSK and ZSK key pairs, synchronized between all DMs (see Section 4.4). 💜

  1. SOA.MNAME is set to www.yeti-dns.org.
  2. SOA.RNAME is set to <dm-operator>.yeti-dns.org, where <dm-operator> is currently one of "wide", "bii", or "tisf".
  3. All DNSKEY, RRSIG, and NSEC records are removed.
  4. The apex Name Server (NS) RRset is removed, with the corresponding root server glue (A and AAAA) RRsets.
  5. A Yeti DNSKEY RRset is added to the apex, comprising the public parts of all Yeti KSK and ZSKs.
  6. A Yeti NS RRset is added to the apex that includes all Yeti-Root servers.
  7. Glue records (AAAA only, since Yeti-Root servers are v6-only) for all Yeti-Root servers are added.
  8. The Yeti-Root zone is signed: the NSEC chain is regenerated; the Yeti KSK is used to sign the DNSKEY RRset; and the shared ZSK is used to sign every other RRset.

Note that the SOA.SERIAL value published in the Yeti-Root zone is identical to that found in the root zone. 🌟

4.2.2. Unique ZSK per DM; No Shared KSK ✨

Okay so this approach was the second one to get implemented and maintained as stable state 💖. Each DM gets its own dedicated ZSK key pairs, and they are NOT shared with other DMs 🔐. A Yeti-Root DNSKEY RRset gets constructed and signed upstream of all the DMs. It's basically the union of the set of active Yeti-Root KSKs and the set of active ZSKs for every individual DM 🧩. Now each DM only needs the secret part of its own dedicated ZSK key pairs available locally, and no other secret key material is shared. Literally so clean 😭 Figure 3 shows the high-level vibe:

                        ,----------.         ,-----------.
                   .--------> BII ZSK  +---------> Yeti-Root |
                   | signs  `----------'  signs  `-----------'
                   |
     ,-----------. |        ,----------.         ,-----------.
     | Yeti KSK  +-+--------> TISF ZSK +---------> Yeti-Root |
     `-----------' | signs  `----------'  signs  `-----------'
                   |
                   |        ,----------.         ,-----------.
                   `--------> WIDE ZSK +---------> Yeti-Root |
                     signs  `----------'  signs  `-----------'

                        Figure 3: Unique ZSK per DM

The whole process of retrieving the Root Zone from the Root Server system and replacing and signing the apex DNSKEY RRset no longer happens on the DMs 🙅‍♀️. Instead, it happens on a central Hidden Master 🏰. Producing these signed DNSKEY RRsets is kinda like the Signed Key Responses (SKRs) made during ICANN KSK key ceremonies [ICANN2010] 🎀.

Each DM now pulls its source data from the Yeti DNS Hidden Master instead of the Root Server system 📦. That data has a premodified, Yeti-signed DNSKEY RRset, but everything else is unchanged ✨.

Each DM does a similar transformation to the one in Section 4.2.1, except DMs no longer need to modify or sign the DNSKEY RRset 🙌. The DM's unique local ZSK signs every other RRset instead 💅.

4.2.3. Preserving Root Zone NSEC Chain and ZSK RRSIGs ✨

Okay, now for a plot twist 👀. A change to the transformation from Section 4.2.2 has been proposed as a Yeti experiment called PINZ [PINZ] 🦋. The idea is to preserve the NSEC chain from the Root Zone and all RRSIG RRs generated using the Root Zone's ZSKs 🧚. The DNSKEY RRset would still get modified to replace the Root Zone KSKs, but the Root Zone ZSKs would be kept intact. The Yeti KSK would be used to generate replacement signatures over the apex DNSKEY and NS RRsets 🔑. Source data would keep flowing from the Root Server system through the Hidden Master to the set of DMs. No DNSSEC operations would be required on the DMs, and the source NSEC and most RRSIGs would stay intact 🌸.

The reason this is so iconic is that it keeps changes minimal from the IANA Root zone 🍓. It also gives cryptographically verifiable confidence that no owner name in the root zone was changed while producing the Yeti-Root zone from the Root Zone 🔍. That addresses one of the concerns described in Appendix E, and it can be verified automatically 🤖. Main character energy 👑.

4.3. Yeti-Root Zone Distribution ✨

Each Yeti DM is configured with a full list of Yeti-Root server addresses to send NOTIFY [RFC1996] messages to 📬. This also forms the basis for an address-based access-control list for zone transfers. Authentication by address could be replaced with more rigorous mechanisms, like Transaction Signatures (TSIGs) [RFC2845] 🔐. This hasn't been done at the time of writing 🕰️. Address-based controls avoid the need to distribute shared secrets among the Yeti-Root server operators, so it's lowkey the smarter move right now 🫶.

Individual Yeti-Root servers are configured with a full set of Yeti DM addresses. SOA and AXFR queries may be sent to those addresses in the conventional manner 💌.

4.4. Synchronization of Service Metadata ✨

Changes in the Yeti DNS testbed infrastructure, like adding or removing Yeti-Root servers, renumbering Yeti-Root servers, or DNSSEC key rollovers, require coordinated changes on all DMs 🔄. The Yeti DNS testbed changes way more often than the Root Server system does, and it includes substantially more Yeti-Root servers than there are IANA Root Servers 📈. So a manual change process would be more likely to suffer from human error 😬. That's why an automated and cooperative process was implemented 🤖💜.

The theory is that each DM operator runs a Git repository locally, containing all the service metadata involved in operating each DM 🗂️. When a change is desired and approved among all Yeti coordinators, one DM operator (usually BII) updates the local Git repository ✍️. A serial number in the future (in two days) is chosen for when the changes become active ⏳. The DM operator then pushes the changes to the Git repositories of the other two DM operators, who get a chance to check and edit the changes 👀. When the serial number of the root zone passes the number chosen, the changes are pulled automatically to individual DMs and promoted to production 🚀.

The three Git repositories are synchronized by configuring them as remote servers. For example, at BII we push to all three DMs' repositories like this:

             $ git remote -v
             origin yeticonf@yeti-conf.dns-lab.net:dm (fetch)
             origin yeticonf@yeti-conf.dns-lab.net:dm (push)
             origin yeticonf@yeti-dns.tisf.net:dm (push)
             origin yeticonf@yeti-repository.wide.ad.jp:dm (push)

For more detailed info on DM synchronization, check out this document in Yeti's GitHub repository: https://github.com/BII-Lab/Yeti-Project/blob/master/doc/Yeti-DM-Sync.md 📖🌈.

4.5. Yeti-Root Server Naming Scheme ✨

The current naming scheme for Root Servers was normalized to use single-character host names ("A" through "M") under the domain ROOT-SERVERS.NET, as described in [RSSAC023] 🔤. The principal benefit was that DNS label compression could be used to produce a priming response that would fit within 512 bytes when it was introduced 📏. And 512 bytes is the maximum DNS message size using UDP transport without EDNS(0) [RFC6891], so that was a big deal back then 💀.

Yeti-Root servers don't use this optimization. They use free-form nameserver names chosen by their respective operators 🎨. In other words, no attempt is made to minimize the size of the priming response through label compression. This approach aims to challenge the need to minimize the priming response in a modern DNS ecosystem where EDNS(0) is prevalent 💅.

Priming responses from Yeti-Root servers (unlike those from Root Servers) don't always include server addresses in the additional section 🙈. In particular, Yeti-Root servers running BIND9 return an empty additional section if the configuration parameter "minimum-responses" is set. That forces resolvers to finish the priming process with a set of conventional recursive lookups to resolve addresses for each Yeti-Root server 🔁. Yeti-Root servers running NSD were observed to return a fully populated additional section, depending, of course, on the EDNS buffer size in use 🧁.

Various approaches to normalize the composition of the priming response were considered, including:

  • Require use of DNS implementations that exhibit a desired behavior in the priming response.
  • Modify nameserver software or configuration as used by Yeti-Root servers.
  • Isolate the names of Yeti-Root servers in one or more zones that could be slaved on each Yeti-Root server, renaming servers as necessary. That gives each server a source of authoritative data, so the authority section of a priming response could be fully populated. This is the approach used in the Root Server system with the ROOT-SERVERS.NET zone 🧠.

The potential mitigation of renaming all Yeti-Root servers using a scheme that would allow their names to exist directly in the root zone was not considered 🚫. That approach implies inventing new top-level labels not present in the Root Zone 🙅‍♀️.

Given the relative infrequency of priming queries by individual resolvers, and the additional complexity or other compromises implied by each of those mitigations, the decision was made to make no effort to ensure that priming responses were identical across servers 🤷‍♀️. Even the empty additional sections generated by Yeti-Root servers running BIND9 seem to be sufficient for all resolver software tested. Resolvers simply perform a new recursive lookup for each authoritative server name they need to resolve 🦋✨.

4.6. Yeti-Root Servers 🌸

Okay so volunteers have donated authoritative servers to act as Yeti-Root servers, and it's giving community energy 🫶. At the time of writing there are 25 Yeti-Root servers spread out globally. One of them is named using a label from IDNA2008 [RFC5890], and it's shown in punycode in the list below ✨

+-------------------------------------+---------------+-------------+
| Name                                | Operator      | Location    |
+-------------------------------------+---------------+-------------+
| bii.dns-lab.net                     | BII           | CHINA       |
| yeti-ns.tsif.net                    | TSIF          | USA         |
| yeti-ns.wide.ad.jp                  | WIDE Project  | Japan       |
| yeti-ns.as59715.net                 | as59715       | Italy       |
| dahu1.yeti.eu.org                   | Dahu Group    | France      |
| ns-yeti.bondis.org                  | Bond Internet | Spain       |
|                                     | Systems       |             |
| yeti-ns.ix.ru                       | Russia        | MSK-IX      |
| yeti.bofh.priv.at                   | CERT Austria  | Austria     |
| yeti.ipv6.ernet.in                  | ERNET India   | India       |
| yeti-dns01.dnsworkshop.org          | dnsworkshop   | Germany     |
|                                     | /informnis    |             |
| dahu2.yeti.eu.org                   | Dahu Group    | France      |
| yeti.aquaray.com                    | Aqua Ray SAS  | France      |
| yeti-ns.switch.ch                   | SWITCH        | Switzerland |
| yeti-ns.lab.nic.cl                  | NIC Chile     | Chile       |
| yeti-ns1.dns-lab.net                | BII           | China       |
| yeti-ns2.dns-lab.net                | BII           | China       |
| yeti-ns3.dns-lab.net                | BII           | China       |
| ca...a23dc.yeti-dns.net             | Yeti-ZA       | South       |
|                                     |               | Africa      |
| 3f...374cd.yeti-dns.net             | Yeti-AU       | Australia   |
| yeti1.ipv6.ernet.in                 | ERNET India   | India       |
| xn--r2bi1c.xn--h2bv6c0a.xn--h2brj9c | ERNET India   | India       |
| yeti-dns02.dnsworkshop.org          | dnsworkshop   | USA         |
|                                     | /informnis    |             |
| yeti.mind-dns.nl                    | Monshouwer    | Netherlands |
|                                     | Internet      |             |
|                                     | Diensten      |             |
| yeti-ns.datev.net                   | DATEV         | Germany     |
| yeti.jhcloos.net.                   | jhcloos       | USA         |
+-------------------------------------+---------------+-------------+

A participating resolver first picks up the current list of Yeti-Root servers from a substitute hints file (see Appendix A), and after that it gets them through the usual resolver priming process [RFC8109] 💅

All Yeti-Root servers are IPv6-only 🦋 That's because of the IPv6-only Internet of the foreseeable future. So the Yeti-Root hints file has no IPv4 addresses, and the Yeti-Root zone has no IPv4 glue records. Note that the whole point of an IPv6-only testbed is to find out whether an IPv6-only root can survive any problem or impact when IPv4 is turned off. It's very much in the spirit of the IETF SUNSET4 WG [SUNSET4] 🧚

At the time of writing, every root server in the Root Server system serves the ROOT-SERVERS.NET zone along with the root zone, and all but one of them also serve the ARPA zone. Yeti-Root servers serve the Yeti-Root zone only ✨

The volunteer operators reported how much software diversity there is across the Yeti-Root servers at the time of writing, and it's honestly so iconic 👑

  • Platform: 18 of 25 Yeti-Root servers are implemented on a Virtual Private Server (VPS) rather than bare metal.
  • Operating System: 15 Yeti-Root servers run on Linux (Ubuntu, Debian, CentOS, Red Hat, and ArchLinux); 4 run on FreeBSD; 1 on NetBSD; and 1 on Windows Server 2016.
  • DNS software: 16 of 25 Yeti-Root servers use BIND9 (versions varying between 9.9.7 and 9.10.3); 4 use NSD (4.10 and 4.15); 2 use Knot (2.0.1 and 2.1.0); 1 uses Bundy (1.2.0); 1 uses PowerDNS (4.1.3); and 1 uses MS DNS (10.0.14300.1000).

4.7. Experimental Traffic 🎀

For the Yeti DNS testbed to be useful as an experimentation platform, it needs to carry statistically representative traffic 📊 Several approaches have been tried to load the system, including real-world traffic triggered by end-users and synthetic traffic.

Resolvers that have been explicitly configured to take part in the testbed, as described in Section 4, are a source of real-world, end-user traffic. Thanks to an efficient cache mechanism, the mean query rate is less than 100 qps in the Yeti testbed. Still, a variety of sources were observed as active during 2017, as summarized in Appendix C 🍓

Synthetic traffic has been added to the system from time to time to boost the traffic loads 💜 Approaches include using distributed measurement platforms such as RIPE ATLAS to send DNS queries to Yeti-Root servers. They also include capturing traffic that was sent from non-Yeti resolvers to the Root Server system, then modifying it and replaying it towards Yeti-Root servers 🌈

4.8. Traffic Capture and Analysis 🔍

Traffic capture of queries and responses is available in the testbed, on both Yeti resolvers and Yeti-Root servers. It's there in anticipation of experiments that need packet-level visibility into DNS traffic 👀

Traffic capture is performed on Yeti-Root servers using either:

PCAP-format files containing packet captures are uploaded using rsync to central storage 📦

5. Operational Experience with the Yeti DNS Testbed ✨

The following sections give commentary on the operation and impact analyses of the Yeti DNS testbed described in Section 4 💖 More detailed descriptions of observed phenomena are available in the Yeti DNS mailing list archives at http://lists.yeti-dns.org/pipermail/discuss/ and on the Yeti DNS blog at https://yeti-dns.org/blog.html 🫶

5.1. Viability of IPv6-Only Operation 🌈

All Yeti-Root servers were deployed with IPv6 connectivity. No IPv4 addresses for any Yeti-Root server were made available, for example in the Yeti hints file or in the DNS itself. This implementation decision constrained the Yeti-Root system to be v6 only 🦋

DNS implementations are generally great at using both IPv4 and IPv6 when both are available. Servers that can't be reliably reached over one protocol might be better queried over the other, which helps end-users in the common case where DNS resolution is on the critical path for how fast things feel to them. However, this optimization also means that systemic problems with one protocol can be masked by the other. By forcing all traffic over IPv6, the Yeti DNS testbed aimed to expose any such problems and make them easier to identify and understand 🔍 Several examples of IPv6-specific phenomena observed during the operation of the testbed are described in the sections that follow.

Although the Yeti-Root servers themselves were only reachable using IPv6, real-world end-users often have no IPv6 connectivity 😭 The testbed was also able to explore how well IPv6-only Yeti-Root servers could serve single-stack, IPv4-only end-user populations through the use of dual-stack Yeti resolvers 💅

5.1.1. IPv6 Fragmentation 🧩

In the Root Server system, structural changes that could increase response sizes (and so cause fragmentation, fallback to TCP transport, or both) have been handled with great care, since their impact on clients is hard to predict or measure. The Yeti DNS testbed is experimental and has the luxury of a known client base, which makes it far easier to make such changes and measure their impact 🧪

Many of the experimental design choices described in this document were expected to trigger larger responses. For example, the choice of naming scheme for Yeti-Root servers described in Section 4.5 defeats label compression. It makes a large priming response, up to 1754 octets with 25 NS records and their corresponding glue records. The Yeti-Root zone transformation approach described in Section 4.2.2 greatly enlarges the apex DNSKEY RRset, especially during the KSK rollover, reaching up to 1975 octets with 3 ZSKs and 2 KSKs. So an increased incidence of fragmentation was expected 😬

The Yeti DNS testbed provides service on IPv6 only. However, middleboxes (such as firewalls and some routers) are not friendly to IPv6 fragments. There are reports of a notable packet drop rate caused by middleboxes mistreating IPv6 fragments [FRAGDROP] [RFC7872]. One APNIC study [IPv6-frag-DNS] reported that 37% of endpoints using IPv6-capable DNS resolvers cannot receive a fragmented IPv6 response over UDP 🙈

To study the impact, RIPE Atlas probes were used. For each Yeti-Root server, an Atlas measurement was set up using 100 IPv6-enabled probes from five regions. Each one sent a DNS query for "./IN/DNSKEY" using UDP transport with DO=1. This measurement was carried out concurrently with a Yeti KSK rollover, which further exacerbated the potential for fragmentation. It identified a 7% failure rate compared with a non-fragmented control. A failure rate of 2% was observed with response sizes of 1414 octets, which was surprising given the expected prevalence of 1500-octet (Ethernet-framed) MTUs 🤯

The consequences of fragmentation were not limited to failures in delivering DNS responses over UDP transport. There were two cases where a Yeti-Root server failed when using TCP to transfer the Yeti-Root zone from a DM. DM log files revealed "socket is not connected" errors corresponding to the zone transfer requests. Further experimentation revealed that combinations of NetBSD 6.1, NetBSD 7.0RC1, FreeBSD 10.0, Debian 3.2, and VMWare ESXI 5.5 resulted in a high TCP Maximum Segment Size (MSS) value of 1440 octets being negotiated between client and server, despite the presence of the IPV6_USE_MIN_MTU socket option, as described in [USE_MIN_MTU]. The mismatch appears to cause outbound segments larger than 1280 octets to be dropped before sending. Setting the local TCP MSS to 1220 octets was observed to be a pragmatic mitigation. That number was chosen as 1280 - 60, the size of the IPv6 TCP header with no other extension headers 🩹

5.1.2. Serving IPv4-Only End-Users 🌸

okay so real-world end-users were actually using Yeti resolvers 💅 for general name resolution at a bunch of participant orgs. that included resolving names to IPv4 addresses, and even IPv4-only end-user devices were doing lookups through them. literally iconic 🦋

some participants were like "reliability in resolver infrastructure is SO important" 🫶 and they were also kinda worried about how stable their IPv6 connectivity was. so they deployed Yeti resolvers in parallel with their conventional ones and made both available to end-users. ngl this approach is a useful data point ✨, but having end-users use Yeti resolvers exclusively was a much better way to spot and understand any failures in the Yeti DNS testbed infrastructure 🔥

resolvers in IPv4-only environments were still able to join the Yeti DNS testbed 🧚 through upstream, dual-stack Yeti resolvers. in one case (CERNET2), they did this by giving IPv4 addresses to Yeti-Root servers and mapping them in dual-stack IVI translation devices [RFC6219]. the way that worked is giving me main character energy 👑

5.2. Zone Distribution 🎀

the Yeti DNS testbed uses multiple DMs to distribute the Yeti-Root zone 💖. this approach lets the number of Yeti-Root servers scale higher than a single distribution source could support, and it also gives redundancy 🫶. multiple DMs did come with some operational challenges though 💀, and those are described in the following sections.

5.2.1. Zone Transfers 🦋

Yeti-Root servers were set up to serve the Yeti-Root zone as slaves. each slave had all DMs configured as masters, which gives redundancy in zone synchronization 🌈

each DM in the Yeti testbed served a Yeti-Root zone that was functionally equivalent to the others but not congruent 🙈 (see Section 4.3). the differences included variations in the SOA.MNAME field and, more critically, in the RRSIGs for everything other than the apex DNSKEY RRset. that's because signatures for all other RRsets are generated using a private key that's only available to the DM serving its particular variant of the zone (see Sections 4.2.1 and 4.2.2) 😭

Incremental Zone Transfer (IXFR), as described in [RFC1995], is a viable mechanism for zone synchronization between any Yeti-Root server and a consistent, single DM ✨. but if that Yeti-Root server ever selected a different DM, IXFR would no longer be a safe mechanism. structural changes between the incongruent zones on different DMs wouldn't be included in any transferred delta, and the result would be a zone that was not internally self-consistent 💀. for this reason, the first transfer after a change of DM would require AXFR not IXFR 🫶

none of the DNS software in use on Yeti-Root servers supports this mixture of IXFR/AXFR depending on the master server in use 😭. honestly that's not surprising, because the Yeti-Root system is idiosyncratic 🙃. conventional zone transfer graphs involve zones that are congruent between all nodes. so for this reason, all Yeti-Root servers are configured to use AXFR at all times, and never IXFR, to make sure the zones being served are internally self-consistent 💅

5.2.2. Delays in Yeti-Root Zone Distribution 🌸

each Yeti DM polled the Root Server system for a new revision of the root zone on an interleaved schedule, as described in Section 4.1 ⏰. so different DMs were expected to retrieve each revision of the root zone, and make a matching revision of the Yeti-Root zone available, at different times. the availability of a new revision of the Yeti-Root zone on the first DM would typically come 40 minutes before the last one 🕐

given that distribution mechanism, you might expect the maximum latency between publication of a new root zone revision and the corresponding Yeti-Root zone being available on any Yeti-Root server to be 20 minutes. since in normal operation at least one DM should serve that Yeti-Zone within 20 minutes of root zone publication, right? 🤔 well, in practice this was not observed 💀

in one case, a Yeti-Root server running Bundy 1.2.0 on FreeBSD 10.2-RELEASE was found to lag root zone publication by as much as ten hours 😭. upon investigation, it turned out to be due to software defects, which were corrected later 🛠️

more generally, Yeti-Root servers were routinely observed lagging root zone publication by more than 20 minutes, and relatively often by more than 40 minutes 🙈. whilst in some cases this might be assumed to be a connectivity problem, like perhaps suppressing the delivery of NOTIFY messages, it was also observed that Yeti-Root servers receiving a NOTIFY from one DM would often send SOA queries and AXFR requests to a different DM. if that DM wasn't yet serving the new revision of the Yeti-Root zone, a delay in updating the Yeti-Root server would naturally follow 🫠

5.2.3. Mixed RRSIGs from Different DM ZSKs 🧚

the second approach for doing the transformation of Root Zone to Yeti-Root zone (Section 4.2.2) introduces a situation where mixed RRSIGs from different DM ZSKs get cached in one resolver 🌈

here's the thing 👀: the Yeti-Root zone served by any particular Yeti-Root server will include signatures generated using the ZSK from the DM that served the Yeti-Root zone to that Yeti-Root server. signatures cached at resolvers might be retrieved from any Yeti-Root server, so they're expected to be a mix of signatures generated by different ZSKs. but since all ZSKs can be trusted through the signature by the Yeti KSK over the DNSKEY RRset (which includes all ZSKs), the mixture of signatures was predicted not to be a threat to reliable validation 💖

it was first tested in BII's lab environment as a proof of concept 🧪. in the resolver's DNSSEC log, verifying an RDATA set showed "success" with a key (keyid) in the DNSKEY RRset 😌. it was implemented later in three DMs that were carefully coordinated and made public to all Yeti resolver operators and participants on Yeti's mailing list 📬. at least 45 Yeti resolvers (deployed by Yeti operators) were being monitored and had set a reporting trigger in case anything was wrong. the Yeti mailing list is also open for error reports from other participants 🙌. so far, the Yeti testbed has been run in this configuration (with multiple ZSKs) for 2 years, and it has proven workable and reliable, even when rollovers of individual ZSKs are on different schedules 🏆

another consequence of this approach is that the apex DNSKEY RRset in the Yeti-Root zone is much larger than the corresponding DNSKEY RRset in the Root Zone. that means it takes more space and produces a larger response to the query for the DNSKEY RRset, especially during the KSK rollover 😅

5.3. DNSSEC KSK Rollover 🔐

at the time of writing, the Root Zone KSK is expected to undergo a carefully orchestrated rollover as described in [ICANN2016]. ICANN has commissioned various tests and has published an external test plan [ICANN2017] 📋

three related DNSSEC KSK rollover exercises were carried out on the Yeti DNS testbed, somewhat concurrent with the planning and execution of the rollover in the root zone 🎢. brief descriptions of these exercises are included below 👇

5.3.1. Failure-Case KSK Rollover 💥

the first KSK rollover executed on the Yeti DNS testbed deliberately ignored the 30-day hold-down timer specified in [RFC5011] before retiring the outgoing KSK 🙈

it was confirmed that clients of some (but not all) validating Yeti resolvers experienced resolution failures (received SERVFAIL responses) after this change 😭. those resolvers needed administrator intervention to install a functional trust anchor before resolution was restored 🚨

5.3.2. KSK Rollover vs. BIND9 Views 🧩

the second Yeti KSK rollover was designed with similar phases to ICANN's KSK rollover, although with modified timings to cut down the time needed to complete the process ⏱️. the "slot" used in this rollover was ten days long, like this:

          +-----------------+----------------+----------+
          |                 | Old Key: 19444 | New Key  |
          +-----------------+----------------+----------+
          | slot 1          | pub+sign       |          |
          | slot 2, 3, 4, 5 | pub+sign       | pub      |
          | slot 6, 7       | pub            | pub+sign |
          | slot 8          | revoke         | pub+sign |
          | slot 9          |                | pub+sign |
          +-----------------+----------------+----------+

during this rollover exercise, a problem was observed on one Yeti resolver running BIND 9.10.4-p2 [KROLL-ISSUE] 🐛. that resolver was configured with multiple views serving clients in different subnets when the KSK rollover began. DNSSEC validation failures showed up after the KSK rollover completed, and they were triggered by adding a new view that was meant to serve clients from a new subnet 😩

BIND 9.10 requires "managed-keys" configuration to be specified in every view 📝. this detail wasn't apparently obvious to the operator in this case, and the Internet Systems Consortium (ISC) later highlighted it in their general advice about KSK rollover in the root zone for BIND 9 users [ISC-BIND] 📣. when the "managed-keys" configuration is present in every view that is configured to perform validation, trust anchors for all views get updated during a KSK rollover 🎉🫶

5.3.3. Large Responses during KSK Rollover ✨

Okay so a KSK rollover necessarily means publishing the outgoing and incoming public keys at the same time, which means DNSKEY responses are expected to get bigger 📈💅. The third KSK rollover on the Yeti DNS testbed came with a whole coordinated effort to watch response sizes and see how they affected end-users 👀.

Remember Section 4.2.2? In the Yeti DNS testbed, each DM can keep control of its own set of ZSKs, and those can roll over independently 🦋. So if all three DMs rolled their ZSKs at once during a KSK rollover, the maximum number of apex DNSKEY RRs present would be eight (incoming and outgoing KSK, plus incoming and outgoing of each of three ZSKs) 😭. Lowkey wild. BUT in practice that concurrency never happened. Only the BII ZSK got rolled during the KSK rollover, so only three DNSKEY RRset configurations were observed 🎀:

  • 3 ZSKs and 2 KSKs, DNSKEY response of 1975 octets;
  • 3 ZSKs and 1 KSK, DNSKEY response of 1414 octets; and
  • 2 ZSKs and 1 KSK, DNSKEY response of 1139 octets.

RIPE Atlas probes were used as described in Section 5.1.1 to send DNSKEY queries directly to Yeti-Root servers 🛰️. The numbers of queries and failures were recorded and sorted by response size at the time each query was sent. Here's the tea from [YetiLR] ☕:

+---------------+----------+---------------+--------------+
| Response Size | Failures | Total Queries | Failure Rate |
+---------------+----------+---------------+--------------+
| 1139          | 274      | 64252         | 0.0042       |
| 1414          | 3141     | 126951        | 0.0247       |
| 1975          | 2920     | 42529         | 0.0687       |
+---------------+----------+---------------+--------------+

The general approach shown here is a great example of how the Yeti DNS testbed is designed 🧚. It's separate from the Root Server system but built as a live testbed on the Internet, which makes it easy to use general-purpose active measurement tools (like RIPE Atlas probes) as well as internal passive measurement (like packet capture) 💜.

5.4. Capture of Large DNS Response

Packet capture is a common approach in production DNS systems, where operators need fine-grained insight into traffic so they can understand what production traffic is actually doing 🔍. For authoritative servers, capturing inbound query traffic is often enough, since responses can be synthesized using knowledge of the zones being served when the query came in. Queries are generally small enough not to get fragmented, and even over TCP they usually fit in a single segment 🌸.

The Yeti DNS testbed has different requirements 🙈. In particular, there's a desire to compare responses from the Yeti infrastructure with those received from the Root Server system for a single query stream (e.g., using the "Yeti Many Mirror Verifier" (YmmV) described in Appendix D). Some Yeti-Root servers could recover complete DNS messages from inside nameservers, e.g., using dnstap. However, not all servers offered that, and a consistent approach was desirable 💅.

The need to passively capture responses from the wire, together with experiments that were expected (and in some cases designed) to trigger fragmentation and TCP transport, led to a brand-new tool called PcapParser 🛠️. It performs fragment and TCP stream reassembly from raw packet capture data. A brief description of PcapParser is included in Appendix D ✨.

5.5. Automated Maintenance of the Hints File

Renumbering events in the Root Server system are relatively rare 🌙. Each one comes with an updated hints file published in standard locations, but updating the local copies that DNS resolvers use is a manual job, and the process has an observably long tail 😩. For example, in 2015 J-Root was still receiving traffic at its old address some thirteen years after renumbering [Wessels2015].

The observed impact of these old, deployed hints files is minimal, likely because such renumbering events are very infrequent 🤷‍♀️. Even the oldest hints files still contain some accurate root server addresses from which priming responses can be obtained.

Yeti-Root servers are a different story 🧪. Because the system is experimental and mainly operated by volunteers, Yeti-Root servers are added, removed, and renumbered much more often. So a tool to automatically maintain hints files was created: [hintUpdate] 🫶.

The automated procedure followed by the hintUpdate tool goes like this 📋:

  1. Use the local resolver to obtain a response to the query ./IN/NS.
  2. Use the local resolver to obtain a set of IPv4 and IPv6 addresses for each name server.
  3. Validate all signatures obtained from the local resolvers and confirm that all data is signed.
  4. Compare the data obtained to that contained within the currently active hints file; if there are differences, rotate the old one away and replace it with a new one.

This tool would not work unmodified in the Root Server system, since the names of individual Root Servers (e.g., A.ROOT-SERVERS.NET) are not DNSSEC signed 😬. All Yeti-Root server names are DNSSEC signed, though, so this tool works as expected in that environment 🎉.

5.6. Root Label Compression in Knot DNS Server

[RFC1035] specifies that domain names can be compressed when encoded in DNS messages, and can be represented as one of 🧩:

  1. a sequence of labels ending in a zero octet;
  2. a pointer; or
  3. a sequence of labels ending with a pointer.

The whole point of this flexibility is to reduce the size of domain names encoded in DNS messages 📉.

Now the drama 🍿. It was observed that Yeti-Root servers running Knot 2.0 would compress the zero-length label (the root domain, often written as .) using a pointer to an earlier example. Although legal, this encoding increases the encoded size of the root label from one octet to two 😭. It was also found to break some client software, in particular the Go DNS library 💀. Bug reports were filed against both Knot and the Go DNS library, and both were resolved in subsequent releases 👑.

6. Conclusions ✨

okay so Yeti DNS was designed and built as a live DNS root system testbed 🧪. It serves a root zone ("Yeti-Root" in this document) that's derived from the root zone published by the IANA, with only those structural tweaks needed to make it work in the testbed system 💅. And honestly? It's been a super useful platform for answering questions that would be really hard to answer with the production Root Server system, like the ones in Section 3 🤓🫶.

Here are the indicative findings from building and running Yeti DNS:

  • Operation in a pure IPv6-only environment 🌐. We confirmed a significant failure rate in transmitting large responses (~7%) 😭, but no other persistent failures were observed. There were two cases where Yeti-Root servers failed to retrieve the Yeti-Root zone because of fragmentation of TCP segments. This got mitigated by setting a TCP MSS of 1220 octets (see Section 5.1.1) 🎀.
  • Successful operation with three autonomous Yeti-Root zone signers and 25 Yeti-Root servers ✨. We also confirmed that IXFR is not an appropriate transfer mechanism for zones that are structurally incongruent across different transfer paths (see Section 5.2) 🙈.
  • ZSK size increased to 2048 bits, and multiple KSK rollovers were executed to exercise support of RFC 5011 in validating resolvers 🔑. We also identified pitfalls relating to views in BIND9 when configured with "managed-keys" (see Section 5.3) 🫣.
  • Using natural (non-normalized) names for Yeti-Root servers exposed some differences between implementations in whether additional-section glue gets included in responses to priming queries 👀. Still, despite this inefficiency, Yeti resolvers were observed to function adequately (see Section 4.5) 💖.
  • Knot 2.0 was observed to perform label compression on the root (empty) label 🙃. That increased the encoding size for references to the root label, since a pointer is encoded as two octets whilst the root label itself only requires one (see Section 5.6) 💀.
  • Some tools got developed from the operational experience of running and using the Yeti DNS testbed 🛠️: DNS fragment and DNS Additional Truncated Response (ATR) for large DNS responses, a BIND9 patch for additional-section glue, YmmV, and IPv6 defrag for capturing and mirroring traffic. On top of that, a tool to facilitate automatic maintenance of hints files was created (see Appendix D) 🧚.

The Yeti DNS testbed was only used by end-users whose local infrastructure providers had made the conscious decision to use it, which is exactly what you want for an experimental, non-production system 🧪. So far, no serious user complaints have reached Yeti's mailing list during normal Yeti operation 🥹. Adding more instances into the Yeti root system may help improve the quality of service, but it's generally accepted that Yeti DNS performance is good enough to serve the purpose of a DNS Root testbed 👑.

The experience gained during operation suggested several topics worthy of further study 📚:

  • Priming truncation and TCP-only Yeti-Root servers. The idea is to observe and measure the worst-possible case for priming truncation by responding with TC=1 to all priming queries received over UDP transport, forcing clients to retry using TCP 😈. This should also give some insight into the usefulness of TCP-only DNS in general.
  • KSK ECDSA Rollover. One possible way to reduce DNSKEY response sizes is to change to an elliptic curve signing algorithm 🧮. While in principle this can be done separately for the KSK and the ZSK, the RIPE NCC has done recent research and discovered that some resolvers require both KSK and ZSK to use the same algorithm. That means an algorithm roll also involves a KSK roll. Performing an algorithm roll at the root would be an interesting challenge 🏔️.
  • Sticky Notify for zone transfer. The non-applicability of IXFR as a zone transfer mechanism in the Yeti DNS testbed could be mitigated by implementing a sticky preference for master server for each slave 📌. That way, an initial AXFR response could be followed up with IXFR requests without compromising zone integrity, even in the case (as with Yeti) where equivalent but incongruent versions of a zone are served by different masters.
  • Key distribution for zone transfer credentials. Using a shared secret between slave and master requires key distribution and management, and its scaling properties aren't ideally suited to systems with large numbers of transfer clients 🔐. Other approaches for key distribution and authentication could be considered.
  • DNS is a tree-based hierarchical database 🌳. Mathematically, it has a root node and a dependency between parent and child nodes. So any failures and instability of parent nodes (Root in Yeti's case) may impact their child nodes, whether that's from a human mistake, a malicious attack, or even an earthquake 🌍. It is proposed to define technology and practices that allow any organization, from the smallest company to nations, to be self-sufficient in their DNS 🗽.
  • In Section 3.12 of [RFC8324], a "Centrally Controlled Root" is viewed as an issue of DNS 🏛️. In future work, it would be interesting to test some technical tools like blockchain [BC] to either remove the technical requirement for a central authority over the root, or to enhance the security and stability of the existing Root ⛓️.

7. Security Considerations

Security is a big deal here, so buckle up 😭🔒. As introduced in Section 4.4, service metadata is synchronized among 3 DMs using the Git tool. Any security issue around Git may affect Yeti DM operation. For example, a hacker may compromise one DM's Git repository and push unwanted changes to the Yeti DM system. That could introduce a bad root server or a bad key for a period of time 🚨.

The Yeti resolver needs bootstrapping files to join the testbed, like the hints file and the trust anchor of Yeti 🧷. All of the required information is published on <yeti-dns.org> and <github.com>. If a hacker tampers with those websites by creating a fake page, a new resolver may lose its way and get configured with a bad root 💀.

DNSSEC is an important research goal in the Yeti DNS testbed 🔏. To reduce the central function of DNSSEC for the Root zone, we sign the Yeti-Root zone using multiple, independently operated DNSSEC signers and multiple corresponding ZSKs (see Section 4.2) 🧑‍🤝‍🧑. To verify ICANN's KSK rollover, we rolled the Yeti KSK three times according to RFC 5011, and we do have some observations (see Section 5.3) 👀. In addition, larger RSA key sizes were used in the testbed before 2048-bit keys were used in the ZSK signing process of the IANA Root zone 🗝️.

8. IANA Considerations

Good news, bestie: this document has no IANA actions 🎉💅.

9. References 📚

9.1. Normative References

9.2. Informative References 📚

okay bestie, this is the reading list 🤓 these aren't required to implement anything, they're just the receipts and background sources the doc leans on 💅

Appendix A. Yeti-Root Hints File 🧚

this hints file is the main character here ✨ it's complete and accurate as of when it was written 💖 and if a resolver uses it, it gets pointed at the Yeti DNS testbed instead of the real production Root Server system, so it can join in on experiments running on the testbed 🧪

heads up: some lines got wrapped below so they fit the page 🙈 wrapped lines are marked with a backslash ("") at the end, which is the usual convention 💅

   .                     3600000  IN   NS     bii.dns-lab.net
   bii.dns-lab.net       3600000  IN   AAAA   240c:f:1:22::6
   .                     3600000  IN   NS     yeti-ns.tisf.net
   yeti-ns.tisf.net      3600000  IN   AAAA   2001:559:8000::6
   .                     3600000  IN   NS     yeti-ns.wide.ad.jp
   yeti-ns.wide.ad.jp    3600000  IN   AAAA   2001:200:1d9::35
   .                     3600000  IN   NS     yeti-ns.as59715.net
   yeti-ns.as59715.net   3600000  IN   AAAA   \
                              2a02:cdc5:9715:0:185:5:203:53
   .                     3600000  IN   NS     dahu1.yeti.eu.org
   dahu1.yeti.eu.org     3600000  IN   AAAA   \
                              2001:4b98:dc2:45:216:3eff:fe4b:8c5b
   .                     3600000  IN   NS     ns-yeti.bondis.org
   ns-yeti.bondis.org    3600000  IN   AAAA   2a02:2810:0:405::250
   .                     3600000  IN   NS     yeti-ns.ix.ru
   yeti-ns.ix.ru         3600000  IN   AAAA   2001:6d0:6d06::53
   .                     3600000  IN   NS     yeti.bofh.priv.at
   yeti.bofh.priv.at     3600000  IN   AAAA   2a01:4f8:161:6106:1::10
   .                     3600000  IN   NS     yeti.ipv6.ernet.in
   yeti.ipv6.ernet.in    3600000  IN   AAAA   2001:e30:1c1e:1::333
   .                     3600000  IN   NS     yeti-dns01.dnsworkshop.org
   yeti-dns01.dnsworkshop.org \
                         3600000  IN   AAAA   2001:1608:10:167:32e::53
   .                     3600000  IN   NS     yeti-ns.conit.co
   yeti-ns.conit.co      3600000  IN   AAAA   \
                             2604:6600:2000:11::4854:a010
   .                     3600000  IN   NS     dahu2.yeti.eu.org
   dahu2.yeti.eu.org     3600000  IN   AAAA   2001:67c:217c:6::2
   .                     3600000  IN   NS     yeti.aquaray.com
   yeti.aquaray.com      3600000  IN   AAAA   2a02:ec0:200::1
   .                     3600000  IN   NS     yeti-ns.switch.ch
   yeti-ns.switch.ch     3600000  IN   AAAA   2001:620:0:ff::29
   .                     3600000  IN   NS     yeti-ns.lab.nic.cl
   yeti-ns.lab.nic.cl    3600000  IN   AAAA   2001:1398:1:21::8001
   .                     3600000  IN   NS     yeti-ns1.dns-lab.net
   yeti-ns1.dns-lab.net  3600000  IN   AAAA   2001:da8:a3:a027::6
   .                     3600000  IN   NS     yeti-ns2.dns-lab.net
   yeti-ns2.dns-lab.net  3600000  IN   AAAA   2001:da8:268:4200::6
   .                     3600000  IN   NS     yeti-ns3.dns-lab.net
   yeti-ns3.dns-lab.net  3600000  IN   AAAA   2400:a980:30ff::6
   .                     3600000  IN   NS     \
                           ca978112ca1bbdcafac231b39a23dc.yeti-dns.net
   ca978112ca1bbdcafac231b39a23dc.yeti-dns.net \
                         3600000  IN   AAAA   2c0f:f530::6
   .                     3600000  IN   NS     \
                           3e23e8160039594a33894f6564e1b1.yeti-dns.net
   3e23e8160039594a33894f6564e1b1.yeti-dns.net \
                         3600000  IN   AAAA   2803:80:1004:63::1
   .                     3600000  IN   NS     \
                           3f79bb7b435b05321651daefd374cd.yeti-dns.net
   3f79bb7b435b05321651daefd374cd.yeti-dns.net \
                         3600000  IN   AAAA   2401:c900:1401:3b:c::6
   .                     3600000  IN   NS     \
                           xn--r2bi1c.xn--h2bv6c0a.xn--h2brj9c
   xn--r2bi1c.xn--h2bv6c0a.xn--h2brj9c \
                         3600000  IN   AAAA   2001:e30:1c1e:10::333
   .                     3600000  IN   NS     yeti1.ipv6.ernet.in
   yeti1.ipv6.ernet.in   3600000  IN   AAAA   2001:e30:187d::333
   .                     3600000  IN   NS     yeti-dns02.dnsworkshop.org
   yeti-dns02.dnsworkshop.org \
                         3600000  IN   AAAA   2001:19f0:0:1133::53
   .                     3600000  IN   NS     yeti.mind-dns.nl
   yeti.mind-dns.nl      3600000  IN   AAAA   2a02:990:100:b01::53:0

Appendix B. Yeti-Root Server Priming Response 🎀

okay so this is the actual reply a Yeti root name server gives to a priming request 🧚 the authoritative server here runs NSD, so you're seeing its real output, no edits 💅

...
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62391
;; flags: qr aa rd; QUERY: 1, ANSWER: 26, AUTHORITY: 0, ADDITIONAL: 7
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1460
;; QUESTION SECTION:
;.                      IN NS

;; ANSWER SECTION:
.            86400 IN NS bii.dns-lab.net.
.            86400 IN NS yeti.bofh.priv.at.

.            86400 IN NS yeti.ipv6.ernet.in.
.            86400 IN NS yeti.aquaray.com.
.            86400 IN NS yeti.jhcloos.net.
.            86400 IN NS yeti.mind-dns.nl.
.            86400 IN NS dahu1.yeti.eu.org.
.            86400 IN NS dahu2.yeti.eu.org.
.            86400 IN NS yeti1.ipv6.ernet.in.
.            86400 IN NS ns-yeti.bondis.org.
.            86400 IN NS yeti-ns.ix.ru.
.            86400 IN NS yeti-ns.lab.nic.cl.
.            86400 IN NS yeti-ns.tisf.net.
.            86400 IN NS yeti-ns.wide.ad.jp.
.            86400 IN NS yeti-ns.datev.net.
.            86400 IN NS yeti-ns.switch.ch.
.            86400 IN NS yeti-ns.as59715.net.
.            86400 IN NS yeti-ns1.dns-lab.net.
.            86400 IN NS yeti-ns2.dns-lab.net.
.            86400 IN NS yeti-ns3.dns-lab.net.
.            86400 IN NS xn--r2bi1c.xn--h2bv6c0a.xn--h2brj9c.
.            86400 IN NS yeti-dns01.dnsworkshop.org.
.            86400 IN NS yeti-dns02.dnsworkshop.org.
.            86400 IN NS 3f79bb7b435b05321651daefd374cd.yeti-dns.net.
.            86400 IN NS ca978112ca1bbdcafac231b39a23dc.yeti-dns.net.
.            86400 IN RRSIG NS 8 0 86400 (
                         20171121050105 20171114050105 26253 .
                         FUvezvZgKtlLzQx2WKyg+D6dw/pITcbuZhzStZfg+LNa
                         DjLJ9oGIBTU1BuqTujKHdxQn0DcdFh9QE68EPs+93bZr
                         VlplkmObj8f0B7zTQgGWBkI/K4Tn6bZ1I7QJ0Zwnk1mS
                         BmEPkWmvo0kkaTQbcID+tMTodL6wPAgW1AdwQUInfy21
                         p+31GGm3+SU6SJsgeHOzPUQW+dUVWmdj6uvWCnUkzW9p
                         +5en4+85jBfEOf+qiyvaQwUUe98xZ1TOiSwYvk5s/qiv
                         AMjG6nY+xndwJUwhcJAXBVmGgrtbiR8GiGZfGqt748VX
                         4esLNtD8vdypucffem6n0T0eV1c+7j/eIA== )

;; ADDITIONAL SECTION:
bii.dns-lab.net.        86400 IN AAAA 240c:f:1:22::6
yeti.bofh.priv.at.      86400 IN AAAA 2a01:4f8:161:6106:1::10
yeti.ipv6.ernet.in.     86400 IN AAAA 2001:e30:1c1e:1::333
yeti.aquaray.com.       86400 IN AAAA 2a02:ec0:200::1
yeti.jhcloos.net.       86400 IN AAAA 2001:19f0:5401:1c3::53
yeti.mind-dns.nl.       86400 IN AAAA 2a02:990:100:b01::53:0

;; Query time: 163 msec
;; SERVER: 2001:4b98:dc2:45:216:3eff:fe4b:8c5b#53
;; WHEN: Tue Nov 14 16:45:37 +08 2017
;; MSG SIZE  rcvd: 1222

Appendix C. Active IPv6 Prefixes in Yeti DNS Testbed 🌈

this table is the list of IPv6 prefixes that were active during 2017 🦋 bestie, the whole thing is literally a map of who was originating what 😭

+----------------------+---------------------------------+----------+
| Prefix               | Originator                      | Location |
+----------------------+---------------------------------+----------+
| 240c::/28            | BII                             | CN       |
| 2001:6d0:6d06::/48   | MSK-IX                          | RU       |
| 2001:1488::/32       | CZ.NIC                          | CZ       |
| 2001:620::/32        | SWITCH                          | CH       |
| 2001:470::/32        | Hurricane Electric, Inc.        | US       |
| 2001:0DA8:0202::/48  | BUPT6-CERNET2                   | CN       |
| 2001:19f0:6c00::/38  | Choopa, LLC                     | US       |
| 2001:da8:205::/48    | BJTU6-CERNET2                   | CN       |
| 2001:62a::/31        | Vienna University Computer      | AT       |
|                      | Center                          |          |
| 2001:67c:217c::/48   | AFNIC                           | FR       |
| 2a02:2478::/32       | Profitbricks GmbH               | DE       |
| 2001:1398:1::/48     | NIC Chile                       | CL       |
| 2001:4490:dc4c::/46  | NIB (National Internet          | IN       |
|                      | Backbone)                       |          |
| 2001:4b98::/32       | Gandi                           | FR       |
| 2a02:aa8:0:2000::/52 | T-Systems-Eltec                 | ES       |
| 2a03:b240::/32       | Netskin GmbH                    | CH       |
| 2801:1a0::/42        | Universidad de Ibague           | CO       |
| 2a00:1cc8::/40       | ICT Valle Umbra s.r.l.          | IT       |
| 2a02:cdc0::/29       | ORG-CdSB1-RIPE                  | IT       |
+----------------------+---------------------------------+----------+

Appendix D. Tools Developed for Yeti DNS Testbed 🛠️

so the Yeti DNS testbed had a whole toolbox built around it 🧰 here are a few of the highlights, described quick and dirty ✨

YmmV ("Yeti Many Mirror Verifier") is built to make life easy and safe for a DNS administrator 💖 it captures traffic sent from a resolver to the Root Server system, then replays it toward Yeti-Root servers. Both sets of responses get recorded and compared, and any differences get logged. See https://github.com/BII-Lab/ymmv.

PcapParser is a module that YmmV uses 🍓 it puts back together fragmented IPv6 datagrams and TCP segments from a PCAP archive, then pulls out the DNS messages inside. See https://github.com/RunxiaWan/PcapParser.

DNS-layer-fragmentation has DNS proxies that do application-level fragmentation of DNS messages, based on [FRAGMENTS] 🦄 the vibe is exploring how to split DNS messages in the protocol itself, so they won't get fragmented by the IP layer. See https://github.com/BII-Lab/DNS-layer-Fragmentation.

DNS_ATR is an implementation of DNS Additional Truncated Response (ATR), as described in [ATR] and [HOW_ATR_WORKS] 🫧 it sits between resolver and authoritative servers as a silent, transparent listener, forwarding queries and responses along the way. Responses larger than a nominated threshold (1280 octets by default) set off additional truncated responses sent right after the big one. See https://github.com/songlinjian/DNS_ATR.

Appendix E. Controversy 💜

okay the spicy part 🌶️ but let's keep it respectful, it's a real discussion

The Yeti DNS Project, its infrastructure, and the experiments run on that infrastructure have been explained by the people involved in lots of public meetings at technical venues since it started ✨ the mailing lists used to coordinate the infrastructure are open to join, and their archives are public. The project as a whole has had robust public discussion, which honestly is giving transparency 🙌

Some commentators are worried that the Yeti DNS Project is basically running an alternate root, which goes against the IAB's comments published in [RFC2826] 👀 other alternate roots have been seen to confuse end users and make the DNS namespace unstable, by adding new top-level labels or using top-level labels differently from the Root Server system. The coordinators of the Yeti DNS Project say it's not an alternate root in that sense, since by design the namespace enabled by the Yeti-Root zone is identical to that of the Root Zone 💅

Some commentators are also concerned that the Yeti DNS Project is trying to influence or undermine administrative policy for the Root Server system 😬 they point to DNSSEC trust anchors that the IANA doesn't publish, and to Yeti-Root servers in regions where governments or other organizations have shown interest in running a Root Server. The coordinators say their mandate is entirely technical and has no ambition to influence policy directly 🫶 they do hope, though, that the technical findings could be a useful resource for the wider technical community 🌸

Acknowledgments 🫶

first things first, the authors want to thank everyone who donated their time and resources to implementing and operating the Yeti DNS 🥹 they are:

Tomohiro Ishihara, Antonio Prado, Stephane Bortzmeyer, Mickael
Jouanne, Pierre Beyssac, Joao Damas, Pavel Khramtsov, Dmitry
Burkov, Dima Burkov, Kovalenko Dmitry, Otmar Lendl, Praveen Misra,
Carsten Strotmann, Edwin Gomez, Daniel Stirnimann, Andreas
Schulze, Remi Gacogne, Guillaume de Lafond, Yves Bovard, Hugo
Salgado, Kees Monshouwer, Li Zhen, Daobiao Gong, Andreas Schulze,
James Cloos, and Runxia Wan.

thanks also to everyone who gave important advice and comments to Yeti, whether in face-to-face meetings or virtually by phone or mailing list 💖 some of those people are:

Wu Hequan, Zhou Hongren, Cheng Yunqing, Xia Chongfeng, Tang
Xiongyan, Li Yuxiao, Feng Ming, Zhang Tongxu, Duan Xiaodong, Wang
Yang, Wang JiYe, Wang Lei, Zhao Zhifeng, Chen Wei, Wang Wei, Wang
Jilong, Du Yuejing, Tan XiaoSheng, Chen Shangyi, Huang Chenqing,
Ma Yan, Li Xing, Cui Yong, Bi Jun, Duan Haixing, Marc Blanchet,
Andrew Sullivan, Suzanne Wolf, Terry Manderson, Geoff Huston, Jaap
Akkerhuis, Kaveh Ranjbar, Jun Murai, Paul Wilson, and Kilnam
Chonm.

The authors also thank the Independent Submissions Editorial Board, and these reviewers, whose opinions helped make the document clearer 🎀

Joe Abley, Paul Mockapetris, and Subramanian Moonesamy.

Authors' Addresses 💅

here are the authors and how to reach them 📬

Linjian Song (editor)
Beijing Internet Institute
2nd Floor, Building 5, No.58 Jing Hai Wu Lu, BDA
Beijing  100176
China
Email: songlinjian@gmail.com
URI:   http://www.biigroup.com/


Dong Liu
Beijing Internet Institute
2nd Floor, Building 5, No.58 Jing Hai Wu Lu, BDA
Beijing  100176
China
Email: dliu@biigroup.com
URI:   http://www.biigroup.com/


Paul Vixie
TISF
11400 La Honda Road
Woodside, California  94062
United States of America
Email: vixie@tisf.net
URI:   http://www.redbarn.org/


Akira Kato
Keio University/WIDE Project
Graduate School of Media Design, 4-1-1 Hiyoshi, Kohoku
Yokohama  223-8526
Japan
Email: kato@wide.ad.jp
URI:   http://www.kmd.keio.ac.jp/


Shane Kerr
Antoon Coolenlaan 41
Uithoorn  1422 GN
The Netherlands
Email: shane@time-travellers.org
📦 where this page comes from