RFC 1536

Common DNS Implementation Errors and Suggested Fixes

📅 published October 1993, from ietf

RFC 1536: Common DNS Implementation Errors and Suggested Fixes 💖

Authors: A. Kumar, J. Postel, C. Neuman (ISI), P. Danzig, S. Miller (USC) Category: Informational Date: October 1993

Status of this Memo ✨

bestie, this memo is informational 🎀 It gives info to the Internet community. It does not specify an Internet standard. Distribution of this memo is unlimited.

Abstract 🦋

okay so this memo covers the common DNS implementation errors people keep running into, plus some fixes 💅 It calls out where things go against STD 13, RFC 1034 and STD 13, RFC 1035. Where it's relevant, it also walks through how BIND (versions 4.8.3 and 4.9, the ones the authors looked at) does things, as an example 🫶

Introduction 🌸

ngl the last few years were a total DNS traffic explosion on the NSFnet backbone 😭 Lots of DNS implementations, and lots of versions of them, talk to each other and make a huge pile of unnecessary traffic. Researchers all over the internet are trying to document these interactions, the symptomatic traffic patterns, and ways to fix the broken software.

This draft is an attempt to write down fixes for known DNS problems, so you know what to watch out for and how to repair broken software 🧚

1. Fast Retransmissions 🔥

DNS uses the classic request-response model of client-server interaction. So UDP is the chosen protocol, even though TCP is used for zone transfers. If no response shows up within a "reasonable" period of time, it's the client's job to requery. RFC 1034 and 1035 don't recommend any retransmission policy, but RFC 1035 does recommend that resolvers cycle through a list of servers. So both name servers and stub resolvers should implement some kind of retransmission policy based on round trip time estimates of the name servers. The client should back off exponentially, probably up to a maximum timeout value ⏳

But clients might not do either of these. They might not wait long enough before retransmitting, or they might not back off their inter-query times enough 🙈

So what the server sees is a series of queries from the same querying entity, all spaced super close together. Sure, a correctly implemented server throws away all the duplicate queries, but those queries still add to wide-area traffic anyway 💀

We classify a retransmission of a query as a pure Fast retry timeout problem when a series of query packets meets all of these conditions:

  • a. Query packets are seen within less than a "reasonable waiting period" of each other.
  • b. No response to the original query was seen. In other words, we see two or more queries, back to back.
  • c. The query packets share the same query identifier.
  • d. The server eventually responds to the query.

A GOOD IMPLEMENTATION: ✨

BIND (we looked at versions 4.8.3 and 4.9) has a good retransmission algorithm that solves or limits all of these problems 👑 The Berkeley stub-resolver queries servers at an interval that starts at the greater of 4 seconds and 5 seconds divided by the number of servers the resolver queries. The resolver cycles through servers, and at the end of a cycle, it backs off the timeout exponentially.

The Berkeley full-service resolver (built into the program "named") starts with a timeout equal to the greater of 4 seconds and two times the round-trip time estimate of the server. The timeout gets backed off with each cycle, exponentially, up to a ceiling value of 45 seconds 🎀

FIXES: 💅

  • a. Estimate round-trip times, or set a reasonably high initial timeout.
  • b. Back off timeout periods exponentially.
  • c. Yet another fundamental, though difficult, fix is to send the client an acknowledgement of a query, along with a round-trip time estimate.

Since UDP is used, the client doesn't expect any response until the query is complete. So it's less likely to have info about previous packets to estimate its back-off time from 🥺 Unless you maintain state across queries, so later queries to the same server can use info from earlier ones. Unfortunately, those estimates are likely to be inaccurate for chained requests, since the variance is likely to be high 📉

The fix chosen in the ARDP library used by Prospero is that the server sends an initial acknowledgement to the client in cases where it expects the query to take a long time (which might happen with chained queries). That initial acknowledgement can include an expected time to wait before retrying 🧚

This fix is harder, though, because the client software also has to be trained to expect the acknowledgement packet. In an internet of millions of hosts, that's at best a hard problem 😭

2. Recursion Bugs 🌈

So when a server gets a client request, it first checks its zone data and its cache to see if it can answer the query. If the answer isn't available in either place, the server looks for names of servers that are more likely to have the info, in its cache or zone data. Then it does one of two things. If the client wants the server to recurse and the server architecture allows recursion, the server chains the request to the known servers closest to the queried name. If the client doesn't ask for recursion, or the server can't handle it, the server returns the list of name servers to the client, assuming the client knows what to do with those records 🍓

The client then queries this new list of name servers to get either the answer, or names of another set of name servers to query. This keeps going until the client is satisfied. Servers might also go through this chaining process if the server returns a CNAME record for the queried name. Some servers reprocess this name to try to get the record type they want 💜

Now, in certain cases this chain of events can go sideways. For example, a broken or malicious name server might list itself as one of the name servers to query again. The unsuspecting client then resends the same query to the same server 🙈

In another situation, which is harder to detect, a set of servers might form a loop where A refers to B and B refers to A. This loop might involve more than two servers 🔁

Yet another error is when the client doesn't know how to process the list of name servers it gets back, and requeries the same server, since that's one of the few servers it knows 🤷‍♀️

So we, therefore, sort recursion bugs into three distinct categories:

  • a. Ignored referral: the client did not know how to handle NS records in the AUTHORITY section.
  • b. Too many referrals: the client called on a server too many times, beyond a "reasonable" number, with the same query. This is different from a Fast retransmission problem and from a Server Failure detection problem, because a response is seen for every query. Also, the identifiers are always different. It implies the client is in a loop and should have detected that and broken it. (RFC 1035 mentions that a client should not recurse beyond a certain depth.)
  • c. Malicious Server: a server refers to itself in the authority section. If a server doesn't have an answer now, it's very unlikely to be any better the next time you query it, especially when it claims to be authoritative over a domain 🚩

RFC 1034 warns against situations like this, on page 35:

"Bound the amount of work (packets sent, parallel processes started) so that a request can't get into an infinite loop or start off a chain reaction of requests or queries with other implementations EVEN IF SOMEONE HAS INCORRECTLY CONFIGURED SOME DATA."

A GOOD IMPLEMENTATION: ✨

BIND fixes at least one of these problems 👏 It puts an upper limit on the number of recursive queries it will make to answer a question. It chases a maximum of 20 referral links and 8 canonical name translations.

FIXES: 💅

  • a. Set an upper limit on the number of referral links and CNAME links you're willing to chase.

Note that this is not guaranteed to break only recursion loops. In a rare case, it could prune off a very long search path prematurely. We know, however, with high probability, that if the number of links crosses a certain metric (two times the depth of the DNS tree), it's a recursion problem.

  • b. Watch out for self-referring servers, and avoid them whenever possible.
  • c. Make sure you never pass off an authority NS record with your own name on it! 🙅‍♀️
  • d. Fix clients to accept iterative answers from servers that aren't built to provide recursion. Such clients should either be happy with the non-authoritative answer, or be willing to chase the referral links themselves.

3. Zero Answer Bugs: 🎀

Name servers sometimes return an authoritative NOERROR with no ANSWER, AUTHORITY or ADDITIONAL records. This happens when the queried name is valid but doesn't have a record of the desired type. Of course, the server has authority over the domain 💖

However, once again, some resolver implementations don't interpret this kind of response reasonably. They always expect an answer record when they see an authoritative NOERROR. These entities keep resending their queries, possibly endlessly 😩

A GOOD IMPLEMENTATION 🦋

BIND resolver code doesn't query a server more than 3 times. If it can't get an answer from 4 servers, querying each of them three times, it returns an error.

Of course, it treats a zero-answer response the way it should be treated: with respect! 👑

FIXES: 💅

  • a. Set an upper limit on the number of retransmissions for a given query, at the very least.
  • b. Fix resolvers to interpret such a response as an authoritative statement that the record type doesn't exist for the given name ✨

4. Inability to detect server failure ✨

Okay so servers on the internet are not very reliable 😭. They go down every once in a while, and resolvers are supposed to adapt by not bugging the server for a bit. So when a server doesn't respond, resolvers should try another server. Non-stub resolvers should also bump their round trip time estimate for that server up to a large value, so it doesn't get tried again before the faster servers. 🏃‍♀️💨

Stub resolvers, though, just cycle through a fixed set of servers. If one server is down and the others aren't answering for other reasons (high load, recursive resolution taking longer than the resolver's time-out, ....), the resolver queries the dead server again! 💀 Some resolvers don't even set an upper limit on how many query retransmissions they'll send, so they keep knocking on the dead server's door forever. Not it. 🙅‍♀️

Name servers running system or chained queries can have the same problem. They store the names of servers they should query for a given domain and cycle through them. If none of them answers, they hit each one more than once. Again, it's really important to have an upper limit on retransmissions, so the network doesn't get overloaded. 🌪️

This behavior clearly goes against the rule in RFC 1035 (page 46):

"If a resolver gets a server error or other bizarre response
 from a name server, it should remove it from SLIST, and may
 wish to schedule an immediate transmission to the next
 candidate server address."

Removing a server from SLIST means it isn't queried again for some time. 🫶

Correctly implemented full-service resolvers should, like we said before, update round trip time values for servers that don't respond and only query them after the other good servers. But full-service resolvers might not follow any of these common sense directives. They query dead servers, and they query them endlessly. Obsessed with the wrong behavior, honestly. 🙈

A GOOD IMPLEMENTATION:

BIND puts an upper limit on how many times it queries a server. Both the stub-resolver and the full-service resolver code do this. 🎀 Also, since the full-service resolver estimates round-trip times and sorts name server addresses by those estimates, it doesn't query a dead server again until and unless all the other servers in the list are dead too! On top of that, BIND implements exponential back-off too. 👑

FIXES:

  • a. Set an upper limit on the number of retransmissions.
  • b. Measure round-trip time from servers (some estimate is better than none). Treat no response as a "very large" round-trip time.
  • c. Maintain a weighted rtt estimate and decay the "large" value slowly, with time, so that the server is eventually tested again, but not after an indefinitely long period.
  • d. Follow an exponential back-off scheme so that even if you don't restrict the number of queries, you don't overload the net excessively.

5. Cache Leaks 💧

Every resource record returned by a server gets cached for TTL seconds, where the TTL value comes back with the RR. Full-service (or stub) resolvers cache the RR and answer any future queries from that cached info until the TTL expires. After that, one more trip to the wide-area network gets the RR back into the cache. 🔄

Full-service resolvers might not implement this caching mechanism well, though. They might put a limit on the cache size, or they might not interpret the TTL value correctly. Either way, queries repeated within a TTL period of an RR count as a cache leak. 🚰

A GOOD/BAD IMPLEMENTATION:

BIND has no restriction on cache size. The size is governed by the limits on the virtual address space of the machine it's running on. BIND caches RRs for the duration of the TTL returned with each record. ✨

BUT it doesn't follow the RFCs when it comes to interpreting a 0 TTL value. If a record has a TTL value of 0 seconds, BIND uses the minimum TTL value for that zone, from the SOA record, and caches it for that long. Saving some wide-area traffic is nice, but this is not correct behavior. 🙅‍♀️

FIXES:

  • a. Look over your caching mechanism to make sure TTLs are interpreted correctly.
  • b. Don't restrict cache sizes (come on, memory is cheap!). Expired entries are reclaimed periodically anyway. Of course, the cache size is bound to have some physical limit. But when possible, this limit should be large (run your name server on a machine with a large amount of physical memory). 💾
  • c. Possibly, a mechanism is needed to flush the cache when it's known, or even suspected, that the information has changed. 🧹

6. Name Error Bugs 🙃

This bug is very similar to the Zero Answer bug. A server returns an authoritative NXDOMAIN when the queried name is known to be bad by the server that's authoritative for the domain, in the absence of negative caching. This authoritative NXDOMAIN response usually comes with the SOA record for the domain, in the authority section. 📜

Resolvers should recognize that the name they queried for was a bad name and stop querying further. 🛑

Some resolvers might not interpret this correctly, though, and keep querying servers, expecting an answer record. Literally a lost cause. 😤

Some applications actually prompt NXDOMAIN answers! When given a perfectly good name to resolve, they append the local domain to it. For example, an application in the domain foo.bar.com, when trying to resolve the name usc.edu, first tries usc.edu.foo.bar.com, then usc.edu.bar.com, and finally the good name usc.edu. This causes at least two queries that return NXDOMAIN for every good query. 🫠 The problem gets worse because the negative answers from the previous queries aren't cached. So when the same name is sought again, the whole process repeats. 🔁

Some DNS resolver implementations suffer from this problem too. They append successive sub-parts of the local domain using an implicit searchlist mechanism, when certain conditions are satisfied, and only try the original name when this first set of iterations fails. This behavior recently caused pandemonium on the internet when the domain edu.com was registered and a wildcard CNAME record was placed at the top level. All machines from com domains trying to connect to hosts in the edu domain ended up connecting to the local machine in the edu.com domain! 😱💀

GOOD/BAD IMPLEMENTATIONS:

Some local versions of BIND already implement negative caching. They typically cache negative answers with a very small TTL, just enough to answer a burst of queries spaced close together, which is typically what gets seen. 🧠

The next official public release of BIND (4.9.2) will have negative caching as an ifdef'd feature. 🎉

The BIND resolver appends the local domain to the given name when one of two conditions is met:

  • i. The name has no periods and the flag RES_DEFNAME is set.
  • ii. There is no trailing period and the flag RES_DNSRCH is set.

The flags RES_DEFNAME and RES_DNSRCH are default resolver options in BIND, but they can be changed at compile time. ⚙️

Only if the name generated this way returns an NXDOMAIN is the original name tried as a Fully Qualified Domain Name. And only if it contains at least one period. 🔍

FIXES:

  • a. Fix the resolver code. 🛠️
  • b. Negative Caching. Negative caching servers will restrict the traffic seen on the wide-area network, even if they don't curb it altogether. 📉
  • c. Applications and resolvers should not append the local domain to names they seek to resolve, as far as possible. Names interspersed with periods should be treated as Fully Qualified Domain Names.

In other words, use searchlists only when explicitly specified. No implicit searchlists should be used. A name that contains any dots should first be tried as a FQDN, and if that fails, with the local domain name (or searchlist if specified) appended. A name containing no dots can be appended with the searchlist right away, but once again, no implicit searchlists should be used. 🌷

Associated with the name error bug is another problem, where a server might return an authoritative NXDOMAIN even though the name is valid. A secondary server, on start-up, reads the zone information from the primary through a zone transfer. While it's in the process of loading the zones, it doesn't have information about them, even though it's authoritative for them. So any query for a name in that domain gets answered with an NXDOMAIN response code. 😬 This problem wouldn't be a disaster if it weren't for negative caching servers that cache this answer and spread incorrect information across the internet. 🌐

BAD IMPLEMENTATION:

BIND apparently suffers from this problem. 🙈

Also, a new name added to the primary database takes a while to propagate to the secondaries. Until then, they return NXDOMAIN answers for a good name. Negative caching servers store this answer too and make the problem worse. This is probably a more general DNS problem, but it's apparently more harmful in this situation. 😩

FIX:

  • a. Servers should start answering only after loading all the zone data. A failed server is better than a server handing out incorrect information. 🚫
  • b. Negative cache records for a very small time, just enough to ward off a burst of requests for the same bad name. This could be related to the round-trip time of the server from which the negative answer was received. Alternatively, a statistical measure of how long queries for such names keep coming in could be used. The minimum TTL value from the SOA record is not advisable, since those tend to be pretty large. ⏱️
  • c. A "PUSH" (or, at least, a "NOTIFY") mechanism should be allowed and implemented, so the primary server can tell secondaries that the database has been modified since it last transferred zone data. To ease the problem of "too many zone transfers" that this might cause, Incremental Zone Transfers should also be part of DNS. Also, the primary shouldn't NOTIFY/PUSH with every update, but bunch a good number together. 📦

7. Format Errors 📝

Some resolvers send query packets that don't necessarily conform to the standards laid out in the relevant RFCs. This unnecessarily increases net traffic and wastes server time. 😤

FIXES:

  • a. Fix resolvers.
  • b. Each resolver verifies the format of packets before sending them out, using a mechanism outside of the resolver. This is, obviously, needed only if step 1 cannot be followed. 🔐

References 📚

[1] Mockapetris, P., "Domain Names Concepts and Facilities", STD 13,
    RFC 1034, USC/Information Sciences Institute, November 1987.

[2] Mockapetris, P., "Domain Names Implementation and Specification",
    STD 13, RFC 1035, USC/Information Sciences Institute, November
    1987.

[3] Partridge, C., "Mail Routing and the Domain System", STD 14, RFC
    974, CSNET CIC BBN, January 1986.

[4] Gavron, E., "A Security Problem and Proposed Correction With
    Widely Deployed DNS Software", RFC 1535, ACES Research Inc.,
    October 1993.

[5] Beertema, P., "Common DNS Data File Configuration Errors", RFC
    1537, CWI, October 1993.

Security Considerations 🔒

Security issues are not discussed in this memo. 🫣

Authors' Addresses 💌

Okay bestie, these are the humans who wrote this one, and yes, the contact info is all here. 🫶 Every detail is copied exactly, so you can reach out to any of them.

Anant Kumar 🌸

Anant Kumar
USC Information Sciences Institute
4676 Admiralty Way
Marina Del Rey CA 90292-6695

Phone:(310) 822-1511
FAX:  (310) 823-6741
EMail: anant@isi.edu

Jon Postel 🦋

Jon Postel
USC Information Sciences Institute
4676 Admiralty Way
Marina Del Rey CA 90292-6695

Phone:(310) 822-1511
FAX:  (310) 823-6714
EMail: postel@isi.edu

Cliff Neuman 🎀

Cliff Neuman
USC Information Sciences Institute
4676 Admiralty Way
Marina Del Rey CA 90292-6695

Phone:(310) 822-1511
FAX:  (310) 823-6714
EMail: bcn@isi.edu

Peter Danzig 🧚

Peter Danzig
Computer Science Department
University of Southern California
University Park

EMail: danzig@caldera.usc.edu

Steve Miller 💜

Steve Miller
Computer Science Department
University of Southern California
University Park
Los Angeles CA 90089

EMail: smiller@caldera.usc.edu

Five people, one USC crew, and a whole lot of protocol brainpower. The ISI folks share one address and phone line, so if you're trying to reach Anant, Jon, or Cliff, it's the same building. 🏫✨

📦 where this page comes from