RFC 2517

Building Directories from DNS: Experiences from WWWSeeker 💖

Okay bestie, this is RFC 2517 🎀. It's titled "Building Directories from DNS: Experiences from WWWSeeker," and it's written by R. Moats and R. Huber from AT&T. It's Informational (so it shares info, it doesn't set a standard), it's from the Network Working Group, and it's dated February 1999 🦋. There's no Obsoletes or Updates line on it, so it doesn't replace or change any other RFC. It's giving "here's what we learned the hard way" energy 💅.

---

Status of this Memo ✨

Okay, so this memo is just giving info to the Internet community 🌸. It does not specify an Internet standard of any kind, and it can be shared freely with no limits.

Copyright Notice 🌈

The Internet Society owns the copyright (1999). All rights reserved 💜.

Abstract 🔥

So there's been a LOT of chatter, plus several documents, about needing an Internet Directory 💅. Lately everyone's been focused on finding an organization's domain name without using DNS as a directory service 🧚. This memo is about the lessons learned while InterNIC Directory and Database Services built and ran WWWSeeker ✨, an app that finds a web site when you've got an organization's name and location. The backend database that powered it was built from info pulled from domain registries using WHOIS and other protocols 🍓. We're sharing this so future implementors can skip the blind alleys we already wandered into 🙈. All of this builds on Netfind, which was created by Mike Schwartz and his team at the University of Colorado at Boulder [1] 🫶.

1. Introduction 🎀

Over time there have been several RFCs [2, 3, 4] about ways to do Internet Directories 🌸. A lot of the early docs were about white pages directories, which map a person's name to their telephone number, email address, and so on 📞.

More recently people have been talking about directories that map a company name to a domain name or web site 🦋. Honestly, lots of people use DNS as a directory right now to find this info about a company 👀. The usual move is to guess the company's domain name and stick www. in front of it 💅. That makes it SUPER desirable for a company to have an easy-to-guess name, iykyk 😭.

There are two big problems here 💀. First, as more names get assigned, it gets harder to snag an easy-to-guess one 😩. Second, you also have to guess the TLD, not just the name. A lot of people just guess .COM as the default 💖, but there are tons of two-letter country code top-level domains in use now, plus other gTLDs (.NET, .ORG, and possibly .EDU), and more gTLDs might be coming 🌈. The more TLDs people use, the harder guessing gets 🙃.

Between July 1996 and the shutdown in March 1998, the InterNIC Directory and Database Services project kept the Netfind search engine [1] running, along with the database that maps organization info to domain names 🧚. So that database basically acted as the kind of Internet directory that links company names to domain names 👑. The team also built WWWSeeker, which used the Netfind database to find web sites tied to a given organization ✨. Maintaining and growing that database taught them a ton about running a directory service 🤓. They're sharing it so future implementors can avoid the blind alleys they already explored 💜.

2. Directory Population 🌸

2.1 What to do?

Populating a directory has two parts ✨: finding all the domain names (building the skeleton) and linking those domains to entities (adding the meat) 🍓. Both get covered below 🫶.

2.2 Building the skeleton

So for "building the skeleton," a popular suggestion is some version of a "tree walk" to figure out which domains should go in the directory 🌳. Their experience says this is neither reasonable nor efficient for keeping a directory up to date 😬. Aside from some infrequent, long-standing DNS surveys [5], the Internet community has been pretty discouraged about DNS "tree walks" 🙅‍♀️. Plus, DNS changes so often that you'd need a new tree walk every month (if not more often) 📆. Instead, they found that data on allocated DNS domains can usually be grabbed in bulk with FTP, HTTP, or Gopher, and they used each of those for particular TLDs 🦋. That's a bonus too, because you get "building the skeleton" and "adding the meat" done at the same time 🎀.

Their favorite way to find a server with allocated DNS domain info is to start with the list at http://www.alldomains.com/countryindex.html and go from there 🗺️. Before that list existed, you had to hunt for a registry by trial and error 😭.

When you're maintaining the database, you can verify existing domains with direct DNS lookups instead of doing a "tree walk" 🧚. So "tree walks" should honestly be the absolute last resort for populating a directory 🙈, and bulk retrieval should be used whenever possible 💅.

2.3 Adding the meat

One way to populate a directory ("adding the meat") is to run an automated system that makes repeated WHOIS queries to gather info about the organization that owns a domain 🍓. Those queries would go to a WHOIS server found using the method above 🔍. When the InterNIC Directory and Database Services project wrapped up, the backend database had about 2.9 million records built from data that could be pulled via WHOIS 📊. The whole database had 3.25 million records, with the rest coming from sources other than WHOIS 🌈.

Their experience is that this info has a lot of factual and typographical errors, so it needs more examination and processing to get the quality up 😩. Also, TLD registrars that support WHOIS usually only support WHOIS info for second-level domains (like ne.us), not lower-level domains (like windrose.omaha.ne.us) 🫠. And some TLDs don't have registrars, some don't support WHOIS at all, and others use HTTP, FTP, or gopher to provide org info 🙃. So the takeaway is that an implementor of an internet directory needs to support multiple protocols for directory population 💜. An automated WHOIS search tool is necessary, but it isn't enough 💅.

3. Directory Updating: Full Rebuilds vs Incremental Updates ⏰

Check this out 👀: given how big the database was in April 1998 when it was last generated, a complete rebuild from WHOIS lookups would take between 134.2 and 167.8 days, just for the WHOIS lookups, on a Sun SPARCstation 20 💀. That number doesn't even include everything else, like inverting the token tree, which took about 24 hours of processing on a Sun SPARCstation 20 🥲. All of that would make the total rebuild time even longer 😭.

Whether a rebuild is even doable depends on how often the database gets updated 📅. The allocated domain count was growing by 150K-200K new domains per month in early 1998 📈, so they did monthly updates 🌸. Rebuilding the whole thing every month (based on that time estimate) would need 3 to 5 machines dedicated full time, no matter the machine architecture 🖥️💀. So instead, they checkpointed the allocated domain list and rebuilt incrementally during one weekend each month ✨. That let them finish the update on between 1 and 4 machines (3 Sun SPARCstation 20s and a dual-processor Sparcserver 690) over a couple of days, without fully dedicating the hardware 🙌.

They also paired incremental updates with periodic refreshes of existing data 🔁. Those refreshes could happen in a different part of the month and didn't need full dedication of the hardware either, so older records got updated over time whenever the underlying info changed 🦋. The tradeoff is timeliness and accuracy (some data in the database may be old) versus hardware and processing costs ⚖️. It's a real balancing act, bestie 💅✨.

9. Full Copyright Statement ✨

okay bestie, this part is the legal stuff, so it's a little dry, but the vibe is still clear 💅

Copyright (C) The Internet Society (1999).  All Rights Reserved.

The Internet Society owns the copyright 🎀 and you're allowed to copy this document and share it with others. You can also write derivative works that comment on it, explain it, or help people implement it, and you can prepare, copy, publish, and distribute those, in whole or in part, with no restrictions of any kind 🦋. The one catch is that the copyright notice and this paragraph have to stay on every copy and every derivative work 🫶.

Here's the tea 🍵: this document itself may not be modified in any way. You can't remove the copyright notice or the references to the Internet Society or other Internet organizations 🙅‍♀️. There are two exceptions. First, you can change it when that's needed for developing Internet standards, but then the copyright procedures in the Internet Standards process must be followed. Second, you can change it to translate it into languages other than English 🌸.

The limited permissions above are perpetual 💜. The Internet Society and its successors or assigns will never revoke them ✨.

And now the disclaimer, which is very serious, so I'm keeping the exact wording 🙈

This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

basically the document is handed over "AS IS," and the Internet Society and the Internet Engineering Task Force don't promise anything about it. That includes no promise that using the info won't infringe anyone's rights, and no implied promise of merchantability or fitness for a particular purpose 💀

periodt 💅