RFC 1913
Architecture of the Whois++ Index Service
RFC 1913: Architecture of the Whois++ Index Service 🦋
Okay bestie, RFC 1913 is giving main character energy 💅 It's titled Architecture of the Whois++ Index Service, and it was published in February 1996 by the Network Working Group. The authors are C. Weider (Bunyip), J. Fullton (CNIDR), and S. Spero (EIT). It's on the Standards Track, and the title block doesn't list anything it obsoletes or updates 🌸
Status of this Memo
This document specifies an Internet standards track protocol for the Internet community, and it requests discussion and suggestions for improvements 💜 For the standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited ✨
Abstract
The authors describe an architecture for indexing in distributed databases, and apply this to the WHOIS++ protocol 🎀
1. Purpose ✨
Okay so the WHOIS++ directory service [Deutsch, et al, 1995] is meant to be simple and extensible 🌈 It's built on a template-based information model and a flexible query language, which is so clean. This document describes a general architecture for indexing distributed databases, and then applies that architecture to link lots of these WHOIS++ servers together into one distributed, searchable wide area directory service 🧚
2. Scope 🦋
This doc details a distributed, easy-to-maintain architecture that gives you a unified index over a huge number of distributed WHOIS++ servers 💖 The architecture isn't locked to WHOIS++, though. Other systems can use it too, to build a distributed directory service that's also searchable 🫶
3. Motivation and Introduction 🌸
Honestly, with the massive amount of directory information out there on the Internet, a single centralized directory just isn't feasible 😭 If we want to distribute the directory service, the easiest way (not necessarily the best one, ngl) is to build a hierarchy of directory information collection agents. A query gets delivered to one agent in the tree, then gets handed up or down as needed, until it reaches the agent that actually holds the information that fills the query. People have tried this before, most notably in some implementations of the X.500 standard. But there are a number of major flaws in the way it was done, and this new Index Service is designed to fix them 💅
3.1. The search problem 🔍
One of the main assumptions behind recent distributed directory services is that every entry lives in some location in a hierarchical name space. That's ideal for reading an entry once you already know where it is, but it's not as good when you're searching for locations in the namespace of entries that match some set of criteria 🙈 If the only criteria you know about an entry are things that don't show up in the namespace, you're stuck doing a global query. Whenever we issue a global query (at the root of the namespace), or a query at the top of a given subtree, that query gets replicated to "all" subtrees of the starting point. Replicating the query to every subtree isn't necessarily a problem, since queries are cheap. But every server the query got replicated to has to process it, even if it has no entries matching the criteria, and that part of global query processing is quite expensive 💀 A poorly designed or thin namespace can cause the vast majority of queries to be replicated globally, and a very broad namespace can cause its own navigation problems. Because of all this, search has been turned off at high levels of the X.500 namespace 😭
3.2. The location problem 📍
With global search turned off, you have to know in advance how the name space is laid out so you can guide a query to the right location. The layout then becomes critical to a user's ability to find what they want, so there are endless battles over how to lay out the name space to serve a given set of users, plus enormous headaches whenever it becomes obvious that the current namespace doesn't suit current usage and has to change (which recently happened in X.500). And if you do impose multiple hierarchies on the entries through the namespace, the mechanisms to maintain those multiple hierarchies don't exist yet in X.500, so it's possible to move entries out from under their pointers 😤 On top of that, there's still no agreement on how the X.500 namespace should look, even for the White Pages types of information currently installed in the X.500 pilot project.
3.3. The Yellow Pages problem 📒
Current implementations of this hierarchical architecture also aren't great at the Yellow Pages problem. That's the problem of easily and flexibly building special-purpose directories (say, for molecular biologists) and automatically maintaining them once they exist 🧬 The attributes for a new directory have to be built into the namespace, because that's the only way to group related entries somewhere they can be found without a global search. There's also a classification problem: how do you specify the right categories so that people other than the directory's creator can find the correct subtree? And then there's the problem of actually getting the data into the subtree. If you have to traverse the hierarchy to find the data, you end up searching globally for the right entries anyway 🙃
3.4. Solutions ✨
The problems covered in this section can be tackled with a combination of two new techniques: directory meshes and forward knowledge 💜
4. Directory meshes and forward knowledge 🕸️
Let's hold off for a sec on the actual architecture behind our solution and focus on a high-level view of what our conceptual approach solves 💖 First off, even though every WHOIS++ entry does have a unique identifier (it lives in a specific location in the namespace), the navigation algorithms for reaching a specific entry don't necessarily depend on the identifier the entry was assigned. The Index Service gets around the namespace and hierarchy problems by building a directory mesh on top of the entries. Each layer of the mesh has a set of "forward knowledge" about what the servers at the next lower layer of the mesh contain. So when a server in a given layer gets a query, it can prune the search tree and hand the query off only to the lower-level servers that said they might be able to answer it 🔥 That's how search becomes feasible at all levels of the mesh. In the current version of this architecture, we've picked a certain set of information to hand up the mesh as forward knowledge. This may or may not be exactly the set of information needed to build a truly searchable directory, but the protocol itself doesn't restrict the types of information that can be passed around 🌈
The protocols designed to maintain the forward knowledge will also work perfectly well for replicating servers for redundancy and robustness 🫶 In that case, the forward knowledge handed around is the entire database of entries held by the replicated server.
Another perk of the mesh of index servers is that, because the entry identification scheme is decoupled from the navigation service, multiple hierarchies can be built and easily maintained on top of the existing data 🦋 Also, the user doesn't need to know in advance where in the mesh the entry lives. 🧚
And the Yellow Pages problem finally becomes tractable! Index servers can pick and choose between the information a given server offers, and because the architecture allows automatic polling of data, special purpose directories become easy to construct and to maintain 🎀
5. Components of the Index Service 🧩
5.1. WHOIS++ servers 🌸
The whois++ service is described in [Deutsch, et al, 1995]. That service only specifies the query language, the information model, and the server responses, so whois++ services can be provided by a wide variety of databases and directory services 💅 To participate in the Index Service, though, the underlying database must also be able to generate a 'centroid', or some other type of forward knowledge, for the data it serves ✨
5.2. Centroids as forward knowledge 💎
The centroid of a server is made up of a list of the templates and attributes that server uses, plus a word list for each attribute. The word list for a given attribute contains one occurrence of every word that appears at least once in that attribute in some record in that server's data, and nothing else 🫶
A word is any token delimited by blank spaces, newlines, or the '@' character, in the value of an attribute 🔤
Here's an example. Say a whois++ server contains exactly three records, like this:
Record 1 Record 2
Template: User Template: User
First Name: John First Name: Joe
Last Name: Smith Last Name: Smith
Favourite Drink: Labatt Beer Favourite Drink: Molson Beer
Record 3
Template: Domain
Domain Name: foo.edu
Contact Name: Mike FoobarThen the centroid for this server would be:
Template: User
First Name: Joe
John
Last Name: Smith
Favourite Drink: Beer
Labatt
Molson
Template: Domain
Domain Name: foo.edu
Contact Name: Mike
FoobarThis is the information that gets handed up the tree to provide forward knowledge 🌈 Like we said above, it may not turn out to be the ideal solution for forward knowledge, and we suspect a number of different sets of forward knowledge may get used in the Index Service. Still, the directory architecture is in a very real sense independent of which types of forward knowledge get handed around, so it's totally possible to build a unified directory that uses many types 💖
5.3. Index servers and Index server Architecture 🏗️
A whois++ index server collects and collates the centroids (or other forward knowledge) of either a number of whois++ servers or a number of other index servers 🧚 An index server must be able to generate a centroid for the information it contains. On top of that, an index server can index any other server it wishes, which means one base level server (or index server) can participate in many hierarchies in the directory mesh 👑
5.3.1. Queries to index servers ✨
Okay so when a query lands on an index server 💖, it takes the query in standard whois++ format and searches through its collections of centroids and other forward information 🧠. It figures out which servers might hold records that fill that query 🔍. Then it doesn't answer the question itself. It tells the user's client which servers to go ask next 👉, and that's referral, same idea as the X.500 model 🦋. An index server can also hold primary data of its own 🎀, so it can act as both an index server and a base level server at once 👑. When it does both, its reply can be a mix of actual records and referral pointers 🌈.
5.3.2. Index server distribution model and centroid propogation 🌸
The diagram below shows how a mesh of index servers might be built for a set of whois++ servers. Even though it looks like a neat little tree 🌳, the protocols let server A get indexed by both server D and by server H (for example) 🤯.
whois++ index index
servers servers servers
for for
whois++ lower-level
servers index servers
_______
| |
| A |__
|_______| \ _______
\----------| |
_______ | D |__ ______
| | /----------|_______| \ | |
| B |__/ \----------| |
|_______| | F |
/----------|______|
/
_______ _______ /
| | | |-
| C |--------------| E |
|_______| |_______|-
\
\
_______ \ ______
| | \----------| |
| G |--------------------------------------| H |
|_______| |______|
Figure 1: Sample layout of the Index Service meshSo in the part of the index tree above 🫶, whois++ servers A and B hand their centroids up to index server D, whois++ server C hands its centroid up to index server E, and index servers D and E hand their centroids up to index server F. Servers E and G also hand their centroids up to H.
How many levels of index servers you get, and how many sit at each level, depends on how many whois++ servers are deployed and on the response time of each layer of the server tree ⏱️. These numbers will have to be determined in the field. 🧪
5.3.3. Centroid propogation and changes to centroids 🦋
Centroid propogation starts with an authenticated POLL command (sec. 5.2) 🔐. The POLL command lets the poller ask for the centroid of any or all templates and attributes that the polled server holds. Once the polled server has authenticated the poller, it works out which of the requested centroids the poller is allowed to request, and then sends back a CENTROID-CHANGES report (sec. 5.3) with the data 📦. When the poller gets that CENTROID-CHANGES report, it can authenticate the pollee and decide whether to add those centroid changes to its own data ✅.
Also, if a pollee knows which pollers are holding its centroids, it can tell them that its centroid changed by issuing a DATA-CHANGED command 📣. The poller can then decide if and when to send a new POLL request to grab the updated info 🏃♀️. The DATA-CHANGED command is included in this protocol so that critical information can get 'interactive' updating 💜.
5.3.4. Centroid propogation and mesh traversal 🗺️
When an index server sends a POLL request, it can tell the polled server what relationship it has to that server 🤝. That info can help everyone move around the directory mesh. The current proposal specifies two fields for sending the relationship information, though richer relationship info is expected to be shared in future revisions of this protocol 🔮.
The first is the Hierarchy field, and it can be one of three values 🏗️:
'topology'means the indexing server sits higher up in the network topology (e.g. it indexes the whole regional ISP) 🌐.'geographical'means the polling server covers a geographical area that subsumes the pollee 📍.'administrative'means the indexing server covers an administrative domain that subsumes the pollee 🏛️.
The second is the Description field, which holds the DESCRIBE record of the polling server 📝. This gives users richer metainformation about the directory mesh, so they can expand their queries more effectively 🚀.
5.3.5. Query handling and passing algorithms 🧩
When an index server gets a query, it searches its collection of centroids and figures out which servers hold records that might fill it 🔍. As whois++ gets widely deployed, some index servers may specialize in indexing certain whois++ templates, or even certain fields inside those templates 🎯. If an index server finds a match with the query "for those template fields and attributes the server indexes", that counts as a match for the purpose of forwarding the query 💌.
5.3.5.1. Query referral 👉
Query referral is the process of telling a client which servers to contact next to resolve a query 📬. The syntax for notifying a client is outlined in section 5.5 📖.
5.3.6 Loop control 🔁
Okay this one's important 😭. There are no a priori restrictions on which servers may poll which other servers, and a given server can be part of many sub-meshes. So mechanisms have to be installed to detect cycles in the polling relationships 🌀. The current protocol does this by putting a hop-count on polling relationships 🔢. Each time a polled server generates forward information, it tells the polling server its current hopcount. That hopcount is the maximum of the hopcounts of all the servers it polls, plus 1 ➕. A base level server (one that polls no other servers) has a hopcount of 0 🐣. When a server decides to poll a new server and its hopcount goes up, then it has to inform all the other servers that poll it about its new hopcount 📢. A maximum hopcount (8 in the current version) helps the servers detect polling loops 🛑.
A second approach to loop detection is to do all the work in the client 🧑💻. The client figures out which new referrals already showed up in the referral list, and then just keeps iterating the referral process until there are no new servers left to ask 🔄. An algorithm that does this in WHOIS++ is detailed in [Faltstrom 95] 📚.
6. Syntax for operations of the Index Service: 💅
Each protocol component has its syntax listed below 📋. Each section also lists which of its attributes are required and which are optional for that component 👀. All timestamps have to be in the format YYYYMMDDHHMM and in GMT ⏰.
6.1. Data changed syntax 📣
Here's what the data changed template looks like 💕:
# DATA-CHANGED
Version-number: // version number of index service software, used to
// insure compatibility. Current value is 1.0
Time-of-latest-centroid-change: // time stamp of latest centroid
// change, GMT
Time-of-message-generation: // time when this message was generated,
// GMT
Server-handle: // IANA unique identifier for this server
Host-Name: // Host name of this server (current name)
Host-Port: // Port number of this server (current port)
Best-time-to-poll: // For heavily used servers, this will identify
// when the server is likely to be lightly
// loaded so that response to the poll will be
//speedy, GMT
Authentication-type: // Type of authentication used by server, or NONE
Authentication-data: // data for authentication
# END // This line must be used to terminate the data changed
// messageRequired/optional table 📊
Version-Number REQUIRED
Time-of-latest-centroid-change REQUIRED
Time-of-message-generation REQUIRED
Server-handle REQUIRED
Host-Name REQUIRED
Host-Port REQUIRED
Best-time-to-poll OPTIONAL
Authentication-type OPTIONAL
Authentication-data OPTIONAL6.2. Polling syntax 🎀
# POLL:
Version-number: // version number of poller's index software, used to
// insure compatibility
Type-of-poll: // type of forward data requested. CENTROID or QUERY
// are the only one currently defined
Poll-scope: // Selects bounds within which data will be returned.
// See note.
Start-time: // give me all the centroid changes starting at this
// time, GMT
End-time: // ending at this time, GMT
Template: // a standard whois++ template name, or the keyword ALL,
// for a full update.
Field: // used to limit centroid update information to specific
// fields, is either a specific field name, a list of field
// names, or the keyword ALL
Server-handle: // IANA unique identifier for the polling server.
// this handle may optionally be cached by the polled
// server to announce future changes
Host-Name: // Host name of the polling server.
Host-Port: // Port number of the polling server.
Hierarchy: // This field indicates the relationship which the poller
// bears to the pollee. Typical values might include
// 'Topology', 'Geographical", or "Administrative"
Description: // This field contains the DESCRIBE record of the
// polling server
Authentication-type: // Type of authentication used by poller, or NONE
Authentication-data: // Data for authentication
# END // This line must by used to terminate the poll message📝 Note: For poll type CENTROID, the allowable values for Poll Scope are FULL and RELATIVE. Support of the FULL value is required, since it gives a complete listing of the centroid or other forward information 💎. RELATIVE means these are the relative changes in the centroid since the last report to the polling server 🔄.
For poll type QUERY, the allowable values for Poll Scope are a blank line, which means all records are to be returned, or a valid WHOIS++ query, which means just the records that satisfy that query are to be returned 🎯.
🚨 N.B. Security considerations may require additional authentication for a successful response to the Blank Line Poll Scope. This value has been included for server replication. Don't skip this one, bestie 💀.
A polling server might want to index different types of information than the polled server has collected 🧐. The POLLED-FOR command tells you which servers the polled server has contacted 📞.
Required/Optional Table
Okay so here's the rest of the POLL table, which is literally the sweetest little list of rules 🎀
Version-Number REQUIRED, value is 1.0
Type-Of-Poll REQUIRED, values CENTROID and QUERY are required
Poll-scope REQUIRED If Type-of-poll is CENTROID, FULL is required,
RELATIVE is optional
If Type-of-poll is QUERY, Blank line is
required, and WHOIS++-type queries are
required
Start-time OPTIONAL
End-Time OPTIONAL
Template REQUIRED
Field REQUIRED
Server-handle REQUIRED
Host-Name REQUIRED
Host-Port REQUIRED
Hierarchy OPTIONAL
Description OPTIONAL
Authentication-Type: OPTIONAL
Authentication-data: OPTIONALHere's a POLL command example, and it's giving clean 🧚
# POLL:
Version-number: 1.0
Type-of-poll: CENTROID
Poll-scope: FULL
Start-time: 199501281030+0100
Template: ALL
Field: ALL
Server-handle: BUNYIP01
Host-Name: services.bunyip.com
Host-Port: 7070
Hierarchy: Geographical
# END6.3. Centroid change report ✨
Okay so the centroid change report has nested blocks that can show up more than once 🌸 Each of those repeating blocks gets wrapped in curly braces, '{' and '}', in this paper. The braces are NOT part of the syntax, they're only there so you can tell what's what 🫶
A Data: item is either a list of words, one word per line, or the keyword 💖
ANYThe keyword ANY, when it's the only item in a Data: list, means that any value for that field gets treated as a hit by the indexing server 👑
The Any-field: thing needs more room than the syntax body can give it, so here's the tea 🍵 It can be True or False. If it's True, the pollee is saying there are fields in this template that aren't being exported to the polling server, but it still wants to treat them as a hit. So when the polling server gets a query with a term asking for a field that isn't in this list for this template, it'll count that term as a 'hit'. If it's False, the pollee is saying there are no other fields for this template that should count as a hit. This field is required because the WHOIS++ query model needs the results of each search term to be 'and'ed together. This field lets polled servers only export data for non-sensitive fields, and still get referrals for queries that have sensitive terms 🦋
IMPORTANT: The data listed in the centroid must be in the ISO-8859-1 character set in this version of the indexing protocol. Use of any other character set is a violation of the protocol. Note that the base-level server is also specified to use ISO-8859-1 [Deutsch, et al, 1995].
Here's the syntax for the whole report 💅
# CENTROID-CHANGES
Version-number: // version number of pollee's index software, used to
// insure compatibility
Start-time: // change list starting time, GMT
End-time: // change list ending time, GMT
Server-handle: // IANA unique identifier of the responding server
Case-sensitive: // states whether data is case sensitive or case
// insensitive. values are TRUE or FALSE
Authentication-type: // Type of authentication used by pollee, or NONE
Authentication-data: // Data for authentication
Compression-type: // Type of compression used on the data, or NONE
Size-of-compressed-data: // size of compressed data if compression
// is used
Operation: // One of 3 keywords: ADD, DELETE, FULL
// ADD - add these entries to the centroid for this server
// DELETE - delete these entries from the centroid of this
// server
// FULL - the full centroid as of end-time follows
{ // The multiply occurring template block starts here
# BEGIN TEMPLATE
Template: // a standard whois++ template name
Any-field: // TRUE or FALSE. See beginning of 6.3 for explanation.
{ // the template contains multiple field blocks
# BEGIN FIELD
Field: // a field name within that template
Data: // Either the keyword *ANY*, or
// the word list itself, one per line, cr/lf terminated,
// each line starting with a dash character ('-').
# END FIELD
} // the field ends with END FIELD
# END TEMPLATE
} // the template block ends with END TEMPLATE
# END CENTROID-CHANGES // This line must be used to terminate the
// centroid change reportFor each template, all fields must be listed, or queries won't get referred correctly 💀
Required/Optional table
Version-number REQUIRED, value is 1.0
Start-time REQUIRED (even if the centroid type is FULL)
End-time REQUIRED (even if the centroid type is FULL)
Server-handle REQUIRED
Case-Sensitive OPTIONAL
Authentication-Type OPTIONAL
Authentication-Data OPTIONAL
Compression-type OPTIONAL
Size-of-compressed-data OPTIONAL (even if compression is used)
Operation OPTIONAL, if used, upport for all three values is required
Tokenization-type OPTIONAL
#BEGIN TEMPLATE REQUIRED
Template REQUIRED
Any-field REQUIRED
#BEGIN FIELD REQUIRED
Field REQUIRED
Data REQUIRED
#END FIELD REQUIRED
#END TEMPLATE REQUIRED
#END CENTROID-CHANGES REQUIREDExample time, and this one's so cute 🌈
# CENTROID-CHANGES
Version-number: 1.0
Start-time: 197001010000
End-time: 199503012336
Server-handle: BUNYIP01
# BEGIN TEMPLATE
Template: USER
Any-field: TRUE
# BEGIN FIELD
Field: Name
Data: Patrik
-Faltstrom
-Malin
-Linnerborg
#END FIELD
#BEGIN FIELD
Field: Email
Data: paf@bunyip.com
-malin.linnerborg@paf.se
# END FIELD
# END TEMPLATE
# END CENTROID-CHANGES6.4 QUERY and POLLEES responses 🍓
A QUERY command gets its response in WHOIS++ format, no drama 🧚
6.5. Query referral 🦋
When referrals show up in the body of a response to a query, each referral gets its own SERVER-TO-ASK block, like this 💖
# SERVER-TO-ASK
Version-number: // version number of index software, used to insure
// compatibility
Body-of-Query: // the original query goes here
Server-Handle: // WHOIS++ handle of the referred server
Host-Name: // DNS name or IP address of the referred server
Port-Number: // Port number to which to connect, if different from the
// WHOIS++ port number
# ENDRequired/Optional table
Version-number REQUIRED, value should be 1.0
Body-of-query OPTIONAL
Server-Handle REQUIRED
Host-Name REQUIRED
Port-Number OPTIONAL, must be used if different from port 63Example 🫶
# SERVER-TO-ASK
Version-Number: 1.0
Server-Handle: SUNETSE01
Host-Name: sunic.sunet.se
Port-Number: 63
# END7: Reply Codes 💅
Besides the reply codes listed in [Deutsch 95] for the basic WHOIS++ client/server interaction, version 1.0 of this protocol uses the following reply codes 🔥
113 Requested method not available Unable to provide a requested
compression method. Contacted
server will send requested
data in different format.
227 Update request acknowledged A DATA-CHANGED transmission
has been accepted and logged
for further action.
503 Required attribute missing A REQUIRED attribute is
missing in an interaction.
504 Desired server unreachable The desired server is
unreachable.
505 Desired server unavailable The desired server fails to
respond to requests, but host
is still reachable.8. References 📚
[Deutsch 95] Deutsch, et al., "Architecture of the WHOIS++ service", RFC 1835, August 1995.
[Faltstrom 95] Faltstrom, P., et al., "How to Interact with a WHOIS++ Mesh, RFC 1914, February 1996.
9. Security Considerations 🔒
Security issues are not discussed in this memo. 🙈
10. Authors' Addresses 🫶
Chris Weider
Bunyip Information Systems, Inc.
310 St. Catherine St. West
Montreal, PQ H2X 2A1
CANADA
Phone: +1-514-875-8611
Fax: +1-514-875-6134
EMail: clw@bunyip.com
Jim Fullton
MCNC Center for Communications
Post Office Box 12889
3021 Cornwallis Road
Research Triangle Park
North Carolina 27709-2889
Phone: 410-795-5422
Fax: 410-795-5422
EMail: fullton@cnidr.org
Simon Spero
EMail: ses@eit.com📦 where this page comes from
- ietf: rfc1913 1996-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms