RFC 1625

WAIS over Z39.50-1988 🎀

Okay bestie, this is RFC 1625 🦋 It's an Informational doc, dated June 1994, and it was written by a whole squad ✨

• M. St. Pierre (WAIS, Inc.)
• J. Fullton (CNIDR)
• K. Gamiel (CNIDR)
• J. Goldman (Thinking Machines Corp.)
• B. Kahle (WAIS, Inc.)
• J. Kunze (UC Berkeley)
• H. Morris (WAIS, Inc.)
• F. Schiettecatte (FS Consulting)

The title block doesn't list any RFCs it obsoletes or updates, so it's just doing its own thing 🌸

---

Status of this Memo 📜

This memo is giving information for the Internet community 🌈 It is not a standard, and it doesn't specify an Internet standard of any kind. Distribution is unlimited, so share it with everyone 💖

1. Introduction ✨

Let's talk about Wide Area Information Servers (WAIS), a network publishing system built to help people find information over a computer network 🔍 Its guiding principles are honestly iconic:

1. A wide-area networked-based information system for searching, browsing, and publishing.
2. Based on standards.
3. Easy to use.
4. Flexible and growth oriented.

A big group of developers, publishers, standards bodies, libraries, government agencies, schools, and users have been helping WAIS grow, and that's so wholesome 🫶

The WAIS software has four main components: the client, the server, the database, and the protocol 💻 The client is the user-interface program that sends requests for information to local or remote servers, and clients are available for most popular desktop environments. The server is the program that handles client requests, and it's available on a variety of UNIX platforms. It generally runs on a machine that holds one or more information sources, which are called WAIS databases. The protocol is Z39.50-1988, which connects WAIS clients and servers. It's based on the 1988 Version of the NISO Z39.50 Information Retrieval Service and Protocol Standard. The big goal is an open architecture of information clients and servers, all talking through a standard computer-to-computer protocol 👑

WAIS development began in October 1989, and the first Internet release came out in April 1991. From day one, WAIS committed to Z39.50-1988 as its information retrieval protocol between clients and servers. That implementation is still in use today by existing WAIS clients and servers, with over 50,000 users of Z39.50-1988 on the Internet 🥹

2. Purpose 🎯

The point of this memo is to start a discussion about a migration path for WAIS technology. The path goes from Z39.50-1988 Information Retrieval Service Definitions and Protocol Specification for Library Applications [1] to Z39.50-1992 [2], and then to Z39.50-1994 [3] 🦋

This memo isn't a detailed implementation spec, though. It's about the high-level design goals and functional assumptions behind the WAIS implementation of Z39.50-1988. How WAIS will use Z39.50-1992 and Z39.50-1994 will be the subject of future RFCs, so stay tuned 🍓

3. Historical Design Goals of WAIS 🕰️

To understand the original WAIS implementation and how it used Z39.50-1988, we need to look at its historical design goals. Each goal comes with a quick rundown of the assumptions used to meet it 💅

1. Provide users access to bibliographic and non-bibliographic information, including full-text and images.

Z39.50-1988 grew out of the bibliographic community, so extra assumptions were needed to serve non-bibliographic information. They were also needed to serve documents that exist in multiple formats, like rtf, postscript, gif, and so on 📚

2. Keep the client/server interface simple and independent of changes in the functionality of the server.

The text string the user typed was sent to the server without being parsed into a Type-1 RPN (reverse-polish notation) query, which is what's common for bibliographic applications. Instead, WAIS defined a brand-new Type-3 query that contains the text string. Because of that, neither the client nor the user had to know which Z39.50 Attributes the server supported, which is not true of many other Z39.50 implementations. The client software also didn't need changes to keep up with the server's evolving functionality. That's main character energy 👑

3. Provide relevance feedback capability.

Relevance feedback means you select a document, or a portion of one, and then find a set of documents similar to your selection. WAIS included the documents used in relevance feedback as part of the Type-3 query 🔁

4. Permit the server to operate in a stateless manner.

WAIS servers were designed to be "stateless," which means they didn't store search result sets. In Z39.50 terms, the server used its right to delete a result set on its own as soon as it sent the search response. So the Present Facility of Z39.50 wasn't used, and retrievals were done with the Search Facility instead. Relaxing this constraint in future implementations may turn out to be the most prudent path 🤔

5. Provide the ability for a client to retrieve documents in pieces.

Retrieving part of a document could be done several ways with Z39.50-1988, so specific assumptions were made to build this. Accessing a portion of a document was needed for both retrieval and relevance feedback 🧩

6. Run over TCP.

Z39.50-1988 was designed to run in the application layer, using the presentation services provided by the Open Systems Interconnection (OSI) Reference Model. But TCP/IP and the Internet got super popular, so WAIS was designed to run over TCP instead. Use of Z39.50 over TCP is described in [4] 🌐

4. WAIS Implementation of Z39.50-1988 🛠️

WAIS developers worked with the Z39.50 Implementors Group (ZIG). Together they used a recommended subset of Z39.50-1988, plus specific assumptions, to meet WAIS's requirements. Over time, many of those requirements ended up in later versions of Z39.50. As new requirements show up, WAIS will document any additional assumptions and work with the ZIG on extensions 🫶

WAIS supported the Init and Search Facilities of Z39.50-1988. Both search and retrieval were implemented with the Search Facility, as described in this section ✨

Search starts when the client sends a Search Request APDU (Application Protocol Data Unit) using a Type-3 query. The query has two main fields:

1. The "seed words," or text, typed by the user.
2. A list of document objects. A document object is a full document, or a portion of one, used in relevance feedback. Each document object contains a document identifier (Doc-ID) [5], type, chunk-code, and start and end locations. The Doc-ID and type tell you the location and format of the document, respectively. The chuck-code determines the unit of measure for the start and end locations. Examples of chunk-codes include byte, line, paragraph, and full document. If the chunk code is a full document, the start and end locations get ignored 🙈

A Search Response APDU sent back by the server contains a relevance-ranked list of records, called WAIS Citations. A WAIS Citation refers to a document on the server. Each WAIS Citation has these fields:

1. Headline - a set of words that convey the main idea of the document.
2. Rank - the numerical score of the document based on its relevance to the query, normalized to a top score of 1000.
3. List of available formats - e.g. text, postscript, tiff, etc.
4. Doc-ID - the location of the document.
5. Length - the length of the document in bytes.

The number of WAIS Citations returned was limited by the preferred message size negotiated during the Init 📏

Retrieval of a document starts when the client sends a Search Request APDU using a Type-1 query. The query can have up to four terms 💖

1. Term: Doc-ID
   Use Attribute: system-control-number      code = "un"
   Relation Attribute: equal                 code = "re"

2. Term: the requested document format
   Use Attribute: data-type                  code = "wt"
   Relation Attribute: equal                 code = "re"
3. Term: the start location
   Use Attribute: paragraph, line, byte      code = "wp", "wl",
                                                    "wb"
   Relation Attribute: greater-than-or-equal code = "ro"
4. Term: the end location
   Use Attribute: paragraph, line, byte      code = "wp", "wl",
                                                    "wb"
   Relation Attribute: less-than             code = "rl"

Full-text and images were often bigger than the client's receive buffer, so clients were designed to optionally retrieve documents in chunks. They do this by specifying the start and end positions of the chunk in the query. Here's an example of a fully-specified retrieval query 🍰

query = ( ( use = "un", relation = "re", term = <Doc-ID> )
          AND
          ( use = "wt", relation = "re", term = postscript )
          AND
          ( use = "wb", relation = "ro", term = 0 )
          AND
          ( use = "wb", relation = "ro", term = 2000 )
         )

A retrieval response is sent by the server as a Search Response APDU. In this case, a single record matching the requested document, or the portion of it, comes back in the specified format 📦

---

5. Security Considerations 🔒

Security issues are not discussed in this memo. Yeah bestie, that's a huge gap 😭 Please don't assume this document covers security, because it really doesn't.

okay bestie, last stretch!! 🎀 Here's the paper trail 📚 and the squad who wrote this thing 🫶

6. References ✨

These are the sources the authors leaned on, copied exactly so nothing gets lost in translation 💅

   [1] National Information Standards Organization (NISO).  American
       National Standard Z39.50, Information Retrieval Service
       Definition and Protocol Specifications for Library Applications,
       New Brunswick, NJ, Transaction Publishers; 1988.

   [2] ANSI/NISO Z30.50-1992 (version 2) Information Retrieval Service
       and Protocol: American National Standard, Information Retrieval
       Application Service Definition and Protocol Specification for
       Open Systems Interconnection, 1992.

   [3] Z39.50 Version 3: Draft 8", October 1993.  Maintenance Agency
       Reference: Z39.50MA-034.

   [4] Lynch, C., "Using the Z39.50 Information Retrieval Protocol
       in the Internet Environment", Work in Progress, November 1993.

   [5] "Document Identifiers, or International Standard Book Numbers
       for the Electronic Age", Brewster Kahle, Thinking Machines
       Corporation, see URL=<ftp://wais.com/pub/protocol/doc-ids.txt>,
       September 1991.

7. Authors' Addresses 💌

And now the people behind it 🧚 these are the contact details exactly as they were published, so if you ever need to reach out, here you go 🌸

   Margaret St. Pierre
   WAIS Incorporated
   1040 Noel Drive
   Menlo Park, California  94025

   Phone: (415) 327-WAIS
   Fax:   (415) 327-6513
   EMail: saint@wais.com


   Jim Fullton
   Clearinghouse for Networked Information
   Discovery & Retrieval
   3021 Cornwallis Road
   Research Triangle Park, North Carolina  27709-2889

   Phone: (919)-248-9247
   Fax:   (919)-248-1101
   EMail: jim.fullton@cnidr.org


   Kevin Gamiel
   Clearinghouse for Networked Information
   Discovery & Retrieval
   3021 Cornwallis Road
   Research Triangle Park, North Carolina  27709-2889

   Phone: (919)-248-9247
   Fax:   (919)-248-1101
   EMail: kevin.gamiel@cnidr.org

   Jonathan Goldman
   Thinking Machines Corporation
   1010 El Camino Real, Suite 310
   Menlo Park, California  94025

   Phone: (415) 329-9300 x229
   Fax:   (415) 329-9329
   EMail: jonathan@think.com


   Brewster Kahle
   WAIS Incorporated
   1040 Noel Drive
   Menlo Park, California  94025

   Phone: (415) 327-WAIS
   Fax:   (415) 327-6513
   EMail: brewster@wais.com


   John A. Kunze
   UC Berkeley
   289 Evans Hall
   Berkeley, California  94720

   Phone: (510) 642-1530
   Fax: (510) 643-5385
   EMail: jak@violet.berkeley.edu


   Harry Morris
   WAIS Incorporated
   1040 Noel Drive
   Menlo Park, California  94025

   Phone: (415) 327-WAIS
   Fax:   (415) 327-6513
   EMail: morris@wais.com


   Francois Schiettecatte
   FS Consulting
   435 Highland Avenue
   Rochester, New York  14620

   Phone: (716) 256-2850
   EMail: francois@wais.com

that's the whole RFC, bestie!! 🌈💖