RFC 1691
The Document Architecture for the Cornell Digital Library
The Document Architecture for the Cornell Digital Library ✨
Okay bestie, this is RFC 1691 📚 from the Network Working Group, and it's giving digital library architecture 💅. It's Request for Comments: 1691, written by W. Turner of LTD 🎀. The category is Informational, so it's sharing info rather than setting a standard, and it's dated August 1994. The title block doesn't list anything it Obsoletes or Updates, so there's nothing to track there 🦋.
Status of this Memo 📜
This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited, so you can share it freely 🌸.
Abstract 🌸
So this memo lays out how a huge, organized digital library stores and retrieves digital versions of books, journals, photographic images, etc. ✨ Two features make it iconic: it can generate reference documents, and it can create multiple views of a single document 💅.
Introduction 🧚
Okay so, back in 1989, Cornell University and Xerox Corporation teamed up to see how digital tech could help preserve library material 💻. They got help from the Commission on Preservation and Access, and later Sun Microsystems 🫶. Xerox built the College Library Access and Storage System (CLASS), and Cornell built software for network access to the CLASS Digital Library 🌈.
The Xerox and Cornell University Library staff worked super closely to define requirements for storing both low- and high-resolution versions of images 🖼️. The low-res ones were for browsing over the network, and the high-res ones were for printing 🖨️. They also put a lot of work into documents with internal structures you could navigate. Xerox made the software to create and store documents, while Cornell made complementary software so library users could browse the documents and request printed copies over the network 🔥.
Cornell then came up with a document architecture built on everything they learned from CLASS, and they're keeping their digital library materials in that form 👑.
Document Architecture Overview 📖
Just like a regular library holds books instead of loose pages, the electronic library has to hold documents instead of images 📚. During scanning, images get automatically linked into documents. The system creates document structure files that put the image files in order, the same way a book's binding puts pages in order 🧵.
So a digital book right now has two parts: the individual pages, stored as discrete bit map image files, and the document structure files that "bind" those image files into one document 💜. Each digital document also gets a database entry so you can search by author and title (aka bibliographic info) 🔍.
Beyond page order, a physical book's layout tells readers stuff too. The title page and publication info come first, the table of contents usually comes before the text, the text is split into sections or chapters, and any index comes after the text 🤓. Readers look at these bits when browsing the shelves to decide whether to read the book 📕.
The document structure gives direct access to the parts of an electronic document. It keeps the info that would otherwise be lost when the book gets taken apart for scanning ✨.
Document Architecture Requirements 📝
Here are the requirements they set at the start for the Cornell Digital Library Architecture 💅:
- The architecture MUST be open, meaning published and freely available. 🔓
- The architecture SHOULD be as simple as possible, to make product development easier 🛠️.
- The architecture SHOULD assume data storage in UNIX file systems 🐧.
- The architecture SHOULD allow for standard data usage, like via FTP and Gopher servers. That means all the pages of a document MUST exist in a single directory, and the naming convention used MUST order them in the standard collating sequence, like
0001.TIF,0002.TIF, ...,0411.TIF. Heads up: a series like1.TIF,2.TIF, ...,10.TIFwould get ordered1.TIF,10.TIF,2.TIF, ..., which is not acceptable 🙈. - The architecture SHOULD provide for storing the same information in different formats, like when a page of a document is available at several different resolutions 🎞️.
- Low-resolution "thumbnail" images of each page MUST be stored, so browsing and sharing data is easy 🍓.
- The architecture MUST support distributing files so that similar files can be stored together, which helps with storage use and performance 🌈.
- The architecture MUST support documents made of references to all or part of other documents 🧩.
- The architecture MUST support document components stored on separate servers spread across the network 🌐.
- The architecture MUST support not just a hierarchical structure for each document, but also the ability to define multiple views of each document 🦋.
- The architecture SHOULD accept, rather than dictate, the directory structures documents are stored in. That way, documents created in other ways can join the Digital Library just by adding database info, instead of copying or moving files 🫶.
Document Architecture Description 🏛️
A digital library is made of a Digital Library Server, networked storage, and a referencing database 💖. One library can contain one or more collections, and each collection contains one or more documents.
The referencing database lets you search for documents by author, title, and document ID 🔍. In the current implementation, it's a relational SQL database, and each collection is represented by a table in that database 🗂️. They plan to migrate to Z39.50 database searching as the preferred method, since it's been established as the standard for library applications 👑.
Authorization will be mostly collection-based, but the design will allow authorization checks at any level, all the way down to the individual file 🔐. Notification only comes when the patron tries to open the document or access a particular component 🔔.
Each document has three components: the logical structure, the physical references, and the data files ✨.
The logical structure is a logical description of the document. Think of a document as a tree whose leaves are the data files (pages) 🌳. At minimum, every document has a logical structure that lists its pages and the order they appear in. Most documents have a fancier structure. The logical structure connects a document's logical layout to the physical references that make it up.
The physical references map the lowest levels of the document's logical structure (the leaves of the tree) to the files that hold the data 🔗. If a page has multiple representations, like images at various resolutions, those get linked together in the physical references file.
The data files hold the data that makes up a document. Any format can work: image files, ASCII text, PostScript, and so on 🖨️. There's one assumption, though: there's a one-to-one correspondence between data files for a given physical reference. So if a single page has several file types, those files SHOULD represent exactly the same information 🙈.
Physical References File 📂
The Physical References file is the part of the document that relates logical structures (the logical pieces of documents) to physical files 🧩. Document references, where a document is made of all or part of other documents that might live on different servers, are handled right here in the Physical References file 🌍.
A document can have multiple document objects, and each one holds one or more data objects. If a document contains actual physical data (like when it's created by scanning or importing images), a Master Document Object is created 📸. If a document pulls in components from other documents, a Reference Document Object is created for each of those other documents 📑. Document Objects get numbered with internal reference numbers, and those numbers show up in the matching Data Object lines 🔢.
Data Object lines include the Document Object number, the file reference number, and the file type 🏷️. The Document Object number points to a Document Object line, which is where you find the library name, collection name, and document ID 🔎. The tuple <libraryID>+<collectionID>+<documentID>+<filetype>+<file reference> is guaranteed to locate a file ✨. Each Data Object line refers to a single file, so if a page has multiple file types, that page gets multiple Data Object lines 🧾.
In the file, all the Document Object lines come before all the Data Object lines for a given document 📋. Document Object lines MAY be grouped together at the beginning of the file, or they MAY immediately precede the first Data Object line for their Document Object. Document Object lines appear in order by Document Object number. Data Object lines appear in order by sequence number, not by Document Object number 🧠.
The fields in the Physical References file are delimited by vertical bars 🚧.
Document Object Lines 🗃️
Here's the field breakdown for Document Object lines 💜:
Field Description Comments
----- ---------------------- ----------------------------
1 Document Object number 0 => Master Document Object
1-9 => Reference Document Object
2 Library name Server name
3 Collection name
4 Document ID 8-digit number
5 Author name
6 Volume
7 Title
8 EditionData Object Lines 🎞️
And here's the field breakdown for Data Object lines, which is where the file type codes live 🔥:
Field Description Comments
----- ---------------------- ----------------------------
1 Document Object number Corresponds to above
2 Sequence number
3 File reference Reference number used to locate
file in filing system
4 Physical reference number Equal to Logical Structure file
5 File type 1 = TIFF 600dpi
2 = TIFF thumbnail
3 = ASCII version of page
(i.e., OCR output)
4 = ASCII notes
5 = Other
6 = TIFF 300dpi
6 NoteLogical Structure File
okay so the Logical Structure file is the part of the document structure that gives you "views" of a doc and links images together logically so they make an actual document 🦋 it's basically an unloaded tree, and when someone "opens" a document, the file gets read and the tree gets rebuilt from scratch ✨ by convention, every Logical Structure file has one logical structure called "PAGES" that defines the document by listing the pages in the order they showed up in the original 📖
Document Structure lines
this table is the vibe check for every line 💅
Field Description Comments
----- ---------------------- ----------------------------
1 Parent structure number Structure is a child of...
2 Sequence number
3 Logical Structure name Label for this structure
4 Structure number Equal to Physical Reference file
5 Logical Children # of logical children of this
structureDocument Structure lines (continued)
ok the second half of the table picks up right where we left off 🎀
Field Description Comments
----- ---------------------- ----------------------------
6 Physical Children # of physical children of this
structure
7 References # of references to this
structure within this document
(for how many structures is this
a substructure)Logical Structure File Example
this example is literally giving main character energy 🌸 it walks through a whole tree, from the ROOT all the way down to the little leaves, so you can see how every line hooks up
|0|0|ROOT|0|4|0|0| Structure 0, ROOT, has 4 logical children
|0|1|PAGES|1|100|0|1| Str. 1, PAGES, has 100 logical children
|0|2|CONTENTS|2|22|0|1| Str. 2, CONTENTS, has 22 logical children
...has no physical children
...
|1|1|Production note|5|0|2|2| Str. 5 is child of structure 1
...has a label "Production note"
...has no logical children
...has 2 physical references
...is referenced twice in this document
|1|2||6|0|2|1| Str. 6 has no label
|1|3||7|0|2|1| Str. 7 has 2 physical references
|1|4||8|0|2|1| Str. 8 is referenced only here
|1|5||9|0|2|1| Str. 9 is 5th sequential child of PAGES
...
|1|99||103|0|2|2|
|1|100||104|0|2|2|
|2|1|Production note|105|1|0|1| Str. 105 is a child of str. 2
|2|2|Title page|106|1|0|1| Str. 106 has 1 logical child
|2|3|Table of contents|107|2|0|1|
|2|4|Chapter 1. From Arithmetic to Algebra|108|6|0|1|
|2|5|Chapter 2. The Making of Algebras|109|4|0|1|
|2|6|Chapter 3. Simultaneous Problems|110|4|0|1|
|2|7|Chapter 4. Partial Solutions...|111|3|0|1|
|2|8|Chapter 5. Mathematical Certainty...|112|3|0|1|
|2|9|Chapter 6. The First Hebrew Algebra|113|8|0|1|
|2|10|Chapter 7. How to Choose our Hypotheses|114|9|0|1|
|2|11|Chapter 8. The Limits of the Teachers Function|115|5|0|1|
|2|12|Chapter 9. The Use of Sewing Cards|116|4|0|1|
...
|2|20|Chapter 17. From Bondage to Freedom|124|5|0|1|
|2|21|Appendix|125|2|1|1|
|2|22|advertisements|126|4|1|2|
|105|1|Production note|5|0|2|2| Str. 5 is a child of str. 105
|106|1|Title page|11|0|2|2| 2nd reference to str. 11
|107|1|7|15|0|2|2|
|107|2|8|16|0|2|2|
...
|126|4||104|0|2|2|Implementation Details
so here's the deal 🔥 the tuple <library ID>+<collection ID>+<document ID>+<filetype>+<file reference> is guaranteed to locate a file. a file locator program does the translating between that tuple and the full path and file name in the underlying file system ✨ a library will always be hierarchical, like UNIX file systems, but the order of that hierarchy is flexible so people can optimize things 💖
every level of the hierarchy has an INFO file that describes the order of the levels below it. the file locator reads these files as it walks the directory structure when a library, collection, or document gets opened. here are two examples to make it click 🧚
Example 1. Hierarchy is LIBRARY, COLLECTION, DOCUMENT, FILETYPE.
/<library name>
LIBINFO.TXT Description of library
/<collection name>
COLINFO.TXT Description of collection
/<document ID>
DOCINFO.TXT Description of document
LOGSTR.000 Logical structure file
PHYSREF.000 Physical reference file
/<filetype1>
00001.TIF
00002.TIF
...
/<filetype2>
00001.TIF
00002.TIF
...Example 2. Hierarchy is LIBRARY, FILETYPE, COLLECTION, DOCUMENT.
/<library name>
LIBINFO.TXT Description of library
/<filetype1>
/<collection name>
COLINFO.TXT Description of collection
/<document ID>
DOCINFO.TXT Description of document
LOGSTR.000 Logical structure file
PHYSREF.000 Physical reference file
00001.TIF
00002.TIF
...
/<filetype2>
/<collection name>
COLINFO.TXT Description of collection
/<document ID>
DOCINFO.TXT Description of document
LOGSTR.000 Logical structure file
PHYSREF.000 Physical reference file
00001.TIF
00002.TIF
....this setup has some redundancy, but it lets complete copies of a collection be mounted on different file systems for performance reasons 🌈 the second scheme especially helps: all the low-resolution images can live on high-speed magnetic disk for fast access, while the high-resolution images sit on slower, cheaper storage. it also makes it easier to authorize access to low-resolution images from other software systems (FTP, Gopher) while still restricting access to the high-resolution ones 🫶
Security Considerations
😱 okay bestie, deep breath. this memo does not discuss security issues. at all. not one word. so if you're building anything on top of this, you're on your own for the security part, and you need to figure that out before you ship anything 💀
References
[1] Turner, W., "Cornell Digital Library Document Architecture, Version 1.1 - 3/22/94", Library Technology Department, Cornell University.
Author's Address
William Turner
Library Technology
502 Olin Library
Cornell University
Ithaca, NY 14853
Phone: 607-255-9098
Fax: 607-255-9346
EMail: wrt1@cornell.eduand that's a wrap on the whole doc 🌈✨💖 the legend William Turner, Library Technology, Cornell 👑
📦 where this page comes from
- ietf: rfc1691 1994-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms