RFC 219

User's View of the Datacomputer 🎀

okay bestie, so this is RFC 219 and it's giving 1971 memo energy 💅 It was written by R. Winter (R.A. Winter) at CCA, dated 3 September 1971, with NIC number 7549. It's a MEMORANDUM addressed TO the Datacomputer Design File and FROM R.A. Winter. The category field is blank, so no category is listed. It updates nothing and obsoletes nothing, so it's totally standalone ✨

---

Introduction 🦋

so the datacomputer is a specialized node on the ARPA network that's dedicated to managing one big shared database 🗄️ and when we say big, we mean several trillion bits of data, with at least one trillion of them on-line. sometimes "shared" means some files are shared by pretty much everyone on the ARPA network 🌈

the name datacomputer comes from the idea that the system is all about data handling 💖 the processor can do general computation, but it's not going to be used for that purpose. the processor, like the mass storage device, is just one component of an integrated system, and to the user it looks like a black box 🖤

there's one language for talking to the black box: data language 🗣️ and that language defines everything it can do. literally everything 💀

this memo is about the first of a series of service offerings. we use the term access method to mean a structure together with the operations on it. being too modest to call the first one AM-1 (Access Method-1), we named it DCAM-1 (Datacomputer Access Method-d) 🙈 we expect later DCAMs to generalize DCAM-1. if the need comes up, we'll design parallel services, and all services will use the same data language 🫶

System Overview 🌸

the users of the datacomputer are programs running on other computers, pulling data out of the data base and putting data into it 🖥️ these programs have wildly different environments, capabilities, and applications, but a chief design goal is to let them all share the data. that's the whole vibe 💜

users also differ in how they connect. remote users get in over a narrow link to the datacomputer's low-speed port 🐢 local users plug into the high-speed port through a link that's 80 times wider 🚀

through its ports, the datacomputer accepts two kinds of input: data, and requests for services. data gets output through the ports as requested ✨

in the data base, descriptions are stored separately from the data, and data elements are named, typed and ordered according to those descriptions 📋 a measure of structure independence comes from writing access requests in terms of the symbolic names of items in the data description. it's lowkey genius 🧠

directories are maintained by the system. it uses a hierarchical naming scheme, and access controls for privacy and data integrity are provided 🔐

redundant copies of data and/or journals of changes are kept by the system and used to recover under system control in case of system error 🛟 these facilities can be operated under user control if there's an external error. okay this part is giving responsibility 😭

the datacomputer's only interface with the outside world is its ports, so it sees the whole universe as a group of data streams 🌊 specifically, these are record streams, if you look at every transaction (in the data transfer protocol sense) as a record. each record stream has a data description attached, which lets the datacomputer parse the records into named, typed elements 🎀

so all data elements, both stream elements and data base elements, are named and fully described 💅 data type conversion happens automatically, as a function of old and new data types and any optional information the user supplies. reconfiguring above the element level is just arranging elements in records, and a full set of capabilities is provided for that 🧩 in general, the using program only cares about how the stream records are configured, since those make up its interface with the datacomputer. the internal configuration of data only affects the user to the extent that it limits how accessible or malleable the data is 🫠

honestly, the user shouldn't generally have to know about the internal data configuration 🙈

although the system tries to support all types of applications on some level, the first implementation pays particular attention to large, simply-organized, shared files 📁 the big focus is letting a user of such files describe exactly what data is really of interest to them, so that nothing but the desired information gets transmitted ✨ this matters a lot for avoiding overloading the narrow link, and it's accepted as a central design goal 👑

Data Base Organization 🍓

the database contains all the information stored in the datacomputer. it's a set of files, which are named, physically distinct collections of data 🗂️

the location of one file, the file directory, is known to the system. it holds the names, locations, and certain attributes of all the other files. access to this file is restricted 🔒

internally, each file has its own organization, but every organization is a particular application of a general model. the particular application is defined by a file description attached to the file 🧾

in the general model, each file contains uniquely numbered records. each record contains named fields. a field of a given name can show up more than once in a record, and a unique number goes with each occurrence. a field holds an elementary piece of data, which is its value 💖

records are variable in format and size, and fields are variable in length 🌈

besides the records themselves, each file can contain an index. the system maintains the index to the user's specifications. conceptually, the index contains lists of pointers to records that have certain properties. a typical list might point to the records containing the field STATE with the value MASSACHUSETTS 🗺️

the system supplies a unique, permanent identifier for each record. this identifier maps trivially into a location in the file, or at worst into a small region where the record can be found quickly 🔍 the identifier is used for pointers to the record, both from the index and from other records. it's giving connections ✨

besides the physical ordering, which is defined by record location, a logical ordering will be maintained on request by the system. it can be based on some simple function of record contents, like the value of certain fields. or the user can compute the function and just supply the result (for example, by saying "insert this record after that one") 🧚 retrieval from ordered files can be done in either physical or logical order.

in all such ordered files, if insertions are made, space must be reserved for them, and garbage collection must be done periodically 🧹 a single field value is viewed as a homogeneous string of characters or basic data units. it's described by giving its type (e.g., ASCII, BIT, binary integer, etc.), and its length is measured in some unit associated with the type 📏 when a field's length is constant throughout the file, it's stored in the file description. otherwise it appears with each occurrence of the field. the type of a field is constant 🔒

the information in the file description is enough to parse a record into (field name, value) pairs. also, given a set of those pairs and a file description, the system can produce a record that satisfies the description. mapping in either direction, there's only one possible result 🪞

with a record, a file description, and a (field name, value) pair to store in the record, there is also only one new record that can result 🎯

so a file description defines all the possible formats for a record from a particular file. iconic, honestly 👑

Stream Organization 🌊

streams are sequences of records passed from using programs to the datacomputer, or the other way around 🔁 the format of the records is defined as in the file description. so streams have the same organization as files, except they can't be indexed. the operations defined on streams are more limited than those on files, since the records have to be accessed in sequence 🚶‍♀️

there's no concept of permanent storage for streams. the records move past the datacomputer one at a time, like they're on a conveyor belt 🏭

one record, the current record, is available to the datacomputer in each stream. to start formatting the next record in an output stream, the datacomputer transmits the current record. to access the next record in an input stream, the datacomputer gives up access to the current one 🫶

(part 1 of 2, more coming bestie 🧚)

Operations 💅

okay so when the user wants the contents of their WHOLE file to solve the problem at hand, the datacomputer's job is lowkey simple 🫶 there might be some reformatting or reordering, but finding the right data to work on is trivial. however, this won't be the standard usage, especially for the remote user 🧚

for most problems, the datacomputer expects to subset the file before doing anything else 💖 the bigger the file compared to the subset, the less acceptable it is to transact with the full file just to form the subset. and the datacomputer will have such enormous files that using anything but a very small subset in one problem is most unusual 😭 so subsetting WITHOUT examining the entire file is a fundamental requirement. no cap, it's the whole vibe ✨

normally, the subset is considered formed once a list of the relevant record id's or record addresses is known 🦋

the index of the datacomputer file can be thought of as a collection of primitive record id lists that the file designer expected to be useful in forming interesting subsets 🌸 the values of all important fields can be indexed. for example, every word in a field containing a string of text might be indexed. in fact, an arbitrary function of the contents of the record, or the relation of the record to other records, can be indexed. iconic, honestly 👑

the common logical operators (AND, OR and NOT) are defined for record subsets. arbitrarily complex expressions of them can be evaluated with relatively little processor time or I/O 🔥 the ease of this comes from careful design of the index and strategies, the most important of which is the parallel evaluation of the Boolean functions on large groups of records. certain statistical operations, like counting the number of records satisfying a certain Boolean condition, can be done directly on the index. this can be used to derive question-answering strategies heuristically, or it can be the direct input to a statistical study 🎀

once the index has done all it can in subsetting, attention turns to the records themselves 🍓 certain conditions can't be evaluated using the index. an obvious case is selecting records based on the value of an unindexed field. also, certain data structures can't be explicitly represented in the file:record:field model. these must be constructed by the user, out of groups of records linked by pointers, or using other special mechanisms. the class of operations that is useful in further record selection consists of field content testing, pointer chasing, simple computation in the numerical and symbolic senses, and various operations below the data element level, such as pattern matching, string manipulation, etc. these operations need a control structure approaching that of a general purpose higher level language. it is our intention to make all of this available 💜 though NOT with the goal of providing a computation facility, but rather a data management facility that is capable of using as much knowledge as the programmer can supply 🌈

a simple set of primitives is required for file maintenance in the data structure we are talking about. the operations are:

1. add a field/record
2. delete a field/record
3. replace a field/record.

the difficult part, as in retrieval, is locating the element to be operated on 🙈 notice that individual record formats can be changed at will: the set of possible formats is limited only by the file description ✨

when record contents change, index entries that are a function of them must change too. when the function determining what is to be indexed is part of the file description, maintaining the index is automatically performed by the system. otherwise, it's the responsibility of the user 🫶

all fields in a record can be optional, variable length, and allowed to occur an arbitrary number of times (up to some fixed limit for each field) 🦋 fields can be present and later be deleted from any record. fields can be added to the file description at any time. the only reason for limiting the flexibility of a record format is to reduce storage. bestie, that's the whole trade-off 💅

---

Applications 🌈

the system outlined here is meant to be suitable for many applications. some examples are:

1. Storage and retrieval of dumps and other unstructured files. The system will happily pack away your one enormous record, as quickly and painlessly as possible 📦💖
2. Applications that would normally be set up on tape: sequentially accessed files that are copied over when they are changed. Most record formats should be able to stay exactly as they are. if you want to operate this way, the datacomputer imposes no overhead (such as indexing) on you 😌 it willingly acts as unsophisticated as a tape drive. it will pass your file, adding and changing records as it copies them, pull off the interesting ones, reconfigure them if desired, and transmit them to you. when you describe the data, you've solved the data sharing problem for this application 🎀
3. Simple-minded direct access applications. The great hairy index structure neatly degenerates to imitate indexed sequential, simple directly-addressed files, and other old standbys in the direct access world 🍓
4. Text/document retrieval. The indexing is made for this kind of application. in addition, documents can point to subdocuments, related documents, etc. 📚✨
5. Content-oriented, rapid retrieval applications. this is the specialty of the house 👑🔥
6. Large data bases used for statistical analysis or modeling, such as the census, the common social science data bases, etc. 📊🌸
7. Applications in which data element groups (such as records) are related in a complex fashion, and the intelligence of the datacomputer, which is close to the data and remote from the computational facility, can be put to good use 🧚💜

in all of these, an important consideration is size. we hope to handle these applications properly on the datacomputer, even when the files are of extraordinary size 😭🫶

the original document's machine-readable note and Postel's handwritten receipt stamp are kept below, exactly as they appear.

     [ This RFC was put into machine readable form for entry   ]
     [ into the online RFC archives by Sandy Ginoza 9/2001.    ]
     [ Original has hand-written note in Postel's handwriting: ]
     [ "Received 21 Sept 71".                                  ]