RFC 993
PCMAIL: A distributed mail system for personal computers
RFC 993: PCMAIL, A Distributed Mail System for Personal Computers 💌
Title: PCMAIL: A Distributed Mail System for Personal Computers 🎀 Authors: David D. Clark (MIT) and Mark L. Lambert (MIT) 👑 Date: December 1986 🗓️ Series: Network Working Group, Request for Comments: 993 🧚 Obsoletes: RFC-984, so this one replaces it 🔄 Updates: nothing listed in the header 🤷♀️ Category: not stated in the header 🌸
1. Status of this Document ✨
okay so this doc is a discussion of the Pcmail workstation-based distributed mail system 💅. It's a revision of the design published in NIC RFC-984. The revision is based on discussion and comment from a bunch of different sources, plus further research into the design of interactive Pcmail clients and the use of client code on machines other than IBM PCs 🧠. Since this design may change, implementation of this document is not advised ⚠️. Distribution of this memo is unlimited 🦋.
2. Introduction 🌈
Pcmail is a distributed mail system that gives mail service to an arbitrary number of users, and each user owns one or more workstations 💻. Its motivation is to provide very flexible mail service to a wide variety of workstations, ranging from small, resource-limited machines like IBM PCs to resource-rich machines like Suns or Microvaxes 🍓. ("Resources" here mainly means processor speed and disk space.) It tries to give limited service to resource-limited workstations while still giving full service to resource-rich machines. It's meant to work well with machines that are only infrequently connected to a network, and also with machines that are permanently connected 🔌. It's also designed so diskless workstations get full mail service 🙌.
The system is split into two halves 🍰. The first half is a single entity called the "repository" 🏛️. The repository is a storage center for incoming mail. Mail for a Pcmail user can arrive from outside, from the Internet, or from other repository users internally 📬. The repository also keeps a stable copy of each user's mail state, which from here on is called the user's "global mail state" 🔒. So the repository is typically a computer with a large amount of disk storage 💾.
The second half is one or more "clients" 🖥️. Each Pcmail user can have an arbitrary number of clients, usually single-user workstations. The clients give the user a friendly way to reach their global mail state over a network 🫶. To make the repository and the user's clients work together more efficiently, each client keeps a local copy of its user's global mail state, called the "local mail state" 📋. Clients might be small personal computers, so they may not always have network access, and that means they can't always reach the global mail state in the repository 📴. Because of this, the local and global mail states may not match all the time, so syncing them is necessary 🔁.
Clients talk to the repository through the Distributed Mail System Protocol (DMSP), and the spec for that protocol is in appendix A 📜. The repository is therefore a DMSP server, and it's also a mail end-site and storage facility 🗄️. DMSP gives a complete set of mail manipulation operations ("send a message", "delete a message", "print a message", and so on) 💌. It also has special operations that make it easy to sync a user's global mail state with their clients' local mail states 🔄. The team paid special attention to how DMSP operations act on a user's mail state 🔍. All DMSP operations are failure-atomic, meaning they're guaranteed to either succeed completely or leave the user's mail state unchanged 🛡️. A client can be abruptly disconnected from the repository without leaving inconsistent or damaged mail states, which is honestly iconic 😭✨.
Pcmail's design was shaped by what workstations were actually like at the time 🕰️. Some workstations are fairly portable and can be packed up and moved in the back seat of a car 🚗. A few are truly portable, about the size of a briefcase, and battery-powered 🔋. Some workstations have constant access to a high-speed local-area network, and Pcmail should allow "on-line" mail delivery for those machines. At the same time, it should provide "batch" mail delivery for other workstations that aren't always connected to a network 📦. Portable and semi-portable workstations tend to be resource-poor. A typical IBM PC has a small amount of main memory (typically less than one megabyte) and very little mass storage (floppy-disk drives that can access perhaps 360 kilobytes of data) 💀. Pcmail has to give machines like this adequate mail service without hurting its performance on more resource-rich workstations 💪. Finally, all workstations share some common traits that Pcmail should take advantage of 🙌. For instance, workstations are fairly inexpensive compared to the time-shared systems most people use for mail. So people may own more than one workstation, like a Microvax in the office and an IBM PC at home 🏠.
Pcmail's design reflects these differences between workstations 🎨. Since one person can own several workstations, Pcmail lets users have multiple access points to their mail state 🚪. Each Pcmail user can have several client workstations, and each one can reach the user's mail by talking to the repository over a network 🌐. All the clients keep local copies of the user's global mail state, and they sync the local and global states using DMSP 🔁.
Some workstations will only occasionally be connected to a network, which means they can only sometimes talk to the repository 🌙. So the Pcmail design allows two modes of communication between repository and client 💬. "Interactive mode" is used when the client is always connected to the network. Any change to the client's local mail state is made to the repository's global mail state right away, and any incoming mail is sent from repository to client immediately ⚡. "Batch mode" is used by clients that have infrequent access to the repository 🗂️. Users work on the client's local mail state, queueing their changes locally. When the client next connects to the repository, the changes are executed and the local mail state is synced with the repository's global mail state 🎉.
Last but not least, the Pcmail design keeps the cost of using a resource-poor workstation as a client as low as possible 🪶. Mail messages are split into two parts: a "descriptor" and a "body" 🧩. The descriptor is a capsule summary of the message, and its length (typically about 100 bytes) doesn't depend on how long the actual message is. The body is the actual message text, including an RFC-822 standard message header 📨. A client might not have enough storage for a complete set of messages, but it can usually hold a complete set of descriptors. That gives the user at least a summary of their mail state 👀. For clients with extremely limited resources, Pcmail allows storing partial sets of descriptors. So the user doesn't get a complete local mail state, but they can still look at summaries of some messages 📝. When a client can't immediately store message bodies, it can always pull them over from the repository once storage frees up 🧲.
The rest of this document is broken up into sections covering the following 🗺️:
- The repository architecture
- DMSP, its operations, and the motivation for its design
- The client architecture
- A typical DMSP session between the repository and a client
- The current Pcmail implementation
3. Repository architecture 🏛️
A typical machine running repository code has a relatively powerful processor and a large amount of disk storage 💻. It also has to be a permanent network site, for two reasons 🔐. First, clients talk to the repository over a network and rely on the repository being available at any time. Second, people sending mail to repository users rely on the repository being available to receive mail at any time 📬.
The repository has several jobs 🧑💼. First, and most importantly, it has to efficiently manage a potentially large number of users and their mail states 🗃️. Mail must be stored reliably, in a way that makes it easy for multiple clients to access the global mail state and sync their local mail states with it 🔁. Since a big category of electronic mail is bulletin boards (bboards), the repository should efficiently manage bboard mail too. It should use a minimum of storage for bboard messages while still letting any user subscribed to the bboard read the mail 📌. Second, the repository has to communicate efficiently with its clients 📡. The protocol used between repository and client must be reliable, and it must provide operations that (1) allow typical mail manipulation and (2) support Pcmail's distributed nature by allowing efficient synchronization between local and global mail states 💞. Third, the repository must be able to process mail from sources outside its own user community. A primary outside source is the Internet 🌍. Internet mail will arrive with a NIC RFC-822 standard message header, and the recipient names in the message must be properly translated from the RFC-822 namespace into the repository's namespace 🔤✨.
3.1. Management of user mail state ✨
Okay bestie, so Pcmail sees the whole world as a community of users 🌍💖. Each user gets a user object, which is basically their ID card. It has a unique name, a password (their clients use it to log in before they touch the global mail state 🔐), a list of "client objects" for their devices, and a list of "mailbox objects" 📬.
A client object has a unique name and a status. Each user gets one client object for every client they own, and a client literally cannot talk to the repository unless it's on that user's client list 🚫. So client objects are the bouncers 💅. They also keep local and global mail state in sync. For every global change, the repository keeps a list of the client objects that haven't recorded that change locally yet 🔄.
A client's status is either "active" or "inactive". The repository counts a client as inactive if it hasn't connected within a set time period (one week in the current repository implementation) 🗓️. When an inactive client finally connects, the repository tells it that it's been "reset" 😭. A reset means every message in the user's mail state gets marked as changed since that client last logged in. The next time the client syncs, it receives a complete copy of the repository's global mail state. The reason is that they assume enough global changes happen in a week that the client would spend way too long doing an ordinary local-global state sync 🥲.
Messages live in mailboxes, and users can have an arbitrary number of them 📂. They're used to store and categorize messages. A mailbox object both names a mailbox and describes what's in it. Mailboxes have a unique name, and their contents are described by three numbers. The first is the total number of messages. The second is the total number of unseen messages (messages the user has never seen via any client). The third is the mailbox's next available message unique identifier (UID). This info sits in the mailbox object so clients can get a summary without reading every single message ✨.
Some mailboxes are special because other users are allowed to read their messages. These are called "bulletin board mailboxes" or "bboard mailboxes" 📌, and the repository uses them to store bboard mail. Bboard mailboxes are different from ordinary mailboxes in these ways:
- Their names are unique across the entire repository. For instance, only one bboard mailbox named
sf-loverscan exist in the whole repository community. This doesn't stop other users from having an ordinary mailbox namedsf-lovers🙈. - Subscribers get read-only access to the messages in the bboard mailbox. The bboard mailbox's owner (typically the system manager) has read/update/delete access to it 👑.
A bboard subscriber keeps track of the messages they've looked at with a bboard object. The bboard object holds the name of the bboard, its owner (the user who owns the bboard mailbox where all the messages are stored), and the UID of the first message the subscriber hasn't seen yet 👀.
Users gain read-only access to a bboard by "subscribing" to it, and they lose that access when they "unsubscribe" 🔓🔒.
Each mailbox can hold an arbitrary number of message objects 💌. Every message is split into two parts. The first is the "descriptor", which is a summary of useful info about the message. The second is the "body", which is the message text itself, including its NIC RFC-822 message header. Each message is assigned a monotonically increasing UID based on the owning mailbox's next available UID. Each mailbox has its own set of UIDs, and those UIDs, together with the mailbox name and user name, uniquely identify the message within the repository 🔑.
A descriptor holds the following info: the message UID, the message size in bytes and lines, four "useful" message header fields (the date:, to:, from:, and subject: fields), and sixteen flags. The flags are numbered 0 through 15 🚩. Eight of them are reserved for the repository's use. Some of those are actually used by the repository, and others are just held for informational purposes. In the current repository implementation, the flags mark:
#0whether it has been deleted#1whether the message has been seen#2whether it has been forwarded to the user#3whether it has been forwarded by the user#4whether it has been filed (written to a text file outside the repository)#5whether it has been printed (locally or remotely)#6whether it has been replied to#7whether it has been copied to another mailbox
The remaining eight flags are reserved for future use 🔮.
Descriptors are an efficient way for clients to get message info without wasting time retrieving the whole message from the repository ⚡.
3.2. Repository-to-RFC-822 name translation
"Address objects" are how the repository translates the RFC-822-style mail addresses in Internet messages into repository names 🔁. The repository has its own namespace for identifying messages. Any message is uniquely identified by the triple (user-name, mailbox-name, message-UID). Any mailbox is uniquely identified by the pair (user-name, mailbox-name). So to send a message between two repository users, you address it to (user-name, mailbox-name). The repository delivers the message to the named user and mailbox, then assigns it a UID based on that mailbox's next available UID 💫.
To translate between RFC-822-style mail addresses and repository names, the repository keeps a list of address objects. Each address object is an association between an RFC-822-style address and a (user-name, mailbox-name) pair. When mail arrives from the Internet, the repository uses the address object list to translate the recipients into (user-name, mailbox-name) pairs and route the message correctly 🛣️✨.
4. Communication between repository and client: DMSP 🦋
The Distributed Mail System Protocol (DMSP) is a block-stream protocol that defines and manipulates the objects from the previous section 🧚. It was designed for Pcmail's single-repository/multiple-client model of the world. On top of typical mail manipulation functions, DMSP has functions that make it easy to sync global and local mail states 🔄💕.
DMSP runs on top of the Unified Stream Protocol (USP), specified in MIT-LCS RFC-272. USP gives you a reliable virtual circuit block-stream connection between two machines 🔗. It defines a basic set of data types ("strings", "integers", "booleans", etc.). Instances of these types are grouped in an application-defined order to form USP blocks. Each USP block is defined by a numeric "block type", so a USP application can interpret a block's contents based on its type 🧩. DMSP is a set of operations, and each operation is made up of one or more USP blocks sent between the repository and the client.
Here's how a DMSP session goes 🎀. The client begins by opening a USP connection to the repository's machine. Then it authenticates both itself and its user to the repository with a "login" operation. If authentication succeeds, the user can perform an arbitrary number of DMSP operations, then end the session with a "logout" operation. At that point, the repository closes the connection 👋.
DMSP can manipulate a pair of mail states (local and global) at once, so it is extremely important that all DMSP operations are failure-atomic 😤. If any DMSP operation fails, both states must be left in a consistent, known state. That's why a DMSP operation counts as failed unless the operation initiator receives an explicit acknowledgement 🙅♀️. There are two basic forms of acknowledgement, based on the two broad categories all DMSP operations fall into:
- Mail state modification. The repository acknowledges the request with either an "ok" or a "failure". If it's a failure, the reason for it is returned too.
- Information requests. The repository acknowledges the request by providing the information to the client.
Operations like "delete a message" are in the first category, and operations like "send a list of mailboxes" are in the second 📋.
Next up is a general discussion of all the DMSP operations 🔍. They're broken down by type: general operations, user operations, client operations, mailbox operations, address operations, bboard operations, and message operations.
4.1. General operations
The first group of DMSP operations handles general functions that don't operate on one particular class of object 🌸. DMSP has two general operations, and this is the first one 💅.
To prevent protocol version skew between clients and the repository, DMSP provides a send-version operation. The client supplies its DMSP version number as an argument. The operation succeeds if the supplied version number matches the repository's DMSP version number, and it fails if the two don't match 🚦. Version numbers are unsigned quantities, like 100, 101, 200. The send-version operation should be the first thing a client sends to the repository, since no other operation might work if the client and repository are using different versions of DMSP 🙃.
Users can send mail to other users with the send-message operation 📨. The message must have an Internet-style header as defined by NIC RFC-822. The repository takes the message and distributes it to the mailboxes listed in the to:, cc:, and bcc: fields of the message header. If one or more of those mailboxes exists outside the repository's user community, the repository is responsible for handing the message to a local SMTP server 🌐.
An OK is sent from the repository only if the entire message was successfully transmitted. If the message was destined for the Internet, the send-message operation is successful if the message was successfully transmitted to the local SMTP server 💜.
4.2. User operations 💖
okay so the next batch of DMSP operations is all about user objects 🫶 the big ones are "login" and "logout" ✨ a client has to do a login before it can touch a user's mail state, no exceptions bestie 💅
a DMSP login block has five things in it:
- the user's name
- the user's password
- the name of the client doing the login
- a flag telling the repository to make a client object for that client if one doesn't already exist (the
create-this-client?flag, more on that in 4.3 🧚) - a flag that's TRUE if the client wants "batch mode" and FALSE if it wants "interactive" mode
that flag value lets the repository tune its internal settings for whichever mode is in play 🎀
when a client logs in, the repository can send back an OK block (authentication worked 🥳), a FAILURE block (authentication didn't work 💀), or a FORCE-RESET block. the FORCE-RESET one is the real drama 😭 it gets sent if the client logging in has been marked "inactive" by the repository. clients get marked inactive if they haven't connected in over a week. with FORCE-RESET, the client has to erase its local mail state and pull down a complete copy of the repository's mail state. the reason is that so many changes happened in a week that a normal sync would be way too slow ✨
when a client is done with its session, it does a logout. that lets the repository do any cleanup it needs before the USP connection closes 🌸
users can change their password with "set-password" 🔐 it works kind of like the UNIX change-password operation. you give it your current password and the new one you want. if the current password you gave matches the user's current password, the password gets changed to the new one 💖
4.3. Client operations 🦋
DMSP has four operations for manipulating client objects. the first one, "list-clients", tells the repository to send the user's client list to the requesting client. the list comes as a series of (client-name, status) pairs, where status is 0 (inactive) or 1 (active) 📋
"create-client" lets a user add a client object to their list of client objects 🧁 the login can do this too through the create-this-client? flag, but the add-client operation is super handy for making a bunch of new client objects while you're already logged in through an existing client. create-client needs the name of the client to add.
"delete-client" removes an existing client object from a user's client list. the client being removed can't be in use by anyone at the time 🚫
the last client operation is "reset-client", and it's giving clean slate 🧼 it causes the repository to mark all messages in the user's mail state as changed since the client last logged in. when the client next syncs, it gets a complete copy of the repository's global mail state. this is useful for two reasons 🧠 first, a client's local mail state can easily get lost or damaged, especially if it's stored on a floppy disk 💾 second, if a client has been marked inactive, reset-client is a fast way to resync, assuming there are so many differences between the local and global mail states that a normal sync would take way too much time 🐢
4.4. Mailbox operations 📬
DMSP supports five operations that manipulate mailbox objects. first, "list-mailboxes" has the repository send info on each mailbox to the requesting client. that info is the mailbox name, total message count, unseen message count, and "next available UID" 📊 this is super useful for syncing, since a client can compare the user's global mailbox list with its local mailbox list. it's also a quick summary of each mailbox's contents without actually having the contents present ✨
"create-mailbox" has the repository create a new mailbox and attach it to the user's list of mailboxes. an address object binding the (user-name, mailbox-name) pair to an RFC-822-style address is automatically created and placed in the repository's list of address objects 💌 that's how mail coming from the Internet gets correctly routed to the new mailbox 🛣️
"delete-mailbox" removes a mailbox from the user's list. all messages inside it are also deleted and permanently removed from the system 💀 any address objects binding the mailbox name to RFC-822-style mailbox addresses get removed too 🫠
"reset-mailbox" causes the repository to mark all the messages in a named mailbox as changed since the current client last saw them. when the client next syncs, it receives a complete copy of that mailbox's mail state. it's basically a more specific version of reset-client (which pulls over a complete copy of the user's global mail state). its main use is for mailboxes whose contents got accidentally destroyed locally 🩹
finally, DMSP has an "expunge-mailbox" operation 🗑️ any message can be deleted and "undeleted" at will. deletions only get made permanent when you perform an expunge-mailbox operation. it goes through a named mailbox and removes from the system any messages marked "deleted" 🧹
4.5. Address operations 💌
DMSP provides three operations that let users manipulate address objects 🔗 first, "list-address" returns a list of address objects associated with a particular (user-name, mailbox-name) pair.
"create-address" adds a new address object that associates a (user-name, mailbox-name) pair with a given RFC-822-style mailbox address ✨
finally, "delete-address" destroys the address object binding the given RFC-822-style mail address and the given (user-name, mailbox-name) pair 🫠
4.6. Bboard operations 📌
DMSP provides seven bulletin board operations 🎀 the first, "list-bboards", gives the user a list of the bboards they're currently subscribed to. the list has an entry for each bboard the user subscribes to, and each entry contains:
- the bulletin board's name
- the UID of the first message the subscriber has not yet seen
- the highest message UID in the bulletin board
- the number of messages the subscriber has not yet seen
"list-all-bboards" gives the user a list of all the bboards they can subscribe to 🌍
"create-bboard" lets a user create a bboard mailbox. the name given must be unique across the entire repository user community 🏷️ once the bboard mailbox exists, other users can subscribe to it, using bboard objects to keep track of which messages they've read on which bboards 📖
"delete-bboard" lets a bboard's owner delete a bboard mailbox. subscribers who try to read from a bboard mailbox after it's been deleted are told that the bboard no longer exists 🪦
DMSP also has operations to subscribe to, and unsubscribe from, any bboard 💜 "subscribe-bboard" adds a bboard object to the user's bboard object list, and "unsubscribe-bboard" removes a bboard object from that list. heads up 👀 this does not delete the bboard mailbox (obviously only the bboard's owner can do that). it just removes the user from the list of the bboard's subscribers.
last but not least, DMSP lets the user tell the repository which messages in a bboard they've seen 🥹 every bboard object contains the UID of the first message the user has not yet seen, and the "set-first-unread-message-UID" operation updates that number, ensuring the user sees a given bboard message only once 🌈
4.7. Message operations ✨
okay so messages are the main characters 💅 They're the most-manipulated Pcmail objects, so DMSP gives them their own special operations for efficient synchronization, plus a set of operations for the standard message-manipulation stuff 🦋 Heads up: from here on, "message" and "descriptor" mean the same thing.
A user can ask for a series of descriptors with get-descriptors 📬 The series is identified by a pair of message UIDs, the lower and upper bounds of the list. UIDs are defined to be monotonically increasing numbers, so a pair of UIDs is enough to completely pin down the series 🔥 If the lower bound UID doesn't exist, the repository starts the series with the first message whose UID is greater than the lower bound. Same vibe on the other end: if the upper bound doesn't exist, the repository ends the series with the last message whose UID is less than the upper bound. If certain UIDs in the series no longer exist, the repository (obviously) doesn't send them 💀
The repository sends the descriptors back as a sequence of "choices" 🎀 Each element is either a descriptor, in which case the choice is tagged as a descriptor, or a notification that the requested message has been expunged after the client's last connection to the repository. A descriptor choice is a record with the message's UID, flags, to, from, date, and subject fields, length in bytes, and length in lines. An expunged choice only contains the expunged message's UID 🪦
Next up is get-changed-descriptors, which is built for state synchronization 🔄 Whenever a descriptor changes state (say, it gets deleted), the repository notes which clients haven't recorded the change locally yet. get-changed-descriptors has the repository send the client a given number of descriptors that have changed since the client's last synchronization. The list starts with the earliest-changed descriptor ⭐ Also, the list is only guaranteed to be monotonically increasing within a single call to get-changed-descriptors. Messages with lower UIDs can get changed by other clients in between calls, so don't be shocked 🙈
Once you've looked at the changed descriptors, you'll want to tell the repository that your client has recorded the change locally ✅ That's what reset-changed-descriptors is for. It makes the repository mark a given series of descriptors as "seen by current client." The series is identified by a low UID and a high UID. UIDs in the series that no longer exist just don't get reset 🫶
Message bodies come over from repository to user with get-message-text 📩 Splitting get-descriptors and get-message-text into separate operations is the whole point. Clients with small amounts of disk storage can grab a small message summary (via get-descriptors or get-changed-descriptors) without pulling down the entire message 🧚
Sometimes a message is just too big for some clients to store locally, and that's valid 😭 Users can still look at the contents via print-message. This operation has the repository send a copy of the message to a named printer 🖨️ The printer name only needs to mean something to the particular repository implementation. DMSP transmits the name only as a means of identification, so it's basically a label 🏷️
Copying a message into another mailbox is done with copy-message 💜 If the copy works, you get back a descriptor list of length one, containing a descriptor for the copied message. This descriptor is required, because the copy gets a different UID than the original. The client can't be expected to know which UID got assigned to the copy, so the repository sends a descriptor containing that UID. Iconic behavior, honestly 👑
5. Client Architecture ✨
Clients can be all sorts of workstations, so Pcmail's architecture has to account for a whole range of characteristics 🌈 First, most workstations are way more affordable than the big computers that handle mail service, so a user might have more than one 💖 Second, some workstations are portable and aren't expected to stay hooked up to a network all the time 🦋 Finally, lots of the smaller workstations are resource-poor, so they aren't expected to store much state information locally 🥺 The subsections below cover the parts of Pcmail's client architecture that address these different characteristics ✨
5.1. Multiple clients
The fact that Pcmail users can own more than one workstation is the justification for the multiple client model 💅 A user might have one client at home, another at the office, and maybe even a third portable one 🎀 Each client keeps its own copy of the user's mail state, which is where Pcmail's distributed nature comes from 🌸 Separate clients let users reach their mail state from several different places. Pcmail doesn't put any restrictions on a user talking to the repository from several clients at the same time. Instead, the repository implementation decides whether to let several clients have concurrent access to a user's mail state 👑
5.2. Synchronization
Some workstations are small and pretty portable, so the odds of them always being connected to a network are fairly low 📉 That's another reason each client keeps a local copy of the user's mail state 🍓 The user can then manipulate the local mail state while offline (not connected to the network or the repository). This immediately brings up the problem of synchronizing local and global mail states 😭 The repository is always in a position to receive global mail state updates, whether that's incoming mail or changes from other clients. A client that isn't always connected can't receive those global changes right away. On top of that, the client's user can make their own changes to the local mail state 🙃
Pcmail's architecture allows fast synchronization between each client's local mail state and the repository's global mail state ⚡ Each client is identified in the repository by a client object attached to the user, and that object is the basis for synchronizing local and global mail states 🫶 Some of the less common state changes include adding and deleting user mailboxes, and adding and deleting address objects. Synchronizing these changes happens through DMSP list operations, which let clients compare their local versions of the mailbox and address object lists with the repository's global version and make any appropriate changes 📋 Most possible changes to a user's mail state come in the form of changed descriptors. Since most users have a large number of messages, and message states change pretty often, message synchronization needs special attention 💀
An existing descriptor can change in one of two ways. First, one of its sixteen flags values can change. This covers reading an unseen message, deleting a message, and expunging a message 🗑️ Second, a descriptor can change through the arrival of incoming mail or by copying a message from one mailbox to another. Both of these result in a new message being added to a mailbox 📨
In both of those cases, synchronization is required between the repository and every client that hasn't previously noted the change 🔥 To keep track of which clients have noticed a global mail state change and updated their local states to match, each mailbox has a list of active clients associated with it 🧾 Each client also has a (potentially empty) "update list" of messages that have changed since that client last read them 🌈
When a client connects to the repository, it runs the DMSP get-changed-descriptors operation 🔌 This makes the repository return a list of all descriptor objects on that client's update list. As the client receives the changed descriptors, it can store them locally, which updates the local mail state 💾 After a changed descriptor has been recorded, the client uses the DMSP reset-descriptors operation to remove that message from its update list ✅ That descriptor won't be sent to the client again unless (1) it's explicitly requested, or (2) it changes again 🫶
That's how it works in practice: a client runs through its user's mailboxes, gets all the changed descriptors, incorporates them into the local mail state, and marks each change as recorded 💖
5.3. Batch operation versus interactive operation
Because some workstations are portable, they might not always be connected to a network (and able to talk to the repository) 🧳 Since each client keeps a local mail state, Pcmail users can manipulate that local state while disconnected from the repository 💅 This is known as "batch" operation, since all changes get recorded by the client and applied to the repository's global state in one batch the next time the client connects 📦 Interactive operation is when a client is always connected to the repository. In interactive mode, changes made to the local mail state are also immediately made to the global state through DMSP operations ⚡
In batch mode, the client and repository interact like this: the client connects to the repository and sends over all of the changes the user made to the local mail state. The repository updates its global mail state to match. Once all the changes have been processed, the client starts synchronization so that newly arrived mail, plus mail state changes made by other clients, get incorporated into the local state 🎀
In interactive mode, local changes go straight to the repository, so the first part of batch-type operation is eliminated 🦋 Synchronization works differently too. The client synchronizes once when it first opens a connection to the repository. After that, subsequent synchronizations can happen either when the user asks for one or automatically every so often, as the client decides 🔄 Totally seamless, bestie 🌸
5.4. Message summaries ✨
Okay so, smaller workstations can have basically no disk space 😭 and clients on those machines may never have room for a full local copy of a user's global mail state. That means Pcmail's client architecture has to let users get a clear picture of their mail state without having every message on the machine. Iconic design requirement, honestly 💅
This is where descriptors come in 🦋 A descriptor gives you message info without eating up storage. Each one holds a summary of a single message: the message UID, its length in bytes and lines, its status (encoded in the eight system-defined and eight user-defined flags), and pieces of its RFC-822 header (the "to:", "from:", "subject:" and "date:" fields). All of that fits in a tiny data structure (around 100 bytes) whose size doesn't depend on how big the message is. Main character efficiency 🧚
Most clients should be able to store a full list of message descriptors without much trouble 🌸 That lets a user see their whole mail state without having the messages locally. And if a client is extremely low on disk, it can also grab just a subset of the descriptors from the repository. Short messages can live on the client alongside the descriptors, while long messages can either be printed via the DMSP print-message operation or pulled over with the fetch-message-text operation. Choices, bestie ✨
6. Typical interactive-style client-repository interaction 🎀
Let's walk through a typical communication session between the repository and a client 💜 The client belongs to one of three clients owned by a user named "Fred". It's called "office-client", and since Fred uses it all the time to check his mail, it's marked as "active". Fred has two mailboxes: "main" holds all of his current mail, and "archive" holds the messages that really matter long-term 📁 The example runs a simple synchronization, then a bunch of typical mail state manipulations. Usually the synchronization is done by an application program that connects to the repository, logs in, synchronizes, and logs out.
For the example, every DMSP operation is shown in a human-readable format. In reality, the operations would be sent as a stream of USP blocks, each one a block-type number followed by a stream of bytes representing the block's components 🤓
Before anything else, the client has to authenticate Fred to the repository so it can get at his global mail state. That happens with the DMSP login operation:
login ["fred", "fred-password", "office-client", F, F]This tells the repository that Fred is logging in through "office-client", and that "office-client" is an existing client object attached to Fred's user object. Breaking it down: the first component is Fred's repository user name, and the second is his password. The third is the name of the client talking to the repository. The fourth tells the repository not to create "office-client" if it can't find its client object. And the last one says the client isn't running in batch mode but in interactive mode 🙌
Fred's credentials check out, so the repository logs him in and acknowledges the request with an OK block 🥳
Now that Fred's logged in, the client does an initial synchronization. First it asks for an up-to-date list of mailboxes:
list-mailboxes []The repository replies with:
mailbox-list [["main", 10, 1, 253],
["archive", 100, 0, 101]]So the client learns there are two mailboxes, "main" and "archive" 📬 "Main" has 10 messages, one of which is unseen, and the next incoming message will get UID 253. "Archive" has 100 messages, none unseen, and the next message sent there gets UID 101. There are no new mailboxes in the list. (If there had been, the client program would create them. On the flip side, if any mailboxes in the client's local list weren't in the repository's list, the program would assume another client deleted them and delete them locally too 💀)
To synchronize, the client only needs to look at each mailbox's contents to see whether (1) any new mail has arrived, or (2) Fred changed any messages on his other two clients since "office-client" last connected to the repository 👀
The client asks for changed descriptors with the "get-changed-descriptors" operation. It asks for at most ten, since "office-client" is very low on storage 🫶
get-changed-descriptors ["main", 10]The repository responds with:
descriptor-list [[descriptor[
6,
[T T F F F F F F F F F F
F F F F],
"Fred@borax",
"Joe@fab",
"Wed, 23 Jan 86 11:11 EST",
"tomorrow's meeting",
621,
10]]
[descriptor[
10,
[F T F F F F F F F F F F
F F F F],
"Fred",
"Freds-secretary",
"Fri, 25 Jan 86 11:11 EST",
"Monthly progress report",
13211,
350]]
]The first descriptor is a message Fred deleted on another client yesterday, so "office-client" marks its local copy as deleted 🗑️ The second is a brand-new message, so "office-client" adds the descriptor to its local list ✨ Both changes are now recorded locally, which means the descriptors can be reset:
reset-descriptors ["main", 6, 10]The repository removes every message with UIDs between 6 and 10 from "office-client"'s update list (in this case, just two messages). "Main" is now synchronized 🎉 Next, the client moves on to Fred's "archive" mailbox and asks for the first ten changed descriptors:
get-changed-descriptors ["archive", 10]The repository responds with:
descriptor-list []An empty list tells "office-client" that nothing in "archive" has changed since its last synchronization, so no new synchronization is needed for it 😌
Now Fred's client is ready to pull down the new message. It's 320 lines long, and there might not be enough storage on "office-client" to hold it. The client tries anyway 😬
fetch-message-text ["main", 10]The repository starts sending the message:
message ["From: Fred's-secretary",
"To: Fred",
"Subject: Monthly progress report",
"Date: Fri, 25 Jan 86 11:11 EST",
"",
"Dear Fred,",
"Here is this month's progress report",
...
]Halfway through the transfer, "office-client" runs out of disk space 😭 But every DMSP operation is defined to be failure-atomic, so the part of the message already received is destroyed locally and the operation fails. "Office-client" tells Fred the message can't be pulled over because there isn't enough disk space. The synchronization is now finished, and Fred can start reading his mail 📖 The message that was too big is marked "off line". Fred can either remote-print it, or delete other messages until he has enough room to store it ✂️
Since he's in interactive mode, any changes he makes to messages are immediately turned into DMSP operations and sent to the repository ⚡ Depending on the client implementation, Fred either has to run a "synchronize" command every so often, or the client synchronizes for him automatically at intervals 🔁
7. A current Pcmail implementation 🖥️
This section gives a quick look at a current Pcmail system serving a small community of users 🌈 The Pcmail repository runs under UNIX on a DEC VAX-750 that's connected to the Internet. The clients run on IBM PCs, XTs, and ATs, as well as Sun workstations, Microvaxes, and VAX-750s.
7.1. IBM PC client code 💾
The client code for the IBM machines only operates in batch mode 🙃 Users make local state changes, which get queued until the client connects to the repository. Then the queued changes are performed and the local and global states are synchronized. After that, the client disconnects from the repository.
Users access and modify their local mail state through a user interface program. The program uses windows and a full-screen mode of operation, and users get a variety of commands for acting on individual messages as well as mailboxes 🪟 The interface lets you use any text editor to compose messages, and it adds its own features to make RFC-822-style header composition easier ✍️
Synchronization and processing of queued changes is handled by a separate program, which the user runs whenever they want to 🔄 That program takes any actions queued while using the user interface and converts them into DMSP operations. All queued changes are made before any synchronization happens.
Limiting IBM PC clients to batch mode was a choice made because of development environment limitations 😤 The user interface couldn't work with the network code inside it because of program size limits. And since MS-DOS has no multi-processing facilities, the two programs couldn't run side by side either. The only solution was a two-part client: one part read the mail, and the other part interacted with the repository 🧩
7.2. UNIX client code 🌟
The client code for the Suns, Microvaxes, and VAX-750s runs on 4.2/4.3BSD UNIX 🐚 It's fully interactive, with a powerful user interface living inside Richard Stallman's GNU-EMACS editor. UNIX-based workstations have plenty of main memory and disk storage, so no effort went into shrinking the local mail state by keeping message descriptors instead of message text 💅
The local mail state is made up of a number of BABYL-format mailboxes. The interface looks a lot like the RMAIL mail reader that's already in GNU-EMACS 📧
The user interface talks to the repository through a DMSP implementation built into the GNU-EMACS kernel. Changes to the local mail state are made on the repository right away, and the repository is fast enough that the network round trip is barely noticeable ⚡
There's no automatic synchronization when new mail arrives or old mail is changed by another client 🙈 Instead, users have to fetch new mail explicitly. A simple "notification" program runs in the background and wakes up every minute to check for new mail. When mail arrives, the user runs a command to get it, which also synchronizes the mailbox at the same time 🔔
7.3. Repository code 🧚
okay so the repository is written in C and runs on 4.2/4.3BSD UNIX 💻. Right now it lives on DEC VAX-750s and Microvaxes, but other repositories will soon be running on IBM RT machines and Sun workstations 🌈. The repository code is designed so that several clients belonging to one particular user can "concurrently" modify that user's state 💅. To keep things from getting messy, a mailbox locking scheme stops one client from changing a mailbox while another client is busy changing that same mailbox 🔒.
8. Conclusions 👑
Pcmail is now used by a small community of people at the MIT Laboratory for Computer Science 🎓. ngl the repository design is slaying, because it gives an efficient way to store and maintain mail state for several users 💖. Performance is quite good when up to ten users are connected 🙌. It remains to be seen whether or not the repository will be efficient at managing the state of ten or a hundred times that many users 🤔. Given enough disk storage, it should be able to, since communication between users' clients and the repository is likely to be very asynchronous and to happen in short bursts, with long "quiet intervals" in between as people are busy doing other things 😴.
Members of another research group at LCS are currently working on a replicated, scalable version of the repository. It's designed to support a very large community of users with high availability 🌐. This repository also uses DMSP and has successfully communicated with clients that use the current repository implementation 🤝. DMSP therefore seems to be usable over several flavors of repository design 🍦.
The IBM PC clients are unfortunately very limited in resources, which makes local mail state manipulation difficult at times 😭. Synchronization is also relatively time consuming because the PCs have low performance 🐢. The "batch-mode" that the PCs use tends to be good for PCs that spend a large percentage of their time unplugged and away from a network 🔌. It's somewhat inconvenient for PCs that are always connected to a network, since those could make good use of an "interactive-mode" state manipulation 💜.
The UNIX-based clients are far easier to use than their PC counterparts 💅. Synchronization is much faster, and there is far more functionality in the user interface. Having an interface that runs within GNU-EMACS helps a lot with this 🫶. Most people using the Pcmail system use the UNIX-based client code 🐧.
APPENDIX ✨
A. DMSP Protocol Specification 🦋
okay so here's the list of DMSP operations, sorted by object type 📋. For each one you get the block types, the arguments, and the acknowledgement block types you should expect back 🎀. Every DMSP block has a different number, and the first digit of each block type tells you which object is being manipulated 🔢. Operations numbered 5xx are general, 6xx are user operations, 7xx are client operations, 8xx are mailbox operations, 9xx are address operations, 10xx are bboard operations, and 11xx are message operations 🌸.
Failure blocks contain two fields, a "code" and a "why" 💀. The "code" is an unsigned number that places the error into one of several broad categories (listed below) 🗂️. The "why" is a text string that might explain the error in greater detail 📝.
Error codes:
- 1: network error while reading or writing data 🌐
- 2: internal repository error. This can be due to lack of memory, a fatal bug, lack of disk space, etc. 💥
- 3: requested object already exists. For example, you tried to create a mailbox that already exists 🙈
- 4: requested object not found. For example, you tried to delete a message or a mailbox that doesn't exist 👻
- 5: protocol error. Typically DMSP protocol version skew 🔀
- 6: block argument error. For example, a
set-message-flagoperation was attempted on a bboard by someone other than the bboard's owner 🚫 - 7: data read error. The repository was unable to read the mail state information requested 📖
- 8: data write error. The repository was unable to write out changed mail state information, perhaps because the disk was full 💾
- 9: operating system error. This one should be reserved for things like fork or pipe call errors 🛠️
- 10: unexpected or unknown block type received. For example, you sent a
delete-mailboxblock and received amailbox-listblock in response 🤯
Blocks marked => flow from client to repository, and blocks marked <= flow from repository to client 🏃. If more than one block can be sent, the choices are delimited by "or" ("|") characters 🔀.
For clarity, each block type is put in a human-understandable form 🧠. The block number is followed by an operation name, but this name is never transmitted as part of a USP block. Block arguments are identified by name and type, and enclosed in square brackets. "Record" data types are described by a list of "field-name:field-type" pairs contained in square brackets. "Choice" data types are described by a list of "tag-name:tag-type" pairs contained in square brackets. USP data types are defined as follows (the definitions are brief, so refer to the USP specification for more detailed descriptions) 📚.
A.1. Primitive data types 🧁
string (S): a series of bytes, null-byte padded to even length and preceded by a 16-bit length specifier. Strings are sent in "net-ascii" format. The newline sequence is carriage return followed by linefeed, and single carriage returns are to be followed by a null byte 💌.
- cardinal (C): a 16-bit unsigned number.
- long-cardinal (LC): a 32-bit unsigned number.
- integer (I): a 16-bit signed number.
- long-integer (LI): a 32-bit signed number.
- boolean (B): a 16-bit number with either a 1 or a 0 in the 16th bit.
A.2. Compound data types 🍰
- sequence (SEQ): a list of data items, all the same type, preceded by a 16-bit sequence length specifier.
- array (AR): a fixed-length list of data items, all the same type. A particular array's length is fixed by the application.
- record (REC): a list of data items of any type. A particular record's format is fixed by the application.
- choice (CH): one of a list of possible data types. The data type contained in the choice is identified by a 16-bit numeric tag. The application interprets the data item based on the tag value.
A.3. DMSP Abstract Data Types 🧩
Following are data types defined and used only by DMSP 💜.
client: a record with the following format: REC[name:S, status:C] Status is either 1 (active) or 0 (inactive).
mailbox: a record with the following format: REC[name:S, next-uid:LC, #msgs:C, #new-msgs:C]
bboard: a record with the following format:
REC[name:S, first-unread-message-UID:LC
number-of-unseen-messages:C highest-UID:LC]descriptor: a record with the following format:
REC[UID:LC, flags:SEQ[B], from, to, date, subject:S,
#bytes:LC, #lines:LC]desc-choice: a choice with the following format: CH[expunged-message-UID:LC, desc:descriptor] The descriptor tag number is 1. The expunged-message tag number is 0.
A.4. General operations 📨
=> 502 (send-version) [version:C]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 503 (send-message) [message:SEQ[S]]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]A.5. User operations 👤
=> 600 (login) [name:S, password:S, client:S,
create-client-object?:B
batch-mode?:B]
<= 500 (ok) [] |
501 (failure) [code:C, why:S] |
705 (force-client-reset) []
=> 601 (logout) []
<= 500 (ok) []
=> 602 (set-password) [old:S, new:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]A.6. Client operations 🦄
=> 701 (list-clients) []
<= 700 (client-list) [client-list:SEQ[client]]
=> 702 (create-client) [client:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 703 (delete-client) [client:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 704 (reset-client) [client:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]A.7. Mailbox operations 📫
=> 801 (list-mailboxes) []
<= 800 (mailbox-list) [mailbox-list:SEQ[mailbox]]
=> 802 (create-mailbox) [mailbox:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 803 (delete-mailbox) [mailbox:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 804 (reset-mailbox) [mailbox:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 805 (expunge-mailbox) [mailbox:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:SA.8. Address operations 🏡
=> 901 (list-addresses) [mailbox:S]
<= 501 (failure) [code:C, why:S] |
900 (address-list) [address-list:SEQ[S]]
=> 902 (create-address) [mailbox:S, address:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 903 (delete-address) [mailbox:S, address:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]A.9. Bboard operations 📌
=> 1001 (list-bboards) []
<= 1000 (bboard-list) [bboard-list:SEQ[bboard]]
501 (failure) [code:C, why:S]
=> 1002 (create-bboard) [name:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 1003 (delete-bboard) [name:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 1004 (subscribe-bboard) [name:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 1005 (unsubscribe-bboard) [name:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 1006 (set-bboard-first-unread) [name:S, UID:LC]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 1007 (list-all-bboards) []
<= 1008 (bboard-name-list) [bboard-name-list:SEQ[S]]
501 (failure) [code:C, why:S]A.10. Message operations 📬
okay bestie, this is the message ops section and it's where the mail actually lives 💌 every operation here follows the same pattern: a request goes out with =>, and the reply comes back with <=. 🫶
=> 1102 (get-descriptors) [mailbox:S,
low-uid:LC,
high-uid:LC]
<= 501 (failure) [code:C, why:S] |
1100 (desc-list) [desc-list:SEQ[desc-choice]]
=> 1103 (get-changed-descriptors) [mailbox:S,
max-to-send:C]
<= 501 (failure) [code:C, why:S] |
1100 (desc-list) [desc-list:SEQ[desc-choice]]
=> 1104 (reset-changed-descriptors) [
mailbox:S,
start-uid:LC,
end-uid:LC]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 1105 (get-message-text) [mailbox:S,
uid:LC]
<= 501 (failure) [code:C, why:S] |
1101 (message) [message:SEQ[S]]
=> 1106 (print-message) [mailbox:S,
uid:LC,
printer-name:S]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]
=> 1107 copy-message[source-mailbox:S,
target-mailbox:S,
source-uid:LC]
<= 501 (failure) [code:C, why:S]
<= 501 (failure) [code:C, why:S] |
1100 (desc-list) [desc-list:SEQ[desc-choice]]
=> 1108 (set-flag) [mailbox:S,
uid:LC,
flag-number:C,
flag-setting:B]
<= 500 (ok) [] |
501 (failure) [code:C, why:S]So that's the whole message crew: getting descriptors, grabbing changed ones, resetting those changes, pulling message text, printing, copying, and setting flags 💅✨ each one has its own block number, and that's what the table below is for 🌸
DMSP block types by number 🔢
Here's the full lineup of block types with their numbers, grouped by what they're for. Bookmark this one, it's the cheat sheet you'll keep coming back to 🧚💜
DMSP block types by number
General block types
ok 500
failure 501
send-version 502
send-message 503
User operation block types
login 600
logout 601
set-password 602
Client operation block types
client-list 700
list-clients 701
create-client 702
delete-client 703
reset-client 704
force-client-reset 705
Mailbox operation block types
mailbox-list 800
list-mailboxes 801
create-mailbox 802
delete-mailbox 803
reset-mailbox 804
expunge-mailbox 805
Address operation block types
address-list 900
list-addresses 901
create-address 902
delete-address 903
Bboard operation block types
bboard-list 1000
list-bboards 1001
create-bboard 1002
delete-bboard 1003
subscribe-bboard 1004
unsubscribe-bboard 1005
set-bboard-first-unread 1006
get-n-new-bboard-descriptors 1007
list-all-bboards 1008
bboard-name-list 1009
Message operation block types
descriptor-list 1100
message 1101
get-descriptors 1102
get-changed-descriptors 1103
reset-changed-descriptors 1104
get-message-text 1105
print-message 1106
copy-message 1107
set-flag 1108And that's a wrap, bestie! 🌈💖 The whole DMSP spec is served, every block number accounted for, and now you're officially a DMSP girlie 👑✨
📦 where this page comes from
- ietf: rfc993 1986-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms