RFC 677

Maintenance of duplicate databases

📅 published January 1975, from ietf

The Maintenance of Duplicate Databases 🎀

okay so this is an RFC from January 27, 1975, written by Paul R. Johnson and Robert H. Thomas (both at BBN-TENEX), filed as RFC 677 (NIC 31507) in the Network Working Group 🦋. it's a working paper, not a finished spec, and the header doesn't list anything it obsoletes or updates. the whole thing is about keeping duplicate databases in sync across an ARPA-like network, and honestly it's giving early-internet main character energy 💅✨

Preface:

okay so this paper is a working paper on keeping duplicated databases consistent in an ARPA-like network 🌸 it talks about the general duplicate database problem first, then goes pretty deep into one solution for one specific kind of duplicate database. the ideas here were used to design the User Identification Database for the TIP user authentication and accounting system. the authors think these ideas are generally useful for distributed database problems too, so it's not just a one-off thing 🧚

Introduction

there are a bunch of reasons you'd want redundant, duplicate copies of a database in a distributed network. two big ones:

reliability of data access 🛡️

critical data gets way more accessible when you keep redundant copies of it. the database used for TIP login and accounting is spread out redundantly for exactly this reason 💖

efficiency of data access ⚡

data is faster to reach when it's "near" the process that needs it 🏃‍♀️ a copy of the TIP user ID database lives at every site that offers TIP login, so access is quick. (reliability says this database should be redundant, and efficiency says there should be a copy at each authentication site. so it's both, obviously 🙈)

designing a system to keep redundant, duplicate databases in sync is genuinely hard 😭 the copies are separated by communication delay, and the real world keeps throwing system crashes, operator error, channel failures, and more at you. this paper goes through some of the problems the authors ran into and outlines a design that solves them for one particular kind of database 🫶

The Model

the model is a group of independent database management processes (DBMPs), and each one keeps its own copy of the database 🍓 they talk to each other over network communication paths. every DBMP has complete control over its own copy. it handles all the reads and changes to that copy in response to requests from other processes. the DBMPs only act when asked, but the paper will often say they're the ones causing or originating the changes anyway. it's a vibe thing 💅

here's the important part: the communication paths between DBMPs can fail. so a DBMP might get its conversations with others interrupted, or it might have to wait until paths are back up before it can talk to anyone again 😩 this paper assumes that messages from one process to another are delivered in the same order they were sent. the authors say this is true of the ARPANET 🛰️ and for networks that don't promise that, the DBMPs can use communication protocols to put the messages back in the right order.

to go further, we need to be more precise about the duplicated database and what operations are allowed on it 🎀 at one extreme you've got a constant, read-only database. that's easy mode 🥱 the DBMPs just answer value retrieval requests. at the other extreme is a general shared database where functional modification requests (like X := f(X,Y,Z)) are allowed, and/or where entries have to be completely locked while they're being changed 💀 that gets you all the problems of shared databases on a single computer (synchronization, the possibility of deadlock), plus the extra mess of having copies spread across independent computers.

for example, a fully general system has to deal with communication failures that split the network into two or more sub-networks 🌪️ any solution that relies on locking an element for synchronized changes has to handle processes in non-communicating sub-networks trying to lock the same element. so you've got three options, and none of them are great:

both get to lock it, which breaks the lock discipline 🙃
both have to wait until the partition goes away, which could take arbitrarily long ⏳
some form of centralized or hierarchical control, which means some DBMPs depend on others for every modification and maybe for every access too 🕸️

The Database

the type of database this paper looks at is a collection of entries, each one a (Selector,Value) pair 🌈 every selector is unique, and as far as the DBMPs are concerned, the values are atomic. the mechanisms here could be extended to handle databases with more structure, where a value is itself a collection of (selector,value) pairs, but the paper won't go there 🙈

there are four operations allowed on the database:

Selection - given a selector, the current associated value is returned. 👀
Assignment - a selector and a value are given, and the given value replaces the old value associated with that selector. ✏️
Creation - a new selector and an initial value are given, and a new (selector,initial value) entry gets added to the database. 🌱
Deletion - a selector is given, and the existing (selector,value) entry is removed from the database. 🗑️

note that value modification is limited to assignment 💅 functional modification requests, like "Change X to be Factorial(X)", are specifically ruled out. allowing them would force the use of system-wide synchronization interlocks, and nobody wants that 🔒

Consistency

the paper has to figure out how identical the copies of the database can be kept 🪞 because communication between DBMPs has inherent delay, it's impossible to guarantee the databases are identical at all times. so the goal is more modest: the copies should be "consistent" with each other. here's what that means: if update activity on an entry stops, and there's enough time for each DBMP to talk to all the other DBMPs, then that entry's state (whether it exists and what its value is) will be identical in every copy 🫶 eventually, everyone agrees. that's the energy we're going for 🌸

Timestamps

modifications can start at any of the DBMPs maintaining the database 🎉 those changes obviously have to get communicated to the other DBMPs. to keep things consistent, every DBMP has to make the same decision about which modification to a given entry counts as "final" 👑 it's desirable to pick the "most recent" change. but there's no absolute way to determine the order of events in a distributed system without a universal, always-accessible sequence number generator (a network time standard would do the trick) ⏰ so "most recent" can only be approximated.

the approximation works like this: every modification gets a timestamp, and every entry gets a timestamp too, which is the timestamp of the modification that set its current value.(1)

(1) Time is useful in this context because it has the desired properties of being monotonically increasing, and of being available with a reasonable degree of accuracy. Any other sequence numbering scheme with these properties can be used, "time-of-day" was chosen because it is simple to obtain. Its main faults are that it is often manually set (and thus prone to error), and it may stop during system service interruptions. As computer systems learn to adapt to a network environment, the use of an independent time source should become more common. This is beginning to happen with the TENEX sites on the ARPANET.

okay so here's the catch 🙃 two different DBMPs can't be guaranteed to hand out different times, so a "DBMP of origin" gets tacked onto each timestamp too 🧩 by (arbitrarily) ordering the DBMPs, you get a way to uniquely order timestamps. each timestamp is a pair (T,D), where T is a time and D is a DBMP identifier. for two timestamps (T1,D1) and (T2,D2):

(T1,D1)>(T2,D2) <=> (T1>T2) or (T1=T2 and D1>D2)
(T1,D1) is said to be "more recent" than (T2,D2)

if D1=D2 and T1=T2, the two modifications are assumed to be really two copies of the same modification request 🔁

to keep timestamps unique, each individual DBMP has to associate different times with different modifications 🕰️ that's totally doable, though how fine the unit of time is might limit how often a single DBMP site can make modifications.

so every entry in the database is now a triple:

E ::= (S,V,T), where
  S is the selector
  V is the associated value
  T is the timestamp (a Time,DBMP pair) of the last change to the
    entry

the job of each DBMP is to keep its copy up to date, based on the modification information it's received so far 📬 at the same time, it has to make sure each of its entries stays consistent with the entries of all the other DBMPs. this is tricky, because the order a DBMP receives modifications in can be really different from the order other DBMPs receive them in 🌀 the rest of the paper looks at the allowed database operations and how each one affects DBMP operation.

Assignment

let's start with assignment to an existing entry 💖 when an assignment starts (from a person or a process), the DBMP involved makes the change locally. then it creates a copy of the modified entry, plus a list of the DBMPs the change has to be sent to. as the modification gets delivered to each of the other DBMPs, they're removed from the list. once no DBMPs remain, the copy of the modification gets deleted 🧹 this distribution mechanism has to be error free: a DBMP must positively confirm receipt of a modification before it's removed from the recipients list.(2)

(2) [footnote text as it appears in the original is not reproduced in this part; see Part 2 if it continues there]

hmm wait, let me put that more carefully, because it's important 👀 the original footnote marker (2) sits right here, and the footnote text itself isn't in this part of the document. so just know the marker is there. ✨

when a DBMP receives an assignment modification (either locally or from another DBMP), it compares the timestamp of the modification with the timestamp of its copy of the entry, and keeps whichever one is more "recent" based on the ordering we already covered 🏆 so once every existing assignment to a given entry has been distributed to all the DBMPs, they're guaranteed to have the same "latest" value for that entry. no drama, just math 🧮💅

Creation

creation and deletion of entries are where things get messy 😩 the ability to create brand new, previously unknown entries means a DBMP has to be able to handle assignments to entries it doesn't know about yet 🤯 for example, say an entry XYZ is created by DBMP A, and this happens:

DBMP A tells DBMP B about the new entry 📣
later, B assigns a new value to XYZ
B then tells DBMP C about that assignment before C has heard from A about the creation 😳

now DBMP C has two options. it can save the assignment to XYZ until it hears about the creation, or it can just assume the creation is coming and use the "new" entry right away 🤷‍♀️ the paper says the second option is more in the spirit of keeping the database as "up-to-date" as possible, and it leads to no inconsistencies. so that's the move 💅

Deletion

okay bestie, deletion is the main problem for this whole database system 💀 if deletion means immediate removal from the database, things go sideways. check out this scenario:

XYZ is an entry known by all DBMPs.
XYZ is deleted at DBMP A.
XYZ is modified at DBMP B (before B is notified of the deletion
    by A).

and that's where part 1 ends 😭🎀 the paper is about to dig into what goes wrong here and how the design handles it, so stay tuned for part 2 🦋✨

Now, consider a third DBMP, C. C may hear from DBMP B before DBMP A, in which case XYZ ends up deleted at DBMP C. C may however hear from DBMP A before DBMP B. In this case, if C removes XYZ from its database, then the assignment to XYZ initiated by DBMP B will result in the re-creation of XYZ at DBMP C. To prevent this, C must remember that XYZ has been deleted until it is "safe" to completely forget about XYZ 🫶

(2) This same process (local modification and queuing for remote distribution) is, of course, performed for the other possible operations on the database. The details of how the local modification is done safely, how the messages are queued, how confirmation of delivery is done, etc., though important, will not be discussed here. The use of an addressee list attached to the modification to be delivered is conceptually easy to work with and not difficult to implement in practice.

One approach to this problem is to never actually remove an entry from the database. Deletion just marks the entry as being deleted by setting a "deleted" flag associated each entry. However, the problem of receiving assignments to deleted entries still exists. For example, DBMP A may receive an assignment from DBMP B to an entry which A has marked as deleted. DBMP A can not tell whether B has not heard about the deletion, or has heard about it but has also received a re-creation request for the entry, which hasn't reached DBMP A. To enable A to resolve such situations, we include another timestamp in all entries: the timestamp of the entry's creation. Thus in the above example, DBMP A can compare the creation timestamps of the assignment and the deleted entry. The one with the later creation timestamp is kept. Indeed, whenever a modification with an old creation timestamp is received, it can be ignored 💅

We now have a 5-tuple for entries and modifications:

    E ::= (S,V,F,CT,T)
    S is the selector
    V is the associated value
    F is the deleted/not-deleted flag
    CT is the timestamp of creation
    T is the timestamp of this (last) modification

Note that the values of the F, CT, and T components of a modification uniquely specify the type of modification. Thus only the 5-tuple to become the new entry for a selector, not the type of modification, need be communicated:

    F = not deleted, CT = T => creation
    F = not deleted, CT < T => assignment
    F = deleted => deletion

The mechanism described above handles all the desired operations on the distributed database in a way that guarantees the consistency of all copies. A modification to the database will take effect at each DBMP as soon as it receives the request from the DBMP originating the change ✨

A deficiency with this scheme is that deleted entries are never removed from the database. A method which permits "garbage collection" of deleted entries is discussed below.

Removal of Deleted Entries 🧹

The basic constraint is that a DBMP should not remove a deleted entry until it will never receive any assignments with the same selector (S) and the same or older create time (CT). If it fails to do this, then it will be unable to distinguish these "out of date" assignments from assignments to a newly created entry for the same S. To be able to do this, each DBMP must know for each deleted entry whether all other DBMPs have heard about the deletion. To accomplish this, each DBMP could notify the other DBMPs whenever it hears about a deletion. If these notifications are transmitted in order with the "normal" sequence of modifications, then upon receipt of such a notification a DBMP can be sure that the sending DBMP has delivered any outstanding assignments to the deleted entry, has marked it as deleted, and will not generate any new assignments to it. Keeping track of, and exchanging messages about, each individual deleted entry in this manner is, however, somewhat more elaborate than necessary 🙄

Having each DBMP deliver all its own modifications in sequential order (by timestamp) allows the following simplification. We have all DBMPs maintain a table of the timestamps of the last modification received from each other DBMP. Any DBMP, say A, is then guaranteed to have received all modifications originating at another DBMP, say B, which have timestamps earlier than (or equal to) the entry for B in A s copy of this table. If this table is exchanged between DBMPs, then all DBMPs would have a second N*N (N= number of DBMPs) table where entry (I,J) would be the timestamp of the last modification received by DBMP I from DBMP J. Thus DBMP A can remove a deleted entry whose deletion originated at DBMP K when all entries in the Kth column of this table at DBMP A are later than or equal to the timestamp of the deleted entry. When a DBMP receives a deletion modification, in addition to acting on it and acknowledging receipt of it, the DBMP should also send its table of last timestamps received to all other DBMPs. This is sent in a timestamped message which is queued with the "normal" modification messages 📬

A refinement to this approach, which reduces the amount of information exchanged and the size of the tables, is to have the DBMPs exchange only the oldest of the entries in the first table (of timestamps of last modifications received from other DBMPs). These would then be saved in a 1*N table, replacing the N*N table. A DBMP receiving a modification with a timestamp equal to or older than the oldest timestamp in its second table knows that the modification has been confirmed as being received by all other DBMPs. A deleted entry can thus be removed when its timestamp satisfies this condition. A DBMP would, upon receipt of a deletion modification, queue up a message with this "timestamp of oldest last modification received" for delivery to all other DBMPs 🌸

Summary of solution 📝

An entry in the database is a 5-tuple:

    (S,V,F,CT,T) where
    S is an selector used in all references to this entry.
    V is its present value.
    F is a deleted/undeleted flag.
    CT is the timestamp of the creation of this entry.
    T is the timestamp of the modification which set the current V
      and/or F of the entry.
    A timestamp is a pair (time,DBMP) where the DBMP identifies the
      site at which the time was generated, and the DBMPs are
      (arbitrarily) ordered, so that timestamps are completely
      ordered.

A modification is a pair (Set-of-DBMPs,Entry) where Set-of-DBMPs is the set of DBMPs to which the Entry has yet to be delivered.

An ordered (by timestamp) list of modifications is kept at each DBMP. The DBMP periodically attempts to deliver modification requests to those DBMPs which remain in the Set-of-DBMPs associated with each modification. Modification entries are removed from this list when they have been delivered to all DBMPs.

When a DBMP receives a modification request from another DBMP, it compares the timestamps of the request with the timestamps of the corresponding entry (if any) in its database, and acts upon or disregards the new entry accordingly.

Each DBMP keeps a vector of the timestamp (T) of the last modification received by it from each other DBMP.

When a DBMP receives a deletion modification, it sends a timestamped message to all other DBMPs containing the oldest timestamp in its vector of timestamps of last modification received. Each DBMP keeps a second vector of the last of these timestamps received from each other DBMP.

A deleted entry may be removed from the database when its timestamp (T) is older than all the timestamps in this second vector of timestamps received from other DBMPs.

Conclusion 👑

This paper has presented techniques by which a number of loosely coupled processes can maintain duplicate copies of a database, despite the unreliability of their only means of communication. The copies of the database can be kept "consistent'. However, it is possible for seemingly anomalous behavior to occur. For example, a user may assign a value to a selector using one DBMP, later use another DBMP and assign a new value, and still later find that the first value is the one that remains in the database. This can occur if the clocks used by the two DBMPs for their timestamps are sufficiently out of synchrony that the second assignment is timestamped as having taken place before the first assignment. To the extent that the communication paths can be made reliable, and the clocks used by the processes kept close to synchrony, the probability of seemingly strange behavior can be made very small. However, the distributed nature of the system dictates that this probability can never be zero 🥹

The major innovation presented here is the explicit representation of the time sequence of modifications through timestamps for both modifications and entries. This enables the each DBMP to select the same "most-recent" modification of an entry. In addition, timestamps enable the DBMPs to decide when a deleted entry is no longer referenced (i.e., still active anywhere) and can be deallocated. These techniques should have broader utility in building and modeling other systems of concurrent, cooperating processes 🦋

       [ This RFC was put into machine readable form for entry ]
       [ into the online RFC archives by Alex McKenzie with    ]
       [ support from GTE, formerly BBN Corp.            12/99 ]

📦 where this page comes from

ietf: rfc677 1975-01
💖 girlified by claude-haiku-5-5 on 2026-10-17: a language model's rewrite of the original, under its terms