RFC 3384

Lightweight Directory Access Protocol (version 3) Replication Requirements

📅 published October 2002, from ietf

RFC 3384: Lightweight Directory Access Protocol (version 3) Replication Requirements ✨

Okay bestie, let's get into it! 💖 This is RFC 3384, written in October 2002 by four people: E. Stokes from IBM, R. Weiser from Digital Signature Trust, R. Moats from Lemur Networks, and R. Huber from AT&T Laboratories. It's an Informational doc, which means it shares information with the Internet community and doesn't set any Internet standard. 🦋 The title block doesn't list any RFCs it obsoletes or updates, so it's standing all on its own.

Status of This Memo

This memo is here to give info to the Internet community 🌸. It is not a standard of any kind. Anyone can distribute it freely, with no limits.

Copyright Notice

Abstract

This doc talks about the basic requirements for replicating data that you can reach through LDAPv3 (the Lightweight Directory Access Protocol, version 3) 📚. Think of it as a meeting spot for the general replication requirements that let informational directories work together smoothly 🫶.

1    Introduction...................................................2
2    Terminology....................................................3
3    The Models.....................................................5
4    Requirements...................................................7
4.1  General........................................................7
4.2  Model..........................................................8
4.3  Protocol.......................................................9
4.4  Schema........................................................10
4.5  Single Master.................................................10
4.6  Multi-Master..................................................11
4.7  Administration and Management.................................11
4.8  Security......................................................12
5    Security Considerations.......................................13
6    Acknowledgements..............................................13
7    References....................................................13
A    Appendix A - Usage Scenarios..................................15
A.1  Extranet Example..............................................15
A.2  Consolidation Example.........................................15
A.3  Replication Heterogeneous Deployment Example..................16
A.4  Shared Name Space Example.....................................16
A.5  Supplier Initiated Replication................................16
A.6  Consumer Initiated Replication................................17
A.7  Prioritized attribute replication.............................17
A.8  Bandwidth issues..............................................17
A.9  Interoperable Administration and Management...................18
A.10 Enterprise Directory Replication Mesh.........................18
A.11 Failure of the Master in a Master-Slave Replicated Directory..19
A.12 Failure of a Directory Holding Critical Service Information...19
B    Appendix B - Rationale........................................20
B.1  Meta-Data Implications........................................20
B.2  Order of Transfer for Replicating Data........................20
B.3  Schema Mismatches and Replication.............................21
B.4  Detecting and Repairing Inconsistencies Among Replicas........22
B.5  Some Test Cases for Conflict Resolution in Multi-Master
     Replication...................................................23
B.6  Data Confidentiality and Data Integrity During Replication....27
B.7  Failover in Single-Master Systems.............................27
B.8  Including Operational Attributes in Atomic Operations.........29
     Authors' Addresses............................................30
     Full Copyright Statement......................................31

1. Introduction 🌈

Okay so, spreading directory info across the network is a total win for two reasons 💜: (1) it makes the directory more reliable through fault tolerance, and (2) it puts the directory content closer to the clients who actually use it. LDAP has become so popular as a way to access directory info that people now want to spread LDAP directory content around, both inside enterprises and across the Internet 🔥. Right now, though, LDAP doesn't define a replication mechanism. It only briefly mentions LDAP shadow servers (see [RFC2251]). A standard way to replicate directories across multiple vendors is basically essential if LDAP wants to keep winning in the market 👑.

This doc lays out the requirements for replication between multiple LDAP servers 🎀. RFC 2251 and RFC 2252 [RFC2252] already set the standards for how LDAP clients and servers talk to each other, but server-to-server communication needs extra requirements too. Some of those are covered here 🧚.

First, this doc explains the terminology it uses, and then it introduces the different replication models being considered. After that come the requirements, followed by security considerations. The reasoning behind the requirements lives in the Appendices, which keeps the requirements cleanly separate from their justification ✨.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 🫶.

2. Terminology 📖

Here are the terms this doc uses, bestie. Learn these and you'll be fluent in replication 💅:

Anonymous Replication - Replication where the endpoints know who each other are, but nobody authenticates anyone. Also called "unauthenticated replication" 🙈.
Area of replication - A whole chunk or part of a Directory Information Tree (DIT) that forms one distinct unit of data to replicate. An area of replication is defined by a replication base entry and includes all or some of the depending entries inside it, on a single server. It splits directory data into partitions whose propagation behavior can be configured separately from other partitions. Areas of replication can overlap or be nested. This is a subset of the definition of a "replicated area" in X.525 [X.525] 🌸.
Atomic operation - A set of changes to directory data that the LDAP standards guarantee will be treated as one unit: either all the changes get made, or all of them fail 💖.
Atomicity Information - Info about atomic operations that gets passed along as part of replication 🧚.
Conflict - A situation that happens when the same directory data gets changed on different directory servers before replication can sync them up. When the servers finally sync, their data doesn't match, and that mismatch is the conflict 💥.
Conflict resolution - Deterministic procedures used to settle change information conflicts that might pop up during replication ⭐.
Critical OID - Attributes or object classes that the replication agreement marks as critical to how the system runs. Changes that touch critical OIDs immediately kick off a replica cycle. A password or certificate is a good example 🔐.
Fractional replication - The ability to filter a subset of attributes for replication 🦋.
Incremental Update - An update that only includes the attributes or entries that changed 🌈.
Master Replica - A replica that LDAP operations can update directly. In a Master-Slave Replication system, the Master Replica is the only directly updatable replica in the replica-group 👑.
Master-Slave, or Single Master Replication - A replication model that assumes only one server, the master, gets LDAP write access to the replicated data. Master-Slave replication can be seen as a proper subset of multi-master replication 🎀.
Meta-Data - Data the replication system collects that describes the status or state of replication 📊.
Multi-Master Replication - A replication model where entries can be written and updated on any of several master replica copies, without having to talk to the other master replicas before the write or update happens 🔥.
One-way Replication - Synchronization in a single direction, where the authoritative source info gets handed over to a replica 🚀.
Partial Replication - Partial Replication is Fractional Replication, Sparse Replication, or both 🍓.
Propagation Behavior - How the synchronization process between a consumer and a supplier behaves 💫.
Replica - An instance of an area of replication on a server 🌸.
Replica-Group - The servers that hold instances of a particular area of replication. A server can belong to several replica-groups 🫂.
Replica (or Replication) Cycle - The interval during which update info gets exchanged between two or more replicas. It starts when someone tries to push data to, or pull data from, another replica or set of replicas, and it ends when the data has been exchanged successfully or an error shows up 🔁.
Replication - The process of syncing data spread across directory servers and sorting out update conflicts 💜.
Replication Agreement - A collection of info describing the parameters of replication between two or more servers in a replica-group 📝.
Replication Base Entry - The distinguished name of the root vertex of a replicated area 🌳.
Replication Initiation Conflict - A situation where two sources want to update the same replica at the same time. It's a whole drama 😭.
Replication Session - A session set up between two servers in a replica-group to pass update info as part of a replica cycle 🤝.
Slave (or Read-Only) Replica - A replica that LDAP requests can't update directly. Changes can only come in through replication from a master replica. Read-only replicas can show up in both single-master and multi-master systems 🪞.
Sparse Replication - The ability to filter some subset of entries (other than the complete collection) in an area of replication 🌙.
Topology - The shape of the directed graph that describes the relationships between replicas 🗺️.
Two-way Replication - Synchronization where change info flows in both directions between two replicas ↔️.
Unauthenticated Replication - See Anonymous Replication 🙋.
Update Propagation - The protocol-based process by which directory replicas get reconciled 💅.

3 The Models ✨

okay so the whole goal here 🎯 is an interoperable LDAPv3 directory synchronization protocol that is simple, efficient, and flexible 💅. it has to support both multi-master and master-slave replication, and it has to work for the Internet and for enterprise environments 🌈.

there are five data consistency models 🫶

Model 1: Transactional Consistency -- environments that exhibit all four of the ACID properties (Atomicity, Consistency, Isolation, Durability) [ACID].

Model 2: Eventual (or Transient) Consistency -- environments where definite knowledge of the topology is provided through predetermined replication agreements. examples include X.500 Directories (the X.500 model is single-master only) [X.501, X.525], Bayou [XEROX], and NDS (Novell Directory Services) [NDS]. in this model, every update propagates to every replica that it can reach via a path of stepwise eventual connectivity 🦋

Model 3: Limited Effort Eventual (or Probabilistic) Consistency -- environments that provide a statistical probability of convergence with knowledge of topology. an example is the Xerox Clearinghouse [XEROX2]. this model is similar to "Eventual Consistency", except where replicas may purge updates. purging drops propagation changes when some replica time boundary is exceeded, so some changes end up replicated to only a portion of the topology 🙈. transactional consistency is not preserved, though some weaker constraints on consistency are available.

Model 4: Loosest Consistency -- environments where information is provided from an opportunistic or simple cache until stale. complete knowledge of topology may not be shared among all replicas.

Model 5: Ad hoc -- a copy of a data store where no follow up checks are made for the accuracy/freshness of the data 💀

consistency models 1, 2 and 3 involve the use of prearranged replication agreements among servers. model 1 may simplify support for atomicity in multi-master systems, but the added complexity of the distributed 2-phase commit required for Model 1 is significant. therefore, model 1 will not be considered at this time. models 4 and 5 involve unregistered replicas that "pull" updates from another directory server without that server's knowledge. these models violate a directory's security policies 🚨

models 2 and 3 illustrate two replication scenarios that must be handled: policy configuration through security management parameters (model 2), and hosting relatively static data and address information as in white-pages applications (model 3). so replication requirements are presented for models 2 and 3 ✨

interoperability among directories using LDAP replication may be limited for implementations that add semantics beyond those specified by the LDAP core documents (RFC 2251-2256, 2829, 2830). also, the "core" specifications include numerous features that are not mandatory-to-implement (e.g., RECOMMENDED or OPTIONAL). there are also numerous elective extensions. so LDAP replication interoperability between independent implementations that support different options may be limited 🥲. use of applicability statements to improve interoperability in particular application spaces is RECOMMENDED 🎀

4 Requirements 💜

4.1 General

G1. LDAP Replication MUST support models 2 (Eventual Consistency) and 3 (Limited Effort Eventual Consistency) above.

G2. LDAP Replication SHOULD NOT preclude support for model 1 (Transactional Consistency) in the future.

G3. LDAP replication SHOULD have minimal impact on system performance.

G4. The LDAP Replication Standard SHOULD NOT limit the replication transaction rate.

G5. The LDAP replication standard SHOULD NOT limit the size of an area of replication or a replica.

G6. Meta-data collected by the LDAP replication mechanism MUST NOT grow without bound.

G7. All policy and state data pertaining to replication MUST be accessible via LDAP.

G8. LDAP replication MUST be capable of replicating the following:

all userApplication attribute types
all directoryOperation and distributedOperation attribute types defined in the LDAP "core" specifications (RFCs 2251-2256, 2829-2830)
attribute subtypes
attribute description options (e.g., ";binary" and Language Tags [RFC2596])

G9. LDAP replication SHOULD support replication of directoryOperation and distributedOperation attribute types defined in standards track LDAP extensions.

G10. LDAP replication MUST NOT support replication of dsaOperation attribute types, as such attributes are DSA-specific 🙅‍♀️

G11. The LDAP replication system should limit impact on the network by minimizing the number of messages and the amount of traffic sent.

4.2 Model

M1. The model MUST support the following triggers for initiation of a replica cycle:

a) A configurable set of scheduled times
b) Periodically, with a configurable period between replica cycles
c) A configurable maximum amount of time between replica cycles
d) A configurable number of accumulated changes
e) Change in the value of a critical OID
f) As the result of an automatic rescheduling after a replication initiation conflict
g) A manual request for immediate replication

with the exception of manual request, the specific trigger(s) and related parameters for a given server MUST be identified in a well-known place defined by the standard, e.g., the Replication Agreement(s).

M2. The replication model MUST support both master-slave and multi-master relationships.

M3. An attribute in an entry MUST eventually converge to the same set of values in every replica holding that entry.

M4. LDAP replication MUST encompass schema definitions, attribute names and values, access control information, knowledge information, and name space information.

M5. LDAP replication MUST NOT require that all copies of the replicated information be complete, but MAY require that at least one copy be complete. the model MUST support Partial Replicas 🧩

M6. The determination of which OIDs are critical MUST be configurable in the replication agreement.

M7. The parameters of the replication process among the members of the replica-group, including access parameters, necessary authentication credentials, assurances of confidentiality (encryption), and area(s) of replication, MUST be defined in a standard location (e.g., the replication agreements).

M8. The replication agreements SHOULD accommodate multiple servers receiving the same area of replication under a single predefined agreement.

M9. LDAP replication MUST provide scalability to both enterprise and Internet environments, e.g., an LDAP server must be able to provide replication services to replicas within an enterprise as well as across the Internet 🌍

M10. While different directory implementations can support different/extended schema, schema mismatches between two replicating servers MUST be handled. one way of handling such mismatches might be to raise an error condition.

M11. There MUST be a facility that can update, or totally refresh, a replica-group from a standard data format, such as LDIF format [RFC2849].

M12. An update received by a consumer more than once MUST NOT produce a different outcome than if the update were received only once 🔁

4.3 Protocol

P1. The replication protocol MUST provide for recovery and rescheduling of a replication session due to replication initiation conflicts (e.g., consumer busy replicating with other servers) and or loss of connection (e.g., supplier cannot reach a replica).

P2. LDUP replication SHOULD NOT send an update to a consumer if the consumer has previously acknowledged that update.

P3. The LDAP replication protocol MUST allow for full update to facilitate replica initialization and reset loading, utilizing a standardized format such as LDIF [RFC2849] format.

P4. Incremental replication MUST be allowed.

P5. The replication protocol MUST allow either a master or slave replica to initiate the replication process.

P6. The protocol MUST preserve atomicity of LDAP operations as defined in RFC2251 [RFC2251]. in a multi-master environment this may lead to an unresolvable conflict. MM5 and MM6 discuss how to handle this situation 😬

P7. The protocol MUST support a mechanism to report schema mismatches between replicas discovered during a replication session.

4.4 Schema

SC1. A standard way to determine what replicas are held on a server MUST be defined.

SC2. A standard schema for representing replication agreements MUST be defined.

SC3. The semantics associated with modifying the attributes of replication agreements MUST be defined.

SC4. A standard method for determining the location of replication agreements MUST be defined.

SC5. A standard schema for publishing state information about a given replica MUST be defined.

SC6. A standard method for determining the location of replica state information MUST be defined.

SC7. It MUST be possible for appropriately authorized administrators, regardless of their network location, to access replication agreements in the DIT.

SC8. Replication agreements of all servers containing replicated information MUST be accessible via LDAP.

SC9. An entry MUST be uniquely identifiable throughout its lifetime 🪪

4.5 Single Master

SM1. A Single Master system SHOULD provide a fast method of promoting a slave replica to become the master replica 👑

SM2. The master replica in a Single Master system SHOULD send all changes to read-only replicas in the order in which the master applied them.

4.6. Multi-Master 🦋

okay so multi-master is where it gets spicy 🌶️ because EVERY server can be a boss, and that means we need rules so nobody's fighting over the same entry. here's the vibe:

MM1. The replication protocol SHOULD NOT saturate the network with redundant or unnecessary entry replication. 🙅‍♀️ no spam, bestie.

MM2. The initiator MUST be allowed to determine whether it will become a consumer or supplier during the synchronization startup process. 🎀 you get to pick your role!

MM3. During a replica cycle, it MUST be possible for the two servers to switch between the consumer and supplier roles. 🔄 role swap, iconic.

MM4. When multiple master replicas want to start a replica cycle with the same replica at the same time, the model MUST have an automatic and deterministic mechanism for resolving or avoiding replication initiation conflict. 🚦 when two people try to start a sync at the same time, there has to be a built-in way to sort it out, no drama.

MM5. Multi-master replication MUST NOT lose information during replication. 😭 this one is SERIOUS. If conflict resolution would result in the loss of directory information, the replication process MUST store that information, notify the administrator of the nature of the conflict and the information that was lost, and provide a mechanism for possible override by the administrator. so nothing gets quietly deleted, the admin gets told, and they can override it if they need to.

MM6. Multi-master replication MUST support convergence of the values of attributes and entries. ✨ everyone ends up agreeing eventually. Convergence may result in an event as described in MM5.

MM7. Multi-master conflict resolution MUST NOT depend on the in-order arrival of changes at a replica to assure eventual convergence. 🌀 changes can show up out of order and it still has to work out.

MM8. Multi-master replication MUST support read-only replicas as well as read-write replicas. 📖👀 both kinds of replicas get a seat at the table.

4.7. Administration and Management 🧚

the admin side is giving organized. here's what the admins can count on:

AM1. Replication agreements MUST allow the initiation of a replica cycle to be administratively postponed to a more convenient period. ⏰ hit snooze when it's not a good time.

AM2. Each copy of a replica MUST maintain audit history information of which servers it has replicated with and which servers have replicated with it. 📒 receipts for everything, always.

AM3. Access to replication agreements, topologies, and policy attributes MUST be provided through LDAP. 🗝️

AM4. The capability to check the differences between two replicas for the same information SHOULD be provided. 🔍 spot the difference, but make it official.

AM5. A mechanism to fix differences between replicas without triggering new replica cycles SHOULD be provided. 🛠️ fix it without starting the whole thing over again.

AM6. The sequence of updates to access control information (ACI) and the data controlled by that ACI MUST be maintained by replication. 🔐 the order of permission changes and the data they guard has to stay in sync.

AM7. It MUST be possible to add a 'blank' replica to a replica-group, and force a full update from (one of) the Master(s), for the purpose of adding a new directory server to the system. 🆕 new server joins the squad, and it gets a full copy from one of the Masters.

AM8. Vendors SHOULD provide tools to audit schema compatibility within a potential replica-group. 🧰 vendors, please bring the tools to check that everyone's schema is compatible before they join.

4.8. Security 🔒

this part is SO important, bestie. no cap. 💀

The terms "data confidentiality" and "data integrity" are defined in the Internet Security Glossary [RFC2828]. 📚 go check the glossary if you want the exact meanings.

S1. The protocol MUST support mutual authentication of the source and the replica directories during initialization of a replication session. 🤝 both sides prove who they are.

S2. The protocol MUST support mutual verification of authorization of the source to send and the replica to receive replicated data during initialization of a replication session. 🪪 both sides also check that the other is allowed to send or receive.

S3. The protocol MUST also support the initialization of anonymous replication sessions. 👻 anonymous sessions have to be possible too.

S4. The replication protocol MUST support transfer of data with data integrity and data confidentiality. 🔐 the data arrives unchanged and private.

S5. The replication protocol MUST support the ability during initialization of a replication session for an authenticated source and replica to mutually decide to disable data integrity and data confidentiality within the context of and for the duration of that particular replication session. 🙈 two authenticated buddies can agree to turn off integrity and confidentiality, but only for that one session.

S6. To promote interoperability, there MUST be a mandatory-to-implement data confidentiality mechanism. 🤝 everyone has to support at least one confidentiality mechanism so they can talk to each other.

S7. The transport for administrative access MUST permit assurance of the integrity and confidentiality of all data transferred. 🚚 admin traffic has to be protected too.

S8. To support data integrity, there must be a mandatory-to-implement data integrity mechanism. 🧾 same deal for integrity: there has to be one mechanism everyone implements.

5. Security Considerations 💀

this part is dramatic, and it's giving "pay attention" 🚨

This document includes security requirements (listed in section 4.8 above) for the replication model and protocol. As noted in Section 3, interoperability may be impacted when replicating among servers that implement non-standard extensions to basic LDAP semantics. 🧩 if your server adds its own weird LDAP extensions, it can break interop, so don't do that without a reason.

Security-related and general LDAP interoperability will be significantly impacted by the degree of consistency with which implementations support existing and future standards detailing LDAP security models, such as a future standard LDAP access control model. 🔗 the more consistently everyone follows the standards, the safer and more compatible everything is.

6. Acknowledgements 🫶

This document is based on input from IETF members interested in LDUP Replication. 💜 thank you to everyone who helped out!

7. References 📚

[ACID] T. Haerder, A. Reuter, "Principles of Transaction-Oriented Database Recovery", Computing Surveys, Vol. 15, No. 4 (December 1983), pp. 287-317.

[NDS] Novell, "NDS Technical Overview", 104-000223-001, http://developer.novell.com/ndk/doc/ndslib/dsov_enu/data/h6tvg4z7.html, September, 2000.

[RFC2119] Bradner, S., "Key Words for Use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2251] Wahl, M., Howes, T. and S. Kille, "Lightweight Directory Access Protocol", RFC 2251, December 1997.

[RFC2252] Wahl, M., Coulbeck, A., Howes, T. and S. Kille, "Lightweight Directory Access Protocol (v3): Attribute Syntax Definitions", RFC 2252, December 1997.

[RFC2253] Kille, S., Wahl, M. and T. Howes, "Lightweight Directory Access Protocol (v3): UTF-8 String Representation of Distinguished Names", RFC 2253, December 1997.

[RFC2254] Howes, T., "The String Representation of LDAP Search Filters", RFC 2254, December 1997.

[RFC2255] Howes, T. and M. Smith, "The LDAP URL Format", RFC 2255, December 1997.

[RFC2256] Wahl, M., "A Summary of the X.500(96) User Schema for use with LDAPv3", RFC 2256, December 1997.

[RFC2596] Wahl, M. and T. Howes, "Use of Language Codes in LDAP", RFC 2596, May 1999.

[RFC2828] Shirey, R. "Internet Security Glossary", FYI 36, RFC 2828, May 2000.

[RFC2829] Wahl, M., Alvestrand, H., Hodges, J. and R. Morgan, "Authentication Methods for LDAP", RFC 2829, May 2000.

[RFC2830] Hodges, J., Morgan, R. and M. Wahl, "Lightweight Directory Access Protocol (v3): Extension for Transport Layer Security", RFC 2830, May 2000.

[RFC2849] Good, G., "The LDAP Data Interchange Format (LDIF)", RFC 2849, June 2000.

[X.501] ITU-T Recommendation X.501 (1993), | ISO/IEC 9594-2: 1993, Information Technology - Open Systems Interconnection - The Directory: Models.

[X.525] ITU-T Recommendation X.525 (1997), | ISO/IEC 9594-9: 1997, Information Technology - Open Systems Interconnection - The Directory: Replication.

[XEROX] C. Hauser, "Managing update conflicts in Bayou, a weakly connected replicated storage system". Palo Alto, CA: Xerox PARC, Computer Science Laboratory; 1995 August; CSL-95-4.

[XEROX2] Alan D. Demers, Mark Gealy, Daniel Greene, Carl Hauser, Wesley Irish, John Larson, Sue Manning, Scott Shenker, Howard Sturgis, Daniel Swinehart, Douglas Terry, Don Woods, "Epidemic Algorithms for Replicated Database Maintenance". Palo Alto, CA, Xerox PARC, January 1989.

A. APPENDIX A - Usage Scenarios 🌈

The following directory deployment examples are intended to validate our replication requirements. A heterogeneous set of directory implementations is assumed for all the cases below. This material is intended as background; no requirements are presented in this Appendix. 🧠 this part is just story time to show the requirements in action.

A.1. Extranet Example 🤝

A company has a trading partner with whom it wishes to share directory information. This information may be as simple as a corporate telephone directory, or as complex as an extranet workflow application. For performance reasons, the company wishes to place a replica of its directory within the Partner Company, rather than exposing its directory beyond its firewall. 🧱