RFC 2334
Server Cache Synchronization Protocol (SCSP)
RFC 2334: Server Cache Synchronization Protocol (SCSP) 🎀 (part 1 of 9)
- Working group: Network Working Group 💖
- Authors: J. Luciani (Bay Networks), G. Armitage (Bellcore), J. Halpern (Newbridge), N. Doraswamy (Bay Networks) ✨
- Category: Standards Track 👑
- Date: April 1998 🌸
- Obsoletes / Updates: the title block doesn't list anything here 🫶
Status of this Memo 📜
okay so this document is an Internet standards track protocol spec for the Internet community. it's also literally asking for discussion and suggestions for improvements 🙈 to check where this protocol is in standardization, look at the "Internet Official Protocol Standards" (STD 1). distribution of this memo is unlimited, so everyone can read it 🌈
Copyright Notice 💅
Copyright (C) The Internet Society (1998). All Rights Reserved.
Abstract 🦋
this doc describes the Server Cache Synchronization Protocol (SCSP) and explains it through how it's used in Non Broadcast Multiple Access (NBMA) networks 💜 a pretty straightforward version of the same idea also works on BMA networks, which is iconic 🔥 SCSP is trying to solve the big general cache synchronization/cache-replication problem for distributed protocol entities. in this doc though, it's framed around the client/server paradigm. the distributed server entities are bound to a Server Group (SG) somehow, and they want to sync the contents (or just part of the contents) of their caches. those caches hold info about the state of the clients they serve ✨
1. Introduction 🧚
the keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they show up in this document, are to be read as described in [10]. 📖
it's kind of an obvious goal for any protocol to not be stuck with a single point of failure, like one server in a client/server setup 🙄 but even with redundant servers, you still have the cache synchronization problem. like, when one server notices a change in cache state, it has to pass that news along to every server that's actively mirroring that state info 😭 and it has to happen quickly without wrecking the servers with resource strain 💀 if the state in the server cache is the state of the server's clients, then it's also super desirable to keep the burden on the client light, so clients don't need full knowledge of every server they might use 🍓 still, the sync mechanism shouldn't block a client from reaching several (or all) servers. and obviously the whole thing has to be reasonably scalable, able to use some auto-configuration service, and flexible enough for a wide range of authentication methods 🔥
this doc describes the Server Cache Synchronization Protocol (SCSP). SCSP solves the general server synchronization/cache-replication problem and handles all the issues we just talked about 💖 it syncs caches (or parts of caches) for a set of server entities of one particular protocol, and those servers are bound to a Server Group (SG) somehow. for example, all NHRP servers in a Logical IP Subnet (LIS)[1] 🫶 the client/server protocol a server uses is identified by a Protocol ID (PID). SGs get an ID too, which is literally called a SGID 😂 so the PID/SGID combo tells you both which client/server protocol the SG's servers are syncing for and which instance of that protocol it is ✨ that means you can have multiple instances of the same protocol running at once, each synced independently 🦋 an example of info that needs syncing shows up in NHRP[2] over IP, where it's the registered clients' IP to NBMA mappings in the SG LIS 💜
the easiest way to get SCSP is to know that its algorithm is pretty similar to the one in OSPF[3] 🔥 if you want the deeper details on SCSP's tradeoffs and reliability, you should check out the Hello, Database Synchronization, and Flooding Procedures in OSPF [3] 🙈
as we'll see later, the protocol has three phases. the first is the super brief hello phase, where two devices figure out they can talk to each other 💅 then comes database synchronization. SCSP assumes that until new info shows up, two entities already have the same data, and the database sync phase is what makes that true 🎀
in database sync, the two neighbors swap summary info about each entry in their databases 🧚 summaries are used because the database itself can be really big 🍓 from those summaries, the neighbors can tell whether there's info each one needs from the other. if so, it gets requested and handed over ✨ by the end of this phase, both neighbors have the same data in their databases 💖
after that, the entities enter and stay in the flooding state 🌊 in flooding, any new info that's learned gets sent to all neighbors, except the one (if any) it came from 🦋 that way all new info spreads to every node, and everyone ends up knowing the same stuff 👑 flooding is reliable on each link, so no pattern of low-rate packet loss can cause a disruption 💅 (obviously, a high enough rate of packet loss will bring the whole neighbor relationship down, but if the link isn't working, that's the outcome you want anyway 💀)
the database sync procedure runs every time a link comes up, so all the participating nodes robustly end up with all the available info 🫶 it recovers from partitions properly and copes with other failures too 🌈
the SCSP spec isn't useful on its own 🙅 it has to be paired with an SCSP Protocol Specific specification, which says how a given protocol uses the sync primitives SCSP provides 💜 those specs get written in separate documents, e.g., [8] [9] ✨
2. Overview 🌸
SCSP puts no topology requirements on the SG 💖 the resulting graph still has to span the set of servers being synced, obviously 🙄 SCSP borrows its cache distribution mechanism from the link state protocols [3,4] 🔥 unlike those, though, there's no mandatory Shortest Path First (SPF) calculation, and SCSP adds no memory requirements beyond what's needed to save the cached info, which would exist no matter what sync technology you use 🦋
to set up a frame of reference for the rest of the doc, we introduce three terms: Local Server (LS), Directly Connected Server (DCS), and Remote Server (RS) 🧚 the LS is the server under scrutiny. every statement is made from the LS's point of view when we talk about SCSP 👑 a DCS is a server directly connected to the LS, e.g., there's a VC between the LS and the DCS 💅 so every server is a DCS from the view of every other server that connects to it directly, and every server is an LS with zero or more DCSs connected to it 🎀 from the LS's perspective, an RS is a server that's separate from the LS and not directly connected to it. an RS is always two or more hops away from an LS, whereas a DCS is always exactly one hop away ✨
SCSP has three sub-protocols: the "Hello" protocol, the "Cache Alignment" protocol, and the "Cache State Update" protocol 💜 the "Hello" protocol figures out whether a DCS is operational and whether the LS-DCS connection is bidirectional, unidirectional, or non-functional 🫶 the "Cache Alignment" (CA) protocol lets an LS sync its entire cache with its DCSs' caches 🌈 the "Cache State Update" (CSU) protocol updates the state of cache entries on servers for a given SG 🍓 Sections 2.1, 2.2, and 2.3 go deeper into the Hello, CA, and CSU protocols and the messages they use 😭
SCSP sync happens per protocol instance 🦋 so a separate SCSP instance runs for each instance of the given protocol running on a given box 💖 the protocol is identified in SCSP by a Protocol ID, and the instance is identified by a Server Group ID (SGID). the PID/SGID pair uniquely identifies one SCSP instance ✨ usually this isn't a big deal, because it's rare for lots of instances of a distributed protocol that needs cache sync to run on the same physical box 🙈 but when it does happen, there's a mechanism called the Family ID (briefly described in the Hello Protocol) that cuts maintenance traffic way down at little real cost in control 🔥 whether to use the Family ID mechanism is up to the protocol using SCSP, and it'll be fully defined in that protocol's SCSP-specific spec 💅
+---------------+
| |
+------->| DOWN |<-------+
| | | |
| +---------------+ |
| | ^ |
| | | |
| | | |
| | | |
| @ | |
| +---------------+ |
| | | |
| | WAITING | |
| +--| |--+ |
| | +---------------+ | |
| | ^ ^ | |
| | | | | |
| @ | | @ |
+---------------+ +---------------+
| BIDIRECTIONAL |---->| UNIDIRECTIONAL|
| | | |
| CONNECTION |<----| CONNECTION |
+---------------+ +---------------+
Figure 1: Hello Finite State Machine (HFSM)Figure 1 is the Hello state machine, so you can see how the DOWN, WAITING, BIDIRECTIONAL, and UNIDIRECTIONAL states hook together 🌸
2.1. Hello Protocol 👋
Okay so, "Hello" messages are the vibe check 💅 They tell an LS whether a DCS is operational, and whether the connection between them is bidirectional, unidirectional, or totally non-functional 💀 To do this, every LS MUST periodically send a Hello message to its DCSs 🔥
An LS has to be configured with a list of NBMA addresses. These are the addresses of peer servers in a SG that the LS wants a direct connection with so it can run SCSP 🦋 Basically, these addresses belong to would-be DCSs 🎀 How the LS gets configured with these NBMA addresses is out of scope for this document 🙈 One possible way is an autoconfiguration server, though, so there's that ✨
Each LS has a Hello Finite State Machine (HFSM) for each of its DCSs (see Figure 1) in a given SG 🧚 The HFSM is the one keeping tabs on the connectivity between the servers, so it's giving main character energy 👑
The HFSM starts in the "Down" State and moves to the "Waiting" State once NBMA level connectivity has been established 🌸 Once it's in Waiting, the LS starts sending Hello messages to the DCS. Each Hello message includes a Sender ID, which is set to the LS's ID (LSID), plus zero or more Receiver IDs. Those Receiver IDs are the DCSs the LS has recently heard a Hello from (more on that below) 💜 It also includes a HelloInterval and a DeadFactor, which we'll get to in a sec 🍓 At this point, the DCS may or may not already be sending its own Hello messages to the LS. No pressure 😭
When the LS gets a Hello message from one of its DCSs, it checks whether its LSID shows up in one of the Receiver ID fields of that message 🔍 It also saves the Sender ID from that Hello. If the LSID is in one of the Receiver ID fields, the LS moves the HFSM to the Bidirectional Connection state. If it isn't, the LS moves the HFSM to the Unidirectional Connection state 🦋 The saved Sender ID is the DCS's ID (DCSID). Before the LS next sends its own Hello to that DCS, it checks the saved DCSID against the list of Receiver IDs the LS uses in its own Hello messages. If the DCSID isn't on that list, it gets added right before the Hello goes out ✨
Now the timing stuff 🕰️ The HelloInterval says how many seconds pass between consecutive Hello messages from the server that's sending the "current" Hello. So if the gap between Hellos from a DCS is longer than the HelloInterval that DCS advertised, the next Hello counts as late from the LS's point of view 😬 If the LS doesn't receive a Hello that has its LSID in one of the Receiver ID fields within HelloInterval*DeadFactor seconds (where the DeadFactor was advertised by the DCS in an earlier Hello), then the LS MUST consider the DCS to be stalled 💀 At that point, one of two things happens:
- If any Hello messages arrived during the last HelloInterval*DeadFactor seconds, the LS should move the HFSM for that DCS to the Unidirectional Connection State.
- Otherwise, the LS should move the HFSM for that DCS to the Waiting State and remove the DCSID from the Receiver ID list.
Quick note: the Hello Protocol runs on a per PID/SGID basis 🌈 So imagine two servers (one in SG A and one in SG B) tied to NBMA address X, plus another two servers (again one in SG A and one in SG B) tied to NBMA address Y. If there's a suitable point-to-point VC between X and Y, then there are two HFSMs running on each side of that VC, one per PID/SGID. Two HFSMs each side, it's giving multitasking 💅
Hello messages carry a list of Receiver IDs instead of just one, so the protocol can make use of point to multipoint connections 🫶 Each DCS has its own HFSM, but the LS MUST send only a single Hello message to its DCSs that are attached as leaves of a point to multipoint connection 🔥 It does this by putting the DCSIDs into the Receiver ID list of its next Hello. Only DCSIDs from non-stalled DCSs that the LS has heard a Hello from get included 🌸
Any abnormal event, like receiving a malformed SCSP message, sends the HFSM to the Waiting State. But losing NBMA connectivity sends it to the Down State instead 😭 Until the HFSM is in the Bidirectional Connection State, any properly formed SCSP messages that aren't Hello messages MUST be ignored 🙈 (This is for the case where, for example, there's a point to multipoint connection involved.)
+------------+
| |
+--->| DOWN |
| | |
| +------------+
| |
^ |
| @
| +------------+
| |Master/Slave|
|-<--| |<---+
| |Negotiation | |
| +------------+ |
| | |
^ | ^
| @ |
| +------------+ |
| | Cache | |
|-<--| |-->-|
| | Summarize | |
| +------------+ |
| | |
^ | ^
| @ |
| +------------+ |
| | Update | |
|-<--| |-->-|
| | Cache | |
| +------------+ |
| | |
^ | ^
| @ |
| +------------+ |
| | | |
+-<--| Aligned |-->-+
| |
+------------+Figure 2: Cache Alignment Finite State Machine
2.2. Cache Alignment Protocol 🧚
"Cache Alignment" (CA) messages are how an LS syncs its cache with the cache of each of its DCSs 💖 So when an LS is booting up, CA messages let it line up with every DCS it has. A CA message is a CA header followed by zero or more Cache State Advertisement Summary records (CSAS records) 🎀
Each LS has a Cache Alignment Finite State Machine (CAFSM) for each of its DCSs, tracked on a per PID/SGID basis (see Figure 2) 🌸 The CAFSM is the one watching the cache alignment status between the servers. It starts in the Down State. It's tied to an HFSM, and once that HFSM reaches the Bidirectional State, the CAFSM moves to the Master/Slave Negotiation State 🦋 That state decides whether the LS or the DCS becomes the master of the cache alignment process. Basically, the master sets the tempo for the whole thing 👑
When the LS's CAFSM hits the Master/Slave Negotiation State, the LS sends a CA message to the DCS tied to that CAFSM 💅 The format of CA messages is described in Section B.2.1. The very first CA message the LS sends has no CSAS records. Its CA header holds the LSID in the Sender ID field, the DCSID in the Receiver ID field, a CA sequence number, and three bits: the M (Master/Slave) bit, the I (Initialization of master) bit, and the O (More) bit ✨ In the first CA message the LS sends to a particular DCS, the M, O, and I bits are all set to one 🔥 If the LS doesn't get a CA message back from the DCS within CAReXmtInterval seconds, it resends the CA message it just sent 🕰️ It keeps doing this until the CAFSM moves to the Cache Summarize State, or until the HFSM moves out of the Bidirectional State. And any time the HFSM leaves the Bidirectional State, the CAFSM goes back to the Down State 😭
2.2.1. Master Slave Negotiation State 💜
When the LS gets a CA message from the DCS while it's in the Master/Slave Negotiation State, the role the LS plays depends on how it processes the packet 🔍
- If the CA from the DCS has the M, I, and O bits all set to one, there are no CSAS records in the CA message, and the Sender ID in the DCS's CA message is larger than the LSID, then:
a) The timer counting down the CAReXmtInterval gets stopped ⏹️ b) The CAFSM for that DCS moves to the Cache Summarize State, and the LS takes on the role of slave 🌈 c) The LS adopts the CA sequence number it got in the CA message as its own CA sequence number 🎀 d) The LS sends a CA message to the DCS that's set up like this: the M and I bits are set to zero, the Sender ID field is set to the LSID, the Receiver ID field is set to the DCSID, and the CA sequence number is set to the one from the DCS's CA message. If there are CSAS records to send (meaning the LS's cache isn't empty) and they won't all fit in this CA message, the O bit is set to one and the initial batch of CSAS records goes in the message. Otherwise, the O bit is set to zero, and any CSAS records that need sending get included in the message 🍓
- If the CA message from the DCS has the M and I bits off, and the Sender ID in the DCS's CA message is smaller than the LSID, then:
a) The timer counting down the CAReXmtInterval gets stopped ⏹️ b) The CAFSM for that DCS moves to the Cache Summarize State, and the LS takes on the role of master 👑 c) The LS must process the received CA message. An explanation of CA message processing is given below 💅 d) The LS sends a CA message to the DCS that's set up like this: the M bit is set to one, the I bit is set to zero, the Sender ID field is set to the LSID, the Receiver ID field is set to the DCSID, and the LS's current CA sequence number gets incremented by one and placed in the message. If the LS has any CSAS records to send to the DCS (meaning its cache isn't empty), the O bit is set to one and the initial set of CSAS records goes in the CA message the LS is sending 🦋
- Otherwise, the packet must be ignored 🙈
2.2.2 The Cache Summarize State ✨
Okay bestie, this is where the real drama happens 💅 At any given moment, the master or the slave can have at most one CA message out in the wild 🦋 Once the LS's CAFSM hits the Cache Summarize State, the CA message ping-pong goes like this:
- If the LS gets a CA message with the M bit set incorrectly (like, the M bit is set in the CA of the DCS and the LS is master) or the I bit is set, the CAFSM transitions back to the Master/Slave Negotiation State 🙈
- If the LS is master and the LS receives a CA message with a CA sequence number that is one less than the LS's current CA sequence number, the message is a duplicate and it MUST be discarded 🗑️
- If the LS is master and the LS receives a CA message with a CA sequence number that is equal to the LS's current CA sequence number, the CA message MUST be processed. We'll explain "CA message processing" in a sec. Since the LS got the CA message from the DCS, these things happen:
- a) The timer counting down the CAReXmtInterval is stopped.
- b) The LS must process any CSAS records in the received CA message.
- c) Increment the LS's CA sequence number by one.
- d) The cache exchange keeps going like this:
- If the LS has no more CSAS records to send and the received CA message has the O bit off, the CAFSM transitions to the Update Cache State.
- If the LS has no more CSAS records to send and the received CA message has the O bit on, the LS sends back a CA message (with the new CA sequence number) that has no CSAS records and the O bit off. Then reset the timer counting down the CAReXmtInterval.
- If the LS still has more CSAS records to send, the LS sends its next CA message with its next set of CSAS records. If that's the LS's last set of CSAS records, the O bit is set off. Otherwise the O bit is set on. Then reset the timer counting down the CAReXmtInterval.
- If the LS is slave and the LS receives a CA message with a CA sequence number that is equal to the LS's current CA sequence number, the CA message is a duplicate and the LS MUST resend the CA message it had just sent to the DCS 🔁
- If the LS is slave and the LS receives a CA message with a CA sequence number that is one more than the LS's current CA sequence number, the message is valid and MUST be processed. Again, "CA message processing" is explained below. Since the LS got the CA message from the DCS, these things happen:
- a) The LS must process any CSAS records in the received CA message.
- b) Set the LS's CA sequence number to the CA sequence number in the CA message.
- c) The cache exchange keeps going like this:
- If the LS had just sent a CA message with the O bit off and the received CA message has the O bit off, the CAFSM transitions to the Update Cache State and the LS sends a CA message with no CSAS records and the O bit off.
- If the LS still has CSAS records to send, the LS MUST send a CA message with CSAS records in it.
- a) If the message the LS is sending to the DCS does not contain the last CSAS records the LS needs to send, the CA message is sent with the O bit on.
- b) If the message the LS is sending to the DCS does contain the last CSAS records the LS needs to send, and the CA message just received from the DCS had the O bit off, the CA message is sent with the O bit off and the LS transitions the CAFSM to the Update Cache State.
- c) If the message the LS is sending to the DCS does contain the last CSAS records the LS needs to send, and the CA message just received from the DCS had the O bit on, the CA message is sent with the O bit off and the alignment process continues.
- If the LS is slave and the LS receives a CA message with a CA sequence number that is neither equal to nor one more than the current LS's CA sequence number, an error has happened and the CAFSM transitions to the Master/Slave Negotiation State 😱
Note that if the LS was slave during the CA process, then when the LS moves the CAFSM to the Update Cache State, it MUST keep a copy of the last CA message it sent, and the LS SHOULD set a timer equal to CAReXmtInterval 🕰️ If either the timer expires or the LS gets a CSU Solicit (CSUS) message from the DCS (we'll get to CSUS messages in Section 2.2.3), the LS lets go of its copy of the CA message. Here's why: if the DCS (which is master) loses the last CA message the LS sent, the DCS will resend its previous CA message with the last CA sequence number used. If that happens, the LS would need to resend its last sent CA message too. Keeping it all straight is so important 🧠
2.2.2.1 "CA message processing":
The LS makes a list of those cache entries that are more "up to date" in the DCS than in the LS's own cache. This list is called the CSA Request List (CRL). For what it means for a CSA (Client State Advertisement) record or CSAS record to be more "up to date" than an LS's cache entry, see Section 2.4 🌸
2.2.3 The Update Cache State
If the CRL of the associated CAFSM of the LS is empty when the CAFSM transitions into the Update Cache State, the CAFSM immediately transitions into the Aligned State ✅
If the CRL is not empty when the CAFSM transitions into the Update Cache State, the LS asks the DCS to send the CSA records that match the summaries (aka the CSAS records) the LS is holding in its CRL 🙏 The solicited CSA records will contain the entirety of the cached information the DCS holds for the given cache entry. The LS asks for the relevant CSA records by forming CSU Solicit (CSUS) messages out of the CRL. For the CSUS message format, check out Section B.2.4. Then the LS sends the CSUS messages to the DCS. The DCS answers each CSUS message by sending the LS one or more CSU Request messages that contain the entirety of the newer cached information identified in the CSUS message. Once the LS gets a CSU Request, it sends back one or more CSU Replies, as described in Section 2.3. Heads up: the LS can have at most one CSUS message outstanding at any given time 🚦
Right before the first CSUS message is sent from an LS to the DCS associated with the CAFSM, a timer is set to CSUSReXmtInterval seconds ⏱️ If not all of the CSA records matching the CSAS records in the CSUS message have arrived by the time the timer expires, a new CSUS message gets created. It contains all the CSAS records for which no appropriate CSA record has been received, plus extra CSAS records that weren't covered in the previous CSUS message. That new CSUS message is then sent to the DCS. If, before the timer expires, all CSA record updates have been received for all the CSAS records in the previously sent CSUS message, the timer is stopped. Once the timer is stopped, if there are additional CSAS records that weren't in the previous CSUS message but were in the CRL, the timer is reset and a new CSUS message is created. It contains only those CSAS records from the CRL that haven't been sent to the DCS yet. This keeps going until the LS has received all the CSA records that match the CSAS records that were in the CRL. When the LS has a completely updated cache, the LS transitions the CAFSM associated with the DCS to the Aligned State 🎉
If an LS gets a CSUS message or a CA message with a Receiver ID that isn't the LS's LSID, the message must be discarded and ignored. This matters because an LS might be a leaf of a point to multipoint connection with other servers in the SG 🫶
2.2.4 The Aligned State
While in the Aligned state, an LS will perform the Cache State Update Protocol, as described in Section 2.3 💜
Note that an LS may get a CSUS message while in the Aligned State. When that happens, the LS MUST respond to the CSUS message with the appropriate CSU Request message, in a similar way to the method described earlier in Section 2.2.3 🍓
2.3 Cache State Update Protocol ✨
okay so Cache State Update (CSU) messages are how we keep the state of cache entries up to date on servers, per PID/SGID 💅. A CSU message carries zero or more "Cache State Advertisement" (CSA) records, and each one is its own snapshot of what a particular cache entry looks like right now 🦋. An LS is only allowed to send or receive a CSU to or from a DCS when the matching CAFSM is in either the Aligned State or the Update Cache State 🌸.
There are two kinds of CSU messages: CSU Requests and CSU Replies. Check Sections B.2.2 and B.2.3 for their formats 📋. A CSU Request goes from an LS to one or more DCSs for one of two reasons. Either the LS got a CSUS message, in which case it MUST respond only to the DCS that originated that CSUS message 🙈, or the LS noticed a change of state in a cache entry 👀. An LS finds out about a change of state either by receiving a CSU Request from one of its DCSs, or because it saw a change in a cached entry that the LS itself originated. In the first case, the LS sends a CSU Request to each of its DCSs except the one it heard the change from. In the second case, it sends a CSU Request to each of its DCSs. The change is noted in a CSA record, which gets tacked onto the end of the CSU Request message mandatory part. That's how state changes spread through the whole SG 🌈.
Examples of these changes in state look like this:
1) a server receives a request from a client to add an entry to
its cache,
2) a server receives a request from a client to remove an entry
from its cache,
3) a cache entry has timed out in the server's cache, has been
refreshed in the server's cache, or has been administratively
modified.When an LS gets a CSU Request from one of its DCSs, it acknowledges one or more of the CSA Records in that request by sending back a CSU Reply 💖. The CSU Reply holds one or more CSAS records that match the CSA records being acknowledged. So if an LS drops a CSA record (or is slow to process it) because it doesn't have enough resources, the matching CSAS record just doesn't go into the CSU Reply to that DCS 😭.
An LS can send multiple CSU Requests before it gets a CSU Reply acknowledging any of the CSA Records in them. A CSU Reply can also acknowledge CSA Records from multiple CSU Requests. So yeah, "request" and "reply" is lowkey confusing, ngl 🙃.
A CSA Record is a CSAS Record followed by the client/server protocol specific info stored in a cache entry (see Section B.2.0.2 for CSAS record format info and Section B.2.2.1 for CSA record format info) 📦. When the LS decides that a CSA Record represents information that's more "up to date" (see Section 2.4) than what's in its own cache, two things happen ✨. First, the LS updates its cache with the newer info. Second, the LS sends a CSU Request containing that CSA Record to each of its DCSs except the one it came from. That's how state changes get propagated within the PID/SGID 🦋. At some point the LS also acknowledges receiving the CSA Record by sending the right DCS a CSU Reply with the matching CSAS Record 💅.
When an LS sends a new CSU Request, it keeps track of the outstanding CSA records in that request and which DCSs it was sent to 🧚. For each DCS the request went to, a timer set to CSUReXmtInterval seconds starts right before the send ⏱️. That timer is tied to the CSA Records in that request. If it expires before all of those records get acknowledged by that DCS, then (and only then) the LS re-sends a CSU Request to that DCS 🔁. The re-sent request only includes the CSA Records that still haven't been acknowledged 🎀. If every CSA Record on a timer gets acknowledged, the timer stops ✅. Re-sent CSA Records follow the same time-out and retransmit rules as brand-new ones. Retransmission happens a configured number of times for each CSA Record. If acknowledgment still doesn't come through, an "abnormal event" has happened, and the HFSM for that DCS moves to the Waiting State 🚨.
A CSA Record instance is "on a DCS retransmit queue" when it's tied to that timer 🍓. Only the most up-to-date CSA Record is allowed to sit on a given DCS retransmit queue. So if an older CSA Record is already queued when a newer instance is about to be queued, the older one gets dequeued and disassociated from its timer right before the newer one gets enqueued 🫶.
When an LS gets a CSU Reply from one of its DCSs, it checks each CSAS record in that reply against the CSAS Record portion of the CSA Records queued on that DCS retransmit queue 🔍.
- If there's an exact match between the CSAS record portion of a CSA record and a CSAS Record in the CSU Reply, that CSA Record counts as acknowledged. It gets dequeued from the DCS retransmit queue and disassociated from its timer 💖.
- If there's a match except for the CSA Sequence number, then:
- a) If the queued CSA Record has a CSA Sequence Number greater than the CSA Sequence Number in the CSAS Record of the CSU Reply, the CSAS Record in the CSU Reply gets ignored 🙅.
- b) If the queued CSA Record has a CSA Sequence Number less than the CSA Sequence Number in the CSAS Record of the CSU Reply, the queued CSA Record gets dequeued and disassociated from its timer. On top of that, a CSUS Message gets sent to the DCS that sent the more up-to-date CSAS Record. All normal CSUS processing happens as if the CSUS were sent as part of the CA protocol 🔥.
When an LS gets a CSU Request containing a CSA Record whose CSA Sequence Number is smaller than the CSA Sequence number of the cached CSA, the LS MUST still acknowledge that CSA record in the CSU Request. But it MUST do so by sending a CSU Reply containing the CSAS Record portion of the CSA Record stored in its cache, not the CSAS Record portion of the CSA Record from the CSU Request 🙌.
An LS answers CSUS messages from its DCSs by sending CSU Requests with the right CSA records to that DCS 💌. Now, what if an LS gets a CSUS message with a CSAS record for an entry that's no longer in its database? For example, the entry timed out and got discarded after Cache Alignment finished but before the entry was requested through a CSUS message 💀. In that case the LS copies the CSAS Record from the CSUS message into a CSU Request, and it sets the N bit. That's the signal that this is a NULL record, since the cache entry no longer exists in the LS's cache 🫥. Note that here, the "CSA Record" in the CSU Request that stands in for the NULL cache entry is literally just a CSAS Record, because there's no client/server protocol specific info left for that entry 🧚♀️.
If an LS gets a CSA Record in a CSU Request from a DCS, and it already has an identical CSA record posted to that DCS's retransmit queue, then the record on the retransmit queue counts as implicitly acknowledged 🤝. So it gets dequeued and disassociated from its timer. The CSA Record the DCS sent still MUST be acknowledged by the LS in a CSU Reply, though 💅. This matters on point to multipoint connections, where the rule "when an LS receives a CSA record from a DCS, that LS floods the CSA Record to every DCS except the DCS from which it was received" might get broken 🌈.
If an LS gets a CSU with a Receiver ID that isn't equal to the LSID and isn't set to all 0xFFs, the CSU gets thrown out and ignored 🗑️. This is needed because the LS might be a leaf of a point to multipoint connection with other servers in the LS's SG 🌸.
An LS MAY send a CSU Request to the all 0xFFs Receiver ID when it's the root of a point to multipoint connection with a set of its DCSs 🌳. If an LS gets a CSU Request with the all 0xFFs Receiver ID, it MUST use the Sender ID in that CSU Request as the Receiver ID of the CSU Reply. In other words, it MUST unicast its response back to the sender of the request when it responds 💜. If the LS wants to send a CSU Request to the all 0xFFs Receiver ID, it MUST create a time-out and retransmit timer for each of the DCSs that are leaves of the point to multipoint connection, and it has to do that before sending the CSU Request ⏰. If the time-out and retransmit timer for a given DCS expires before that DCS acknowledges a given CSA Record, the LS MUST use that specific DCSID as the Receiver ID instead of the all 0xFFs Receiver ID 🎯. Same deal if the LS needs to re-send a CSA Record. Then the LS MUST specify the specific DCSID as the Receiver ID instead of the all 0xFFs Receiver ID 💅.
Now for the full mesh moment 🦋. Say a set of servers are in a full mesh of point to multipoint connections, and one server in that mesh sends a CSU Request into the mesh. If the sending server puts the CSA Records in that CSU Request addressed to the all 0xFFs Receiver ID, then no other server in the mesh needs to send its own CSU Request with those CSA Records into the mesh just to flood them properly 🙅♀️. Why? Because every server in the mesh already heard the CSU Request and processed the included CSA Records as appropriate ✨. So a server in a full mesh can treat the mesh as one single logical port, and the rule "when an LS receives a CSA record from a DCS, that LS floods the CSA Record to every DCS except the DCS from which it was received" isn't broken 👑. A receiving server in the full mesh still needs to acknowledge those CSA records with CSU Reply messages. Those replies carry the LSID of the replying server as the Sender ID, and the ID of the server that sent the CSU Request as the Receiver ID field 💖. In the time-out and retransmit case, the Receiver ID of the CSU Request gets set to the specific DCSID that didn't acknowledge the CSA Record, instead of the all 0xFFs Receiver ID 🔁. And since a full mesh emulates a broadcast medium for the servers attached to it, using SCSP on a broadcast medium might use this technique too 📡. Further discussion of this use of a full mesh or of a broadcast medium is left to the client/server protocol specific documents 🫶.
2.4 The meaning of "More Up To Date"/"Newness" 💅
okay so during cache alignment AND normal CSU processing, an LS takes a CSAS Record and compares it against what its own cache entry already says 👀 the goal is to figure out if the record is more "up to date" than the entry, aka is this newer tea ☕
there are three pieces of info the LS checks: 1) the Cache Key, 2) the Originator, which is described by an Originator ID (OID), and 3) the CSA Sequence number 🎀 if you want the deets on these fields, peek at Section B.2.0.2 🧚
so a CSAS record (whether it's part of a CSA Record or flying solo) counts as more "up to date" than the LS's cache when ALL of these are true:
- The Cache Key in the CSAS Record matches the stored Cache Key in the LS's cache entry 🔑
- The OID in the CSAS Record matches the stored OID in the LS's cache entry 🪪
- The CSA Sequence Number in the CSAS Record is greater than the CSA Sequence Number in the LS's cache entry 📈
Discussion and Conclusions ✨
okay so even though all of this is written in terms of keeping a client's state synced across the caches of the servers in an SG, this solution generalizes super easily to any number of database synchronization problems (LECS synchronization is one example) 🌈 it's giving reusable 😭
SCSP defines a generic flooding protocol 🌊 there are a bunch of related issues around cache maintenance and topology maintenance that are better defined in the client/server protocol specific documents. for example, a protocol might want a generic cache entry time-out mechanism, or it might want servers to advertise adjacency info so you could get a topo-map of the servers in an SG 🗺️ when mechanisms like these are wanted, they'll be defined in those client/server protocol specific documents, not here 💅
Appendix A: Terminology and Definitions 📖
CA Message - Cache Alignment Message These messages let an LS synchronize its entire cache with the cache of one of its DCSs 🧩
CAFSM - Cache Alignment Finite State Machine The CAFSM watches the state of the cache alignment between an LS and one particular DCS. There's one CAFSM per DCS as seen from an LS 👀
CSA Record - Cache State Advertisement Record A CSA is a record inside a CSU message that identifies an update to the status of a "particular" cache entry 🎀
CSAS Record - Cache State Advertisement Summary Record A CSAS contains a summary of the info in a CSA. A server sends CSAS records describing its cache entries to another server during cache alignment. CSAS records are also included in CSUS messages when an LS wants the entire CSA from the DCS. The LS asks for the CSA because it believes the DCS has a more recent view of the state of that cache entry 🥹
CSU Message - Cache State Update message This is a message an LS sends to its DCSs when the LS becomes aware of a change in the state of a cache entry 📣
CSUS Message - Cache State Update Solicit Message An LS sends this to its DCS after the two have exchanged CA messages. The CSUS message holds one or more CSAS records, which act as requests for entire CSA records (instead of just the summary info in the CSAS) 🙋♀️
DCS - Directly Connected Server The DCS is a server directly connected to the LS, e.g., there's a VC between the LS and DCS. This term, along with LS and RS, gives us a frame of reference when we talk about servers and their synchronization. Unless explicitly stated otherwise, there's no implied difference in functionality between a DCS, LS, and RS 🤝
HFSM - Hello Finite State Machine An LS has an HFSM for each of its DCSs. The HFSM watches the state of the connectivity between the LS and one particular DCS 🛎️
LS - Local Server The LS is the server under scrutiny, so all statements are made from its perspective. This term, along with DCS and RS, gives us a frame of reference for talking about servers and their synchronization. Unless explicitly stated otherwise, there's no implied difference in functionality between a DCS, LS, and RS 👑
LSID - Local Server ID The LSID is a unique token that identifies an LS. This value might be taken from the protocol address of the LS 🏷️
PID - Protocol ID This field holds an identifier for the client/server protocol that's using SCSP for the given message. Assignment of Protocol IDs for this field is handed over to IANA, as described in Section C 🌸
RS - Remote Server (RS) From an LS's perspective, an RS is a server separate from the LS that isn't directly connected to it. An RS is always two or more hops away from an LS, whereas a DCS is always one hop away 🛰️ Unless otherwise stated, an RS means a server in the SG. Same deal as before: no implied difference in functionality between a DCS, LS, and RS unless explicitly said otherwise 💜
SG - Server Group SCSP synchronizes the caches (or a portion of the caches) of a set of server entities bound to an SG through some means (e.g., all servers belonging to a Logical IP Subnet (LIS)[1]). So an SG is just a grouping of servers around some commonality 🧁
SGID - Server Group ID This is a 16 bit identification field that uniquely identifies the instance of the client/server protocol whose servers in the SG are being synchronized. This means multiple instances of the same protocol can run at the same time and have their servers synchronized independently of each other 🦋
Appendix B: SCSP Message Formats 🎀
This section of the appendix has the message formats for SCSP 📦 SCSP protocols are LLC/SNAP encapsulated with an LLC=0xAA-AA-03, OUI=0x00-00-5e, and PID=0x00-05.
SCSP has 3 parts to every packet: the fixed part, the mandatory part, and the extensions part 🧩 The fixed part shows up in every packet and is shown below. The mandatory part is specific to the message type (CA, CSU Request/Reply, Hello, CSUS). It includes (among other packet elements) a Mandatory Common Part and zero or more records, each holding info about the state of a particular cache entry (except in the case of a Hello message) whose info is being synchronized within an SG. The extensions part holds the set of extensions for the SCSP message 💫
In the message formats below, fields marked as "unused" MUST be set to zero when the message is transmitted, and they MUST be ignored when the message is received 🙈
B.1 Fixed Part 🍓
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version | Type Code | Packet Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum | Start Of Extensions |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Version : This is the version of the SCSP protocol being used. The current version is 1 🔢
Type Code : This is the code for the message type (e.g., Hello (5), CSU Request(2), CSU Reply(3), CSUS (4), CA (1)) 🏷️
Packet Size : The total length of the SCSP packet, in octets (excluding link layer and/or other protocol encapsulation) 📏
Checksum : The standard IP checksum over the entire SCSP packet, starting at the fixed header. If the packet is an odd number of bytes in length, this calculation is performed as if a byte set to 0x00 is appended to the end of the packet 🧮
Start Of Extensions : This field is coded as zero when no extensions are present in the message. If extensions are present, this field is coded with the offset from the top of the fixed header to the beginning of the first extension 🧚
B.2.0 Mandatory Part 💖
The mandatory part of the SCSP packet holds the operation-specific info for a given message type (e.g., SCSP Cache State Update Request/Reply, etc.). It includes (among other packet elements) a Mandatory Common Part (described in Section B.2.0.1) and zero or more records, each holding info about the state of a particular cache entry (except in the case of a Hello message) whose info is being synchronized within an SG. Depending on the message type, these records are either Cache State Advertisement Summary (CSAS) Records (described in Section B.2.0.2) or Cache State Advertisement (CSA) Records (described in Section B.2.2.1). CSA Records contain a summary of a cache entry's info (i.e., a CSAS Record) plus some additional client/server protocol specific info 🌷 The mandatory common part format and CSAS Record format are shown immediately below, before we see how they're used in SCSP messages, so we don't have to repeat ourselves in the message descriptions 🫶
B.2.0.1 Mandatory Common Part
Okay bestie, sections B.2.1 through B.2.5 are basically twins 👯♀️ they share a ton of the same format, so they all pull from one shared chunk called the mandatory common part 🎀 here's the layout:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Protocol ID | Server Group ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| unused | Flags |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sender ID Len | Recvr ID Len | Number of Records |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sender ID (variable length) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Receiver ID (variable length) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Protocol ID 🌸 this tells you which client/server protocol is using SCSP for the message. IANA handles assigning these values (see Section C) ✨ protocols with current documents are already set up like this:
1 - ATMARP
2 - NHRP
3 - MARS
4 - DHCP
5 - LNNIServer Group ID 💜 this uniquely identifies the one instance of a client/server protocol whose servers are getting synced. Iconic uniqueness 😤
Flags 🦋 the meaning depends on the message, so we'll get into the details in the specific message format sections below.
Sender ID Len 📏 the length of the Sender ID, in octets.
Recvr ID Len 📏 the length of the Receiver ID, in octets.
Number of Records 🍓 how many extra records are attached to the message. The exact format of those records depends on the message type, and each one gets its own description in the sections below.
Sender ID 🫶 an identifier for the server sending the message. One possible choice is that server's protocol address.
Receiver ID 🧚 an identifier for the server that's supposed to receive the message. One possible choice is the protocol address of that receiving server.
B.2.0.2 Cache State Advertisement Summary Record (CSAS record)
Okay, CSAS records are giving summary 📝 they carry a summary of a cache entry from a client/server database that SCSP is keeping in sync. The summary has just enough info for SCSP to find the right cache entry in that database and then compare the "newness" of the summary against the "newness" of what's already cached 👀
Quick note: CSAS records don't have a Server Group ID (SGID) or a Protocol ID 🙈 those two IDs are how you figure out which protocol and which instance the summary is for, and they already live in the mandatory common part of each message, so no need to repeat them.
Another quick note: the Hop Count and Record Length values in a CSAS record depend on whether it's a "stand-alone" record or "embedded" in a CSA Record. We'll unpack that below 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop Count | Record Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Cache Key Len | Orig ID Len |N| unused |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CSA Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Cache Key ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Originator ID ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Hop Count 🦘 this is how many hops the record can make before it gets dropped. Every server it passes through knocks the Hop Count down by one. For a "stand-alone" CSAS record (one that isn't embedded in a CSA record), this is set to 1, because summaries don't go past one hop during cache alignment 🙅♀️ if the CSAS record is "embedded" in a CSA record, the Hop Count gets set to an administratively defined value that's almost certainly greater than or equal to the cardinality of the SG minus one. Now the twist: if the CSA Record is carried in a CSU Request that was sent in response to a solicitation (meaning a CSAS Record that came in a CSUS message), the Hop Count SHOULD be set to 1 💖
Record Length 📐 for a "stand-alone" CSAS record, this is 12+"Cache Key Leng"+"Orig ID Len" in bytes. Otherwise it's 12+"Cache Key Leng"+"Orig ID Len"+ sizeof("Client/Server Protocol Specific Part for cache entry"). You can find the size of the Client/Server Protocol Specific Part in the client/server protocol specific document for the given Protocol ID 🧁
Cache Key Len 🔑 the length of the Cache Key field, in bytes.
Orig ID Len. 🏷️ the length of the Originator ID field, in bytes.
N 🙃 the "N" bit means this CSAS Record is actually a Null record. It's only used in a CSAS Record inside a CSU Request/Reply that's responding to a CSUS message. Here's the scenario: an LS might get a solicitation for a CSA record, but the cache entry that CSA record was talking about doesn't exist in its cache anymore (see Section 2.3 for details). In that case, the LS copies the CSAS Record straight from the CSUS message into the CSU Request and sets the N bit to say the cache entry is gone. The DCS that solicited the CSA record still responds with a CSU Reply, so nobody's left on read 📬 this bit is usually set to zero.
CSA Sequence Number 🔢 this is the sequence number that tells you how "new" a CSA record instance is. A "larger" number means a more recent advertisement. So if part (or all) of a cache entry needs an update, the CSA record advertising the new state MUST contain a CSA Sequence Number that's larger than the one from the previous advertisement. The originator of the CSA record assigns this number, and the CSA Sequence Number can come from either the originating server or the client that made its server advertise its existence ✨
The CSA Sequence Number is a signed 32 bit number. Within that space, the number -2^31 (0x80000000) is reserved. That means the usable range for a given Cache Key is -2^31+1 (0x80000001) through 2^31-1 (0x7fffffff). An LS starts with -2^31+1 the first time it originates a CSA Record for a cache entry it created. Every time that cache entry gets modified in a way that needs syncing with the other servers in the SG, the LS increments the CSA Sequence Number for that Cache Key and uses the new number when advertising the update 📈
Heads up, this is the spicy part 🌶️ if a given CSA Sequence Number has reached 2^31-2 and the cache entry has changed so that an update has to go out to the rest of the SG, then the cache entry MUST first be purged from the SG by the LS. The LS does this by sending a CSA Record that gets the cache entry removed from the other servers, and that purging CSA Record carries a CSA Sequence Number of 2^31-1. The exact packet format and the mechanism for purging a cache entry are defined in the appropriate protocol specific document. Once every DCS has acknowledged the purging CSA Record, the LS sends a new CSA Record with the updated information, and that new record carries a CSA Sequence Number of -2^31+1. Protecting the numbering like this is giving careful planning 💅
After a restart, and after the restarting LS's CAFSM has reached the Aligned state, if an update to an existing cache entry needs to be synced, or a brand new cache entry needs to be synced, the CSA Record that follows MUST contain a CSA Sequence Number that's unique within the SG for the given OID and Cache Key. The RECOMMENDED way to get this number (unless the protocol specific document says otherwise) is to take the CSA Sequence Number of the existing cache entry (or zero, if there's no out of date cache entry yet) and add a configured constant to it 🧮 also, the protocol specific document might require that every cache entry containing the OID of the restarting LS gets purged before the cache entries are updated. If so, the updating CSA Record still carries a CSA Sequence Number, set to the number from the previously existing cache entry plus a configured constant. Same energy, different timing 🕰️
Cache Key 🗝️ this is a database lookup key that uniquely identifies a piece of data the originator of a CSA Record wants to sync with its peers, for a given "Protocol ID/Server Group ID" pair. It's usually a small opaque byte string that SCSP associates with a piece of data in a cache. For example, an originator might give a particular 4 byte string to the binding of an IP address with an ATM address. In general, the originating server of a CSA record is responsible for generating a Cache Key for every piece of data it originates and wants to keep in sync with its peers in the SG 💜
Originator ID 🪪 this field holds an ID that's administratively assigned to the server that originates the CSA Records. Main character server energy 👑
B.2.1 Cache Alignment (CA)
okay bestie, the CA message is how an LS syncs its entire cache with its DCSs inside a server group 🫶 it's giving full reset energy. The CA message type code is 1 🔥 and here's the mandatory part layout:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CA Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Mandatory Common Part |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CSAS Record |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.......
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CSAS Record |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+CA Sequence Number 🧚 A value that uniquely identifies a step in the cache alignment process, so everything stays in order. A larger sequence number means a more recent CA message ✨. The slave server always copies the sequence number from the master server's previous CA message into the CA message it's currently sending, which is how the slave acknowledges the master's CA message 💜. Since the initial CA process is lock-step, if the slave doesn't receive the same sequence number it got last time, then what was in the slave's previous CA message is implicitly acknowledged 🙈. Also, every CAFSM gets its own separate CA Sequence Number space.
Whenever cache alignment needs to be (re)started and the CAFSM enters the Master/Slave Negotiation state, the CA Sequence Number should be set to a value the DCS hasn't seen before. One possible scheme is the machine's time of day counter ⭐.
Mandatory Common Part 🎀 The mandatory common part is explained in full in Section B.2.0.1. Two of its fields get coded differently depending on the message type, and those are the "Number of Records" field and the "Flags" field 💅.
Number of Records: for the CA message, this tells you how many CSAS Records are tacked onto the end of the CA message mandatory part 🌸.
Flags: the CA message's Flags field looks like this:
0 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|M|I|O| unused |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+M: part of the negotiation for cache alignment. When it's set, the sender is saying it wants to lead the alignment process 👑. This one's the "Master/Slave bit".I: when set, the sender thinks it's currently negotiating for master or slave status. This one's the "Initialization bit" 🦋.O: tells you the sender still has more CSAS records to send, which means the cache alignment process has to keep going. This is the "mOre bit", even though the name is kinda dubious 💀.
All the other fields in the mandatory common part are coded the way Section B.2.0.1 describes ✨.
CSAS record 🌈 The CA message tacks CSAS records onto the end of its mandatory part. These CSAS records are NOT embedded in CSA records, so don't mix them up 🙅. For the details on CSAS records, see Section B.2.0.2 💖.
B.2.2 Cache State Update Request (CSU Request)
the CSU Request is how a server tells its directly connected servers that cache entries changed 📣. It goes from one server (the LS) to a directly connected server (the DCS) whenever the LS notices a change. The LS can notice in two ways: either it receives a CSU request that updates its own database, or it sees a change to a cached entry that it originated itself 🔥. Each change gets noted by adding a "Cache State Advertisement" (CSA) record to the end of the CSU Request's mandatory part, like this 💅.
The CSU Request message type code is 2 🍓. Here's the mandatory part layout:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Mandatory Common Part |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CSA Record |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.......
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CSA Record |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Mandatory Common Part 🎀 Same deal as before: the full story is in Section B.2.0.1, and two fields change meaning per message type, "Number of Records" and "Flags" 💜.
Number of Records: for the CSU Request, this is how many CSA Records are appended to the CSU Request mandatory part 🌸.
Flags: there are currently no flags defined for the CSU Request's Flags field 🙈. Literally nothing, bestie.
Everything else in the mandatory common part is coded as described in Section B.2.0.1 ✨.
CSA Record 🦋 See Section B.2.2.1 👑.
B.2.2.1 Cache State Advertisement Record (CSA record)
CSA records hold everything needed to connect the current state of a cache entry in an SG with the servers being synchronized 🔥. A CSA record has a CSAS Record header plus a client/server protocol specific part 💅. The CSAS Record gives SCSP enough info to find the matching cache entry in the client/server database, and then compare how "new" the summary is against how "new" the cached entry is 🧚. If the CSA is newer than the receiving server's cached entry, that cached entry gets updated with the CSA Record's contents ✨. The protocol specific part is documented separately for each protocol. For examples, NHRP's protocol specific part is shown in [8] and ATMARP's is shown in [9] 🌈.
Here's the wild part: the info carried by one CSA record can be bigger than a link layer PDU 😭. So CSA records MUST be fragmented across several CSU Request messages. How that fragmenting works is client/server protocol specific, and it's documented in the matching protocol specific document 🫶.
The content of a CSA record looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| CSAS Record |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Client/Server Protocol Specific Part for cache entry ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+CSAS Record 💖 For the rules and format of a CSAS Record when it's "embedded" in a CSA Record, see Section B.2.0.2 ✨.
Client/Server Protocol Specific Part for cache entry 🌸 This field holds the stuff that's specific to the protocol-specific bit of SCSP processing. Each protocol that uses SCSP defines its own set of fields in a separate document 📚. The Protocol ID, which says which protocol is using SCSP in a given packet, lives in the message's mandatory part 🔥.
B.2.3 Cache State Update Reply (CSU Reply)
the CSU Reply is how a DCS says "got it" to an LS, acknowledging one or more CSA records that came in a CSU Request 💅. A CSA record gets acknowledged by putting a CSAS record that matches it into the CSU Reply 🦋.
The CSU Reply has the same format as the CSU Request, with a few differences 👑. Its type code is 3. It only returns CSAS Records, not CSA records. And it only returns CSAS Records for the CSA Records it's actually acknowledging ✨. So what this means is an LS that sent a CSU Request might not get an acknowledgment for every CSA Record in that request in just one CSU Reply 🙈. Plan for multiple replies, babe 🫶.
B.2.4 Cache State Update Solicit Message (CSUS message)
this message lets one server (the LS) ask a directly connected server (the DCS) for its entire pile of CSA record data 🧚. The DCS answers with CSU Request messages that carry the right CSA records 🔥. The CSUS message type code is 4 💜. Its format is the same as the CSU Reply message 🌈.
A CSUS message only solicits CSU Requests from one server, the one named in the Receiver ID field of the message's Mandatory Part 🦋. Just one server, no group chat drama 💅.
B.2.5 Hello ✨
okay bestie, the Hello message is how a server checks that its neighbour is still around 🫶 The sending server (the LS) uses it to check connectivity with one of its directly connected neighbour servers (the DCSs). The Hello message type code is 5 🎀 Here's the mandatory part:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| HelloInterval | DeadFactor |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| unused | Family ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Mandatory Common Part |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Additional Receiver ID Record |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.........
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Additional Receiver ID Record |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+HelloInterval 💖 This is the time between sending consecutive Hello Messages. If the LS doesn't get a Hello message from the DCS (one that has the LSID as a Receiver ID) within the HelloInterval the DCS advertised, then that DCS's Hello is considered to be late. Also, the LS MUST send its own Hello message to a DCS within the HelloInterval it advertised to that DCS in the LS's previous Hello message to it. Otherwise the DCS would consider the LS's Hello to be late 😭
DeadFactor 🔥 This is a multiplier on the HelloInterval. If an LS doesn't get a Hello message containing its LSID as a Receiver ID within HelloInterval*DeadFactor from a given DCS (the one that advertised the HelloInterval and DeadFactor in a previous Hello message), then the LS MUST consider that DCS stalled. At this point, one of two things MUST happen: 1) if the LS has received any Hello messages from the DCS during this time, the LS transitions the corresponding HFSM to the Unidirectional State; otherwise, 2) the LS transitions the corresponding HFSM to the Waiting State. Dramatic, I know 💀
Family ID 🦋 This is an opaque bit string used to refer to an aggregate of Protocol ID/SGID pairs. Only a single HFSM is run for all the Protocol ID/SGID pairs assigned to a Family ID. So there's a one-to-many mapping between that single HFSM and the CAFSMs for each of the Protocol ID/SGID pairs. This might have the net effect of substantially reducing HFSM maintenance traffic ✨ See the protocol specific SCSP documents for further details.
Mandatory Common Part 🌸 The mandatory common part is described in detail in Section B.2.0.1. Two fields in it have codings that are specific to this message type: the "Number of Records" field and the "Flags" field.
- Number of Records 🍓 For the Hello message, this field holds the number of "Additional Receiver ID" records included in the Hello. Those records each contain a length field and a Receiver ID field. Note that the count in "Number of Records" does NOT include the Receiver ID that's already in the Mandatory Common Part.
- Flags 🧚 Currently, no flags are defined for the Flags field of the mandatory common part for the Hello message.
All other fields of the mandatory common part are coded as described in Section B.2.0.1.
Additional Receiver ID Record 💜 This record has a length field followed by a Receiver ID. Even within an SG, a Receiver ID's length might change, so each additional Receiver ID heard (beyond the first one) gets its length in bytes and its value encoded in an "Additional Receiver ID Record". Receiver IDs are the IDs of a DCS that the LS has recently heard a Hello from (that is, within DeadFactor*HelloInterval as advertised by the DCS in a previous Hello message). The format looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Rec ID Len | Receiver ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Now the fun part, the three cases 🎀
If the LS hasn't heard from any DCS, it sets the Hello message fields like this: Recvr ID Len is set to zero, no storage is allocated for the Receiver ID in the Common Mandatory Part, "Number of Records" is set to zero, and no storage is allocated for "Additional Receiver ID Records".
If the LS has heard from exactly one DCS, it sets the fields like this: the Receiver ID of that DCS and the length of that Receiver ID are encoded in the Common Mandatory Part, "Number of Records" is set to zero, and no storage is allocated for "Additional Receiver ID Records".
If the LS has heard from two or more DCSs, it sets the fields like this: the Receiver ID of the first DCS that was heard and the length of that Receiver ID are encoded in the Common Mandatory Part, "Number of Records" is set to the number of "Additional" DCSs heard, and for each additional DCS an "Additional Receiver ID Record" is formed and appended to the end of the Hello message 🌈
B.3 Extensions Part 🎀
The Extensions Part, if it's there, carries one or more extensions in {Type, Length, Value} triplets 💅 Extensions have this format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type | Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Value... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Type 🦋 The extension type code (see below).
Length 💖 The length in octets of the value. This doesn't include the Type and Length fields. A null extension will have only an extension header and a length of zero.
When extensions exist, the extensions part gets terminated by the End of Extensions extension, which has Type = 0 and Length = 0 ✨
Extensions may occur in any order, but any particular extension type may occur only once in an SCSP packet. An LS MUST NOT change the order of extensions 🙈
B.3.0 The End Of Extensions 🧚
Type = 0
Length = 0When extensions exist, the extensions part is terminated by the End Of Extensions extension. That's the bestie that closes the list 👑
B.3.1 SCSP Authentication Extension 🔐
Type = 1 Length = variable
The SCSP Authentication Extension is carried in SCSP packets to convey authentication information between an LS and a DCS in the same SG 🫶
Authentication is done pairwise on an LS to DCS basis, so the authentication extension is generated at each LS. If a received packet fails the authentication test, then an "abnormal event" has occurred. The packet is discarded and this event is logged 💀 Absolutely no drama, but it's serious!
The presence or absence of authentication is a local matter 🌸
B.3.1.1 Header Format
The authentication header has the following format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Security Parameter Index (SPI) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+ Authentication Data... -+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The Security Parameter Index (SPI) can be thought of as an index into a table that keeps track of the keys and other info, like the hash algorithm 🔑 LS and DCS communicate either off-line using manual keying, or online using a key management protocol, to populate this table. The receiving SCSP entity always allocates the SPI and the parameters associated with it.
The authentication data field contains the MAC (Message Authentication Code) calculated over the entire SCSP payload. The length of this field depends on the hash algorithm used to calculate the MAC ✨
B.3.1.2 Supported Hash Algorithms
The default hash algorithm to be supported is HMAC-MD5-128 [11]. HMAC is safer than normal keyed hashes 🔥 Other hash algorithms MAY be supported by def.
IANA will assign the numbers to identify the algorithm being used, as described in Section C 🦋
B.3.1.3 SPI and Security Parameters Negotiation
SPIs can be negotiated either manually or using an Internet Key Management protocol. Manual keying MUST be supported. The following parameters are associated with the tuple <SPI, DCS ID>: lifetime, Algorithm, Key. Lifetime indicates the duration in seconds for which the key is valid. With manual keying, this duration can be infinite. Also, to better support manual keying, there may be multiple tuples active at the same time (with the DCS ID being the same) 🌈
Any Internet standard key management protocol MAY be used to negotiate the SPI and parameters 💜
B.3.1.4 Message Processing
At the time of adding the authentication extension header, the LS looks up a table to fetch the SPI and the security parameters based on the DCS ID. If there are no entries in the table and key management is supported, the LS initiates the key management protocol to fetch the necessary parameters. The LS then calculates the hash by zeroing the authentication data field before calculating the MAC on the sending end. The result replaces the zeroed authentication data field 🍓 If key management isn't supported and authentication is mandatory, the packet is dropped and this information is logged.
When receiving traffic, an LS fetches the parameters based on the SPI and its ID. The authentication data field is extracted before zeroing it out to calculate the hash. It computes the hash on the entire payload, and if the hash does not match, then an "abnormal event" has occurred 😭
B.3.1.5 Security Considerations 🔐
Okay bestie, this part is SERIOUS, so listen up 😭🔐 It is important that the keys you pick are strong, because the security of the whole system depends on the keys being chosen properly AND on the algorithms being implemented correctly. No cap.
SCSP has a peer to peer trust model 🤝 It is recommended to use an Internet standard key management protocol to negotiate the keys between the neighbors. If other methods of negotiation are used and the keys get transmitted in clear text, that compromises the security completely 💀
Data integrity covers the entire SCSP payload ✨ This guarantees that the message was not modified, and that the source is authenticated as well. If the authentication extension is not used, or if the security is compromised, then SCSP servers are liable to spoofing attacks, active attacks, and passive attacks 🙈
There is no mechanism to encrypt the messages 🫣 It is assumed that a standard layer 3 confidentiality mechanism will be used to encrypt and decrypt messages. Since integrity is calculated on an SCSP message and not on each record, there is an implied trust between all the servers in a domain. It is recommended to use the security extension between all the servers in a domain, and not just a subset of servers.
Any SCSP server is susceptible to Denial of Service (DOS) attacks 🌪️ A rogue host can inundate its neighboring SCSP server with SCSP packets. However, if the authentication option is used, SCSP databases will not become corrupted, because the bogus packets will be discarded when the authentication check fails 🛡️
Due to the pairwise authentication model of SCSP, the information received from any properly authenticated server is trusted and propagated throughout the server group. Consequently, if the security of any SCSP server is compromised, the entire database becomes vulnerable to corruption originating from the compromised server 😱
B.3.2 SCSP Vendor-Private Extension 🎀
Type = 2
Length = variableThe SCSP Vendor-Private Extension is carried in SCSP packets to convey vendor-private information between an LS and a DCS in the same SG, and is thus of limited use. If a finer granularity (e.g., CSA record level) is desired, then the given client/server protocol specific SCSP document MUST define such a mechanism. Obviously, though, such a protocol specific mechanism might look exactly like this extension. The Vendor Private Extension MAY NOT appear more than once in an SCSP packet for a given Vendor ID value. 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Vendor ID | Data.... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- Vendor ID: 802 Vendor ID as assigned by the IEEE [7].
- Data: The remaining octets after the Vendor ID in the payload are vendor-dependent data.
If the receiver does not handle this extension, or does not match the Vendor ID in the extension, then the extension may be completely ignored by the receiver. 💅
C. IANA Considerations 📋
Any and all requests for value assignment from the various number spaces described in this document require proper documentation. Possible forms of documentation include, but are not limited to, RFCs or the product of another cooperative standards body (e.g., the MPOA and LANE subworking group of the ATM Forum). Other requests may also be accepted, under the advice of a "designated expert". (Contact the IANA for the contact information of the current expert.) 📋✨
References 📚
[1] Laubach, M., and J. Halpern, "Classical IP and ARP over ATM", Laubach, RFC 2225, April 1998.
[2] Luciani, J., Katz, D., Piscitello, D., Cole, B., and N. Doraswamy, "NMBA Next Hop Resolution Protocol (NHRP)", RFC 2332, April 1998.
[3] Moy, J., "OSPF Version 2", STD 54, RFC 2328, April 1998.
[4] "P-NNI V1", Dykeman, Goguen, 1996.
[5] Armitage, G., "Support for Multicast over UNI 3.0/3.1 based ATM Networks", RFC 2022, November 1996.
[6] Keene, "LAN Emulation over ATM Version 2 - LNNI specification", btd-lane-lnni-02.08
[7] Reynolds, J., and J. Postel, "Assigned Numbers", STD 2, RFC 1700, October 1994.
[8] Luciani, J., "A Distributed NHRP Service Using SCSP", RFC 2335, April 1998.
[9] Luciani, J., "A Distributed ATMARP Service Using SCSP", Work In Progress.
[10] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[11] Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed Hashing for Message Authentication", RFC 2104, February 1997.
Acknowledgments 🫶
This memo is a distillation of issues raised during private discussions, on the IP-ATM mailing list, and during the Dallas IETF (12/95). Thanks to everyone who contributed, but particular thanks go to the following people (in no particular order): Barbara Fox of Harris and Jeffries; Colin Verrilli of IBM; Raj Nair, and Matthew Doar of Ascom Nexion; Andy Malis of Cascade; Andre Fredette of Bay Networks; James Watt of Newbridge; and Carl Marcinik of Fore. You all ate 👑💖
Authors' Addresses 🏠
James V. Luciani
Bay Networks, Inc.
3 Federal Street, BL3-03
Billerica, MA 01821
Phone: +1-978-916-4734
EMail: luciani@baynetworks.comGrenville Armitage
Bell Labs Lucent Technologies
101 Crawfords Corner Road
Holmdel, NJ 07733
Phone: +1 201 829 2635
EMail: gja@lucent.comJoel M. Halpern
Newbridge Networks Corp.
593 Herndon Parkway
Herndon, VA 22070-5241
Phone: +1-703-708-5954
EMail: jhalpern@Newbridge.COMNaganand Doraswamy
Bay Networks, Inc.
3 Federal St, BL3-03
Billerice, MA 01821
Phone: +1-978-916-1323
EMail: naganand@baynetworks.comFull Copyright Statement ⚖️
Now the legal bits, which I'm keeping word for word so nothing gets twisted 🙈
Copyright (C) The Internet Society (1998). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 💜
📦 where this page comes from
- ietf: rfc2334 1998-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms