RFC 2651

The Architecture of the Common Indexing Protocol (CIP)

๐Ÿ“… published August 1999, from ietf

RFC 2651: The Architecture of the Common Indexing Protocol (CIP) ๐Ÿ’–

Okay bestie, this is RFC 2651 ๐Ÿ“š, a Standards Track doc from the Network Working Group, dated August 1999 ๐ŸŒธ. The authors are J. Allen (WebTV Networks) and M. Mealling (Network Solutions, Inc.) ๐Ÿซถ. The title block doesn't list anything it obsoletes or updates, so nothing to track there โœจ.


Status of This Memo โœจ

So this doc specifies an Internet standards track protocol for the whole Internet community, and it's literally asking for discussion and suggestions to make it better ๐Ÿ’…. If you want to know where this protocol stands in the official standardization stuff, check the current "Internet Official Protocol Standards" (STD 1) ๐Ÿ“‹. And the best part: distribution of this memo is unlimited, so share away ๐Ÿฆ‹.

Copyright (C) The Internet Society (1999). All Rights Reserved.

Abstract ๐ŸŒˆ

CIP is used to pass indexing information from server to server so query routing gets easier ๐Ÿ”€. Query routing is the process of redirecting and replicating queries through a distributed database system toward the servers holding the results you want ๐ŸŽฏ. This doc describes the CIP framework, including its architecture and the protocol specifics of exchanging indices, so it's basically the full glow-up guide ๐Ÿ’–.


1. Introduction ๐ŸŽ€

1.1. History and Motivation ๐Ÿ•ฐ๏ธ

Okay so CIP is an evolution and refinement of distributed indexing concepts first introduced in the Whois++ Directory Service [RFC1913, RFC1914] ๐Ÿ‘‘. Indexing was super useful in that system for query routing, but the centroid index object passed among Whois++ servers was specifically designed for template-based databases searchable by token-based matching ๐Ÿงฉ. With alternative index objects, index-passing is going to be useful for way more application domains, not just Directory Services and stuff that can be cast into template collections ๐Ÿš€.

The indexing part of Whois++ is integrated with the data access protocol, and the goal with CIP is to extract that indexing portion while abstracting the index objects so they apply more broadly to information retrieval ๐Ÿง . Another kind of tech reuse happened too: the ad-hoc data representations Whois++ used got converted into structures based on the MIME specification for structured Internet mail ๐Ÿ’Œ.

Whois++ had a version number field in centroid objects for future growth, and the initial version was "1" ๐Ÿ”ข. Version 1 of CIP (back when it was still embedded in Whois++ and not called CIP on its own) only supported ISO-8895-1 characters, and only the centroid index object type ๐Ÿ™ˆ.

Version 2 of the Whois++ centroid was used in the Digger software by Bunyip Information Systems to tell recipients that the centroid carried extra character set information โœจ. Digger's centroids can carry UTF-8 encoded 16-bit Unicode characters, or ISO-8859-1 characters, and a field in the headers decides which one ๐ŸŒธ.

This spec is for CIP version 3, and it's a major overhaul ๐Ÿ’…. But here's the slay: by using a short negotiation sequence, CIP version 3 servers can still interoperate with earlier servers in an index-passing mesh ๐Ÿซถ.

If any terms are confusing, the glossary in Appendix A has your back ๐Ÿ“–.

1.2 CIP's place in the Information Retrieval world ๐Ÿ”

CIP is all about facilitating query routing ๐Ÿ›ฃ๏ธ. It's a protocol used between servers in a network to pass hints that make data access by clients later on way more efficient ๐Ÿ’จ. Query routing is the act of redirecting and replicating queries through a distributed database system toward the servers that hold the actual results, using references to indexing information ๐Ÿ“.

CIP is a "backend" protocol, which means it's implemented in and "spoken" only among network servers ๐Ÿ–ฅ๏ธ. These same servers still have to speak some kind of data access protocol to talk to clients ๐Ÿ—ฃ๏ธ. During query resolution in the native protocol implementation, the server checks the indexing info collected by the CIP implementation for guidance on how to route the query ๐Ÿงญ.

Data access protocols used with CIP must have some provision for control information in the form of a referral ๐ŸŽ. The syntax and semantics of those referrals are outside the scope of this spec, so that's a whole other vibe ๐Ÿคทโ€โ™€๏ธ.


This doc is one of three documents ๐Ÿ‘ฏ. This one covers the fundamental concepts and framework of CIP ๐Ÿ›๏ธ.

The "MIME Object Definitions for the Common Indexing Protocol" [CIP-MIME] doc describes the MIME objects that make up the items passed by the transport system ๐Ÿ“ฆ.

Requirements and examples for several transport systems are specified in the "CIP Transport Protocols" [CIP-TRANSPORT] doc ๐Ÿšš.

A second set of documents covers the specs for specific index types ๐Ÿ—‚๏ธ.


3. Architecture ๐Ÿ—๏ธ

3.1 CIP in the Information Retrieval World ๐ŸŒŽ

3.1.1 Information Retrieval in the Abstract ๐Ÿงƒ

To get how CIP fits into the information retrieval world, we first need the unifying abstract features of existing IR tech ๐Ÿ”‘. Then we'll talk about why adding indexing tech to this model makes a system capable of query routing, and why query routing is worth it ๐Ÿ’–.

So an abstract view of the client/server data retrieval process includes data sets and data access protocols ๐Ÿ“Š. Each individual server handles queries over a fixed domain of data, and for CIP we call that domain the dataset ๐Ÿ—ƒ๏ธ. Clients search the dataset and retrieve parts of it through a data access protocol ๐Ÿ“ฅ. There are lots of data access protocols, and each one is optimized for the data in question ๐Ÿ’…. For example, LDAP and Whois++ are access protocols that reflect the needs of the directory services domain ๐Ÿ“‡. Other data access protocols include HTTP and Z39.50 ๐ŸŒ.

3.1.2 Indexing Information Facilitates Query Routing ๐Ÿช„

The description above is a world without indexing, where no server knows about any other server ๐Ÿ™ˆ. In some cases (like X.500 referrals and HTTP redirects) a server will, as part of its reply, pull another server into resolving the query ๐Ÿค. But those servers only generate replies based on their own local knowledge ๐Ÿ . When indexing info gets introduced into a server's local database, that server now knows not just answers from its local dataset, but also answers based on external indices ๐ŸŒŸ. Those indices come from peer servers through an indexing protocol, and CIP is one such indexing protocol ๐Ÿซถ.

Replies based on index info might not be the complete answer, because an index isn't a replicated version of the remote dataset. It's a possibly reduced version of it ๐Ÿค. So alongside complete replies from the local dataset, the server may give referrals to other datasets ๐ŸŽ€. Those referrals are the core feature needed for effective query routing ๐Ÿ‘‘. When servers use CIP to pass indices from server to server, they make a kind of investment ๐Ÿ’ธ. It costs some resources to create, transmit, and store the indices, but in return query routing becomes possible ๐Ÿ”ฅ.

Query routing is the process of replicating and moving a query closer to the datasets that can satisfy it ๐ŸŽฏ. In some distributed systems, widely distributed searches have to be done by replicating the query to all sub-datasets ๐Ÿ˜ฉ. That approach can be wasteful of resources, both on the network and on the servers, so it's sometimes explicitly disabled ๐Ÿ™…โ€โ™€๏ธ. Using indexing in a system like that opens the door to way more efficient distributed searching ๐Ÿšชโœจ.

CIP-equipped servers provide the referrals needed to make query routing work ๐Ÿงš. But it's always the client's responsibility to collate, filter, and chase the referrals it receives ๐Ÿƒโ€โ™€๏ธ. That gives the end-user (or an agent, if no human is involved in the search) the greatest control over the query resolution process ๐Ÿ‘‘. The added client complexity is weighed against the benefits of total control over query resolution โš–๏ธ. In some cases, it may also be possible to decouple referral chasing from the client by introducing a proxy, which lets existing simple clients use query routing ๐Ÿ”Œ. That proxy would transparently resolve referrals into concrete results before handing them back to the simple-minded client ๐Ÿซ .

3.1.3 Abstracting the CIP index object ๐Ÿงฉ

As useful as indices seem, not all queries can benefit from the same type of index ๐Ÿ™ƒ. For example, say the index is a simple list of keywords. With that kind of index, it's impossible to answer queries about whether two keywords were near one another, or whether a keyword was present in a certain context (like in the title) ๐Ÿ“ฐ.

Because of the need for application domain specific indices, CIP index objects are abstract, which means they must be defined by a separate specification ๐Ÿ“. The basic protocols for moving index objects are widely applicable, but the specific design of the index, and the structure of the mesh of servers that pass a particular type of index, depends on the application domain ๐ŸŒ. This doc describes only the protocols for moving indices among servers ๐Ÿšš. Companion docs describe the initial index objects ๐ŸŽ.

The requirements that index type specifications must address are specified in the [CIP-MIME] doc ๐Ÿ“‹.

3.2 Architectural Details ๐Ÿ”ง

CIP implements index passing, which provides the forward knowledge needed to generate the referrals used for query routing ๐Ÿ”ฎ. The core of the protocol is the index object ๐Ÿ’Ž. The next sections present the structure of the index objects themselves, then how and why indices get passed from server to server, and finally the circumstances under which a server may synthesize an index object based on incoming ones ๐Ÿงช.

3.2.1 The CIP Index Object ๐Ÿ“ฆ

A CIP index object is made of two parts: the header and the payload ๐ŸŽ€. The header has the metadata needed to process and make use of the index object being transmitted ๐Ÿ—๏ธ. The actual index lives in the payload ๐Ÿ’œ.

Three particular headers deserve a shout-out right now ๐Ÿ“ฃ. The type of the index object selects one of many distinct CIP index object specifications, which define exactly how the index blocks get created, parsed, and used to facilitate query routing ๐Ÿง . Another header is DSI, the Dataset Identifier, which uniquely identifies the dataset the index was created from ๐Ÿ†”. And another one that's crucial for generating referrals is Base-URI ๐ŸŒˆ. The URI (or URIs) in this header form the basis of any referrals generated from this index block ๐Ÿ”—. The URI is also used as input during the index aggregation process to constrain what kinds of aggregation are possible, because of multiprotocol constraints โ›“๏ธ. How that URI gets used is defined by the aggregation algorithm ๐Ÿงฎ. The exact syntax of these headers is specified in the CIP MIME specification doc [CIP-MIME] ๐Ÿ“–.

The payload is opaque to CIP itself ๐Ÿ™ˆ. It's defined exclusively by the index object specification associated with the object's MIME type ๐Ÿท๏ธ. Specs on how to parse and use the payload are published separately as "CIP index object specifications" ๐Ÿ“š. This abstract definition of the index object is the foundation of CIP's applicability to indexing needs across multiple application domains ๐ŸŒŸ๐Ÿ’–.

A precise definition of the content and form of a CIP index block can be found in the Protocol doc [CIP-MIME]. ๐Ÿฅฐ

3.2.2 Moving Index Objects: How to Build a Mesh

Okay so indices get passed around between all the servers in a CIP mesh ๐Ÿ’– Why? Because if you spread that info out before a query even shows up, routing can be efficient AND accurate the second the query lands ๐Ÿฆ‹

A CIP mesh is a squad of CIP servers that pass indices of the same type between themselves ๐Ÿซถ Usually it's set up like a hierarchical tree, where servers closer to the root have bigger, more comprehensive indices ๐Ÿ‘‘ See Figure 1 ๐Ÿ‘€ BUT a mesh is explicitly allowed to have lateral links too, and there can be more than one part of the mesh that acts like a "root" ๐ŸŒˆ Mesh admins are encouraged to avoid loops, but they're not obliged to keep a strict tree structure, so it's not that deep ๐Ÿ’… Clients who want to fully resolve every referral they get should protect against referral loops while they traverse the mesh, otherwise they're just wasting time and network resources ๐Ÿ˜ญ See the section on "Navigating the Mesh" for the tea on this โœจ

     base level             index                    index
     directory             servers                  servers
      servers                for                      for
                          base level               lower-level
                           servers                index servers
     _______
    |       |
    |   A   |__
    |_______|  \            _______
                \---CIP----|       |
     _______               |   D   |__
    |       |   /---CIP----|_______|  \             ------
    |   B   |__/                       \--CIP------|      |
    |_______|                                      |  F   |
                                       /--CIP------|______|
                                      /
     _______                _______  /
    |       |              |       |-
    |   C   |-------CIP----|   E   |
    |_______|              |_______|-
                                |    \
                                r     \
     _______                    e      \            ______
    |       |                   f       \--CIP-----|      |
    |   G   |-------CIP---------e------------------|  H   |
    |_______|                   r                  |______|
            \--referral---|     r      --referral-/

                          |     a     |

                          |     l     |

                          \ 3   | 2   | 1

                            \--------/

                            |        |

                            | client |

                            |        |

                             --------


             Figure 1: Sample layout of the Index Service mesh

As of right now, every index passed in a given mesh is assumed to be the same type, meaning it's governed by the same CIP index object specification ๐ŸŽ€ Could you make gateways between meshes that carry different index objects? Maybe, but that's totally undefined at this point and declared out of scope for this spec ๐Ÿ™ˆ

If a CIP server gets an index of a type it doesn't understand, it can pass that index forward untouched ๐Ÿฆ‹ If a server implementation decides it won't accept unknown indices, it should send back an appropriate error message to the server that sent the index ๐Ÿ’… This is so mesh implementations can attempt heterogeneous meshes. BUT heterogeneous meshes are considered ill defined, so they should be treated as dangerous ๐Ÿ’€ Read that again, bestie โš ๏ธ

Index passing is experienced as a parallel (and possibly lower-priority) job next to the server's main gig, which is answering queries ๐Ÿงš Index objects travel between CIP servers through protocol exchanges explicitly defined in this document, NOT through the server's native protocol. This distinction is so important it's worth repeating ๐Ÿ”ฅ

Queries are answered (and referrals are sent) via the native data access protocol. Index objects are transferred via alternative means, as defined by this document.

When two servers team up to move indexing info, they're in a "polling relationship" ๐Ÿค The server holding the data of interest and generating the index is the "polled server" ๐ŸŒธ The one collecting that generated index is the "polling server" ๐Ÿ“

In a polling relationship, the polled server is responsible for telling the polling server when it has a new index the polling server might care about ๐Ÿ“ฃ The polling server can then grab the index object right away, OR schedule a job to pick up a copy later when it's more convenient ๐Ÿ’œ But the polling server is not required to wait around for the polled server to announce changes. It can request a new index at any time ๐Ÿซถ

There's also a second way servers can pass indices, totally separate from the polling thing โœจ In an "index pushing" relationship, a CIP server just sends the index to a peer whenever it's necessary, and the receiver gets to handle it however it wants ๐Ÿ‘‘ It can refuse it, accept it and then silently discard it, accept only parts of it (by accepting it as is and then filtering it), or accept it without question ๐Ÿ’…

Index pushing is meant for dumb leaf nodes that just want their index available to the global mesh of servers, but have zero interest in implementing the full CIP transaction protocol ๐Ÿงš It lowers the barrier to entry for CIP leaf nodes, which is iconic ๐ŸŒˆ If you want the details on joining a CIP mesh in this more restricted way, check the section below on "Protocol Conformance" ๐Ÿ‘€ CIP index passing operations happen across a reliable transport mechanism, including both TCP connections and Internet mail messages ๐Ÿฆ‹ The precise mechanisms are described in the Transport document [CIP-Transport] ๐Ÿ“š

3.2.3 Index Object Synthesis

From everything we just covered, it should be clear that indexing servers read and write index objects as they pass them around the mesh ๐Ÿงš BUT a CIP server doesn't have to just pass the inbound indices through as the outbound ones ๐Ÿ’… Passing an index object through to other servers is always permissible, but a server can also choose to aggregate two or more of them. That cuts down on redundancy in the index, at the cost of longer referral chains ๐Ÿฆ‹

The big premise of index passing is that even when you squish a whole body of data into an index with lossy compression methods, the hints useful for routing queries should survive ๐Ÿซถ The index isn't a complete copy of the original dataset, so it has less info, obviously ๐Ÿ™ˆ Index objects can be passed along unchanged, but as more and more info piles up in the resulting index object, redundancy creeps back in ๐Ÿ˜ญ At that point it might be useful to compress again by aggregating two or more index objects into one ๐ŸŽ€

Aggregation has to be done WITHOUT compromising the ability to route queries correctly, while also avoiding excessive numbers of missed results ๐Ÿ’– How likely false negatives are allowed to be has to be established per application domain, and it's controlled by the granularity of the index and the aggregation rules defined for it by the particular specification ๐Ÿ”ฅ

Now, when CIP is used in a multi-protocol application domain, like a Directory Service (with contenders including Whois++, LDAP, and Ph), things get way trickier ๐ŸŒ€ The core problem is avoiding forcing a referral chain through part of the mesh that doesn't support the protocol the client used to make the query ๐Ÿ˜ฑ If that happens, the client loses access to any hits beyond that point in the chain, since it can't resolve the referral in its native data access protocol. That's a query routing failure, and it's something to avoid ๐Ÿšซ

Beyond multi-protocol stuff, server managers might choose not to allow index object aggregation for performance reasons ๐Ÿ“‰ As referral chains get longer, a client has to do more transactions to resolve a query. More transactions means more user-perceived delays, more system load, and more global bandwidth demand ๐Ÿ˜ฉ In general there's a tradeoff between aggressive aggregation (which reduces indexing overhead) and aggressive referral chain optimization ๐Ÿ’œ This tradeoff is also sensitive to the particular application domain, and it still needs to be explored more in actual operational situations ๐Ÿง

Conceptually, a CIP index server has several index objects on hand at any given time ๐Ÿง  If it holds data in addition to indexing info, the server has an index object built from its own data, called the "local index" ๐Ÿ  It may also have one or more indices from remote servers that it collected via the index passing mechanisms. Those are called "in-bound indices" ๐Ÿ“ฅ

Implementor's Note: It may not be necessary to keep all of these structures intact and distinct in the local database. It is also not required to keep the out-bound index (or indices) built and ready to distribute at all times. The previous paragraph merely introduces a useful model for expressing the aggregation rules. Implementors are free to model index objects internally however they see fit. ๐ŸŒธ

The following two rules control how a CIP server formulates its outgoing indices ๐Ÿ“ค

  1. An index server may pass any of the index objects in its local index and its in-bound indices through unchanged to polling servers. ๐Ÿฆ‹
  2. If and only if the following three conditions are true, an index server can aggregate two or more index objects into a single new index object, to be added to the set of out-bound indices. ๐Ÿ”ฅ
    • a. Each index object to be aggregated covers exactly the same set of protocols, as defined by the scheme component of the Base-URI's in each index object.
    • b. The index server supports every one of the data access protocols represented by the Base-URI's in the index objects to be aggregated.
    • c. The specification for the index object type specified by the type header of the index objects explicitly defines the aggregation operation.

The resulting index object must have Base-URI's characteristic of the local server for each protocol it supports. The outgoing objects should have the DSI of the local server. ๐Ÿ’…

4. Navigating the mesh ๐Ÿงญ

With the CIP infrastructure in place to manage index objects, the only problem left is how to actually use the indexing info to do efficient searches ๐Ÿ” CIP facilitates query routing, which is basically a client activity ๐Ÿ’ A client connects to one server, which redirects the query to servers "closer to" the answer. That redirection message is called a referral ๐ŸŽ€

4.1. The Referral ๐ŸŽ€

okay so, the idea of a referral and the rules for when to issue one are described by CIP ๐Ÿ’…. But the referral itself has to get back to the client in the native protocol, so its syntax isn't really a CIP thing. The part that decides a referral is needed and builds it lives in the CIP implementation on the server ๐Ÿง . The part that actually sends it to the client lives in the server's native protocol implementation ๐Ÿ“ฎ.

A referral happens when a search against the index objects the server holds shows that there may be hits in one of the datasets those index objects represent ๐Ÿ‘€. If more than one index object says a referral to a given dataset is needed, the server should generate only one referral to that dataset, because the client might not be able to spot duplicates ๐Ÿ™ˆ. one and done, bestie โœจ

The format depends on the native protocol(s) the CIP server speaks, but the baseline contents are the same across every protocol ๐Ÿ’–. At the very least, a DSI and a URI must be returned. The DSI is the one tied to the dataset that caused the hit, and it has to be shown to the client so it can avoid referral loops ๐Ÿ”. The other required piece comes from the Base-URI parameter that travels with index objects ๐Ÿงฉ.

The extra info in the Base-URI might be needed so the server receiving the referred query can handle it properly ๐Ÿซถ. A great example is an LDAP server, which needs a base X.500 distinguished name to search from ๐ŸŒณ. When an LDAP server sends a centroid-format index object up to a CIP indexing server, it sends a Base-URI along with the name of the X.500 subtree the index was made for. Then, when a referral gets made, that Base-URI gets passed back to the client so it can hand it to the original LDAP server ๐Ÿ’œ

Same as always, along with the DSI you can optionally send a DSI-Description header ๐ŸŒธ. Because a client might want to check with the user before chasing a referral, and because this string is the friendliest version of the DSI that CIP has to offer, it should be included in referrals whenever it's available (aka when it came along with the index object) โœจ

4.2. Cross-protocol Mappings ๐Ÿฆ‹

Every data access protocol that uses CIP is going to need a clearly defined set of rules for mapping queries in the native protocol to searches against an index object ๐Ÿ“œ. Those rules change based on the data domain. In theory this could get messy, since N protocols times M data domains would mean N x M mappings ๐Ÿ˜ญ. In practice it shouldn't go that way, because some access protocols are just totally unsuited to some data domains ๐Ÿ’€. Like, picture an LDAP server trying to search an index object built from unorganized text pages. What would the results even be? How would the client make any sense of them? ๐Ÿค”

That said, as older protocols get hooked up to CIP and new ones get built for it, this issue has to be looked at ๐Ÿ‘€. Whois++ and the CENTROID index type are an extremely close match, since they were designed together ๐Ÿ’…. Hooking LDAP up to the CENTROID index type is a different story. You'd need to map the attribute names LDAP uses to attribute names that are already in use across the CENTROID mesh ๐Ÿ—บ๏ธ. You'd also need to tokenize LDAP queries under the same rules as the CENTROID indexing policy so searches run correctly ๐ŸŽฏ. All of these application- and protocol-specific steps must be spelled out in the index object specification, as covered in the [CIP-MIME] document ๐Ÿ“–โœจ

4.3. Moving through the mesh ๐Ÿงš

From a client's point of view, CIP basically dumps all the "hard work" on its shoulders, since the client is the one that has to track down the real data ๐Ÿ˜ค. And yeah, that's true, but it's also super misleading ๐Ÿ™ƒ. Because the client controls the query routing process, the client has a lot of say over how big the result set gets, how fast the query moves, and how deep the search goes ๐ŸŒˆ.

The simplest client just hands referrals to the user in a raw, ready-to-reuse form, without trying to follow them ๐Ÿ“. For example, one Whois++ client that talks to the user through a Web-based form turns referrals into HTML hypertext links ๐Ÿ”—. The link carries the referral data through the HTML forms interface GET encoding rules: the hostname, the port, and the query ๐Ÿ’Œ. If the user clicks, they run a brand new search on the new host. A savvier client might show the referrals to the user and ask which ones to follow ๐Ÿง. And as long as there are sensible limits on search time and bandwidth usage, it could also be reasonable to program a client to follow every referral automatically ๐Ÿค–

When a client follows all the referrals, it has to be a little smart about it ๐Ÿง . Remember, the mesh is an interconnected graph of CIP servers, and that graph can have cycles ๐Ÿ”„. Those cycles could set off an endless loop of referrals, wasting time for the servers and for the client too ๐Ÿ˜ฉ. So when a client takes on the job of chasing down every referral, it must use some form of mesh traversal algorithm ๐Ÿ—บ๏ธ. One has already been documented for Whois++ in RFC-1914, and the same algorithm can easily be used with this version of CIP โœจ. In Whois++, the equivalent of a DSI is called a handle. With that swap, the Whois++ mesh traversal algorithm works unchanged with CIP ๐Ÿ’–

Last but not least, the mesh entry point (aka the first server queried) can seriously affect whether the query succeeds ๐Ÿšช. To avoid scaling problems, it is not acceptable to use a single "root" node and force every client to connect to it ๐Ÿ™…โ€โ™€๏ธ. Instead, clients should connect to a reasonably well connected local server (well connected with respect to the CIP mesh, not the Internet infrastructure) ๐Ÿ . If no match turns up from that entry point, the client can widen the search by asking the original server that polls it ๐Ÿ”. In general those servers have a better "vantage point" on the mesh, so they'll find answers the first search missed ๐Ÿ”ญ. The mechanism for figuring out the mesh structure dynamically like this exists, but it isn't documented here for brevity. For more on the POLLED-BY and POLLED-FOR commands, see RFC-1913 ๐Ÿ“š

One more thing, babe: while these mesh operations are important for making the client's searches as good as they can be, the client still speaks its native protocol ๐Ÿ’…. That info has to get to the client without making the client understand CIP ๐Ÿซถ


5. Security Considerations ๐Ÿ”’

okay, this section is the serious one, so buckle up ๐Ÿ˜ญ In here we talk about the security considerations you need when using this specification ๐Ÿ›ก๏ธ. There are at least three levels where security comes into play. First, indexing info can leak way more proprietary stuff than it should unless it's carefully controlled ๐Ÿซฃ. Second, at a more fundamental level, the CIP protocol itself needs external security services to run safely ๐Ÿ”. And lastly, CIP itself can be used to spread false information ๐Ÿšจ. Not gonna lie, this is kind of a big deal ๐Ÿ’€

5.1. Secure Indexing ๐Ÿ•ต๏ธโ€โ™€๏ธ

CIP is designed to index all kinds of data ๐Ÿ—‚๏ธ. Some of that data might be seen as valuable, proprietary, or even highly sensitive by whoever maintains it ๐Ÿ’Ž. Take a human resources database as an example. Certain bits of it, in moderation, can be super helpful for a company to make public ๐Ÿข. But the whole database is a seriously valuable asset, and the company has to protect it ๐Ÿ”’. The directory service community has a lot of experience over the years with how to walk that fine line between fully revealing a database and making useful pieces of it available โš–๏ธ. There are also legal considerations about what data can be collected and shared ๐Ÿ“œ.

Another case where security becomes a problem is a data publisher who wants to take part in a CIP mesh ๐Ÿ“ฆ. The data that publisher creates and manages is the company's prime asset ๐Ÿ‘‘. There's a financial incentive to join a CIP mesh, since exporting indices of your data makes people more likely to search your database ๐Ÿ’ธ. (Making profit off of the search activity is left as an exercise to the entrepreneur ๐Ÿ˜‰.) Once again, the index has to be designed carefully to protect the database while still giving a useful synopsis of the data ๐ŸŽ€.

One of the basic premises of CIP is that data providers will be willing to provide indices of their data to peer indexing servers ๐Ÿค. But unless they're carefully built, those indices could be a threat to the security of the database ๐Ÿ˜ฌ. So security of the data must be a prime consideration when developing a new index object type ๐Ÿงฑ. The risk of reverse engineering a database based only on the index exported from it must be kept at a level that matches the value of the data and the need for fine-grained indexing ๐Ÿ”.

Lastly, mesh organizers should know that inserting false data into a mesh can be used as part of an attack ๐Ÿšฉ. Depending on the type of mesh and the aggregation algorithms, an index can selectively prune parts of a mesh ๐ŸŒฟ. Also, since CIP is used to discover information, it will be a target for advertisements of false information ๐Ÿ“ข. CIP does not provide a method for trusting the data it contains ๐Ÿ™…โ€โ™€๏ธ. so keep your guard up, bestie ๐Ÿซถ


Acknowledgments ๐ŸŒธ

Thanks to the many helpful members of the FIND working group for the discussions that led to this specification ๐Ÿ’œ.

Specific acknowledgment goes to Jeff Allen, formerly of Bunyip Information Systems ๐Ÿ‘. His original version of these documents helped enormously in crystallizing the debate and the consensus ๐Ÿง . Most of the actual text in this document was originally written by Jeff ๐Ÿซถ. Jeff is no longer involved with the FIND Working Group or with editing this document. His authorship is preserved by a specific decision of the current editor ๐ŸŽ€.

Authors' Addresses ๐Ÿ“ฎ

Jeff R. Allen
246 Hawthorne St.
Palo Alto, CA 94301

EMail: jeff.allen@acm.org


Michael Mealling
Network Solutions, Inc.
505 Huntmar Park Drive
Herndon, VA 22070

Phone: (703) 742-0400
EMail: michael.mealling@RWhois.net

References ๐Ÿ“š

[RFC1913]       Weider, C., Fullton, J. and S. Spero, "Architecture
                of the Whois++Index Service", RFC 1913, February
                1996.

[RFC1914]       Faltstrom, P., Schoultz, R. and C. Weider, "How to
                Interact with a Whois++ Mesh", RFC 1914, February
                1996.

[CIP-MIME]      Allen, J. and  M. Mealling, "MIME Object Definitions
                for the Common Indexing Protocol (CIP)", RFC 2652,
                August 1999.

[CIP-TRANSPORT] Allen, J. and  P. Leach, "CIP Transport Protocols",
                RFC 2653, August 1999.

Appendix A: Glossary โœจ

application domain: okay so this is a problem domain that CIP gets applied to, with indexing needs that no existing problem domain already covers ๐Ÿงš. Separate application domains need separate index object specifications, and potentially separate CIP meshes. See index object specification.

centroid: an index object type used with Whois++ ๐Ÿ’…. In CIP versions before version 3, the index wasn't extensible and could only take the form of a centroid. A centroid is a list of (template name, attribute name, token) tuples with duplicates removed.

dataset: a collection of data (real or virtual) that an index gets built over ๐Ÿ“. When a CIP server aggregates two or more indices, the resulting index represents the index from a "virtual dataset" spanning the previous two datasets.

Dataset Identifier: an identifier picked from any part of the ISO/CCITT OID space, which uniquely identifies a given dataset among all datasets indexed by CIP ๐Ÿ”ฅ.

DSI: see Dataset Identifier.

DSI-description: a human readable string that can optionally ride along with DSIs to make them more user-friendly ๐ŸŽ€. See dataset Identifier.

index: a summary or compressed form of a body of data. Examples include a unique list of words, a codified full text analysis, a set of keywords, etc.

index object: the embodiment of the indices passed by CIP ๐Ÿฆ‹. An index object consists of some control attributes and an opaque payload.

index object specification: a document describing an index object type for use with the CIP system described in this document. See index object and payload.

index pushing: the act of presenting, unsolicited, an index to a peer CIP server.

MIME: see Multipurpose Internet Mail Extensions

Multipurpose Internet Mail Extensions: a set of rules for encoding Internet Mail messages that gives them richer structure ๐Ÿ’œ. CIP uses MIME rules to simplify object encoding issues. MIME is specified in RFC-1521 and RFC-1522.

payload: the application domain specific indexing information stored inside an index object. The format of the payload is specified externally to this document, and depends on the type of the containing index object.

polled server: a CIP server which receives a request to generate and pass an index to a peer server.

polling server: a CIP server which generates a request to a peer server for its index.

referral chain: the set of referrals generated by the process of routing a query. See query routing.

query routing: based on reference to indexing information, redirecting and replicating queries through a distributed database system towards the servers holding the actual results ๐Ÿงญ.


Copyright (C) The Internet Society (1999). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

legal bit alert ๐Ÿ™ˆ but it matters, so read it carefully bestie. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement ๐Ÿซถ

Funding for the RFC Editor function is currently provided by the Internet Society ๐ŸŒˆ.

๐Ÿ“ฆ where this page comes from