RFC 2967
TISDAG - Technical Infrastructure for Swedish Directory Access Gateways
RFC 2967: TISDAG, the Technical Infrastructure for Swedish Directory Access Gateways ✨🎀
Okay bestie, let's get into this one 💅. RFC 2967 is an Informational document from the Network Working Group (so it's not an Internet standard, just useful info for the community 🌸). It's dated October 2000 and was written by L. Daigle of Thinking Cat Enterprises and R. Hedberg of Catalogix 🦋. The title block doesn't list anything it obsoletes or updates, so there's nothing to chase down there 💜.
Status of this Memo 🧚
Okay so this part is basically the disclaimer 💅. This memo gives information to the Internet community. It does not specify an Internet standard of any kind, and it's free for anyone to pass around, with distribution unlimited 🌈.
Copyright Notice 💖
Copyright is held by The Internet Society, with the notice reading: Copyright (C) The Internet Society (2000). All Rights Reserved. 🎀
Abstract 🦋
Okay so the big idea behind TISDAG (Technical Infrastructure for Swedish Directory Access Gateways) is that it's the infrastructure that makes the DAG proposal actually work ✨. The whole point is a single access point for information about Swedish Internet users, and it's giving main character energy 👑.
Here's why it's so iconic 🔥: everyone gets the same access no matter who runs the service. That means the same level of access (7x24, so always on 💀), and the same information showing up, even if different providers keep different pieces of it. It doesn't matter what directory service protocols they're using, or what client access protocol the end user has. The experience stays consistent for everyone 🫶.
Table of Contents 💖
Okay bestie, here's the whole map of the doc so you know where everything lives. Every section number is exactly as it's cited elsewhere, so don't change a single digit 🫶
1.0 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1 Project Goal. . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Executive Summary of Technical Study Result . . . . . . . . . 5
1.3 Document Overview . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.0 Requirements. . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 End-User Requirements . . . . . . . . . . . . . . . . . . . . 8
2.2 WDSPs Requirements. . . . . . . . . . . . . . . . . . . . . . 8
2.3 DAG-System Requirements . . . . . . . . . . . . . . . . . . . 9
3.0 Functional Specification. . . . . . . . . . . . . . . . . . . 9
3.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 The DAG Core. . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3 Client Interface. . . . . . . . . . . . . . . . . . . . . . . 11
3.3.1 Acceptable User Input . . . . . . . . . . . . . . . . . . . 12
Supported Query Types. . . . . . . . . . . . . . . . . . . . . 12
Matching Semantics . . . . . . . . . . . . . . . . . . . . . . 12
Character Sets . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3.2 Data Output Spec. . . . . . . . . . . . . . . . . . . . . . 14
Schema Definition. . . . . . . . . . . . . . . . . . . . . . . 14
Referral Definition. . . . . . . . . . . . . . . . . . . . . . 14
Error conditions . . . . . . . . . . . . . . . . . . . . . . . 14
3.4 Directory Server Interface. . . . . . . . . . . . . . . . . . 14
4.0 Architecture. . . . . . . . . . . . . . . . . . . . . . . . . 15
4.1 Software Components . . . . . . . . . . . . . . . . . . . . . 15
4.1.1 Internal Communications . . . . . . . . . . . . . . . . . . 15
4.1.2 Referral Index. . . . . . . . . . . . . . . . . . . . . . . 15
4.1.3 DAG-CAPs. . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.1.4 DAG-SAPs. . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 Important Architectural Notes . . . . . . . . . . . . . . . . 17
4.2.1 2 Distinct Functions: Referrals and Chaining . . . . . . . 17
4.2.2 Limited Query and Response Semantics. . . . . . . . . . . . 17
4.2.3 Visibility. . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2.4 Richness of Query semantics . . . . . . . . . . . . . . . . 18
4.2.5 N+M Protocol Mappings . . . . . . . . . . . . . . . . . . . 18
4.2.6 DAG-CAPs and DAG-SAPs are completely independent of each
other. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2.7 The Role of the DAG-CAP . . . . . . . . . . . . . . . . . . 18
4.2.8 The Role of the DAG-SAP . . . . . . . . . . . . . . . . . . 19
4.2.9 DAG/IP is internal. . . . . . . . . . . . . . . . . . . . . 19
4.2.10 Expectations . . . . . . . . . . . . . . . . . . . . . . . 19
4.2.11 Future Extensions. . . . . . . . . . . . . . . . . . . . . 19
5.0 Software Specifications . . . . . . . . . . . . . . . . . . . 19
5.1 Notational Convention . . . . . . . . . . . . . . . . . . . . 19
5.2 DAG-CAP Basics. . . . . . . . . . . . . . . . . . . . . . . . 20
5.2.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . 20
5.2.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . 21
5.2.3 Error handling. . . . . . . . . . . . . . . . . . . . . . . 21
5.2.4 Pruning of results. . . . . . . . . . . . . . . . . . . . . 22
5.3 DAG-SAP Basics. . . . . . . . . . . . . . . . . . . . . . . . 22
5.3.1 Functionality . . . . . . . . . . . . . . . . . . . . . . . 22
5.3.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . 23
5.3.3 Error handling. . . . . . . . . . . . . . . . . . . . . . . 23
5.3.4 Pruning of results. . . . . . . . . . . . . . . . . . . . . 23
5.3.5 Constraint precedence . . . . . . . . . . . . . . . . . . . 23
5.4 The Referral Index. . . . . . . . . . . . . . . . . . . . . . 24
5.4.1 Architecture. . . . . . . . . . . . . . . . . . . . . . . . 24
5.4.2 Interactions with WDSPs (CIP) . . . . . . . . . . . . . . . 24
5.4.3 Index Object Format . . . . . . . . . . . . . . . . . . . . 24
5.4.4 DAG-Internal I/O. . . . . . . . . . . . . . . . . . . . . . 24
5.4.5 The Index Server. . . . . . . . . . . . . . . . . . . . . . 24
5.4.6 Configuration . . . . . . . . . . . . . . . . . . . . . . . 25
5.4.7 Security. . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.5 Mail (SMTP) DAG-CAP . . . . . . . . . . . . . . . . . . . . . 25
5.5.1 Mail DAG-CAP Input. . . . . . . . . . . . . . . . . . . . . 26
5.5.2 Translation from Mail query to DAG/IP . . . . . . . . . . . 28
Querying the Referral Index. . . . . . . . . . . . . . . . . . 28
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 29
5.5.3 Chaining queries in Mail DAG-CAP. . . . . . . . . . . . . . 31
5.5.4 Expression of results in Mail DAG-CAP . . . . . . . . . . . 31
5.5.5 Expression of Errors in Mail DAG-CAP. . . . . . . . . . . . 31
5.6 Web (HTTP) DAG-CAP. . . . . . . . . . . . . . . . . . . . . . 32
5.6.1 Web DAG-CAP Input . . . . . . . . . . . . . . . . . . . . . 32
5.6.2 Translation from Web query to DAG/IP. . . . . . . . . . . . 33
Querying a DAG-SAP Directly. . . . . . . . . . . . . . . . . . 33
Querying the Referral Index. . . . . . . . . . . . . . . . . . 33
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 35
5.6.3 Chaining queries in Web DAG-CAP . . . . . . . . . . . . . . 36
5.6.4 Expression of results in Web DAG-CAP. . . . . . . . . . . . 36
text/html results. . . . . . . . . . . . . . . . . . . . . . . 36
application/whoispp-response Results . . . . . . . . . . . . . 37
5.6.5 Expression of Errors in Web DAG-CAP . . . . . . . . . . . . 37
Standard Errors. . . . . . . . . . . . . . . . . . . . . . . . 38
5.7 Whois++ DAG-CAP . . . . . . . . . . . . . . . . . . . . . . . 38
5.7.1 Whois++ DAG-CAP Input . . . . . . . . . . . . . . . . . . . 38
5.7.2 Translation from Whois++ query to DAG/IP. . . . . . . . . . 39
Querying the Referral Index. . . . . . . . . . . . . . . . . . 39
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 39
5.7.3 Chaining in Whois++ DAG-CAP . . . . . . . . . . . . . . . . 40
5.7.4 Expression of results in Whois++. . . . . . . . . . . . . . 41
5.7.5 Expression of Errors in Whois++ DAG-CAP . . . . . . . . . . 41
5.8 LDAPv2 DAG-CAP. . . . . . . . . . . . . . . . . . . . . . . . 42
5.8.1 LDAPv2 DAG-CAP Input. . . . . . . . . . . . . . . . . . . . 42
5.8.2 Translation from LDAPv2 query to DAG/IP . . . . . . . . . . 44
Querying the Referral Index. . . . . . . . . . . . . . . . . . 44
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 46
5.8.3 Chaining queries in LDAPv2 DAG-CAP. . . . . . . . . . . . . 48
5.8.4 Expression of results in LDAPv2 . . . . . . . . . . . . . . 48
5.8.5 Expression of Errors in LDAPv2 DAG-CAP. . . . . . . . . . . 48
5.9 LDAPv3 DAG-CAP. . . . . . . . . . . . . . . . . . . . . . . . 50
5.9.1 LDAPv3 DAG-CAP Input. . . . . . . . . . . . . . . . . . . . 50
5.9.2 Translation from LDAPv3 query to DAG/IP . . . . . . . . . . 51
Querying the Referral Index. . . . . . . . . . . . . . . . . . 51
Querying a DAG-SAP . . . . . . . . . . . . . . . . . . . . . . 54
5.9.3 Chaining queries in LDAPv3 DAG-CAP. . . . . . . . . . . . . 55
5.9.4 Expression of results in LDAPv3 . . . . . . . . . . . . . . 55
5.9.5 Expression of Errors in LDAPv3 DAG-CAP. . . . . . . . . . . 56
5.10 Whois++ DAG-SAP. . . . . . . . . . . . . . . . . . . . . . . 57
5.10.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.10.2 Translation from DAG/IP to Whois++ query . . . . . . . . . 58
5.10.3 Translation of Whois++ results to DAG/IP . . . . . . . . . 58
5.11 LDAPv2 DAG-SAP . . . . . . . . . . . . . . . . . . . . . . . 59
5.11.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.11.2 Translation from DAG/IP to LDAPv2 query. . . . . . . . . . 59
5.11.3 Translation of LDAPv2 results to DAG/IP. . . . . . . . . . 61
5.12 LDAPv3 DAG-SAP . . . . . . . . . . . . . . . . . . . . . . . 62
5.12.1 Input. . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.12.2 Translation from DAG/IP to LDAPv3 query. . . . . . . . . . 62
5.12.3 Translation of LDAPv3 results to DAG/IP. . . . . . . . . . 64
5.13 Example Queries. . . . . . . . . . . . . . . . . . . . . . . 64
5.13.1 A Whois++ Query. . . . . . . . . . . . . . . . . . . . . . 65
What the Whois++ DAG-CAP Receives. . . . . . . . . . . . . . . 65
What the Whois++ DAG-CAP sends to the Referral Index . . . . . 65
What the Whois++ DAG-CAP Sends to an LDAP DAG-SAP. . . . . . . 65
5.13.2 An LDAP Query. . . . . . . . . . . . . . . . . . . . . . . 66
What the LDAP DAG-CAP Receives . . . . . . . . . . . . . . . . 66
5.13.3 What the LDAP DAG-CAP sends to the Referral Index. . . . . 67
What the LDAP DAG-CAP Sends to a Whois++ DAG-SAP . . . . . . . 67
What the LDAP DAG-CAP Sends to an LDAP DAG-SAP . . . . . . . . 68
6.0 Service Specifications. . . . . . . . . . . . . . . . . . . . 68
6.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6.2 WDSP Participation. . . . . . . . . . . . . . . . . . . . . . 69
6.3 Load Distribution . . . . . . . . . . . . . . . . . . . . . . 69
6.4 Extensibility . . . . . . . . . . . . . . . . . . . . . . . . 72
7.0 Security. . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.1 Information credibility . . . . . . . . . . . . . . . . . . . 73
7.2 Unauthorized access . . . . . . . . . . . . . . . . . . . . . 73
8.0 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 74
Appendix A - DAG Schema Definitions . . . . . . . . . . . . . . . 75
A.1 DAG Personal Information Schema (DAGPERSON Schema). . . . . . 76
A.2 DAG Organizational Role Information Schema (DAGORGROLE
Schema). . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Appendix B - Schema Mappings for Whois++ and LDAP . . . . . . . . 77
B.1 LDAP and the DAG Schemas. . . . . . . . . . . . . . . . . . . 78
B.2 Whois++ and the DAG Schemas . . . . . . . . . . . . . . . . . 81
Appendix C - DAG-Internal Protocol (DAG/IP) . . . . . . . . . . . 82
C.1 A word on the choice of DAG/IP. . . . . . . . . . . . . . . . 83
C.2 DAG/IP Input and Output -- Overview . . . . . . . . . . . . . 83
C.3 BNF for DAG/IP input and output . . . . . . . . . . . . . . . 83
C.3.1 The DAG/IP Input Grammar. . . . . . . . . . . . . . . . . . 84
C.3.2 The DAG/IP Response Grammar . . . . . . . . . . . . . . . . 87
C.4 DAG/IP Response Messages. . . . . . . . . . . . . . . . . . . 89
Appendix D - DAG/IP Response Messages Mapping . . . . . . . . . . 93
Appendix E - DAG CIP Usage. . . . . . . . . . . . . . . . . . . . 95
E.1 CIP Index Object. . . . . . . . . . . . . . . . . . . . . . . 95
E.2 CIP Index Object Creation . . . . . . . . . . . . . . . . . . 97
E.3 CIP Index Object Sharing. . . . . . . . . . . . . . . . . . . 98
E.3.1 Registration of Servers . . . . . . . . . . . . . . . . . . 98
E.3.2 Transmission of Objects . . . . . . . . . . . . . . . . . .100
Appendix F - Summary of Technical Survey Results. . . . . . . . .100
Appendix G - Useful References. . . . . . . . . . . . . . . . . .102
Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . .102
Authors' Addresses. . . . . . . . . . . . . . . . . . . . . . . .104
Full Copyright Statement. . . . . . . . . . . . . . . . . . . . .105List of Tables 🎀
List of Tables
Table 3.1 DAG-supported queries . . . . . . . . . . . . . . . . .12
Table 5.1 Allowable Whois++ Queries . . . . . . . . . . . . . . .38
Table A.1 DAGPERSON schema attributes . . . . . . . . . . . . . .76
Table A.2 DAGORGROLE schema attributes. . . . . . . . . . . . . .77
Table B.1 Canonical DAGPERSON schema & LDAP inetorgPerson
attributes . . . . . . . . . . . . . . . . . . . . . . . . . .79
Table B.2 Reasonable Approximations for LDAP organizationalRole
attributes . . . . . . . . . . . . . . . . . . . . . . . . . .79
Table B.3 Canonical mappings for LDAP organizationalRole
attributes . . . . . . . . . . . . . . . . . . . . . . . . . .81
Table B.4 Canonical DAGPERSON schema & Whois++ USER attributes. .81
Table B.5 Canonical mappings for Whois++ ORGROLE attributes . . .82
Table C.1 List of system response codes . . . . . . . . . . . . .90
Table D.1 LDAPv2/v3 resultcodes to DAG/IP response codes
mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . .93
Table D.2 Mapping from DAG/IP response codes to LDAPv2/v3
resultcodes. . . . . . . . . . . . . . . . . . . . . . . . . .94
Table D.3 Mapping between DAG/IP and Whois++ response codes . . .94
Table F.1 Summary of TISDAG Survey Results: Queries . . . . . . 101
Table F.2 Summary of TISDAG Survey Results: Operational
Information. . . . . . . . . . . . . . . . . . . . . . . . . 1011.0 Introduction ✨
1.1 Project Goal 🎯
okay so the big goal here is building the tech backbone for a single-access-point service where you can search whitepages info on Swedish internet users 🇸🇪💖 The service must be uniform for all information -- same level of access (7x24, like, always on 🌙) and the same whitepages info available, no matter which service provider is in charge of keeping that info ✨
1.2 Executive Summary of Technical Study Result 📝
lowkey the best part of the TISDAG project's DAG proposal is that it defines the technical infrastructure for a single-access-point service for info on Swedish internet users 🦋 You get uniform access for everything: same level of access (7x24), same info available, whoever the service provider is, whatever their directory service protocols are, and whatever client access protocol the end-user is using 💅
Instead of making Swedish directory providers mirror their whole records to one central place, the DAG uses a well-defined index object summary of that data, updated whenever the directory service provider is ready 🫶 When an end-user queries the DAG, the referral info gets used (by the end-user's software, or by a module inside the DAG, depending on what's appropriate) to finish the query directly on the directory service provider's system 🔥 This makes sure the end-user gets the most up-to-date complete info, and it also supports the provider's main interest: its service 👑 The DAG architecture is very modular, so support for future protocols can be added in the operational system 🧚
1.3 Document Overview 📚
This doc is broken into 5 major sections:
- Requirements: okay so as a service, the DAG system will have lots of different types of users 👯 To be successful, those users' needs (requirements) have to be met. That creates certain constraints, or system requirements, that must be met too. This section captures the baseline requirement assumptions the system has to address, and lays the groundwork for everything else built on top 🍓
- Functional Specification Overview: starting from the users' requirements, specific technologies and functionality details get outlined to architect a system that meets them 💜 That includes a conceptual architecture. The Requirements section covers what each user group needs from the eventual DAG system, while implementing and providing the service comes with constraints or conditions that have to be met in order to participate in the overall system ✨
- Architecture: once the system has been defined conceptually, a proposed software architecture is specified to produce the desired functionality and meet the stated requirements 🏗️
- Software Specifications: this section gives the specifications for the software components that meet the architecture described above 💻
- Service Specifications: once the software has been designed, the success of the DAG system rests on its operational characteristics 🌸 Details of service requirements are given in this section.
1.4 Terminology 📖
- DAG-CAP: Client Access Point -- point of communication between client-access software and the DAG system.
- DAG-System: The Directory Access Gateway system resulting from the TISDAG project. A collection of infrastructural software and services for the purpose of providing unified access to Swedish whitepages information.
- DAG/IP: DAG-Internal Protocol -- communication protocol used between software components of the DAG.
- End-User: People performing White Pages searches and look-ups (via various forms of client software).
- DAG-SAP: Service Access Point -- point of communication between the DAG and WDSP software.
- WDSP: Whitepages Directory Service Provider -- ISPs, companies, or other interested entities.
- Whitepages Information: Collected information coordinates for individual people. This typically includes (but is not limited to) a person's name, and e-mail address.
2.0 Requirements 🎀
there are 2 primary classes of users for the proposed Whitepages directory access gateway:
- End-users
- WDSPs
as outlined below, the needs of each of these user classes impose a set of constraints on the design of the DAG system itself 💅 Some of the requirements shown below are assumed starting criteria for the DAG service, and others were derived from data collected in the Technical Survey or other expertise input 🔍
2.1 End-User Requirements 🙋
The End-User is getting a specific set of search types:
- Name
- Name + Organization
- Role + Organization
- Name + Locality
- Name + Organization + Locality
- Role + Organization + Locality
The search results will, if available, include the following info for each "hit":
- Full name
- E-mail address
- Role
- Organization
- Locality
- Full address
- Telephone numbers
Access to the service must be available through reasonable and current protocols -- so that directory-service-aware software can use it seamlessly, and there are no reasonable technological impediments to making this service useful to all Swedish Internet users 🌐
Following on that, its responses are expected to be timely. A standard search should not take more time than the average access to a web-server ⚡
2.2 WDSPs Requirements 🏢
Since the WDSPs that take part in this service are already in the business of providing whitepages information, they have certain requirements that must be respected to make this a successful and useful service for everyone involved 🫶
The DAG system must provide reasonable assurances of data integrity for WDSPs. The information the End-User sees should correspond directly to what the WDSPs provided 🔒 The DAG system should be non-preferential when providing whitepages information -- the service is for the End-User, and the source of the whitepages information should not influence the search and information presentation processes 💖
The DAG system must be able to reflect information updates within a reasonable time after receipt from WDSPs. On the flip side, while the DAG system will function best with regular updates from WDSPs, the update and participation overhead for WDSPs should be kept within reasonable bounds of what the WDSP should do to support regular access to its information 🔄
Furthermore, since WDSPs provide directory service information with an eye to value-added service, wherever possible End-Users should be redirected to the WDSP responsible for individual directory service entries for final and further information 🦋
2.3 DAG-System Requirements ⚙️
To address the requirements of End-Users and WDSPs, the DAG system itself has certain design constraints that must be taken into account 🧩
The system must be implementable/operational by Dec 31/98 -- which implies that it must be designed and constructed with already extant technologies 📅
The System will have certain requirements for participation -- e.g., 7x24 WDSP availability ⏰
In terms of scaling, the system should be able to handle 8M records at the outset, with a view to handling larger information systems in the future 📈
The system must also be capable of extension to other, related applications (e.g., serving security certificate information) 🔐
3.0 Functional Specification 🧚
In the TISDAG pilotservice we have decided to apply some limitations as to what is specified for the DAG/IP. These limitations are presented in this text in the following manner:
TISDAG: This is a TISDAG comment3.1 Overview 🌈
The conceptual environment of the DAG system can be described in three major components:
- client access software for end-users
- the DAG system core
- WDSP directory service software
This is illustrated in Figure 3.1.
The DAG (Directory Access Gateway) is the infrastructural core of the service 🏛️ It maintains the necessary data and transformation facilities to let diverse directory service Client Software connect smoothly to the existing WDSPs' directory servers. The key challenges in designing this portion of the system are:
- Quantity of data -- the amount of whitepages information that will be made available, and the diversity of its sources (different WDSPs), create challenges in finding a structure that allows efficient searching and keeps the information updated in a timely way 📊
- Multiplicity of access protocols -- to support the use of existing whitepages-aware software with a minimum of perturbation, the DAG system will have to present a uniform face across several different access protocols, each with its own information search and representation paradigm 🎭
This specification will outline the following areas:
- the functioning of the DAG core itself
- the interface between the DAG core and End-Users' Directory Service Access software
- the interface between the DAG core and Directory Services Servers
3.2 The DAG Core ✨
okay so the DAG is basically the brain of the whole operation 🧠 and it does NOT want to hoard all the data, bestie. it keeps only index info, so the whitepages stay close to where they come from (the WDSPs themselves) 🫶 when an End-User asks something, the DAG uses "query routing" to point the query at the right WDSPs for search refinement and retrieval. this was originally built for Whois++, but the IETF's FIND WG is doing it in a protocol-independent way, so picking this approach doesn't lock anyone into a specific whitepages access protocol 🦋 it's giving flexibility.
the DAG will also handle queries for access protocols that don't support referral mechanisms 🧚 to support multiple access protocols and different data paradigms, the DAG will be designed to specifically support a limited set of whitepages queries. the vibe is focused, not everything-for-everyone 💅
+---------+ @
+ ->| | -+-
/|Protocol| | |
/ | / +---------+ / \
/ | "B"
+ | /
| |<-
+-------+ | |
O | | | |
-+- | |<--------->| |
| | | Protocol | |
/ \ | | "A" | |<-
+-------+ | |Protocol
| | \
+ | "A" +---------+ @
\ | \ | | -+-
\ | ->| | |
\| +---------+ / \
+
The
End Client DAG Directory Directory
Users Software System Server Service
Core Software ProvidersFigure 3.1 The role of the DAG system 🎀
3.3 Client Interface 🌸
the DAG will respond to End-User queries in:
- e-mail (SMTP)
- WWW (HTTP)
- LDAPv2
- Whois++
- LDAPv3
the DAG will provide responses that include the agreed-upon data. for access protocols that can handle referrals, responses will be data and/or referrals in that query protocol. those are Whois++ and LDAPv3 🍓 N.B.: the LDAPv3 proposal defines a referral as a URL, and no limitation is placed on the access protocol. however, it can't be assumed that every client can handle every access protocol, so only referrals to LDAPv3 servers will be returned 🙈
3.3.1 Acceptable User Input 💖
User Input is defined in terms of:
- Searchable Attributes
- Matching semantics
- Character sets
these, together with the DAG schema defined in Appendix A, form the basis of the required query expression. the individual queries get more detail in the Client Access Point (DAG-CAP) component descriptions for each supported protocol 🫶
Supported Query Types 💅
the DAG system is designed to support fragment-matching queries on a limited set of data attributes: "Name", "Organizational Role", "Organization", and "Locality". the permitted query combinations of attributes are listed in Table 3.1. not every combination of the three attributes is supported. only the ones needed for the desired functionality made the cut ✂️
Symbol Description
------- -----------
N Name
NL Name + Locality
NO Name + Organization
NOL Name + Organization + Locality
RO Role + Organization
ROL Role + Organization + LocalityTable 3.1 DAG-supported queries
the RO and ROL queries are separated from the rest because they search for "virtual" persons, meaning roles within an organization (for example, president or customer service desk) where you might want to find contact info 👑
Matching Semantics 🦋
as suits the individual client query protocols, more string matching expressions may be provided. the basic semantics of the DAG expect the following to be available in all client access software (where relevant):
- Full word, exact match
- Word substring match (E.g., "cat" would match "scatter")
- Case-sensitive and case-insensitive matching
TISDAG: LDAP/X.500 supports case-sensitivity as such, but some of the most-used attributes, such as the commonName attribute, are defined in the standard as case-insensitive attributetypes. the impact on the DAG system is that even if the index collected from an LDAP/X.500 server has upper and lower case letters in the tokens, they can't be handled as such, since that would be inferring meaning in something natively regarded as meaningless. the conclusion is that the Referral Index should be case-insensitive, and case-sensitivity should be supported by the SAPs if the native access protocol supports it. 🌈
Character Sets 🧚
wherever possible, the DAG System supports and promotes the use of Unicode Version 2.0 for character sets (see [21]), specifically the UTF-8 encoding (see Appendix A.2 of [21] or [20]). accommodation is made, where necessary, to support the deployed base of existing software.
specifically:
DAG/IP: all internal communications using the DAG/IP are carried out in UTF-8.
TISDAG: not just UTF-8, but UTF-8 based on composed UNICODE version 2 character encodings. 💜
DAG-CAP input: where specific access protocols permit selection of character sets, DAG-CAPs must support UTF-8. they may additionally support other anticipated character set encodings.
DAG-SAP communications with WDSPs: where specific access protocols permit selection of character sets, DAG-SAPs must support UTF-8 and use UTF-8 whenever the remote WDSP supports it. they may additionally support other character set encodings.
CIP Index Objects: the Index Objects supplied by the WDSPs to the DAG system shall contain data encoded in UTF-8.
TISDAG: the same limitation as for DAG/IP applies, meaning the basic data should be UTF-8 encoded composed UNICODE version 2 character encodings. 🔥
3.3.2 Data Output Spec 🍓
Schema Definition 💅
the schema used for the DAG service is defined in Appendix A. it's a very basic information schema, meant to carry the info the DAG service needs and nothing more. generic "whitepages" schema definitions do exist, but the more sophisticated and detailed the information presentation, the harder it is to map the schema seamlessly across protocols with different paradigms. so the "KISS" ("Keep it simple, sir") principle seems right here ✨
individual DAG-CAPs define how they express this schema.
Referral Definition 🌸
for client access protocols that use the concept of referrals, DAG-CAP definitions will define how referrals get expressed in those protocols. the DAG/IP defines the expression of referrals (see Appendix C).
Error conditions 🙈
each DAG-CAP may provide more detailed error messages, but it will minimally define support for these error conditions:
- unrecognized query
- too many hits
apart from these errors, the DAG-CAP may choose to refuse a query by redirecting the End-User to a different DAG-CAP of the same protocol 🦋
3.4 Directory Server Interface 👑
the DAG will use the Common Indexing Protocol (CIP) server-server protocol to get updated index objects from WDSPs. for query-routing purposes, WDSPs are expected to provide a Whois++, LDAPv2, or LDAPv3 interface to their data (even if their preferred access is something completely different). N.B.: in the responses from the technical survey, all respondents currently provide access to their service in one of these protocols 🫶
to provide a useful and uniform service, WDSPs are expected to provide 7x24 access to their whitepages information. WDSPs are also expected to implement operations, administration, maintenance, and provisioning processes designed to minimize service downtime for both planned and unplanned administration and maintenance activities. ⭐
4.0 Architecture 🌈
4.1 Software Components 🧠
the conceptual architecture of the DAG is represented in Figure 4.1. the general architectural specs are described below, followed by the individual component specs in Sections 5.5 through 5.12.
4.1.1 Internal Communications 🛸
communications between components of the DAG will be over TCP/IP connections, using the DAG-Internal Protocol (DAG/IP). DAG/IP is used by DAG-CAPs to talk to the Referral Index and DAG-SAPs. so the DAG/IP defines:
- the DAG-CAPs' range of query ability in the Referral Index (to gather referrals in response to the End-User's requests)
- the responses (and their formats) of the Referral Index to the DAG-CAP requests
- the DAG-CAPs' range of query ability to the DAG-SAPs for pursuing referrals when the DAG-CAP needs to do chaining for the client access software
- the responses (and their formats) of the DAG-SAPs to the DAG-CAPs
the detail of the planned DAG/IP is given in Appendix C. the detail of the DAG-CAP--Referral Index and DAG-CAP--DAG-SAP interactions is given in the definitions of the individual DAG-CAPs and DAG-SAPs, below (Sections 5.5 through 5.12) 💜
4.1.2 Referral Index 🍓
the Referral Index is responsible for maintaining the index of WDSP information and for providing a list of reasonable referrals in response to DAG-CAP search requests. these "referrals" are pointers that identify WDSPs which may have information matching the End-User's query 🎀
4.1.3 DAG-CAPs ✨
okay so DAG-CAPs are the ones who handle the client access protocol interface bc they're the front door for the DAG service 🚪💖 they take end-user queries in whatever query access protocol the user is speaking, translate that into a query for the Referral Index (aka expressed in DAG/IP), and then translate the Referral Index's answer back into something the client's protocol can understand 🦋 the answer might just be passing the referrals straight back, or the DAG-CAP can call on DAG-SAPs to turn the referral into actual results ("chaining") 🔗, or it can do a mix of both. it's giving flexibility, bestie 💅
+-------------------------------------+
|+====+ |
HTTP <-->+| |<------+ (Full chaining) |
|| | | |
|+====+ | |
| | +----+|
| | Referral-->| ||
| | Result <--| |+<--> Whois++
| | +----+|
|+====+ | |
SMTP <-->+| |<------+ (Full chaining) |
|| | | |
|+====+ | |
| | +----+|
| | Referral-->| ||
| | Result <--| |+<--> LDAPv2
| | +----+|
|+====+ | |
Whois++<-->+| |<------+ (Chain LDAPv2/3) |
|| | | |
|+====+ | |
| | +----+|
| | Referral-->| ||
| | Result <--| |+<--> LDAPv3
| | +----+|
|+====+ | |
LDAPv2 <-->+| |<------+ (Full chaining) |
|| | | |
|+====+ | |
| | |
|+====+ | |
LDAPv3 <-->+| |<------+ (Chain Whois++) |
|| | | |
|+====+ | |
| | |
| v |
| +-----------------------+ |
| | Referral Index |<---------------> Common
| | | | Indexing Protocol
| +-----------------------+ | (CIP)
+-------------------------------------+
All internal communications are in DAG/IP.
Figure 4.1 Conceptual Architecture of the DAG4.1.4 DAG-SAPs 🧚
okay and now the DAG-SAPs are the ones who get called in (by DAG-CAPs) to take those DAG-generated referrals and actually chase them down 🏃♀️✨ they issue the indicated query at the specified WDSP service, then convert whatever comes back from the WDSP into DAG/IP-specific format so the DAG-CAP that asked can use it. each DAG-SAP is in charge of handling referrals to one particular kind of WDSP protocol (like LDAPv2, Whois++, etc) 🌸
4.2 Important Architectural Notes 📚
this section is where the spec spills its tea on the thinking behind the architecture and software design of the DAG system 🫖 it's basically the context you need before reading the software specs, and it should give you clues about how the DAG system could be extended later 🔮 it also kinda works as an FAQ, since the content got shaped by the questions people asked during the tech spec phase. so if something in the software specs feels confusing on a first read, this is where it gets illuminated 💡
4.2.1 2 Distinct Functions: Referrals and Chaining 🎀
at all times you gotta remember that the main job of the DAG system is to give users referrals to WDSP services that might have the info they're after 👑 not every client protocol can handle referrals though, so the DAG system also offers a chaining service that pursues referrals the user's client can't handle on its own 🫶 the chaining service does try to match the user's query against data from WDSPs, but that's a secondary, support function. in the perfect future, all access protocols will be able to handle all referrals! 🥹✨
4.2.2 Limited Query and Response Semantics 🍓
the DAG system is NOT trying to be a chameleon or the ultimate whitepages query service 🦎🙅♀️ it focuses on giving referrals for a limited number of query types, the ones outlined in the functional specifications of the DAG service. so it's a great place to start a search, but refinements and detailed inquiries are beyond its scope 💅
4.2.3 Visibility 👀
given how limited the DAG query syntax is, it won't always be possible to exactly match a query posed to a CAP into a query posed to a SAP 😬 which means that, for instance, a LDAPv2 client that sends a query to the DAG system which then gets chained to an LDAP server might not get the same results as it would if the client were directly connected to that server 💀
4.2.4 Richness of Query semantics 🌈
even the limited query syntax of the DAG system can express queries that might NOT be possible to represent in the access protocols of the WDSPs 🤯 in those cases, the DAG-SAP either can refuse the query or try to emulate it 🎭
4.2.5 N+M Protocol Mappings 🧩
when the DAG system does chaining, a certain amount of mapping between protocols is required 🔀 in theory there are "N" allowable end-user query access protocols and "M" supported WDSP server protocols. so the software is built to use a single internal protocol (the DAG/IP) and data schema, which gives every component a common language 🗣️💜 without that, each input protocol module (DAG-CAP) would have to handle every WDSP protocol, which means NxM protocol mappings. that would make the system complex and way harder to expand with new protocols in the future 😵💫
4.2.6 DAG-CAPs and DAG-SAPs are completely independent of each other 🧁
for all the reasons above, the DAG-CAP and DAG-SAP modules are meant to be completely independent of each other 🙌 a DAG-SAP responds to a query posed to it in the DAG/IP, without caring about the protocol of the DAG-CAP that passed the query along 🫠
4.2.7 The Role of the DAG-CAP 🎭
so the DAG-CAP is the one responsible for using the DAG/IP to get referral information and, where necessary, chained responses 💅 where needed, it makes adjustments to handle the differences in semantics between the DAG/IP and its native protocol. this might involved doing post-filtering of the results returned by the DAG-SAPs, since the query issued in DAG/IP to the DAG-SAP might be "broader" then the original query 🔍
thus, the DAG-CAP "knows" only 2 protocols: its native protocol, and the DAG/IP 🫶
4.2.8 The Role of the DAG-SAP 🛠️
similarly, the DAG-SAP is responsible for answering DAG/IP queries by contacting the designated WDSP server 📞 where necessary, it makes adjustments to handle the differences in semantics between the DAG/IP and its native protocol. those adjustments might mean that the DAG-SAP ends up with results that don't match the original query 😭 in that case the DAG-SAP should attempt to do post-pruning to reduce the mismatch between the original query and the results returned ✂️
so the DAG-SAP "knows" only 2 protocols too: its native protocol, and the DAG/IP 💖
4.2.9 DAG/IP is internal 🔒
no module outside of the DAG system should be aware of the DAG/IP's construction 🤫 end-users use the query protocols supported by the DAG-CAPs, and WDSPs get contacted using the query protocols supported in the DAG-SAPs. the DAG/IP stays behind the curtain 🎀
4.2.10 Expectations 🌍
the expectation is that the DAG system, even though it's defined as a single construct, will run its modules on several different computers that might be widely distributed in terms of geography and ownership 🗺️✨ for that reason, the DAG/IP is specified so that it works over inter-machine communications 🖥️💕
4.2.11 Future Extensions 🔮
the DAG system architecture was built with extensibility in mind 🦋 at any time, an individual component can be improved (for example, the Mail DAG-CAP might get a different query interface) without disrupting the system 🙌
on top of that, future versions of the DAG system might support other access protocols, both for end-users and for WDSPs 🌟
5.0 Software Specifications 💻
5.1 Notational Convention 📝
it's always a challenge to accurately represent a text protocol in a printed document 😩 like, when is a new line a "newline," and when is it just an effect of the text formatter? to illustrate things properly, this document includes lots of segments of protocol grammars, sample data, and sample input/output in a text protocol. to tell apart the newlines that actually matter in a protocol, the symbol <NL> is used 🫧 for example:
This is an example of a very long line of input. There is only one
newline in it (at the end), in spite of the fact that this document
shows it spanning several lines of text.<NL>5.2 DAG-CAP Basics 🌷
5.2.1 Functionality ✨
okay bestie, every DAG-CAP has to support the FULL range of DAG queries from 3.3.1 💅 no skipping, no cap 🫶
each DAG-CAP takes queries in its own native protocol 🦋 and each individual DAG-CAP definition spells out what those native queries are supposed to look like 🌸
the DAG-CAP is then responsible for:
- converting that expression into a query in the DAG/IP to get the relevant referrals from the Referral Index 🧚 this might mean that parts of the original query get ignored (like if it had attributes the DAG application doesn't support, or if the query algebra wasn't supported by the DAG application) 🙈
- returning referrals in the client's native protocol, where possible 💖
- expressing the client query to the necessary DAG-SAPs, given the limitations above, so they can chain the referrals that can't usefully be expressed in the client's native protocol 🔥
- possibly doing post-filtering on the DAG-SAP results 💜 and
- converting the collected DAG-SAP results so they're expressed in the client's native protocol (and schema, where that applies) ✨
each DAG-CAP defines what the interaction with the end-user looks like (e.g., synchronous or asynchronous, etc) 🎀 also, every DAG-CAP must be able to do the following so the DAG system can do load-limiting and load-balancing 👑
- direct the client to a different DAG-CAP of the same type (for load-balancing)
- decline to return results because too many referrals were generated (to discourage data-mining). ideally, this should include generating a message to refine the query so it produces a more manageable number of referrals/replies 💀
DAG-CAPs must be capable of accepting and respecting DAG-SAP service referrals (for DAG-SAP load-sharing) 🌈
in protocols that allow it, the DAG-CAP should tell the end-user which services were unavailable for chaining referrals 🥺 (so they know there were parts of the search that couldn't be completed, and information might be missing 😭)
TISDAG: any CAP that receives commands other than queries, like help, answers those on its own. a CAP should not pass any system command on to the RI. 🍓
5.2.2 Configuration 🎀
it must be possible to change the expected address of the DAG-CAP by configuring the software (i.e., host and port, e-mail address, etc) 🌸
for DAG-CAPs that need to access DAG-SAPs for query chaining, the DAG-CAP must be configurable for each type (protocol) of DAG-SAP it needs, in terms of:
- at least one known DAG-SAP of every necessary protocol to contact
- for each DAG-SAP, the host and port of the DAG-SAP software
the DAG-CAPs must also be configurable in terms of a maximum number of referrals to handle for a user transaction. (so, to prevent data mining, the DAG-CAP will refuse to reply if the query is too general and too many hits come back from the Referral Index 🙅♀️)
the DAG-CAP must be configurable in terms of alternate DAG-CAPs of the same type that the end-user software can be sent to if this one is too busy 🦋
5.2.3 Error handling 💅
apart from error conditions that come from the DAG-CAP's own operation, DAG-CAPs are responsible for communicating error conditions that happen elsewhere in the system and affect the outcome of the user's query (e.g., in the DAG-RI, or in one or more DAG-SAPs) 🚨
if the DAG-CAP sends a query to the DAG-RI and gets an error message back, it should try to match the received DAG errorcode to its native access protocol's error codes 🔍 the same goes when the DAG-CAP is "chaining" the query to one DAG-SAP 🔗
sometimes the DAG-CAP may have to combine multiple errorcodes into a single expression for the user 🧩 when the DAG-CAP is "chaining" the query through DAG-SAPs to one or more WDSPs, there can be a mix of responsecodes from the DAG-SAPs 😵💫 if that happens, the DAG-CAP should try to forward information to the end-user software that is as specific as possible, like which of the WDSPs couldn't fulfill the query and why 🫠
see Appendix D for more info on error condition message mappings 📖
5.2.4 Pruning of results 🌸
there's no perfect match between the query syntaxes of the DAG system and the different access protocols that the DAG-CAPs and DAG-SAPs support 🥲 so there will be times when the results a DAG-CAP has to collect are "broader" than they would be with a perfect match 🕸️ this might have bad effects on the system, like administrative limits getting "unnecessarily" exceeded on WDSPs, or the collected results going over the sizelimit of the DAG-CAP 📈
BUT, the DAG-CAP is the only part of the DAG system that actually knows what the original query was 👀 so it can prune the results it got from the DAG-SAPs so the results shown to the client better match the original question 💖✨
5.3 DAG-SAP Basics 🧚
5.3.1 Functionality 💜
every DAG-SAP must support the FULL range of DAG queries from 3.3.1 🫶 results must be complete DAG schemas in well-formed DAG/IP result formats (see Appendix C) 📋 each DAG-SAP takes queries in DAG/IP and converts them to the native schema and protocol it's designed to proxy 🔁
the DAG-SAP is then responsible for
- converting the query into the native schema and protocol of the WDSP that the referral points to. (if the query can't be represented in the native protocol, it must return an error message. if it can be emulated, the DAG-SAP can try emulating it by posing a related query to the WDSP and post-pruning the results it gets back 🪄)
- contacting that WDSP, using the host, port, and protocol information provided in the referral 📞
- negotiating the query with the remote WDSP 🤝
- accepting results from the WDSP, possibly doing post-filtering on the result set 🧹 and
- conveying the results back to the calling DAG-CAP using the DAG/IP and its schema 💌
note that this implicitly means the DAG-SAP is responsible for chaining and pursuing any referrals it gets from WDSP services 🔗 the DAG-SAP only returns search results to the DAG-CAP that called it 🎯
5.3.2 Configuration 🎀
DAG-SAPs must be configurable to accept connections only from recognized DAG components 🔐
DAG-SAPs that have service limits must be configurable to redirect DAG-CAPs to alternate DAG-SAPs of the same type when necessary 🦋
5.3.3 Error handling 🚨
a DAG-SAP must translate error codes it gets from a WDSP server into DAG error codes according to Appendix D 🔄
5.3.4 Pruning of results ✂️
since it might not be possible to exactly map a DAG query into a query in the access protocol the DAG-SAP supports, the DAG-SAP should try to translate it into a more general query (or, if necessary, into a set of queries) 🌀 if so, the DAG-SAP must then prune the result set it received before passing it on to the DAG-CAP 🧼
5.3.5 Constraint precedence 👑
some constraints, search and case, can appear both as local and global constraints 🤔 if this happens in a query, then the local constraint specification overrides the global one 💅 for a query like the following:
fn=leslie;search=exact and org=think:search=substringthe resulting search constraint for fn=leslie will be exact, while the one for org=think will be substring 🌈✨
5.4 The Referral Index 🧠
5.4.1 Architecture 🏗️
the Referral Index contains (only) the information needed to deliver referrals to DAG-CAPs based on the query types the DAG itself supports 🎀 it creates an index over these objects so it can respond to DAG-CAP queries using the DAG/IP 📇 the information is pulled directly from interactions with participating WDSPs' software, using the Common Indexing Protocol (CIP) 🫶
5.4.2 Interactions with WDSPs (CIP) 🤝
WDSPs that want to take part in the DAG system must register themselves (see Section 5.4.6) 📝 once registered, the Referral Index will interact with the WDSPs using the Common Indexing Protocol as defined in [1], using the Index Object defined in Section 5.4.3 💖
5.4.3 Index Object Format 🧾
the CIP index object type is based on the Tagged Index Object as defined in [12] 🏷️ Appendix E details the expected content of the index objects as the WDSPs are to provide them 📦
TISDAG: the tokens in the Tagged Index Object should be UTF-8 encoded composed UNICODE version 2 character encoding. 🌸
5.4.4 DAG-Internal I/O 🔁
the Referral Index interacts with the rest of the DAG internal modules (DAG-CAPs) by listening for queries and responding in the DAG/IP (defined in Appendix C) 👂
5.4.5 The Index Server 🖥️
the Referral Index must index the necessary attributes of the CIP index object so it can respond to queries of the form described in Table 3.1 📊
the semantics of the chosen CIP object (defined in Appendix E) are such that a referral to a WDSP server is sent back if (and only if)
- the index object of the WDSP contains all the tokens of the query, in the attributes specified, according to the logic of the DAG/IP query, and
- all of those tokens are found with a common tag.
so, a query for the name "Fred Flintstone" (2 tokens) will yield a referral to a server that has a record for "Fred Amadeus Flintstone", but NOT to a WDSP with 2 differently tagged records, one for "Fred Amadeus" and one for "Julie Flintstone" 🙃 depending on the access protocol and the original end-user query, the referral to the WDSP with "Fred Amadeus Flintstone" may or may not yield a successful result 🤷♀️ but it's known that the other WDSP would NOT have yielded successful searches 🙅♀️ so the referral approach may give false-positive results, but it will not miss appropriate WDSPs 💜✨
5.4.6 Configuration 🎀
the Referral Index must provide the ability to register interested WDSPs, as outlined in Appendix E 📋
the Referral Index must be able to configure the port for DAG/IP communications 🔌 also, it must be configurable to recognize only registered DAG-CAPs 🔒
5.4.7 Security 🛡️
the Referral Index will accept queries only from recognized (registered) DAG-CAPs 🙌 this will reduce "denial of service" attack types 🚫 but it's also a reflection of the fact that the Referral Index uses the DAG/IP (i.e., internal) protocol, which should not be exposed to non-DAG software 🫣
the Referral Index must be able to use authenticated communication to receive data from WDSPs (see Appendix E) 🔐
5.5 Mail (SMTP) DAG-CAP 📬
this is the default Mail DAG-CAP 💌 more sophisticated ones could definitely be written, e.g., for pretty-printed output, or for handling different philosophies of case-matching 🌈
this DAG-CAP has been designed on the assumption that mail queries will be human-generated (i.e., using a mail program or text editor), as opposed to queries formulated by software agents 🙋♀️ the input grammar should therefore be simple and liberal in accepting variations of whitespace formatting 🧸🎀
5.5.1 Mail DAG-CAP Input ✨
okay so the Mail DAG-CAP takes input as a regular or MIME-encoded ([9] and [10]) SMTP mail message 💌 sent to an advertised mail address. It parses your message and replies with a MIME-encoded message holding the DAG search results 🦋
bestie, only one query per email. Anything you write after a single valid query gets read and then ignored 🙈
The query in the body has to follow this syntax 💅 Also, all the input control terms ("type=", "name=" etc) are written in lower case here just for convenience. You can type them in upper case or mixed case and it's totally fine 🫶
mailquery = [mnl] [controls] mnl terms mnl
controls = [msp] "searchtype" [msp] "=" [msp]
( matchtype /
casetype /
matchtype msp casetype /
casetype msp matchtype /
<nothing> )
matchtype = "substring" / "exact"
; default: substring
casetype = "ignore" / "sensitive"
; default: ignore
terms = n / n-l / n-o / n-o-l / r-o / r-o-l
n = n-term
n-l = ( n-term l-term / l-term n-term)
n-o = ( n-term o-term / o-term n-term )
n-o-l = ( n-term o-term l-term /
n-term l-term o-term /
l-term n-term o-term /
l-term o-term n-term /
o-term l-term n-term /
o-term n-term l-term )
r-o = ( r-term o-term / o-term r-term )
r-o-l = ( r-term o-term l-term /
r-term l-term o-term /
l-term o-term r-term /
l-term r-term o-term /
o-term l-term r-term /
o-term r-term l-term )
n-term = [msp] "name" [msp] "=" [msp] string mnl
o-term = [msp] "org" [msp] "=" [msp] string mnl
l-term = [msp] "loc" [msp] "=" [msp] string mnl
r-term = [msp] "role" [msp] "=" [msp] string mnl
string = <US-ASCII or quoted-printable encoded
<ISO-8859-1 or UTF-8 except nl and sp>
msp = 1*(sp)
sp = " "
mnl = 1*(nl)
nl = <linebreak>Here are some valid mail queries 🎀
Example 1:
searchtype = <NL>
name = thinking cat<NL>Example 2:
searchtype = exact ignore<NL>
name=thinking cat<NL>Example 3:
role=thinking cat<NL>
org =space colonization<NL>Example 4:
name=thinking cat <NL>
<NL>
<NL>
My signature line follows here in the most annoying
fashion <NL>And these are not acceptable queries 💀
Example 5:
searchtype= exact substring <NL>
name = thinking cat <NL>Example 6:
name=thinking cat org= freedom fighters anonymous<NL>In Example 5 there are two conflicting searchtypes, which is a no-go 🙅 In Example 6 there's no linebreak after the n-term, so that's also a no-go 😭
5.5.2 Translation from Mail query to DAG/IP 🔥
Querying the Referral Index
The big trick when turning Mail DAG-CAP input into the DAG/IP query format is to "tokenize" the input terms into single token elements 🧩 For example, this n-term:
name= thinking cat<NL>gets tokenized into 2 n-tokens:
thinking
catThose then get mapped into the following in the DAG/IP query (dag-n-terms):
FN=thinking and FN=cat<NL>It's the same deal for all r-terms, l-terms and o-terms 💖 The main steps for translating mail input into a DAG/IP query are:
- translate quoted-printable encoding, if necessary
- translate base64 encoding, if necessary
- tokenize the strings for each term
- construct the DAG/IP query from the resulting components, as described in more detail below
DAG/IP constraints get built from the searchtype info in the query ✨
dag-matchtype = "search=" <matchtype> /
"search=substring" ; if matchtype not
; specified
dag-casetype = "case=ignore" / ; if casetype not
; specified or
; casetype=ignore
"case=consider" ; if casetype=sensitive
constraints = ":" dag-matchtype ";" dag-casetypeThe terms for the DAG/IP query come from the tokenized strings in the mail input 🌸
dag-n-terms = "FN=" n-token 0*( " and FN=" n-token)
dag-o-terms = "ORG=" o-token 0*( " and ORG=" o-token)
dag-l-terms = "LOC=" l-token 0*( " and LOC=" l-token)
dag-r-terms = "ROLE=" r-token 0*( " and ROLE=" r-token)So the relevant DAG/IP queries come in one of two types 👑
dagip-query = ( ( ( n-query / nl-query / no-query /
nol-query ) [" and template=DAGPERSON"]":"
dag-matchtype ";" dag-casetype) /
( ( ro-query / rol-query )
[" and template=DAGORGROLE"]":"
dag-matchtype ";" dag-casetype) )
n-query = dag-n-terms
nl-query = dag-n-terms " and " dag-l-terms
no-query = dag-n-terms " and " dag-o-terms
nol-query = dag-n-terms " and " dag-o-terms " and "
dag-l-terms
ro-query = dag-r-terms " and " dag-o-terms
rol-query = dag-r-terms " and " dag-o-terms " and "
dag-l-termsHere's how the examples from earlier get translated 🦋
Example 1:
FN=thinking and FN=cat:search=substring;case=ignore<NL>Example 2:
FN=thinking and FN=cat:search=exact;case=ignore<NL>Example 3:
ROLE=thinking and ROLE=cat and ORG=space and
ORG=colonization:search=substring;case=ignore<NL>Querying a DAG-SAP
When you're querying a DAG-SAP (no matter which protocol that DAG-SAP speaks), the DAG/IP query has to include info about the target WDSP server 🎯 That info is pulled from the SERVER-TO-ASK referral info in the Referral Index, and gets appended to the query as specified in Appendix C):
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charsetThe response from the Referral Index included:
"# SERVER-TO-ASK " serverhandle nl
" Server-info: " serverinfo nl
" Host-Name: " hostname nl
" Host-Port: " number nl
" Protocol: " prot nl
" Source-URI: " source nl
" Charset: " charset nl
"# END" nlAnd the "quoted-hostname" and "quoted-serverinfo" come from "hostname" and "serverinfo" respectively, by quoting the DAG/IP special characters 🧁
For example, this referral:
# SERVER-TO-ASK dagsystem01<NL>
Server-info: o=thinkingcat, c=se<NL>
Host-Name: thinkingcat.com<NL>
Host-Port: 2839<NL>
Protocol: ldapv2<NL>
Source-URI: http://www.thinkcat.com
Charset: T.61<NL>
# END<NL>would yield this addition:
:host=thinkingcat\.com;port=2839;server-info=o\=thinkingcat\,\
c\=se;charset=T\.61in its query to an LDAPv2 DAG-SAP 💜
(N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response) 🤓
Note that it's the DAG-SAP's job to pull these terms out of the query and use them to figure out which WDSP server to contact 🫶 See the individual DAG-SAP definitions, below.
5.5.3 Chaining queries in Mail DAG-CAP 🔗
The Mail DAG-CAP has to chain all referrals -- to the Whois++ DAG-SAP, LDAPv2 DAG-SAP, or LDAPv3 DAG-SAP as appropriate for the referral 🌈
5.5.4 Expression of results in Mail DAG-CAP 💌
The results message gets sent to the "Reply-To:" address of the originating mail, if there is one (see [4] for appropriate interpretation of mail originator headers) 📬 The original query is repeated, along with the message-id. The rest of the body is the concatenation of responses from the DAG-SAP calls. Each result has the WDSP's SOURCE URI (from the referral) tacked on the end, and the system messages get removed 🧹
At the end of the message, the WDSP servers that failed to respond (meaning the DAG-SAP handling the referral returned the % 403 Information Unavailable message) get listed with their server-info 😭
5.5.5 Expression of Errors in Mail DAG-CAP 🚨
If the mail DAG-CAP gets a message that it can't parse with the query grammar above, it sends back an explanatory message to the query mail's reply address. That message says the query couldn't be interpreted and describes what valid queries look like 🙈
If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (this is for detecting data-mining efforts, or otherwise refusing over-general queries, such as "FN=svensson"), the mail DAG-CAP sends an explanatory message to the query mail's reply address. That message describes the "over-generalized query" problem, suggests the user resubmit a more precise query, and describes the list of valid query types 💅
If the mail DAG-CAP gets several different result codes from the DAG-SAPs, it should represent those in an appropriate manner in the response message 🎀
A mail DAG-CAP may redirect a connection to another mail DAG-CAP for reasons of load-balancing 🌸 This is done simply by forwarding the mail query to the address of the alternate mail DAG-CAP 🦋
5.6 Web (HTTP) DAG-CAP ✨
5.6.1 Web DAG-CAP Input
okay so the web DAG-CAP talks over plain old standard HTTP 💖 The general vibe is a form page with radio buttons to pick "substring or exact match" and "consider case or ignore case" 🎀 Everything else (name, role, organization, locality) is just free-form text boxes 🌸
The DAG-CAP takes queries through an HTTP "post" method. That's the form action from the page above, or the query could be generated somewhere else ✨ The rest of this section lists the variables that go in that post. The actual page layout and most UI stuff is up to the builder 💅 Also, the Web DAG-CAP might have to send responses in different content encodings, so it must therefore address the Accept-Encoding: request header in the HTTP connection 🫶
HTTP can't do referrals on its own 🙃 but two extra variables let the client ask for referral info and then chase each referral through the Web DAG-CAP itself, like it's a proxy for those referrals 🦋 These are the extra "control variables": one to request referrals only, and one to show that the transaction is a continuation of an earlier query that's pursuing a referral.
Now, there's been a call for a "machine-readable" version of the search output 🤖 HTML is built for visual layout, so user agents that want to do something with the results other than show them in an HTML browser don't have many cues for pulling out the useful bits. Also, "minor" visual changes that come with big HTML updates can break user agents that were written to blindly parse the original HTML 😭 So provision has been made to return "raw" format results 💜 You ask for these by putting Accept-Content: application/whoispp-response in the request header of the HTTP message sent to the HTTP DAG-CAP.
The variables that are expected are:
transaction = "new" / "chain" ; default is "new". This
; should not be user-settable. It is used
; in constructed URLs
resulttype = "all" / "referrals" ; default is "all"
matchtype = "substring" / "exact"
casetype = "case ignore" / "case sensitive"
n-term = string
o-term = string
l-term = string
r-term = string
host-term = string
port-term = string
servinfo-term = string
prot-term = string ; the protocol of the referral
string = <UNICODE-2-0-UTF-8> / <UNICODE-1-1-UTF-8> /
<ISO-8859-1>5.6.2 Translation from Web query to DAG/IP
Querying a DAG-SAP Directly
okay, this one's a glow-up 🌟 If the transaction variable is "chain", the info in the POST is used to chase down one particular referral, not to search the Referral Index. The right DAG-SAP (figured out from the prot-term) gets contacted and sent the query directly 💅
Results from this type of query are always full results (i.e., not referrals) 🍓
Querying the Referral Index
a big part of turning Web DAG-CAP input into a DAG/IP query is to "tokenize" the input terms into single token elements 🧚 For example, this n-term:
name= thinking catgets tokenized into 2 n-tokens:
thinking
catand those get mapped into this in the DAG/IP query (dag-n-terms):
FN=thinking and FN=catThe same goes for the r-term, l-term and o-term 💖
The main steps for translating the HTTP input into a DAG/IP query are:
- translate encodings, if necessary
- tokenize the strings for each term
- construct the DAG/IP query from the resulting components, as described in more detail below
DAG/IP constraints get built from the searchtype info in the query ✨
dag-matchtype = "search=" <matchtype> /
"search=substring" ; if matchtype not
; specified
dag-casetype = "case=ignore" / ; if casetype not
; specified or
; casetype="case ignore"
"case=consider" ; if casetype=
; "case sensitive"
constraints = ":" dag-matchtype ";" dag-casetypeThe terms for the DAG/IP query come from the tokenized strings in the HTTP post input 🎀
dag-n-terms = "FN=" n-token 0*( " and FN=" n-token)
dag-o-terms = "ORG=" o-token 0*( " and ORG=" o-token)
dag-l-terms = "LOC=" l-token 0*( " and LOC=" l-token)
dag-r-terms = "ROLE=" r-token 0*( " and ROLE=" r-token)So the relevant DAG/IP queries end up as one of two types:
dagip-query = ( ( ( n-query / nl-query / no-query / nol-query )
[" and template=DAGPERSON"]":" dag-matchtype
";" dag-casetype) /
( ( ro-query / rol-query )
[" and template=DAGORGROLE"]":" dag-matchtype
";" dag-casetype) )
n-query = dag-n-terms
nl-query = dag-n-terms " and " dag-l-terms
no-query = dag-n-terms " and " dag-o-terms
nol-query = dag-n-terms " and " dag-o-terms " and "
dag-l-terms
ro-query = dag-r-terms " and " dag-o-terms
rol-query = dag-r-terms " and " dag-o-terms " and "
dag-l-termsQuerying a DAG-SAP
no matter which protocol the DAG-SAP speaks, the DAG/IP query has to include info about the target WDSP server 🫶 That info comes from the Referral Index SERVER-TO-ASK referral information, and gets appended to the query as specified in Appendix C:
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charsetThe response from the Referral Index included:
"# SERVER-TO-ASK " serverhandle <NL>
" Server-info: " serverinfo <NL>
" Host-Name: " hostname <NL>
" Host-Port: " number <NL>
" Protocol: " prot <NL>
" Source-URI: " source <NL>
" Charset: " charset <NL>
"# END" <NL>The "quoted-hostname" and "quoted-serverinfo" come from "hostname" and "serverinfo" respectively, after quoting the DAG/IP special characters 🦋
For example, this referral:
# SERVER-TO-ASK dagsystem01<NL>
Server-info: o=thinkingcat, c=se<NL>
Host-Name: thinkingcat.com<NL>
Host-Port: 2839<NL>
Protocol: ldapv2<NL>
Source-URI: http://www.thinkingcat.com
Charset: T.61<NL>
# END<NL>would add this to its query to an LDAPv2 DAG-SAP:
:host=thinkingcat\.com;port=2839;server-info=o\=thinkingcat\,\
c\=se;charset=T\.61(N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response 📚)
Note that it's the DAG-SAP's job to pull these terms out of the query and use them to figure out which WDSP server to contact. See the individual DAG-SAP definitions, below 🌈
5.6.3 Chaining queries in Web DAG-CAP
so if the resulttype was "all", every referral that came back from the Referral Index gets chained through the appropriate DAG-SAPs 🔥 If only referrals were requested, the Referral Index results are returned instead 💅
5.6.4 Expression of results in Web DAG-CAP
text/html results
the default response encoding is text/html 🌸 If the resulttype was "all", the content of the chaining responses from the DAG-SAPs (minus the system messages) gets collated into one page response, with one result entry per demarcated line (like a bullet item) ✨ The FN or ROLE value should be shown first and clearly 👑 The SOURCE URI for each WDSP referral should be presented as an HREF for each of that WDSP's results 🔗
At the end of the message, the WDSP servers that failed to respond get listed with their server-info 😭 (that's when the DAG-SAP handling the referral returned the % 403 Information Unavailable message)
If, however, the resulttype was "referrals", the results from the Referral Index come back as HREF URLs pointing at the Web DAG-CAP itself. They carry the info needed to run the query, including the "HOST=", etc, for the referral 🦋 For example, if the original query was:
n-term="thinking cat"
resulttype="referrals"and it pulled this referral from the Referral Index:
# SERVER-TO-ASK DAG-Serverhandle<NL>
Server-Info: c=se, o=tce<NL>
Host-Name: answers.tce.com<NL>
Host-Port: 1111<NL>
Protocol: ldapv3<NL>
Source-URI: http://some.service.se/
Charset: UTF-8<NL>
# END<NL>then the response would be an HTML page with an HREF HTTP "POST" URL to the Web DAG-CAP, with these variables set:
n-term="thinking cat"
transaction="chain"
servinfo-term="c=se, o=tce"
host-term="answers.tce.com"
port-term="1111"
prot-term="ldapv3"The Source-URI should be set up in the response as its own HREF URI 🌟
application/whoispp-response Results
if the Accept-Encoding: " HTTP request header had the value "application/whoispp-response", the content of the HTTP response gets built in the same syntax and attribute mapping as the Whois++ DAG-CAP 💜
If the resulttype was "all", the Web DAG-CAP has already chained all the referrals, so the response includes only full data records 🍓
If the resulttype was "referrals", then all referrals get passed straight back in a single response, in correct Whois++ referral format (super convenient, since that's how they're already formulated in the DAG/IP) 🎀 Note that this includes referrals to LDAP-based services as well as Whois++ servers 🙈
5.6.5 Expression of Errors in Web DAG-CAP
so a Web DAG-CAP may redirect a connection to another web DAG-CAP for load-balancing reasons ⚖️ That's done simply by using an HTTP redirect 🦋
Standard Errors
if the web DAG-CAP gets a message it can't parse with the query grammar described above, it sends back an explanatory HTML page saying the query couldn't be interpreted, plus a description of valid queries 💀
If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (this is for catching data-mining efforts, or otherwise refusing over-general queries like FN=svensson), the web DAG-CAP sends a page with an explanatory message about the "over-generalized query" problem 🙅♀️ It suggests the user resubmit a more precise query and describes the list of valid query types 💅
If the web DAG-CAP receives more than one result code from the DAG-SAPs, it must represent them all in an appropriate manner in the response 🫶
application/whoispp-response Errors
an invalid query gets a simple text response with the error: % 500 Syntax Error 😭
If too many referrals are generated from the Referral Index, the simple text response will have the message % 503 Query too general 🙈
5.7 Whois++ DAG-CAP
TISDAG: The system commands polled-for/-by should elicit the empty set as a return value until we better understand the implications of doing otherwise. 🌸
5.7.1 Whois++ DAG-CAP Input
okay so input to the Whois++ DAG-CAP follows the Whois++ standard ([6]) 💅 and the Whois++ DAG-CAP MUST support these queries at the very least 🎀
Query Type Expression in Whois++
----------- ------------------------------------
N One or more "name=" and
template=USER
NL One or more "name=" and
One or more "address-locality=" and template=USER
NO One or more "name=" and
one or more "organization-name=" and template=USER
NOL One or more "name=" and
one or more "organization-name=" and
one or more "address-locality=" and template=USER
RO One or more "org-role=" and
one or more "organization-name=" and template=ORGROLE
ROL One or more "org-role=" and
one or more "organization-name=" and
one or more "address-locality=" and template=ORGROLE
Table 5.1 Allowable Whois++ QueriesThe following constraints MUST be supported for queries, bestie 🦋
"search=" (substring / exact)
"case=" (ignore / consider)If no constraints are defined in a query, the default is exact and ignore. It's giving simple defaults ✨ For example,
FN=foo and loc=kista and fn=bar<NL>
is a perfectly valid Whois++ NL query for "Foo Bar" in "Kista" 🍓
5.7.2 Translation from Whois++ query to DAG/IP
Querying the Referral Index
The Whois++ DAG-CAP formulates a DAG/IP query by forwarding the search terms it received (as defined in Table 5.1). Main character energy, it just passes the tea along 🫶
For example, the above query would be expressed as:
FN=foo and LOC=kista and FN=bar and template=DAGPERSON<NL>Querying a DAG-SAP
In querying a DAG-SAP (irrespective of the protocol of that DAG-SAP), the DAG/IP query MUST include information about the target WDSP server. This information is drawn from the Referral Index SERVER-TO-ASK referral information, and is appended to the query as specified in appendix C 🧚
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charsetwhere the response from the Referral Index included:
# SERVER-TO-ASK serverhandle<NL>
Server-info: serverinfo<NL>
Host-Name: hostname<NL>
Host-Port: number<NL>
Protocol: prot<NL>
Source-URI: source<NL>
Charset: charset<NL>
# END<NL>and the "quoted-hostname" and "quoted-serverinfo" are obtained from "hostname" and "serverinfo" respectively, by quoting the DAG/IP special characters. Quoting is literally the whole vibe here 💖
For example, the referral
# SERVER-TO-ASK dagsystem01<NL>
Server-info: o=thinkingcat, c=se<NL>
Host-Name: thinkingcat.com<NL>
Host-Port: 2839<NL>
Protocol: ldapv2<NL>
Source-URI: http://www.thinkingcat.com/
Charset: T.61<NL>
# END<NL>would yield the addition
:host=thinkingcat\.com;port=2839;server-info=o\=thinkingcat\,\
c\=se;charset=T\.61in its query to an LDAPv2 DAG-SAP 👑
(N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response). Read the appendix, it's iconic 📖
Note that it is the DAG-SAP's responsibility to extract these terms from the query and use them to identify the WDSP server to be contacted. See the individual DAG-SAP definitions, below 🌸
5.7.3 Chaining in Whois++ DAG-CAP
The Whois++ DAG-CAP relies on DAG-SAPs to chain any non-Whois++ referrals (currently, the LDAPv2 and LDAPv3 DAG-SAPs) 🔗✨
5.7.4 Expression of results in Whois++
Results are expressed in Whois++ by collating the DAG/IP results received from DAG-SAPs (using the FULL response), and using the template and attribute mappings defined in Appendix B. For each result from a given referral, the SOURCE attribute is added, with the value of the SOURCE-URI from the referral. Tiny detail, huge payoff 💅
Any referrals to other Whois++ servers provided by the Referral Index are sent directly to the Whois++ client as follows:
server-to-ask = "# SERVER-TO-ASK " DAG-Serverhandle<NL>
" Server-Handle: " SERVER-INFO<NL>
" Host-Name: " HOST<NL>
" Host-Port: " PORT<NL>
" Protocol: " PROTOCOL<NL>
"# END"<NL>where SERVER-INFO, HOST, PORT, PROTOCOL are drawn from the referral provided in the DAG/IP, and the SOURCE-URI information is lost. Yes it's sad, but that's the deal 😭
5.7.5 Expression of Errors in Whois++ DAG-CAP
As appropriate, the Whois++ DAG-CAP will express operational errors following the Whois++ standard. There are 4 particular error conditions of the DAG system that the DAG-CAP will handle as described below 🚨
When the Whois++ DAG-CAP receives a query that it cannot reply to within the (data) constraints of the DAG, it sends an error message and closes the connection. The error message includes
% 502 Search expression too complicated<NL>If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (for detecting data-mining efforts, or otherwise refusing over-general queries, such as "FN=svensson"), the Whois++ DAG-CAP will send an error message and close the connection. The error message includes
% 503 Query too general<NL>(N.B.: this is different from the "Too many hits" reply, which does send partial results.) Not the same vibe at all 🙈
A Whois++ DAG-CAP may redirect a connection to another Whois++ DAG-CAP for reasons of load-balancing. This is expressed to the end-user client software using the SERVER-TO-ASK response with appropriate information to reach the designated alternate DAG-CAP. Traffic cop energy 🚦
If a Whois++ DAG-CAP receives several different response codes from DAG-SAPs, it SHOULD try to represent them all in the response to the end-user client. Don't leave anyone out, bestie 🌈
The proposed mapping between DAG/IP response codes and Whois++ response codes is given in Appendix D. 🎀
5.8 LDAPv2 DAG-CAP
5.8.1 LDAPv2 DAG-CAP Input
Input to the LDAPv2 DAG-CAP follows the LDAPv2 standard ([19]). Minimally, the LDAPv2 DAG-CAP MUST support the following queries (adapted from the ASN.1 grammar of the standard) 🧁
BindRequest ::=
[APPLICATION 0] SEQUENCE {
version INTEGER (1 .. 127),
name LDAPDN,
authentication CHOICE {
simple [0] OCTET STRING,
krbv42LDAP [1] OCTET STRING,
krbv42DSA [2] OCTET STRING
}
}
BindResponse ::= [APPLICATION 1] LDAPResult
SearchRequest ::=
[APPLICATION 3] SEQUENCE {
baseObject "dc=se",
scope wholeSubtree (2),
derefAliases ENUMERATED {
neverDerefAliases (0),
derefInSearching (1),
derefFindingBaseObj (2),
derefAlways (3)
},
sizeLimit INTEGER (0 .. maxInt),
timeLimit INTEGER (0 .. maxInt),
attrsOnly BOOLEAN,
filter Filter,
attributes SEQUENCE OF AttributeType
}
Filter ::=
CHOICE {
and [0] SET OF Filter,
or [1] SET OF Filter,
not [2] Filter,
equalityMatch [3] AttributeValueAssertion,
substrings [4] SubstringFilter
}
SubstringFilter ::=
SEQUENCE {
type AttributeType,
SEQUENCE OF CHOICE {
initial [0] LDAPString,
any [1] LDAPString,
final [2] LDAPString
}
}Queries against attributes in the prescribed LDAP standard schema (see Appendix B) are accepted. Schema-approved queries only, no freelancing 📚
N.B., this is a minimal set of supported queries, to achieve the basic DAG-defined queries. An LDAP DAG-CAP MAY choose to support more complex queries than this, if it undertakes to do the translation from the DAG/IP to the LDAPv2 client in a way that responds to the semantics of those queries. Level up if you want, but you have to handle it right 🌟
TISDAG: Since LDAPv2 didn't specify any characterset but relied on X.500 to do so, in practice several different charactersets are in use in Sweden today. That the LDAPv2 CAP has no way of knowing which characterset that are in use by a connecting client is a problem that the TISDAG project can not solve. Sorry bestie, that one's outside their control 😭 Users of the DAG system will have to configure their specific client according to information on the TISDAG web page. That page provides very specific information (including port number) that can be given to LDAPv2 users. The LDAP DAG-CAP listening on the default port (389) will be the LDAPv3 one. Port 389 is LDAPv3 now, periodt 💜
5.8.2 Translation from LDAPv2 query to DAG/IP ✨
Querying the Referral Index 🔍
okay so the big trick for turning LDAP queries into DAG/IP Referral Index queries is to chop the string-y LDAP AttributeValueAssertions or SubstringFilters into tokens 🧩, then build a token-oriented DAG/IP query out of those pieces. this makes the query more general, so you'll get some false-positive referrals 🙈, but it should not miss any appropriate referrals. no ghosting allowed 💅
There are 3 particular cases to be considered:
- equalityMatch queries
- substring queries
- combination equalityMatch and substring queries
TISDAG: If the LDAP filter contains a cn-term and no objectclass specification it is unclear if the search is for a person or a role. When this happens the DAG query should cover all bases and map the query into a query for both people and roles. 🤔
EqualityMatch queries are the easy ones 🥰. you just tokenize the AttributeValueAssertions, make one DAG/IP query term per token (using the right DAGSchema attribute), and then do an exact match in the DAG/IP.
Consider this example, written in the ASCII expression of LDAP Filters as described in [13]):
(& (cn=Foo Bar)(objectclass=inetOrgPerson))This query can be represented in the DAG/IP as:
FN="Foo" and FN="Bar":search=exact<NL>N.B. The search is set up to be "case=ignore" (the DAG/IP's default) because the relevant LDAP schema attributes are all derivatives of the "name" attribute element, which is defined to have a case insensitive match. 🔤
If no objectclass were defined, the query in DAG/IP would have been:
(FN="Foo" and FN="bar") or (ROLE="Foo" and ROLE="bar"):search=exactinetOrgPerson is used as the objectclass in this and the following examples, although person or organizationalPerson could also have been used. 🧑💼
This query will yield false-positive referrals, and honestly it's giving over-generalized 😭. the original LDAP query should only match records where the "cn" attribute is exactly the phrase "Foo Bar", but the DAG/IP query will refer you to any WDSP containing records that include the two tokens "foo" and "bar" in any order.
For example, this DAG/IP query will yield referrals to WDSPs with records including:
cn: Bar Foo
cn: Le Bar Foo
cn: Foo Bar ABLDAP substring queries also have to be tokenized to build a DAG/IP query 🧠. the extra thing to keep in mind is that LDAP substring expressions are aimed at phrases, which hides where the token boundaries really are. so all the points between substring components have to be treated as potential token boundaries. main character energy for the boundaries, honestly 💜
So the LDAP query:
(& (cn=black) (o=c*t) (objectclass=inetOrgPerson))could be expressed as a DAG/IP query with 3 tokens, in a substring search:
FN=black and ORG=c and ORG=t:search=substring<NL>This query will yield false-positive results, because the tokenized query does not preserve the order the pieces showed up in the LDAP substring, and it doesn't preserve phrase boundaries either. so:
ORG=c and ORG=t:search=substringwill match:
tabaccowhich is not a match under the LDAP query semantics. not it 🙅♀️
Combined EqualityMatch and Substring queries need special attention 👀. when an LDAP query has both EqualityMatch components and substring filter components, you can build the DAG/IP query with the same tokenization steps, but the whole search becomes a substring search. that's because DAG/IP only defines search types across the entire query for Referral Index queries.
So:
(& (cn=Foo Bar) (o=c*t) (objectclass=inetOrgPerson))can be expressed as:
FN=Foo and FN=Bar and ORG=c and ORG=t:search=substring<NL>Alternatively, the LDAP DAG-CAP could run two separate queries and take the intersection (the logical "AND") of the two sets of referrals returned by the Referral Index. two queries, one intersection, very organized 🗂️✨
Note that DAG/IP can accept phrases for searches. the query:
FN=Foo\ bar<NL>(note the escaped space) is perfectly valid 💖. however, it would only match things that were tokenized in a way that keeps the space, and with the data stored here that's the empty set 😔.
Querying a DAG-SAP 📡
It is never invalid to send the same substantive query to a DAG-SAP that you used to get referral information from the Referral Index 🫶. the catch is that these queries are so over-generalized they can return a pile of results, so the LDAP DAG-CAP has to prune them to match the semantics of the original LDAP query. the LDAP DAG-CAP is the one doing this pruning, since it received the original query and is the one responsible for answering to its semantics 🧹.
Concretely, when making the DAG/IP query that gets sent to a DAG-SAP, the queries above are still valid, but there's also a finer-grained option:
FN=foo and FN=bar and ORG=c;search=lstring and ORG=t;search=tstringEspecially with the LDAPv2 DAG-CAP, though, you'll be using LDAP(v2/v3) DAG-SAPs. since those DAG-SAPs also deal in phrase-oriented data, you can pass them a less over-generalized query:
FN=Foo\ Bar:search=exact<NL>When querying a DAG-SAP (no matter which protocol that DAG-SAP speaks), the DAG/IP query must include information about the target WDSP server 🎯. this info comes from the Referral Index SERVER-TO-ASK referral information, and gets appended to the query as specified in Appendix C:
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charsetwhere the response from the Referral Index included:
"# SERVER-TO-ASK " serverhandle<NL>
" Server-info: " serverinfo<NL>
" Host-Name: " hostname<NL>
" Host-Port: " number<NL>
" Protocol: " prot<NL>
" Source-URI: " source<NL>
" Charset: " charset<NL>
"# END<NL>and the "quoted-hostname" and "quoted-serverinfo" come from "hostname" and "serverinfo" respectively, with the DAG/IP special characters quoted 🎀.
For example, this referral:
# SERVER-TO-ASK dagsystem01<NL>
Server-info: o=thinkingcat, c=se<NL>
Host-Name: thinkingcat.com<NL>
Host-Port: 2839<NL>
Protocol: ldapv2<NL>
Source-URI: http://www.thinkingcat.com <NL>
Charset: T.61<NL>
# END<NL>would yield this addition:
:host=thinkingcat\.com;port=2839;server-info=o\=thinkingcat\,\
c\=se;charset=T\.61in its query to an LDAPv2 DAG-SAP 🦋.
(N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response 📖.)
Note that it's the DAG-SAP's job to pull these terms out of the query and use them to figure out which WDSP server to contact. see the individual DAG-SAP definitions, below 👇.
5.8.3 Chaining queries in LDAPv2 DAG-CAP 🔗
The LDAPv2 DAG-CAP relies on DAG-SAPs to resolve every referral 💁♀️.
5.8.4 Expression of results in LDAPv2 🎁
As described above, results from DAG-SAPs will have to be post-processed wherever the original query was generalized for expression in DAG/IP 🧼.
Acceptable results are expressed in the LDAP search response:
SearchResponse ::=
CHOICE {
entry [APPLICATION 4] SEQUENCE {
objectName LDAPDN,
attributes SEQUENCE OF SEQUENCE
{
AttributeType,
SET OF AttributeValue
}
},
resultCode [APPLICATION 5] LDAPResult
}where:
LDAPDN = DN / "cn=" (FN/ROLE) [",o="ORG] ",dc=se"
attributes = <all attributes mapped from DAG schema, and
"objectClass = inetOrgPerson",
"objectClass = top",
"objectClass = person" or
"objectClass = organizationalRole", as
appropriate, and "labeledURI = <SOURCE-URI>"
for each result from a given referral>(Where DN, FN, ORG and ROLE are the values from the DAG schema.)
I.e., where it's available, the entry's true DN gets used 🏆. otherwise (e.g., for data coming from Whois++ servers), a reasonable facsimile is constructed 🪞.
5.8.5 Expression of Errors in LDAPv2 DAG-CAP 🚨
As appropriate, the LDAPv2 DAG-CAP will express system responses following the LDAPv2 standard 📜.
Appendix D gives the proposed mapping between DAG/IP response codes and LDAPv2 resultcodes 🗺️.
There are 4 particular error conditions of the DAG system that the DAG-CAP will handle as described below 🛠️.
When the LDAPv2 DAG-CAP gets a query it can't reply to within the (data) constraints of the DAG queries, it sends an error message and closes the connection 🔌. the error message includes the LDAPv2 resultCode:
noSuchAttribute (for incorrect schema attributes)
inappropriateMatching (when a match type other than those
supported is used, e.g. approxMatch)
unwillingToPerform (when the query is not one of the
defined types)If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (used for detecting data-mining efforts, or otherwise refusing over-general queries, such as "FN=svensson"), the LDAPv2 DAG-CAP will send an error message 🚫. the error message includes one of the following resultCodes:
sizeLimitExceeded
timeLimitExceededAn LDAPv2 DAG-CAP may redirect a connection to another LDAPv2 DAG-CAP for load-balancing reasons ⚖️. this gets expressed to the end-user client software using the "umich referral" convention, which tells the client software to go to an alternate DAG-CAP by passing the URL in an error message.
Since an LDAPv2 DAG-CAP can only send one resultcode back to a client, if it receives several different result codes from the DAG-SAPs, it has to build a result message that somehow represents the combination of them 🧃. the proposed approach is:
- All the response codes are collected
- Each response code is translated into the corresponding LDAPv2 resultcode.
- A resultcode is chosen to represent the collected response on the following grounds:
- If "success" is the only resultcode represented after these steps, return that result code.
- If apart from "success" there is one other resultcode represented, return that other resultcode.
- If apart from "success" there are two or more resultcodes represented, return the resultcode "other".
5.9 LDAPv3 DAG-CAP 💜
5.9.1 LDAPv3 DAG-CAP Input ✨
Okay bestie, so the LDAPv3 DAG-CAP takes its input from the LDAPv3 definition (currently defined in [17]) 💅 At the very least it has to handle these queries, which are adapted from the standard's ASN.1 grammar 🧚
BindRequest ::= [APPLICATION 0] SEQUENCE {
version INTEGER (1 .. 127),
name LDAPDN,
authentication AuthenticationChoice }
AuthenticationChoice ::= CHOICE {
simple [0] OCTET STRING,
-- 1 and 2 reserved
sasl [3] SaslCredentials }
SaslCredentials ::= SEQUENCE {
mechanism LDAPString,
credentials OCTET STRING OPTIONAL }
BindResponse ::= [APPLICATION 1] SEQUENCE {
COMPONENTS OF LDAPResult,
serverSaslCreds [7] OCTET STRING OPTIONAL }
SearchRequest ::= [APPLICATION 3] SEQUENCE {
baseObject c=se,
scope wholeSubtree (2) },
derefAliases ENUMERATED {
neverDerefAliases (0),
derefInSearching (1),
derefFindingBaseObj (2),
derefAlways (3) },
sizeLimit INTEGER (0 .. maxInt),
timeLimit INTEGER (0 .. maxInt),
typesOnly BOOLEAN,
filter Filter,
attributes AttributeDescriptionList }
Filter ::= CHOICE {
and [0] SET OF Filter,
or [1] SET OF Filter,
not [2] Filter,
equalityMatch [3] AttributeValueAssertion,
substrings [4] SubstringFilter }
SubstringFilter ::= SEQUENCE {
type AttributeDescription,
-- at least one must be present
substrings initial [0] LDAPString,
substrings any [1] LDAPString,
substrings final [2] LDAPString}Queries against attributes in the proscribed LDAP standard schema (see Appendix B) are accepted 🫶
N.B., this is a minimal set of queries, just enough to hit the basic DAG-defined queries 🎀 An LDAP DAG-CAP can totally go further and support fancier queries too! 🌈 But it has to take on the job of translating the DAG/IP answers back to the LDAPv3 client in a way that actually respects what those fancier queries mean 💅
5.9.2 Translation from LDAPv3 query to DAG/IP 🦋
Querying the Referral Index
Okay so the big move here is to chop the string-y LDAP AttributeValueAssertions or SubstringFilters into tokens, then build a token-based DAG/IP query using the DAGschema 🧩 This makes the query more general, so you'll get some false-positive referrals 🙈 but you won't miss any referrals you actually wanted 💖
There are 3 particular cases to be considered 👑
- equalityMatch queries
- substring queries
- combination equalityMatch and substring queries
TISDAG: If the LDAP filter has a cn-term and no objectclass specification, it's genuinely unclear whether the search is for a person or a role 😭 In that case the DAG query should cover all bases and map to a query for both people and roles 💀
EqualityMatch queries are pretty chill: you tokenize the AttributeValueAssertions, make one DAG/IP query term per token (using the right DAGSchema attribute), and do an exact match in the DAG/IP ✨
Take this example, written in the ASCII expression of LDAP Filters as described in [13]):
(& (cn=Foo Bar)(objectclass=person))This can be represented in the DAG/IP as
FN="Foo" and FN="Bar":search=exact<NL>N.B. The search is set up as case=ignore (the DAG/IP's default) 💅 because the relevant LDAP schema attributes are all derivatives of the "name" attribute element, which has a case-insensitive match. Lowkey so clean 🌸
If no objectclass were defined, the query in DAG/IP would have looked like this 🍓
(FN="Foo" and FN="bar") or ( ROLE="Foo" and ROLE="bar"):search=exactEven though person is used as the objectclass in this and the following examples, inetOrgPerson or organizationalPerson could also have been used 🦋
This query is going to yield false-positive referrals 🙃 The original LDAP query should only match records where the "cn" attribute is exactly the phrase "Foo Bar". The DAG/IP query, though, will yield referrals for any WDSP containing records with the two tokens "foo" and "bar" in any order 😭
For example, this DAG/IP query would give referrals to WDSPs with records including:
cn: Bar Foo
cn: Le Bar Foo
cn: Foo Bar ABLDAP substring queries also need tokenizing to build a DAG/IP query 🌈 The extra thing to keep in mind is that LDAP substring expressions are aimed at phrases, which hides the real token boundaries 👀 So every point between substring components has to be treated as a possible token boundary 💜
So this LDAP query:
(& (cn=black) o=c*t) (objectclass=person))should be written as a DAG/IP query with 3 tokens, in a substring search:
FN=black and ORG=c and ORG=t:search=substring<NL>This will yield false positives, because the tokenized query doesn't keep the order the pieces showed up in the LDAP substring, and it doesn't keep the phrase boundaries either 😭 That means
ORG=c and ORG=t:search=substringwill match
tabaccowhich is NOT a match under the LDAP query semantics 💀
Combined EqualityMatch and Substring queries need extra care 🎀 When an LDAP query has both EqualityMatch components and substring filter components, you can still tokenize the same way to build the DAG/IP query to the Referral Index. The catch is that the whole search becomes a substring search, because DAG/IP defines search types across the entire query 💅
So
(& (cn=Foo Bar) (o=c*t) (objectclass=person))can be expressed as
FN=Foo and FN=Bar and ORG=c and ORG=t:search=substring<NL>Alternatively, the LDAP DAG-CAP can run two separate queries and take the intersection (the logical "AND") of the two sets of referrals returned by the Referral Index 🧠
Also, DAG/IP can take phrases for searches 🌷 The query
FN=Foo\ bar<NL> (note the escaped space)is perfectly valid! 🫶 However, it would only match things that were tokenized in a way that keeps the space, which is the empty set for the data stored here 😭
Querying a DAG-SAP
It's never invalid to send the same substantive query to a DAG-SAP that you used to get referral info from the Referral Index 🥰 But because those queries are so generalized, you can get a huge pile of results, so you'll need to prune them to match the semantics of the original LDAP query 💅 The LDAP DAG-CAP is the one responsible for this pruning, since it received the original query and has to answer to its semantics 👑
Concretely, when building the DAG/IP query that goes to a DAG-SAP, the queries above are still valid, but there's also a finer-grained option, which is:
FN=foo and FN=bar and ORG=c;search=lstring and ORG=t;search=tstringWhen querying a DAG-SAP (no matter which protocol that DAG-SAP speaks), the DAG/IP query has to include info about the target WDSP server 🌸 This info comes from the Referral Index's SERVER-TO-ASK referral information, and gets appended to the query as specified in Appendix C, like this:
"host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charsetWhere the response from the Referral Index included:
# SERVER-TO-ASK " serverhandle <NL>
" Server-info: " serverinfo<NL>
" Host-Name: " hostname<NL>
" Host-Port: " number<NL>
" Protocol: " prot<NL>
" Source-URI: " source<NL>
" Charset: " charset<NL>
"# END"<NL>The "quoted-hostname" and "quoted-serverinfo" come from "hostname" and "serverinfo" respectively, by quoting the DAG/IP special characters 🎀
For example, this referral:
# SERVER-TO-ASK dagsystem01<NL>
Server-info: o=thinkingcat, c=se<NL>
Host-Name: thinkingcat.com<NL>
Host-Port: 2839<NL>
Protocol: ldapv2<NL>
Source-URI:http://www-thinkingcat.se/
Charset: T.61<NL>
# END<NL>would yield this addition:
:host=thinkingcat\.com;port=2839;server-info=o\=thinkingcat\,\
c\=se;charset=T\.61in its query to an LDAPv2 DAG-SAP 🧚
(N.B.: See Appendix C for further definitions of the terms used in the SERVER-TO-ASK response 📖)
Note that it's the DAG-SAP's job to pull these terms out of the query and use them to figure out which WDSP server to contact 💖 See the individual DAG-SAP definitions, below 🫶
5.9.3 Chaining queries in LDAPv3 DAG-CAP 🔗
The LDAPv3 DAG-CAP leans on DAG-SAPs to resolve all referrals, except the ones pointing to LDAPv3 servers (i.e., Whois++ referrals, currently) 💜
5.9.4 Expression of results in LDAPv3 🌈
As mentioned above, results from DAG-SAPs have to be post-processed whenever the original query got generalized for DAG/IP 🧹 Acceptable results come back in LDAPv3 messages containing search result entries (see the standard for more detail) ✨
SearchResultEntry ::= [APPLICATION 4] SEQUENCE {
objectName LDAPDN,
attributes PartialAttributeList }
PartialAttributeList ::= SEQUENCE OF SEQUENCE {
type AttributeDescription,
vals SET OF AttributeValue }
SearchResultReference ::= [APPLICATION 19] SEQUENCE OF LDAPURL
-- at least one LDAPURL element must be present
SearchResultDone ::= [APPLICATION 5] LDAPResultwhere
LDAPDN = DN / "cn=" (FN/ROLE) [",o=" ORG] ",dc=se"
attributes = <all attributes mapped from the DAG schema, and
"objectClass = inetOrgPerson",
"objectClass = person",
"objectClass = top" or
"objectClass = organizationalRole", as
appropriate, and "labeledURI = <SOURCE-URI>"
for each result from a given referral>
LDAPResult = success(Where DN, FN, ROLE, and ORG are the values from the DAG schema 🧠)
I.e., where it's available, the entry's true DN is used 💅 Otherwise (like for data coming from Whois++ servers), a reasonable facsimile gets built 🎀
Referral URLs get built from the DAG/IP's SERVER-TO-ASK info like this 🦋
refurl = "ldap://" HOST [":" PORT] "/" (SERVER-INFO / "dc=se")The idea is that WDSPs using LDAPv3 servers will put an appropriate LDAPDN for their server in the SERVER-INFO 🌸 Clients are then expected to repeat their query at the server that URL points to (i.e., the refURL does not include the query) 💖
5.9.5. Expression of Errors in LDAPv3 DAG-CAP ✨
okay so the LDAPv3 DAG-CAP will express operational errors the LDAPv3 standard way 💅. There are 4 particular error conditions of the DAG system that it handles specially, and I'm obsessed with how clean they are 😭
When the LDAPv3 DAG-CAP gets a query it can't reply to within the (data) constraints of the DAG queries, it sends an error message and closes the connection. The error message includes the LDAPv3 resultCode, which is one of these:
noSuchAttribute (for incorrect schema attributes chosen)
inappropriateMatching (when a match type other than those
supported is used e.g., approxMatch)
unwillingToPerform (when the query is not one of the defined
types)If the number of referrals sent by the Referral Index is greater than the pre-determined maximum (for detecting data-mining efforts, or otherwise refusing over-general queries, such as FN=svensson), the LDAPv3 DAG-CAP will send an error message. The error message includes the following resultCode:
adminLimitExceededAn LDAPv3 DAG-CAP may redirect a connection to another LDAPv3 DAG-CAP for reasons of load-balancing 🦋 In this case, the LDAPv3 DAG-CAP sends a result message including only:
SearchResultReference ::= [APPLICATION 19] AltURL
SearchResultDone ::= referral
where
AltURL = "ldap://" <althostport> ":" <altbase>Now the tricky bit 🧠 Since a LDAPv3 DAG-CAP only can send one resultcode back to a client; if it receives several different result codes from the DAG-SAPs, it has to build a result message that somehow represents the combo of all of them. It is proposed that in these cases the following actions are taken:
- All the response codes are collected
- Each response code is translated into the corresponding LDAPv3 resultcode.
- A resultcode is chosen to represent the collected response on the following grounds:
- If "success" is the only resultcode represented after these steps, return that result code.
- If apart from "success" there is one other resultcode represented, return that other resultcode.
- If apart from "success" there are two or more resultcodes represented, return the resultcode "other".
5.10. Whois++ DAG-SAP 🌸
5.10.1. Input
The Whois++ DAG-SAP expects valid DAG/IP communications. Queries must include referral information (see below) and search terms that conform to the DAG-allowed query types (e.g., not searches for organization alone, etc).
The referral information is added to the end of the DAG-SAP query, as defined in the DAG-CAP definition sections:
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charset5.10.2. Translation from DAG/IP to Whois++ query
The HOST and PORT information are used to make a TCP/IP-based connection to the remote (presumed) Whois++ server. The query expressed to the remote Whois++ server is the remainder of the DAG/IP query the Whois++ DAG-SAP received, with the following template ID translations:
template=DAGPERSON becomes template=USERand
template=DAGROLE becomes template=ORGROLEAdditional mappings for attributes are defined in Appendix B.
Heads up, this part is a little spicy 🌶️ Not all of the search types used in DAG/IP are required by the Whois++ syntax. So some Whois++ WDSPs may be running servers that don't support any searches other than exact and lstring (the search types required by the Whois++ protocol standard). The Whois++ DAG-CAP may:
- send the DAG/IP query as constructed (e.g., with
search=substring), and pass back the% 502 Search expression too complicatedfrom the WDSP's server, - translate the DAG/IP query into a construct using only these search types (which will yield incomplete results, as not all queries are expressible with those search types),
- attempt to ascertain what search types are supported by the remote server and reformulate using them (e.g., regular expressions). This would work, but would entail an excessively complicated Whois++ DAG-SAP, and might not yield any better results if the remote server doesn't support any optional search types.
5.10.3. Translation of Whois++ results to DAG/IP
Any referrals that the remote WDSP server returns are pursued, following the usual Whois++ (client) fashion, by the Whois++ DAG-SAP.
If it is not possible to establish a Whois++ session with the remote server, or if the session is interrupted before results are received, the DAG-SAP will itself return no results and an error message, including:
% 403 Information Unavailable<NL>If the remote server issues any other Whois++ error message and does not yield any results, the remote server's error message will be included in the DAG-SAP's own error message; no results will be returned.
If results are successfully received from the remote server, they will be expressed using the DAG/IP. Basically, all FULL response information received from the remote server gets passed through, mapped into the DAGSchema using the mappings defined in Appendix A. ✨
5.11. LDAPv2 DAG-SAP 💜
5.11.1. Input
The LDAPv2 DAG-SAP expects valid DAG/IP communications. Queries must include referral information (see below) and search terms that conform to the DAG-allowed query types (e.g., not searches for organization alone, etc).
The referral information is added to the end of the DAG-SAP query, as defined in the DAG-CAP definition sections (as additional terms in the DAG/IP query):
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charset5.11.2. Translation from DAG/IP to LDAPv2 query
The HOST and PORT information are used to make a TCP/IP-based connection to the remote (presumed) LDAPv2 server. The DAG-SAP will establish a connection with the remote server, following standard LDAPv2 message exchanges.
The search request itself gets built from the DAG/IP query (without the HOST, SERVER-INFO and PORT terms) like this:
SearchRequest ::=
[APPLICATION 3] SEQUENCE {
baseObject LDAPDN, -- from the DAG/IP query
scope baseObject (0) },
derefAliases ENUMERATED {
neverDerefAliases (0),
derefInSearching (1),
derefFindingBaseObj (2),
derefAlways (3)
},
sizeLimit INTEGER (0 .. maxInt),
timeLimit INTEGER (0 .. maxInt),
attrsOnly FALSE
filter Filter,
attributes SEQUENCE OF AttributeType
-- all DAGschema attributes
equivalents in the defined
standard LDAP schema
}
Filter ::=
CHOICE {
and [0] SET OF Filter,
or [1] SET OF Filter,
not [2] Filter,
substrings [4] SubstringFilter,
}
SubstringFilter
SEQUENCE {
type AttributeType,
SEQUENCE OF CHOICE {
substrings initial [0] LDAPString,
substrings any [1] LDAPString,
substrings final [2] LDAPString}
}where and, or and not filters are constructed to preserve the logic of the DAG/IP query.
For the purposes of matching token-based DAG/IP queries to reasonable LDAP queries, all searches should be passed to the LDAP WDSP as substring searches. The WDSP results must then be pruned to respect token boundaries, where necessary.
So, for example, the DAG/IP query:
FN=Foo\ Bar and ORG=Thinking\ Cat:search=substring<NL>would be sent to the designated LDAP WDSP as:
(& (fn=*Foo Bar*) (o=*Thinking Cat*) (objectclass=person))Interestingly 🤔 the query:
FN=Foo\ Bar and ORG=Thinking\ Cat:search=exact<NL>would also be sent to the designated LDAP WDSP as:
(& (fn=*Foo Bar*) (o=*Thinking Cat*) (objectclass=person))but the WDSP's returned results would have to be pruned to remove any results that had non-tokenizing characters on either side of "Foo Bar" and "Thinking Cat".
The final consideration for mapping DAG/IP queries into LDAP queries is character case. In LDAP, individual attribute syntaxes define the consideration of case. All of the attributes used here are case-insensitive in their definitions. Therefore, all LDAP WDSP queries are inherently case-insensitive; if the DAG/IP query calls for a case-sensitive match, the LDAP DAG-SAP will have to do pruning of the results from the DAG-SAP.
5.11.3. Translation of LDAPv2 results to DAG/IP
If it is not possible to establish an LDAPv2 session with the remote server, or if the session is interrupted before results are received, or if the remote server issues any kind of error message and produces no result, the DAG-SAP will itself return no results and an error message, including:
% 403 Information Unavailable<NL>If results are successfully received from the remote server, the attributes and values that are provided for each result message will be incorporated into the DAG/IP result, according to the schema mappings laid out in Appendix B.
One particular adjustment has to be made to handle the differences between LDAP and DAG/IP. The attributes that searches are keyed on ("cn", "l", and "o" in the LDAP schemas) are all defined as being case-insensitive for equality matching. So if the DAG/IP query includes the constraint case=consider, the results from the remote server must be post-processed to remove any wrong-cased ones.
TISDAG: The serverhandle and localhandle in the DAG/IP response should be constructed as follows:
serverhandle is: <hostname-without-periods><port> (because
server DN's are not enforceably unique). E.g., a
services.bunyip.com server on 7778 would
become servicesbunyipcom7778.
localhandle is: the RDN (relative distinguished name), with
spaces replaced by "_". E.g., cn=leslie_daigle5.12. LDAPv3 DAG-SAP 🎀
5.12.1. Input
The LDAPv3 DAG-SAP expects valid DAG/IP communications. Queries must include referral information (see below) and search terms that conform to the DAG-allowed query types (e.g., not searches for organization alone, etc).
The referral information is added to the end of the DAG-SAP query, as defined in the DAG-CAP definition sections:
":host=" quoted-hostname ";port=" number ";server-info="
quoted-serverinfo ";charset=" charset5.12.2 Translation from DAG/IP to LDAPv3 query
okay so the HOST and PORT info is what we use to open a TCP/IP connection to the remote (presumed) LDAPv3 server 🌐. The DAG-SAP sets up that connection itself, following the standard LDAPv3 message exchanges ✨
The search request gets built from the DAG/IP query, minus the HOST, SERVER-INFO and PORT terms, and it looks like this 🎀
SearchRequest ::=
[APPLICATION 3] SEQUENCE {
baseObject LDAPDN, -- from the DAG/IP query
scope baseObject (0) },
derefAliases ENUMERATED {
neverDerefAliases (0),
derefInSearching (1),
derefFindingBaseObj (2),
derefAlways (3)
},
sizeLimit INTEGER (0 .. maxInt),
timeLimit INTEGER (0 .. maxInt),
attrsOnly FALSE
filter Filter,
attributes SEQUENCE OF AttributeType
-- all DAGschema attributes equivalents in
the defined standard LDAP schema
}
Filter ::=
CHOICE {
and [0] SET OF Filter,
or [1] SET OF Filter,
not [2] Filter,
substrings [4] SubstringFilter,
}
SubstringFilter
SEQUENCE {
type AttributeType,
SEQUENCE OF CHOICE {
substrings initial [0] LDAPString,
substrings any [1] LDAPString,
substrings final [2] LDAPString}
}The and, or and not filters get built so the logic of the DAG/IP query stays intact 🧠
Now for matching token-based DAG/IP queries to sensible LDAP queries 🔍, every search should be passed to the LDAP WDSP as a substring search. After that, the WDSP results must be pruned so they respect token boundaries, but only where that's actually needed.
So for example, this DAG/IP query:
FN=Foo\ Bar and ORG=Thinking\ Cat:search=substring<NL>would get sent to the designated LDAP WDSP as:
(&(fn=*Foo Bar*)(o=*Thinking Cat*)(objectClass=person))Interestingly, this query:
FN=Foo\ Bar and ORG=Thinking\ Cat:search=exact<NL>would also get sent to the designated LDAP WDSP as:
(&(fn=*Foo Bar*)(o=*Thinking Cat*)(objectClass=person))but the WDSP's returned results would have to be pruned to remove any results that had non-tokenizing characters on either side of "Foo Bar" and "Thinking Cat" 🙈
The last thing to think about when mapping DAG/IP queries into LDAP queries is character case 💅 In LDAP, individual attribute syntaxes decide how case is handled. All of the attributes used here are case-insensitive in their definitions, so all LDAP WDSP queries are inherently case-insensitive. If the DAG/IP query calls for a case-sensitive match, the LDAP DAG-SAP will have to do pruning of the results from the DAG-SAP.
5.12.3 Translation of LDAPv3 results to DAG/IP
Any referrals that the remote WDSP server sends back get followed by the LDAPv3 DAG-SAP, the usual LDAPv3 (client) way 🧭
If it's not possible to set up an LDAPv3 session with the remote server, or if the session gets interrupted before results arrive, or if the remote server throws any kind of error and produces no result, then the DAG-SAP itself returns no results plus an error message, including:
% 403 Information Unavailable<NL>If results do come back successfully from the remote server, the attributes and values in each result message get folded into the DAG/IP result 🎀 That result gets expressed using the DAG/IP and schema mappings outlined in Appendix A.
There's one special adjustment needed because LDAP and the DAG/IP aren't exactly the same 🌸 The attributes that searches are keyed on ("cn", "l", and "o" in the LDAP schemas) are all defined as case-insensitive for equality matching. So if the DAG/IP query includes the constraint "case=consider", the results from the remote server must be post-processed to remove any wrong-cased ones.
TISDAG: The serverhandle and localhandle in the DAG/IP response should be constructed as follows:
- serverhandle is:
<hostname-without-periods><port>(because server DN's are not enforceably unique). E.g., aservices.bunyip.comserver on7778would becomeservicesbunyipcom7778. - localhandle is: the RDN (relative distinguished name), with spaces replaced by "_". E.g.,
cn=leslie_daigle
5.13 Example Queries
Here are some sample end-user queries that show off the trickier bits of query/schema semantics translation in the DAG system ✨
N.B.: the data in these examples is often senseless 🙃 It's only here to show matching on word-ordering, case sensitivity, and stuff like that.
5.13.1 A Whois++ Query
What the Whois++ DAG-CAP Receives
In this example, the Whois++ DAG-CAP receives this query:
name=thinking and name=cat:search=exact;case=consider<NL>The expected answer is basically: any USER templates that contain the tokens "thinking" and "cat" in a name attribute 💖
For example, different records:
name: the thinking cat
name: sublime cat thinkingor a single record with 2 or more name attributes:
name: thinking felines
name: erudite catbut not:
name: Thinking Cat EnterprisesThis last one wouldn't match, because the query called for case sensitivity and the case of the name attribute's value doesn't match 😭
What the Whois++ DAG-CAP Sends to the Referral Index
After schema translation, this is what goes to the Referral Index:
fn=thinking and fn=cat:search=exact<NL>What the Whois++ DAG-CAP Sends to an LDAP DAG-SAP
Note that the Whois++ DAG-CAP never talks to a Whois++ DAG-SAP directly, since the Whois++ referrals from the Referral Index get passed straight back to the Whois++ client 🧚
The Whois++ DAG-CAP should send the same substantive query to the DAG-SAP that it sent to the Referral Index, except it can include the case sensitivity constraint:
fn=thinking and fn=cat:search=exact;case=consider<NL>The DAG-SAP translates that into an LDAP query of this form:
(&(cn=*thinking*)(cn=*cat*)(objectclass=inetOrgPerson))which will match a record with:
cn: Thinking
cn: Cat(i.e., 2 different cn attributes, with the 2 values; LDAP defines case sensitivity matching by the schema attribute definition 🤓)
or a record with:
cn: I wish I had a thinking dog and a singing catThe first record should be pruned by the LDAP DAG-SAP, so it respects the semantics of the DAG/IP query 💅
5.13.2 An LDAP Query
What the LDAP DAG-CAP Receives
In this example, the LDAP DAG-CAP receives this query (using RFC1960 notation):
(& (cn=th*c*t) (o=green groceries) (objectClass=person))What the LDAP user wants here is every record within the "green groceries" organization that has a cn attribute starting with "th", ending with "t", and having a "c" somewhere in the middle 👀
cn values that would match include:
cn: thinkingcat
cn: Thinking Cat
cn: The Black Cat
cn: Thick Mat5.13.3 What the LDAP DAG-CAP sends to the Referral Index
The LDAP DAG-CAP must formulate a token-based query for the Referral Index that won't accidentally exclude records that would match 🫶 The tricky part is that the "*" characters in an LDAP string-based query can cover token boundaries.
A suitable query to the Referral Index would be:
FN=th AND FN=C AND FN=T AND ORG=green AND
ORG=groceries:search=substring<NL>This will generate some false positive referrals, sending the query to WDSPs with records like these (the match letters are in capitals so you can spot them):
cn: wiTH three blaCk poTs
o: peaGREEN and cyan GROCERIES
o: GROCERIES are GREENer than electronicsOther approaches were considered too: breaking the original query into several queries to the Referral Index, so the DAG-CAP only uses the referrals that show up in every response. But honestly that's:
- overkill -- the purpose of the Referral Index is to give direction on where there may be more information
- difficult to code into the DAG-CAP in a general way -- it has to identify, by LDAP query type, when and how to do so
- likely to generate Referral Index queries that are complex and time-consuming to process
What the LDAP DAG-CAP Sends to a Whois++ DAG-SAP
The LDAP DAG-CAP may send the same query to a Whois++ DAG-SAP that it sent to the Referral Index. False positives here are results that the LDAP client wouldn't expect as a match. The LDAP DAG-CAP should prune these results from what the Whois++ DAG-SAP returns 🧹
Or it might rewrite the query into:
FN=th;search=lstring AND FN=C;search=substring AND
FN=T;search=tstring AND ORG=green AND ORG=groceries:case=ignore<NL>What the LDAP DAG-CAP Sends to an LDAP DAG-SAP
As an architectural principle, it is never wrong to send the same query to a DAG-SAP that was formulated for the Referral Index 🌈 It's also worth remembering that all DAG-SAPs are handled equally by all DAG-CAPs. So an LDAP DAG-CAP won't need to send a different query to an LDAP DAG-SAP than it would to any other DAG-SAP.
So in this case, the LDAP DAG-CAP could either send the same query to the LDAP DAG-SAP that it sent to the Referral Index, or it could send the augmented version that's allowed to be used with DAG-SAPs, namely:
FN=th;search=lstring AND FN=C;search=substring AND
FN=T;search=tstring AND ORG=green\ groceries:case=ignore<NL>The LDAP DAG-SAP translates this into a query of the form:
(&(cn=*th*)(cn=*c*)(cn=*t*)(o=*green groceries*)
(objectClass=person))which is still more general than the original query 🦋
Note the translation from "FN=th;search=lstring" into "cn=th". That's needed because the DAG/IP lstring constraint is based on tokens, whereas "cn=th*" refers to the beginning of the attribute's value (phrase, not token). So the DAG-SAP should prune out anything like "oTHer plaCes for visiTors" to match the semantics of the DAG/IP query it received. Then the DAG-CAP should prune those results to match the semantics of the original LDAP query ✨
6.0 Service Specifications
6.1 Overview
To meet the requirements laid out for the TISDAG project, the software built for the DAG system must be able to meet the following service specifications 👑
- primary designated DAG-CAPs of all types (but not necessarily secondary ones set up for load-balancing) must be available to provide service or redirect queries on a 7x24 basis.
- in general, responses to queries should be available in under 10 seconds; very generalized queries (i.e., when the user truly cannot specify enough information to focus the search) can be deferred to take much longer (having results is more important than having a quick answer) ⏳
- the data provided from each WDSP should be updated in the DAG at least once every 7 days 🍓
6.2 WDSP Participation 🎀
okay so WDSPs who wanna join the DAG party do it by giving DAG-compatible access to their service, and DAG-compatible means:
- access in (exactly) one of LDAPv2, LDAPv3, or Whois++
- 7x24 service for responding to referrals generated in the DAG core
- (minimally) weekly updates of the index object describing the information their service indexes
- use of USER and ROLE templates for Whois++ servers
- use of inetorgperson and organizationalrole objectclasses for LDAP servers
to join, WDSPs must register each DAG-compliant server with the DAG system and give details for each data set it covers: ✨
- the host, port and protocol of the server
- an identifier for the dataset
- a URL for the service of preference for accessing the data (preferred source)
- protocol-specific information
- administrative contact information
- CIP object exchange information
ngl this part is so good 🫶 if a WDSP wants their data in the DAG system but can't meet these requirements, they can still provide the info through an agreement with a third-party that does meet them. that's how data gets replicated between cooperating WDSPs 🌈 the DAG referral index does not claim ownership of personal information. it just points queries at the services that do, under whatever agreements with whichever relevant parties. and in that case, the SOURCE-URI may direct end-users to the WDSP's existing services, not the third party's service 💅
6.3 Load Distribution 💪
it's giving "this is gonna be SO popular" 🔥 so measures must be available to distribute the load of answering queries.
the DAG system is presented as one conceptual whole made up of several component parts: DAG-CAPs, DAG-SAPs and the Referral Index. each of these parts must be replicable, and service must be shared between replicas.
it might be interesting to let large-scale service providers (large companies, ISPs) mirror the Referral Index or set up alternate DAG-CAPs/DAG-SAPs for their personnel/customers. policies and possibilities for that are beyond the scope of this report, but the software architecture has been designed to support such activity 🦋
Figure 6.1 shows that individual components of the DAG system may each run on non-co-located server hardware, connected by TCP/IP networks. these components can be replicated as needed 🧚
+====+
| | DAG-CAP (Client Access Point)
| |
+====+
+----+
| | DAG-SAP (Service Access Point)
| |
+----+
+====+
HTTP <-->| |
| | +----+
+====+ | |<--> Whois++
| |
+====+ +----+
SMTP <-->| |
| | +----+
+====+ | |<--> LDAPv2
| |
+====+ +----+
Whois++<-->| |
| |
+====+ +----+
| |<--> LDAPv3
| |
+----+
| |<--> LDAPv3
| |
+----+
| |<--> LDAPv3
| |
+====+ +----+
LDAPv2 <-->| |
| |
+====+
+====+
LDAPv3 <-->| |
| |
+====+
+------------------------+
| Referral Index |<--> Common Indexing Protocol
| | (CIP)
+------------------------+
+------------------------+
| Referral Index |
| |
+------------------------+Figure 6.1 Distributable nature of DAG components
thus, the software built to this specification must be configurable to permit the following actions: 🎯
- DAG-CAP software must be able to handle or redistribute the primary load. depending on the DAG-CAP software, this may be handled by having multiple processes attending to incoming queries, or the DAG-CAP at the primary address for the protocol may be nothing more than a reflector that redirects incoming queries to the address of the least-loaded server at the moment.
- this is particularly necessary in synchronous connection protocols, such as Whois++ and LDAP, where the goal is to minimize the amount of time a requesting client is connected to the well-advertised address port.
- DAG-CAP software must be able to direct referrals to different DAG-SAPs of the same protocol type.
- DAG-CAP software must be able to detect overly general queries (i.e., have some metric to decide that the number of referrals generated by the Referral Index is too great).
- DAG-SAPs must be able to redirect DAG-CAP queries at their discretion, or just refuse service because of loading (therefore DAG-CAPs must also be able to find other DAG-SAPs)
6.4 Extensibility 🧩
the DAG system has been designed to allow for extensibility in certain key areas:
it's possible to add new DAG-CAPs and DAG-SAPs transparently 🌸 beyond replicating the software of existing DAG-CAPs, you can add new implementations for particular protocols (e.g., building a more elaborate mail-based query system), or implementations for altogether different protocols (e.g., PH), as long as you stick to the basic principles of DAG-CAPs and DAG-SAPs defined in the software specification. the new DAG-CAP is responsible for translating queries into DAG/IP (post-processing results, if necessary) and returning results in the new protocol. no other part of the DAG system is affected. main character energy 👑
more functionality may be added to the DAG system service (e.g., adding security certificate references to the schema of returned information) by updating the DAG schema ✨
depending on how the load on the service goes, it may be interesting to consider reducing the number of queries that are chained for protocols that inherently can handle the concept of pursuing referrals. specifically, LDAPv3 and Whois++ both handle referrals, but the current system calls for chaining LDAPv3 (and LDAPv2) referrals for the Whois++ DAG-CAP, and vice versa. alternatively, "virtual" DAG-CAPs could be established for each participating WDSP for each protocol the WDSP doesn't support, and referrals to those DAG-CAPs could be given to the calling client. for example, a Whois++ client would be given a Whois++ referral to the virtual Whois++ DAG-CAP for a WDSP that supports only LDAP. the importance of having one virtual DAG-CAP per WDSP is that the point of connection is the only way to distinguish which WDSP the Whois++ client thought it was connecting to 🦋
7.0 Security 🔐
7.1 Information credibility 🫧
okay this one's serious bestie 😭 security, in the context of "read-only" directory services, is primarily concerned with maintaining data integrity as it passes from an originating server to the end-user making an inquiry. some server(s) hold correct user information, and a client accessing a directory service should be certain that, whichever servers the information has to pass through before reaching the client, it receives a true representation of the original information.
the DAG system as such MUST be completely invisible as the mediator of the information from the WDSPs to the querying directory access client. the only possible modifications that can appear are translations from one characterset into another. hopefully, this does not alter the meaning of the information 💀
7.2 Unauthorized access 🚪
in keeping with the public nature of the proposed TISDAG service, the DAG system does not provide any access control system beyond components' configuration to accept connections from recognized other components. for more detailed access control, it is up to the connected WDSPs to apply the access control 🙈
since the DAG system only supports searching and retrieving information, no updates can occur through the DAG client access points 🌈
security in updates (CIP index objects) is provided by encryption and signature of objects from registered WDSPs 🔥
8.0 Acknowledgments 🫶
this work came from ideas originally put forward by Patrik Faltstrom 🍓 the TISDAG project was supported by the Swedish KK Foundation.
thanks especially to Jens Lundstrom, Thommy Eklof, Bjorn Larsson and Sandro Mazzucato for their comments on draft versions of this document 💜
Appendix A - DAG Schema Definitions 📚
the DAG makes use of 2 information schemas -- the DAGPERSON schema for information about specific people, and the DAGORGROLE schema for organizational roles that may or may not be job positions occupied by people at any given time (e.g., an organization's president, customer service desk, etc).
this appendix defines the schemas in terms of the attributes used within the DAG/IP. mappings to the standard LDAP and Whois++ object classes and templates (respectively) are described in Appendix B.
because the role of the DAG schemas is to act as an intermediary between information provided in different access protocols, with different underlying schema paradigms, the attributes in the schema are identified as being required or optional. the required attributes are so designated because they are involved in the DAG search types and/or the minimal returned response. they have defined mappings in the selected access protocols. the optional attributes have proposed mappings in those protocols.
it is important to note that the DAG/IP is constructed to carry any alternative attribute information that may be provided by a given WDSP. individual DAG-SAPs and DAG-CAPs may choose to pass along, interpret, or ignore any attributes not defined in this appendix 🧠
additionally, note that the order of attributes in the DAG/IP is significant, which means it is possible to use one attribute to carry the information describing the type of subsequent ones (e.g., see the "ADR-TYPE" attribute below) ✨
finally, attributes may be repeated. for example, this schema structure can carry multiple phone numbers of different types for one person 📞🎀
A.1 DAG Personal Information Schema (DAGPERSON Schema) 💖
okay so this is the DAGPERSON schema, the person-shaped entry that DAG-SAPs send back to DAG-CAPs 🎀 Some fields are required and some are optional, and the table below is the full list.
Attribute Designation Specific Description
--------- ----------- -------------------------------------
FN Required Free-text representation of full name
EMAIL Required Internet e-mail address
LOC Required Locality -- geographic region
ORG Required Person's organization
ADR-TYPE Optional Type of address that follows
("org", "home", "org-postal",
"home-postal", "unqualified")
ADR Optional Full address
ADR-STREET Optional Street address component
ADR-ROOM Optional Suite or room number component
ADR-CITY Optional City name
ADR-STATE Optional Region of address
ADR-COUNTRY Optional Country
ADR-CODE Optional Postal code component
TEL-TYPE Optional Type of telephone number (
"work", "home", "mobile",
"fax" ,"pager", "unqualified")
in the following attribute
TEL Optional A phone number for the person
SOURCE Optional The WDSP's preferred access to
their service -- a URL
DN Optional Entry's "distinguished name"
(for LDAP)
Table A.1 DAGPERSON schema attributesA.2 DAG Organizational Role Information Schema (DAGORGROLE Schema) 🧚
this one is the role version, so it's for things like a shared mailbox or a department rather than a specific human 💅 same vibe, different fields.
Attribute Designation Specific Description
--------- ----------- ---------------------
ROLE Required Name of organizational role
EMAIL Required E-mail address associated with role
ORG Required Name of organization
LOC Required Locality -- geographic region
TEL-TYPE Optional Type of telephone number
in the TEL attribute immediately
following("org" or "fax")
TEL Optional Phone number
FN Optional Full name of current role occupant
SOURCE Optional The WDSP's preferred access to their
service -- a URL
DN Optional Entry's "distinguished name" (for LDAP)
Table A.2 DAGORGROLE schema attributesAppendix B - Schema Mappings for Whois++ and LDAP 🌸
okay this is where it gets spicy 🌶️ the DAG/IP uses those two schemas, but the access protocols have their own schemas and real user queries and chained requests need to be translated between them 🦋 the standard schemas that get recognized are listed here:
- the USER template for Whois++ ([8])
- the ORGROLE template for Whois++ ([8])
- the inetOrgperson objectclass for LDAP ([16])
- the organizationalrole objectclass for LDAP ([18])
so the DAGPERSON and DAGORGROLE schemas were built from two things: what the TISDAG project wanted to return in results, and what the basic WDSP access protocols (LDAPv2/v3 and Whois++) already use 🧠 the catch is that address info in those standard schemas can be represented in a bunch of different ways. so different WDSPs might only use some sub-parts, or add their own local customizations 💀
that's why Appendix A exists: it's a very basic schema that can carry all the info that's needed. the basic DAG-CAPs and DAG-SAPs are designed to work with that structure. this appendix describes the expected behaviour for DAG-SAPs mapping into the DAG/IP schema, and for DAG-CAPs pulling information back out to hand to client software after a chained operation returns results ✨
B.1 LDAP and the DAG Schemas 🫶
the DAG schemas only carry information when a DAG-SAP is sending back data it got from WDSP servers to a DAG-CAP over the DAG/IP. the "canonical" mappings between the standard LDAP object classes (inetOrgPerson from [16] and organizationalRole from [18]) and the DAGPERSON and DAGORGROLE schemas are set up so that anything passed from an LDAP DAG-SAP to an LDAP DAG-CAP comes out in the same attributes it went in with. e.g. an LDAPv3 DAG-SAP returning info for an LDAPv2 DAG-CAP 🙌
but address stuff is wildly different across protocols. the "reasonable approximation" mappings are there to give DAG-CAPs a basic way to pass along info from non-LDAP DAG-SAP sources. they might not be perfect, but the info will still reach the end user in something LDAP can understand, and that's the whole goal of this project 🎯
the canonical mappings for the LDAP inetOrgPerson object class and the DAGPERSON schema are in Table B.1. a few reasonable approximation mappings come after that, in Table B.2. past that, DAG-SAPs can send along any extra attributes over the DAG/IP, and DAG-CAPs can choose to forward or interpret whichever ones they recognize. like, the sn ("surname") attribute isn't in these tables, but a DAG-SAP might still return it over the DAG/IP, and a DAG-CAP that recognizes the string could decide to include it in its LDAP response to the client 🌈
DAGPERSON Attribute LDAP inetorgPerson attribute
------------------- ----------------------------
FN cn
EMAIL mail
LOC l
ORG o
ADR-TYPE=org
ADR-STREET street
ADR-ROOM roomNumber
ADR-STATE st
ADR-COUNTRY c
ADR-TYPE=org-postal
ADR postalAddress
ADR-ROOM postOfficeBox
ADR-CODE postalCode
ADR-TYPE=home-postal
ADR homePostalAddress
TEL-TYPE=work
TEL telephoneNumber
TEL-TYPE=home
TEL homePhone
TEL-TYPE=fax
TEL facsimileTelephoneNumber
TEL-TYPE=mobile
TEL mobile
TEL-TYPE=pager
TEL pager
DN dn
SOURCE labeledURI
Table B.1 Canonical DAGPERSON schema & LDAP inetorgPerson attributes DAGROLE Attribute LDAP organizationalRole attribute
----------------------- ---------------------------------
ADR-TYPE=unqualified
ADR street
ADR-STREET street
ADR-ROOM room
ADR-STATE st
ADR-COUNTRY c
TEL-TYPE=unqualified
TEL telephoneNumber
Table B.2 Reasonable Approximations for LDAP organizationalRole
attributeshere's an example LDAP record in LDIF [11] format, so you can see how it gets carried over into the DAGPERSON schema 👀
dn: cn=Barbara Jensen, ou=Product Development, o=Ace Industry,
c=US
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetorgperson
cn: Barbara Jensen
cn: Barbara J Jensen
cn: Babs Jensen
sn: Jensen
uid: bjensen
telephonenumber: +1 408 5551212
description: A big sailing fanand this is how that would validly be carried in the DAGPERSON schema:
DN: cn=Barbara Jensen, ou=Product Development, o=Ace Industry,
c=US
FN: Barbara Jensen
FN: Barbara J Jensen
FN: Babs Jensen
SN: Jensen
TEL-TYPE: work
TEL: +1 408 5551212the canonical mappings for the LDAP organizationalRole object class and the DAGORGROLE schema are in Table B.3. after that, DAG-SAPs can choose to send along any attributes, and DAG-CAPs can interpret any they recognize. heads up though 🚨 the organizationalRole class doesn't have a spot for an e-mail address. this mapping just kinda assumes the mail attribute from inetOrgPerson is available, which is a bit of a leap 🙈
DAGORGROLE Attribute LDAP organizationalRole attribute
-------------------- ---------------------------------
ROLE cn
EMAIL mail
ORG o
LOC l
TEL-TYPE=org
TEL telephoneNumber
TEL-TYPE=fax
TEL facsimileNumber
FN roleOccupant
DN dn
SOURCE labeledURI
Table B.3 Canonical mappings for LDAP organizationalRole attributesB.2 Whois++ and the DAG Schemas 🍓
the "canonical" mappings between the standard Whois++ templates from [8] and the DAGPERSON and DAGORGROLE schemas are in Tables B.4 and B.5 ✨ same deal as before: DAG-SAPs can pass along any extra attributes over the DAG/IP, and DAG-CAPs can forward or interpret whatever they recognize.
DAGPERSON Attribute Whois++ USER template attribute
------------------- -------------------------------
FN name
EMAIL email
LOC address-locality
ORG organization-name
ADR-TYPE=unqualified
ADR address
ADR-TYPE=org
ADR organization-address
ADR-STREET organization-address-street
ADR-ROOM organization-address-room
ADR-CITY organization-address-city
ADR-STATE organization-address-state
ADR-COUNTRY organization-address-country
ADR-CODE organization-address-zip-code
ADR-TYPE=home address-type=home
ADR address
ADR-STREET address-street
ADR-ROOM address-room
ADR-CITY address-city
ADR-STATE address-state
ADR-COUNTRY address-country
ADR-CODE address-zip-code
TEL-TYPE=work phone-type=work
TEL phone
TEL-TYPE=home phone-type=home
TEL phone
TEL-TYPE=fax
TEL fax
TEL-TYPE=mobile
TEL cellular
TEL-TYPE=pager
TEL pager
Table B.4 Canonical DAGPERSON schema & Whois++ USER attributes DAGORGROLE Attribute Whois++ ORGROLE attribute
-------------------- -------------------------
ROLE org-role
EMAIL email
ORG organization-name
LOC organization-address-locality
FN name
TEL-TYPE=org
TEL phone
TEL-TYPE=fax
TEL fax
Table B.5 Canonical mappings for Whois++ ORGROLE attributesAppendix C - DAG-Internal Protocol (DAG/IP) 🦋
okay last bit of this part 💖 the DAG-Internal Protocol (DAG/IP) is currently defined as a derivative of the query-interaction protocol of Whois++, as laid out in RFC1835 ([6]) 🎀
C.1 A word on the choice of DAG/IP ✨
Okay so, the DAG/IP is strictly internal to the DAG system 🫶. That means it's totally possible to use any query language, or even define a brand-new one 💅.
The Whois++ protocol was picked as the basis of DAG/IP for a few reasons, bestie 🎀:
- it has the power and flexibility to convey all necessary queries
- it is a simple, text-based protocol; clients need not implement the full functionality of the protocol in order to carry out minimal queries
- the power of the full-fledge directory service query protocol will give DAG-CAP writers the ability to express more sophisticated queries if desired (e.g., to produce more intricate "intelligent" matching of spellings, common character substitutions, etc). 🧠
- the text-based, delimited attribute results expression facilitates optional inclusion of extra data supplied by WDSPs -- DAG-CAPs can easily ignore any unknown information and continue to interpret the rest of the result information. 🦋
Also, using an existing protocol means you get to borrow the experience and time of the people who created it 😭. They already hammered out the elusive-but-necessary stuff, like handling line-endings and quoting special characters 🙌.
There's also a freely-available test suite of tools for testing servers' Whois++ protocol conformance (for the Referral Index, and for DAG-SAPs). Send mail to digger-info@bunyip.com for further information 💌.
C.2 DAG/IP Input and Output -- Overview 🌸
Input interactions in DAG/IP are as defined in RFC1835, "Architecture of the WHOIS++ service" ([6]), sections 2.2 and 2.3. Section C.3 of this document adapts the grammar used in more recent descriptions of the Whois++ protocol to illustrate the syntax of the DAG/IP.
DAG/IP output will be a subset of what is defined in RFC1835, section 2.4, except that referral responses ("SERVER-TO-ASK") contain more information 🔥.
C.3 BNF for DAG/IP input and output 🧚
The following sections are adapted from the Whois++ grammar. For discussion of the semantic intent of the query protocol, and other matters, see Whois++ RFC 1835 [6].
C.3.1 The DAG/IP Input Grammar 💜
The following grammar, which uses the Augmented BNF (ABNF) notation as defined in [5], defines the set of acceptable DAG/IP input.
N.B.: As outlined in the ABNF definition, rule names and string literals are in the US-ASCII character set, and are case-insensitive. Also, when a character is written explicitly in the grammar, as for example ";", it represents the byte value of that character in all of the allowed character sets in their encodings used in this protocol. Specifically in UNICODE, ";" means the character U+003B, which when encoding the character in UTF-8 will generate the byte value 0x3B which is then used in the DAG/IP protocol. 🤓
dagip-command = ( system-command [":" "hold"]
/ ri-query
/ sap-query ) nl
ri-query = ri-terms [":" globalcnstrnts]
sap-query = sap-terms [":" [sapcnstrnts][ ":" wdspinfo]]
system-command = "constraints"
/ "describe"
/ "commands"
/ "polled-by"
/ "polled-for"
/ "version"
/ "list"
/ "show" [1*sp datastring]
/ "help" [1*sp datastring]
/ "<NL>" [string]
ri-terms = ri-and-expr *(1*sp "or" 1*sp ri-and-expr)
ri-and-expr = ri-basic-expr *(1*sp "and" 1*sp ri-basic-
expr)
ri-basic-expr = ["not" 1*sp] ri-term / ( "(" ri-terms ")" )
ri-term = generalterm / specificterm / combinedterm
sap-terms = sap-and-expr *(1*sp "or" 1*sp sap-and-expr)
sap-and-expr = sap-basic-expr *(1*sp "and" 1*sp
sap-basic-expr)
sap-basic-expr = ["not" 1*sp] sap-term / ( "(" sap-terms ")" )
sap-term = ( generalterm / specificterm / combinedterm)
localcnstrnts
generalterm = datastringTISDAG: Since the DAG system only supports certain attribute combinations in its queries, (Table 3.1). The use of generalterm may lead to unexpected behaviour and is therefore deprecated. CAPs should therefore not use it even if it is in the protocol.
specificterm = specificname "=" datastring
specificname = "handle" / "value"
combinedterm = attributename "=" datastring
sapcnstrnts = sapcnstrnt *(";" sapcnstrnt)
sapcnstrnt = localcnstrnt / globalcnstrnt
localcnstrnts = [";search=" sap-searchvalue] [";case="
sap-casevalue]
localcnstrnt = "search=" sap-searchvalue / "case="
sap-casevalue
;N.B.: in the case where local and global constraints
; conflict, local constraints take precedence
; and overrides the global constraint
sap-searchvalue = "tstring" / searchvalue
sap-casevalue = "consider" / "ignore"
globalcnstrnts = globalcnstrnt *(";" globalcnstrnt)
globalcnstrnt = "search" "=" searchvalue
/ opt-globalcnst
opt-globalcnst = "hold"
/ "case" "=" casevalue
/ "maxfull" "=" 1*digit
/ "maxhits" "=" 1*digit
/ "language" "=" language
/ "incharset" "=" characterset
/ "ignore" "=" attributename
/ "include" "=" attributename
; N.B.: If an attribute is named both with the "include" and "ignore"
; constraints, the attribute is to be included in the result, but the
; system message must be "% 112 Requested constraint not fulfilled".
language = <The language code defined in RFC1766>
characterset = "UNICODE-2-0-UTF-8"
searchvalue = "exact" / "substring" / "lstring"
casevalue = "ignore" / "consider"
wdspinfo = attrValAss *( ";" attrValAss )
attrValAss = attributename "=" datastringTISDAG: Within the boundaries of the TISDAG project it has been decided that the only permitted attributes for wdspinfo are "host","port","server-info" and "charset". Regarding "charset" the values for this attribute are defined to be one of "UTF-8", "ISO8859-1","T.61" or "US-ASCII".
datastring = 1*data-elt
attributename = 1*(<%d32-126 except specialbyte>)
; omit 127, which is DEL
data-elt = "\" specialbyte / normalbyte
normalbyte = <%d32-255, except specialbyte>
specialbyte = " " / tab / "=" / "," / ":" / ";" / "\" /
"*" / "." / "(" / ")" / "[" / "]" / "^" /
"$" / "!" / "<NL>"
number = 1*digit
digit = "0" / "1" / "2" / "3" / "4" /
"5" / "6" / "7" / "8" / "9"
tab = %d09
sp = %d32 ; space
nl = %d13 %d10 ; CR LFNOTE: Spaces (sp) that are significant to a query must be escaped. The following characters, when significant to the query, may be preceded and/or followed by a single space:
: ; , ( ) = !C.3.2 The DAG/IP Response Grammar 🎀
The following grammar, which uses the Augmented BNF (ABNF) notation as defined in RFC2234 (see [5]),
N.B.: As outlined in the ABNF definition, rule names and string literals are in the US-ASCII character set, and are case-insensitive. Also, when a character is written explicitely in the grammar, as for example ";", it represents the byte value of that character in all of the allowed character sets in their encodings used in this protocol. Specifically in UNICODE, ";" means the character U+003B which when encoding the character in UTF-8 will generate the byte value 0x3B which is then used in the DAG/IP protocol. 💅
server-resp = goodmessage mnl output mnl endmessage
/ badmessage nl endmessageclose
output = 0*(full-record / server-to-ask)
full-record = "# FULL " template " " serverhandle " "
localhandle system-nl
1*fulldata
"# END" system-nlTISDAG: serverhandle is: - Whois++, whatever the server-handle on the record returned by the WDSP. - LDAP, <hostname-without-periods><port> (because server DN's are not enforceably unique). E.g., a services.bunyip.com server on 7778 would become servicesbunyipcom7778. localhandle is: - Whois++: the localhandle on the record returned by the WDSP - LDAP, it is the RDN (relative distinguished name), with spaces replaced by "_". E.g., cn=leslie_daigle
server-to-ask = "# SERVER-TO-ASK " serverhandle system-nl
server-to-askdata
"# END" system-nl
fulldata = " " attributename ": " attributevalue
system-nl
server-to-ask-data = " Server-Info: " serverinfo system-nl
" Host-Name: " hostname system-nl
" Host-Port: " number system-nl
" Protocol: " prot system-nl
" Source-URI: " source system-nl
" Charset: " characterset system-nl
attributename = r-string
attributevalue = longstring
template = <%d32-%d255 except specialbyte>
serverhandle = <%d32-%d255 except specialbyte>
localhandle = <%d32-%d255 except specialbyte>
serverinfo = string
hostname = string
prot = string ; currently one of "ldapv2"
; "ldapv3" "whois++"
characterset = "UTF-8" / "T.61" / "ISO8859-1" / "US-ASCII"
source = string
longstring = string 0*( nl ( "+" / "-" ) string )
string = 0*(%d32-255)
r-string = 0*(<%d32-126 except specialbyte>)
; omit 127 which is DEL
specialbyte = ":" / " "
mnl = 1*system-nl
system-nl = nl [ 1*(message nl) ]
nl = %d13 %d10 ; CR and LF
message = [1*( messagestart "-" string nl)]
messagestart " " string nl
messagestart = "% " digit digit digit
goodmessage = [1*( goodmessagestart "-" string nl)]
goodmessagestart " " string nl
goodmessagestart= "% 200"
badmessage = [1*( badmessagestart "-" string nl)]
badmessagestart " " string nl
badmessagestart = "% 5" digit digit
endmessage = endmessageclose / endmessagecont
endmessageclose = [endmessagestart " " string nl]
byemessage
endmessagecont = endmessagestart " " string nl
endmessagestart = "% 226"
byemessage = byemessagestart " " string nl
byemessagestart = "% 203"
number = 1*( digit )
digit = "0" / "1" / "2" / "3" / "4" / "5" / "6" /
"7" / "8" / "9"C.4 DAG/IP Response Messages ✨
okay so this part is all about the response codes, and bestie, they're derived from the Whois++ protocol definition in RFC1835 ([6]) 🫶 so you already know the vibe if you've seen that one 💅
every system message starts with a %, then a space, a three digit number, another space, and then an optional text message 🎀 the whole line must be no more than 81 bytes long, counting the terminating CR LF pair, and there's no limit on how many system messages can be generated 🦋
multiline system messages are a little different ✨ they put a hyphen instead of a space in column 6, right after the numeric code, on every line except the last one. the last line uses the space instead 🌸
Example 1
% 200 Command okayExample 2
% 220-Welcome to
% 220-the Whois++ server
% 220 at ACME inc.the client isn't expected to parse the text part of the response 💖 the one exception is replies 600 and 601. for 600, the text is the name of a character set the server will use for the rest of the response. for 601, it tells you what language the attribute value is in 🌈 the valid values for character sets are listed in the "characterset" list in the BNF listing in Appendix C 🧚
the theory behind reply codes is described in Appendix E in STD 10, RFC821 ([15]) 📚 (that's the reference, not a new rule, just the homework link 😭)
Here's the big list of system response codes 👑
System response code Description
---------------------------- ------------------------------
110 Too many hits The number of matches exceeded
the value specified by the
maxhits constraint. Server
will still reply with as many
records as "maxhits" allows.
111 Requested constraint not One or more constraints in query
supported is not implemented, but the
search is still done.
112 Requested constraint not One or more constraints in query
fulfilled has unacceptable value and was
therefore not used, but the
search is still done.
200 Command Ok Command accepted and executed.
The client must wait for a
transaction end system message.
201 Command Completed Command accepted and executed.
successfully
203 Bye Server is closing connection
204 Overgeneralized The server could not exactly
match the DAG query into its
native access protocol. The
resulting native query was
"looser".
220 Service Ready Greeting message. Server is
accepting commands.
226 Transaction complete End of data. All responses to
query are sent.
401 Service not available
402 Search expression
too complicated
403 Information Unavailable When a remote service is not
(currently) available.
404 Time out
500 Syntax error
502 Search expression too This message is sent when the
complicated server is not able to resolve a
query (i.e. when a client sent a
regular expression that is too
deeply nested).
503 Query to general This is like the "too many hits"
situation, but the server does
not send along any results. This
message is used to deflect data
mining.
505 Operations error Permanent operations error
600 <token> Subsequent attribute values are
encoded in the character set
specified by <token>.
601 <token> Subsequent attribute values are
in the language specified by
<token>.
601 DEF Subsequent attribute values are
default values, i.e. they should
be used for all languages not
specified by "601 <token>" since
last "601 ANY" message.
601 ANY Subsequent attribute values are
for all languages.Table C.1 List of system response codes
Appendix D - DAG/IP Response Messages Mapping ✨
this appendix is the translation cheat sheet 🍓 it maps LDAPv2/v3 result codes onto DAG/IP response codes, then goes the other way, then lines DAG/IP up against Whois++ 💜
LDAPv2/v3 DAG/IP
--------------------------------------- ---------------------
success (0) v2&v3 200 Command Ok
operationsError (1) v2&v3 505 Operations error
protocolError (2) v2&v3 505 Operations error
timeLimitExceeded (3) v2&v3 404 Timeout
sizeLimitExceeded (4) v2&v3 110 To many hits
compareFalse (5) v2&v3 200 OK
compareTrue (6) v2&v3 200 OK
authMethodNotSupported (7) v2&v3 505 Operations error
strongAuthRequired (8) v2&v3 505 Operations error
referral (10) v3 200 OK
adminLimitExceeded (11) v3 110 Too many hits
unavailableCriticalExtension (12) v3 505 Operations error
confidentialityRequired (13) v3 505 Operations error
saslBindInProgress (14) v3 N.A.
noSuchAttribute (16) v2&v3 200 OK
undefinedAttributeType (17) v2&v3 500 Syntax error
inappropriateMatching (18) v2&v3 500 Syntax error
constraintViolation (19) v2&v3 111 Requested constraint
not supported
attributeOrValueExists (20) v2&v3 200 OK
invalidAttributeSyntax (21) v2&v3 500 Syntax error
noSuchObject (32) v2&v3 200 OK
aliasProblem (33) v2&v3 505 Operations error
invalidDNSyntax (34) v2&v3 500 Syntax error
isLeaf (35) v2 N.A.
aliasDereferencingProblem (36) v2&v3 505 Operations error
inappropriateAuthentication (48) v2&v3 500 Syntax error
invalidCredentials (49) v2&v3 403 Information Unavailable
insufficientAccessRights (50) v2&v3 403 Information Unavailable
busy (51) v2&v3 403 Information Unavailable
unavailable (52) v2&v3 401 Service not available
unwillingToPerform (53) v2&v3 505 Operations error
loopDetect (54) v2&v3 505 Operations error
namingViolation (64) v2&v3 N.A.
objectClassViolation (65) v2&v3 N.A.
notAllowedOnNonLeaf (66) v2&v3 N.A.
notAllowedOnRDN (67) v2&v3 N.A.
entryAlreadyExists (68) v2&v3 N.A.
objectClassModsProhibited (69) v2&v3 N.A.
affectsMultipleDSAs (71) v3 N.A.
other (80) v2&v3 403 Information UnavailableTable D.1 LDAPv2/v3 resultcodes to DAG/IP response codes mapping
DAG/IP LDAP v2/v3
--------------------------------------- --------------------------
110 Too many hits sizeLimitExceeded (4)
111 Requested constraint not supported constraintViolation (19)
112 Requested constraint not fullfilled constraintViolation (19)
200 Command Ok Success (0)
201 Command Completed successfully N.A.
203 Bye N.A.
204 Overgeneralized N.A.
220 Service Ready N.A.
226 Transaction complete N.A.
401 Service not available unavailable (52)
402 Search expression too complicated unwillingToPerform (53)
403 Information Unavailable busy (51)
404 Time out timeLimitExceeded (3)
405 Operations error operationsError (1)
500 Syntax error protocolError (2)
502 Search expression too complicated unwillingToPerform (53)
503 Query to general unwillingToPerform (53)
505 Operations error operationsError (1)
600 <token> N.A.
601 <token> N.A.
601 DEF N.A.
601 ANY N.A.Table D.2 Mapping from DAG/IP response codes to LDAPv2/v3 resultcodes
DAG/IP Whois++
-------------------------------------- -----------------------------
110 Too Many hits 110 Too Many hits
111 Requested constraint not supported 111 Requested constraint not
supported
112 Requested constraint not fullfilled 112 Requested constraint not
fullfilled
200 Command Ok 200 Command Ok
201 Command Completed successfully 201 Command Completed
successfully
401 Service not available 401 Service not available
403 Information Unavailable 403 Information not available
404 Timeout 404 Timeout
405 Operations error 405 Operations error
500 Syntax error 500 Syntax error
502 Search expression too complicated 502 Search expression too
complicated
503 Query to general 506 Query to general
505 Operations error 505 Operations errorTable D.3 Mapping between DAG/IP and Whois++ response codes
Appendix E - DAG CIP Usage 🌸
E.1 CIP Index Object
okay so the CIP object the DAG system uses is based on the Tagged Index Object from [12] 💅 the grammar below is adapted from that Work in Progress and tailored to the exact object the DAG uses ✨
index-object = 0*(io-part SEP) io-part
io-part = header SEP schema-spec SEP index-info
header = version-spec SEP update-type SEP this-update SEP
last-update context-size
version-spec = "version:" *SPACE "x-tagged-index-1"
update-type = "updatetype:" *SPACE ( "total" |
( "incremental" [*SPACE "tagbased"|"uniqueIDbased" ])
this-update = "thisupdate:" *SPACE TIMESTAMP
last-update = [ "lastupdate:" *SPACE TIMESTAMP SEP]
context-size = [ "contextsize:" *SPACE 1*DIGIT SEP]
schema-spec = "BEGIN IO-Schema" SEP 1*(schema-line SEP)
"END IO-Schema"
schema-line = attribute-name ":" token-type
token-type = "TOKEN"
index-info = full-index | incremental-index
full-index = "BEGIN Index-Info" SEP 1*(index-block SEP)
"END Index-Info"
incremental-index = 1*(add-block | delete-block | update-block)
add-block = "BEGIN Add Block" SEP 1*(index-block SEP)
"END Add Block"
delete-block = "BEGIN Delete Block" SEP 1*(index-block SEP)
"END Delete Block"
update-block = "BEGIN Update Block" SEP
0*(old-index-block SEP)
1*(new-index-block SEP)
"END Update Block"
old-index-block = "BEGIN Old" SEP 1*(index-block SEP)
"END Old"
new-index-block = "BEGIN New" SEP 1*(index-block SEP)
"END New"
index-block = first-line 0*(SEP cont-line)
first-line = attr-name ":" *SPACE taglist "/" attr-value
cont-line = "-" taglist "/" attr-value
taglist = tag 0*("," tag) | "*"
tag = 1*DIGIT ["-" 1*DIGIT]
attr-value = 1*(UTF8)
attr-name = dag-searchattr / "objectclass"
dag-searchattr = "FN" / "LOC" / "ROLE" / "ORG"
TIMESTAMP = 1*DIGIT
NAMECHAR = DIGIT | UPPER | LOWER | "-" | ";" | "."
SPACE = <ASCII space, %x20>;
SEP = (CR LF) | LF
CR = <ASCII CR, carriage return, %x0D>;
LF = <ASCII LF, line feed, %x0A>;
DIGIT = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" |
"8" | "9"
UPPER = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" |
"I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" |
"Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" |
"Y" | "Z"
LOWER = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" |
"i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" |
"q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" |
"y" | "z"
US-ASCII-SAFE = %x01-09 / %x0B-0C / %x0E-7F
;; US-ASCII except CR, LF, NUL
UTF8 = US-ASCII-SAFE / UTF8-1 / UTF8-2 / UTF8-3
/ UTF8-4 / UTF8-5
UTF8-CONT = %x80-BF
UTF8-1 = %xC0-DF UTF8-CONT
UTF8-2 = %xE0-EF 2UTF8-CONT
UTF8-3 = %xF0-F7 3UTF8-CONT
UTF8-4 = %xF8-FB 4UTF8-CONT
UTF8-5 = %xFC-FD 5UTF8-CONTN.B. the only tokenization type allowed is "TOKEN" 🙅♀️ the Tagged Index Object memo does allow "FULL" (where the whole attribute value is kept as one single token), but that's a trap 😭 it gives you a unique token for every record. Studies on how centroid sizes grow with the number of records (see [14]) show that unique tokens, like phone numbers, should be avoided 📞 tags do cost a few extra bytes per token index entry, but unique tokens make the number of token entries grow linearly with the number of records, which tanks search efficiency 💀
Also, tags are applied per entry 🧚 so if two index lines in the same index object share a tag, those lines always point to the same "record" in the directory. In LDAP terms, the two lines would point to the same directory object 🎀
And if two index lines in the same index object have different tags, those lines always point to different records in the directory ✨
The attribute "objectclass" is used to mark the record/object types summarized in this index object 🌸
Values for the objectclass attribute should be restricted to: dagperson or dagrole, the two DAG schema object types 💜
E.2 CIP Index Object Creation
WDSPs are expected to make index objects following the general principles in the Whois++ protocol docs (how centroids get created) and the Tagged Index Object docs ([12]) 📚 Following the syntax from above, the index object holds token info for each attribute in the DAGSchema, which means:
- a list of all the unique tokens (strings delimited by the specified characters) that show up in the WDSP database for the attribute
- for each token in that list, which records the token shows up in
So, for example, say we have these two records:
Record #1:
FN: Foo Bar
ORG: The Snack Bar
Record #2:
FN: Bar Smith
ORG: Snack ShackThat yields (conceptually) the following info for the attribute FN:
Foo (1), Bar (1,2), Smith (2)and the following info for the attribute ORG:
The (1), Snack (1, 2), Bar (1), Shack (2)Note that the record numbers here are just tags, or virtual record identifiers, that show when 2 tokens are in the same record 🧩 The record identifiers are never used in any part of a query to the WDSP 🙅♀️
There's some debate about whether using the same record tag across all attributes makes it too easy to "decompile" the index object 👀 meaning rebuilding the WDSP's data by reordering the tokens for each attribute and tag number. But we're only dealing with the search attributes here, which is a tiny slice of everything the WDSP holds 🫶 So the conclusion is that the efficiency win from using the same tag numbers across attributes beats the (remote) chance of info reconstruction ✨
This would yield the index object:
version: x-tagged-index-1
update-type: total
this-update: 855938804
last-update:
context-size:
BEGIN IO-Schema
objectclass: TOKEN
FN: TOKEN
ORG: TOKEN
END IO-Schema
BEGIN Index-Info
objectclass: */dagperson
FN: 1/Foo
-1,2/Bar
-2/Smith
ORG: 1/The
-1,2/Snack
-1/Bar
-2/Shack
End Index-InfoTISDAG: Within the project it has been decided to base consistency between updates on consistent tags 🚨 This means that if the update-type is "incremental" the specifier must be "tagbased".
E.3 CIP Index Object Sharing
E.3.1 Registration of Servers
It is beyond the scope of this document to define how WDSP servers shall be registered with the DAG Referral Index 🙈 Such a procedure must be defined, and the following info has to be set up for each WDSP dataset (adapted from the Tagged Index Object spec, [12]) 📋
dsi: An OID which uniquely identifies the subtree and scope of the dataset the index object is created for.base-uri: One or more URI's which will form the base of any referrals created from the index object governed by this agreement. For example, for LDAP the base-uri would specify (among other items): the LDAP host, the base object the index object refers to (e.g., c=SE), and the scope of the index object (e.g., single container).supplier: The hostname and listening port number of the supplier server, plus any alternative servers holding that same naming contexts, in case the supplier is unavailable 🫠source-uri: The URI of the WDSP's preferred source of directory service info. This might be, for instance, an HTTP-based service.consumeraddr: A URI of the "mailto:" form, with the RFC 822 email address of the consumer server 💌updateinterval: The maximum duration in seconds between occurrences of the supplier server generating an update. If the consumer server hasn't received an update from the supplier server after waiting this long since the previous update, the index info is likely out of date ⏰ A typical value for a server with frequent updates would be 604800 seconds, or every week 🗓️attributeNamespace: Every set of index servers that together wants to support a specific usage of indices has to agree on which attribute names to use in the index objects. The participating directory servers also have to agree on the mapping from local attribute names to the attribute names used in the index. Since one specific index server might be involved in several such sets, it has to have some way to connect an update to the proper set of indexes. One possible solution would be to use different DSIs 🗂️consistencybase: How consistency of the index is maintained over incremental updates:complete- every change or delete concerning one object has to contain all tokens connected to that object. This method must be supported by any server that wants to comply with this standard.tagbased- starting at a full update, every incremental update referring back to this full update has to maintain state information regarding tags, so that an object within the original database is assigned the same tagnumber every time. This method is optional.uniqueID- every object in the Dataset has to have a unique value for a specific attribute in the index. An example of such an attribute could be the distinguishedName attribute. This method is also optional.
securityoption: Whether and how the supplier server should sign and encrypt the update before sending it to the consumer server. Options for this version of the DAG service are"none": the update is sent in plaintext; and"PGP/MIME": the update is digitally signed and encrypted using PGP (see [7])."PGP/MIME"is recommended 🔐security credentials: The long-term cryptographic credentials used for key exchange and authentication of the consumer and supplier servers, if a security option was selected. For"PGP/MIME", this will be the trusted public keys of both servers 🔑
E.3.2 Transmission of Objects
CIP Index Objects are sent to the DAG Referral Index by MIME-encoded SMTP, following the Common Indexing Protocol spec (see [2] and [3]) 📨✨
Appendix F - Summary of Technical Survey Results 📊✨
okay so, as part of the TISDAG project, there was a technical survey 🎀 It got announced on the tisdag@swip.net mailing list, and all Swedish WDSPs (and potential WDSPs) were encouraged to fill out and submit the WWW-based survey form (see http://tisdag.sunet.se/tisdag-survey.html) 💌
The survey ran in May, 1997. Response was not as good as they'd hoped 😭 In the end, 5 WDSPs participated. They wanted more answers so they could really get a feel for where directory service providers are right now and where they're headed 📈 But informal "hallway" chats with a few people at Interoperabilitet'97 in Sollentuna showed that, while everyone sees TISDAG as an important and timely step, they don't necessarily get how it'll affect them yet, or what they can or should contribute 🤷♀️ So these results are informational, not a definitive statement of the whole directory service picture in Sweden. Just keep that in mind, bestie 🫶
The tea ☕ is that even though there were only 5 respondents, these are clearly major players 👑 4 of them expect to have more than 100 000 records to contribute by 12 months from now 🤯 There were no real surprises when it came to the supported protocols or search types 💅
Table E.1 sums up the survey info about the types of queries WDSPs support right now, and what they planned to support over the next 12 months. Quick note: when the survey happened, the requirement of searching by ROLE hadn't been proposed yet 🙈 So the survey didn't specifically ask whether WDSPs supported both DAGPERSON schema protocol-equivalents (that's the USER template in Whois++ and the inetorgperson objectclass in LDAP). In the table, the column "Complete info?" tells you whether or not the WDSP currently returns at least as much information as is required for a DAG reply.
Resp Search Types Complete info? Access Protocols Access Protocols
(now) (12 months)
---- ------------ -------------- ---------------- ----------------
1 NOL Except ROLE Whois++ Whois++
2 N,NO,NL,NOL Except ROLE LDAPv2,DAP,PH, LDAPv2,LDAPv3,DAP,
HTTP,Gopher PH,HTTP,Gopher
3 N,NL,NOL Except ROLE LDAPv2,DAP,HTTP LDAPv2,LDAPv3,DAP,
HTTP
4 N,NO,NL,NOL Except ROLE Whois++,HTTP LDAPv3,Whois++,
HTTP,E-mail
5 N,NO,NL,NOL Except ROLE LDAPv2,Whois LDAPv2,LDAPv3,
Whois++,HTTP Whois,Whois++,PH,
Finger,HTTPTable F.1 Summary of TISDAG Survey Results: Queries
Resp # of Records (now) # of Records (12 months) Character Sets
----- ------------------ ------------------------ --------------
1 94 280 120 000 - 130 000 ISO-8859-1
2 88 000 100 000 ISO-8859-1
3 N/A 100 000 T.61 (Telex)
4 150 000 250 000 ISO-8859-1
UTF-8 UNICODE
5 4 300 10 000 ISO-8859-1Table F.2 Summary of TISDAG Survey Results: Operational Information
Appendix G - Useful References 📚🌸
N.B.: The following is a collection of Internet standards documents (RFCs) and Internet-Drafts that the material in this report was drawn from. Internet-Drafts are works-in-progress and are not meant to be cited. Where they're used in this document, references point to the text in the Internet-Draft. They aren't meant to imply standards so much as useful starting points for this project's work 💜
Electronic copies of the versions of the Internet-Drafts used in preparing this report are available from the project web page, http://tisdag.sunet.se.
Bibliography
[1] Allen, J. and M. Mealling, "The Architecture of the Common Indexing Protocol", RFC 2651, August 1999.
[2] Allen, J. and M. Mealing, "MIME Object Definitions for the Common Indexing Protocol (CIP)", RFC 2652, August 1999.
[3] Allen, J. and P. Leach, "CIP Transport Protocols", RFC 2653, August 1999.
[4] Crocker, D., "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, August 1982.
[5] Crocker, D., "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, November 1997.
[6] Deutsch, P., Schoultz, R., Falstrom, P. and C. Weider, "Architecture of the WHOIS++ Service", RFC 1835, July 1995.
[7] Elkins, M., "MIME Security with Pretty Good Privacy (PGP)", RFC 2015, October 1996.
[8] Patrik Faltstrom, Martin Hamilton, Leslie L. Daigle, "WHOIS++ templates", Work in Progress.
[9] Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Interent Message Bodies", RFC 2045, November 1996.
[10] Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046, November 1996.
[11] Good, G., "The LDAP Data Interchange Format (LDIF) - Technical Specification", RFC 2849, June 2000.
[12] Hedberg, R., Greenblatt, B., Moats, R. and M. Wahl, "A Tagged Index Object for use in the Common Indexing Protocol", RFC 2654, August 1999.
[13] Howes, R., "A String Representation of LDAP Search Filters", RFC 1960, June 1996.
[14] Paul Panotzki, "Complexity of the Common Indexing Protocol: Predicting Search Times in Index Server Meshes", Master's Thesis, KTH, September 1996.
[15] Postel, J., "Simple Mail Transfer Protocol", STD 10, RFC 821, August 1982.
[16] Smith, M., "Definition of the inetOrgPerson Object Class", RFC 2798, April 2000.
[17] Wahl, M., Howes, T. and S. Kille, "Lightweight Directory Access Protocol (v3)", RFC 2251, December 1997.
[18] Wahl, M., "A summary of the X.500(96) User Schema for use with LDAPv3", RFC 2256, December 1997.
[19] Yeong, W., Howes, T. and S. Kille, "Lightweight Directory Access Protocol", RFC 1777, March 1995.
[20] Yergeau, F., "UTF-8, a transformation format of ISO 10646", RFC 2279, January 1998.
[21] The Unicode Consortium, "The Unicode Standard -- Version 2.0", Addison-Wesley, 1996.
Authors' Addresses 💌
Leslie L. Daigle Thinking Cat Enterprises
EMail: leslie@thinkingcat.com
Roland Hedberg Catalogix Jegerveien 25 0777 Oslo Norway
Phone: +47 23 08 29 96 EMail: Roland@catalogix.se
Full Copyright Statement ©️
Copyright (C) The Internet Society (2000). All Rights Reserved.
This document and translations of it may be copied and shared with others. Derivative works that comment on or explain it, or help with implementing it, may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, as long as the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations. The exception is when it's needed for developing Internet standards, in which case the copyright procedures defined in the Internet Standards process must be followed, or when it's translated into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement 🫶
Funding for the RFC Editor function is currently provided by the Internet Society. 🌈
📦 where this page comes from
- ietf: rfc2967 2000-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms