RFC 8977

Registration Data Access Protocol (RDAP) Query Parameters for Result Sorting and Paging

📅 published January 2021, from ietf

RFC 8977 is a Standards Track doc, so it's official and it matters 💅✨ The title is Registration Data Access Protocol (RDAP) Query Parameters for Result Sorting and Paging 🎀 It's by M. Loffredo and M. Martinelli (IIT-CNR/Registro.it) and S. Hollenbeck (Verisign Labs) 👑 It's dated January 2021 and its ISSN is 2070-1721 🦋 The title block doesn't list anything it obsoletes or updates 🧚

Abstract ✨

RDAP doesn't have core functionality that lets clients ask for sorting and paging parameters to manage big result sets 💀 That gap makes it unpredictable how servers process queries and how clients process responses 😭 It gets way less chaotic if clients can tell servers what they prefer for handling big responses 🫶 This document describes RDAP query extensions that let clients say what they want for sorting and paging result sets 💖

Status of This Memo 🌸

This is an Internet Standards Track document 💅

It's a product of the Internet Engineering Task Force (IETF) 🦋 It represents the consensus of the IETF community and got public review. The Internet Engineering Steering Group (IESG) approved it for publication 👑 More info on Internet Standards is in Section 2 of RFC 7841 📚

Info about the current status, any errata, and how to send feedback is at https://www.rfc-editor.org/info/rfc8977 🔗

Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved 🌈

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the date of publication 📜 Please read these carefully, because they describe your rights and restrictions for this document 🙈 Code Components extracted from this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're provided without warranty, as described in the Simplified BSD License 💜

Table of Contents 🧚

1.  Introduction
  1.1.  Conventions Used in This Document
2.  RDAP Query Parameter Specification
  2.1.  Sorting and Paging Metadata
    2.1.1.  RDAP Conformance
  2.2.  "count" Parameter
  2.3.  "sort" Parameter
    2.3.1.  Sorting Properties Declaration
    2.3.2.  Representing Sorting Links
  2.4.  "cursor" Parameter
    2.4.1.  Representing Paging Links
3.  Negative Answers
4.  Implementation Considerations
5.  IANA Considerations
6.  Security Considerations
7.  References
  7.1.  Normative References
  7.2.  Informative References
Appendix A.  JSONPath Operators
Appendix B.  Approaches to Result Pagination
  B.1.  Specific Issues Raised by RDAP
Appendix C.  Implementation Notes
  C.1.  Sorting
  C.2.  Counting
  C.3.  Paging
Acknowledgements
Authors' Addresses

1. Introduction 🌷

Result sorting and paging are a big deal for clients and servers building RESTful services [REST] 💖 Here's why it's worth it:

  • reducing the server response bandwidth requirements
  • improving server response time
  • improving query precision and, consequently, obtaining more relevant results
  • decreasing server query processing load
  • reducing client response processing time

There are two main ways people do sorting and paging 🧠

  1. Sorting and paging go through extra parameters in the query string, like the Open Data Protocol (ODATA) [ODATA-PART1] 🎀
  2. Info about the number of results and which portion of the result set is returned, plus a set of ready-made links for scrolling through the result set, goes in the HTTP header of the request/response [RFC7231] 🦋

The HTTP header approach has some drawbacks, though 😤 First, you can't set header properties directly from a web browser 🙈 Also, in an HTTP session, the status info (the session identifier) usually goes in the header or a cookie, while the resource identification or search type goes in the query string 🍓 Last, putting custom info in HTTP headers assumes the client already knows the server implementation, and that's widely considered a Representational State Transfer (REST) design anti-pattern 💀 So this document goes with query parameters instead ✨

RDAP [RFC7482] defines two query types right now 🧚

  • lookup: the server returns only one object
  • search: the server returns a collection of objects

Lookup queries don't raise issues around response size, but search queries can generate a large result set that's often truncated based on server limits 😭 It's also not possible to get the total number of objects a search might return [RFC7483] 💔 And there's no way to specify sort criteria so the most relevant objects show up at the start of the result set 🥲 So a client might have to traverse the whole result set to find what it needs, or, because of truncation, might never find it at all 🫠

This spec extends RDAP query capabilities so you can sort and page results 💅 It does that by adding new query parameters that can be applied to RDAP search path segments. The service uses the Hypertext Transfer Protocol (HTTP) [RFC7230] and the conventions in [RFC7480] 🔗

Implementing these new parameters is technically feasible, since the major relational database management systems already support operators for counting, sorting, and paging rows 🧠✨

1.1. Conventions Used in This Document 📚

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🎀

2. RDAP Query Parameter Specification 👑

The new query parameters are OPTIONAL extensions of path segments defined in [RFC7482] ✨ Here they are:

  • "count": a boolean value that allows a client to request the return of the total number of objects found 🔢
  • "sort": a string value that allows a client to request a specific sort order for the result set 🪄
  • "cursor": a string value representing a pointer to a specific fixed-size portion of the result set 🧭

Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following sections to describe the formal syntax of these new parameters 📐

2.1. Sorting and Paging Metadata 🌈

Modern REST design principles, collectively known as HATEOAS (Hypermedia as the Engine of Application State) [HATEOAS], say that a client entering a REST application through an initial URI should use server-provided links to dynamically discover available actions and reach the resources it needs 🦋 That way, the client doesn't need prior knowledge of the service or have to hard-code URIs for different resources 🫶 The server can change URIs as the API evolves without breaking clients 💖 Honestly, a REST service should be as self-descriptive as possible 💅

So servers implementing the query parameters in this spec SHOULD provide extra info in their responses about both the available sorting criteria and possible pagination 🧁 That info lives in two OPTIONAL response elements, named sorting_metadata and paging_metadata 📦

The sorting_metadata element has these properties:

  • "currentSort": "String" (OPTIONAL): either the value of the "sort" parameter as specified in the query string, or the sort applied by default, if there is one 🎯
  • "availableSorts": "AvailableSort[]" (OPTIONAL): an array of objects, each describing an available sort criterion 🗂️ The AvailableSort object includes these members:
    • "property": "String" (REQUIRED): the name a client can use to request the sort criterion 🏷️
    • "default": "Boolean" (REQUIRED): an indicator of whether the sort criterion is applied by default. An RDAP server MUST define only one default sorting property for each object class 👑
    • "jsonPath": "String" (OPTIONAL): the JSONPath expression of the RDAP field that matches the property 🧩
    • "links": "Link[]" (OPTIONAL): an array of links as described in [RFC8288], containing the query string that applies the sort criterion 🔗

At least one of the "currentSort" and "availableSorts" properties MUST be present 💅

The paging_metadata element has these fields:

  • "totalCount": "Numeric" (OPTIONAL): a numeric value for the total number of objects found. It MUST be provided if and only if the query string contains the "count" parameter 🔢
  • "pageSize": "Numeric" (OPTIONAL): a numeric value for the number of objects that should have been returned in the current page. It MUST be provided if and only if the total number of objects exceeds the page size. This property is redundant for RDAP clients, because the page size can be derived from the length of the search results array, but it can be helpful if the end user interacts with the server through a web browser 🌸
  • "pageNumber": "Numeric" (OPTIONAL): a numeric value for the number of the current page in the result set. It MUST be provided if and only if the total number of objects found exceeds the page size 📄
  • "links": "Link[]" (OPTIONAL): an array of links as described in [RFC8288], containing the reference to the next page. This spec only describes forward pagination, because that's all you need to traverse the result set 🏃‍♀️💨

2.1.1. RDAP Conformance 🎀

Servers that return the paging_metadata element in their response MUST include the string literal "paging" in the rdapConformance array 🧾 Servers that return the sorting_metadata element MUST include the string literal "sorting" 💜

2.2. "count" Parameter ✨

okay so here's the tea 🫖 right now RDAP doesn't let a client figure out the total number of results when the response gets truncated 😭 which is so inefficient bestie, because the user literally can't tell if they've seen everything. the "count" parameter fixes that! it lets a client ask the server for the total number of objects that match the search pattern 💖

here's a query with "count" in it 🎀

https://example.com/rdap/domains?name=example*.com&count=true

and the ABNF (the grammar, so every valid "count" looks the same to every server 🧚)

count = "count=" ( trueValue / falseValue )
trueValue = ("true" / "yes" / "1")
falseValue = ("false" / "no" / "0")

so a trueValue (that's "true", "yes", or "1") means the server MUST provide the total number of objects in the "totalCount" field of the "paging_metadata" element (Figure 1) 🙌 and a falseValue (that's "false", "no", or "0") means the server MUST NOT provide this number 🙅‍♀️

{
  "rdapConformance": [
        "rdap_level_0",
        "paging"
  ],
  ...
  "paging_metadata": {
    "totalCount": 43
  },
  "domainSearchResults": [
    ...
  ]
}

Figure 1: Example of RDAP Response with "paging_metadata" Element Containing the "totalCount" Field 💜

2.3. "sort" Parameter 💅

RDAP has no way at all to say how the results should be sorted 😤 a server could pick a default sorting scheme based on the object class, but that's not mandatory and it might not match what the user needs. the client can sort things itself, but that's rather inefficient. server-side sorting could also help avoid truncation of the results that actually matter 🍓

so the "sort" parameter lets the client ask the server to sort the results by the values of one or more properties, with a sort direction for each property 🦋 the ABNF is:

sort = "sort=" sortItem *( "," sortItem )
sortItem = property-ref [":" ( "a" / "d" ) ]
property-ref = ALPHA *( ALPHA / DIGIT / "_" )

"a" means an ascending sort MUST be applied ⬆️ and "d" means a descending sort MUST be applied ⬇️ and if the sort direction is absent, an ascending sort MUST be applied 🌸

here are some queries that use "sort" 🧚

https://example.com/rdap/domains?name=example*.com&sort=name
https://example.com/rdap/
domains?name=example*.com&sort=registrationDate:d
https://example.com/rdap/
domains?name=example*.com&sort=lockedDate,name
okay this part is the big one 🚨 except for sorting IP addresses and values that denote dates and times, servers MUST implement sorting according to the JSON value type of the RDAP field that the sorting property refers to. so JSON strings MUST be sorted lexicographically, and JSON numbers MUST be sorted numerically. values denoting dates and times MUST be sorted in chronological order. and if IP addresses are represented as JSON strings, they MUST be sorted based on their numeric conversion 🔥

so how do you turn an IP address into a number?? it's actually giving math 🤓 each dotted IPv4 address is a number written in a 256-based manner. for example, 192.168.0.1 means 1*256^0 + 0*256^1 + 168*256^2 + 192*256^3 = 3232235521. IPv6 works the same way but with base 65536 🌈 so the numerical representation of 2001:0db8:85a3:0:0:8a2e:0370:7334 is 42540766452641154071740215577757643572 ✨ and the good news is that built-in functions and libraries for converting IP addresses into numbers exist in most known programming languages and relational database management systems 💖

last bit: if the "sort" parameter presents an allowed sorting property, it MUST be provided in the "currentSort" field of the "sorting_metadata" element 🫶

2.3.1. Sorting Properties Declaration ✨

Okay so in the "sort" parameter ABNF, the element called property-ref is basically a pointer to a property on an RDAP object 💅. You could write that pointer as a JSONPath expression (we'll call it "jsonpath" from here on) 🦋.

JSONPath is giving XPath energy, since it's originally based on the XML XPath notation [W3C.CR-xpath-31-20170321]. It's a way to point at one element (or a whole bunch of them) inside a JSON document [RFC8259]. For example, the jsonpath for the ASCII name in an RDAP domain lookup response is $.ldhName, where $ is the root of the document object model (DOM) 🌸. Another option for selecting a value inside JSON is the JSON Pointer [RFC6901].

Both JSONPath and JSON Pointer are super common for grabbing any value out of JSON data, but honestly neither one is very concise or easy to use 😭. Check out the jsonpath for the registration date in an RDAP domain search response:

$.domainSearchResults[*].events[?(@.eventActi
on='registration')].eventDate

Yikes, bestie 💀. So this specification defines property-ref in terms of names that identify RDAP properties instead. Not every RDAP property is a good fit for sort criteria though 🙈. These are the ones that don't work:

  • properties providing service information (e.g., links, notices, and remarks)
  • multivalued properties (e.g., status, roles, and variants)
  • properties representing relationships to other objects (e.g., entities)

On the flip side, properties that are expressed as values of other properties (e.g., registration date) can be used in this context 🎀.

A list of properties that an RDAP server MAY implement is defined 🧚. The properties are split into two groups: object-common properties and object-specific properties.

  • Object-common properties. These are derived from merging the "eventAction" and the "eventDate" properties 🍓. The following values of the "sort" parameter are defined:
    • registrationDate
    • reregistrationDate
    • lastChangedDate
    • expirationDate
    • deletionDate
    • reinstantiationDate
    • transferDate
    • lockedDate
    • unlockedDate
  • Object-specific properties. Some of these are also defined as query path segments, so keep an eye out 👀. They include:
    • Domain: name
    • Nameserver: name, ipv4, ipv6
    • Entity: fn, handle, org, email, voice, country, cc, city

The correspondence between these sorting properties and the RDAP object classes is shown in Table 1 💜. Some of the sorting properties for the RDAP entity class are related to jCard elements [RFC7095]. But since jCard is the JSON format for vCard, the corresponding definitions live in the vCard specification [RFC6350], so you know where to look 📚.

+============+============+=================+======+=======+======+
| Object     | Sorting    | RDAP property   | RFC  | RFC   | RFC  |
| class      | property   |                 | 7483 | 6350  | 8605 |
+============+============+=================+======+=======+======+
| Searchable | Common     | eventAction     | 4.5  |       |      |
| objects    | properties | values suffixed |      |       |      |
|            |            | by "Date"       |      |       |      |
+------------+------------+-----------------+------+-------+------+
| Domain     | name       | unicodeName/    | 5.3  |       |      |
|            |            | ldhName         |      |       |      |
+------------+------------+-----------------+------+-------+------+
| Nameserver | name       | unicodeName/    | 5.2  |       |      |
|            |            | ldhName         |      |       |      |
+------------+------------+-----------------+------+-------+------+
|            | ipv4       | v4 ipAddress    | 5.2  |       |      |
+------------+------------+-----------------+------+-------+------+
|            | ipv6       | v6 ipAddress    | 5.2  |       |      |
+------------+------------+-----------------+------+-------+------+
| Entity     | handle     | handle          | 5.1  |       |      |
+------------+------------+-----------------+------+-------+------+
|            | fn         | jCard fn        | 5.1  | 6.2.1 |      |
+------------+------------+-----------------+------+-------+------+
|            | org        | jCard org       | 5.1  | 6.6.4 |      |
+------------+------------+-----------------+------+-------+------+
|            | voice      | jCard tel with  | 5.1  | 6.4.1 |      |
|            |            | type="voice"    |      |       |      |
+------------+------------+-----------------+------+-------+------+
|            | email      | jCard email     | 5.1  | 6.4.2 |      |
+------------+------------+-----------------+------+-------+------+
|            | country    | country name in | 5.1  | 6.3.1 |      |
|            |            | jCard adr       |      |       |      |
+------------+------------+-----------------+------+-------+------+
|            | cc         | country code in | 5.1  |       | 3.1  |
|            |            | jCard adr       |      |       |      |
+------------+------------+-----------------+------+-------+------+
|            | city       | locality in     | 5.1  | 6.3.1 |      |
|            |            | jCard adr       |      |       |      |
+------------+------------+-----------------+------+-------+------+

                 Table 1: Definitions of Sorting Properties

An RDAP server MUST NOT use the defined sorting properties with a meaning other than the one described in Table 1 🚫. Strict but fair, periodt 💅.

Now for some extra considerations so there's zero confusion 🤓:

  • Searches on domains or nameservers can return both A-labels and U-labels [RFC5890], so a consistent sorting policy MUST treat the unicodeName and ldhName as two representations of the same value 🌈. The unicodeName value MUST be used while sorting if it's present. If the unicodeName is unavailable, the value of the ldhName MUST be used instead.
  • The jCard "sort-as" parameter MUST be ignored for the sorting capability described in this document. Sorry, sort-as, you're not invited 🙅‍♀️.
  • Nameservers can have multiple IPv4 and IPv6 addresses, but the most common setup is one address per IP version. So this specification assumes nameservers have a single IPv4 and/or IPv6 value 🦋. When more than one address per IP version shows up, sorting MUST be applied to the first value.
  • Multiple events with a given action on an object might come back. If that happens, sorting MUST be applied to the most recent event ⏰.
  • Except for handle values, every sorting property defined for entity objects can be multivalued, following the vCard definition in [RFC6350] 🎀. When more than one value is present, sorting MUST be applied to the preferred value, the one identified by the parameter pref="1". If the "pref" parameter is missing, sorting MUST be applied to the first value.

The "jsonPath" field in the "sorting_metadata" element clarifies which RDAP response field a sorting property refers to 🗺️. Here's the mapping between the sorting properties and the jsonpaths of the RDAP response fields. The JSONPath operators used here are described in Appendix A ✨.

Searchable objects:

registrationDate
   $.domainSearchResults[*].events[?(@.eventAction=="registration"
   )].eventDate

reregistrationDate
   $.domainSearchResults[*].events[?(@.eventAction=="reregistratio
   n")].eventDate

lastChangedDate
   $.domainSearchResults[*].events[?(@.eventAction=="last
   changed")].eventDate

expirationDate
   $.domainSearchResults[*].events[?(@.eventAction=="expiration")]
   .eventDate

deletionDate
   $.domainSearchResults[*].events[?(@.eventAction=="deletion")].e
   ventDate

reinstantiationDate
   $.domainSearchResults[*].events[?(@.eventAction=="reinstantiati
   on")].eventDate

transferDate
   $.domainSearchResults[*].events[?(@.eventAction=="transfer")].e
   ventDate

lockedDate
   $.domainSearchResults[*].events[?(@.eventAction=="locked")].eve
   ntDate

unlockedDate
   $.domainSearchResults[*].events[?(@.eventAction=="unlocked")].e
   ventDate

Domain:

name
   $.domainSearchResults[*].[unicodeName,ldhName]

Nameserver:

name
   $.nameserverSearchResults[*].[unicodeName,ldhName]

ipv4
   $.nameserverSearchResults[*].ipAddresses.v4[0]

ipv6
   $.nameserverSearchResults[*].ipAddresses.v6[0]

Entity:

handle
   $.entitySearchResults[*].handle

fn
   $.entitySearchResults[*].vcardArray[1][?(@[0]=="fn")][3]

org
   $.entitySearchResults[*].vcardArray[1][?(@[0]=="org")][3]

voice
   $.entitySearchResults[*].vcardArray[1][?(@[0]=="tel" &&
   @[1].type=="voice")][3]

email
   $.entitySearchResults[*].vcardArray[1][?(@[0]=="email")][3]

country
   $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][3][6]

cc
   $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][1].cc

city
   $.entitySearchResults[*].vcardArray[1][?(@[0]=="adr")][3][3]

A few additional notes on those jsonpaths 📝:

  • The ones related to event dates are only defined for the "domain" object. To get the equivalent jsonpaths for "entity" and "nameserver", swap the domainSearchResults path segment for entitySearchResults and nameserverSearchResults respectively.
  • The ones related to jCard elements are specified without taking the "pref" parameter into account. Servers that sort those values identified by the "pref" parameter SHOULD update a jsonpath by adding an appropriate filter 🎯. For example, if the email values identified by pref="1" are being considered for sorting, the jsonpath of the "email" sorting property should be:
$.entitySearchResults[*].vcardArray[1][?(@[0]=="email" &&
      @[1].pref=="1")][3].
Okay but read that last one twice, bestie. The filter is what keeps the sort on the right email 😭🫶.

okay bestie, so an RDAP server MAY use the "links" array inside "sorting_metadata" to hand you ready-made references [RFC8288] to the sort criteria it supports (see Figure 2) 💅. Each link is basically a pointer to an alternate view of the results 🦋

The "value", "rel", and "href" JSON values MUST be specified. Everything else is OPTIONAL 🌸, so those three are the must-haves and the rest is extra sparkle ✨

{
  "rdapConformance": [
    "rdap_level_0",
    "sorting"
  ],
  ...
  "sorting_metadata": {
     "currentSort": "name",
     "availableSorts": [
       {
       "property": "registrationDate",
       "jsonPath": "$.domainSearchResults[*]
          .events[?(@.eventAction==\"registration\")].eventDate",
       "default": false,
       "links": [
         {
         "value": "https://example.com/rdap/domains?name=example*.com
                   &sort=name",
         "rel": "alternate",
         "href": "https://example.com/rdap/domains?name=example*.com
                  &sort=registrationDate",
         "title": "Result Ascending Sort Link",
         "type": "application/rdap+json"
         },
         {
         "value": "https://example.com/rdap/domains?name=example*.com
                   &sort=name",
         "rel": "alternate",
         "href": "https://example.com/rdap/domains?name=example*.com
                  &sort=registrationDate:d",
         "title": "Result Descending Sort Link",
         "type": "application/rdap+json"
         }
       ]
       },
       ...
     ]
  },
  "domainSearchResults": [
    ...
  ]
}

Figure 2: Example of a "sorting_metadata" Instance to Implement Result Sorting

2.4. "cursor" Parameter 🧭

the "cursor" parameter in this spec is super flexible 💖 It can be used to encode information about any pagination method. For example, if you build a simple "cursor" that stores offset pagination info, the value "b2Zmc2V0PTEwMCxsaW1pdD01MA==" is the base64 encoding of "offset=100,limit=50". Same vibe for keyset pagination: the "cursor" value "ZXhhbXBsZS1OLmNvbQ==" is the base64 encoding of "key=example-N.com", where the key identifies the last row of the current page 🍓

Heads up though, this spec only uses base64 for cursor obfuscation as an example. RDAP servers are NOT RECOMMENDED to obfuscate the "cursor" value through nothing more than base64 encoding 🙈

This approach lets RDAP providers pick a pagination method that fits their needs, the user's access level, and the submitted query. Plus, servers can switch methods over time without telling clients anything 🧚 The reasoning behind all this is covered in more detail in Appendix B.

The ABNF syntax of the "cursor" parameter is:

cursor = "cursor=" 1*( ALPHA / DIGIT / "/" / "=" / "-" / "_" )

Here's an example of an RDAP query with the "cursor" parameter:

https://example.com/rdap/domains?name=example*.com
&cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=

An RDAP server SHOULD use the "links" array of the "paging_metadata" element to provide a ready-made reference [RFC8288] to the next page of the result set (see Figure 3) 🎀 Some other "rel" values a server MAY implement are "first", "last", and "prev" 💜

{
  "rdapConformance": [
    "rdap_level_0",
    "paging"
  ],
  ...
  "notices": [
    {
      "title": "Search query limits",
      "type": "result set truncated due to excessive load",
      "description": [
      "search results for domains are limited to 50"
      ]
    }
  ],
  "paging_metadata": {
    "totalCount": 73,
    "pageSize": 50,
    "pageNumber": 1,
    "links": [
      {
      "value": "https://example.com/rdap/domains?name=example*.com",
      "rel": "next",
      "href": "https://example.com/rdap/domains?name=example*.com
              &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=",
      "title": "Result Pagination Link",
      "type": "application/rdap+json"
      }
    ]
  },
  "domainSearchResults": [
    ...
  ]
}

Figure 3: Example of a "paging_metadata" Instance to Implement Cursor Pagination

3. Negative Answers 🙅‍♀️

Parameter values are checked against their ABNF syntax, so any request with an invalid value for a parameter SHOULD produce an HTTP 400 (Bad Request) response code. That same response SHOULD be returned in these cases:

  • if sorting by either single or multiple properties, the client provides an unsupported value for the "sort" parameter, as well as a value related to an object property not included in the response
  • if the client submits an invalid value for the "cursor" parameter

Optionally, the response MAY include extra info about either the supported sorting properties or the correct "cursor" value in the HTTP entity body (see Figure 4) 🌈

{
    "errorCode": 400,
    "title": "Domain sorting property 'unknown' is not valid",
    "description": [
        "Supported domain sorting properties are:"
        "'aproperty', 'anotherproperty'"
    ]

}

Figure 4: Example of RDAP Error Response Due to an Invalid Domain Sorting Property Included in the Request

4. Implementation Considerations 🛠️

Honestly, the new parameters are totally doable 🙌 Counting, sorting, and paging operators are already supported by the major relational database management systems. Similar operators are fully or partly supported by the most well-known NoSQL databases (e.g., MongoDB, CouchDB, HBase, Cassandra, Hadoop, etc.). More implementation notes are in Appendix C 📚

5. IANA Considerations 🗂️

IANA has registered the following values in the "RDAP Extensions" registry:

   Extension identifier:  paging
   Registry operator:  Any
   Published specification:  RFC 8977
   Contact:  IETF <iesg@ietf.org>
   Intended usage:  This extension describes a best practice for result
      set paging.

   Extension identifier:  sorting
   Registry operator:  Any
   Published specification:  RFC 8977
   Contact:  IETF <iesg@ietf.org>
   Intended usage:  This extension describes a best practice for result
      set sorting.

6. Security Considerations 🔐

Security services for the operations in this document are covered in [RFC7481] 🛡️

Searches are the heavy lifters 🏋️‍♀️ A search query usually eats more server resources (memory, CPU cycles, and network bandwidth) than a lookup query does. That raises the risk of server resource exhaustion and a denial of service afterward. You can lower that risk by restricting search functionality or rate-limiting search requests. Servers can also cut their load by truncating results in a response. BUT this last policy can cause more inefficiency or risk, because clients end up acting on incomplete information when the RDAP server offers no way to return the truncated results 💀

The new parameters in this document give RDAP operators a way to build a server that avoids those inefficiency risks ✨ The "count" parameter lets the client judge whether a response is complete. The "sort" parameter gets the most relevant info to the front of the result set, which can cut down on unnecessary search requests. And the "cursor" parameter lets the user scroll through the result set with a sequence of sustainable queries that stay within server-acceptable limits 👑

7. References 📖

7.1. Normative References

7.2. Informative References 📚

Appendix A. JSONPath Operators 🧚

The jsonpaths used in this document are provided according to the Goessner proposal [GOESSNER-JSON-PATH].

Okay so, that spec requires that implementations support a set of "basic operators". 💅 These let you access the elements of a JSON structure, like objects and arrays, plus their subelements (object members and array items, respectively). There are no operations for grabbing a parent or sibling element of a given element. The root element is always referred to as $, whether it's an object or an array. 🌸

The spec also lets implementations support arbitrary script expressions. These can index into an object or array, or filter elements out of an array. Script expression behavior is implementation-defined, but most implementations support the basic relational and logical operators, as well as both object member and array item access, and they're similar enough for the purposes of this document. The commonly supported operators and functions, split into "top-level operators" and "filter operators", are documented in Tables 2 and 3. 🎀

For more information on implementation interoperability issues, see [JSONPATH-COMPARISON]. At the time of writing, work is also beginning on a standardization effort (see [JSONPATH-WG]). 🦋

+===================+=========================================+
| Operator          | Description                             |
+===================+=========================================+
| $                 | Root element                            |
+-------------------+-----------------------------------------+
| .<name>           | Object member access (dot-notation)     |
+-------------------+-----------------------------------------+
| ['<name>']        | Object member access (bracket-notation) |
+-------------------+-----------------------------------------+
| [<number>]        | Array item access                       |
+-------------------+-----------------------------------------+
| *                 | All elements within the specified scope |
+-------------------+-----------------------------------------+
| [?(<expression>)] | Filter expression                       |
+-------------------+-----------------------------------------+

             Table 2: JSONPath Top-Level Operators
+====================+========================================+
| Operator           | Description                            |
+====================+========================================+
| @                  | Current element being processed        |
+--------------------+----------------------------------------+
| .<name>            | Object member access                   |
+--------------------+----------------------------------------+
| .[<name1>,<name2>] | Union of object members                |
+--------------------+----------------------------------------+
| [<number>]         | Array item access                      |
+--------------------+----------------------------------------+
| ==                 | Left is equal to right                 |
+--------------------+----------------------------------------+
| !=                 | Left is not equal to right             |
+--------------------+----------------------------------------+
| <                  | Left is less than right                |
+--------------------+----------------------------------------+
| <=                 | Left is less than or equal to right    |
+--------------------+----------------------------------------+
| >                  | Left is greater than right             |
+--------------------+----------------------------------------+
| >=                 | Left is greater than or equal to right |
+--------------------+----------------------------------------+
| &&                 | Logical conjunction                    |
+--------------------+----------------------------------------+
| ||                 | Logical disjunction                    |
+--------------------+----------------------------------------+

               Table 3: JSONPath Filter Operators

Appendix B. Approaches to Result Pagination 🍓

An RDAP query could return a response with hundreds, even thousands, of objects, especially when partial matching is used. 😭 That's why the cursor parameter for result pagination is defined: to make responses easier to handle. 💜

Right now, the most popular ways to do pagination in a REST API are offset pagination and keyset pagination. Neither one requires the server to hold the result set in a storage area across multiple requests, since a new result set is generated every time a request is submitted. So they're preferred over any other method that requires managing a REST session. ✨

The traditional method uses the limit and offset operators. You can use each one on its own:

  • "limit=N": means that the server returns the first N objects of the result set
  • "offset=N": means that the server skips the first N objects and returns objects starting from position N+1

When limit and offset are used together, they let you identify a specific portion of the result set. For example, the pair offset=100,limit=50 returns the first 50 objects, starting from position 101 of the result set. 🦋

Offset pagination is easy to implement, but it has drawbacks too:

  • When offset has a very high value, scrolling the result set could take some time.
  • It always requires fetching all rows before dropping as many rows as specified by offset.
  • It may return inconsistent pages when data are frequently updated (i.e., real-time data).

Keyset pagination [SEEK] adds a query condition that selects only the data not yet returned. 🌈 Some REST API providers [CURSOR-API1] [CURSOR-API2] have based their implementation of a cursor parameter [CURSOR] on this method. The cursor is a URL-safe string that's opaque to the client, and it represents a logical pointer to the first result of the next page. 🫶

Even keyset pagination can be a little troublesome, though:

  • It needs at least one key field.
  • It does not allow sorting simply by any field, because the sorting criterion must contain a key.
  • It works best with full composite values supported by database management systems (i.e., [x,y]>[a,b]); emulation is possible but inelegant and less efficient.
  • It does not allow direct navigation to arbitrary pages, because the result set must be scrolled in sequential order starting from the initial page.
  • Implementing bidirectional navigation is tedious, because all comparison and sort operations have to be reversed. 💀

B.1. Specific Issues Raised by RDAP 🔥

Some additional considerations apply in the RDAP context:

  • An RDAP object is a conceptual aggregation of information generally collected from more than one data structure (e.g., table), and this makes keyset pagination, which is already quite difficult, even harder to implement. For example, the entity object can include information from different data structures (registrars, registrants, contacts, resellers), each one with its key field mapping the RDAP entity handle. 👑
  • Depending on the number of page results, as well as the number and the complexity of the properties of each RDAP object in the response, the time required by offset pagination to skip the previous pages could be much faster than the processing time needed to build the current page. In fact, RDAP objects are usually formed from information belonging to multiple data structures and containing multivalued properties (i.e., arrays); therefore, data selection might be a time-consuming process. This situation occurs even though the selection is supported by indexes. 🙈
  • Depending on the access levels defined by each RDAP operator, the increase in complexity and the decrease in flexibility of keyset pagination compared to offset pagination could be considered impractical. 🌸

Ultimately, both pagination methods have benefits and drawbacks. 🫶

Appendix C. Implementation Notes 💖

This section gives an overview of the main choices made while implementing the capabilities defined in this document in the RDAP public test server of Registro.it at the Institute of Informatics and Telematics of the National Research Council (IIT-CNR). 🎀 The content here can be guidance for implementers who plan to provide RDAP users with those capabilities. You can access the RDAP public test server at <https://rdap.pubtest.nic.it/>, and more documentation about the server features is at <https://rdap.pubtest.nic.it/doc/README.html>. ✨

C.1. Sorting

okay so if the query string doesn't say how to sort, the results come back sorted by a default property 💅: name for domains and nameservers, and handle for entities. The server can sort by multiple properties 🦋, but the sorting_metadata object only lists links to alternative result set views that are sorted by a single property. Those links are just there to show which sorting properties are allowed for each searchable object 🎀. The server supports all of the object-specific sorting properties in the specification, with one exception: nameserver sorting based on unicodeName isn't included. So the name sorting property gets mapped onto the ldhName response field 🌸. For the object-common properties, sorting by registrationDate, expirationDate, lastChangedDate, and transferDate is supported ✨.

C.2. Counting

counting is its own separate query 🧮. Some relational database management systems have custom operators that can grab the total count together with the rows, but that combo query can get way more expensive than the one you'd run without the total count 💀. totalCount is optional response info, so always fetching the total number of rows felt like an inefficient move, and it was not the way to go 🙅‍♀️. To avoid running unnecessary queries, when the count parameter is in the submitted query, it doesn't get repeated in the query strings of the links arrays inside both the paging_metadata and sorting_metadata objects 🫶.

C.3. Paging

the server does cursor pagination through keyset pagination when sorting by a unique property is requested or when the default sort is applied 🔑. Otherwise, it does cursor pagination through offset pagination 🍓. Most relational database management systems don't natively support comparing full composite values, so full keyset pagination seems kinda troublesome 😭. That's why, at least at first, using keyset pagination selectively is the advisable move 🧚. Also, the cursor value doesn't just encode pagination info. It also encodes the search pattern and the other query parameters, so the server can check that the whole query string is consistent 👑. If the cursor value doesn't match the rest of the query string, the server sends back an error response 🚨.

Acknowledgements ✨

The authors would like to give a huge shoutout to Brian Mountford, Tom Harrison, Karl Heinz Wolf, Jasdip Singh, Erik Kline, Éric Vyncke, Benjamin Kaduk, and Roman Danyliw for their contributions to this document 🫶💖.

Authors' Addresses 🦋

Mario Loffredo
IIT-CNR/Registro.it
Via Moruzzi,1
56124 Pisa
Italy

Email: mario.loffredo@iit.cnr.it
URI:   https://www.iit.cnr.it


Maurizio Martinelli
IIT-CNR/Registro.it
Via Moruzzi,1
56124 Pisa
Italy

Email: maurizio.martinelli@iit.cnr.it
URI:   https://www.iit.cnr.it


Scott Hollenbeck
Verisign Labs
12061 Bluemont Way
Reston, VA 20190
United States of America

Email: shollenbeck@verisign.com
URI:   https://www.verisignlabs.com/
📦 where this page comes from