RFC 9309

Robots Exclusion Protocol

📅 published September 2022, from ietf

RFC 9309: Robots Exclusion Protocol 🎀

Okay, bestie, let's get into it! 🦋 This is RFC 9309, the Robots Exclusion Protocol, written by M. Koster, G. Illyes, H. Zeller, and L. Sassman from Google LLC. It's a Standards Track doc from the IETF, dated September 2022, and its ISSN is 2070-1721. The title block doesn't list anything it obsoletes or updates, so nothing gets swapped out here 💅

Abstract

So the Robots Exclusion Protocol got its start with Martijn Koster back in 1994 🕰️ It lets service owners say how automatic clients called crawlers can get at their content, and whether they can at all 🤖 This RFC takes that original idea and builds on it. It adds definition language for the protocol, guidance for handling errors, and guidance for caching 🧚

Status of This Memo

This is an Internet Standards Track document. 🌸

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841. ✨

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc9309. 📬

Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved. ©️🎀

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. 📜💜

Table of Contents

1.  Introduction
  1.1.  Requirements Language
2.  Specification
  2.1.  Protocol Definition
  2.2.  Formal Syntax
    2.2.1.  The User-Agent Line
    2.2.2.  The "Allow" and "Disallow" Lines
    2.2.3.  Special Characters
    2.2.4.  Other Records
  2.3.  Access Method
    2.3.1.  Access Results
      2.3.1.1.  Successful Access
      2.3.1.2.  Redirects
      2.3.1.3.  "Unavailable" Status
      2.3.1.4.  "Unreachable" Status
      2.3.1.5.  Parsing Errors
  2.4.  Caching
  2.5.  Limits
3.  Security Considerations
4.  IANA Considerations
5.  Examples
  5.1.  Simple Example
  5.2.  Longest Match
6.  References
  6.1.  Normative References
  6.2.  Informative References
Authors' Addresses

1. Introduction 🌷

This doc applies to services that offer resources clients can reach through URIs, as defined in [RFC3986]. For example, when we're talking about HTTP, a browser is a client that shows you the content of a web page 🖥️

Crawlers are automated clients. Search engines, for instance, have crawlers that follow links recursively so they can index stuff, as defined in [RFC8288]. 🕷️

Sometimes it's a lot for service owners if crawlers visit their entire URI space 😩 This document lays out the rules, originally defined by the "Robots Exclusion Protocol" [ROBOTSTXT], that crawlers are asked to follow when they access URIs. 🫶

Just so we're clear, these rules are not a form of access authorization. They're not a bouncer, bestie 🚫

1.1. Requirements Language

Okay, so this part is about the BCP 14 key words, the special words that tell you how strict a requirement is. They're interpreted as described in BCP 14 [RFC2119] [RFC8174], but only when they show up in all capitals, like they do right here: "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL". Anything in lowercase is just normal English 📚✨

2. Specification 🎀

2.1. Protocol Definition

The protocol language is made of rule(s) and group(s) that the service makes available in a file named "robots.txt", as described in Section 2.3. 📄

  • Rule: A line with a key-value pair that defines how a crawler may access URIs. See Section 2.2.2. 🔑
  • Group: One or more user-agent lines, followed by one or more rules. The group ends when there's another user-agent line or the end of the file. See Section 2.2.1. The last group may have no rules, which means it implicitly allows everything. 💖

2.2. Formal Syntax

Here's the Augmented Backus-Naur Form (ABNF) description, as described in [RFC5234]. It's the grammar, so don't mess with it 🧐

 robotstxt = *(group / emptyline)
 group = startgroupline                ; We start with a user-agent
                                       ; line
        *(startgroupline / emptyline)  ; ... and possibly more
                                       ; user-agent lines
        *(rule / emptyline)            ; followed by rules relevant
                                       ; for the preceding
                                       ; user-agent lines

 startgroupline = *WS "user-agent" *WS ":" *WS product-token EOL

 rule = *WS ("allow" / "disallow") *WS ":"
       *WS (path-pattern / empty-pattern) EOL

 ; parser implementors: define additional lines you need (for
 ; example, Sitemaps).

 product-token = identifier / "*"
 path-pattern = "/" *UTF8-char-noctl ; valid URI path pattern
 empty-pattern = *WS

 identifier = 1*(%x2D / %x41-5A / %x5F / %x61-7A)
 comment = "#" *(UTF8-char-noctl / WS / "#")
 emptyline = EOL
 EOL = *WS [comment] NL ; end-of-line may have
                        ; optional trailing comment
 NL = %x0D / %x0A / %x0D.0A
 WS = %x20 / %x09

 ; UTF8 derived from RFC 3629, but excluding control characters

 UTF8-char-noctl = UTF8-1-noctl / UTF8-2 / UTF8-3 / UTF8-4
 UTF8-1-noctl = %x21 / %x22 / %x24-7F ; excluding control, space, "#"
 UTF8-2 = %xC2-DF UTF8-tail
 UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2UTF8-tail /
          %xED %x80-9F UTF8-tail / %xEE-EF 2UTF8-tail
 UTF8-4 = %xF0 %x90-BF 2UTF8-tail / %xF1-F3 3UTF8-tail /
          %xF4 %x80-8F 2UTF8-tail

 UTF8-tail = %x80-BF

2.2.1. The User-Agent Line 🤖💅

Crawlers set their own name, called a product token, so they can find the groups that are relevant to them. The product token MUST contain only uppercase and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-"). The product token SHOULD be a substring of the identification string that the crawler sends to the service. For example, with HTTP [RFC9110], the product token SHOULD be a substring in the User-Agent header. The identification string SHOULD describe the purpose of the crawler. 🎯

Here's an example of a User-Agent HTTP request header with a link to a page that describes what the ExampleBot crawler is for. The product token shows up as a substring in the User-Agent HTTP header and as the product token in the robots.txt user-agent line 👀

+==========================================+========================+
| User-Agent HTTP header                   | robots.txt user-agent  |
|                                          | line                   |
+==========================================+========================+
| User-Agent: Mozilla/5.0 (compatible;     | user-agent: ExampleBot |
| ExampleBot/0.1;                          |                        |
| https://www.example.com/bot.html)        |                        |
+------------------------------------------+------------------------+

        Figure 1: Example of a User-Agent HTTP header and robots.txt
              user-agent line for the ExampleBot product token

Notice that the product token (ExampleBot) is a substring of the User-Agent HTTP header. Cute, right? 🥰

Crawlers MUST use case-insensitive matching to find the group that matches the product token, and then obey the rules of the group. If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group and parsed according to Section 2.2.2. 🧩

    +========================================+========================+
    | Two groups that match the same product | Merged group           |
    | token exactly                          |                        |
    +========================================+========================+
    | user-agent: ExampleBot                 | user-agent: ExampleBot |
    | disallow: /foo                         | disallow: /foo         |
    | disallow: /bar                         | disallow: /bar         |
    |                                        | disallow: /baz         |
    | user-agent: ExampleBot                 |                        |
    | disallow: /baz                         |                        |
    +----------------------------------------+------------------------+

        Figure 2: Example of how to merge two robots.txt groups that
                        match the same product token

If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present. 🌈

        +==================================+======================+
        | Two groups that don't explicitly | Applicable group for |
        | match ExampleBot                 | ExampleBot           |
        +==================================+======================+
        | user-agent: *                    | user-agent: *        |
        | disallow: /foo                   | disallow: /foo       |
        | disallow: /bar                   | disallow: /bar       |
        |                                  |                      |
        | user-agent: BazBot               |                      |
        | disallow: /baz                   |                      |
        +----------------------------------+----------------------+

       Figure 3: Example of no matching groups other than the "*" for
                        the ExampleBot product token

And finally, if no group matches the product token, and there's no group with a user-agent line with the "*" value, or there are no groups at all, then no rules apply. 🫧

2.2.2. The "Allow" and "Disallow" Lines 💅

okay so these lines tell you whether accessing a URI that matches the corresponding path is allowed or disallowed 🦋

to check if a URI is allowed, a crawler MUST match the paths in "allow" and "disallow" rules against the URI. The matching SHOULD be case sensitive. The matching MUST start with the first octet of the path. The most specific match found MUST be used, and the most specific match is just the one with the most octets 👑 Duplicate rules in a group MAY be deduplicated. If an "allow" rule and a "disallow" rule are equivalent, then the "allow" rule SHOULD be used. If there's no match among the rules in a group for a matching user-agent, or the group has no rules at all, the URI is allowed. The /robots.txt URI is implicitly allowed 🫶

Octets in the URI and robots.txt paths outside the range of the ASCII coded character set, and those in the reserved range defined by [RFC3986], MUST be percent-encoded as defined by [RFC3986] prior to comparison.

If a percent-encoded ASCII octet shows up in the URI, it MUST be unencoded prior to comparison, unless it's a reserved character in the URI as defined by [RFC3986] or the character is outside the unreserved character range. The match only evaluates positively if the end of the path from the rule is reached before a difference in octets is encountered. Basically, you have to get all the way to the end of the rule's path with zero mismatches 🔥

here's the vibe check with some examples 🧚

   +==================+=======================+=======================+
   | Path             | Encoded Path          | Path to Match         |
   +==================+=======================+=======================+
   | /foo/bar?baz=quz | /foo/bar?baz=quz      | /foo/bar?baz=quz      |
   +------------------+-----------------------+-----------------------+
   | /foo/bar?baz=    | /foo/bar?baz=         | /foo/bar?baz=         |
   | https://foo.bar  | https%3A%2F%2Ffoo.bar | https%3A%2F%2Ffoo.bar |
   +------------------+-----------------------+-----------------------+
   | /foo/bar/        | /foo/bar/%E3%83%84    | /foo/bar/%E3%83%84    |
   | U+E38384         |                       |                       |
   +------------------+-----------------------+-----------------------+
   | /foo/            | /foo/bar/%E3%83%84    | /foo/bar/%E3%83%84    |
   | bar/%E3%83%84    |                       |                       |
   +------------------+-----------------------+-----------------------+
   | /foo/            | /foo/bar/%62%61%7A    | /foo/bar/baz          |
   | bar/%62%61%7A    |                       |                       |
   +------------------+-----------------------+-----------------------+

       Figure 4: Examples of matching percent-encoded URI components

the crawler SHOULD ignore "disallow" and "allow" rules that aren't in any group (like any rule that comes before the first user-agent line). and implementors MAY bridge encoding mismatches if they detect that the robots.txt file isn't UTF-8 encoded 🌸

2.2.3. Special Characters 🎀

crawlers MUST support these special characters, and yes they're all iconic ✨

     +===========+===================+==============================+
     | Character | Description       | Example                      |
     +===========+===================+==============================+
     | #         | Designates a line | allow: / # comment in line   |
     |           | comment.          |                              |
     |           |                   | # comment on its own line    |
     +-----------+-------------------+------------------------------+
     | $         | Designates the    | allow: /this/path/exactly$   |
     |           | end of the match  |                              |
     |           | pattern.          |                              |
     +-----------+-------------------+------------------------------+
     | *         | Designates 0 or   | allow: /this/*/exactly       |
     |           | more instances of |                              |
     |           | any character.    |                              |
     +-----------+-------------------+------------------------------+

          Figure 5: List of special characters in robots.txt files
  • # starts a line comment, whether it's at the end of a line or on its own line 💬
  • $ marks the end of the match pattern 🏁
  • * means 0 or more instances of any character 🌈

if crawlers match special characters verbatim in the URI, they SHOULD use "%" encoding. for example:

    +============================+====================================+
    | Percent-encoded Pattern    | URI                                |
    +============================+====================================+
    | /path/file-with-a-%2A.html | https://www.example.com/path/      |
    |                            | file-with-a-*.html                 |
    +----------------------------+------------------------------------+
    | /path/foo-%24              | https://www.example.com/path/foo-$ |
    +----------------------------+------------------------------------+

                   Figure 6: Example of percent-encoding

2.2.4. Other Records 🦋

crawlers MAY interpret other records that aren't part of the robots.txt protocol, like "Sitemaps" [SITEMAPS]. crawlers MAY be lenient when interpreting other records too. for example, they may accept common misspellings of the record 🙈

parsing other records MUST NOT interfere with parsing the explicitly defined records in Section 2. for example, a "Sitemaps" record MUST NOT terminate a group. no vibes getting ruined here 💅

2.3. Access Method 🔑

the rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service. the file MUST be UTF-8 encoded (as defined in [RFC3629]) and have the Internet Media Type "text/plain" (as defined in [RFC2046]) 💖

as per [RFC3986], the URI of the robots.txt file looks like this:

"scheme:[//authority]/robots.txt"

for example, with HTTP or FTP, the URIs are:

          https://www.example.com/robots.txt

          ftp://ftp.example.com/robots.txt

2.3.1. Access Results 🍓

2.3.1.1. Successful Access ✨

if the crawler successfully downloads the robots.txt file, the crawler MUST follow the parseable rules. simple and clean 🫶

2.3.1.2. Redirects 🧚

a server might respond to a robots.txt fetch request with a redirect, like HTTP 301 or HTTP 302 for HTTP. the crawlers SHOULD follow at least five consecutive redirects, even across authorities (like hosts, for HTTP) 🌈

if a robots.txt file is reached within five consecutive redirects, the robots.txt file MUST be fetched, parsed, and its rules followed in the context of the initial authority 👑

if there are more than five consecutive redirects, crawlers MAY assume that the robots.txt file is unavailable 🤷‍♀️

2.3.1.3. "Unavailable" Status 💀

"unavailable" means the crawler tries to fetch the robots.txt file and the server responds with status codes saying the resource is unavailable. for example, with HTTP, those codes are in the 400-499 range 🙈

if a server status code tells the crawler the robots.txt file is unavailable, then the crawler MAY access any resources on the server. main character energy 🦋

2.3.1.4. "Unreachable" Status 😭

if the robots.txt file is unreachable because of server or network errors, the robots.txt file is undefined and the crawler MUST assume complete disallow. for example, with HTTP, server errors show up as status codes in the 500-599 range 🚫

if the robots.txt file stays undefined for a reasonably long period of time (say, 30 days), crawlers MAY assume that it's unavailable as defined in Section 2.3.1.3, or they can keep using a cached copy 🫠

2.3.1.5. Parsing Errors 🧩

crawlers MUST try to parse each line of the robots.txt file. crawlers MUST use the parseable rules. so even if some lines are messy, the good ones still count 💅

2.4. Caching 🧊

crawlers MAY cache the contents of the fetched robots.txt file. crawlers MAY use standard cache control as defined in [RFC9111]. crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable 🕰️

2.5. Limits 🚧

crawlers SHOULD impose a parsing limit to protect their systems (see Section 3). the parsing limit MUST be at least 500 kibibytes [KiB] 🛡️

3. Security Considerations 🔐

the Robots Exclusion Protocol is not a substitute for valid content security measures. listing paths in the robots.txt file exposes them publicly, which makes the paths discoverable. so if you want to control access to the URI paths in a robots.txt file, the users of the protocol should employ a valid security measure relevant to the application layer the robots.txt file is served on. for example, with HTTP, that's HTTP Authentication as defined in [RFC9110]. robots.txt is NOT a lock 🔒💀

to protect against attacks on their systems, implementors of robots.txt parsing and matching logic should take the following considerations into account:

  • Memory management: Section 2.5 defines the lower limit of bytes that must be processed, which inherently also protects the parser from out-of-memory scenarios 🧠
  • Invalid characters: Section 2.2 defines a set of characters that parsers and matchers can expect in robots.txt files. out-of-bound characters should be rejected as invalid, which limits the available attack vectors that try to compromise the system 🚨
  • Untrusted content: implementors should treat the content of a robots.txt file as untrusted content, as defined by the specification of the application layer used. for example, with HTTP, implementors should follow the Security Considerations section of [RFC9110] 🫣

4. IANA Considerations 🌷

this document has no IANA actions. nothing to see here, bestie 💅

5. Examples 🍰

5.1. Simple Example 🎂

this example shows a few things:

  • *: a group that's relevant to all user agents that don't have an explicitly defined matching group. it allows access to URLs with the /publications/ path prefix, and it restricts access to URLs with the /example/ path prefix and to all URLs with a .gif suffix. the "*" character designates any character, including the otherwise-required forward slash (see Section 2.2) 🌟
  • foobot: a regular case. a single user agent followed by rules. the crawler only has access to two URL path prefixes on the site, /example/page.html and /example/allowed.gif. the rules of the group are missing the optional space character, which is acceptable as defined in Section 2.2 💖
  • barbot and bazbot: a group that's relevant for more than one user agent. the crawlers aren't allowed to access URLs with the /example/page.html path prefix, but otherwise they have unrestricted access to the rest of the URLs on the site 🦋
  • quxbot: an empty group at the end of the file. the crawler has unrestricted access to the URLs on the site 🧚
               User-Agent: *
               Disallow: *.gif$
               Disallow: /example/
               Allow: /publications/

               User-Agent: foobot
               Disallow:/
               Allow:/example/page.html
               Allow:/example/allowed.gif

               User-Agent: barbot
               User-Agent: bazbot
               Disallow: /example/page.html

               User-Agent: quxbot

               EOF

5.2. Longest Match 🏆

this example shows that when there are two rules, the longest one is the one used for matching. in the following case, /example/page/disallowed.gif MUST be used for the URI example.com/example/page/disallow.gif 🎯

               User-Agent: foobot
               Allow: /example/page/
               Disallow: /example/page/disallowed.gif

6. References 📚

6.1. Normative References ✨

okay bestie, these are the normative references, so you literally can't skip them 🫶 here's the whole lineup:

6.2. Informative References 🧚

these ones are the extra reading, not required, but honestly they're iconic:

Authors' Addresses 💌

the squad behind this doc, with their addresses and emails:

Martijn Koster
Stalworthy Manor Farm
Suton Lane
Wymondham, Norfolk
NR18 9JG
United Kingdom
Email: m.koster@greenhills.co.uk


Gary Illyes
Google LLC
Brandschenkestrasse 110
CH-8002 Zürich
Switzerland
Email: garyillyes@google.com


Henner Zeller
Google LLC
1600 Amphitheatre Pkwy
Mountain View, CA 94043
United States of America
Email: henner@google.com


Lizzi Sassman
Google LLC
Brandschenkestrasse 110
CH-8002 Zürich
Switzerland
Email: lizzi@google.com
📦 where this page comes from