RFC 9969

Report from the IAB Workshop on AI-CONTROL

📅 published May 2026, from ietf

Report from the IAB Workshop on AI-CONTROL 🦋

Title block, the tea: This is RFC 9969, and it's an Informational doc from the Internet Architecture Board (IAB). The authors are M. Nottingham and S. Krishnan, and it was published in May 2026. The ISSN is 2070-1721. The header doesn't have an "Obsoletes" or "Updates" line, so this doc doesn't list any RFCs it replaces or changes. 💅

Abstract ✨

The AI-CONTROL Workshop was hosted by the Internet Architecture Board (IAB) in September 2024. This report recaps the major points of discussion and flags topics that might be worth more thought and work later. 🌸

Quick heads-up: this is a report on what happened at the workshop. The views in here belong to the workshop participants and don't necessarily match the IAB's own views and positions. 🫶

Status of This Memo 📝

This doc is not an Internet Standards Track spec. It's published just for information. 🎀

It's a product of the Internet Architecture Board (IAB) and holds information the IAB thought was worth keeping on the permanent record. It represents the IAB's consensus. Documents the IAB approves for publication aren't candidates for any level of Internet Standard, so check Section 2 of RFC 7841 if you want the details. 🧚

For the current status, any errata, and how to send feedback, go to https://www.rfc-editor.org/info/rfc9969. ⭐

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the publication date. Please read those carefully, because they spell out your rights and restrictions for this doc. 📜

Table of Contents 🗂️

1.  Introduction
  1.1.  Chatham House Rule
  1.2.  Views Expressed in This Report
2.  Workshop Scope and Discussion
  2.1.  Crawl Time vs. Inference Time
    2.1.1.  Multiple Uses for Crawl Data
    2.1.2.  Application of Preferences
  2.2.  Trust
  2.3.  Attachment
    2.3.1.  robots.txt (and Similar)
    2.3.2.  Embedding
    2.3.3.  Registries
  2.4.  Vocabulary
3.  Conclusions
  3.1.  Potential Standards Work
    3.1.1.  Out of Initial Scope
4.  IANA Considerations
5.  Security Considerations
6.  Informative References
Appendix A.  About the Workshop
  A.1.  Agenda
    A.1.1.  Thursday, 2024-09-19
    A.1.2.  Friday, 2024-09-20
  A.2.  Attendees
IAB Members at the Time of Approval
Acknowledgements
Authors' Addresses

1. Introduction ✨

The IAB runs occasional workshops to look at long-term issues and strategies for the Internet, and to suggest where the Internet architecture should head next. This long-term planning is a complement to the day-to-day engineering work done by IETF working groups. 🌟

Big picture: the Internet is one of the main sources of data used to train Large Language Models (LLMs) (or, more generally, Artificial Intelligence (AI)). Most publishers putting stuff online never pictured that use, so a way to express owners' preferences about AI crawling has popped up, sometimes backed by law (for example, the European Union's AI Act [AI-ACT]). 🤖

The IAB held the AI-CONTROL Workshop on 19-20 September 2024 to "explore practical opt-out mechanisms for AI and build an understanding of use cases, requirements, and other considerations in this space" [CFP]. One big thread: people have been using the Robots Exclusion Protocol [RFC9309], also known as "robots.txt", but AI crawlers never agreed on how to handle it, so they treat it pretty differently. Also, robots.txt might not be the right tool for controlling AI crawlers at all. 🙈 The talks weren't only about robots.txt, though. Opt-out approaches other than that got a look too. 👀

To make sure a bunch of viewpoints were in the room, the program committee invited a wide mix of technical experts, AI vendors, content publishers, civil society advocates, and policymakers. 🍓

1.1. Chatham House Rule 🤫

Participants agreed to run the workshop under the Chatham House Rule [CHATHAM-HOUSE], so this report doesn't attribute statements to individuals or organizations without their express permission. Most submissions to the workshop were public, which means they're attributable, and they're used here to add substance and context. 💬

Appendix A.2 lists the workshop participants, unless they asked for that info to be kept private. 📋

1.2. Views Expressed in This Report 👀

This doc is a report on the workshop proceedings. The views and positions in it were expressed during the workshop by participants, and they don't necessarily reflect the IAB's views and positions. 🫶

Also, the content comes from presentations given by participants and from notes taken during the discussions, with no interpretation or validation applied. So the report follows the flow and back-and-forth of the workshop, but it doesn't try to capture a consensus. 🧠

2. Workshop Scope and Discussion 🔥

The workshop started by checking out the current state of AI control. 🔍

Right now, Internet publishers share their preferences about how their content gets treated for AI training using a bunch of different mechanisms. There are declarative ones, like terms of service, embedded metadata, and robots.txt [RFC9309], plus active ones, like paywalls and selectively blocking crawlers (for example, by IP address or User-Agent). 🧱

People didn't agree on what AI opt-out means overall. Research presented at the workshop [DECLINE] shows these controls are getting more common, which shrinks the amount of data available to AI (for things like training and inference-time usage). Some participants were worried about that. At least one AI vendor didn't seem too bothered, saying "there are plenty of tokens available" for training even if a lot of people opt out. Others said they need to opt out of AI training because of how they see its effects on their control over content. In their view, AI is taking over their relationships with customers and could be a threat to whole industries. 😤

Still, there was quick agreement that the current state of AI opt-out is hurting both viewpoints. As one participant put it, "no one is better off." 😭

A lot of that mess was blamed on the lack of coordination and standards for AI opt-out. Right now, content publishers have to check with each AI vendor to figure out how to opt out of training their products, because each vendor behaves very differently. On top of that, publishers have to keep watching both new vendors and policy updates from the vendors they already know about. 📆

Underneath all those immediate issues are some big constraints. They could be tied to uncertainty in the legal context, the nature of AI, and what it means to need to opt out of crawling for it. 🌀

2.1. Crawl Time vs. Inference Time ⏱️

The biggest one is the "crawl time vs. inference time" problem. Statements of preference are visible at crawl time, tied to content either by location (for example, robots.txt) or embedded inside the content as metadata. The catch is that the target of those directives is often separate from the crawler. That happens either because the crawl data isn't only used for training AI models, or because the preferences might still apply at inference time. 🧩

2.1.1. Multiple Uses for Crawl Data 🔁

A crawl's data can have multiple uses, either because the vendor has another product that uses it (for example, a search engine), or because a party other than the AI vendor does the crawling. Both patterns are super common. Plenty of Internet search engine operators also train AI models, and lots of AI models use third-party crawl data. Either way, mixing up different uses can change the incentives for publishers to cooperate with the crawler. 🎯

Well-established uses of crawling, like Internet search, were seen by participants as at least partly aligned with publishers' interests. Publishers let their sites get crawled, and in return they get more traffic and attention from being in the search index. But several participants pointed out that this give-and-take doesn't exist for AI training uses. Some see AI as hostile to publishers, since it can pull traffic away from their sites. 🙅‍♀️

So when a crawler has multiple uses that include AI, participants observed that "collateral damage" was likely for non-AI uses. That's especially true when publishers take more active control measures, like blocking or paywalls, to protect their interests. 💥

Several participants were concerned about what this does to the ecosystem. The effect could be "locking down the Web," and one person said it had implications for freedom of expression overall. 🔒

2.1.2. Application of Preferences 🧠

When data gets used to train an LLM, the resulting model can't selectively use just a portion of that data when it performs a task. Inference uses the whole model, and there's no way to point at specific input data that it used for that. 🫠

So while publishers' preferences might be available at crawl time, they generally aren't available when inference happens. Preferences stated in terms of AI use, like "no military uses" or "non-commercial only," can't be applied by a general-purpose "foundation" model. 🚫

That leaves AI vendors that want to honor those preferences with a few unappealing choices. They can leave that data out of foundation models altogether, which makes those models less viable. Or they can build a separate model for every permutation of preferences, and those models would probably multiply as the set of permutations grows. 😵‍💫

It also came up that preferences change over time, while LLMs take a long time to create and can't easily be updated to reflect those changes. One concern was how this makes an opt-out regime "stickier." Content with no associated preference, like content that predates the authors' knowledge of LLMs, is still allowed to be used for these purposes nobody saw coming. 🕰️

2.2. Trust 🤝

Participants felt that the disconnect between stating preferences and applying them adds to a lack of trust in the ecosystem. Other contributors were the usual lack of attribution for data sources in LLMs, the lack of incentive for publishers to contribute data, and finally (and this one got the most mentions) the lack of any means to monitor compliance with preferences. 🚨

That lack of trust made some participants wonder whether communicating preferences is enough in every case without a way to enforce them, or even a way to audit whether they're being followed. Some participants also said that the lack of trust was the main reason behind the growing blocking of AI crawler IP addresses, along with other measures. 🛑

2.3. Attachment 🎀

okay bestie, so the workshop spent a LOT of time on attachment 💅 which is basically how preferences get glued onto content on the Internet. Lots of mechanisms got discussed 🦋

2.3.1. robots.txt (and Similar) 🤖

The Robots Exclusion Protocol [RFC9309] is widely recognized by AI vendors as an attachment mechanism for preferences. Several deficiencies were discussed, and honestly they're kind of a lot 😭

First, it does not scale to offer granular control over large sites where authors might want to express different policies for a range of content (for example, YouTube). Imagine trying to set a different rule for every single video 💀

robots.txt is also typically under the control of the site administrator. If a site has content from many creators (as is often the case for social media and similar platforms), the administrator may not allow them to express their preferences fully, or at all. So the creators are kinda left out of their own vibes 🥺

If content is copied or moved to a different site, the preferences at the new site need to be explicitly transferred because robots.txt is a separate resource. So your preferences don't just come along for the ride 🧳

These deficiencies led many participants to feel that robots.txt cannot be the only solution to opt-out: Rather, it should be part of a larger system that addresses its shortcomings. Main character energy, but it needs a squad 👑

Participants noted that other similar attachment mechanisms have been proposed. However, none appear to have gained as much attention or implementation (both by AI vendors and content owners) as robots.txt. It's the one everybody actually uses 🌈

2.3.2. Embedding 🧚

Another mechanism for associating preferences with content is to embed them into the content itself. Many formats used on the Internet allow this; for example, HTML has the <meta> tag, images have Extensible Metadata Platform (XMP) and similar metadata sections, and XML and JSON have rich potential for extensions to carry such data. Basically the preferences live inside the file, which is giving sneaky-clever 🍓

Embedded preferences were seen to have the advantage of granularity, and of "traveling with" content as it is produced, when the content that embeds the preferences is moved from site to site or when it is stored offline. Honestly the traveling part is so iconic ✨

However, several participants pointed out that embedded preferences are easily stripped from most formats. This is a common practice for reducing the size of a file (thereby improving performance when downloading it) and for assuring privacy (since metadata often leaks information unintentionally). So the stripping isn't random, it's on purpose, and it's a whole thing 🙈

Furthermore, some types of content are not suitable for embedding. For example, it is not possible to embed preferences into purely textual content, and web pages with content from several producers (such as a social media or comment feeds) cannot easily reflect preferences for each one. Not every format can hold the vibe 😭

Participants noted that the means of embedding preferences in many formats would need to be determined by or coordinated with organizations outside the IETF. For example, HTML and many image formats are maintained by external bodies. So the IETF can't do it all alone, it needs its besties 🫶

2.3.3. Registries 📚

In some existing copyright management regimes, it is already common to have a registry of works that is consulted upon use. For example, this approach is often used for photographs, music, and video. It's an established move in those worlds 💜

Typically, registries use hashing mechanisms to create a "fingerprint" for the content that is robust to changes. Think of it like a unique little ID card for each piece of content 🪪

Using a registry decouples the content in question from its location so that it can be found even if moved. It is also claimed to be robust against stripping of embedded metadata, which is a common practice to improve performance and/or privacy. So it can find your stuff even after the metadata got yeeted ✨

However, several participants pointed out issues with deploying registries at the scale of the Internet. While they may be effective for (relatively) closed and well-known ecosystems, such as commercial music publishing, applying them to a diverse and very large ecosystem like the Internet has proven problematic. Works great in a small, tidy group chat, but the whole Internet is a LOT 💀

2.4. Vocabulary 💬

Another major focus area for the workshop was on vocabulary -- the specific semantics of the opt-out signal. Several participants noted that there are already many proposals for vocabularies, as well as many conflicting vocabularies already in use. Several examples were discussed, including where existing terms were ambiguous, did not address common use cases, or were used in conflicting ways by different actors. Y'all really said "what does this word even MEAN" 🙃

Although no conclusions regarding exact vocabulary were reached, it was generally agreed that a complex vocabulary is unlikely to succeed. Keep it simple, bestie, nobody wants a 40-page glossary 📖

3. Conclusions 👑

Participants generally agreed that on its current path, the ecosystem is not sustainable. As one remarked, "robots.txt is broken and we broke it". Oof, the honesty is kinda iconic but also a little painful 😭

Legal uncertainty, along with fundamental limitations of opt-out regimes pointed out above, limit the effectiveness of any technical solution, which will be operating in a system unlike either robots.txt (where there is a symbiotic relationship between content owners and the crawlers) or copyright (where the default is effectively opt-in, not opt-out). So the new system has to live in its own lane 🛣️

However, the workshop ended with general agreement that positive steps could be taken to improve the communication of preferences from content owners for AI use cases. In discussion, it was evident that the discovery of preferences from multiple attachment mechanisms is necessary to meet the diverse needs of content authors and, therefore, that defining how they are combined is important. Mixing and matching the signals is the whole game 🧩

We outline a proposed standard program below. Let's get into the plan ✨

3.1. Potential Standards Work 🌸

The following items were identified as good starting points for IETF work:

  • Attachment to websites by location (in robots.txt or a similar mechanism)
  • Attachment via embedding in IETF-controlled formats (e.g., HTTP headers)
  • Definition of a common core vocabulary
  • Definition of the overall regime, e.g., how to combine preferences discovered from multiple attachment mechanisms

It would be expected that the IETF would coordinate with other Standards Development Organizations (SDOs) to define embedding in other formats (e.g., HTML). Teamwork makes the dream work 🫶

3.1.1. Out of Initial Scope 🚫

It was broadly agreed that it would not be useful to work on the following items, at least to begin with:

  • Enforcement mechanisms for preferences
  • Registry-based solutions
  • Identifying or authenticating crawlers and/or content owners
  • Audit or transparency mechanisms

Not yet, bestie. One thing at a time 💅

4. IANA Considerations 🗂️

This document has no IANA actions. Nothing to register here, no cap 🧁

5. Security Considerations 🔐

This document is a workshop report and does not impact the security of the Internet. Lowkey a chill section, but still, the Internet is fine 🌈

6. Informative References 📎

[AI-ACT]   European Parliament, "Regulation (EU) 2024/1689 of the
           European Parliament and of the Council of 13 June 2024
           laying down harmonised rules on artificial intelligence
           and amending Regulations (EC) No 300/2008, (EU) No
           167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139
           and (EU) 2019/2144 and Directives 2014/90/EU, (EU)
           2016/797 and (EU) 2020/1828 (Artificial Intelligence Act)
           (Text with EEA relevance)", 13 June 2024,
           <https://eur-lex.europa.eu/eli/reg/2024/1689/oj>.

[CFP]      Internet Architecture Board, "IAB Workshop on AI-CONTROL",
           <https://datatracker.ietf.org/group/aicontrolws/about/>.

[CHATHAM-HOUSE]
           Chatham House, "Chatham House Rule",
           <https://www.chathamhouse.org/about-us/chatham-house-
           rule>.

[DECLINE]  Longpre, S., Mahari, R., Lee, A., and C. Lund, "Consent in
           Crisis: The Rapid Decline of the AI Data Commons", 2025,
           <https://www.ietf.org/slides/slides-aicontrolws-consent-
           in-crisis-the-rapid-decline-of-the-ai-data-commons-
           00.pdf>.

[PAPERS]   Internet Architecture Board, "IAB Workshop on AI-CONTROL
           Materials",
           <https://datatracker.ietf.org/group/aicontrolws/
           materials/>.

[RFC9309]  Koster, M., Illyes, G., Zeller, H., and L. Sassman,
           "Robots Exclusion Protocol", RFC 9309,
           DOI 10.17487/RFC9309, September 2022,
           <https://www.rfc-editor.org/info/rfc9309>.

Appendix A. About the Workshop 🎀

The AI-CONTROL Workshop was held on 2024-09-19 and 2024-09-20 at Wilkinson Barker Knauer in Washington, D.C., USA. Two days, one very full calendar 📅

Workshop attendees were asked to submit position papers. These papers are published on the IAB website [PAPERS], unless the submitter requested it be withheld. Receipts, bestie 📄

The workshop was conducted under the Chatham House Rule [CHATHAM-HOUSE], meaning that statements cannot be attributed to individuals or organizations without explicit authorization. So what's said in the room stays in the room, iykyk 🤫

A.1. Agenda 🗓️

This section outlines the broad areas of discussion on each day.

A.1.1. Thursday, 2024-09-19

  • Setting the stage: An overview of the current state of AI opt-out, its impact, and existing work in this space
  • Lightning talks: A variety of perspectives from participants

A.1.2. Friday, 2024-09-20

  • Opt-Out Attachment: robots.txt and beyond: Considerations in how preferences are attached to content on the Internet
  • Vocabulary: what opt-out means: What information the opt-out signal needs to convey
  • Discussion and wrap-up: Synthesis of the workshop's topics and how future work might unfold

A.2. Attendees 👯

Attendees of the workshop are listed with their primary affiliation. Attendees from the program committee (PC) and the Internet Architecture Board (IAB) are also marked. The whole squad is here ✨

  • Jari Arkko, Ericsson
  • Hirochika Asai, Preferred Networks
  • Farzaneh Badiei, Digital Medusa (PC)
  • Fabrice Canel, Microsoft (PC)
  • Lena Cohen, EFF
  • Alissa Cooper, Knight-Georgetown Institute (PC, IAB)
  • Marwan Fayed, Cloudflare
  • Christopher Flammang, Elsevier
  • Carl Gahnberg
  • Max Gendler, The News Corporation
  • Ted Hardie
  • Dominique Hazaël-Massieux, W3C
  • Gary Ilyes, Google (PC)
  • Sarah Jennings, UK Department for Science, Innovation and Technology
  • Paul Keller, Open Future
  • Elizabeth Kendall, Meta
  • Suresh Krishnan, Cisco (PC, IAB)
  • Mirja Kühlewind, Ericsson (PC, IAB)
  • Greg Leppert, Berkman Klein Center
  • Greg Lindahl, Common Crawl Foundation
  • Mike Linksvayer, GitHub
  • Fred von Lohmann, OpenAI
  • Shayne Longpre, Data Provenance Initiative
  • Don Marti, Raptive
  • Sarah McKenna, Alliance for Responsible Data Collection; Sequentum
  • Eric Null, Center for Democracy and Technology
  • Chris Needham, BBC
  • Mark Nottingham, Cloudflare (PC)
  • Paul Ohm, Georgetown Law (PC)
  • Braxton Perkins, NBC Universal
  • Chris Petrillo, Wikimedia
  • Sebastian Posth, Liccium
  • Michael Prorock
  • Matt Rogerson, Financial Times
  • Peter Santhanam, IBM
  • Jeffrey Sedlik, IPTC/PLUS
  • Rony Shalit, Alliance For Responsible Data Collection; Bright Data
  • Ian Sohl, OpenAI
  • Martin Thomson, Mozilla
  • Thom Vaughan, Common Crawl Foundation (PC)
  • Kat Walsh, Creative Commons
  • James Whymark, Meta

The following participants requested that their identity and/or affiliation not be revealed:

  • A government official

IAB Members at the Time of Approval 🌟

Internet Architecture Board members at the time this document was approved for publication were:

  • Matthew Bocci
  • Roman Danyliw
  • Dhruv Dhody
  • Jana Iyengar
  • Cullen Jennings
  • Suresh Krishnan
  • Mirja Kühlewind
  • Warren Kumari
  • Jason Livingood
  • Mark Nottingham
  • Tommy Pauly
  • Alvaro Retana
  • Qin Wu

Acknowledgements ✨

The program committee and the IAB would like to thank Wilkinson Barker Knauer for their generosity in hosting the workshop 🏢💖. Honestly, a venue that shows up for a workshop like this deserves all the flowers 🌸.

We also thank our scribes for capturing notes that assisted in the production of this report 📝🫶. These legends made sure the discussion didn't vanish into the void:

  • Zander Arnao
  • Andrea Dean
  • Patrick Yurky

Authors' Addresses 🦋

Mark Nottingham Melbourne Australia Email: mnot@mnot.net URI: https://www.mnot.net/

Suresh Krishnan Email: suresh.krishnan@gmail.com

📦 where this page comes from