RFC 8963
Evaluation of a Sample of RFCs Produced in 2018
RFC 8963: Evaluation of a Sample of RFCs Produced in 2018 🎀
Part 1 of 12 💖
Okay bestie, so this is RFC 8963, and it's called Evaluation of a Sample of RFCs Produced in 2018. It's an Independent Submission written by C. Huitema from Private Octopus Inc. The category is Informational, which means it's published for information and isn't a standard 💅. It came out in January 2021 and has ISSN 2070-1721. The title block doesn't say it obsoletes or updates any other RFC. 🦋
Abstract ✨
Okay so the author wanted to figure out how long it takes for an idea to get published, whether through the IETF or the Independent Stream, from the very first individual draft all the way to the RFC 😭. They looked at a random batch of RFCs approved in 2018, and also random sets from 2008 and 1998 so they could compare 2018 with the stuff from 10 or 20 years ago. The average 2018 RFC took 3 years and 4 months to produce. Of that, 2 years and 10 months were spent in the working group, 3 to 4 months went to IETF consensus and IESG review, and 3 to 4 months went to RFC production. The biggest wobble in RFC production delays comes from the AUTH48 phase 🌸.
They also counted citations of the chosen RFCs using Semantic Scholar and compared those numbers with what we know about deployment. Citation counts show academic interest, but they only loosely line up with deployment or usage of the specs. Counting web references could help with that too 🧚.
Status of This Memo 🌈
This document is not an Internet Standards Track specification. It's published for informational purposes, so no drama there ✨.
It's a contribution to the RFC Series, independent of any other RFC stream. The RFC Editor chose to publish it at their discretion and makes no statement about its value for implementation or deployment. Documents approved for publication by the RFC Editor aren't candidates for any level of Internet Standard, see Section 2 of RFC 7841 💅.
You can find information about the current status of this document, any errata, and how to give feedback on it at https://www.rfc-editor.org/info/rfc8963 🔥.
Copyright Notice 🎀
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the date this document was published. Please read these carefully, because they describe your rights and restrictions regarding this document 🫶.
Table of Contents 🧚
1. Introduction
2. Methodology
2.1. Defining the Important Milestones
2.2. Selecting a Random Sample of RFCs
2.3. Conventions Used in This Document
3. Analysis of 20 Selected RFCs
3.1. RFC 8411
3.2. RFC 8456
3.3. RFC 8446
3.4. RFC 8355
3.5. RFC 8441
3.6. RFC 8324
3.7. RFC 8377
3.8. RFC 8498
3.9. RFC 8479
3.10. RFC 8453
3.11. RFC 8429
3.12. RFC 8312
3.13. RFC 8492
3.14. RFC 8378
3.15. RFC 8361
3.16. RFC 8472
3.17. RFC 8471
3.18. RFC 8466
3.19. RFC 8362
3.20. RFC 8468
4. Analysis of Process and Delays
4.1. Delays from First Draft to RFC
4.2. Working Group Processing Time
4.3. Preparation and Publication Delays
4.4. Copy Editing
4.5. Independent Stream
5. Citation Counts
5.1. Citation Numbers
5.2. Comparison to 1998 and 2008
5.3. Citations versus Deployments
5.4. Citations versus Web References
6. Observations and Next Steps
7. Security Considerations
8. IANA Considerations
9. Informative References
Acknowledgements
Author's Address1. Introduction 💖
As the organization's website puts it, "The IETF is a large open international community of network designers, operators, vendors, and researchers concerned with the evolution of the Internet architecture and the smooth operation of the Internet." The IETF's specs get published in the RFC series, alongside documents from the IAB, IRTF, and Independent streams (as per RFC 8729) 🌸. In this memo, the author tries to understand the delays involved in publishing an idea in the IETF or through the Independent Stream, from the first individual draft to the RFC. This is an individual effort, so the conclusions are personal to the author. There was no attempt to seek IETF consensus 🙈.
The IETF keeps records of documents and process actions in the IETF Datatracker [TRKR] 📋. The Datatracker has information about RFCs and drafts, so we can work out statistics about the production system. We can measure how long it takes to take a proposal from initial draft to final publication, and how those delays split up between working group discussions, IETF reviews, IESG assessment, RFC Editor delays, and final reviews by the authors. For Independent Stream RFCs, the split is draft production, reviews by the Independent Submissions Editor, conflict reviews, RFC Editor delays, and final reviews. Tracker data is available for all RFCs, not just IETF Stream RFCs 🔥.
Just measuring production delays can be misleading 🤔. If the IETF or the other streams simply rubber-stamped draft proposals and published them, the delays would be short, but the quality and impact might suffer. We hope most of the RFCs that get published are useful, but we need a way to measure that usefulness. We try by counting the references to published RFCs in Semantic Scholar [SSCH], and by asking the authors of each RFC in the sample whether the protocols and technologies defined in the RFCs were implemented and used on the Internet 🧚. The citations Semantic Scholar measures include citations in other RFCs and in Internet-Drafts. We also count references on the web, which gives some results but would be hard to automate 💅.
To keep the resources for this study in check, the author picked 20 RFCs published in 2018 at random, as explained in Section 2.2 🎲. The random sampling picked both IETF Stream and Independent Stream documents. For comparison, the author also picked 20 RFCs published in 1998 and 20 published in 2008 at random. Limiting the sample to 20 out of 209 RFCs published in 2018 allows for in-depth analysis of each RFC, but readers should be reminded that this is a small sample. The sample is too small to apply general statistical techniques and quantify specific ratios, and discussions of correlation techniques would be inappropriate. Instead, the purpose is to spot trends, spot issues, and document future work 🦋.
The information gathered for every RFC in the sample is presented in Section 3. In Section 4, we analyze the production process and the sources of delays, comparing the 2018 sample to the selected samples for 1998 and 2018. In Section 5.1, we present citation counts for the RFCs in the samples, and analyze whether citation counts could be used to evaluate the quality of RFCs 👑.
Measuring delays could be automated by processing dates and events recorded in the Datatracker. Measuring published RFCs could be complemented by statistics on abandoned drafts, which would measure how efficient the IETF triaging process is. More instrumentation would help us understand how large delays happen during working group processes. These potential next steps are developed in Section 6 ✨.
2. Methodology 🎀
The study reported here started with a simple idea: take a sample of RFCs and do an in-depth analysis of the path from the first presentation of the idea to its publication, while also trying to assess the success of the resulting specification. This means defining the key milestones we want to track, and drawing a random sample using an unbiased process 💅.
2.1. Defining the Important Milestones 🌸
The IETF Datatracker records a list of events for each document processed by IETF working groups. It's super granular, and it's also super variable 🙃. Most documents start life as an individual draft, get adopted by a working group, go through a Working Group Last Call, get submitted to the IESG, go through an IETF Last Call and an IESG review, eventually get approved by the IESG, and get processed for publication by the RFC Editor. But there are exceptions. Some documents are first submitted to one working group and then moved to another. Some documents are published through the Independent Stream and get submitted to the Independent Submissions Editor instead of the IESG 🧚.
To make the tabulation simpler, we break the period from the submission of the first draft to the publication of the RFC into three big chunks 📦:
- The working group processing time, from the first draft to the start of the IETF last call;
- The IETF processing time, which lasts from the beginning of the IETF last call to the approval by the IESG, including the reviews by various directorates;
- The RFC production, from approval by the IESG to publication, including the AUTH48 reviews.
Independent Stream submissions don't have a working group. Instead, we look at the progression of the individual draft until the Independent Submissions Editor (ISE) adopts it, and that counts as the equivalent of the "Working Group" period. The delay from ISE adoption until submission to the RFC Editor counts as the equivalent of the IETF processing time 🦋.
We measure the starting point of the process using the date of submission of the first draft listed on that RFC's page in the IETF Datatracker. In most cases, this first draft is an individual draft that was then resubmitted as a working group draft, or maybe resubmitted with a new name while it was searching for a home in an IETF working group, or before the decision to submit on the Independent Stream was made 🌈.
The IETF Datatracker entries for RFCs and drafts do not always list working group events like Working Group Last Call. The only intermediate event we list between the first draft and the submission to the IESG is the working group adoption, for which we use the date of submission of version 00 of the draft eventually published as the RFC. We also use that date (the submission of version 00) for drafts submitted to the Independent Stream 🙈.
2.2. Selecting a Random Sample of RFCs ✨
Okay so, basic production mechanisms could be checked by digging through data from the IETF Datatracker 🔍, but subjective data needs someone to manually assess the results, and that is seriously time-consuming 😩. Our resources are limited, so we're only doing this analysis for a small sample of RFCs, picked at random from the list of RFCs approved in 2018. Specifically, we'll pick 20 RFC numbers at random between:
The list of 20 selected RFCs is: RFC 8411, RFC 8456, RFC 8446, RFC 8355, RFC 8441, RFC 8324, RFC 8377, RFC 8498, RFC 8479, RFC 8453, RFC 8429, RFC 8312, RFC 8492, RFC 8378, RFC 8361, RFC 8472, RFC 8471, RFC 8466, RFC 8362, and RFC 8468 💅
When we evaluate delays and impact, we'll compare 2018 to 2008 and 1998, which is 10 and 20 years ago 🕰️. To drive that comparison, we pick 20 RFCs at random from those published in 2008, and another 20 from those published in 1998.
The list of the 20 randomly selected RFCs from 2008 is: RFC 5227, RFC 5174, RFC 5172, RFC 5354, RFC 5195, RFC 5236, RFC 5348, RFC 5281, RFC 5186, RFC 5326, RFC 5277, RFC 5373, RFC 5404, RFC 5329, RFC 5283, RFC 5358, RFC 5142, RFC 5271, RFC 5349, and RFC 5301 🌸
The list of the 20 randomly selected RFCs from 1998 is: RFC 2431, RFC 2381, RFC 2387, RFC 2348, RFC 2391, RFC 2267, RFC 2312, RFC 2448, RFC 2374, RFC 2398, RFC 2283, RFC 2382, RFC 2289, RFC 2282, RFC 2404, RFC 2449, RFC 2317, RFC 2394, RFC 2297, and RFC 2323 🦋
2.3. Conventions Used in This Document 📝
Here are the abbreviations we use in the tables:
BCP Best Current Practice
Exp Experimental
Info Informational
PS Proposed Standard
DS Draft Standard [This maturity level was retired by RFC 6410.]Also, Status is as defined in RFC 2026, and Stream is as defined in RFC 8729.
3. Analysis of 20 Selected RFCs 🔬
We review each of the RFCs listed in Section 2.2 for the year 2018, trying both to answer the known questions and to gather insight for further analyses. In many cases, the analysis of the data is complemented by direct feedback from the RFC authors 🫶
3.1. RFC 8411
"IANA Registration for the Cryptographic Algorithm Object Identifier Range" [RFC8411]:
Status (Length): Informational (5 pages)
Overview: 4 individual drafts
First draft: 2017-05-08
Last Call start: 2017-10-09
IESG eval. start: 2017-12-28
IESG approved: 2018-02-26 (draft 03)
AUTH48 start: 2018-04-20
AUTH48 complete: 2018-07-17
Published: 2018-08-06
IANA action: create tableThis RFC was published from the individual draft, which was not resubmitted as a working group draft.
The draft underwent minor copy editing before publication ✏️
Some but not all of the long delay in AUTH48 is due to clustering with [RFC8410]. MISSREF state concluded on 2018-05-09 and the document re-entered AUTH48 at once. AUTH48 lasted over two months after that. (For state definitions, see <https://www.rfc-editor.org/about/queue/#state_def>.)
The time after AUTH48 and before publication (3 weeks) partly overlaps with travel for IETF 102, and is partly due to coordinating the cluster 🧳
3.2. RFC 8456
"Benchmarking Methodology for Software-Defined Networking (SDN) Controller Performance" [RFC8456]:
Status (Length): Informational (64 pages)
Overview: 2 individual drafts; 9 WG drafts
First draft: 2015-03-23
WG adoption: 2015-10-18
Last Call start: 2018-01-19
IESG eval. start: 2018-02-27
IESG approved: 2018-05-25
AUTH48 start: 2018-08-31
AUTH48 complete: 2018-10-16
Published: 2018-10-30The draft underwent extensive copy editing, covering use of articles, syntax, and word choice. The changes are enough to cause pagination differences. The "diff" tool marks pretty much every page as changed. Some diagrams see change in protocol elements like message names 👀
According to the author, the experience of producing this document mirrors a typical one in the Benchmarking Methodologies Working Group (BMWG). There were multiple authors in multiple time zones, which slowed down the AUTH48 process somewhat, although the AUTH48 delay of 46 days is only a bit longer than the average draft.
The RFC was part of a cluster with [RFC8455].
BMWG publishes Informational RFCs centered around benchmarking, and the methodologies in RFC 8456 have been implemented in benchmarking products 🔥
3.3. RFC 8446
"The Transport Layer Security (TLS) Protocol Version 1.3" [RFC8446], as the title indicates, defines the new version of the TLS protocol 🔐 From the IETF Datatracker, we extract the following:
Status (Length): Proposed Standard (160 pages)
Overview: 29 WG drafts
First draft: 2014-04-17
Last Call start: 2018-02-15
IESG eval. start: 2018-03-02
IESG approved: 2018-03-21 (draft 28)
AUTH48 start: 2018-06-14
AUTH48 complete: 2018-08-10
Published: 2018-08-10This draft started as a WG effort.
The RFC was a major effort in the IETF 👑 Working group participants developed and tested several implementations. Researchers analyzed the specifications and performed formal verifications. Deployment tests outlined issues that caused extra work when the specification was almost ready. This complexity largely explains the time spent in the working group.
Comparing the final draft to the published version, we find relatively light copy editing. It includes explaining acronyms on first use, clarifying some definitions, standardizing punctuation and capitalization, and spelling out some numbers in text. This generally falls in the category of "style," although some of the clarifications go into message definitions. However, that simple analysis does not explain why the AUTH48 phase took almost two months 🤔
This document's AUTH48 process was part of the "GitHub experiment," which tried to use GitHub pull requests to track the AUTH48 changes and review comments. The RFC Production Center (RPC) staff had to learn to use GitHub for that process, and this required more work than the usual RFC. The author and AD thoroughly reviewed each proposed edit, accepting some and rejecting some. The concern there was that any change in a complex specification might affect a protocol that was extensively reviewed in the working group. Of course, these reviews added time to the AUTH48 delays 😅
There are 21 implementations listed in the Wiki of the TLS 1.3 project [TLS13IMP]. It has been deployed on major browsers, and it is already used in a large fraction of TLS connections 🌈
3.4. RFC 8355
"Resiliency Use Cases in Source Packet Routing in Networking (SPRING) Networks" [RFC8355] is an Informational RFC. It originated from an informational use-case draft; it was mostly used for the BOF creating the WG, and then to drive initial work and evolutions from the WG 🧚
Status (Length): Informational (13 pages)
Overview: 2 individual drafts; 13 WG drafts
First draft: 2014-01-31
WG adoption: 2014-05-13
Last Call start: 2017-04-20
IESG eval. start: 2017-05-04 (draft 09)
IESG approved: 2017-12-19 (draft 12)
AUTH48 start: 2018-03-12
AUTH48 complete: 2018-03-27
Published: 2018-03-28Minor set of copy edits, mostly for style.
There is no implementation of the RFC itself, but the technology behind it (such as Segment Routing Architecture [RFC8402] and TI-LFA [TI-LFA]) is widely implemented, and deployment is ongoing 🚀
According to participants in the discussion, the process of adoption of the source packet routing standards was very contentious. Establishing consensus at both the working group level and the IETF level was difficult and time-consuming 😭
3.5. RFC 8441
"Bootstrapping WebSockets with HTTP/2" [RFC8441]
Status (Length): Proposed Standard (8 pages)
Overview: 3 individual drafts; 8 WG drafts; Updates RFC
6455
First draft: 2017-10-15
WG adoption: 2017-12-19
Last Call start: 2018-05-07 (draft 05)
IESG eval. start: 2018-05-29 (draft 06)
IESG approved: 2018-06-18 (draft 07)
AUTH48 start: 2018-08-13
AUTH48 complete: 2018-09-15
Published: 2018-09-18
IANA action: table entriesThis RFC defines the support of WebSockets in HTTP/2, which is different from the mechanism defined for HTTP/1.1 in [RFC6455]. The process was relatively straightforward, involving the usual type of discussions, some on details and some on important points.
Comparing the final draft and published RFC shows a minor set of copy edits, mostly for style. However, the author recalls a painful process 😩 The RFC includes many charts and graphs that were very difficult to format correctly in the author's production process, which involved conversions from markdown to XML, and then from XML to text. The author had to get substantial help from the RFC Editor 🙏
There are several implementations, including Firefox and Chrome, making RFC 8441 a very successful specification 💖
3.6. RFC 8324
"DNS Privacy, Authorization, Special Uses, Encoding, Characters, Matching, and Root Structure: Time for Another Look?" [RFC8324]. This is an opinion piece on DNS development, published on the Independent Stream 🌸
Status (Length): Informational (29 pages)
Overview: 5 individual drafts; Independent Stream
First draft: 2017-06-02
ISE review start: 2017-07-10 (draft 03)
IETF conflict review start: 2017-10-29
Approved: 2017-12-18 (draft 04)
AUTH48 start: 2018-01-29 (draft 05)
AUTH48 complete: 2018-02-26
Published: 2018-02-27This RFC took only 9 months from first draft to publication, which is the shortest in the 2018 sample set ⚡ In part, this is because the text was privately circulated and reviewed by the ISE's selected experts before the first draft was published. The nature of the document is another reason for the short delay. It is an opinion piece and does not require the same type of consensus building and reviews as a protocol specification.
Comparing the final draft and the published version shows only minor copy edits, mostly for style. According to the author, this is because he knows how to write in RFC style, with the result that his documents often need a minimum of editing. He also makes sure that the document on which the RFC Production Center starts working already has changes discussed and approved during Last Call and IESG review incorporated, rather than expecting the Production Center to operate off of notes about changes to be made. Iconic behaviour 💅
3.7. RFC 8377
"Transparent Interconnection of Lots of Links (TRILL): Multi-Topology" [RFC8377]
Status (Length): Proposed Standard (20 pages)
Overview: 3 individual drafts; 7 WG drafts; Updates RFCs
6325 and 7177
First draft: 2013-09-03
WG adoption: 2015-09-01
Last Call start: 2018-02-19 (draft 05)
IESG eval. start: 2018-03-06 (draft 05)
IESG approved: 2018-03-12 (draft 06)
AUTH48 start: 2018-04-20 (draft 06)
AUTH48 complete: 2018-07-31
Published: 2018-07-31
IANA action: table entriesMinor set of copy edits, mostly for style, also clarity ✨
3.8. RFC 8498 🎀
"A P-Served-User Header Field Parameter for an Originating Call Diversion (CDIV) Session Case in the Session Initiation Protocol (SIP)" [RFC8498]. Bestie, the title alone is a whole mouthful 💀
Status (Length): Informational (15 pages)
Overview: 5 individual drafts; 9 WG drafts
First draft: 2016-03-21
WG adoption: 2017-05-15
Last Call start: 2018-10-12 (draft 05)
IESG eval. start: 2018-11-28 (draft 07)
IESG approved: 2018-12-11 (draft 08)
AUTH48 start: 2019-01-28
AUTH48 complete: 2019-02-13
Published: 2019-02-14
IANA action: table rows added.The editing here was mostly copy edits for style 🖊️ plus some clarification of sentences that were a bit fuzzy. Very cleanup-core, no drama ✨
3.9. RFC 8479 🔐
"Storing Validation Parameters in PKCS#8" [RFC8479] 🧚
Status (Length): Informational (8 pages)
Overview: 5 individual drafts; Independent Stream
First draft: 2017-08-08
ISE review start: 2018-12-10 (draft 00)
IETF conflict review start: 2018-03-29
Approved: 2018-08-20 (draft 03)
AUTH48 start: 2018-09-20 (draft 04)
AUTH48 complete: 2018-09-25
Published: 2018-09-26So the goal here was to write down what the gnutls implementation was doing for storing provably generated RSA keys 🔑. It's a short RFC that got published pretty fast, although the author, the Independent Submissions Editor, and the IESG kept talking for several months 🙈. During the first conflict review, the IESG asked the ISE not to publish until IETF working groups had a chance to pick up the work. The author handled that by presenting to the SECDISPATCH WG during IETF 102. Nobody in a WG wanted to adopt it, so it moved forward on the Independent Stream 🦋
The copy edits were super minor, just moving some references from normative to informative 📚
Also, the author isn't aware of any implementation besides gnutls that relies on this RFC. 🥺
3.10. RFC 8453 🛰️
"Framework for Abstraction and Control of TE Networks (ACTN)" [RFC8453] 💫
Status (Length): Informational (42 pages)
Overview: 3 individual drafts; 16 WG drafts
First draft: 2015-06-15
WG adoption: 2016-07-15
Out of WG: 2018-01-26 (draft 11)
Expert review requested: 2018-02-13
Last Call start: 2018-04-16 (draft 13)
IESG eval. start: 2018-05-16 (draft 14)
IESG approved: 2018-06-01 (draft 15)
AUTH48 start: 2018-08-13
AUTH48 complete: 2018-08-20
Published: 2018-08-23
IANA action: table rows added.Minor copy editing here, nothing wild 🌸
3.11. RFC 8429 🔒
"Deprecate Triple-DES (3DES) and RC4 in Kerberos" [RFC8429] 💅
Status (Length): BCP (10 pages)
Overview: 6 WG drafts
First draft: 2017-05-01
Last Call start: 2017-07-16 (draft 03)
IESG eval. start: 2017-08-18 (draft 04)
IESG approved: 2018-05-25 (draft 05)
AUTH48 start: 2018-07-24
AUTH48 complete: 2018-10-31
Published: 2018-10-31
IANA action: table rows added.This draft started as a working group effort 🫶
The RFC recommends deprecating two encryption algorithms that are now considered obsolete and possibly broken 💀. The document went back to the WG after the first Last Call, got edited, and then had a second Last Call. The gap from first draft to Working Group Last Call looks short, but the number can be misleading 👀. The first draft replaced a similar one in the KITTEN Working Group, which stalled for a while before the CURDLE Working Group picked up the work. Deprecating RC4 was kind of contentious, but the WG had already argued it out before this draft was written, so the draft wasn't held up by that fight 🙌
Most of the 280 days between IETF LC and IESG approval happened because the IESG was debating whether this document should obsolete RFC 4757 or move it to Historic status. Nobody was really pushing that discussion for a while 😭
The 99 days in AUTH48 are mostly because one of the authors was a sitting AD, and those duties took priority over reviewing this document 🙃
Minor copy editing, for style ✨
The implementation of the draft would be the actual removal of support for 3DES and RC4 in major implementations. This is happening, but very slowly 🐢
3.12. RFC 8312 🏎️
"CUBIC for Fast Long-Distance Networks" [RFC8312] 🌈
Status (Length): Informational (18 pages)
Overview: 2 individual drafts; 8 WG drafts
First draft: 2014-09-01
WG adoption: 2015-06-08
Last Call start: 2017-09-18 (draft 06)
IESG eval. start: 2017-10-04
IESG approved: 2017-11-14 (draft 07)
AUTH48 start: 2018-01-08
AUTH48 complete: 2018-02-07
Published: 2018-02-07
IANA action: table rows added.Minor copy editing, for style 🎀
The TCP congestion control algorithm Cubic was first defined in 2005, got implemented in Linux soon after, and then showed up in major OSes after that 🐧. After some debates from 2015 to 2015 (yes, both years are the same, that's what the source says 😅), the TCPM Working Group adopted the draft with the goal of documenting Cubic in the RFC Series. According to the authors, it wasn't a high-priority effort, since Cubic was already running in multiple OSes and written up in research papers 📄. At some point only one of the authors was actively working on the draft, which may explain why another two years went by after WG adoption 🕰️
The RFC publication may or may not have triggered more implementations. On the flip side, several OSes picked up bug fixes from the draft and the RFC 🛠️
3.13. RFC 8492 🔑
"Secure Password Ciphersuites for Transport Layer Security (TLS)" [RFC8492] 🌿
Status (Length): Informational (40 pages)
Overview: 10 individual drafts; 8 WG drafts; Independent
Stream
First draft: 2012-09-07
Targeted to ISE: 2016-08-05
ISE review start: 2017-05-10 (draft 01)
IETF conflict review start: 2017-09-04
Approved: 2017-10-29 (draft 02)
AUTH48 start: 2018-10-19 (draft 05)
AUTH48 complete: 2019-02-19
Published: 2019-02-21
IANA action: table rows added.This RFC has a complicated history 🌀. The first individual draft was submitted to the TLS Working Group on September 7, 2012. It moved along there and got adopted by the WG after 3 revisions. Then it had 8 more revisions in the TLS WG, until the WG decided not to progress it. The WG chairs parked it in 2013 after it failed to reach consensus in WG Last Call. The AD finally pulled the plug in 2016, and the draft was resubmitted to the ISE 😤
At that point the author was busy and was treating the RFC as low priority because, in his words, it would not be a "real RFC" 💀. There were problems that only showed up late. Specifically, it had to wait for a change in registry policy that only came with the publication of TLS 1.3. That made the draft get published after RFC 8446, and it also meant adding references to TLS 1.3 📎. The author also got a very late comment during AUTH48 that caused some rewriting. Finally, there was an IANA issue with the extension registry, where someone else had added a similar extension. The draft was changed to just use that one 🙌
Changes made in AUTH48 include adding a reference to TLS 1.3, copy editing for style, some added requirements, added paragraphs, and changes in the algorithms specification 📝
3.14. RFC 8378 🌊
"Signal-Free Locator/ID Separation Protocol (LISP) Multicast" [RFC8378] is an Experimental RFC, defining how to implement Multicast in the LISP architecture 🧪
Status (Length): Experimental (21 pages)
Overview: 5 individual drafts; 10 WG drafts
First draft: 2014-02-28
WG adoption: 2015-12-21
Last Call start: 2018-02-13 (draft 07)
IESG eval. start: 2018-02-28 (draft 08)
IESG approved: 2018-03-12 (draft 09)
AUTH48 start: 2018-04-23
AUTH48 complete: 2018-05-02
Published: 2018-05-02Preparing this RFC took more than 4 years ⏳. According to the authors, they weren't pushing it aggressively and just let the working group process set the pace. They also did implementations during that time 🛠️
Minor copy editing, for style 💖
The RFC got implemented by lispers.net and Cisco, and it was used for IPv6 multicast over IPv4 unicast/multicast at the Olympics in PyeungChang 🏅. The plan is to work on a Proposed Standard once the experiment concludes 🚀
3.15. RFC 8361 🚦
"Transparent Interconnection of Lots of Links (TRILL): Centralized Replication for Active-Active Broadcast, Unknown Unicast, and Multicast (BUM) Traffic" [RFC8361] 💜
Status (Length): Proposed Standard (17 pages)
Overview: 3 individual drafts; 14 WG drafts
First draft: 2013-11-12
WG adoption: 2014-12-16
Last Call start: 2017-11-28 (draft 10)
IESG eval. start: 2017-12-18 (draft 11)
IESG approved: 2018-01-29 (draft 13)
AUTH48 start: 2018-03-09
AUTH48 complete: 2018-04-09
Published: 2018-04-12According to the authors, the long delays in producing this RFC came down to slow uptake of the technology in the industry 🐌
Minor copy editing, for style ✨
There was at least one partial implementation 🧩
3.16. RFC 8472 🪪
"Transport Layer Security (TLS) Extension for Token Binding Protocol Negotiation" [RFC8472] 🎀
Status (Length): Proposed Standard (8 pages)
Overview: 1 individual draft; 15 WG drafts
First draft: 2015-05-29
WG adoption: 2015-09-11
Last Call start: 2017-11-13 (draft 10)
IESG eval. start: 2018-03-19
IESG approved: 2018-07-20 (draft 14)
AUTH48 start: 2018-09-17
AUTH48 complete: 2018-09-25
Published: 2018-10-08This is a pretty simple document, but it took over 3 years to go from individual draft to RFC 😩. According to the authors, the biggest setbacks happened at the start, because it took a while to find a home for the draft. It was presented in the TLS WG (since it's a TLS extension) and the UTA WG (since it's about applications using TLS). Then the ADs decided a new WG was needed, so the authors had to work through the WG creation process, including running a BOF 📅
Minor copy editing, for style, plus the addition of a reference to TLS 1.3 📎
Perhaps partly because of the delays, some of the implementers lost interest in supporting this RFC 💔
3.17. RFC 8471 🌷
"The Token Binding Protocol Version 1.0" [RFC8471] 🌈
Status (Length): Proposed Standard (18 pages)
Overview: 1 individual draft; 19 WG drafts
First draft: 2014-10-13
WG adoption: 2015-03-15
Last Call start: 2017-11-13 (draft 16)
IESG eval. start: 2018-03-19
IESG approved: 2018-07-20 (draft 19)
AUTH48 start: 2018-09-17
AUTH48 complete: 2018-09-25
Published: 2018-10-08This document presents a Token Binding Protocol for TLS. We can notice a period of 5 months before the WG adopted the draft, and that partly explains the overall time of almost 4 years from first draft to publication ⏱️
Minor copy editing, for style 🖊️
The web references point to adoption in multiple development projects 🧑💻
3.18. RFC 8466 💖
"A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service Delivery" [RFC8466] 🎀
Status (Length): Proposed Standard (158 pages)
Overview: 5 individual drafts; 11 WG drafts
First draft: 2016-09-01
WG adoption: 2017-02-26
Last Call start: 2018-02-21 (draft 07)
IESG eval. start: 2018-03-14 (draft 08)
IESG approved: 2018-06-25 (draft 10)
AUTH48 start: 2018-09-17
AUTH48 complete: 2018-10-09
Published: 2018-10-12Okay, this one is giving "we polished it up and fixed the YANG model" energy 💅. The only changes were copy editing for style and clarity, plus corrections to the YANG model ✨.
3.19. RFC 8362 🦋
"OSPFv3 Link State Advertisement (LSA) Extensibility" [RFC8362] is a major extension to the OSPF protocol. It makes OSPFv3 fully extensible 🔥.
Status (Length): Proposed Standard (33 pages)
Overview: 4 individual drafts; 24 WG drafts
First draft: 2013-02-17
WG adoption: 2013-10-15
Last Call start: 2017-12-19 (draft 19)
IESG eval. start: 2018-01-18 (draft 20)
IESG approved: 2018-01-29 (draft 23)
AUTH48 start: 2018-03-19
AUTH48 complete: 2018-03-30
Published: 2018-04-03Bestie, the journey here was lowkey wild 😭. The spec was first submitted as an individual draft in the IPv6 WG, then moved to the OSPF WG. The long wait before this RFC came out is due to the complexity of the problem, and the need to wait for implementations ⏳.
It's a very important change to OSPF. Since it was non-backward compatible, the developers started out with some very complex migration scenarios, but they ended up with either legacy or extended OSPFv3 LSAs within an OSPFv3 routing domain 🧩. The early attempts at a hybrid mode with both legacy and extended LSAs also slowed implementation down, because it was so complex 🙈.
Copy editing for style and clarity.
This specification either was or will be implemented by all the router vendors 👑.
3.20. RFC 8468 🧚
"IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP Performance Metrics (IPPM) Framework" [RFC8468].
Status (Length): Informational (15 pages)
Overview: 3 individual drafts; 7 WG drafts
First draft: 2015-08-06
WG adoption: 2016-07-04
Last Call start: 2018-04-11 (draft 04)
IESG eval. start: 2018-05-24 (draft 05)
IESG approved: 2018-07-10 (draft 06)
AUTH48 start: 2018-09-13
AUTH48 complete: 2018-11-05
Published: 2018-11-14This one is so special 🌈. It wasn't triggered by a technical reason or by interest in the topic. It came from a formal requirement instead.
While writing RFC 7312, the IP Performance Metrics (IPPM) Working Group realized that RFC 2330, the IP Performance Metrics Framework, supported IPv4 only and explicitly excluded support for IPv6 🤯. People were still using the metrics defined on top of RFC 2330 (and therefore IPv4 only) for IPv6, too. The IPPM WG agreed the work was needed, but interest in progressing (and reading and reviewing) the IPv6 draft was limited 😮💨.
Resolving the IPv6 technical part was straightforward. Then some people asked for a broader scope (topics like header compression, 6LoWPAN, etc.), and it took time to figure out, and then convince people, that these topics were out of scope 🙃. The group also had to resolve contentious topics, for example how to measure the processing of IPv6 extension headers, which is sometimes nonstandard 💀.
The time spent in AUTH48 state for this document was longer than average 😩. According to the authors, the main reasons include:
- Workload and travel caused by busy work periods of all coauthors
- Time zone difference between coauthors and editor (at least US, Europe, and India, not considering travel)
- RFC Production Center proposed and committed some unacceptable modifications that needed to be reverted
- Lengthy discussions on a new document title (required high effort and took a long time, in particular reaching consensus between coauthors and editor was time-consuming and involved the AD)
- RFC Production Center correctly identified some nits (obsoleted personal websites of coauthors) and coauthors attempting to fix them.
The differences between the final draft and the published RFC show copy editing for style and clarity, but they don't account for the back and forth between authors and editors that the authors mentioned 🫶.
4. Analysis of Process and Delays ✨
We examine the 20 RFCs in the sample, measuring things like delay and citation counts, to try to spot patterns in the IETF processes 🔍.
4.1. Delays from First Draft to RFC 📊
We look at how long it takes between submitting the first draft and publishing the RFC. We use the three milestones defined in Section 2.1: processing time in the working group, IETF processing time, and RFC production time. The following table shows the number of days in each phase for the 20 RFCs in the sample 🍓:
+======+============+=======+=========+======+======+======+
| RFC | Status | Pages | Overall | WG | IETF | Edit |
+======+============+=======+=========+======+======+======+
| 8411 | Info | 5 | 455 | 154 | 140 | 161 |
+------+------------+-------+---------+------+------+------+
| 8456 | Info | 64 | 1317 | 1033 | 126 | 158 |
+------+------------+-------+---------+------+------+------+
| 8446 | PS | 160 | 1576 | 1400 | 34 | 142 |
+------+------------+-------+---------+------+------+------+
| 8355 | Info | 13 | 1517 | 1175 | 243 | 99 |
+------+------------+-------+---------+------+------+------+
| 8441 | PS | 8 | 327 | 204 | 31 | 92 |
+------+------------+-------+---------+------+------+------+
| 8324 | Info (ISE) | 29 | 270 | 38 | 161 | 71 |
+------+------------+-------+---------+------+------+------+
| 8377 | PS | 8 | 1792 | 1630 | 21 | 141 |
+------+------------+-------+---------+------+------+------+
| 8498 | Info | 15 | 1059 | 935 | 59 | 65 |
+------+------------+-------+---------+------+------+------+
| 8479 | Info (ISE) | 8 | 414 | 233 | 144 | 37 |
+------+------------+-------+---------+------+------+------+
| 8453 | Info | 42 | 1165 | 1036 | 46 | 83 |
+------+------------+-------+---------+------+------+------+
| 8429 | BCP | 10 | 548 | 76 | 313 | 159 |
+------+------------+-------+---------+------+------+------+
| 8312 | Info | 18 | 1214 | 1113 | 16 | 85 |
+------+------------+-------+---------+------+------+------+
| 8492 | Info (ISE) | 40 | 2358 | 1706 | 172 | 480 |
+------+------------+-------+---------+------+------+------+
| 8378 | Exp | 21 | 1524 | 1446 | 27 | 51 |
+------+------------+-------+---------+------+------+------+
| 8361 | PS | 17 | 1612 | 1477 | 62 | 73 |
+------+------------+-------+---------+------+------+------+
| 8472 | PS | 8 | 1228 | 899 | 249 | 80 |
+------+------------+-------+---------+------+------+------+
| 8471 | PS | 18 | 1228 | 899 | 249 | 80 |
+------+------------+-------+---------+------+------+------+
| 8466 | PS | 158 | 771 | 538 | 124 | 109 |
+------+------------+-------+---------+------+------+------+
| 8362 | PS | 33 | 1871 | 1766 | 41 | 64 |
+------+------------+-------+---------+------+------+------+
| 8468 | Info | 15 | 1196 | 979 | 90 | 127 |
+------+------------+-------+---------+------+------+------+
| average | 35 | 1172 | 948 | 117 | 118 |
+-------------------+-------+---------+------+------+------+
| average (not ISE) | 36 | 1200 | 999 | 110 | 104 |
+-------------------+-------+---------+------+------+------+Table 1
The average time from first draft to publication is about 3 years and 3 months, but it varies a LOT 🫠. Leaving out the Independent Stream RFCs, the average delay from start to finish is 3 years and 4 months. On average, 2 years and 9 months of that is spent getting consensus in the working group, and 3 to 4 months each goes to IETF consensus and to RFC production 🗓️.
The longest delay is for [RFC8492], at 6.5 years from start to finish. This is a very special case, though 🙈. It was a draft prepared for the TLS Working Group that failed to reach consensus. After that, it was resubmitted to the ISE and hit atypical production delays 💀.
On average, 80% of the delay is incurred in WG processing, 10% in IETF review, and 10% for edition and publication 📈.
For IETF Stream RFCs, the delays for Informational documents appear to be slightly shorter than those for protocol specifications, maybe six months shorter on average. But there are lots of differences between individual documents ✨. For protocol specifications the delays range from less than a year to more than 5 years, and for Informational documents they range from a year and 3 months to a bit more than 4 years 🌸.
We can compare the delays in the 2018 samples to those observed 10 years ago and 20 years before:
We're comparing RFC publication delays across 1998, 2008, and 2018 🫶
+============+============+=======+=======+
| RFC (2008) | Status | Pages | Delay |
+============+============+=======+=======+
| 5326 | Exp | 54 | 1584 |
+------------+------------+-------+-------+
| 5348 | PS | 58 | 823 |
+------------+------------+-------+-------+
| 5281 | Info | 51 | 1308 |
+------------+------------+-------+-------+
| 5354 | Exp | 23 | 2315 |
+------------+------------+-------+-------+
| 5227 | PS | 21 | 2434 |
+------------+------------+-------+-------+
| 5329 | PS | 12 | 1980 |
+------------+------------+-------+-------+
| 5277 | PS | 35 | 912 |
+------------+------------+-------+-------+
| 5236 | Info (ISE) | 26 | 1947 |
+------------+------------+-------+-------+
| 5358 | BCP | 7 | 884 |
+------------+------------+-------+-------+
| 5271 | Info | 22 | 1066 |
+------------+------------+-------+-------+
| 5195 | PS | 10 | 974 |
+------------+------------+-------+-------+
| 5283 | PS | 12 | 1096 |
+------------+------------+-------+-------+
| 5186 | Info | 6 | 2253 |
+------------+------------+-------+-------+
| 5142 | PS | 13 | 1005 |
+------------+------------+-------+-------+
| 5373 | PS | 24 | 1249 |
+------------+------------+-------+-------+
| 5404 | PS | 27 | 214 |
+------------+------------+-------+-------+
| 5172 | PS | 7 | 305 |
+------------+------------+-------+-------+
| 5349 | Info | 10 | 1096 |
+------------+------------+-------+-------+
| 5301 | PS | 6 | 396 |
+------------+------------+-------+-------+
| 5174 | Info | 8 | 427 |
+------------+------------+-------+-------+
Table 2Now let's peek at the 1998 crew, which is a totally different vibe 🌸
+============+============+=======+=========+
| RFC (1998) | Status | Pages | Delay |
+============+============+=======+=========+
| 2289 | PS | 25 | 396 |
+------------+------------+-------+---------+
| 2267 | Info | 10 | unknown |
+------------+------------+-------+---------+
| 2317 | BCP | 10 | 485 |
+------------+------------+-------+---------+
| 2404 | PS | 7 | 488 |
+------------+------------+-------+---------+
| 2374 | PS | 12 | 289 |
+------------+------------+-------+---------+
| 2449 | PS | 19 | 273 |
+------------+------------+-------+---------+
| 2283 | PS | 9 | 153 |
+------------+------------+-------+---------+
| 2394 | Info | 6 | 365 |
+------------+------------+-------+---------+
| 2348 | DS | 5 | 699 |
+------------+------------+-------+---------+
| 2382 | Info | 30 | 396 |
+------------+------------+-------+---------+
| 2297 | Info (ISE) | 109 | 28 |
+------------+------------+-------+---------+
| 2381 | PS | 43 | 699 |
+------------+------------+-------+---------+
| 2312 | Info | 20 | 365 |
+------------+------------+-------+---------+
| 2387 | PS | 10 | 122 |
+------------+------------+-------+---------+
| 2398 | Info | 15 | 396 |
+------------+------------+-------+---------+
| 2391 | PS | 10 | 122 |
+------------+------------+-------+---------+
| 2431 | PS | 10 | 457 |
+------------+------------+-------+---------+
| 2282 | Info | 14 | 215 |
+------------+------------+-------+---------+
| 2323 | Info (ISE) | 5 | unknown |
+------------+------------+-------+---------+
| 2448 | Info (ISE) | 7 | 92 |
+------------+------------+-------+---------+
Table 3Next up, the median and quartile numbers, which are the real tea ☕
We can compare the median delay, and the delays observed by the fastest and slowest quartiles in the three years:
+======+=============+========+=============+
| Year | Fastest 25% | Median | Slowest 25% |
+======+=============+========+=============+
| 2018 | 715 | 1221 | 1537 |
+------+-------------+--------+-------------+
| 2008 | 869 | 1081 | 1675 |
+------+-------------+--------+-------------+
| 1998 | 169 | 365 | 442 |
+------+-------------+--------+-------------+
Table 4So the IETF takes three to four times more to produce an RFC in 2018 than it did in 1998, but about the same time as it did in 2008. 😭 We can get a rough estimate of how this translates into "level of attention" per RFC by comparing the number of participants in the IETF meetings of 2018, 2008, and 1998 [IETFCOUNT] with the number of RFCs published in those years [RFCYEAR]. Here's the spread 👇
+======+=========+========+========+======+=========+============+
| Year | Number | Spring | Summer | Fall | Average | Attendees/ |
| | of RFCs | P. | P. | P. | P. | RFC |
+======+=========+========+========+======+=========+============+
| 2018 | 208 | 1235 | 1078 | 879 | 1064 | 5.1 |
+------+---------+--------+--------+------+---------+------------+
| 2008 | 290 | 1128 | 1181 | 962 | 1090 | 3.8 |
+------+---------+--------+--------+------+---------+------------+
| 1998 | 234 | 1775 | 2106 | 1705 | 1862 | 8.0 |
+------+---------+--------+--------+------+---------+------------+
Table 5The last column in the table gives the ratio of average number of participants to the number of RFCs published. If the IETF were a centralized organization, and if all participants and documents were equivalent, this ratio would be the number of participants dedicated to producing an RFC in a given year. Obviously this is a completely abstract figure, because none of those hypotheses are true 🙈 but it still gives a vague indication of the "level of attention" applied to documents. We see that this ratio has gone up from 2008 to 2018, since the number of participants was about the same for those two years but the number of published RFCs decreased. However, this ratio was much higher in 1998. The IETF had many more participants, and there were probably many more eyes available to review any given draft. If we applied the ratios of 1998, the IETF would be producing 119 documents in 2018 instead of 208. 💅✨
4.2. Working Group Processing Time 💅
Okay so the biggest chunk of the delay happens in the working groups 🫠, before the draft ever gets sent to the IESG for IETF review. As we mentioned back in Section 2.1, the only middle milestone we can pull out of the IETF Datatracker is the date the document was adopted by its working group (or targeted for independent submission). Here's how the delays break down for the documents in our sample:
+======+============+======+================+================+
| RFC | Status | WG | Until adoption | After adoption |
+======+============+======+================+================+
| 8411 | Info | 154 | 0 | 154 |
+------+------------+------+----------------+----------------+
| 8456 | Info | 1033 | 209 | 824 |
+------+------------+------+----------------+----------------+
| 8446 | PS | 1400 | 0 | 1400 |
+------+------------+------+----------------+----------------+
| 8355 | Info | 1175 | 102 | 1073 |
+------+------------+------+----------------+----------------+
| 8441 | PS | 204 | 65 | 139 |
+------+------------+------+----------------+----------------+
| 8324 | Info (ISE) | 38 | 0 | 38 |
+------+------------+------+----------------+----------------+
| 8377 | PS | 1630 | 728 | 902 |
+------+------------+------+----------------+----------------+
| 8498 | Info | 935 | 420 | 515 |
+------+------------+------+----------------+----------------+
| 8479 | Info (ISE) | 233 | 0 | 233 |
+------+------------+------+----------------+----------------+
| 8453 | Info | 1036 | 396 | 640 |
+------+------------+------+----------------+----------------+
| 8429 | BCP | 76 | 0 | 76 |
+------+------------+------+----------------+----------------+
| 8312 | Info | 1113 | 280 | 833 |
+------+------------+------+----------------+----------------+
| 8492 | Info (ISE) | 1706 | 1428 | 278 |
+------+------------+------+----------------+----------------+
| 8378 | Exp | 1446 | 661 | 785 |
+------+------------+------+----------------+----------------+
| 8361 | PS | 1477 | 399 | 1078 |
+------+------------+------+----------------+----------------+
| 8472 | PS | 899 | 105 | 794 |
+------+------------+------+----------------+----------------+
| 8471 | PS | 1127 | 153 | 794 |
+------+------------+------+----------------+----------------+
| 8466 | PS | 538 | 178 | 360 |
+------+------------+------+----------------+----------------+
| 8362 | PS | 1766 | 240 | 1526 |
+------+------------+------+----------------+----------------+
| 8468 | Info | 979 | 333 | 646 |
+------+------------+------+----------------+----------------+
| Average | 948 | 285 | 663 |
+-------------------+------+----------------+----------------+
Table 6The time before working group adoption averages a bit over 9 months 🕐, while processing after adoption averages 1 year and almost 10 months 😭. RFC 8492 totally stands out 👀, because it spent a long stretch trying to get published through a working group before it went to the Independent Submissions Editor. If we take RFC 8492 off the list, the average time until adoption drops to just over 7 months ✨, and that becomes just 25% of the total processing time in the WG.
Lowkey the big takeaway here: one outlier can totally warp an average, so RFC 8492 is doing a lot of the work in those numbers 💀
There are a few documents that started out as working group efforts right away, or were targeted for the Independent Stream from the very beginning 🦋. Those tend to move through pretty quickly 🏃♀️, with one exception: RFC 8446, which the TLS Working Group spent a long time on 🫶.
4.3. Preparation and Publication Delays ✨
okay so the preparation and publication delays have three parts, and I'm going to break them down bestie 💅
- the time from submission to the RFC Editor until AUTH48 starts, when the doc gets prepped (we're calling this "RFC edit" 🧚)
- the AUTH48 phase, where the authors look over the changes the editors suggested and eventually sign off on them (called "AUTH48" 🎀)
- the publication delay, from when the authors and editors finally agree to when it's actually published (called "RFC Pub" 🦋)
Here's the breakdown for each RFC in the table 🌸
+=======+========+=======+==========+========+=========+=========+
| RFC | Status | Pages | RFC edit | AUTH48 | RFC Pub | Edit |
| | | | | | | (total) |
+=======+========+=======+==========+========+=========+=========+
| 8411 | Info | 5 | 53 | 88 | 20 | 161 |
+-------+--------+-------+----------+--------+---------+---------+
| 8456 | Info | 64 | 98 | 46 | 14 | 158 |
+-------+--------+-------+----------+--------+---------+---------+
| 8446 | PS | 160 | 85 | 57 | 0 | 142 |
+-------+--------+-------+----------+--------+---------+---------+
| 8355 | Info | 13 | 83 | 15 | 1 | 99 |
+-------+--------+-------+----------+--------+---------+---------+
| 8441 | PS | 8 | 56 | 33 | 3 | 92 |
+-------+--------+-------+----------+--------+---------+---------+
| 8324 | Info | 29 | 42 | 28 | 1 | 71 |
| | (ISE) | | | | | |
+-------+--------+-------+----------+--------+---------+---------+
| 8377 | PS | 8 | 39 | 102 | 0 | 141 |
+-------+--------+-------+----------+--------+---------+---------+
| 8498 | Info | 15 | 48 | 16 | 1 | 65 |
+-------+--------+-------+----------+--------+---------+---------+
| 8479 | Info | 8 | 31 | 5 | 1 | 37 |
| | (ISE) | | | | | |
+-------+--------+-------+----------+--------+---------+---------+
| 8453 | Info | 42 | 73 | 7 | 3 | 83 |
+-------+--------+-------+----------+--------+---------+---------+
| 8429 | BCP | 10 | 60 | 99 | 0 | 159 |
+-------+--------+-------+----------+--------+---------+---------+
| 8312 | Info | 18 | 55 | 28 | 2 | 85 |
+-------+--------+-------+----------+--------+---------+---------+
| 8492 | Info | 40 | 355 | 123 | 2 | 480 |
| | (ISE) | | | | | |
+-------+--------+-------+----------+--------+---------+---------+
| 8378 | Exp | 21 | 42 | 9 | 0 | 51 |
+-------+--------+-------+----------+--------+---------+---------+
| 8361 | PS | 17 | 39 | 31 | 3 | 73 |
+-------+--------+-------+----------+--------+---------+---------+
| 8472 | PS | 8 | 59 | 8 | 13 | 80 |
+-------+--------+-------+----------+--------+---------+---------+
| 8471 | PS | 18 | 59 | 8 | 13 | 80 |
+-------+--------+-------+----------+--------+---------+---------+
| 8466 | PS | 158 | 84 | 22 | 3 | 109 |
+-------+--------+-------+----------+--------+---------+---------+
| 8362 | PS | 33 | 49 | 11 | 4 | 64 |
+-------+--------+-------+----------+--------+---------+---------+
| 8468 | Info | 15 | 65 | 53 | 9 | 127 |
+-------+--------+-------+----------+--------+---------+---------+
| Average | | 74 | 39 | 5 | 118 |
+----------------+-------+----------+--------+---------+---------+
| Average | | 59 | 35 | 5 | 99 |
| (without 8492) | | | | | |
+----------------+-------+----------+--------+---------+---------+
Table 7On average, the total delay is about four months 😭 but that average is skewed by the super extreme numbers for [RFC8492]. If we leave that RFC out of the math, the average drops to just a bit more than 3 months: about 2 months for preparation, a bit more than one month for AUTH48, and 5 days for publishing 🍓
Of course, the delays change from RFC to RFC. To try and explain why, we checked the correlation between the observed delays and a bunch of plausible factors:
- the number of pages in the document,
- the amount of copy editing, as discussed in Section 4.4,
- whether or not IANA actions were required,
- the number of authors,
- the number of draft revisions,
- the working group delay.
Here's what we found 👀
+===================+==========+========+=============+
| Correlation | RFC edit | AUTH48 | Edit(total) |
+===================+==========+========+=============+
| Number of pages | 0.50 | -0.04 | 0.21 |
+-------------------+----------+--------+-------------+
| Copy-Edit | 0.42 | 0.24 | 0.45 |
+-------------------+----------+--------+-------------+
| IANA | -0.14 | -0.21 | 0.12 |
+-------------------+----------+--------+-------------+
| Number of authors | 0.39 | -0.07 | 0.18 |
+-------------------+----------+--------+-------------+
| Number of drafts | 0.18 | -0.33 | -0.19 |
+-------------------+----------+--------+-------------+
| WG delay | 0.03 | -0.16 | -0.15 |
+-------------------+----------+--------+-------------+
Table 8So we've got some plausible explanations for the production delay 🧠 It's somewhat longer for longer documents, or for documents that need a lot of copy editing (see Section 4.4). Kinda surprising, but it also tends to go up with the number of authors 🙈 And it doesn't look significantly correlated with whether IANA action was needed or not.
The analysis of RFC 8324 in Section 3.6 explains its short editing delays by the experience of the author. Honestly that makes sense: if a document needs less editing, the editing delays should be shorter 💖 This is partly backed up by the link between the amount of copy editing and the publication delay.
The AUTH48 delays are way harder to explain 😩 They vary a LOT more than the preparation delay, with a standard deviation of 20 days for AUTH48 versus 10 days for preparation. In theory, AUTH48 is just a final check: the authors get the document from the RFC production center and just have to approve it, or maybe ask for a last-minute fix. The name suggests it should only take two days, but on average it takes more than a month 💀
We often guessed that the number of authors affects the AUTH48 delay, or that authors who've spent ages on the document in the working group get demotivated and take even longer to answer questions during AUTH48. Maybe that happens sometimes, but our stats don't show it. If anything, the numbers point the other way 🤷♀️
After asking the authors of the RFCs in the sample why AUTH48 took so long, we got three explanations:
- Some RFCs have multiple authors in multiple time zones. This slows down coordination when approving changes.
- Some authors found some of the proposed changes unnecessary or undesirable, and asked for them to be reversed. This needed long back-and-forths between authors and editors.
- Some authors weren't giving high priority to AUTH48 responses.
Like I said above, we weren't able to verify these theories by looking at the data 🥲 The author's experience with this document also suggests another possible delay for the Independent Stream RFC: processing delay by the Independent Submissions Editor, discussed in Section 4.5 🔥
4.4. Copy Editing ✍️
okay so we can see how much copy editing each published RFC got by comparing the draft that was approved for publication with the final RFC text 💅 we totally expect differences in the "boilerplate" and the IANA section, but we'll also spot changes from copy editing. ngl deciding how much editing happened is subjective, so we rate it on a scale of 1 to 4:
- 1: Minor editing 🌸
- 2: Editing for style, like capitalization, hyphens, "that" versus "which", and expanding all acronyms at least once ✨
- 3: Editing for clarity on top of style, like rewriting ambiguous sentences and clarifying internal references 🧚 For YANG models, that may include model corrections suggested by the verifier.
- 4: Extensive editing 😭
the table below shows that about half of the RFCs needed editing for style, and the other half got at least some editing for clarity 🍓
+======+============+===========+
| RFC | Status | Copy Edit |
+======+============+===========+
| 8411 | Info | 2 |
+------+------------+-----------+
| 8456 | Info | 4 |
+------+------------+-----------+
| 8446 | PS | 3 |
+------+------------+-----------+
| 8355 | Info | 2 |
+------+------------+-----------+
| 8441 | PS | 2 |
+------+------------+-----------+
| 8324 | Info (ISE) | 2 |
+------+------------+-----------+
| 8377 | PS | 3 |
+------+------------+-----------+
| 8498 | Info | 3 |
+------+------------+-----------+
| 8479 | Info (ISE) | 1 |
+------+------------+-----------+
| 8453 | Info | 2 |
+------+------------+-----------+
| 8429 | BCP | 2 |
+------+------------+-----------+
| 8312 | Info | 2 |
+------+------------+-----------+
| 8492 | Info (ISE) | 3 |
+------+------------+-----------+
| 8378 | Exp | 2 |
+------+------------+-----------+
| 8361 | PS | 2 |
+------+------------+-----------+
| 8472 | PS | 2 |
+------+------------+-----------+
| 8471 | PS | 2 |
+------+------------+-----------+
| 8466 | PS | 3 |
+------+------------+-----------+
| 8362 | PS | 3 |
+------+------------+-----------+
| 8468 | Info | 3 |
+------+------------+-----------+
Table 9this way of counting doesn't include the changes the editors proposed that the authors eventually turned down, since those changes aren't in either the final draft or the published RFC 👻 it might be possible to get an evaluation of these "phantom changes" from the RFC Production Center 🔮
4.5. Independent Stream 🦋
out of 20 randomly selected RFCs, 3 went through the Independent Stream. One is an independent opinion, another describes a non-IETF protocol format, and the third was [RFC8492], which is a special case. apart from that special case, the publication delays were way shorter for the Independent Stream than for the IETF Stream 🏃♀️
the authors of these 3 RFCs are regular IETF contributors 🫶 that observation led to a secondary analysis of all the RFCs published in the Independent Stream in 2018. there are 14 of them: 8507, 8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351, 8328, and 8324. (RFCs 8367 and 8369 were published on 1 April 2018 🃏) most of the documents were written by regular IETF participants, but two weren't. One describes "The BagIt File Packaging Format (V1.0)" [RFC8493], and the other the "Yeti DNS Testbed" [RFC8483]. They document a data format and a system built outside the IETF, and they show off the outreach side of the Independent Stream 🌈 in both cases, the authors included one experienced IETF participant, who presumably helped outsiders figure out the publication process 🧚
the present document had some publication delays because of the Independent Submissions Editor 🥲 The ISE is a bottleneck and a volunteer resource. Even though one volunteer person is still roughly enough resource for the job, the delivery will necessarily be best effort, with delays caused by spikes in ISE load, work commitments, and other life events 🙃 those delays may not be fundamentally critical to RFC delivery, but they can add a big percentage of delay to what might otherwise be a smooth process 💀
5. Citation Counts 📚
in this exploration, we want to find out whether citation counts give a meaningful picture of how popular RFCs are 👀 we get the citation counts from the Semantic Scholar API, using queries shaped like this: https://api.semanticscholar.org/v1/paper/10.17487/rfc8446?include_unknown_references=true
in these queries, each RFC is identified by its DOI reference, which is made of the RFC Series prefix 10.17487 plus the RFC identifier. the queries return a bunch of properties, including a list of citations for the RFC. from that list of citations, we work out three numbers:
- The total number of citations
- The number of citations in the year of publication and the year after that
- For the RFC published in 1998 or 2008 that we use for comparison, the number of citations in the years 2018 and 2019
all the numbers were retrieved on October 6, 2019 📅
5.1. Citation Numbers 🏆
as measured on October 6, 2019, the citation counts for the RFCs in our sample set were:
+============+============+=======+===========+
| RFC (2018) | Status | Total | 2018-2019 |
+============+============+=======+===========+
| 8411 | Info | 1 | 0 |
+------------+------------+-------+-----------+
| 8456 | Info | 1 | 1 |
+------------+------------+-------+-----------+
| 8446 | PS | 418 | 204 |
+------------+------------+-------+-----------+
| 8355 | Info | 3 | 3 |
+------------+------------+-------+-----------+
| 8441 | PS | 1 | 1 |
+------------+------------+-------+-----------+
| 8324 | Info (ISE) | 0 | 0 |
+------------+------------+-------+-----------+
| 8377 | PS | 0 | 0 |
+------------+------------+-------+-----------+
| 8498 | Info | 0 | 0 |
+------------+------------+-------+-----------+
| 8479 | Info (ISE) | 0 | 0 |
+------------+------------+-------+-----------+
| 8453 | Info | 3 | 3 |
+------------+------------+-------+-----------+
| 8429 | BCP | 0 | 0 |
+------------+------------+-------+-----------+
| 8312 | Info | 25 | 16 |
+------------+------------+-------+-----------+
| 8492 | Info (ISE) | 4 | 4 |
+------------+------------+-------+-----------+
| 8378 | Exp | 1 | 1 |
+------------+------------+-------+-----------+
| 8361 | PS | 0 | 0 |
+------------+------------+-------+-----------+
| 8472 | PS | 1 | 1 |
+------------+------------+-------+-----------+
| 8471 | PS | 1 | 1 |
+------------+------------+-------+-----------+
| 8466 | PS | 0 | 0 |
+------------+------------+-------+-----------+
| 8362 | PS | 1 | 1 |
+------------+------------+-------+-----------+
| 8468 | Info | 1 | 1 |
+------------+------------+-------+-----------+
Table 10the results show that [RFC8446] is by far the most cited of the 20 RFCs in our sample 🔥 not surprising, since TLS is a key Internet protocol 🔐 the TLS 1.3 protocol was also studied extensively by researchers, so it got mentioned in a bunch of published papers. Surprisingly, Semantic Scholar lists some citations that predate the publication date 🤔 these are probably citations of the various draft versions of the protocol.
the next most cited RFC in the sample is [RFC8312], which describes the Cubic congestion control algorithm for TCP ⚡ that protocol was also the target of lots of academic publications. The other RFCs in the sample only have a small number of citations.
there's probably a small bias when measuring citations at a fixed date 🕰️ an RFC published in January 2018 would have more time to collect citations than one published in December. That may be true to some extent, since the second most cited RFC in the set was published in January. However, the effect has to be limited. The most cited RFC was published in August, and the second most cited was published in 2019. (That RFC got an RFC number in 2018, but publication was slowed by long AUTH48 delays 🐌)
5.2. Comparison to 1998 and 2008 ✨
okay so now we get to the fun part: comparing against the older batches 💅 to get a baseline, we look at how many references the RFCs from 2008 and 1998 have. but bestie, we have to account for time. older docs have had way more years to pile up citations, so it's not a fair fight otherwise 🙈 the trick is to look at three counts for each doc: the total references across its whole lifetime, the references it picked up in the year after it came out, and the references we've spotted since 2018 🦋
+============+============+=======+===========+===========+
| RFC (2008) | Status | Total | 2008-2009 | 2018-2019 |
+============+============+=======+===========+===========+
| 5326 | Exp | 138 | 14 | 15 |
+------------+------------+-------+-----------+-----------+
| 5348 | PS | 14 | 3 | 0 |
+------------+------------+-------+-----------+-----------+
| 5281 | Info | 69 | 15 | 7 |
+------------+------------+-------+-----------+-----------+
| 5354 | Exp | 17 | 13 | 0 |
+------------+------------+-------+-----------+-----------+
| 5227 | PS | 19 | 1 | 2 |
+------------+------------+-------+-----------+-----------+
| 5329 | PS | 24 | 6 | 1 |
+------------+------------+-------+-----------+-----------+
| 5277 | PS | 32 | 3 | 2 |
+------------+------------+-------+-----------+-----------+
| 5236 | Info (ISE) | 25 | 5 | 4 |
+------------+------------+-------+-----------+-----------+
| 5358 | BCP | 21 | 2 | 0 |
+------------+------------+-------+-----------+-----------+
| 5271 | Info | 7 | 2 | 0 |
+------------+------------+-------+-----------+-----------+
| 5195 | PS | 7 | 4 | 2 |
+------------+------------+-------+-----------+-----------+
| 5283 | PS | 8 | 1 | 0 |
+------------+------------+-------+-----------+-----------+
| 5186 | Info | 14 | 4 | 2 |
+------------+------------+-------+-----------+-----------+
| 5142 | PS | 8 | 4 | 0 |
+------------+------------+-------+-----------+-----------+
| 5373 | PS | 5 | 2 | 0 |
+------------+------------+-------+-----------+-----------+
| 5404 | PS | 1 | 1 | 0 |
+------------+------------+-------+-----------+-----------+
| 5172 | PS | 2 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
| 5349 | Info | 8 | 0 | 2 |
+------------+------------+-------+-----------+-----------+
| 5301 | PS | 5 | 1 | 0 |
+------------+------------+-------+-----------+-----------+
| 5174 | Info | 0 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
Table 11
+============+============+=======+===========+===========+
| RFC (1998) | Status | Total | 1998-1999 | 2018-2019 |
+============+============+=======+===========+===========+
| 2289 | PS | 2 | 0 | 1 |
+------------+------------+-------+-----------+-----------+
| 2267 | Info | 982 | 5 | 61 |
+------------+------------+-------+-----------+-----------+
| 2317 | BCP | 9 | 1 | 2 |
+------------+------------+-------+-----------+-----------+
| 2404 | PS | 137 | 6 | 1 |
+------------+------------+-------+-----------+-----------+
| 2374 | PS | 42 | 4 | 0 |
+------------+------------+-------+-----------+-----------+
| 2449 | PS | 7 | 2 | 0 |
+------------+------------+-------+-----------+-----------+
| 2283 | PS | 17 | 3 | 2 |
+------------+------------+-------+-----------+-----------+
| 2394 | Info | 13 | 2 | 1 |
+------------+------------+-------+-----------+-----------+
| 2348 | DS | 5 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
| 2382 | Info | 17 | 12 | 0 |
+------------+------------+-------+-----------+-----------+
| 2297 | Info (ISE) | 36 | 11 | 0 |
+------------+------------+-------+-----------+-----------+
| 2381 | PS | 39 | 12 | 0 |
+------------+------------+-------+-----------+-----------+
| 2312 | Info | 14 | 3 | 0 |
+------------+------------+-------+-----------+-----------+
| 2387 | PS | 4 | 1 | 0 |
+------------+------------+-------+-----------+-----------+
| 2398 | Info | 17 | 0 | 1 |
+------------+------------+-------+-----------+-----------+
| 2391 | PS | 31 | 3 | 0 |
+------------+------------+-------+-----------+-----------+
| 2431 | PS | 3 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
| 2282 | Info | 8 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
| 2323 | Info (ISE) | 1 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
| 2448 | Info (ISE) | 0 | 0 | 0 |
+------------+------------+-------+-----------+-----------+
Table 12next up, we compare the median number of citations, plus the numbers for the least and most popular quartiles, across those three time windows 📊
+============================+===========+========+============+
| References | Lower 25% | Median | Higher 25% |
+============================+===========+========+============+
| RFC (2018) | 0 | 1 | 3 |
+----------------------------+-----------+--------+------------+
| RFC (2008) | 6.5 | 11 | 21.75 |
+----------------------------+-----------+--------+------------+
| RFC (2008), until 2009 | 1 | 2.5 | 4.5 |
+----------------------------+-----------+--------+------------+
| RFC (2008), 2018 and after | 0 | 0 | 2 |
+----------------------------+-----------+--------+------------+
| RFC (1998) | 4.75 | 13.5 | 32.25 |
+----------------------------+-----------+--------+------------+
| RFC (1998), until 1999 | 0 | 2 | 4.25 |
+----------------------------+-----------+--------+------------+
| RFC (1998), 2018 and after | 0 | 0 | 1 |
+----------------------------+-----------+--------+------------+
Table 13so what's the tea? ☕ the totals show that the newer docs have fewer citations than the older ones. honestly that's mostly just time doing its thing 🌸 and if we only count the citations picked up in the publication year and the year right after, the distribution across all three samples looks pretty much the same 🫶
we also see that citations to RFCs slowly fade over time. only the most popular RFCs from 1998 are still getting cited in 2019 😭 the rest have kind of gone quiet, which is a whole vibe 🧚
5.3. Citations versus Deployments ✨
okay so this table is the tea 🫖. It puts side by side the citation counts from Section 5.1 and the deployment estimates from Section 3, so you can see both at once 👀
+============+============+===========+============+
| RFC (2018) | Status | Citations | Deployment |
+============+============+===========+============+
| 8411 | Info | 1 | medium |
+------------+------------+-----------+------------+
| 8456 | Info | 1 | medium |
+------------+------------+-----------+------------+
| 8446 | PS | 418 | high |
+------------+------------+-----------+------------+
| 8355 | Info | 3 | medium |
+------------+------------+-----------+------------+
| 8441 | PS | 1 | high |
+------------+------------+-----------+------------+
| 8324 | Info (ISE) | 0 | N/A |
+------------+------------+-----------+------------+
| 8377 | PS | 0 | unknown |
+------------+------------+-----------+------------+
| 8498 | Info | 0 | unknown |
+------------+------------+-----------+------------+
| 8479 | Info (ISE) | 0 | one |
+------------+------------+-----------+------------+
| 8453 | Info | 3 | unknown |
+------------+------------+-----------+------------+
| 8429 | BCP | 0 | some |
+------------+------------+-----------+------------+
| 8312 | Info | 25 | high |
+------------+------------+-----------+------------+
| 8492 | Info (ISE) | 4 | one |
+------------+------------+-----------+------------+
| 8378 | Exp | 1 | some |
+------------+------------+-----------+------------+
| 8361 | PS | 0 | one |
+------------+------------+-----------+------------+
| 8472 | PS | 1 | medium |
+------------+------------+-----------+------------+
| 8471 | PS | 1 | medium |
+------------+------------+-----------+------------+
| 8466 | PS | 0 | unknown |
+------------+------------+-----------+------------+
| 8362 | PS | 1 | medium |
+------------+------------+-----------+------------+
| 8468 | Info | 1 | some |
+------------+------------+-----------+------------+
Table 14so from looking at these results, it's pretty obvious that citation counts cannot be used as a proxy for how much an RFC is "worth" 💅 in our sample, the two RFCs with high citation counts were both widely deployed and are definitely successful, BUT we also see tons of RFCs that got real deployment without a high citation count. no cap, that's the vibe 🌸
citation counts are driven by academic interest, but they're only loosely correlated with actual deployment 🎓 we saw that [RFC8446] got cited a lot partly because the standardization process had lots of researchers involved, and the high count for [RFC8312] is mostly because academics love evaluating congestion control protocols 🧠 if we look back at earlier years, the most cited RFC in the 2008 sample is [RFC5326], an experimental RFC defining security extensions for an experimental delay tolerant transport protocol. that protocol doesn't carry a big chunk of Internet traffic, but it's been the subject of a fair number of academic studies 📚 iconic for research, not for the wires 😭
the citation process also tends to favour the first expression of a concept 🥇 that's what we see with the most cited RFC in the 1998 set, which is [RFC2267], an informational RFC defining Network Ingress Filtering. it was obsoleted in May 2000 by [RFC2827], but it's still cited frequently in 2018 and 2019, no matter its formal status in the RFC Series 🙈 same energy with [RFC8441], which gets very few citations even though it updates [RFC6455], and that one has a large number of citations 💀 the same goes for [RFC8468], which is sparsely cited while [RFC2330] is widely cited. bottom line: just counting citations won't tell you whether developers are still using an old specification or have moved over to the revised RFC 🦋
5.4. Citations versus Web References 🌐
okay next up, web references! these might be another hint of how popular an RFC is 🔍 to check, we searched Google and Bing for the term "RFCnnnn" (for example, "RFC8411") and copied down how many results each search engine returned. the table below has the results, from searches performed on April 4, 2020 📅
+============+============+===========+========+=======+
| RFC (2018) | Status | Citations | Google | Bing |
+============+============+===========+========+=======+
| 8411 | Info | 1 | 301 | 94 |
+------------+------------+-----------+--------+-------+
| 8456 | Info | 1 | 266 | 8456 |
+------------+------------+-----------+--------+-------+
| 8446 | PS | 418 | 25900 | 47800 |
+------------+------------+-----------+--------+-------+
| 8355 | Info | 3 | 521 | 114 |
+------------+------------+-----------+--------+-------+
| 8441 | PS | 1 | 2430 | 59500 |
+------------+------------+-----------+--------+-------+
| 8324 | Info (ISE) | 0 | 393 | 138 |
+------------+------------+-----------+--------+-------+
| 8377 | PS | 0 | 264 | 10900 |
+------------+------------+-----------+--------+-------+
| 8498 | Info | 0 | 335 | 10100 |
+------------+------------+-----------+--------+-------+
| 8479 | Info (ISE) | 0 | 564 | 11000 |
+------------+------------+-----------+--------+-------+
| 8453 | Info | 3 | 817 | 11400 |
+------------+------------+-----------+--------+-------+
| 8429 | BCP | 0 | 391 | 41600 |
+------------+------------+-----------+--------+-------+
| 8312 | Info | 25 | 1620 | 2820 |
+------------+------------+-----------+--------+-------+
| 8492 | Info (ISE) | 4 | 323 | 9400 |
+------------+------------+-----------+--------+-------+
| 8378 | Exp | 1 | 418 | 11600 |
+------------+------------+-----------+--------+-------+
| 8361 | PS | 0 | 499 | 92 |
+------------+------------+-----------+--------+-------+
| 8472 | PS | 1 | 496 | 169 |
+------------+------------+-----------+--------+-------+
| 8471 | PS | 1 | 1510 | 11600 |
+------------+------------+-----------+--------+-------+
| 8466 | PS | 0 | 766 | 173 |
+------------+------------+-----------+--------+-------+
| 8362 | PS | 1 | 67 | 147 |
+------------+------------+-----------+--------+-------+
| 8468 | Info | 1 | 453 | 127 |
+------------+------------+-----------+--------+-------+
Table 15okay, the Bing numbers are kinda sus 🙃 why would RFC 8441 pull up 59,500 web references?? when we actually looked at the results, it's a mix. some are logs from dev projects implementing Web Sockets, which is exactly what we want 💖 but others are spurious. like, a shop selling rugby jerseys shows up because its phone number ends in "8441" 🏉 other pages matched because street numbers or product numbers happened to match the RFC number. the same kind of collision probably explains the big Bing counts for RFCs 8377, 8498, 8479, 8453, 8429, 8378, and 8471 😂 so Bing results just don't look like a good metric 🚫
Google is a different story 🔎 on Google, every RFC gets at least 250 references, mostly because the whole RFC catalog is copied onto a ton of web servers. the deviations from that baseline mostly track the number of citations in Semantic Scholar, with a couple of exceptions: RFC 8441 and RFC 8471 get more references than their low citation counts would predict. when we looked, we found lots of references in development databases explaining how these protocols get implemented across different code bases and open source projects 🧑💻 so counting Google results can give some sense of an RFC's popularity, and it complements the citation counts nicely 🫶
BUT there are practical problems with Google counts too 😬 Google searches are personalized, so the results depend on who's searching, and the counts may vary as well. the results also depend on the search algorithm, and nothing guarantees the counts won't change when the algorithm changes 🌀 that said, the results do show that some of the RFCs in our sample are being used by developers or in deployments 💜 which, honestly, is the whole point 🎀
6. Observations and Next Steps ✨
okay bestie, so the author's whole goal was to get a personal understanding of the "chain of production" of the RFCs 🧚, and especially to look at the different causes of delays in the process 🐢💅. As shown in Section 4, the average RFC was produced in 3 years and 4 months, which is similar to the 2008 sample 😭 but more than three times larger than the delays for the 1998 sample 💀.
The working group process appears to be the main source of delays 🔥. So efforts to diminish delays should probably focus there, instead of on the IETF and IESG reviews or the RFC production. For the RFC production phase, most of the variability originates in the AUTH48 process 🌸, which is influenced by a variety of factors such as the number of authors or the level of engagement of these authors 🫶.
Most of the delay is spent in the working group, but the IETF Datatracker doesn't hold much information about what happens inside the working groups 🙈. For example, events like Working Group Last Calls were not recorded in the history of the selected drafts available in the Datatracker. Such information would have been so interesting ✨. Of course, requiring that information would create an administrative burden, so there's clearly a trade-off between requiring more work from working group chairs and providing better data for process analysis 🎀. (It appears that this information can be available in the Datatracker for more recent drafts, if the WG chairs use the Datatracker properly 💜.)
The Independent Stream operates as expected 💖. The majority of the authors of the Independent Stream RFCs appear to be IETF insiders, but there is a significant amount of engagement by outside parties 🌈.
The analysis of citations in Section 5.1 shows that citation numbers are a very poor indication of the "value" of an RFC 💀. Citation numbers measure the engagement of academic researchers with specific topics, but have little correlation with the level of adoption and deployment of a specific RFC 🙃. The result counts of Google searches do capture references outside academia, such as logs of development projects 🦋. This might be informative, but it is not clear that the counts would not change over time due to algorithm changes or personalization 👑.
This document analyses a small sample of RFCs "in depth" 🔍. This allowed gathering of detailed feedback on the process and the deployments. On the other hand, much of the data on delays is available from the IETF Datatracker. It may be worth considering adding automated reporting of delay metrics in the IETF Datatracker 🧠.
This document only considers the RFCs that were published in a given year. This approach can be criticized as introducing a form of "survivor bias" 😬. There are many drafts proposed to the IETF, and only a fraction of them end up being published as RFCs. On one hand, this is expected, because part of the process is to triage between ideas that can gather consensus and those that don't ⚖️. On the other hand, we don't know whether that triage is too drastic and has discouraged progress on good ideas 🥺.
One way to evaluate the triage process would be to look at publication attempts that were abandoned, for example drafts that expired without progressing or being replaced 🗑️. The sampling methodology could also be used for that purpose. Pick maybe 20 drafts at random, among those abandoned in a target year, and investigate why they were abandoned. Was it because better solutions emerged in the working group? Or maybe because the authors discovered a flaw in their proposal? Or was it because some factional struggle blocked a good idea? Was the idea pursued in a different venue? Hopefully, someone will try this kind of investigation 🫶🎀.
7. Security Considerations 🔐
This document does not specify any protocol 🙌.
We might want to analyze whether security issues were discovered after publication of specific standards 👀. This one is serious, bestie, so don't skip it 🚨.
8. IANA Considerations 📋
This document has no IANA actions 🌸.
Preliminary analysis does not indicate that IANA is causing any particular delay in the RFC publication process ✨.
9. Informative References 🧾✨
okay bestie, this is where the receipts live 🧾💅 every outside doc the spec leans on is listed here, with all its details, so nobody has to play detective 🕵️♀️
[IETFCOUNT]IETF, "Past IETF Meetings",<https://www.ietf.org/how/meetings/past/>.[RFC2267]Ferguson, P. and D. Senie, "Network Ingress Filtering: Defeating Denial of Service Attacks which employ IP Source Address Spoofing", RFC 2267, DOI 10.17487/RFC2267, January 1998,<https://www.rfc-editor.org/info/rfc2267>.[RFC2330]Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, DOI 10.17487/RFC2330, May 1998,<https://www.rfc-editor.org/info/rfc2330>.[RFC2827]Ferguson, P. and D. Senie, "Network Ingress Filtering: Defeating Denial of Service Attacks which employ IP Source Address Spoofing", BCP 38, RFC 2827, DOI 10.17487/RFC2827, May 2000,<https://www.rfc-editor.org/info/rfc2827>.[RFC5326]Ramadas, M., Burleigh, S., and S. Farrell, "Licklider Transmission Protocol - Specification", RFC 5326, DOI 10.17487/RFC5326, September 2008,<https://www.rfc-editor.org/info/rfc5326>.[RFC6455]Fette, I. and A. Melnikov, "The WebSocket Protocol", RFC 6455, DOI 10.17487/RFC6455, December 2011,<https://www.rfc-editor.org/info/rfc6455>.[RFC8312]Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", RFC 8312, DOI 10.17487/RFC8312, February 2018,<https://www.rfc-editor.org/info/rfc8312>.[RFC8324]Klensin, J., "DNS Privacy, Authorization, Special Uses, Encoding, Characters, Matching, and Root Structure: Time for Another Look?", RFC 8324, DOI 10.17487/RFC8324, February 2018,<https://www.rfc-editor.org/info/rfc8324>.[RFC8355]Filsfils, C., Ed., Previdi, S., Ed., Decraene, B., and R. Shakir, "Resiliency Use Cases in Source Packet Routing in Networking (SPRING) Networks", RFC 8355, DOI 10.17487/RFC8355, March 2018,<https://www.rfc-editor.org/info/rfc8355>.[RFC8361]Hao, W., Li, Y., Durrani, M., Gupta, S., and A. Qu, "Transparent Interconnection of Lots of Links (TRILL): Centralized Replication for Active-Active Broadcast, Unknown Unicast, and Multicast (BUM) Traffic", RFC 8361, DOI 10.17487/RFC8361, April 2018,<https://www.rfc-editor.org/info/rfc8361>.[RFC8362]Lindem, A., Roy, A., Goethals, D., Reddy Vallem, V., and F. Baker, "OSPFv3 Link State Advertisement (LSA) Extensibility", RFC 8362, DOI 10.17487/RFC8362, April 2018,<https://www.rfc-editor.org/info/rfc8362>.[RFC8377]Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent Interconnection of Lots of Links (TRILL): Multi-Topology", RFC 8377, DOI 10.17487/RFC8377, July 2018,<https://www.rfc-editor.org/info/rfc8377>.[RFC8378]Moreno, V. and D. Farinacci, "Signal-Free Locator/ID Separation Protocol (LISP) Multicast", RFC 8378, DOI 10.17487/RFC8378, May 2018,<https://www.rfc-editor.org/info/rfc8378>.[RFC8402]Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, July 2018,<https://www.rfc-editor.org/info/rfc8402>.[RFC8410]Josefsson, S. and J. Schaad, "Algorithm Identifiers for Ed25519, Ed448, X25519, and X448 for Use in the Internet X.509 Public Key Infrastructure", RFC 8410, DOI 10.17487/RFC8410, August 2018,<https://www.rfc-editor.org/info/rfc8410>.[RFC8411]Schaad, J. and R. Andrews, "IANA Registration for the Cryptographic Algorithm Object Identifier Range", RFC 8411, DOI 10.17487/RFC8411, August 2018,<https://www.rfc-editor.org/info/rfc8411>.[RFC8429]Kaduk, B. and M. Short, "Deprecate Triple-DES (3DES) and RC4 in Kerberos", BCP 218, RFC 8429, DOI 10.17487/RFC8429, October 2018,<https://www.rfc-editor.org/info/rfc8429>.[RFC8441]McManus, P., "Bootstrapping WebSockets with HTTP/2", RFC 8441, DOI 10.17487/RFC8441, September 2018,<https://www.rfc-editor.org/info/rfc8441>.[RFC8446]Rescorla, E., "The Transport Layer Security (TLS) Protocol Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018,<https://www.rfc-editor.org/info/rfc8446>.[RFC8453]Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for Abstraction and Control of TE Networks (ACTN)", RFC 8453, DOI 10.17487/RFC8453, August 2018,<https://www.rfc-editor.org/info/rfc8453>.[RFC8455]Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., and S. Banks, "Terminology for Benchmarking Software-Defined Networking (SDN) Controller Performance", RFC 8455, DOI 10.17487/RFC8455, October 2018,<https://www.rfc-editor.org/info/rfc8455>.[RFC8456]Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., and S. Banks, "Benchmarking Methodology for Software-Defined Networking (SDN) Controller Performance", RFC 8456, DOI 10.17487/RFC8456, October 2018,<https://www.rfc-editor.org/info/rfc8456>.[RFC8466]Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 2018,<https://www.rfc-editor.org/info/rfc8466>.[RFC8468]Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP Performance Metrics (IPPM) Framework", RFC 8468, DOI 10.17487/RFC8468, November 2018,<https://www.rfc-editor.org/info/rfc8468>.[RFC8471]Popov, A., Ed., Nystroem, M., Balfanz, D., and J. Hodges, "The Token Binding Protocol Version 1.0", RFC 8471, DOI 10.17487/RFC8471, October 2018,<https://www.rfc-editor.org/info/rfc8471>.[RFC8472]Popov, A., Ed., Nystroem, M., and D. Balfanz, "Transport Layer Security (TLS) Extension for Token Binding Protocol Negotiation", RFC 8472, DOI 10.17487/RFC8472, October 2018,<https://www.rfc-editor.org/info/rfc8472>.[RFC8479]Mavrogiannopoulos, N., "Storing Validation Parameters in PKCS#8", RFC 8479, DOI 10.17487/RFC8479, September 2018,<https://www.rfc-editor.org/info/rfc8479>.[RFC8483]Song, L., Ed., Liu, D., Vixie, P., Kato, A., and S. Kerr, "Yeti DNS Testbed", RFC 8483, DOI 10.17487/RFC8483, October 2018,<https://www.rfc-editor.org/info/rfc8483>.[RFC8492]Harkins, D., Ed., "Secure Password Ciphersuites for Transport Layer Security (TLS)", RFC 8492, DOI 10.17487/RFC8492, February 2019,<https://www.rfc-editor.org/info/rfc8492>.[RFC8493]Kunze, J., Littman, J., Madden, E., Scancella, J., and C. Adams, "The BagIt File Packaging Format (V1.0)", RFC 8493, DOI 10.17487/RFC8493, October 2018,<https://www.rfc-editor.org/info/rfc8493>.[RFC8498]Mohali, M., "A P-Served-User Header Field Parameter for an Originating Call Diversion (CDIV) Session Case in the Session Initiation Protocol (SIP)", RFC 8498, DOI 10.17487/RFC8498, February 2019,<https://www.rfc-editor.org/info/rfc8498>.[RFCYEAR]RFC Editor, "Number of RFC Published per YEAR",<https://www.rfc-editor.org/rfcs-per-year/>.[SSCH]Allen Institute for AI, "Semantic Scholar | AI-Powered Research Tool",<https://www.semanticscholar.org/>.[TI-LFA]Litkowski, S., Bashandy, A., Filsfils, C., Decraene, B., and D. Voyer, "Topology Independent Fast Reroute using Segment Routing", Work in Progress, Internet-Draft, draft-ietf-rtgwg-segment-routing-ti-lfa-05, 15 November 2020,<https://tools.ietf.org/html/draft-ietf-rtgwg-segment-routing-ti-lfa-05>.[TLS13IMP]TLS WG, "TLS 1.3 Implementations", commit dcb7890, 14 October 2019,<https://github.com/tlswg/tlswg-wiki/blob/master/IMPLEMENTATIONS.md>.[TRKR]IETF, "IETF Datatracker",<https://datatracker.ietf.org/>.
Acknowledgements 🫶
Many thanks to the authors of the selected RFCs who were willing to provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini, Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao Weiguo, and Li Yizhou. 💖
Many thanks also to Adrian Farrel for his useful advice, to Stephen Farrell and Colin Perkins for their guidance on the use of citations, and to Dave Crocker for a comprehensive review. Thanks also to Alice Russo and the RFC Editor team for their work improving this document and checking the accuracy of the data. 🌸
Author's Address 💌
Christian Huitema
Private Octopus Inc.
427 Golfcourse Rd
Friday Harbor, WA 98250
United States of America
Email: huitema@huitema.net📦 where this page comes from
- ietf: rfc8963 2021-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms