RFC 6385

General Area Review Team (Gen-ART) Experiences

📅 published October 2011, from ietf

General Area Review Team (Gen-ART) Experiences 💖

okay bestie, so this is RFC 6385 🎀. It's an Informational doc, which means it's published for info and isn't a Standards Track spec 🌸. It was published in October 2011 and has ISSN 2070-1721. The authors are M. Barnes (Polycom), A. Doria (Research Consultant), H. Alvestrand (Google), and B. Carpenter (University of Auckland) 🧚. The title block doesn't list anything it obsoletes or updates.

Abstract ✨

Gen-ART has been reviewing Internet-Drafts (I-Ds) since 2004 🦋. This doc is all about what they learned over the past 7 years. At first the team reviewed I-Ds before each IESG telechat. Then, since late 2005, reviewers get assigned I-Ds during IETF Last Call, unless the I-D doesn't need one 🔥. The same reviewer then checks any updates when the I-D lands on an IESG telechat agenda 💅.

Status of This Memo 🌈

This doc is not an Internet Standards Track specification. It's published for informational purposes 💜.

It's a product of the Internet Engineering Task Force (IETF) and represents the IETF community's consensus 🫶. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 for that 🙈.

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc6385 ✨.

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved 💖.

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the publication date. Please read these carefully, because they describe your rights and restrictions for this doc 📜. Code Components pulled out of this doc must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 🔥.

Table of Contents 🧚

1. Introduction ....................................................2
2. Who Are the Gen-ART Members? ....................................3
3. Goals of Gen-ART ................................................3
4. Gen-ART Reviews .................................................4
   4.1. IETF Last Call Review Process ..............................4
   4.2. IESG Telechat Review Process ...............................5
   4.3. Form of Review .............................................5
   4.4. Gen-ART Process Overview ...................................8
5. Secretarial Process ............................................10
   5.1. Maintaining Review Spreadsheet ............................10
   5.2. Last Call Assignment Procedure ............................12
   5.3. Telechat Assignment Procedure .............................13
   5.4. Capturing Reviews .........................................14
6. Results ........................................................14
7. Impressions ....................................................15
   7.1. Reviewers' Impressions ....................................15
   7.2. General Area Directors' Impressions .......................17
   7.3. Gen-ART Secretaries' Impressions ..........................18
8. Needed Improvements ............................................18
9. Applicability ..................................................20
10. Security Considerations .......................................20
11. Acknowledgments ...............................................20
12. Informative References ........................................21

1. Introduction ✨

Gen-ART was created personally by the General Area Director in 2004 👑. This doc talks about the experiences and lessons learned as the process evolved over the past 7 years. It describes the process that was in place when this doc was published 🦋.

The process is likely to keep changing over time 🌸. Subsequent General Area Directors have kept the review team around. It has no official role in the IETF standards process, except as a group of individuals who, like everyone, can comment on Internet-Drafts (I-Ds) 💅. Its volunteers, including a secretary and the reviewers, serve at the invitation of the General AD. Both the reviews and the reviewer names are public 🫶.

2. Who Are the Gen-ART Members? 🌈

The reviewers are usually people with a fair amount of experience in various IETF Working Groups (WGs). They've authored WG I-Ds and RFCs, and they're often considered subject matter experts (SMEs) in their areas 🧠✨. The current team is made up of these technical experts, including several WG chairs and past and current Internet Architecture Board (IAB) members 🔥. Several past and current ADs have also been reviewers. Two past General ADs have served as reviewers too, and one is serving right now 👑.

Members sometimes step away from the team for various reasons, usually "day job" demands 💼. They often rejoin for periods of time when their schedules allow 🦋. Also, some reviewers stay on the team with a lighter load, getting just one I-D to review each month (at Last Call time) 🍓. Section 11 has a list of currently active reviewers, along with those who served in the past 💜.

3. Goals of Gen-ART 💅

The original and continuing goal of Gen-ART was, and still is, to take some of the IESG review burden off the General AD 🫶. The bi-weekly IESG reviews are often super heavy. Sometimes there are more than 20 I-Ds scheduled for discussion in one telechat 😭. So ADs often get less than a week's notice for many I-Ds on the telechat agenda 🙈.

Gen-ART was based on a model that worked well in the Operations (OPS) Directorate: quick review close to telechat time, to advise the AD on issues that are still serious 🔥. With a trusted group of reviewers reading and evaluating I-Ds, the General AD could focus on the I-Ds where a reviewer had raised a concern 🎀. Reviewers are expected to give feedback based on a whole set of criteria, including the ones summarised in Section 4.3 💖. The big goal is to make sure I-Ds are well structured, can be understood easily (at least at a high level), and give a reasonable basis for implementation (for I-Ds meant for the Standards Track) ✨.

Other area (and WG) directorates and review teams existed before Gen-ART, and more have been set up since 🌸. Their roles are pretty distinct, so there's little overlap between the goals and review criteria of the teams 🦋. It's also really valuable for these teams to work independently 💜. For example, when both Gen-ART reviews and Security Directorate (SecDir) reviews raise the same sorts of concerns, it's a clear red flag that the I-D needs more work before it moves on 🚩. Because the reviews are typically thorough (and objective), the sponsoring AD and document shepherd can often work with the editors/WG (and vice versa, depending on area and WG structure) to improve the overall quality of the final I-D 🫶. It should also be noted that some ADs take Gen-ART reviews into account when preparing their own evaluations 👑.

Statistics from Gen-ART reviews over the past 6+ years show a trend of increased quality and readiness for progression of I-Ds by the time they hit the telechat agenda 📈✨. More statistics are discussed in Section 6 🍓.

4. Gen-ART Reviews 🦋

4.1. IETF Last Call Review Process 🌈

The original process was only meant for reviews right before the IESG telechat, but the decision was made to include IETF Last Call (LC) reviews in early 2005 🎀. Over time, the LC reviews have proven to be quite effective 🔥. Assigning I-Ds at IETF LC time usually gives a reviewer more time to review an I-D 💖. And sometimes the IETF LC version is the one that shows up on the telechat ✨. So by the time I-Ds are added to the telechat agenda, a majority (typically at least 70%) have already been reviewed 🙌. For I-Ds that have been up-versioned, the time spent on re-review depends on the review summary for the IETF LC review 🧚.

LC assignments evolved to shrink the gap between LC announcements and assignment time, with the secretary doing LC assignments every Thursday night 🌙. This usually gives the reviewer at least one week, and sometimes two to three weeks, to finish the review 💅. Reviews are obviously most helpful when they're done on or before the end of the IETF LC 🫶.

LC assignments are done on a fairly strict round-robin basis, so the workload stays fair across all the reviewers ⚖️. Reviewers who are unavailable (vacations, etc.) during the review period are obviously left out of that round, but they keep their same queue position for the next round 🦋. The order is sometimes changed to avoid giving an editor/author or WG chair their own I-Ds 🙈. A reviewer can also NACK an assignment if they feel they may have some bias (although corporate affiliations are not considered sources of bias) or if they don't feel they can review the I-D in a timely manner 💜.

The assignment process is completely manual, although a spreadsheet makes it way easier 📊. The details are in Section 5. Ideally this could be automated 🤖. But manual intervention would still be needed to keep the available reviewer list up to date (unless reviewers took on the job of maintaining their own data in some kind of database) 🍓. More details on the tools needed to automate the whole process are in Section 8 🔥.

4.2. IESG Telechat Review Process 🌸

Here's how reviewing works when I-Ds show up on the IESG agenda 💅:

  • The "nearly final" IESG meeting agenda usually appears on Thursday night, less than one week before the IESG telechat 🌙. The Gen-ART secretary uses it as the input for the assignment process 💖.
  • For I-Ds reviewed at IETF Last Call, a new review is only requested if the I-D is revised 🫶. In that case the reviewer, usually the person who did the Last Call review, only needs to check that any open issues were resolved ✨. The draft often hasn't changed between the IETF LC and the IESG telechat review 🦋. Section 4.4 has the step-by-step telechat review assignment process, with specific details on maintaining the review assignment data 🎀. That data is kept in review spreadsheets (Section 5) 📊.

4.3. Form of Review ✨

Okay so, instead of inventing brand-new rules, the Gen-ART folks literally borrowed from "Careful Additional Review of Documents (CARD) by Senior IETF Reviewers (SIRS)" [SIRS] 💅 They adapted it for the special "late, quick review" case and for the General Area's own concerns. Iconic recycling, honestly.

Every review must start with a summary statement chosen from, or adapted from, this list:

  • This draft is ready for publication as a [type] RFC, where [type] is Informational, Experimental, etc. (In some cases, the review might recommend publication as a different [type] than requested by the author.) 🎀
  • This draft is basically ready for publication, but has nits that should be fixed before publication. ✨
  • This draft is on the right track but has open issues, described in the review. 🌸
  • This draft has serious issues, described in the review, and needs to be rethought. 😭
  • This draft has very fundamental issues, described in the review, and further work is not recommended. 💀
  • Unfortunately, I don't have the expertise to review this draft. 🙈

The length of a review can vary wildly depending on the situation. It's considered acceptable for purely editorial comments to be sent privately if it's obvious that the I-D needs substantial revision. All substantive comments, though, must be included in the public review. Wherever possible, write comments as suggestions for improvement instead of simple criticism, because nobody likes being roasted 🫶 Explicit references to prior work and prior IETF discussion should be given whenever possible.

Reviewers are asked to look for all kinds of problems: basic architectural or security issues, Internet-wide impact, technical nits, problems of form and format (like IANA Considerations or incorrect references), and editorial issues. These reviews are on I-Ds that are supposed to be finished, so the review should consider "no issue too small" 🔍 but should still cover the whole range, from the general architectural level all the way down to the editorial level.

All reviews should apply generally agreed-upon IETF criteria, such as:

  • [RFC1958]: "Architectural Principles of the Internet" 🌈
  • [RFC3426]: "General Architectural and Policy Considerations"
  • [RFC3439]: "Some Internet Architectural Guidelines and Philosophy"
  • ID-Checklist: The "ID checklist" document maintained by the IESG
  • [RFC2223bis]: "Instructions to Request for Comments (RFC) Authors" as updated by [RFC-STYLE]: "RFC Document Style"
  • [RFC5226]: BCP 26 - "Guidelines for Writing an IANA Considerations Section in RFCs"
  • [RFC3552]: BCP 72 - "Guidelines for Writing RFC Text on Security Considerations"
  • Any other applicable architectural or procedural documents. It is considered important that reviews give precise references to such criteria when relevant to a comment. 📚

Of special interest to the General Area, because they don't fall under any other area, are:

  • A clear description of why the I-D or protocol is useful to the Internet. 👑
  • Adherence to IETF formalities, such as capitalized "must", "should", etc. in normative statements, per the ID-Checklist.
  • Useful and reasonable IANA considerations. Make sure that all necessary registries are defined/referenced, and make sure there's definition and compliance with IANA assignment criteria. 🧚
  • Correct dependencies for normative references.
  • That the I-D is written in reasonably clear English. 🍓
  • Checking the updates/obsoletes information.
  • Running idnits and checking the output.
  • Checking that things imported by reference, especially from other RFCs, make sense (notably definitions of terms, security considerations, and lists of criteria) and ensuring they are used as intended by the referenced document.
  • That examples (e.g., Fully Qualified Domain Names (FQDNs), telephone numbers, IP addresses) are taken from the right spaces. 🦋

4.4. Gen-ART Process Overview 💜

Here's a general overview of the Gen-ART process, plus some basic rules for assignments. The super precise details of the secretary's process are in Section 5, so don't worry, we'll get there 🫶

  • The availability of reviewers and the order of assignments for the next round of Last Call document assignments are updated weekly, and they're available in the directory where all the assignments and reviews are cached. 📅
  • At telechat assignment time, all previously reviewed I-Ds are assigned to the reviewer who reviewed them previously, assuming that reviewer is available. Otherwise, these I-Ds get assigned to a new person through the process described below.
  • The secretary tries to avoid assigning I-Ds that might conflict with other IETF roles, such as WG chairs, other directorates, etc. However, if the secretary doesn't notice the conflict, the reviewer should notify the secretary and the Gen-ART mailing list so another reviewer can be assigned. 🙅‍♀️
  • It should be emphasized that assignment is never made according to a reviewer's technical specialty. Even though it happens, when, for example, routing I-Ds fall on routing experts or MIB documents fall on MIB doctors, it's coincidental. To the reviewer, the choice looks random. 🎲
  • There's an attempt to evenly distribute I-Ds amongst reviewers at LC time by using a round-robin process, starting from where the previous week's assignments stopped. 🔁
  • Typically, there's no attempt made to actually equalize the load, because the length and complexity of the I-Ds aren't taken into account in this process. (So, a reviewer could end up with a couple of hundred-page I-Ds, but that's statistically rare.) However, if a reviewer might receive more than one new LC I-D at one time, the secretary does try to make sure that both aren't large I-Ds. 🙏
  • Once the assignments are made, the web pages that list the reviews and the assignments are posted. Since the telechat agenda isn't published until the end of the day on the Thursdays prior to the telechats (i.e., one week prior), the secretary needs to finish the assignments that Thursday evening. This often means working later into the night, and it also requires an Internet connection even while traveling. 🌙
  • If the reviewers notice any problems or conflicts of interest, a bargaining process takes place, shifting I-Ds from one reviewer to another. The secretary updates the assignment files with any new assignments. 🤝
  • Once the review is completed, the reviewer sends it to the Gen-ART list, ideally using the template provided in the review assignment emails. Typically, reviews are also sent to authors, the responsible AD, and the WG chairs/document shepherd. The only case where this might not be done is when no issues are found for a re-review, and none had been found on an initial review. Sending the review to the authors, ADs, and/or WG chairs/Proto Shepherds was originally voluntary but is now considered standard practice. Reviewers may also send the reviews to the IETF discussion list, but that's entirely at the reviewer's discretion, and in that case the author must be copied on the review so they see any follow-up discussion. Reviewers may also send the comments to the WG; however, this typically causes the review to end up in the moderation queue, since most reviewers don't want to subscribe to the WG lists for the I-Ds they review. So it's expected that any of the original recipients (i.e., authors, WG chairs/document shepherd, or responsible AD) may forward the review to the WG mailing list if they believe it's necessary. In the past, sending these reviews caused confusion among authors, who may not have been expecting a Gen-ART review and may not be familiar with Gen-ART. So reviewers are reminded to prepend to the email a description of Gen-ART and the purpose of the review. This information is part of the standard template provided in the review assignment emails. 📧
  • The secretary gathers the reviews, sometimes edits them for format, and records each review in the spreadsheet on the web pages, including the synopsis. This is typically done on Thursday. This is one part of the process that's easy to delegate, so one volunteer can upload all the reviews and then the secretary only needs to update the fields in the spreadsheet. If the reviewer hasn't provided a synopsis ("Summary" field in the template), the secretary makes a best guess based on the review details. Note that in most cases the reviewers do include a synopsis. 🗂️
  • Ideally, the reviews should be posted to the Gen-ART mailing list by close of business on the East Coast on Tuesday. This is necessary so the General AD has time to consider the reviews before the telechat. If reviews are received after Tuesday, the AD may not read them before the IESG telechat. Due to time constraints, the spreadsheets containing review summaries and assignments are only updated on Thursday evenings, when the new LC assignments and upcoming telechat assignments are done. Ideally, the reviews would get uploaded on the Tuesdays prior to the telechat, along with the updated spreadsheets. ⏰
  • If the AD concludes that the concerns raised by the reviewer warrant placing a DISCUSS comment on the I-D, the AD will do so, and the DISCUSS must be resolved before the I-D advances. Usually the reviewer is involved in the resolution process, but the responsibility for the DISCUSS rests with the AD. 🛑

5. Secretarial Process 🎀

This section summarizes the details of managing the review materials, including the spreadsheet used to track all reviews and the HTML files containing the review assignments. Please note that these details are a snapshot of a process that has been implemented. They're very likely to change over time, especially as the needed improvements highlighted in Section 8 get carried out. 🔮

5.1. Maintaining Review Spreadsheet 📊

okay so the spreadsheet is basically the source of truth 💅 every I-D gets entered when it's assigned, and all the reviews get captured in there too. for IETF LC assignments, the assignments get finished before the I-Ds go into the spreadsheet (see Section 5.2 🎀). for telechat assignments, I-Ds only get added if there's no previous LC assignment, because obviously. everything else just gets its fields updated, see Section 5.3 🦋

every review is reachable from the spreadsheet through hyperlinks on specific fields, so you can click straight through 🔗 here's what gets tracked, in this exact order:

  1. "Chat/LC Date": the date the LC review is due, or the date of the telechat ⏰
  2. "Document": the filename for the I-D, which links to the IETF I-D tracker 🔗
  3. "Assigned": the name of the reviewer who got that I-D 🫶
  4. "Category": this one is self-explanatory, it's one of "Doc - WG", "Doc - Ind/AD", or "IETF LC". Gen-ART does not review I-Ds submitted directly to the RFC Editor, so don't even go there 🙅‍♀️ "IETF LC" gets entered for every I-D at LC time, and then it changes to one of the other values based on the telechat agenda 📋
  5. "Previous Review": this has a link to any earlier reviews. like, if a doc shows up on a telechat agenda and an IETF LC review already happened, this field gets updated with the LC review summary (so the stuff from "Current Review Summary", which is explained below, gets copied over 💜). the field is set to "New" when an I-D first gets assigned or added. for returns, it says "Return" or "Return/IETF-LC" for I-Ds that had an LC review. it's worth noting that since Gen-ART started doing reviews at LC time, there seem to be way fewer returns on the agenda 🌈
  6. "Current Review Summary": this includes the review type and version number of the doc that's to be reviewed or has been reviewed (for example, LC: -02). when the field also has a review summary after the type and version (like Telechat: -06 Ready), the active hyperlink points to the review 🔥 sometimes a reviewer re-reviews an I-D before its telechat assignment. in that case it gets added to the spreadsheet, but the date doesn't change, so the date column stays consistent, since the reviews themselves have the review date 📅

here's the steps for adding a new I-D to the spreadsheet, bestie 🍓

  1. first, blank rows get inserted for the number of new I-Ds being added, so you optimize your steps ✨
  2. a row with default fields (hyperlinks included) lives at the end of the file, to keep data entry minimal 🧚 there's a separate default row for IETF LC assignments and for telechat assignments. that row gets copied into each new blank row, and then the dates get entered (this also lets you double-check that every I-D from the review assignments is accounted for, especially LC 👀)
  3. the I-D name gets copied into the name field and also appended to the hyperlink for the "Review Summary" field. the hyperlink is part of the default row, so entering reviews later is way less work 💅
  4. the data gets sorted by "Chat/LC Date", "Assigned", and "Document". then the file gets saved and closed 💾
  5. the file gets reopened and saved as HTML 🌸
  6. the file gets opened a second time and sorted by "Assigned", "Chat/LC Date", and "Document". this gives the I-D reviewers an easy way to find any outstanding assignments 🔍 iconic behavior honestly

5.2. Last Call Assignment Procedure 📣

the secretary can cache the Last Call assignments as they get announced, and/or check the IETF announcement mailing list archives 📬 assignments happen on Thursday evening, along with any telechat assignments. this batches the file changes, which is so efficient it's giving main character energy ⭐

the assignments live in an HTML file. here's how that file gets made, step by step 🎀

  1. the order of assignment is actually created the week before, using the details below. so before starting the new assignments, the current file gets saved for editing for the following week 🗂️ the current file-naming convention is reviewersyymmdd-lc.html. for example, for July 8th, 2010, the file reviewers100708-lc.html was created, and the file for the following week is named reviewers100715-lc.html 🧚
  2. since the file is already prepared with the right ordering of reviewers, the assignments happen in the order of due dates. the link to the I-D in the Datatracker gets copied into the assignment file, along with the intended RFC status for each new LC I-D 🔗
  3. the "Due Date" paragraph from the Last Call announcement gets shortened to IETF LC ends on:, and the date stays 📅
  4. once the assignment file is complete, the new I-Ds get added to the spreadsheet, like we covered above 💖
  5. the assignment file for the next week then gets updated to reflect the next reviewer in the round-robin. you just cut and paste the names in the list as a block, and remove any "one doc per month" reviewers (the ones marked with an *) who already got their monthly assignment 🙅‍♀️ if the next round of assignments happens at the start of a new month, the "one doc per month" reviewers get added back into the list, in the normal order (alphabetically by first name) 🌈
  6. the assignment files and updated spreadsheets then get cached on the Gen-ART server 🖥️
  7. an email with a link to the assignment file and the updated spreadsheets gets sent to the Gen-ART mailing list 💌 it has a standard form, so reviewers can just cut and paste the template, which includes the Gen-ART context statement and a link to the FAQ 🫶

5.3. Telechat Assignment Procedure 🦋

since LC assignments are now the starting point for Gen-ART I-D reviews, telechat assignments are generally pretty straightforward, because most I-Ds are already in the spreadsheet 😌 here's the play-by-play:

  1. the telechat agenda is usually available around 6PM PDT 🕕 to create the assignment HTML file, the agenda gets built from the email announcing the upcoming telechat agenda. the filename looks like reviewersyymmdd.html, with the date matching the telechat date (not the assignment date, which is the case for Last Call assignments) 📁
  2. rows get added to the agenda for the reviewers' names 🧾
  3. the reviewers' names then get added to the weekly assignment file ✏️
  4. as each reviewer gets added to the assignment file, the review spreadsheet gets updated like this 💜
    • "Chat/LC Date" changes to the telechat date 📅
    • the link to the LC review, if there is one, gets copied as the link for the "Previous Review" column 🔗
    • the text for the "Current Review" gets updated to reflect the new review type (Telechat) and version 💅
  5. for an I-D that didn't go through IETF LC, a reviewer gets assigned using the order in the file that's used for Last Call assignments for the next week 🔄
  6. once the reviewer(s) are figured out, the LC assignment file for the next week gets updated 🗓️
  7. any new I-Ds get added to the spreadsheet (and the updates saved) following the steps in Section 5.1 🌸
  8. the assignment files and updated spreadsheets get cached on the Gen-ART server 🖥️
  9. an email with a link to the assignment file and the updated spreadsheets gets sent to the Gen-ART mailing list 📧 it's the same standard form, so reviewers can cut and paste the template with the Gen-ART context statement and FAQ link ✨

5.4. Capturing Reviews 📥

as noted in Section 4.4, the spreadsheet usually gets updated with the review summaries on Thursday evenings, right before the data for that week's LC and telechat assignments gets entered 🗓️ here's how the reviews get captured:

  1. right now, an extra volunteer is helping the secretary cache the email reviews as they come in 🙌
  2. when the review is inline in the body of the email, it gets cut and pasted into a text file, saved with the reviewer's last name appended to the filename. for example, draft-ietf-xyz-00-smith.txt 📝
  3. when the review is an attachment to the email, the file can be saved and uploaded directly 📎
  4. the volunteer uploads the reviews by around 5PM CST on Thursdays. that way they're ready for the secretary when that week's assignments are done. this order matters, because it keeps the info for I-Ds on the upcoming telechat up to date ⏰
  5. the review summary gets entered into the "Current Summary" field. the hyperlink to the review (added at assignment time) will automatically work once the file is uploaded 🔗✨
  6. once all the reviews are entered and the spreadsheets are formatted, the review spreadsheet gets saved and the files get uploaded, following the last three steps in Section 5.1 💾

6. Results 🏆

okay the numbers are honestly incredible 😭 over the past 7 years, Gen-ART has provided reviewing services to 3 ADs and has done around two thousand publicly available reviews 🎉 the reviews have been done by a team of around a dozen full-time reviewers, plus other reviewers who each get one I-D assignment a month. there are currently 9 reviewers in that second group. the full-time reviewers get 2-3 assignments each month 💪

for quality, here's the big win 🌟 since reviews now start at LC time, the number of I-Ds that are "Ready" at the telechat has gone up. in this doc, "Ready" means the reviewer believes the document has no outstanding editorial or technical issues ✅ based on the 2007 data, over 250 I-Ds were assigned at LC time and went through IESG review. of those 250, 82% of the LC reviews (205 I-Ds) were completed 📈 of the completed reviews, 70% (144 I-Ds) were "Ready" at the telechat. of those 144, roughly 1/4 had been deemed "Ready" (with no nits) at LC time, based on a sample of 50 reviews 👀 for the I-Ds that weren't reviewed at LC time, only about 1/4 of them were deemed "Ready" when they were reviewed for the telechat. so yeah, doing Gen-ART reviews at Last Call time really does seem to improve the I-Ds for the telechat 🫶💖

7. Impressions 🔮

this section is split into 3 subsections: the impressions gathered from the Gen-ART, the impressions of the ADs they worked for, and the impressions of the Gen-ART secretaries 🧚✨

7.1. Reviewers' Impressions 🦋

okay so this bit is the reviewers spilling their tea 🫖. Gen-ART asked each reviewer for a few lines on their personal experience and lessons learned, and here's what they said:

"We'd like to ask you each to write a few lines about your personal experience and lessons learned as a Gen-ART reviewer".
  • We really do find problems, but we don't find problems with most I-Ds. 🔍✨
  • Comments seem to be in three areas: editorial/grammar, editorial/what-the-heck-does-this-mean, and actual problems. I'm seeing fewer reviews in the first category, which is a good thing 🎀
  • It is becoming rarer that we hear back "these guys have suffered enough; I'm voting no objection" 😭 (I'm remembering an LDAP I-D that had been around so long it had 2119 referenced AS A DRAFT -- some people suffered a lot 💀)
  • The direct assignment of reviews is necessary and effective. It does not matter much as far as I can tell what scheme is used to actually do the assignment 💅
  • Folks are very open to the reviews that come out of Gen-ART. This somewhat surprised me, because I have seen resistance to outside reviews in other cases 🌸
  • The improvements that have come about (for example, one of my latest, an I-D about the SIPPING conference) have made a big difference to the comprehensibility and usability of the I-Ds -- and provide a useful incentive to keep going ⭐
  • Some form of review like this is desperately needed. While most of the stuff we see is good, every once in a while really bad errors have made their way all the way to IESG vote 😱
  • Reading this stuff is interesting. I like having a reason to read a wide range of materials 📚🧚
  • I am more than convinced that this can be, and is, a valuable process. It is, in my opinion, a pity that Senior IETF Reviewers (SIRS) and so on did not take off, because this late-stage reviewing is a poor substitute for doing the same thing at a much earlier stage. Very few of the drafts that have come past my screen are truly fully ready for IESG review. It is actually a joy to find the occasional nugget that is both well written and is a proper technical job, such that the reviewer really can say "This is ready" 🥹🫶
  • I have certainly found the process intellectually stimulating! It encourages me to take a wider interest in what is going on in the IETF, but consumes a fair bit of time to do a proper job, and requires a very wide knowledge to be able to properly catch the cross-area implications: I hope (believe!) that this is something that one gets better at with experience and doing a few of these reviews 🌈
  • There is probably a very limited pool of people who have both the time and the inclination to keep on doing these reviews. It does require a fair bit of dedication 🫠
  • It is difficult to avoid correcting the English, even if that is not really the point: Often, really bad English (whether as a result of non-mother-tongue authors with limited grasp or mother-tongue authors using informal language) obscures/corrupts what is being said or just makes it impossible to read 🙈
  • Mostly authors welcome the comments: I think most of them understand the concept of "ego-free reviewing", and we have generally been constructive rather than destructive 💖
  • Part of the job of Gen-ART is to think the unthinkable from another point of view, to challenge (apparently undocumented) assumptions, and apply experience from other fields 🧠👑

7.2. General Area Directors' Impressions 👑

bestie, this part is from multiple General Area Directors, so the "I"s aren't necessarily the same AD. Just vibes from a few different people 💅

It should be noted that these impressions are from multiple General Area Directors; thus, the "I"s are not necessarily associated with a specific AD.

  • It's essential. The reviewing load for the IESG DOES NOT SCALE 😤🔥
  • Without Gen-ART, I would be a much less effective General AD. 🥺🫶
  • On a single fortnight example, the IESG had 21 drafts on the agenda. It is just impossible (to conscientiously review all the documents), and no wonder we sometimes miss serious issues 💀
  • So I think a distributed review team with about 30 trusted reviewers needs to be institutionalized. I suspect that will need to be formalized in a BCP sooner or later -- with their reviews having a formal position in the standards process, and the expectation that the whole IESG truly reviews all I-Ds being relaxed 🦋
  • We've learned that polite, well reasoned, constructive reviews are very positively received by authors and WGs. Dismissive reviews are counter-productive. And reviews sent in private eventually show up in public, so it's better to go public at the start 🌷
  • Normally, LC reviews are available in good time for the draft to be revised before reaching the IESG agenda. It is important that this happens, except for an emergency situation where the responsible AD has good reason to place the draft on the agenda immediately. In that case, it would be preferable for the AD to inform the Gen-ART, so that the review can be expedited ⏰
  • The other problem is a big detail -- between late Thursday or early Friday when the secretary sends out the assignments, and Wednesday when the General AD likes to start filling in ballots based on the reviews received by close of business on Tuesday, there are only three work days (plus possible volunteer time over the weekend). Now even with only one I-D to review, that may be a real challenge. Sometimes, a lucky reviewer will get 130 pages (e.g., draft-ietf-nntpext-base-27). That doesn't compute 🥲📄
  • There are some mechanical issues. The process followed is far too manual. Everything needs to be robotic except for the judgment calls about which reviewer gets which draft. Similarly, the reviewer should be able to just paste the review into a web form, click, and it's sent off to everyone appropriate and posted to the review site 🤖✨

7.3. Gen-ART Secretaries' Impressions 🗂️

Serving as the secretary of Gen-ART is a worthwhile experience. From a personal point of view, it gives the secretary an easy way to track all of the work going through the IESG review process and see how the work flowed through that process. Also, by reviewing and sometimes creating the one-line abstracts that go on the review web page, the secretary has an opportunity to really get a survey of the work being approved by the IETF 🍓

The nature of these reviews is informal, and originally the reviews were only intended for the General AD, though they were made public. During 2004, there was little if any interaction between authors and reviewers. There was some discussion during 2004 about trying to expand the role of Gen-ART to a more formal, early-review model, i.e., to evolve it into a form of SIRS. The original Gen-ART secretary was against such a transformation, because she felt it would put at risk something that worked. She believed that there were risks inherent in formalizing the reviews and adding mechanisms for standardizing the resultant review process. Another concern involves the interaction between reviewers and authors. As discussed above, it has become the practice to send reviews to the authors with an explanation about the nature of Gen-ART reviews. While it is clear that this has resulted in improved RFCs, it has also resulted in increased workload for the reviewers 💜

The secretary thinks that Gen-ART is an experiment that works well, but she also believes it is fragile. The secretary is often concerned about overburdening reviewers, and feels it is her responsibility to keep them from burning out. Adding additional reviewers to the review team would help to alleviate this concern. In terms of the process, adding additional reviewers has minimal impact 🫶🌸

8. Needed Improvements ✨

The current size of the review team introduces a fairly heavy workload for the individual reviewers that are not on the "one doc per month" assignment cycle. Additional reviewers would be really helpful to alleviate this workload. It is also important to note that having additional reviewers adds minimal workload to the secretary's process; thus, the only blocking point is finding the right folks that are interested in this type of volunteer role. As noted in Section 7.2, 30 would be a good size for the review team. This would cut the workload for an individual reviewer in half (given the current model of 9 reviewers on the "one doc per month" assignment cycle) 🙌

Obviously, automation of the process would be a good thing. However, Gen-ART secretaries are not necessarily highly motivated to transition to a more automated approach until a significant part of the process is automated. In more recent consideration of this situation, it likely would be best to first automate the process of entering the reviews, as that benefits the review team as a whole. This automation should allow the reviewers to enter the reviews via a web interface that would automatically generate the appropriate emails -- quite similar to how the draft "Upload" tool currently works. Also, given consistent naming conventions for the review forms, this step would automate some of the process for the secretary, as the reviews would automatically appear via the spreadsheet hyperlinks, although there would still be a need to manually enter the summary. But this would eliminate the need to edit/normalize and upload files and, hopefully, eliminate the problem encountered with unflowed text in emails and getting the review properly formatted using some text editors 🤞💻

Section 5 was written to facilitate the process of determining tools requirements, by providing the very detailed steps currently applied to the process. As noted above, automating the upload of the reviews could be a good first step. This is somewhat starting at the end of the process. However, it seems that by automating in this direction, we may have optimal results; since one of the earliest steps in the process is the task of assigning reviewers, it likely needs the most manual intervention, even with tools available 🧩

The current SecDir secretary does use some tools for assignments and generating assignment emails. These tools could be considered for use by the Gen-ART secretary. Since the SecDir reviews are not cached and the information maintained for those reviews is less detailed, there would be no reusability of that aspect. However, if the Gen-ART spreadsheet can be automatically populated (with assignments and completed reviews), the SecDir may be able to make use of that same tool 🌈

9. Applicability ✨

okay so bestie, here's the tea 🫖. As the process is implemented today, Gen-ART has no formal role in the IETF standards process 💅. But trust in the review team has been building, and the team has learned to deliver reviews that people genuinely appreciate 🥹, so they've had a significant impact on I-D quality and on timeliness 🔥. Honestly, the team isn't a roadblock at all. It's giving confidence 💖: the General AD can make decisions more easily and resolve issues with way more precision ✨.

Reviews now usually show up during IETF Last Call, so like the SecDir reviews, they're generally expected now 🙈. That means the team's role has gotten more formal than it was back when this document was first drafted in 2005 🎀. Still, what happens with the reviews is entirely up to the ADs, document shepherds, WG chairs, and authors, to handle however they think is best 🦋.

10. Security Considerations 🔐

Since this is an informational I-D about an open process, the security considerations are all about the process and the people involved in it 🧚. The big concern is limiting who can create and update the Gen-ART data/files, so the integrity of the data stays intact 🚨. For example, each Gen-ART reviewer should have a unique user name/password, just like folks do to access any other IETF-maintained data, as appropriate 🫶.

11. Acknowledgments 🌸

Okay this part is so sweet 🥺. The initial comments came from the members of Gen-ART, and the experiences in this document come from their hard work over the last 7+ years 👑. We're thanking the past reviewers of Gen-ART:

  • Mark Allman
  • Harald Alvestrand (creator of Gen-ART) 🌟
  • Ron Bonica
  • Scott Brim
  • Gonzalo Camarillo
  • Sharon Chisholm
  • Spencer Dawkins
  • Lakshminath Dondeti
  • Avri Doria (past secretary) 💜
  • Pasi Eronen
  • Dorothy Gellert
  • Eric Gray
  • Avashalom Houri
  • Glenn Kowack
  • John Loughney
  • Lucy Lynch
  • Enrico Marocco
  • Michael Patton
  • Stefan Santesson
  • Robert Sparks
  • Tom Taylor
  • Sean Turner
  • Christian Vogt
  • Suzanne Woolf

And we also thank the current team of reviewers/secretary 🎀:

  • Mary Barnes (past secretary, 2005-2010)
  • Richard Barnes
  • David Black
  • Ben Campbell
  • Brian Carpenter (past General AD)
  • Elwyn Davies
  • Francis Dupont
  • Roni Even
  • Miguel-Angel Garcia
  • Vijay Gurbani (assisting secretary to upload reviews)
  • Wassim Haddad
  • Joel Halpern
  • Suresh Krishnan
  • Peter McCann
  • Jean Mahoney (secretary as of Jan. 2011)
  • Alexey Melnikov
  • Kathleen Moriarty

12. Informative References 📚

Here are the references, all listed in full 🧾:

[RFC1958]     Carpenter, B., Ed., "Architectural Principles of the
              Internet", RFC 1958, June 1996.

[RFC2223bis]  Reynolds, J., Ed., and R. Braden, Ed., "Instructions to
              Request for Comments (RFC) Authors", Work in Progress,
              August 2004.

[RFC-STYLE]   Braden, R., Ginoza, S., and A. Hagens, "RFC Document
              Style", September 2009,
              <http://www.rfc-editor.org/rfc-style-guide/rfc-style>.

[RFC3426]     Floyd, S., "General Architectural and Policy
              Considerations", RFC 3426, November 2002.

[RFC3439]     Bush, R. and D. Meyer, "Some Internet Architectural
              Guidelines and Philosophy", RFC 3439, December 2002.

[RFC3552]     Rescorla, E. and B. Korver, "Guidelines for Writing RFC
              Text on Security Considerations", BCP 72, RFC 3552,
              July 2003.

[RFC5226]     Narten, T. and H. Alvestrand, "Guidelines for Writing
              an IANA Considerations Section in RFCs", BCP 26,
              RFC 5226, May 2008.

[SIRS]        Carpenter, B. and D. Crocker, "Careful Additional
              Review of Documents (CARD) by Senior IETF Reviewers
              (SIRS)", Work in Progress, June 2003.

Authors' Addresses 📮

Mary Barnes
Polycom
TX
USA

EMail: mary.ietf.barnes@gmail.com


Avri Doria
Research Consultant
Providence, RI
USA

EMail: avri@acm.org


Harald Alvestrand
Google
Kungsbron 2
11122 Stockholm
SE

EMail: harald@alvestrand.no


Brian E. Carpenter
University of Auckland
PB 92019
Auckland, 1142
New Zealand

Phone:
EMail: brian.e.carpenter@gmail.com

That's the whole thing, bestie! 🌈✨ Gen-ART is literally the review team that keeps the IETF's documents slaying 💅🦋

📦 where this page comes from