RFC 9307

Report from the IAB Workshop on Analyzing IETF Data (AID) 2021

📅 published September 2022, from ietf

RFC 9307: Report from the IAB Workshop on Analyzing IETF Data (AID) 2021 🎀

Okay bestie, this is RFC 9307 💖. It's a report from the Internet Architecture Board (IAB) on the "Show me the numbers" workshop. The authors are N. ten Oever (University of Amsterdam), C. Cath (University of Cambridge), M. Kühlewind (Ericsson), and C. S. Perkins (University of Glasgow). It's Informational, dated September 2022, with ISSN 2070-1721. It doesn't list any RFCs it obsoletes or updates, so it's standing on its own ✨

Abstract

The "Show me the numbers: Workshop on Analyzing IETF Data (AID)" workshop was run by the IAB from November 29 to December 2, 2021. It was supposed to be hosted by the IN-SIGHT.it project at the University of Amsterdam, but it went fully online 💻 The workshop had two discussion parts with a hackathon in the middle. This report sums up what was discussed and points out topics that deserve future work 🔥

Heads up: this document is a report on what happened at the workshop. The views and positions in it belong to the workshop participants, and they don't necessarily reflect the IAB's views and positions 🙈

Status of This Memo

Real talk, this isn't an Internet Standards Track spec. It's published just for information 📚

This document is a product of the Internet Architecture Board (IAB). It holds information the IAB thought was worth keeping on the permanent record, and it represents the consensus of the IAB. Anything the IAB approves for publication is never a candidate for any level of Internet Standard. Check Section 2 of RFC 7841 for the details 💅

Want the current status, any errata, or a way to send feedback? Head to https://www.rfc-editor.org/info/rfc9307 ✨

Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved 🌈

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info), as they stand on the publication date. Please read these carefully, because they spell out your rights and restrictions for this document 🧚

Table of Contents

   1.  Introduction
   2.  Workshop Scope and Discussion
     2.1.  Tools, Data, and Methods
     2.2.  Observations on Affiliation and Industry Control
     2.3.  Community and Diversity
     2.4.  Publications, Process, and Decision Making
     2.5.  Environmental Sustainability
   3.  Hackathon Report
   4.  Position Papers
     4.1.  Tools, Data, and Methods
     4.2.  Observations on Affiliation and Industry Control
     4.3.  Community and Diversity
     4.4.  Publications, Process, and Decision Making
     4.5.  Environmental Sustainability
   5.  Informative References
   Appendix A.  Data Taxonomy
   Appendix B.  Program Committee
   Appendix C.  Workshop Participants
   IAB Members at the Time of Approval
   Acknowledgments
   Authors' Addresses

1. Introduction ✨

Okay, so the IETF is an international Standards Developing Organization (SDO), and it holds a ton of data 🤩 That covers its history, how it develops, what's happening with standards right now, Internet protocols, and the institutions that make up the IETF. A big chunk of it is public, but honestly it's badly underused as a tool for informing the work inside the IETF, or the wider research crowd who care about Internet governance and how information and communication technologies (ICT) standards get set 📊

The point of the "IAB Workshop on Analyzing IETF Data (AID) 2021" was to see how IETF data gets used right now, figure out what insights you can pull from it, and explore open questions about how it could be used in the future 🔍

These questions could shape a research agenda built on IETF data. It would encourage collaboration between anyone interested, from academia and civil society to industry and IETF leadership 🫶

2. Workshop Scope and Discussion 💜

The workshop had two all-group discussion slots, one at the start and one at the end. In between, participants ran hackathon activities based on topics that came up in the opening discussion and in the position papers people submitted. These are the topic areas that got identified and discussed 👑

2.1. Tools, Data, and Methods 🛠️

The IETF has a huge range of data sources. The main ones are the mailinglist archives [Mail-Arch], RFCs [IETF-RFCs], and the datatracker [Datatracker]. The datatracker gives info on participants, authors, meeting proceedings, minutes, and more [Data-Overview]. There are also statistics for the IETF websites [IETF-Statistics], the working group Github repositories, and the IETF survey data [Survey-Data]. People talked about how useful download statistics for the RFCs themselves are, pulled from different repos 📈

Lots of tools exist for analyzing this data. They're built by IETF participants or by researchers who are into what the IETF does. Two projects presented at the workshop were BigBang [BigBang] and Sodestream's IETFdata [ietfdata] library 💖 The RFC Prolog Database came up in a submitted paper; see [Prolog-Database]. These projects could add to the existing IETF statistics [ArkkoStats] and datatracker statistics [DatatrackerStats], for example around gender-related info. Privacy issues came up too, along with what it means to make this kind of data publicly available 🙈

The datatracker is a community tool, and it welcomes contributions 🎀 For additions to the current interfaces or the statistics page, check the Datatracker Database Overview [Data-Overview]. When the workshop happened, setup instructions for a local development environment were at IAB AID Workshop Data Resources [DataResources]. Questions or discussion about the datatracker and possible enhancements can go to tools-discuss@ietf.org 📬

2.2. Observations on Affiliation and Industry Control 🏢

Lots of the submitted position papers were interested in researching industry control in the standardization process. That's different from individual contributions made in a personal capacity. Here, industry control covers both a) technical contributions and the ability to actually standardize them, and b) competition for leadership roles 👀 To look into these questions, the group talked about investigating participant affiliations, including "indirect" affiliations (for example, by tracking funding and changes in affiliation). People also said they needed to model company characteristics or stakeholder groups 💅

Looking at IETF data shows that affiliation dynamics are hard to capture. That's partly because of how the data is entered, and partly because of bigger social dynamics 🌀 On the data capture side, affiliation is an open text field, so people write their affiliation in lots of different ways (for example, with different capitalization, spacing, word separation, and so on). A common data format could help analyses that compare how SDOs perform and how actors behave inside and across standards bodies. To help with this, a draft data model was built during the hackathon part of the workshop, and you can find it in Appendix A ✨

There's also the question of mergers, acquisitions, and subsidiary companies. Nothing authoritative outside the data gives you a clean source of variation for affiliation changes, so people hand-collect and curate data to analyze how affiliations shift over time. The approach is imperfect, but you can still draw conclusions from it 📊 For example, when a small organization joins a large one through a merger or acquisition, that leads to a statistically significant increase in the likelihood of an individual being put in a working group chair position (see the document by Baron and Kanevskaia [LEADERSHIP-POSITIONS]). Iconic result, honestly 👑

2.3. Community and Diversity 🌸

The workshop participants were super into using existing data to better understand who the current IETF community actually is 🥰 They also cared about the community's diversity, how to potentially increase it, and why that makes the community more inclusive. For example, is there a set of factors that "drive people away"? And if so, why? Inclusivity and transparency about the standardization process are generally important for keeping the Internet and its development process viable 🌈 During the workshop discussion, someone pointed out that when you measure different angles of diversity, you also have to understand the goals you're chasing. Is it about competence (mainly technical diversity from different companies and stakeholder groups), or relevance (which also means regional diversity and an international footprint)? 🧠

The discussion on community and diversity went all over the place. It covered methods like novel text mining, time series clustering, graph mining, and psycholinguistic approaches to understanding the consensus mechanism. It also got into more speculative stuff, like what it would take to build a feminist Internet 💜 The group also talked about the data you'd need to measure who is in the community and how diverse it is 📊

The big challenge, people said, is defining what diversity means and how to measure it. One participant put it as defining "who the average IETFer is" 🤔 There was also a question about what to do with missing data, or with communities that don't participate or are underrepresented. The examples given were women, individuals from the African continent, and network operators 🌍 On how IETF data is structured, several researchers said it's hard to track conversations, since mail threads split, merge, and change. The ICANN-at-large model came up as an example of how to bring in relevant stakeholders who aren't currently in the IETF 🫶 Going the other way, outside communities (especially policy makers) also find it interesting to get a sense of who the IETF community is and to stay updated on it 🎀

The human side of community and diversity got highlighted too. To understand the IETF community's diversity, you have to talk to people, not just analyze text 🗣️ And to make sure the community is inclusive, individual participants have to make an effort. As one participant put it, they need to tell people that their participation is valuable 💖

(Part 2 is coming up next, bestie 🧚)

2.4. Publications, Process, and Decision Making 📚✨

Okay so a bunch of submissions were about the RFC publication process, how standards and other RFCs get developed in the IETF, and how the IETF actually makes decisions 🧠. That covered technical calls on what goes into the standards, procedural and process decisions, and how we can understand, model, and maybe improve the standards process 💅. Some of the work looked at what makes an RFC successful, how RFCs get used and referenced, and what we can learn about how much a topic matters by studying the RFCs, Internet-Drafts, and email discussions 🦋.

There were three sets of questions in this area. The first one was about the success and failure of standards and asked:

  • What makes a successful and good RFC?
  • What makes the process of making RFCs successful?
  • How are RFCs used and referenced once published?

Discussion looked at how to better understand the path from an Internet-Draft to an RFC, to see if there are specific factors that lead to successful development. Participants explored whether this depends on the seniority and experience of the authors, the topic and IETF area, the extent and scope of mailing list discussion, and other factors 👀. The goal was to figure out whether the success of an Internet-Draft can be predicted, and whether interventions can be developed to increase the likelihood of success for work 🌈.

The second question focused on decision making:

  • How does the IETF make design decisions?
  • What are the bottlenecks in effective decision making?
  • When is a decision made? And what is the decision?

The difficulties here are capturing decisions and the results of consensus calls early in the process, and understanding the factors that lead to effective decision making 😭.

Finally, there were questions about what we can learn about protocols by studying IETF publications, processes, and decision making. For example:

  • Are there insights to be gained around how security concerns are discussed and considered in the development of standards?
  • Is it possible to verify correctness of protocols and detect ambiguities?
  • What can be learned by extracting insights from implementations and activities on implementation efforts?

Answers to these questions will come from analysis of IETF emails, RFCs and Internet-Drafts, meeting minutes, recordings, Github data, and external data such as surveys, etc. 🔥

2.5. Environmental Sustainability 🌱

The final discussion session considered environmental sustainability 🌍. Topics included what the IETF's role is with respect to climate change, both in terms of the environmental impact of the way the IETF develops standards and the environmental impact of the standards the IETF develops.

Discussion started by considering how sustainable IETF meetings are. The focus was on the amount of carbon dioxide (CO2) emissions that IETF meetings are responsible for, and how we can make the IETF more sustainable ✈️. Analysis looked at the home locations of participants, meeting locations, and the carbon footprint of air travel and remote attendance, to estimate the CO2 costs of an IETF meeting. While the analysis is ongoing, initial results suggest that the costs of holding multiple in-person IETF meetings per year are likely unsustainable in terms of CO2 emissions 😬.

The extent to which climate impacts are considered during the development and standardization of Internet protocols was also discussed 🌸. RFCs and Internet-Drafts of active working groups were reviewed for relevant keywords, to highlight how much climate change, energy efficiency, and related topics were considered in the design of Internet protocols. This review revealed how limited that consideration has been 💀. There is ongoing work to get a fuller picture by reviewing meeting minutes and mail archives as well, but initial results show only limited consideration of these important issues 🫶.

3. Hackathon Report 💻🎀

The middle two days of the workshop were organized as a hackathon, and it was giving main character energy 👑. The aims of the hackathon were to:

  1. acquaint people with the different data sources and analysis methods,
  2. seek to answer some of the questions that came up during presentations on the first day of the workshop, and
  3. foster collaboration among researchers to grow a community of IETF data researchers 🧚.

At the end of Day 1, the plenary presentation day, people were invited to divide themselves into groups and select their own respective facilitators. All groups had their own work space and could use their own communication methods and channels. Daily check-ins were also organized during the two hackathon days. On the final day, the hackathon groups presented their work in a plenary session 🎤.

According to the co-chairs, the objectives of the hackathon were met, and the output significantly exceeded expectations 🙈. It allowed more interaction than academic conferences do, and it produced some actual research results from people who had not collaborated before the workshop 💖.

Future workshops that choose to integrate a hackathon could consider asking participants to submit issues and questions beforehand (potentially as part of the position papers or the sign-up process), to make it easier to form groups 🍓.

4. Position Papers 📝

4.1. Tools, Data, and Methods 🛠️

  • Sebastian Benthall, "Using Complex Systems Analysis to Identify Organizational Interventions" [COMPLEX-SYSTEMS]
  • Stephen McQuistin and Colin Perkins, "The ietfdata Library" [ietfdata-Library]
  • Marc Petit-Huguenin, "The RFC Prolog Database" [Prolog-Database]
  • Jari Arkko, "Observations about IETF process measurements" [MEASURING-IETF-PROCESSES]

4.2. Observations on Affiliation and Industry Control 🏢

  • Justus Baron and Olia Kanevskaia, "Competition for Leadership Positions in Standards Development Organizations" [LEADERSHIP-POSITIONS]
  • Nick Doty, "Analyzing IETF Data: Changing affiliations" [ANALYZING-AFFILIATIONS]
  • Don Le, "Analysing IETF Data Position Paper" [ANALYSING-IETF]
  • Elizaveta Yachmeneva, "Research Proposal" [RESEARCH-PROPOSAL]

4.3. Community and Diversity 💜

  • Priyanka Sinha, Michael Ackermann, Pabitra Mitra, Arvind Singh, and Amit Kumar Agrawal, "Characterizing the IETF through its consensus mechanisms" [CONSENSUS-MECHANISMS]
  • Mallory Knodel, "Would feminists have built a better internet?" [FEMINIST-INTERNET]
  • Wes Hardaker and Genevieve Bartlett, "Identifying temporal trends in IETF participation" [TEMPORAL-TRENDS]
  • Lars Eggert, "Who is the Average IETF Participant?" [AVERAGE-PARTICIPANT]
  • Emanuele Tarantino, Justus Baron, Bernhard Ganglmair, Nicola Persico, and Timothy Simcoe, "Representation is Not Sufficient for Selecting Gender Diversity" [GENDER-DIVERSITY]

4.4. Publications, Process, and Decision Making 📚

  • Michael Welzl, Carsten Griwodz, and Safiqul Islam, "Understanding Internet Protocol Design Decisions" [DESIGN-DECISIONS]
  • Ignacio Castro et al., "Characterising the IETF through the lens of RFC deployment" [RFC-DEPLOYMENT]
  • Carsten Griwodz, Safiqul Islam, and Michael Welzl, "The Impact of Continuity" [CONTINUITY]
  • Paul Hoffman, "RFCs Change" [RFCs-CHANGE]
  • Xue Li, Sara Magliacane, and Paul Groth, "The Challenges of Cross-Document Coreference Resolution in Email" [CROSS-DOC-COREFERENCE]
  • Amelia Andersdotter, "Project in time series analysis: e-mailing lists" [E-MAILING-LISTS]
  • Mark McFadden, "A Position Paper by Mark McFadden" [POSITION-PAPER]

4.5. Environmental Sustainability 🌏

  • Christoph Becker, "Towards Environmental Sustainability with the IETF" [ENVIRONMENTAL]
  • Daniel Migault, "CO2eq: Estimating Meetings' Air Flight CO2 Equivalent Emissions: An Illustrative Example with IETF meetings" [CO2eq]

5. Informative References ✨

okay bestie, here are the receipts 🧾💅 every one of these is an informative reference, so they're here for context and aren't required reading to implement anything. they're all from the 2021 IAB AID workshop papers and tools, and I've kept every title, author, date, and link exactly as written 🫶

Appendix A. Data Taxonomy 🗂️

here's the draft data taxonomy for SDO data, the whole family tree of organizations, people, patents, and meetings 🌳✨

A Draft Data Taxonomy for SDO Data:

Organization:
  Organization Subsidiary
  Time
  Email domain
  Website domain
  Size
          Revenue, annual
          Number of employees
  Org - Affiliation Category (Labels) ; 1 : N
    Association
    Advertising Company
    Chipmaker
    Content Distribution Network
    Content Providers
    Consulting
    Cloud Provider
    Cybersecurity
    Financial Institution
    Hardware vendor
    Internet Registry
    Infrastructure Company
    Networking Equipment Vendor
    Network Service Provider
    Regional Standards Body
    Regulatory Body
    Research and Development Institution
    Software Provider
    Testing and Certification
    Telecommunications Provider
    Satellite Operator

Org - Stakeholder Group : 1 - 1
    Academia
    Civil Society
    Private Sector -- including industry consortia and associations;
    state-owned and government-funded businesses
    Government
    Technical Community (IETF, ICANN, ETSI, 3GPP, oneM2M, etc)
    Intergovernmental organization

SDO:
  Membership Types (SDO)
  Members (Organizations for some, individuals for others...)
  Membership organization
    Regional SDO
      ARIB
      ATIS
      CCSA
      ETSI
      TSDSI
      TTA
      TTC
    Consortia

Country of Origin:
  Country Code

Number of Participants

Patents
  Organization
  Authors - 1 : N - Persons/Participants
  Time
  Region
  Patent Pool
  Standard Essential Patent
    If so, for which standard

Participant (An individual person)
  Name
  1: N - Emails
    Time start / time end

  1 : N : Affiliation
    Organization
    Position
          Time start / end

  1 : N : Affiliation - SDO
    Position
    SDO
    Time

  Email Domain (personal domain)

  (Contribution data is in other tables)

Document
  Status of Document
          Internet Draft
          Work Item
    Standard
  Author -
    Name
          Affiliation - Organization
    Person/Participant
        (Affiliation from Authors only?)

Data Source - Provenance for any data imported from an external data set

Meeting
  Time
  Place
  Agenda
  Registrations
    Name
    Email
    Affiliation

Appendix B. Program Committee 🎀

the workshop Program Committee squad was Niels ten Oever (Chair, University of Amsterdam), Colin Perkins (Chair, IRTF, University of Glasgow), Corinne Cath (Chair, Oxford Internet Institute), Mirja Kühlewind (IAB, Ericsson), Zhenbin Li (IAB, Huawei), and Wes Hardaker (IAB, USC/ISI) 👑

Appendix C. Workshop Participants 🌈

the Workshop Participants were Bernhard Ganglmair, Carsten Griwodz, Christoph Becker, Colin Perkins, Corinne Cath, Daniel Migault, Don Le, Effy Xue Li, Elizaveta Yachmeneva, Francois Ortolan, Greg Wood, Ignacio Castro, Jari Arkko, Justus Baron, Karen O'Donoghue, Lars Eggert, Mallory Knodel, Marc Petit-Huguenin, Mark McFadden, Michael Welzl, Mirja Kühlewind, Nick Doty, Niels ten Oever, Priyanka Sinha, Safiqul Islam, Sebastian Benthall, Stephen McQuistin, Wes Hardaker, and Zhenbin Li 🫶

IAB Members at the Time of Approval 🧚

these are the Internet Architecture Board members at the time this document was approved for publication 💜

  • Jari Arkko
  • Deborah Brungard
  • Lars Eggert
  • Wes Hardaker
  • Cullen Jennings
  • Mallory Knodel
  • Mirja Kühlewind
  • Zhenbin Li
  • Tommy Pauly
  • David Schinazi
  • Russ White
  • Quin Wu
  • Jiankang Yao

Acknowledgments 🙏

the Program Committee wants to give a huge thank you to Cindy Morgan for logistics support and to Kate Pundyk for note-taking 🍓

we'd also like to thank the Ford Foundation for their support, which made participation of Corinne Cath, Kate Pundyk, and Mallory Knodel possible (grant number, 136179, 2020) 💖

efforts put into this workshop by Niels ten Oever were made possible through funding from the Dutch Research Council (NWO) through grant MVI.19.032 as part of the program 'Maatschappelijk Verantwoord Innoveren (MVI)' ✨

efforts in the organization of this workshop by Colin Perkins were supported in part by the UK Engineering and Physical Sciences Research Council under grant EP/S036075/1 🌸

Authors' Addresses 💌

Niels ten Oever
University of Amsterdam
Email: mail@nielstenoever.net


Corinne Cath
University of Cambridge
Email: corinnecath@gmail.com


Mirja Kühlewind
Ericsson
Email: mirja.kuehlewind@ericsson.com


Colin Perkins
University of Glasgow
Email: csp@csperkins.org
📦 where this page comes from