RFC 5657
Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard
RFC 5657: Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard ✨
Title block, bestie 🎀
- Series: Network Working Group, RFC 5657, and BCP 9 🌸
- Authors: L. Dusseault (Messaging Architects) and R. Sparks (Tekelec) 👑
- Category: Best Current Practice 💖
- Date: September 2009 🦋
- Updates: RFC 2026 (this doc changes some of its rules) 🫶
- Obsoletes: none listed 🧚
Part 1 of 3, more coming! 💅
Abstract
Okay so, advancing a protocol to Draft Standard means you have to document how it was interoperated and implemented. The reports people have sent in over the years look wildly different, and new report writers get basically no guidance. This document updates the existing processes and gives more detail on what belongs in an interoperability and implementation report. It's giving clarity 💅✨
Status of This Memo
This one is a Best Current Practice for the Internet Community, and the IETF is asking for discussion and suggestions for improvements 🌈 Distribution of this memo is unlimited 🦋
Copyright and License Notice
Here's the legal stuff, verbatim because it has to be exact 💜
Copyright (c) 2009 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the BSD License.Table of Contents
1. Introduction ....................................................2
2. Content Requirements ............................................4
3. Format ..........................................................5
4. Feature Coverage ................................................6
5. Special Cases ...................................................8
5.1. Deployed Protocols .........................................8
5.2. Undeployed Protocols .......................................8
5.3. Schemas, Languages, and Formats ............................8
5.4. Multiple Contributors, Multiple Implementation Reports .....9
5.5. Test Suites ................................................9
5.6. Optional Features, Extensibility Features .................10
6. Examples .......................................................10
6.1. Minimal Implementation Report .............................11
6.2. Covering Exceptions .......................................11
7. Security Considerations ........................................11
8. References .....................................................12
8.1. Normative References ......................................12
8.2. Informative References ....................................121. Introduction ✨
The Draft Standard level, and the requirements a standard has to meet to get there, are described in [RFC2026]. For Draft Standard, two implementations have to interoperate, and on top of that, a report has to be given to the IETF. The whole paragraph about this documentation says:
The Working Group chair is responsible for documenting the
specific implementations which qualify the specification for Draft
or Internet Standard status along with documentation about testing
of the interoperation of these implementations. The documentation
must include information about the support of each of the
individual options and features. This documentation should be
submitted to the Area Director with the protocol action request.
(see Section 6)Moving documents along the standards track is a big signal to users and implementors, and going through the advancement process can actually make the standard or its implementations better 🔥 The barriers to Draft Standard seem pretty high, or at least super confusing. This memo is here to help guide people through one part of advancing a spec to Draft Standard. It also changes some of the requirements from RFC 2026, with the goal of keeping or improving report quality while making them less of a burden to write 💅
Having and demonstrating enough interoperability is a gating requirement for getting to Draft Standard. So the main goal of an implementation report is to convince the IETF and the IESG that the protocol is ready. That goal gets met by summarizing the interoperability characteristics and giving just enough detail to back up that conclusion. There are also side benefits for the people writing the report, like bugs found or fixed in the tested implementations, documentation that helps future implementors, or ideas for other documents or future revisions of the protocol being tested 🧚
Different kinds of documentation fit widely deployed standards than fit standards that aren't deployed yet 🌸 Different test approaches fit standards that aren't typical protocols, like languages, formats, schemas, and so on. Section 5 talks about how reports for these standards can vary 💜
Implementation should naturally focus on the final version of the RFC. If there's any evidence that implementations are interoperating based on Internet-Drafts or earlier versions of the specification, or if interoperability was greatly helped by mailing list clarifications, that should be noted in the report 🙈
The level of detail in reports accepted in the past has varied super widely 😭 One example of a submitted report that is not enough to show interoperability is, in its entirety, "A partial list of implementations include: Cray SGI Netstar IBM HP Network Systems Convex". It doesn't say how anyone knows these implementations interoperate (was it public lab testing? internal lab testing? deployment?). It also doesn't capture whether implementors were aware of, or were asked about, any features that turned out to be problematic. On the other end, some reports have a ton of detail about the test methodology but barely any info about what worked and what didn't 💀
This memo is meant to clarify what an implementation report should contain, and to suggest a reasonable form for most of them. It's not meant to rule out good ideas 🦋 For example, this memo can't account for every process variation, like documents going to Draft Standard twice, and it can't consider every type of standard either. Whenever a situation differs a lot from what's described here, the IESG uses judgement to decide whether an implementation report meets the goals above 👑
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] 💖
2. Content Requirements 🎀
The implementation report MUST identify the author of the report, who is responsible for characterizing the interoperability quality of the protocol. The report MAY identify other contributors (testers, those who answered surveys, or those who contributed information) to share credit or blame 😌 The report MAY provide a list of report reviewers who corroborate the characterization of interoperability quality, or name an active working group (WG) that reviewed the report.
Some of the RFC 2026 requirements get relaxed with this update:
- The report MAY name exactly which implementations were tested. A requirement to name implementations was implied by the description of the responsibility for "documenting the specific implementations" in RFC 2026. However, naming implementations usually helps meet the goals of implementation reports. If only a subset of implementations was tested or surveyed, it would also help to explain how that subset was chosen or self-selected. See also the note on implementation independence below.
- The report author MAY choose an appropriate level of detail to document feature interoperability, rather than documenting each individual feature. See the note on granularity of features below.
- A contributor other than a WG chair MAY submit an implementation report to an Area Director (AD).
- Optional features that are not implemented, but are important and do not harm interoperability, MAY, exceptionally and with approval of the IESG, be left in a protocol at Draft Standard. See Section 5.6 for documentation requirements and an example of where this is needed.
Note on independence: Independence of implementations is mentioned in the RFC 2026 requirements for Draft Standard status. Independent implementations should be written by different people at different organizations using different code and protocol libraries. If it's necessary to relax this definition, it can be relaxed as long as there is evidence to show that success is due more to the quality of the protocol than to out-of-band understandings or common code. If there are only two implementations of an undeployed protocol, the report SHOULD identify the implementations and their "genealogy" (which libraries were used or where the codebase came from). If there are many more implementations, or the protocol is in broad deployment, it is not necessary to call out which two of the implementations demonstrated interoperability of each given feature. A reader may conclude that at least some of the implementations of that feature are independent.
Note on granularity: The granularity of features described in a specification is necessarily very detailed. In contrast, the granularity of an implementation report need not be as detailed. A report need not list every "MAY", "SHOULD", and "MUST" in a complete matrix across implementations. A more effective approach might be to characterize the interoperability quality and testing approach, then call out any known problems in either testing or interoperability 🌈
3. Format ✨
Okay so the format rules are super strict, bestie 💅 Implementation and interoperability reports MUST be ASCII text with line breaks so they're readable 📝 Same as Internet-Drafts, no 8-bit characters are allowed right now 🙅♀️ It's acceptable (not required tho) for a report to be formatted as an Internet-Draft ✨
Here's a simple outline an implementation report MAY follow, in part or in full 🦋
Title: Titles of implementation reports are strongly RECOMMENDED to contain one or more RFC number for consistent lookup in a simple archive 🗂️ Also, the name or a common mnemonic of the standard should be in the title. An example might look like "Implementation Report for the Example Name of Some Protocol (ENSP) RFC XXXX" 🎀
Author: Identify the author of the report 🫶
Summary: Attest that the standard meets the requirements for Draft Standard and name who is attesting it 👑 Describe how many implementations were tested or surveyed 🔢 Quickly characterize the deployment level and where the standard can be found in deployment 🌍 Call out, and if possible, briefly describe any notably difficult or poorly interoperable features and explain why these still meet the requirement 😭 Assert any derivative conclusions: if a high-level system is tested and shown to work, then we may conclude that the normative requirements of that system (all sub-system or lower-layer protocols, to the extent that a range of features is used) have also been shown to work. It's giving chain reaction energy ⭐
Methodology: Describe how the information in the report was obtained 🔍 This should be no longer than the summary.
Exceptions: This section might read "Every feature was implemented, tested, and widely interoperable without exception and without question" 💖 If that statement is not true, then this section should cover whether any features were thought to be problematic 🥺 Problematic features need not disqualify a protocol from Draft Standard, but this section should explain why they do not (e.g., optional, untestable, trace, or extension features). See the example in Section 6.2. Not me getting the vibe that honesty is iconic here 🙈
Detail sections: Any other justifying or background information can be included here 📚 In particular, any information that would have made the summary or methodology sections more than a few paragraphs long may be created as a detail section and referenced 🧚
In this section, it would be good to discuss how the various considerations sections played out 🔐 Were the security considerations accurate and dealt with appropriately in implementations? Was real internationalization experience found among the tested implementations? Did the implementations have any common monitoring or management functionality (although note that documenting the interoperability of a management standard might be separate from documenting the interoperability of the protocol itself)? Did the IANA registries or registrations, if any, work as intended? 🤔✨
Appendix sections: It's not necessary to archive test material such as test suites, test documents, questionnaire text, or questionnaire responses 🗃️ However, if it's easy to preserve this information, appendix sections allow readers to skip over it if they are not interested 🙌 Preserving detailed test information can help people doing similar or follow-on implementation reports, and can also help new implementors 💜
4. Feature Coverage 🎯
Okay, so what counts as a "feature" for an interoperability report has been debated forever 😩 Good judgement is required in finding a level of detail that adequately demonstrates coverage of the requirements. Statements made at too high a level will result in a document that can't be verified and hasn't adequately challenged that the testing accidentally missed an important failure to interoperate 🚩 On the other hand, statements at too fine a level result in an exponentially exploding matrix of requirement interaction that overburdens the testers and report writers 💀 The important information in the resulting report would likely be hard to find in the sea of detail, making it difficult to evaluate whether the important points of interoperability have been addressed.
The best interoperability reports organize statements of interoperability at a level of detail just sufficient to convince the reader that testing has covered the full set of requirements and, in particular, that the testing was sufficient to uncover any places where interoperability does not exist ✨ Reports similar to that for RTP/RTCP (an excerpt appears below) are more useful than an exhaustive checklist of every normative statement in the specification 🦋
10. Interoperable exchange of receiver report packets.
o PASS: Many implementations, tested UCL rat with vat,
Cisco IP/TV with vat/vic.
11. Interoperable exchange of receiver report packets when
not receiving data (ie: the empty receiver report
which has to be sent first in each compound RTCP packet
when no-participants are transmitting data).
o PASS: Many implementations, tested UCL rat with vat,
Cisco IP/TV with vat/vic.
...
8. Interoperable transport of RTP via TCP using the
encapsulation defined in the audio/video profile
o FAIL: no known implementations. This has been
removed from the audio/video profile.
Excerpts from
http://www.ietf.org/iesg/implementation/report-avt-rtp-rtcp.txtConsensus can be a good tool to help determine the appropriate level for such feature descriptions 🤝 A working group can make a strong statement by documenting its consensus that a report sufficiently covers a specification and that interoperability has been demonstrated 👑
5. Special Cases 🌈
5.1. Deployed Protocols
When a protocol is deployed, results obtained from laboratory testing are not as useful to the IETF as learning what is actually working in deployment 🚀 To this end, it may be more informative to survey implementors or operators 📋 A questionnaire or interview can elicit information from a wider number of sources. As long as it is known that independent implementations can work in deployment, it is more useful to discover what problems exist, rather than gather long and detailed checklists of features and options 💅
5.2. Undeployed Protocols
It is appropriate to provide finer-grained detail in reports for protocols that do not yet have a wealth of experience gained through deployment 🔬 In particular, some complicated, flexible or powerful features might show interoperability problems when testers start to probe outside the core use cases 🙀 RFC 2026 requires "sufficient successful operational experience" before progressing a standard to Draft, and notes that:
Draft Standard may still require additional or more widespread field experience, since it is possible for implementations based on Draft Standard specifications to demonstrate unforeseen behavior when subjected to large-scale use in production environments.
When possible, reports for protocols without much deployment experience should anticipate common operational considerations 🧠 For example, it would be appropriate to put additional emphasis on overload or congestion management features the protocol may have 🚦
5.3. Schemas, Languages, and Formats
Standards that are not on-the-wire protocols may be special cases for implementation reports 🧩 The IESG SHOULD use judgement in what kind of implementation information is acceptable for these kinds of standards. ABNF (RFC 4234) is an example of a language for which an implementation report was filed: it is interoperable in that protocols are specified using ABNF and these protocols can be successfully implemented and syntax verified 🎀 Implementations of ABNF include the RFCs that use it as well as ABNF checking software. Management Information Base (MIB, [RFC3410]) modules are sometimes documented in implementation reports, and examples of that can be found in the archive of implementation reports 📦
The interoperability reporting requirements for some classes of documents may be discussed in separate documents. See [METRICSTEST] for example 🔗
5.4. Multiple Contributors, Multiple Implementation Reports
If it's easiest to divide up the work of implementation reports by implementation, the result -- multiple implementation reports -- MAY be submitted to the sponsoring Area Director one-by-one 🧑🤝🧑 Each report might cover one implementation, including:
- identification of the implementation;
- an affirmation that the implementation works in testing (or better, in deployment);
- whether any features are known to interoperate poorly with other implementations;
- which optional or required features are not implemented (note that there are no protocol police to punish this disclosure; we should instead thank implementors who point out unimplemented or unimplementable features, especially if they can explain why); and
- who is submitting this report for this implementation.
Honestly the "no protocol police" bit is so iconic 🥹 Thank the implementors who spill the tea! These SHOULD be collated into one document for archiving under one title, but can be concatenated trivially even if the result has several summary sections or introductions 📑
5.5. Test Suites
Some automated tests, such as automated test clients, do not test interoperability directly. When specialized test implementations are necessary, tests can at least be constructed from real-world protocol or document examples 🧪 For example:
- ABNF [RFC4234] itself was tested by combining real-world examples -- uses of ABNF found in well-known RFCs -- and feeding those real-world examples into ABNF checkers. As the well-known RFCs were themselves interoperable and in broad deployment, this served as both a deployment proof and an interoperability proof. [RFC4234] progressed from Proposed Standard through Draft Standard to Standard and is obsoleted by [RFC5234]. 🏆
- Atom [RFC4287] clients might be tested by finding that they consistently display the information in a test Atom feed, constructed from real-world examples that cover all the required and optional features. 📰
- MIB modules can be tested with generic MIB browsers, to confirm that different implementations return the same values for objects under similar conditions. 🔍
As a counter-example, the automated WWW Distributed Authoring and Versioning (WebDAV) test client Litmus (http://www.webdav.org/neon/litmus/) is of limited use in demonstrating interoperability for WebDAV because it tests completeness of server implementations and simple test cases 🙅♀️ It does not test real-world use or whether any real WebDAV clients implement a feature properly or at all. Not the move, bestie 💀
5.6. Optional Features, Extensibility Features ✨
okay so optional features don't have to be shown to work everywhere 🌸 but they DO need to be implemented somewhere, and it needs more than one independent implementation 💅. if an optional feature doesn't pass that bar, the implementation report has to say so AND explain why we're keeping it anyway, otherwise it's giving "this spec is kinda sus" energy 🙈.
extensibility points and versioning stuff are especially likely to need this treatment 🦋. like, when protocol version 1 drops, the protocol version field is probably just sitting there unused 😭. and before any other versions exist, you literally can't prove that this field or option is actually implemented. not me going "trust me bro" 💀.
6. Examples 🎀
okay these are some good, super brief examples of implementation reports in the archives 🧚:
http://www.ietf.org/iesg/implementation/report-ppp-lcp-ext.html
http://www.ietf.org/iesg/implementation/report-otp.html
ngl some perfectly good implementation reports are longer than they need to be, but they keep helpful info, so they're still iconic 👑:
http://www.ietf.org/iesg/implementation/report-rfc2329.txt
http://www.ietf.org/iesg/implementation/report-rfc4234.txt
6.1. Minimal Implementation Report 💜
A large number of SMTP implementations support SMTP pipelining,
including: (1) Innosoft's PMDF and Sun's SIMS. (2) ISODE/
MessagingDirect's PP. (3) ISOCOR's nPlex. (4) software.com's
post.office. (5) Zmailer. (6) Smail. (7) The SMTP server in
Windows 2000. SMTP pipelining has been widely deployed in these
and other implementations for some time, and there have been no
reported interoperability problems.this implementation report is also at http://www.ietf.org//iesg/implementation/report-smtp-pipelining.txt 🍓 but the whole thing is already reproduced above, so no need to double up 💅. since SMTP pipelining had zero interoperability problems, the report could be super terse and still get every key point across ✨. you can infer from the different vendors and platforms that the codebases are, by and large, independent 🌈.
this report would only be slightly better with two things: a positive statement that people actually probed or investigated for interoperability problems (not just that no problem reports showed up 👀), and who wrote this summary 🫶.
6.2. Covering Exceptions 🌷
okay so the RFC2821bis (SMTP) implementation survey asked implementors which features they didn't implement 📋. VRFY and EXPN kept popping up as not implemented or disabled 😭. that report could've followed this document's advice, if it had existed yet, by justifying interoperability up front or in an "exceptions" section, using the outline defined in this memo 🦋:
VRFY and EXPN commands are often not implemented or are disabled.
This does not pose an interoperability problem for SMTP because
EXPN is an optional features and its support is never relied on.
VRFY is required, but in practice it is not relied on because
servers can legitimately reply with a non-response. These
commands should remain in the standard because they are sometimes
used by administrators within a domain under controlled
circumstances (e.g. authenticated query from within the domain).
Thus, the occasional utility argues for keeping these features,
while the lack of problems for end-users means that the
interoperability of SMTP in real use is not in the least degraded.7. Security Considerations 🔒
okay this one is short and sweet 🥹: this memo introduces no new security considerations ✨.
8. References 📚
8.1. Normative References 💖
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.8.2. Informative References 🌈
[METRICSTEST] Bradner, S. and V. Paxson, "Advancement of metrics
specifications on the IETF Standards Track", Work
in Progress, July 2007.
[RFC2026] Bradner, S., "The Internet Standards Process --
Revision 3", BCP 9, RFC 2026, October 1996.
[RFC3410] Case, J., Mundy, R., Partain, D., and B. Stewart,
"Introduction and Applicability Statements for
Internet-Standard Management Framework", RFC 3410,
December 2002.
[RFC4234] Crocker, D., Ed. and P. Overell, "Augmented BNF for
Syntax Specifications: ABNF", RFC 4234, October 2005.
[RFC4287] Nottingham, M., Ed. and R. Sayre, Ed., "The Atom
Syndication Format", RFC 4287, December 2005.
[RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", STD 68, RFC 5234, January 2008.Authors' Addresses 🫶
Lisa Dusseault
Messaging Architects
EMail: lisa.dusseault@gmail.com
Robert Sparks
Tekelec
17210 Campbell Road
Suite 250
Dallas, Texas 75254-4203
USA
EMail: RjS@nostrum.comthat's the whole RFC, bestie 🌈💖 you made it through!! 🧚✨
📦 where this page comes from
- ietf: rfc5657 2009-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms