RFC 6390

RFC 6390 ✨: Guidelines for Considering New Performance Metric Development

Okay bestie, here's the title block 💖

• RFC number: 6390
• Title: Guidelines for Considering New Performance Metric Development
• Authors: A. Clark (Telchemy Incorporated) and B. Claise (Cisco Systems, Inc.)
• Category: Best Current Practice (BCP: 170)
• ISSN: 2070-1721
• Date: October 2011
• Obsoletes / Updates: the title block doesn't list any RFCs this one obsoletes or updates 🦋

Abstract 🌸

Okay so this doc is basically a framework and a process for developing Performance Metrics of protocols and applications that ride on top of IETF-specified protocols. 💅 These metrics can be used to characterize traffic on live networks and services, which is honestly the whole point. 🔥

Status of This Memo 📜

This memo documents an Internet Best Current Practice. 🧚

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on BCPs is available in Section 2 of RFC 5741. ✨

Info about the current status of this document, any errata, and how to send feedback on it can be found at http://www.rfc-editor.org/info/rfc6390. 💜

Copyright Notice 💖

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Simplified BSD License. 🎀

This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English. 🫶

Table of Contents 🦋

   1. Introduction ....................................................4
      1.1. Background and Motivation ..................................4
      1.2. Organization of This Document ..............................5
   2. Terminology .....................................................5
      2.1. Requirements Language ......................................5
      2.2. Performance Metrics Directorate ............................5
      2.3. Quality of Service .........................................5
      2.4. Quality of Experience ......................................6
      2.5. Performance Metric .........................................6
   3. Purpose and Scope ...............................................6
   4. Relationship between QoS, QoE, and Application-Specific
      Performance Metrics .............................................7
   5. Performance Metrics Development .................................7
      5.1. Identifying and Categorizing the Audience ..................7
      5.2. Definitions of a Performance Metric ........................8
      5.3. Computed Performance Metrics ...............................9
           5.3.1. Composed Performance Metrics ........................9
           5.3.2. Index ..............................................10
      5.4. Performance Metric Specification ..........................10
           5.4.1. Outline ............................................10
           5.4.2. Normative Parts of Performance Metric Definition ...11
           5.4.3. Informative Parts of Performance Metric
                  Definition .........................................13
           5.4.4. Performance Metric Definition Template .............14
           5.4.5. Example: Loss Rate .................................15
      5.5. Dependencies ..............................................16
           5.5.1. Timing Accuracy ....................................16
           5.5.2. Dependencies of Performance Metric Definitions on
                  Related Events or Metrics ..........................16
           5.5.3. Relationship between Performance Metric and
                  Lower-Layer Performance Metrics ....................17
           5.5.4. Middlebox Presence .................................17
      5.6. Organization of Results ...................................17
      5.7. Parameters: The Variables of a Performance Metric .........18
   6. Performance Metric Development Process .........................18
      6.1. New Proposals for Performance Metrics .....................18
      6.2. Reviewing Metrics .........................................19
      6.3. Performance Metrics Directorate Interaction with
           Other WGs .................................................19
      6.4. Standards Track Performance Metrics .......................20
   7. Security Considerations ........................................20
   8. Acknowledgements ...............................................20
   9. References .....................................................21
      9.1. Normative References ......................................21
      9.2. Informative References ....................................21

1. Introduction ✨

Okay so, lots of networking technologies, applications, and services are distributed in nature, and their performance can get messed up by IP impairments, server capacity, congestion, and other factors. 😭 That's why it's so important to measure performance: you need to make sure quality objectives are being met and you need to be able to diagnose problems. Standardized metrics keep performance measurement consistent across implementations, which makes it way easier to interpret and compare results. 💅

There are at least three phases in the development of performance standards. They are:

1. Definition of a Performance Metric and its units of measure
2. Specification of a method of measurement
3. Specification of the reporting format

While developing metrics, it's often handy to define performance objectives and expected value ranges. This extra info usually isn't part of the formal spec of the metric, but it's super useful background for implementers and users of the metric. 🌸

The intended audience for this document includes, but is not limited to, IETF participants who write Performance Metrics documents, reviewers of those documents, and members of the Performance Metrics Directorate. 🧚

1.1. Background and Motivation 🦋

Previous IETF work on reporting application Performance Metrics includes "Real-time Application Quality-of-Service Monitoring (RAQMON) Framework" [RFC4710]. This framework extends the remote network monitoring (RMON) family of specifications so you can do real-time quality-of-service (QoS) monitoring of various applications running on devices like IP phones, pagers, Instant Messaging clients, mobile phones, and other handheld computing devices. 📱 Then there's "RTP Control Protocol Extended Reports (RTCP XR)" [RFC3611] and "Session Initiation Protocol Event Package for Voice Quality Reporting" [RFC6035], which define protocols that support real-time Quality of Experience (QoE) reporting for Voice over IP (VoIP) and other applications on devices like IP phones and mobile handsets. 🎀

The IETF is also really active in developing reliable transport protocols, like TCP [RFC0793] or the Stream Control Transmission Protocol (SCTP) [RFC4960], and those would affect how IP performance relates to application performance. 👑

So yeah, there's a gap in the currently chartered coverage of IETF Working Groups (WGs): nobody's currently developing Performance Metrics for protocols above and below the IP layer that can be used to characterize performance on live networks. 🙈

Similar to "Guidelines for Considering Operations and Management of New Protocols and Protocol Extensions" [RFC5706], which is the reference document for the IETF Operations Directorate, this document should be consulted as part of the new Performance Metric review by the members of the Performance Metrics Directorate. 💜

1.2. Organization of This Document 🍓

This document is split into two major sections after the "Purpose and Scope" section. The first one is a definition and description of a Performance Metric and its key aspects. The second one defines a process for developing these metrics that works in the IETF environment. 🌈

2. Terminology 💖

2.1. Requirements Language ✨

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 🫶

2.2. Performance Metrics Directorate 👑

The Performance Metrics Directorate is a directorate that provides guidance for Performance Metrics development in the IETF. 🧚

The Performance Metrics Directorate should be composed of experts in the performance community, potentially selected from the IP Performance Metrics (IPPM), Benchmarking Methodology (BMWG), and Performance Metrics for Other Layers (PMOL) WGs. 🦋

2.3. Quality of Service 🎀

Quality of Service (QoS) is defined in a way similar to the ITU "Quality of Service (QoS)" section of [E.800], i.e., "Totality of characteristics of a telecommunications service that bear on its ability to satisfy stated and implied needs of the user of the service". 💅

2.4. Quality of Experience 🌸

Quality of Experience (QoE) is defined in a way similar to the ITU "QoS experienced/perceived by customer/user (QoSE)" section of [E.800], i.e., "a statement expressing the level of quality that customers/users believe they have experienced". 🥹

NOTE 1 - The level of QoS experienced and/or perceived by the customer/user may be expressed by an opinion rating. ⭐

NOTE 2 - QoE has two main components: quantitative and qualitative. The quantitative component can be influenced by the complete end-to-end system effects (including user devices and network infrastructure). 🔥

NOTE 3 - The qualitative component can be influenced by user expectations, ambient conditions, psychological factors, application context, etc. 🌈

NOTE 4 - QoE may also be considered as QoS delivered, received, and interpreted by a user with the pertinent qualitative factors influencing his/her perception of the service. 💜

2.5. Performance Metric 🍓

A Performance Metric is a quantitative measure of performance, specific to an IETF-specified protocol or specific to an application transported over an IETF-specified protocol. Examples are the FTP response time for a complete file download, the DNS response time to resolve the IP address, a database logging time, etc. 🧚✨

3. Purpose and Scope ✨

okay so bestie, this is the part where the doc tells you what it's actually for 💅 The purpose is to define a framework and a process for developing Performance Metrics for protocols above and below the IP layer (like IP-based applications that run over reliable or datagram transport protocols) 🦋 These metrics can be used to characterize traffic on live networks and services. BUT, and this is the lowkey plot twist, this document does not define any Performance Metrics itself 🙈 it's the rulebook, not the actual metrics!!

The scope covers guidelines for Performance Metrics Directorate members when they're considering new Performance Metrics, and it suggests how the Directorate will interact with the rest of the IETF 🎀 However, this document is NOT trying to replace existing working methods in WGs that already have chartered work in this area. no stepping on toes, periodt 💜

This process is not meant to govern Performance Metric development in existing IETF WGs that are focused on metrics development, like IPPM and BMWG 🌸 However, this guidelines document may be useful in those activities and MAY be applied where appropriate ✨ A typical example is developing Performance Metrics to be exported with the IP Flow Information eXport (IPFIX) protocol [RFC5101], using specific IPFIX information elements [RFC5102], which would benefit from the framework in this document 🧚

The framework in this document applies to Performance Metrics derived from both active and passive measurements 🫶

4. Relationship between QoS, QoE, and Application-Specific Performance Metrics 💖

okay this section is SO good because it untangles three things that people mix up constantly 😭 Network QoS is about network and network protocol performance, while QoE is about assessing a user's experience in the context of a task or a service 🌈 Application-specific Performance Metrics cover measurement at layers between IP and the user 💅

Here's an example. Network QoS metrics (packet loss, delay, and delay variation [RFC5481]) can be used to estimate application-specific Performance Metrics (de-jitter buffer size and RTP-layer packet loss) 🎀 Then you combine those with other known aspects of a VoIP application (like codec type) using an algorithm compliant with ITU-T P.564 [P.564] to estimate a Mean Opinion Score (MOS) [P.800] ✨ But the QoE for a particular VoIP user depends on the specific context, like a casual conversation, a business conference call, or an emergency call 🙈 The vibe changes completely!!

Finally, QoS and application-specific Performance Metrics are quantitative, while QoE is qualitative 💜 Also, network QoS and application-specific Performance Metrics can be directly or indirectly evident to the user, while QoE is directly evident 🦋

5. Performance Metrics Development 🌸

this section is literally the key definitions and qualifications of Performance Metrics, so pay attention bestie 🧚

5.1. Identifying and Categorizing the Audience 🎀

okay so the big thing here is that lots of the stuff about metric definition and reporting, even picking the essential metrics, depends on who's going to use the results and what for 💅 For example, the metric description SHOULD include use cases and example reports that illustrate service quality monitoring and maintenance, or identification and quantification of problems ✨

All documents defining Performance Metrics SHOULD identify the primary audience and its associated requirements 🫶 The audience can influence both the definition of metrics and the methods of measurement 💖

The key areas of variation between different metric users include:

• Suitability of passive measurements of live traffic or active measurements using dedicated traffic
• Measurement in a laboratory environment or on a network of deployed devices
• Accuracy of the results
• Access to measurement points and configuration information
• Measurement topology (point-to-point, point-to-multipoint)
• Scale of the measurement system
• Measurements conducted on-demand or continuously
• Required reporting formats and periods
• Sampling criteria [RFC5474], such as systematic or probabilistic
• Period (and duration) of measurement, as live traffic can have patterns 🌈

5.2. Definitions of a Performance Metric 🦋

okay, a Performance Metric is a measure of an observable behavior of a networking technology, an application, or a service 💅 Most of the time it can be directly measured, but sometimes the value is computed 🙈 The process for determining the value may assume an implicit or explicit underlying statistical process. In that case, the Performance Metric is an estimate of a parameter of that process, assuming the statistical process closely models the behavior of the system 🧚

A Performance Metric should serve some defined purposes ✨ That can include measuring capacity, quantifying how bad some problems are, measuring service level, diagnosing or locating problems, and other uses like that 🌸 A Performance Metric may also be an input to other processes, like computing a composite Performance Metric or modeling or simulating a system 💜

Tests of the "usefulness" of a Performance Metric include:

(i) the degree to which its absence would cause significant loss of information on the behavior or performance of the application or system being measured

(ii) the correlation between the Performance Metric, the QoS, and the QoE delivered to the user (person or other application)

(iii) the degree to which the Performance Metric is able to support the identification and location of problems affecting service quality

(iv) the requirement to develop policies (Service Level Agreement, and potentially Service Level Contract) based on the Performance Metric

For example, say a distributed application is running over a network connection that's subject to packet loss 😭 A Packet Loss Rate (PLR) Performance Metric is defined as the mean packet loss ratio over some time period 🎀 If the application performs poorly over connections with a high packet loss ratio and always performs well when the packet loss ratio is zero, then the PLR Performance Metric is useful to some degree ✨ But some applications are sensitive to short periods of high loss (bursty loss) and are pretty relatively insensitive to isolated packet loss events. For that type of application, there would be very weak correlation between PLR and application performance 🙈 A "better" Performance Metric would consider both the packet loss ratio and the distribution of loss events 💅 If application performance degrades when the PLR exceeds some rate, then a useful Performance Metric may be a measure of the duration and frequency of periods during which the PLR exceeds that rate (as, for example, in RFC 3611) 🦋

5.3. Computed Performance Metrics 🧚

5.3.1. Composed Performance Metrics 💖

some Performance Metrics can't be measured directly, but can be composed from base metrics that have already been measured 🌈 A composed Performance Metric is derived from other metrics by applying a deterministic process or function (like a composition function) 💅 The process may use metrics that are identical to the metric being composed, metrics that are dissimilar, or some combination of both 🫶 Usually the base metrics have a limited scope in time or space, and they can be combined to estimate the performance of some larger entities ✨

Some examples of composed Performance Metrics and their definitions are:

Spatial composition is defined as the composition of metrics of the same type with differing spatial domains [RFC5835] [RFC6049]. Ideally, for spatially composed metrics to be meaningful, the spatial domains should be non-overlapping and contiguous, and the composition operation should be mathematically appropriate for the type of metric.

Temporal composition is defined as the composition of sets of metrics of the same type with differing time spans [RFC5835]. For temporally composed metrics to be meaningful, the time spans should be non-overlapping and contiguous, and the composition operation should be mathematically appropriate for the type of metric.

Temporal aggregation is a summarization of metrics into a smaller number of metrics that relate to the total time span covered by the original metrics. An example would be to compute the minimum, maximum, and average values of a series of time-sampled values of a metric.

In the context of flow records in IP Flow Information eXport (IPFIX), the IPFIX Mediation Framework [RFC6183], based on "IP Flow Information Export (IPFIX) Mediation: Problem Statement" [RFC5982], also discusses some aspects of temporal and spatial composition 🌸

5.3.2. Index 🎀

An index is a metric for which the output value range has been selected for convenience or clarity, and whose behavior is selected to support ease of understanding, for example the R Factor [G.107] 💅 The deterministic function for an index is often developed after the index range and behavior have been determined 🦋

5.4. Performance Metric Specification 👑

5.4.1. Outline ⭐

okay the last bit of this part is the outline rule 🫶 A Performance Metric definition MUST have a normative part that defines what the metric is and how it is measured or computed 💖 It SHOULD also have an informative part that describes the Performance Metric and its application ✨

5.4.2. Normative Parts of Performance Metric Definition ✨

okay so the normative part of a Performance Metric definition MUST define at least the following 💅

(i) Metric Name 🏷️

Metric names are RECOMMENDED to be unique within the set of metrics being defined for the protocol layer and context. Strict uniqueness might not be possible (the IPPM registry [RFC6248] is a good example of an IANA metric registry that doesn't give enough specificity 🙈), so you need to get broad review to avoid naming overlap. The Performance Metrics Directorate can help you pick a unique name for IANA metric registration. The Performance Metric name MAY be descriptive 🎀

(ii) Metric Description 📝

The description MUST explain what the metric is, what's being measured, and how it relates to the performance of the system being measured. This is the part that makes it make sense, bestie 🫶

(iii) Method of Measurement or Calculation 🧮

This MUST define what's being measured or computed, plus the specific algorithm you're using. Does it involve active measurements, passive ones, or both? 👀 Words like "average" need a qualifier (for example, a running average or an average over some interval) or it gets super confusing. Exception cases SHOULD also be defined, with the right handling method. Packet loss metrics are a classic mess here. They often don't say how a packet gets labeled as lost (versus just really delayed) or how duplicate packets get handled. Like, if you report the average PLR over a time interval, and a packet shows up late from one interval into the next, was it "lost" in the interval where it should have arrived, or should it count as received? 😭

Some calculation methods need you to throw out some collected data (because of outliers) so the measurement parameters actually mean something. Burstable billing is the example here: it sorts the 5-min samples and discards the top 5 percentile 💅

Some parameters linked to the method MAY also be reported, so people can fully interpret the Performance Metric. Think the time interval, the load, the minimum packet loss, the potential measurement errors and where they come from, the attainable accuracy of the metric (like +/- 0.1), the method of calculation, and so on 🧠

(iv) Units of Measurement 📏

The units MUST be clearly stated. No vibes-based units, periodt 💖

(v) Measurement Point(s) with Potential Measurement Domain 📍

If the measurement is specific to a measurement point, this SHOULD be defined. The measurement domain MAY also be defined. If your measurement points are spread across domains, then whether it's intra- or inter-domain is another thing to consider 🌍

The definition should also talk about how the value might change depending on which measurement point you pick. For example, the time between a SIP request [RFC3261] and the final response can be really different at the User Agent Client (UAC) than at the User Agent Server (UAS) 🤯

Sometimes you need multiple measurement points. In that case, all measurement points SHOULD be defined, including the measurement domain(s) 🦋

(vi) Measurement Timing ⏱️

The acceptable range of timing intervals or sampling intervals, plus the timing accuracy you need for those intervals, MUST be specified. Short sampling intervals or frequent samples give you a rich source of info for judging application performance, but they can produce way too much measurement data 😵 Long measurement or sampling intervals cut down on the data you report and collect, which can leave you without enough detail to understand app performance or service quality, especially since the measured quantity can change a lot over time 🌊

With multiple measurement points, the need for synchronized clocks must be clearly specified. For the IP delay variation application metric specifically, the different aspects of synchronized clocks are covered in [RFC5481] ⏰

5.4.3. Informative Parts of Performance Metric Definition 🌸

The informative part of a Performance Metric specification exists to help people implement and use the metric. This part SHOULD provide the following data 🧚

(i) Implementation 🛠️

The implementation description MAY be text, an algorithm, or example software. The goal here is to help implementers get consistent results, which is honestly the whole point 💜

(ii) Verification ✅

The definition SHOULD give guidance on verification testing. That can be test vectors, a formal verification test method, or informal advice. Any of these works, no judgment 🍓

(iii) Use and Applications 🔍

This section helps the "user" figure out how, when, and where the metric can be applied, and what the value range actually means. This MAY include a definition of the "typical" and "abnormal" range of the Performance Metric, if that isn't obvious from the metric's nature. The description MAY also include info about how extreme measurement values affect things, like whether the metric is sensitive to outliers. The Use and Application section SHOULD also include the security implications in the description 🔐

Here are some examples:

(a) Honestly, it's pretty intuitive that a lower packet loss ratio means better performance. But the user might not know what a given packet loss ratio actually signifies 🤷‍♀️

(b) The speech level of a telephone signal is usually expressed in dBm0. If the user is shown this:

Speech level = -7 dBm0

that's not intuitive at all unless the user is a telephony expert 😩 But if the metric definition explains that the typical range is -18 to -28 dBm0, then a value higher than -18 means the signal may be too high (loud), and a value below -28 means it may be too low (quiet). Way easier to read now, right? ✨

(iv) Reporting Model 📊

The reporting model definition makes any relationship between the metric and the reporting model clear. There are often implied relationships between how you report metrics and the metric itself, but implementers usually don't get told about them 👀 For example, say the metric is a short-term running average packet delay variation (like the inter-arrival jitter in [RFC3550]), and it gets reported every 6-10 seconds. The result might have limited accuracy when packet delay variation is non-stationary 📉

5.4.4. Performance Metric Definition Template 📋

Normative:

• Metric Name
• Metric Description
• Method of Measurement or Calculation
• Units of Measurement
• Measurement Point(s) with Potential Measurement Domain
• Measurement Timing

Informative:

• Implementation
• Verification
• Use and Applications
• Reporting Model

5.4.5. Example: Loss Rate 🎀

Okay, here's the example, and it's the loss rate metric from RFC 3611 [RFC3611]. The fields look like this:

Metric Name:  LossRate

Metric Description:  The fraction of RTP data packets from the source
   lost since the beginning of reception.

Method of Measurement or Calculation:  This value is calculated by
   dividing the total number of packets lost (after the effects of
   applying any error protection, such as Forward Error Correction
   (FEC)) by the total number of packets expected, multiplying the
   result of the division by 256, limiting the maximum value to 255
   (to avoid overflow), and taking the integer part.

Units of Measurement:  This metric is expressed as a fixed-point
   number with the binary point at the left edge of the field.  For
   example, a metric value of 12 means a loss rate of
   approximately 5%.

Measurement Point(s) with Potential Measurement Domain:  This metric
   is made at the receiving end of the RTP stream sent during a Voice
   over IP call.

Measurement Timing:  This metric can be used over a wide range of
   time intervals.  Using time intervals of longer than one hour may
   prevent the detection of variations in the value of this metric
   due to time-of-day changes in network load.  Timing intervals
   should not vary in duration by more than +/- 2%.

Implementation:  The numbers of duplicated packets and discarded
   packets do not enter into this calculation.  Since receivers
   cannot be required to maintain unlimited buffers, a receiver MAY
   categorize late-arriving packets as lost.  The degree of lateness
   that triggers a loss SHOULD be significantly greater than that
   which triggers a discard.

Verification:  The metric value ranges between 0 and 255.

Use and Applications:  This metric is useful for monitoring VoIP
   calls, more precisely, to detect the VoIP loss rate in the
   network.  This loss rate, along with the rate of packets discarded
   due to jitter, has some effect on the quality of the voice stream.

Reporting Model:  This metric needs to be associated with a defined
   time interval, which could be defined by fixed intervals or by a
   sliding window.  In the context of RFC 3611, the metric is
   measured continuously from the start of the RTP stream, and the
   value of the metric is sampled and reported in RTCP XR VoIP
   Metrics reports.

5.5. Dependencies 🔗

This section covers several Performance Metrics dependencies that the metric designer should keep in mind during development. These dependencies, plus any others not listed here, SHOULD be documented in the Performance Metric specifications 💖

5.5.1. Timing Accuracy ⏳

How accurate your measurement timing is can affect how accurate the Performance Metric is. For a sampled-value metric this may not matter much, but for an interval-based metric it will. Some metrics, like the number of events per time interval, are directly affected: a 10% variation in the time interval leads straight to a 10% variation in the measured value 📈 Other metrics, like the average packet loss ratio over some time interval, are affected to a lesser extent.

If you need to correlate sampled values or intervals, then the accuracy of sampling times and interval start/stop times is essential. It has to be good enough for the application (for example, +/- 2%) 🎯

5.5.2. Dependencies of Performance Metric Definitions on Related Events or Metrics 🧩

Performance Metric definitions may rely on factors, explicitly or implicitly, that aren't obvious. For example, recognizing a packet as "lost" depends on having some way of knowing the packet was actually lost (like an RTP sequence number), plus some time threshold after which a packet that hasn't arrived gets declared lost. It's really important to recognize these dependencies and build them into the metric definition 🫶

5.5.3. Relationship between Performance Metric and Lower-Layer Performance Metrics ✨

okay so lower-layer performance metrics can be used to compute or infer how higher-layer applications are performing 💅, potentially with an application performance model 🎀. how accurate that ends up being depends on a bunch of factors, like:

• (i) The completeness of the set of metrics (i.e., are there metrics for all the input values to the application performance model?)
• (ii) Correlation between input variables (being measured) and application performance
• (iii) Variability in the measured metrics and how this variability affects application performance

5.5.4. Middlebox Presence 🧚

a middlebox [RFC3303] like a proxy, network address translation (NAT), redirect server, session border controller (SBC) [RFC5853], or application layer gateway (ALG) can add variability to a measurement or restrict its scope 👀. for example, an SBC that doesn't process RTP loopback packets might block that traffic or terminate it locally instead of passing it through to its target. the drama is real 😭

5.6. Organization of Results 🌸

the IPPM Framework [RFC2330] sorts metric results into three related notions:

• singleton: an elementary instance, or "atomic" value.
• sample: a set of singletons with some common properties and some varying properties.
• statistic: a value derived from a sample through deterministic calculation, such as the mean.

Performance Metrics MAY use this organization for the results, with or without the term names used by the IPPM WG. Section 11 of RFC 2330 [RFC2330] should be consulted for further details. 📚

5.7. Parameters: the Variables of a Performance Metric 🦋

metrics are completely defined once all options and input variables have been identified and considered 💖. sometimes these variables are left unspecified in a metric definition, and their general name means the user has to set them and report them along with the results. the IPPM metric template calls these "parameters". the scope of the metric, the time it was run, the length interval of a sliding-window measurement, the timer settings, and the counter thresholds are all examples of parameters ✨.

All documents defining Performance Metrics SHOULD identify all key parameters for each Performance Metric. this one's important, bestie 🫶

6. Performance Metric Development Process 💜

6.1. New Proposals for Performance Metrics

this process adds more considerations on top of the processes for adopting new work described in RFC 2026 [RFC2026] and RFC 2418 [RFC2418] 🍓. new Performance Metrics work item proposals SHALL be approved using the existing IETF process. the following entry criteria get considered for each proposal 🔥.

Proposals SHOULD be prepared as Internet-Drafts, describing the Performance Metric and conforming to the qualifications above as much as possible. Proposals SHOULD be deliverables of the corresponding protocol development WG charters. as such, the proposals SHOULD be vetted by that WG before the Performance Metrics Directorate discusses them. this part of the process includes an assessment of the need for the proposed Performance Metric and an assessment of how much support there is for developing it in the IETF 🧚.

Proposals SHOULD include an assessment of interaction and/or overlap with work in other Standards Development Organizations (SDOs). Proposals SHOULD identify additional expertise that might be consulted. 🌈

Proposals SHOULD specify the intended audience and users of the Performance Metrics. the development process encourages participation from members of the intended audience 💅.

Proposals SHOULD identify any security and IANA requirements. security issues could potentially involve revealing data that identifies a user, or the potential misuse of active test tools 💀. IANA considerations may involve the need for a Performance Metrics registry. ✨

6.2. Reviewing Metrics

Each Performance Metric SHOULD be assessed according to the following list of qualifications 👑:

• Are the performance metrics unambiguously defined?
• Are the units of measure specified?
• Does the metric clearly define the measurement interval where applicable?
• Are significant sources of measurement errors identified and discussed?
• Does the method of measurement ensure that results are repeatable?
• Does the metric or method of measurement appear to be implementable (or offer evidence of a working implementation)?
• Are there any undocumented assumptions concerning the underlying process that would affect an implementation or interpretation of the metric?
• Can the metric results be related to application performance or user experience, when such a relationship is of value?
• Is there an existing relationship to metrics defined elsewhere within the IETF or within other SDOs?
• Do the security considerations adequately address denial-of-service attacks, unwanted interference with the metric/measurement, and user data confidentiality (when measuring live traffic)?

6.3. Performance Metrics Directorate Interaction with Other WGs

The Performance Metrics Directorate SHALL provide guidance to the related protocol development WG when it considers an Internet-Draft that specifies Performance Metrics for a protocol 🎀. a sufficient number of individuals with expertise must be willing to consult on the draft. if the related WG has already wrapped up, comments on the proposal should still be sought from key RFC authors and former chairs 🦋.

As with expert reviews performed by other directorates, a formal review is recommended by the time the document gets reviewed by the Area Directors or an IETF Last Call is being conducted 📚.

Existing mailing lists SHOULD be used; however, a dedicated mailing list MAY be started if necessary to help the work on a draft 💌.

In some cases, it makes sense to have the IETF session discussion during the related protocol WG session, so the effort gets more visibility with that WG and the review gets bigger 🌟.

6.4. Standards Track Performance Metrics

The Performance Metrics Directorate will help RFCs move along the Standards Track. See [IPPM-STANDARD-ADV-TESTING]. this can include preparing test plans to examine different implementations of the metrics, making sure the metric definitions are clear and unambiguous (depending on the final form of the draft mentioned above) 🧁.

7. Security Considerations 🔥

generally speaking, having a framework for Performance Metric development is not a security issue for the Internet. Performance Metric definitions, though, can introduce security issues, so this framework recommends that persons defining Performance Metrics should identify any such risk factors 🚨.

the security considerations that apply to any active measurement of live networks are relevant here. See [RFC4656].

the security considerations that apply to any passive measurement of specific packets in live networks are relevant here too. See the security considerations in [RFC5475]. 😱

8. Acknowledgements 🫶

the authors want to thank Al Morton, Dan Romascanu, Daryl Malas, and Loki Jorgenson for their comments and contributions, and Aamer Akhter, Yaakov Stein, Carsten Schmoll, and Jan Novak for their reviews 💖.

9. References 📚

9.1. Normative References

[RFC2026]  Bradner, S., "The Internet Standards Process --
           Revision 3", BCP 9, RFC 2026, October 1996.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2418]  Bradner, S., "IETF Working Group Guidelines and
           Procedures", BCP 25, RFC 2418, September 1998.

[RFC4656]  Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
           Zekauskas, "A One-way Active Measurement Protocol
           (OWAMP)", RFC 4656, September 2006.

9.2. Informative References ✨

Okay bestie, this is the reading list for the extra-credit crowd 📚🧚 Every entry below is a source the spec points to, not a must-read, so pick your fave and go 💅

• [E.800] "ITU-T Recommendation E.800. E SERIES: OVERALL NETWORK OPERATION, TELEPHONE SERVICE, SERVICE OPERATION AND HUMAN FACTORS", September 2008. 📞
• [G.107] "ITU-T Recommendation G.107. The E-model: a computational model for use in transmission planning", April 2009. 🧮
• [IPPM-STANDARD-ADV-TESTING] Geib, R., Ed., Morton, A., Fardid, R., and A. Steinmitz, "IPPM standard advancement testing", Work in Progress, June 2011. 🧪
• [P.564] "ITU-T Recommendation P.564. Conformance Testing for Voice over IP Transmission Quality Assessment Models", November 2007. 🎙️
• [P.800] "ITU-T Recommendation P.800. Methods for subjective determination of transmission quality", August 1996. 👂
• [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981. 🔌
• [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998. 📏
• [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002. ☎️
• [RFC3303] Srisuresh, P., Kuthan, J., Rosenberg, J., Molitor, A., and A. Rayhan, "Middlebox communication architecture and framework", RFC 3303, August 2002. 🛡️
• [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003. 🎥
• [RFC3611] Friedman, T., Ed., Caceres, R., Ed., and A. Clark, Ed., "RTP Control Protocol Extended Reports (RTCP XR)", RFC 3611, November 2003. 📊
• [RFC4710] Siddiqui, A., Romascanu, D., and E. Golovinsky, "Real-time Application Quality-of-Service Monitoring (RAQMON) Framework", RFC 4710, October 2006. 🧐
• [RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", RFC 4960, September 2007. 🌊
• [RFC5101] Claise, B., Ed., "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information", RFC 5101, January 2008. 📤
• [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, "Information Model for IP Flow Information Export", RFC 5102, January 2008. 🗂️
• [RFC5474] Duffield, N., Ed., Chiou, D., Claise, B., Greenberg, A., Grossglauser, M., and J. Rexford, "A Framework for Packet Selection and Reporting", RFC 5474, March 2009. 🎯
• [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F. Raspall, "Sampling and Filtering Techniques for IP Packet Selection", RFC 5475, March 2009. 🎲
• [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation Applicability Statement", RFC 5481, March 2009. ⏱️
• [RFC5706] Harrington, D., "Guidelines for Considering Operations and Management of New Protocols and Protocol Extensions", RFC 5706, November 2009. 🛠️
• [RFC5835] Morton, A., Ed., and S. Van den Berghe, Ed., "Framework for Metric Composition", RFC 5835, April 2010. 🧩
• [RFC5853] Hautakorpi, J., Ed., Camarillo, G., Penfield, R., Hawrylyshen, A., and M. Bhatia, "Requirements from Session Initiation Protocol (SIP) Session Border Control (SBC) Deployments", RFC 5853, April 2010. 🚧
• [RFC5982] Kobayashi, A., Ed., and B. Claise, Ed., "IP Flow Information Export (IPFIX) Mediation: Problem Statement", RFC 5982, August 2010. 🧠
• [RFC6035] Pendleton, A., Clark, A., Johnston, A., and H. Sinnreich, "Session Initiation Protocol Event Package for Voice Quality Reporting", RFC 6035, November 2010. 🎧
• [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of Metrics", RFC 6049, January 2011. 🗺️
• [RFC6183] Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi, "IP Flow Information Export (IPFIX) Mediation: Framework", RFC 6183, April 2011. 🖼️
• [RFC6248] Morton, A., "RFC 4148 and the IP Performance Metrics (IPPM) Registry of Metrics Are Obsolete", RFC 6248, April 2011. 🗑️

---

Authors' Addresses 🎀

Here's where you can find the authors, bestie 💖

Alan Clark
Telchemy Incorporated
2905 Premiere Parkway, Suite 280
Duluth, Georgia  30097
USA

EMail: alan.d.clark@telchemy.com


Benoit Claise
Cisco Systems, Inc.
De Kleetlaan 6a b1
Diegem  1831
Belgium

Phone: +32 2 704 5622
EMail: bclaise@cisco.com