RFC 2721

RTFM: Applicability Statement

📅 published October 1999, from ietf

Okay bestie, let's get into RFC 2721 🎀✨

RTFM: Applicability Statement 🦋

It's from the Network Working Group, written by N. Brownlee of The University of Auckland. It's Category: Informational (so it's giving "here's some info," not "this is a standard" 💅), and it's dated October 1999. Nothing in the header says it obsoletes or updates another RFC.

Status of this Memo 📜

This memo is here to give information to the Internet community 🌸 It does not specify an Internet standard of any kind, and distribution of this memo is unlimited.

Copyright (C) The Internet Society (1999). All Rights Reserved.

Abstract 💜

This doc gives you the full overview of Realtime Traffic Flow Measurement, covering where it applies and where it falls short 🧚

Table of Contents 🗂️

   1  The RTFM Documents . . . . . . . . . . . . . . . . . . . . . .  2
   2  Brief Technical Specification (TS) . . . . . . . . . . . . . .  3
   3  Applicability Statement (AS) . . . . . . . . . . . . . . . . .  3
   4  Limitations  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   5  Security Considerations  . . . . . . . . . . . . . . . . . . .  5
   6  Policy Considerations  . . . . . . . . . . . . . . . . . . . .  6
   7  Soundness  . . . . . . . . . . . . . . . . . . . . . . . . . .  6
   8  Appendix A: WG Report on the Meter MIB . . . . . . . . . . . .  8
   9  References . . . . . . . . . . . . . . . . . . . . . . . . . .  9
   10 Author's Address . . . . . . . . . . . . . . . . . . . . . . .  9
   11 Full Copyright Statement . . . . . . . . . . . . . . . . . . . 10

1 The RTFM Documents 📚

The RTFM Traffic Measurement System was built by the Realtime Traffic Flow Measurement Working Group 🙌 It's explained across six other documents, which go like this:

  • [ACT-BKG] Internet Accounting: Background (Informational)
    • This one lays out the requirements for a usage reporting system for network traffic 📊 It sketches the RTFM Architecture (meters, meter readers and managers), which allows for multiple meters and meter readers, with asynchronous reading from the meters. It also proposes ways of classifying traffic flows, says flows need to be bi-directional (with separate sets of counters for each direction), and says each packet needs to be counted in a single flow (the 'count in one bucket' principle) 🪣
  • [RTFM-ARC] RTFM Architecture (Informational)
    • This is the architecture doc, describing each component 🏗️ It explains how traffic flows are seen as logical entities described by their address-attribute values, so each flow is defined by the attributes of its end-points. It also gives a detailed description of the RTFM traffic meter, including how flows are stored in the meter's flow table and how packets are matched against rules stored in a ruleset.
  • [RTFM-MIB] RTFM Meter MIB (Proposed Standard)
    • This one describes the SNMP Management Information Base for an RTFM meter 🧠 That covers its flow table, its rule table (which stores the meter's rulesets), and the control tables used for managing a meter and reading flow data from it.
  • [RTFM-SRL] SRL: A Language for Describing Traffic Flows and Specifying Actions for Flow Groups (Informational)
    • An RTFM ruleset is an array of rules. The meter uses it to decide which flows are of interest, which end-point is the flow source, and how much detail (i.e. what attribute values) must be saved for each flow 🔍 SRL is a high-level language that gives you a clear, logical way to write rulesets. It should also be useful for other applications that select flows and perform actions on them, e.g. packet-marking gateways, RSVP policy agents, etc. 🦋
  • [RTFM-NEW] RTFM New Attributes (Experimental)
    • Users have shown a lot of interest in extending the RTFM Architecture so a meter can report on more flow-related measures ✨ This RFC documents work on specifying those measures (the 'new' attributes) and reports on experience implementing them.
  • [RTFM-NTM] RTFM: Experiences with NeTraMet (Informational)
    • NeTraMet is a free software implementation of the RTFM Architecture, available since 1993 🎉 This RFC records RTFM implementation experience gained with NeTraMet up to late 1996. One especially important result: groups of rules that test the same attribute using the same mask can be implemented as a single hashed comparison. That lets the meter quickly figure out whether a packet belongs to one of a large number of networks 🤯

2 Brief Technical Specification (TS) 💖

RTFM lets you measure network traffic 'flows', i.e.

  • a method of specifying traffic flows within a network
  • a hierarchy of devices (meters, meter readers, managers) for measuring the specified flows
  • a mechanism for configuring meters and meter readers, and for collecting the flow data from remote meters

RTFM gives you high time resolution for flow first- and last-packet times ⏱️ Counters for long-duration flows can be read at intervals decided by a manager. The RTFM Meter is designed to do as much data reduction work as possible, which keeps the amount of data to read, and the processing needed to make useful reports from it, as small as it can be 🍓

RTFM flow data can be used for all kinds of things, like usage accounting, long-term recording of network usage (classified by IP address attributes), and real-time analysis of traffic flows at remote metering points 📈

3 Applicability Statement (AS) 🌈

To use RTFM for collecting network traffic information, you first have to think about where in the network the traffic flows will be measured 🗺️ Once that's decided, an RTFM Meter must be installed at each chosen measurement point.

You need at least one Meter Reader to collect the measured data from the meters, plus a single Manager to control the meters and meter readers 🫶

RTFM Meters can be single- or multi-user hosts running a meter program (one is available as free software, and a second is under development at IBM Research) 💻 Or meters could run as firmware in switches or routers 🔌 A hybrid approach, where an RTFM meter takes raw traffic data from a router, is another useful implementation path.

RTFM Managers are programs running on a host, talking to meters and meter readers over the network 🌐 For this, meters are SNMP agents implementing the RTFM Meter MIB, and managers are SNMP clients that use the Meter MIB to store and access the flow data 💅

4 Limitations 🫧

RTFM is designed to measure traffic flows for traffic passing a point in a network 🚦 If packets for a flow pass the metering point in both directions, the meter will match them up and provide counters for each direction. If packets only pass in one direction, the meter can only provide counts for that direction.

Users of RTFM should note that installing meters, meter readers and managers only gives you the capability to collect flow data 🙈 You'll still need more installation work to develop configuration files (RTFM rulesets) for each meter, data processing applications to analyse the flow data, and various scripts, cron jobs, etc. so you end up with a useful production-quality measurement system that suits your particular needs 🛠️

One of RTFM's strengths is that it can collect flow data at whatever level of detail (or 'granularity') you need ✨ It's tempting to just collect 'all possible data', but there are severe resource constraints 😭 If you try to save the complete address-attribute value for all attributes of every possible flow, a very large amount of data can be produced really fast, and the meter only has a finite amount of memory for its flow table. A better approach is to save the minimum amount of data needed to reach your measurement system goals 🧁

For example, if you want to collect usage data to bill subscribers identified by their IP address, you could just save the full IP address and nothing more 💸 The RTFM meter would produce flow data for each subscriber IP address, with PDU and Octet counts for data sent and received, which is the minimum needed to produce bills. In practice you'd probably want to save at least part of the Destination IP address too, which lets you produce usage logs showing subscriber activity over time 📅

The simplest way to figure out how much detail you can collect is to create an initial ruleset that collects the minimum amount, then modify it step by step, gradually increasing the amount of information saved for each flow 🪜 An RTFM meter ought to provide some measures of its own performance (e.g. number of active flows, percentage idle processor time, packets metered, packets not metered). These measures will be implementation-specific, but should let you assess the impact of each change to the ruleset 👀

If the network data rate is too high, i.e. the meter reports that it cannot meter all the packets even with the initial ruleset above, you may be able to use other strategies 🌊 For example, you could:

  • run the meter on a faster computer, e.g. move from a DOS PC to a workstation, or perhaps use a meter implemented in firmware within a switch or router 🖥️
  • use sampling. The details of such sampling are not defined within the RTFM Architecture, but the Meter MIB provides one simple method: it lets you specify that only every nth packet on an interface will be metered 🎲 This would probably not be acceptable for producing billing data, but might well be acceptable for traffic engineering purposes 🚗

6. Policy Considerations ✨

okay so before you collect ANY traffic data, you need well-defined operations policies 📋 covering stuff like:

  • Exactly what data is to be collected, at what level of detail?
  • How long will the data be kept?
  • What may the data be used for?
  • Who will be allowed to see the raw data?
  • May summaries of the data be shown to other people?

these policy questions should normally be part of an organisation's Network Security Policy 🔐 bestie, this is the boring-but-important stuff 🙈

then there are policy issues that are more directly about the traffic data itself. they're basically part of how you design the measurement system 🧚 like:

  • How much time resolution is required for the data? (less resolution means longer collection intervals, but that may need more memory in the meters to hold flow data between collections 💜)
  • What level of hardware redundancy is needed? (a single meter and meter reader is generally enough 🌸 for extra reliability, meters and meter readers can be duplicated)
  • Who is allowed to use the system? (approved users will need permissions to download rulesets to the meters, and to collect their data, possibly via their own meter readers 🔑)

7. Soundness 🔥

NeTraMet, the first implementation of the RTFM Architecture, has been in use worldwide since 1994. like, it's been out there for YEARS and it's still going strong 👑 right now there are lots of organisations, big and small, using it to collect traffic data for billing purposes 💅

one example is Kawaihiko, the New Zealand Universities' Network 🇳🇿 which has seven RTFM meters located at sites throughout New Zealand. one of those sites is NZIX, the New Zealand Internet eXchange at the University of Waikato 🦋 where Kawaihiko has a meter (attached to a 100baseT network) observing traffic flows across the exchange to each of Kawaihiko's three international Internet Service Providers. 5-minute Octet counts are collected from all the Kawaihiko meters by a single meter reader at Auckland. and the traffic data from the meters is used to figure out the cost per month for each of the Kawaihiko sites 💖

it's kinda hard to guess how many organisations are using RTFM traffic measurement 🤷‍♀️ there are about 250 people on the NeTraMet mailing list, and it often gets questions like 'why doesn't this ruleset do what I meant'? 😭 but once new users have the system running, they tend to just use it without saying much more about it 🫶

the list does give useful feedback sometimes ✨ like early in 1998, there were two HUGE user contributions:

  • Jacek Kowalski (Telstra, Melbourne) described an improved hash algorithm for NeTraMet's flow table, which gave almost an order of magnitude improvement in packet-handling performance. absolutely iconic 🔥
  • Kevin Hoadley (JANET, U.K.) reported having problems with very large rulesets. these were resolved, and better methods of downloading rules were developed, so NeTraMet now works well for rulesets with more than 32,000 rules 👑

maybe one reason there's so little talk about NeTraMet being used for collecting billing data is that users may consider the way collect their data is a commercially sensitive matter 🙈 totally fair tbh 💅

8. Appendix A: WG Report on the Meter MIB 🎀

the Meter MIB (in its current form) was developed early in 1996 ✨ it was made as an SNMPv2 MIB, after lots of detailed (and still ongoing) discussions with David Perkins, starting at the Dallas IETF meeting in December 1995 💜

there are two current implementations:

  • NeTraMet (Nevil Brownlee, The University of Auckland)
  • IBM Meter (Sig Handelman & Stephen Stibler, IBM Research, N.Y, Bert Wijnen provided further help with SNMP)

the NeTraMet meter is a stand-alone SNMP agent using an SNMPv2C implementation derived from CMU SNMPv2 🧁

the IBM meter runs as a sub-agent on an AIX system. all of the meter code was written by Stephen Stibler, and it was NOT derived from the NeTraMet code 🙌 Stephen also found it useful to use nifty, one of NeTraMet's manager/reader programs, to test the IBM meter 🦋

as mentioned above, there have only been two implementors so far, and the Working Group consensus has been very strong 💅

the MIB has one unusual aspect 👀 which is the method used to read large amounts of data from its Flow Table. an earlier SNMPv1 version of the MIB was in use from 1992 to 1997, and it used opaque objects to read column slices from the flow table for flows that had been active since a specified time. honestly this was very non-standard (or at least very application-specific) 💀

when they moved to SNMPv2, they were able to use 64-bit counters for PDUs and Octets, RowStatus variables for control tables, and GETBULK requests to read rows from the flow table ⭐ they also use the TimeFilter convention from the RMON2 MIB to select which flows to read, so the meter MIB gives off a strong RMON2 resemblance 🌈

the current MIB has a better way of reading large amounts of data from the flow table. this is the 'DataPackage' convention, which specifies the attribute values to be read from a flow table row. the meter returns the values for each required attribute inside a BER-encoded sequence. so there's only one object identifier for the whole sequence, which hugely cuts down the number of bytes needed to retrieve the data 🧚 the combo of these three things:

TimeFilter:  to select the flows to be read
DataPackage: to select the attributes required for each flow
GetBulk:     to read many flows with a single SNMP PDU

provides a very effective way to read flow data from a traffic meter 😍 literally chef's kiss 👑

9. References 📚

[ACT-BKG]  Mills, C., Hirsch, G. and G. Ruth, "Internet Accounting
           Background", RFC 1272, November 1991.

[RTFM-ARC] Brownlee, N., Mills, C. and G. Ruth, "Traffic Flow
           Measurement: Architecture", RFC 2722, October 1999.

[RTFM-MIB] Brownlee, N., "Traffic Flow Measurement: Meter MIB", RFC
           2720, October 1999.

[RTFM-NEW] Handelman, S., Stibler, S., Brownlee, N. and G. Ruth,
           "RTFM: New Attributes for Traffic Flow Measurement", RFC
           2724, October 1999.

[RTFM-NTM] Brownlee, N., "Traffic Flow Measurement: Experiences with
           NeTraMet", RFC 2123, March 1997.

[RTFM-SRL] Brownlee, N., "SRL: A Language for Describing Traffic
           Flows and Specifying Actions for Flow Groups", RFC 2723,
           October 1999.

10. Author's Address 💌

Nevil Brownlee
Information Technology Systems & Services
The University of Auckland
Private Bag 92-019
Auckland, New Zealand

Phone: +64 9 373 7599 x8941
EMail: n.brownlee@auckland.ac.nz

Copyright (C) The Internet Society (1999). All Rights Reserved. 💖

so you can copy and share this document, and you can make derivative works that comment on or explain it or help with implementing it 🫶 you can publish and distribute those in whole or in part with no real restrictions, as long as the copyright notice and this paragraph stay on all copies and derivative works. BUT the document itself can't be modified in any way, like removing the copyright notice or the references to the Internet Society or other Internet organisations 🚫 the only exceptions are developing Internet standards (in which case you follow the copyright procedures in the Internet Standards process) or translating it into languages other than English 🌍

the limited permissions above are perpetual, and the Internet Society and its successors or assigns won't revoke them 💜

this document and the information in it is provided on an "AS IS" basis, and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 💀

Acknowledgement 🌸

Funding for the RFC Editor function is currently provided by the Internet Society. thank you, Internet Society 🙏✨

📦 where this page comes from