RFC 2123
Traffic Flow Measurement: Experiences with NeTraMet
RFC 2123: Traffic Flow Measurement: Experiences with NeTraMet 🎀
Okay bestie, first things first 💖 this one is by N. Brownlee from The University of Auckland, dated March 1997. It's Informational, so it's giving "here's what we learned" energy, not "here's a rule you must follow" 🧚 It doesn't obsolete or update anything, since the title block doesn't list anything it obsoletes or updates.
Status of This Memo 🌸
This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
Abstract ✨
Okay so this memo is basically a diary of what happened when people actually built the Traffic Flow Measurement Architecture and Meter MIB 📖 It goes through how NeTraMet (a traffic meter) and NeMaC (a combined manager and meter reader) were built, talks about writing meter rule sets, and gives some tips on setting up a traffic flow measurement system with NeTraMet 🦋
Table of Contents 📋
1 Introduction 2
1.1 NeTraMet structure and development . . . . . . . . . . . . . . 3
1.2 Scope of this document . . . . . . . . . . . . . . . . . . . . 4
2 Implementation 4
2.1 Choice of meter platform . . . . . . . . . . . . . . . . . . . 4
2.2 Programming support requirements . . . . . . . . . . . . . . . 5
2.2.1 DOS environment . . . . . . . . . . . . . . . . . . . . . 6
2.2.2 Unix environment . . . . . . . . . . . . . . . . . . . . . 7
2.3 Implementing the meter . . . . . . . . . . . . . . . . . . . . 7
2.3.1 Data structures . . . . . . . . . . . . . . . . . . . . . 7
2.3.2 Packet matching . . . . . . . . . . . . . . . . . . . . . 8
2.3.3 Testing groups of rule addresses . . . . . . . . . . . . . 8
2.3.4 Compression of address masks . . . . . . . . . . . . . . . 9
2.3.5 Ignoring unwanted flow data . . . . . . . . . . . . . . . 10
2.3.6 Observing meter reader activity . . . . . . . . . . . . . 11
2.3.7 Meter memory management . . . . . . . . . . . . . . . . . 12
2.4 Data collection . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 Restarting a meter . . . . . . . . . . . . . . . . . . . . . . 15
2.6 Performance . . . . . . . . . . . . . . . . . . . . . . . . . 16
3 Writing rule sets 16
3.1 Rule set to observe all flows . . . . . . . . . . . . . . . . 17
3.2 Specifying flow direction, using computed attributes . . . . . 18
3.3 Subroutines . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.4 More complicated rule sets . . . . . . . . . . . . . . . . . . 23
4 Flow data files 26
4.1 Sample flow data file . . . . . . . . . . . . . . . . . . . . 27
4.2 Flow data file features . . . . . . . . . . . . . . . . . . . 28
4.3 Terminating and restarting meter reading . . . . . . . . . . . 29
5 Analysis applications 30
6 Using NeTraMet in a measurement system 31
6.1 Examples of NeTraMet in production use . . . . . . . . . . . . 31
7 Acknowledgments 33
8 References 33
9 Security Considerations 34
10 Author's Address 341. Introduction 💅
Early in 1992 my University needed a system to get back the costs of its Internet traffic 💸 That March I went to the Internet Accounting Working Group's session at the San Diego IETF, and I was so excited to find out they'd already come up with a detailed architecture for measuring network traffic and were just waiting for someone to try building it 🤩
So during 1992 I made a prototype measurement system that used balanced binary trees to store traffic flow info. I presented it at the Washington IETF in November 1992. It performed well, but it never tried to free up memory from old flows, and managing the balanced trees cost way too much 😭 So I moved on and built a production-quality system, this time using hash tables to index the flow information 🧚
That version is called NeTraMet (the Network Traffic Meter), and it was released as free software in October 1993 🎉 I've kept working on it ever since, putting out new releases two or three times a year. NeTraMet is now in production use at lots of sites around the world. Nobody can really count how many sites there are, but there's an active NeTraMet mailing list, and it had about 130 subscribers in March 1996 🌈
Early in 1996 the Realtime Traffic Flow Measurement Working Group (RTFM) was chartered to take the Traffic Flow Measurement Architecture onto the IETF standards track 🚀 This document records the traffic flow measurement experience I got over three years with NeTraMet 🫶
1.1. NeTraMet structure and development
The Traffic Flow Architecture document [1] describes four components:
- METERS, which are attached to the network at the points where it is desired to measure the traffic,
- METER READERS, which read data from meters and store it for later use,
- MANAGERS, which configure meters and control meter readers, and
- ANALYSIS APPLICATIONS, which process the data from meter readers so as to produce whatever reports are required.
NeTraMet is a computer program that implements the Traffic Meter, stores the measured flow data in memory, and provides an SNMP agent so the data is available to Meter Readers 🤖 The NeTraMet distribution files include NeMaC, which is a combined Manager and Meter Reader that can manage an arbitrary number of meters. Each meter can use its own rule set and have its flow data collected at its own specified intervals ✨ The distribution also has a few basic Analysis Applications, so you can make simple plots from NeMaC's flow data files (fd_filter and fd_extract) and watch the flows at a remote meter in real time (nm_rc and nifty) 👀
Since the first release, the Traffic Meter MIB [2] has been improved and simplified a lot 🔥 The big changes include better ways to specify traffic flows (more actions and better control structures for the Packet Matching Engine) and computed attributes (class and kind). Operational needs at sites running NeTraMet pushed these changes along, and they've been tested heavily across successive versions of NeTraMet 💜
NeTraMet is widely used to collect usage data for Internet Service Providers, especially in Australia and New Zealand 🇦🇺🇳🇿 There are also active users all over the world, for example in Canada, France, Germany and Poland 🌍
NeTraMet is seriously useful for understanding exactly where traffic is flowing in big networks 💖 Since the Traffic Meters do a lot of data reduction (as their rule sets say to), they cut down a lot on the amount of data Meter Readers have to read. That makes NeTraMet especially good for networks with lots of remote sites. An example of this, the Kawaihiko network, is briefly described below 🧚
NeTraMet can also be used for real-time network monitoring and trouble-shooting, not just post-observation analysis 🙈 The distribution includes 'nifty,' an X/Motif application that monitors traffic flows and tries to highlight the 'interesting' ones 🦋
1.2. Scope of this document
This document shares what I learned from three years of working with the Traffic Flow Measurement Architecture 📚 Its contents are grouped like this:
- Implementation issues for NeTraMet and NeMaC,
- How rule files work, and how to write them for particular purposes, and
- How to use NeTraMet and NeMaC for short-term and long-term flow measurement.
2. Implementation 🎀
2.1. Choice of meter platform
As the Architecture document [1] points out, the Realtime Traffic Flow Measurement Working Group's goal is to create a standard for the Traffic Meter, so it can end up built into network devices like hubs, switches and routers 🖥️ Until the Architecture is developed enough to make that happen, it was good enough to run the meter as a program on a general-purpose computer 💻
NeTraMet's computer had to be one that's widely available across the Internet community. A Unix system was a strong contender, since they're commonly used for all sorts of network support and management tasks. For the first implementation, though, Unix had some downsides 🙃
- The wide variety of different Unix systems can make software support harder.
- A Unix system costs too much to run as a meter if you want meters running at many points in your network at once.
Performance was another factor in picking the platform 🔍 When I first started building NeTraMet, nobody could predict how much processing a viable meter needed. I also had no clue how much memory the code or data would take. So I went with a DOS PC, because:
- It's a minimum system in every way. If the meter works well on something this small, it can be built on almost any hardware (including routers, switches, etc.) 💪
- It's inexpensive. Sites can easily afford to have lots of meters around their networks 💸
- It's a simple system, and one I had complete control over. That let me build in effective instrumentation to monitor the meter's performance and include a wide variety of performance optimisations in the code 🫶
Once the meter was running, I needed a manager to download rule files to it 📥 Since one manager and meter reader can effectively support a large number of meters, a Unix environment for NeMaC was a natural choice 🐧 NeMaC has way fewer software support problems than NeTraMet does, since it has minimal support needs. It only needs to open a UDP socket to the SNMP port on each controlled meter 🧚
Early NeTraMet distributions only had the PC meter and the Unix manager. In later releases I ported NeTraMet (the meter) to Unix and extended NeMaC's control features (the combined manager and meter reader) 🌸 I've also tried porting NeMaC to DOS. It's not difficult, but it doesn't seem worth pursuing 🙈
The current version of NeTraMet is a production-quality traffic measurement system that has been in continuous use at the University of Auckland for nearly two years 👑
2.2. Programming support requirements
To build the Traffic Flow Meter, I needed a programming environment with good support for the following:
- observation of packet headers on the network;
- system timer with better than 10 ms resolution;
- IP (Internet Protocol), for communications with manager and meter reader;
- SNMP, for the agent implementing the Meter MIB.
2.2.1 DOS environment 💾🎀
okay so for the PC I went with Ethernet as the physical network medium 🌐. That was just the first pick, because it's what the University of Auckland's data network ran on. Interfaces for other media can totally be added later when they're needed ✨.
Now the drivers. There's a bunch of "generalised" network interfaces out there. I looked at the ones from Novell, DEC and Microsoft and said nope 🙅♀️, partly because they're proprietary and partly because they didn't seem easy to use. Instead I picked the CRYNWR Packet Drivers [3] . They work with a ton of interface cards, they're simple, and the docs are actually clear 🥹. They support Ethernet's promiscuous mode, so you can look at the headers of every packet going past, super straightforward 👀. The downside is they're harder to use with newer user shells (like Microsoft Windows) 💀. But that didn't matter, because I planned to run the meter as the only program on a dedicated machine 🫶.
Timing was a whole thing 😭. The BIOS timer routines only tick about 18 times each second, so the resolution is kinda limited. At first I was like "eh, one second per packet is fine, flows last many seconds" 🙈. Then it became clear that lots of flows last well under a second. To measure those properly with the Traffic Flow Meter, you need times resolved to 10 millisecond intervals. That's the size of TimeTicks, the most common time unit in SNMP [4]. Since all the details of the original PC are readily available [5], understanding the underlying hardware wasn't too hard. So I wrote PC timer routines for NeTraMet that read the hardware timer with 256 times the resolution of the DOS clock ticks, i.e. about 5 ticks per millisecond ⏱️✨.
Ngl, that 256x trick is the most iconic part of this whole section 👑.
There are loads of TCP/IP implementations for DOS, but most are commercial. So I chose Waterloo TCP [6], which was available (including full source code) as public domain software 🎉. I needed to modify it so I could save incoming packet headers while also forwarding packets destined for the meter to the IP handler routines 🔧. For SNMP I picked CMU SNMP [7], which was also public domain with full source. That made porting it from Unix to the PC pretty simple 💜.
Finally, NeTraMet development was done in Borland's Turbo C and Turbo Assembler 🩷. Newer C programming environments are around now, but honestly, Turbo C version 2 has been perfectly happy with me for the NeTraMet project! 💅
2.2.2 Unix environment 🐧
For the Unix meter, the one obvious problem was "how do I get access to packet headers?" 🤔. Early versions used various system-specific interfaces on a SunOS 4.2 system. Later versions use libpcap [8], which gives a portable way to get at packet headers on a wide range of Unix systems 🌸. I've verified it works really well for ethernet interfaces on Solaris, SunOS, Irix, DEC Unix and Linux, and for FDDI interfaces on Solaris 🙌. libpcap timestamps each packet header with resolution set by the system clock, which is definitely better than 10 ms!! 🤯
All Unix systems have TCP/IP capabilities, so that wasn't an issue. For SNMP I used CMU SNMP, exactly like on the PC 🫶.
2.3 Implementing the meter ⚙️
This section gives a quick tour of the meter's data structures and the packet matching process 🦋. One super big concern during NeTraMet's evolution has been getting the highest possible meter performance 🔥. I developed a bunch of interesting optimisations for that, as discussed below. Another big concern was efficient and effective memory management, which gets its own deep dive below 🧠.
2.3.1 Data structures 🧱
Everything in NeTraMet, NeMaC and their supporting utility programs is written in C 💖. That's partly because C and its run-time libraries give good access to the underlying hardware, and partly because it's a super portable language.
Each flow's data lives in a C structure 📦. It holds all the flow's attribute values (including packet and byte counts), plus a link field so flows can be strung together into lists. NeTraMet assumes Adjacent addresses are 802 MAC Addresses, which are six bytes long. Transport addresses are assumed to be two bytes long, which is the case for port numbers in IP. Peer addresses are normally four bytes or less, but they can be as long as 20 bytes (for CLNS) 🌈. I chose a fixed Peer address size, defined at compile time, to avoid the hassle of variable-sized flow structures 🙃.
The flow table is an array of pointers to flow data structures, so you can index flows via their flow numbers 🔢. There's also one big hash table, which the Architecture document [1] calls the flow table's 'search index' 🔍. Each hash value in the table points to a circular chain of flows. To find a flow, you compute its hash value, then search that value's flow chain 🕵️♀️.
Each rule is stored in a C structure too 📜. All the rule components have fixed sizes, but address fields must be wide enough to hold any type of address (Adjacent, Peer or Transport) 💅. Like flow Peer addresses, the rule address width is defined at compile time. Each rule set is an array of pointers to rule data structures, and the rule table is an array of pointers to the rule sets. NeMaC specifies the size of each rule set before it starts downloading it, but the maximum number of rule sets is defined at compile time 💜.
2.3.2 Packet matching 🎯
Packet matching in NeTraMet works exactly as described in the Architecture document [1] ✨. Each incoming packet header is analysed to determine its attribute values. Those values get stored in a structure that's passed to the Packet Matching Engine 🚀. To make matching with source and destination reversed easy 🔄, this structure has two substructures: one holds the source Adjacent, Peer and Transport address values, and the other holds the destination address values 💖.
2.3.3 Testing groups of rule addresses 🧩
As the Architecture [1] describes, each rule's address will usually be tested: ANDed with the rule's mask and compared with the rule's value. If the comparison fails, the next rule in sequence runs 🏃♀️. This lets you write rule sets where a group of rules checks whether one of an address (e.g. its SourcePeerAddress) is one of a set of specified IP addresses 🎀. Those groups can get quite large, with hundreds of rules 😳. Running them sequentially would be slow, so something better was essential 🔥.
The NeTraMet optimisation is to find groups of rules that test the same attribute with the same mask, and convert them into a single hashed search of their values 🪄. The overhead of setting up the hash tables (one for each group) is paid once, just before the meter starts running a new rule set. When a 'group' test is due, the meter ANDs the incoming attribute value, computes a hash from it, and uses that to search the group's hash table 🔎. Early tests showed the rule hash chains were usually very short, with only one or two members 💅. The effect is that large sequences of tests shrink to a hash computation and a lookup, with a very small number of compares. In short, this is an essential optimisation for any traffic meter!! 👑
Hashed compares have overhead too, so NeTraMet has a minimum group size defined at compile time. If a group is too small, it doesn't get combined into a hashed group 🙅♀️.
In early versions of NeTraMet I didn't allow Gotos into a hashed group, which proved to be an unnecessarily conservative position 🙈. NeTraMet stores each group's hash table in a separate memory area, and keeps a pointer to the hash table in the first rule of the group. (The rules data structure has an extra component to hold this hash table pointer) 📦. Rules within the group don't have hash table pointers. When they're executed as the target of a Goto rule, they behave as ordinary rules, so their tests are performed normally 💜.
2.3.4 Compression of address masks 🗜️
When the Packet Matching Engine has decided that an incoming packet belongs to a flow that's being measured, it searches the flow table to see whether the flow is already there 🔍. It does this by computing a hash from the packet and using it to access the flow table's search index.
When designing a hash table, you normally assume the objects in the table have a constant size 📏. For NeTraMet's flow table, that would mean each flow contains a value for every attribute. But that's not the case, since only the attribute values 'pushed' by rules during packet matching are stored for a flow 🙌.
Let's say every flow in the table contains a value for only one attribute, SourcePeerAddress, and the rule set decides whether flows belong to a specified list of IP networks. In that case, only their network numbers get pushed 🌐. The rules do this test using a variety of masks, since the network number allocations range from 16 to 24 bits in width 🤓. When searching the flow table, the meter must distinguish between zeroes in the address and 'don't care' bits that had been ANDed out 🧐. To manage that, it has to store SourcePeerMask values in the flow table as well as the ANDed SourcePeerAddress values 😭.
In early versions of NeTraMet this was side-stepped by using multiple hash tables and relying on the user to write rules that used the same set of attributes and masks for all the flows in each table. It worked, but it was clumsy and difficult to explain 🙃. Later versions switched to a single hash table and stored the mask values for all the address attributes in each flow.
The current version stores the address masks in compressed form 🗜️. After examining a large number of rule sets, I realised that although a rule set may have many rules, it usually has a very small number of address masks 💡. So it's a simple matter to build a table of address masks, and store an index to this 'mask table' instead of a complete mask 🗂️. NeTraMet's maximum number of masks is defined at compile time, up to a maximum of 256. That allows a single byte for each mask in the flow data structure, which significantly shrinks the structure 📉. As well as the size reduction, two masks can be compared by comparing their indices in the mask table, which reduces to a single-byte comparison. Overall, using a mask table seems to give useful improvements in storage efficiency and execution speed 🦋💖
2.3.5 Ignoring unwanted flow data ✨
okay so, as the Architecture document [1] explains, every incoming packet gets tested against the current rule set by the Packet Matching Engine 💅. This section is about how I made NeTraMet faster on the PC by cutting down how much work each incoming packet causes 🚀.
on the PC, every incoming packet triggers an interrupt, and NeTraMet has to handle it so it can collect info about the packet 😭. in the early versions I used a ring buffer with 512 slots for packet headers. I just copied each packet's first 64 bytes into the next free slot 🎀. later on, the headers got pulled out of the buffer, attribute values were extracted from them, and the resulting 'incoming attribute values' records were passed to the Packet Matching Engine.
then I changed the interrupt handling code to pull out the attribute values and store them in a 'buffer slot.' 🧚 this shrank the storage each slot needed, which left more room for storing flows 🍓. it did make the processing for each packet interrupt go up a little bit, but honestly that hasn't caused any problems at all 🙈.
in later versions I realised something iconic: if you only care about measuring IP packets, there's zero point in storing (and then processing) Novell or EtherTalk packets! 💀 the meter can just look at a rule set and figure out which Peer types are of interest. if there are PushRule rules that test SourcePeerType (or DestPeerType), those rules say which types matter. if there aren't any rules like that, every packet type is of interest. the PC NeTraMet has a set of Boolean variables, one for each protocol it can handle 👑. the values of these 'protocol' variables get set when the meter starts running a new rule set. for each incoming packet, the interrupt handler works out the Peer type. if the protocol isn't of interest, no further processing happens and the packet is just ignored 🫶. same idea here: if Adjacent addresses are never tested, there's no reason to copy them into the packet buffer slot 🦋.
the overall effect of these optimisations is most noticeable for rule files that measure IP flows on a network segment that also carries a ton of traffic for other network protocols 🌈. this is super common on multiprotocol Local Area networks, so it's a big deal. on the Unix version of NeTraMet, the Operating System does all the packet interrupt processing, and libpcap [8] hands packet headers straight to NeTraMet. the 'protocol' and 'adjacent address' optimisations still happen, just at the point where NeTraMet receives the packet headers from libpcap ✨.
2.3.6 Observing meter reader activity 👀
the Architecture document [1] says a flow data record has to be held in the meter until a meter reader has read its data 🔒. so a meter needs a reliable way of deciding when flow data has been read. this gets messy because there can be more than one meter reader, and they collect their data asynchronously 😵💫.
early versions of NeTraMet fixed this with a single MIB variable that a meter reader could set to say it was starting a data collection 💌. when an SNMP SET request came in, NeTraMet would update its 'collectors' table. this table had an entry for each meter reader, plus variables recording the start time of the last two collections. the most recent collection might still be running, but its start time is a safe estimate of when the one before it actually finished ⏰. space used for flows that have been idle since the penultimate collection started can be recovered by the meter's garbage collector, which I'll explain below 🗑️.
the Meter MIB [2] specifies a more general table of meter reader info 📋. a meter reader that wants to collect data from a meter has to tell the meter what it's planning by creating a row in the table. then it sets a LastTime variable in that row to show when the collection starts 🕰️. the meter handles that SET request exactly like we just described. if there are multiple meter readers, the meter can easily find the earliest time any of them started its penultimate collection, and it can recover flows that have been idle since then 🙌. if a meter reader fails, NeTraMet will eventually time out its entry in the meter reader info table and delete it 🧹. this stops a messy situation where the meter can't recover flows until several meter readers have collected them, and one of those readers has failed 😤.
2.3.7 Meter memory management 🧠
in principle, the size of the flow table (i.e. the maximum number of flows) could be changed dynamically 🌸. that would mean allocating space for the flow table's new pointer array and copying the old pointers into it. NeTraMet doesn't do this. instead, the maximum number of flows is set on the command line when it starts running. if no maximum is specified, a compile-time default number is used 💜.
memory for flow data structures (i.e. 'flows') is allocated dynamically 💖. NeTraMet asks the C run-time system for blocks of several hundred flows and links them into a free list. when it needs a new flow, NeTraMet takes memory from the free list and then searches the flow table's pointer array for an unused flow pointer. in practice, a 'last-allocated' index points into the flow table, so a simple linear search is enough 🔍. the flow index gets saved in the flow's data record, and its other attribute values are set to zero.
to release a flow data record, it must first be removed from any hash list it belongs to. this is easy, since those lists are circular 🔄. the flow's entry in the flow table pointer array is then set to zero (NULL pointer), and its space goes back onto the free list.
once a flow data record is created, it could keep existing forever. eventually, though, the meter would run out of space 😱. to deal with this, NeTraMet uses an incremental garbage collector to reclaim memory 🧹✨.
the garbage collector procedure runs at regular intervals set by a GarbageCollectInterval variable. it searches through the flow table looking for flows that might be recovered. to keep the resources used by garbage collection under control, there are limits on how many in-use and idle flows the garbage collector may inspect. these limits are set either when NeTraMet starts (as options on the command line) or dynamically by NeMaC (using variables in an Enterprise MIB for NeTraMet) 🎛️.
to decide whether a flow can be recovered, the garbage collector looks at how long it has been idle (no packets in either direction) and when its data was last collected 🕵️♀️. if it has been collected by all known meter readers since its LastTime, its memory can be recovered 💅. this algorithm uses a variable called GarbageCollectTime, which normally holds the meter's UpTime when the penultimate collection (i.e. the one before last) started. see the section on observing meter reader activity (above) for more details 📖.
if flows don't get collected often enough, the meter could run out of space 🥺. NeTraMet tries to prevent this with a low-priority background process. it checks the percentage of flows that are active and compares it with the HighWaterMark MIB variable. if the percentage of active flows is greater than the high-water mark, GarbageCollectTime is incremented by the current value of the InactivityTimeout MIB variable 📈.
the Meter MIB [2] specifies that a meter should switch to a 'standby' rule set if the percentage of active flows rises above HighWaterMark 🔁. when we used NeTraMet to measure traffic flows to and from the University of Auckland, it wasn't hard to make standby rules that were very similar to the 'production' rule file. the only difference was that they pushed much less information about flows 🍰. on several occasions this let the meter keep running for one or two days after the meter reader failed 🙌. when the meter reader restarted, it was able to collect all the accumulated flow data! 😭🫶
the MIB also specifies that the meter should take some action when the active flow percentage rises above its FloodMark value 🌊. if that didn't happen, the meter could spend a rapidly growing share of its time garbage collecting, to the point where it couldn't respond properly to requests from its manager 💀. NeTraMet switches to the default rule set when FloodMark is reached 🔔.
a potentially large number of new flows may be created when the meter switches to a standby rule set 🌷. so it's important to set a HighWaterMark that leaves enough flow table space for this. in practice, a HighWaterMark of 65% and a FloodMark of 95% seem to work well 🏆.
2.4 Data collection 🦋
Okay so, when a meter reader wants to collect flows, it starts by setting the LastTime variable in its ReaderInfoTable row, then goes through the flow table collecting data. There are a bunch of algorithms for scanning that table, and we'll look at them below 💅
The simplest option is a linear scan, reading LastTime for every row. If the read fails, the row is inactive. If it works, the row is interesting only if its LastTime value is greater than the time of the last collection. It's simple, but it's also rather slow, since it needs an SNMP GET request for every possible flow. That's giving impractical 🙈
Early versions of NeTraMet used two 'windows' into the flow table to find the interesting flows 🪟 Both windows were SNMP tables, indexed by a variable that specified a time. NeMaC (the meter reader) could send a succession of GETNEXT requests to one of these windows and find the flow indices for every flow that had been active since the specified time. The two windows were the ActivityTime window, which located active flows, and the CreateTime window, which located new flows. Once it knew the index of an active flow, the meter reader could GET the values for all the attributes it cared about. NeMaC lets the user pick which attributes those are, instead of just reading all of them ✨
Having two windows meant NeMaC could read attributes that stay constant, like a flow's address attributes, only when the flow is created. Then it could read only the attributes that change over time, like packet and byte counts, during later collections. Experience has shown, though, that a lot of flows have pretty short lifetimes. So the efficiency boost from using two windows doesn't lead to any worthwhile improvement in collection performance. Not the glow-up we expected 😭
The current version of the Meter MIB [2] uses a TimeFilter variable in the flow table entries. You can use it with GETNEXT requests to find all the flows that have been active since a specified time directly, with no need for those extra 'window' SNMP variables. It can also be combined with SNMPv2's GETBULK request to cut down the number of SNMP packets needed for each collection even more. I have yet to implement this in NeTraMet 🌸
Now, the downside of using SNMP to collect data from the meter is that SNMP packets have a lot of overhead 📦 For example, if we want to read an Integer32 variable (four bytes of data), it comes back with its object identifier, type and length. That's at least ten bytes of superfluous data, bestie. One way to cut this overhead down is to use an Opaque object to return a whole collection of data at once. NeTraMet does exactly this to retrieve 'column activity data' from the meter, and here's how it works 👀
Each packet of column activity data holds data values for one specified attribute, and each value is preceded by its flow number. You can think of the flow table as a two-dimensional array, with a column for each flow attribute. Column activity data objects let the meter reader read columns of the flow table, so it only collects the attributes the user asked for. It's a bit messy in practice, though, because the flow table gets read column by column, which means rows can become active after the first column has already been read. NeMaC reads the widest columns first (the ones with the greatest size in bytes, e.g. PeerAddress), and then ignores any rows that show up in later columns. Newly active rows will, of course, get picked up in the next collection 🔄
Using Opaque objects like this dramatically reduces the number of SNMP packets needed to read a meter. This has been worth it where the number of flows is large (for example on busy LANs) and the meters are physically spread out over slow WAN links 🌐 The catch is that general-purpose MIB browsers can't understand the column activity variables. Honestly, that feels like a small price to pay for the improved collection performance 💖
2.5 Restarting a meter 🔁
Okay, so if a meter fails, say because of a power failure, it restarts and begins running rule set 1, which is the default rule set built into the meter. Its manager has to notice that this happened and respond with some suitable action 🚨
NeMaC lets the user set a 'keepalive' interval. After every one of those intervals, NeMaC reads the meter's sysUptime and compares it with the last sysUptime. If the new sysUptime is less than the last one, NeMaC decides the meter has restarted. It then downloads the meter's backup rule set and production rule set, and asks the meter to start running the production rule set. In normal use, we use a keepalive interval of five minutes and a collection interval of 15 minutes. If a meter restarts, we lose up to five minutes of data before the rule sets get downloaded 🥲
Running the default rule set on startup is part of the Traffic Flow Measurement Architecture [1]. It fits the idea that meters are very simple devices with no disk storage 💾 Disks are super cheap these days, though, so it might be worth considering whether the architecture should let a meter save its configuration (including rule sets) to disk. Just a thought, besties 🧠
2.6 Performance 🚀
The PC version of the meter, NeTraMet, keeps constantly measuring how much processor time is being used. Whenever there's no incoming packet data to process, it generates 'dummy' packets and places them in the input buffer. These packets get processed normally by the Packet Matching Engine, and they have a PeerType of 'dummy.' The meter counts the dummy and normal packets, and their ratio is used as an estimate of the processor time that's 'idle,' meaning it isn't being used to process incoming packets. The Unix version is meant to run as a process in a multiprocessing system, so it can't busy-wait in this way 🙅♀️
The meter also collects several other performance measures. You can display these on the meter console by making keyboard requests ⌨️
Now for the receipts 🧾 The PC meter can run on a 10 MHz 286 machine, where it can handle a steady load of about 750 packets per second. On a 25 MHz 386SX, it handles about 1250 packets per second. Users have reported that a 40 MHz 486 can handle peaks of about 3,000 packets per second without packet loss 🔥 The Unix meter was tested metering traffic on a (lightly loaded) FDDI interface, and it uses about one percent of the processor time on a SPARC 10 system running Solaris. One percent, periodt 💅
3 Writing rule sets 🎀
The Traffic Meter is a versatile device for measuring a user-specified set of traffic flows and doing useful data reduction on them. This data reduction doesn't just minimise the volume of data that meter readers have to collect. It also simplifies the later processing of traffic flow data 🧹
The flows you care about, and the processing to be done on them, are specified in a 'rule set.' The manager (NeMaC) downloads that rule set to the meter (NeTraMet). This section explains what's involved in writing rule sets 📝
NeTraMet can only meter packets observed on a network segment. That means that for all the observed flows, the Source and Dest Type attributes (e.g. SourcePeerType and DestPeerType) have the same value 🤝
The NeTraMet implementation uses single variables in its flow data structure for AdjacentType, SourceType and TransType. Even so, the rule sets discussed below push values for both Source and Dest Type attributes. This makes sure packet matching works properly with the directions reversed, even for a meter that allows Source and Dest Type values to be different 🔄
3.1 Rule set to observe all flows 👀
NeMaC reads rule sets from text files. Those files contain the rules, the set number the meter (and meter reader) will identify them by, and a 'format,' which is a list specifying which attributes the meter reader should collect and write to the flow data file. The # character starts a comment, and NeMaC ignores the rest of the line 💬
SET 2
#
RULES
#
SourcePeerType & 255 = Dummy: Ignore, 0;
Null & 0 = 0: GotoAct, Next;
#
SourcePeerType & 255 = 0: PushPkttoAct, Next;
DestPeerType & 255 = 0: PushPkttoAct, Next;
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
SourceTransType & 255 = 0: PushPkttoAct, Next;
DestTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: PushPkttoAct, Next;
DestTransAddress & 255.255 = 0: CountPkt, 0;
#
FORMAT FlowRuleSet FlowIndex FirstTime " "
SourcePeerType SourcePeerAddress DestPeerAddress " "
SourceTransType SourceTransAddress DestTransAddress " "
ToPDUs FromPDUs " " ToOctets FromOctets;The first rule tests the incoming packet's SourcePeerType to see whether it's 'dummy.' If it is, the packet is ignored. Otherwise, the next rule runs 🗑️
The second rule tests the Null attribute. That kind of test always succeeds, so the rule just jumps to the action of the next rule. (NeMaC turns the keyword 'next' into the number of the following rule, so you don't have to 🧚)
The third rule pushes the packet's SourcePeerType value, then jumps to the action of the next rule. The user doesn't know in advance what the values of PushPkt rules will be, which is why the value in them is always zero. The user must take care not to write rule sets that try to perform the test in a PushPkt rule. This is a very common error in a rule set, so NeMaC tests for it and displays an error message. Caught in 4K 🚫
The following rules push a series of attribute values from the packet. The last rule also Counts the packet, which means it tells the Packet Matching Engine (PME) that the packet has been successfully matched. The PME responds by searching the flow table to see whether the flow is already current (i.e. in the table). If it's needed, it creates a new flow data record, and then it increments that flow's packet and byte counters 📈
Overall, this rule set simply classifies the packet (i.e. decides whether or not it's to be counted), then pushes all the Peer and Transport attribute values for it. It makes no attempt to specify a direction for the flow. That's left to the PME, as described in [1] 🧭 So the resulting flow data file will have each flow's source and destination addresses in the order of the first packet the meter observed for that flow. First impressions are everything, bestie ✨
3.2 Specifying flow direction, using computed attributes ✨
Okay so, like we said up top, the Packet Matching Engine (PME) will reliably figure out the flow, and the direction inside that flow, for every packet a meter sees 🫶. If the rule set doesn't specify a direction, the PME just assumes the first packet it sees for a flow is travelling forward, i.e. from source to destination. But when we analyse the flow data later, we're usually obsessed with traffic to or from one particular source 💅
One easy way to get that is to write a rule set that specifies the source for flows. All you need is rules that succeed if the packet is going the direction you want, and run a 'Fail' action otherwise. The next two examples show this 🎀
(Ps, early versions of NeMaC let you use 'Retry' as a synonym for 'Fail.' The current version also allows 'NoMatch,' which honestly sounds like a better way to say "fail, allowing PME to try a second match with directions reversed." 🙈)
# Count IP packets from network 130.216.0.0
#
SourcePeerType & 255 = IP: Pushto, ip_pkt;
Null & 0 = 0: Ignore, 0;
#
ip_pkt:
SourcePeerAddress & 255.255.0.0 = 130.216.0.0: Goto c_pkt;
Null & 0 = 0: NoMatch, 0;
#
c_pkt:
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.255 = 0: CountPkt, 0;The rule labelled ip_pkt checks whether the packet came from network 130.216. If it didn't, the test fails and the next rule runs a NoMatch action, which makes the PME retry the match with the directions reversed. If the second match fails too, the packet didn't have 130.216 as an end-point, so it gets ignored 💀
The next rule set meters IP traffic on a network segment that connects two routers, g1 and g2. It sorts flows into three groups: ones travelling from g1 to g2, ones whose source is g1, and ones whose source is g2 🌈
# Count IP packets between two gateways
#
# -------+-------------------+------------------+-------
# | | |
# +----+-----+ +----+-----+ +---+---+
# | g1 | | g2 | | meter |
# +-+-+-+-+--+ +-+-+-+-+--+ +-------+
#
SourcePeerType & 255 = IP: Pushto, ip_pkt;
Null & 0 = 0: Ignore, 0;
#
ip_pkt:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-80-48-81-0E-7C:
Goto, s1;
Null & 0 = 0: Goto, s2;
s1:
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 02-07-01-04-ED-4A
GotoAct, g3;
Null & 0 = 0: GotoAct, g1;
s2:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 02-07-01-04-ED-4A:
Goto, s3;
Null & 0 = 0: NoMatch, 0;
s3:
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-80-48-81-0E-7C:
NoMatch, 0;
Null & 0 = 0: GotoAct, g2;
#
g1: FlowClass & 255 = 1: PushtoAct, c_pkt; # From g1
g2: FlowClass & 255 = 2: PushtoAct, c_pkt; # From g2
g3: FlowClass & 255 = 3: PushtoAct, c_pkt; # g1 to g2
#
c_pkt:
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 0:
PushPkttoAct, Next;
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 0: PushPkttoAct, Next;
SourcePeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.255 = 0: PushPkttoAct, Next;
Null & 0 = 0: Count, 0The first two rules ignore non-IP packets. The next two rules Goto s1 if the packet's source was g1, or to s2 otherwise. The rule labelled s2 checks whether the packet's source was g2; if it wasn't, a NoMatch action runs so the PME can try the match with the packet's direction reversed. If the match fails on the second try, the packet didn't come from (or go to) g1 or g2, and it gets ignored.
Packets which come from g1 are tested by the rule labelled s1, and the PME will Goto either g3 or g1.
Packets which came from g2 are tested by the rule labelled s3. If they aren't going to g1, the PME will Goto g2. If they are going to g1, a NoMatch action runs, because we want them counted as backward-travelling packets for the g1-g2 flow 🔥
The rules at g1, g2 and g3 push the value 1, 2 or 3 from their rule into the flow's FlowClass attribute. An Analysis Application can use this value to split the flows into the three groups of interest. FlowClass is an example of a 'computed' attribute, i.e. one whose value is Pushed by the PME during rule matching ✨
The remaining rules Push the values of other attributes needed for later analysis, then Count the flow.
3.3 Subroutines 🧚
Subroutines work in the PME pretty much the way they do in BASIC. A subroutine body is just a sequence of statements, supported by the GoSub and Return actions. 'GoSub' saves the PME's running environment and jumps to the first rule of the subroutine body. Subroutine calls can nest as deeply as you need, though NeTraMet defines the maximum nesting at compile time 💜. 'Return n' restores the environment and jumps to the action part of the nth rule after the Gosub, where n is the index value from the Return rule.
The Return action is a way of steering control flow in a rule set, kind of like a FORTRAN Computed Goto. It's one way a subroutine can return a result. The other way is by Pushing a value into either a computed attribute (like we saw in the last section) or a flow attribute 🦋.
One common use for a subroutine is testing whether a packet attribute matches one of a set of values. That subroutine gets way more useful if it can test any one of several attributes. The PME architecture handles this with 'meter variables' that hold the names of the attributes to be tested. The meter variables are called V1, V2, V3, V4 and V5, and the Assign action sets their values. Say we need a subroutine that tests either SourcePeerAddress or DestPeerAddress: we write its rules to test V1 instead. Before calling the subroutine we Assign SourcePeerAddress to V1, and later tests of V1 are converted by the PME into tests on SourcePeerAddress. Note that since meter variables may be reassigned inside a subroutine, their values are part of the environment which must be saved by a Gosub action 🌸
The following rule set demonstrates the use of a subroutine ..
# Rule specification file to tally IP packets in three groups:
# UA to AIT, UA to elsewhere, AIT to elsewhere
#
# -------+-------------------+-----------------+--------
# | | |
# +----+-----+ +----+-----+ +---+---+
# | UA | | AIT | | meter |
# +-+-+-+-+--+ +-+-+-+-+--+ +-------+
#
SourcePeerType & 255 = IP: PushtoAct, ip_pkt;
Null & 0 = 0: Ignore, 0;
#
ip_pkt:
v1 & 0 = SourcePeerAddress: AssignAct, Next;
Null & 0 = 0: Gosub, classify;
Null & 0 = 0: GotoAct, from_ua; # 1 ua
Null & 0 = 0: GotoAct, from_ait; # 2 ait
Null & 0 = 0: NoMatch, 0; # 3 other
#
from_ua:
v1 & 0 = DestPeerAddress: AssignAct, Next;
Null & 0 = 0: Gosub, classify;
Null & 0 = 0: Ignore, 0; # 1 ua-ua
Null & 0 = 0: GotoAct, ok_pkt; # 2 ua-ait
Null & 0 = 0: Gotoact, ok_pkt; # 3 ua-other
#
from_ait:
v1 & 0 = DestPeerAddress: AssignAct, Next;
Null & 0 = 0: Gosub, classify;
Null & 0 = 0: NoMatch, 0; # 1 ait-ua
Null & 0 = 0: Ignore, 0; # 2 ait-ait
Null & 0 = 0: GotoAct, ok_pkt; # 3 ait-other
#
ok_pkt:
Null & 0 = 0: Count, 0;The subroutine begins at the rule labelled classify (shown below). It returns to the first, second or third rule after the invoking Gosub rule, depending on whether the tested PeerAddress is in the UA, AIT, or 'other' group of networks. In the listing below only one network is tested in each of the groups. It's trivial to add more rules (one per network) into either of the first two groups. In this example the subroutine Pushes the network number from the packet into the tested attribute before returning 🌈
The first invocation of classify (above) begins at the rule labelled ip_pkt. It Assigns SourcePeerAddress to V1 then executes a Gosub action. Classify returns to one of the three following rules. They will Goto from_ua or from_ait if the packet came from the UA or AIT groups, otherwise the PME will retry the match. This means that matched flows will have a UA or AIT network as their source, and flows between other networks will be ignored.
The next two invocations of 'classify' test the packet's DestPeerAddress. Packets from AIT to UA are Retried, forcing them to be counted as AU to AIT flows. Packets from UA to UA are ignored, as are packets from AIT to AIT.
classify:
v1 & 255.255.0.0 = 130.216.0.0: GotoAct, ua; # ua
v1 & 255.255.0.0 = 156.62.0.0: GotoAct, ait; # ait
Null & 0 = 0: Return, 3; # other
ua:
v1 & 255.255.0.0 = 0: PushPkttoAct, Next;
Null & 0 = 0: Return, 1;
ait:
v1 & 255.255.0.0 = 0: PushPkttoAct, Next;
Null & 0 = 0: Return, 2;3.4 More complicated rule sets
okay so this next example is where it gets spicy 🌶️💅 it groups IP flows by their Transport Address, which is basically the IP port number. If we just pushed every flow's SourceTransAddress and DestTransAddress, we'd get a huge pile of flows that only differ in whichever transport address isn't the well-known port. chaotic, bestie 😭
So instead we push the well-known port number into each flow's SourceTransAddress. DestTransAddress just stays zero by default 🎀
SourcePeerType & 255 = dummy: Ignore, 0;
SourcePeerType & 255 = IP: Pushto, IP_pkt;
Null & 0 = 0: GotoAct, Next;
SourcePeerType & 255 = 0: PushPkttoAct, Next;
Null & 0 = 0: Count, 0; # Count others by protocol type
#
IP_pkt:
SourceTransType & 255 = tcp: Pushto, tcp_udp;
SourceTransType & 255 = udp: Pushto, tcp_udp;
SourceTransType & 255 = icmp: CountPkt, 0;
SourceTransType & 255 = ospf: CountPkt, 0;
Null & 0 = 0: GotoAct, c_unknown; # Unknown transport type
#
tcp_udp:
s_domain:
SourceTransAddress & 255.255 = domain: PushtoAct, c_well_known;
s_ftp:
SourceTransAddress & 255.255 = ftp: PushtoAct, c_well_known;
s_imap:
SourceTransAddress & 255.255 = 113: PushtoAct, c_well_known;
s_nfs
SourceTransAddress & 255.255 = 2049: PushtoAct, c_well_known;
s_pop:
SourceTransAddress & 255.255 = 110: PushtoAct, c_well_known;
s_smtp:
SourceTransAddress & 255.255 = smtp: PushtoAct, c_well_known;
s_telnet:
SourceTransAddress & 255.255 = telnet: PushtoAct, c_well_known;
s_www:
SourceTransAddress & 255.255 = www: PushtoAct, c_well_known;
s_xwin
SourceTransAddress & 255.255 = 6000: PushtoAct, c_well_known;
#
DestTransAddress & 255.255 = domain: GotoAct, s_domain;
DestTransAddress & 255.255 = ftp: GotoAct, s_ftp;
DestTransAddress & 255.255 = 113: GotoAct, s_imap;
DestTransAddress & 255.255 = 2049: GotoAct, s_nfs;
DestTransAddress & 255.255 = 110: GotoAct, s_pop;
DestTransAddress & 255.255 = smtp: GotoAct, s_smtp;
DestTransAddress & 255.255 = telnet: GotoAct, s_telnet;
DestTransAddress & 255.255 = www: GotoAct, s_www;
DestTransAddress & 255.255 = 6000: GotoAct, s_xwin;
#
Null & 0 = 0: GotoAct, c_unknown; # 'Unusual' port
#
c_unknown:
SourceTransType & 255 = 0: PushPkttoAct, Next;
DestTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: PushPkttoAct, Next;
DestTransAddress & 255.255 = 0: CountPkt, 0;
#
c_well_known:
Null & 0 = 0: Count, 0
#so the first few rules ignore dummy packets, pick out the IP packets, and count everything else per protocol in one flow for each PeerType ✨ TCP and UDP packets make the PME Push their TransType and Goto tcp_udp, while ICMP and OSPF packets get counted in flows where only their TransType was pushed 🦋
at tcp_udp, the SourceTransAddress gets checked to see if it's one of the 'interesting' port numbers. if it is, the port number gets pushed from the rule into SourceTransAddress and the packet is counted at c_well_known. (NeMaC takes Pushto as a synonym for PushRuleto, which is a fun little detail 💜)
then the exact same test runs on the packet's DestTransAddress. if one of those succeeds, the PME Goes to the matching rule above and Pushes that port number into the flow's SourceTransAddress. if none of the tests pass, the packet is counted at c_unknown, where all the flow's Trans attributes get pushed. real talk though: for production use you'd want way more well-known ports in those tests. c_unknown is only meant for little-used exception flows, so don't let it become the main character 🙈
⚠️ these rules only push a value into a flow's SourceTransAddress, and they don't have any NoMatch actions. that means they don't pin down a packet's direction, so other rule sets can reuse them to group flows by well-known ports 🫶
the last example is the one that meters flows from a remote router, and it shows a totally different way of grouping well-known ports 🌈
SourceAdjacentAddress & FF-FF-FF-FF-FF-FF =
00-60-3E-10-E0-A1: Goto, gateway; # tmkr2 router
DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-60-3E-10-E0-A1:
Goto, gateway; # Source is tmkr2
Null & 0 = 0: Ignore, 0;
#
gateway:
SourcePeerType & 255 = IP: GotoAct, IP_pkt;
Null & 0 = 0: GotoAct, Next;
SourcePeerType & 255 = 0: CountPkt, 0;
#
IP_pkt:
SourceTransType & 255 = tcp: PushRuleto, tcp_udp;
SourceTransType & 255 = udp: PushRuleto, tcp_udp;
Null & 0 = 0: GotoAct, not_wkp; # Unknown transport type
#
tcp_udp:
SourceTransAddress & FC-00 = 0: GotoAct, well_known_port;
DestTransAddress & FC-00 = 0: NoMatch, 0;
Null & 0 = 0: GotoAct, not_wkp;
#
not_wkp:
DestTransAddress & 255.255 = 0: PushPkttoAct, Next;
well_known_port:
SourcePeerType & 255 = 0: PushPkttoAct, Next;
DestPeerType & 255 = 0: PushPkttoAct, Next;
SourcePeerAddress & 255.255.255.0 = 0: PushPkttoAct, Next;
DestPeerAddress & 255.255.255.0 = 0: PushPkttoAct, Next;
SourceTransType & 255 = 0: PushPkttoAct, Next;
DestTransType & 255 = 0: PushPkttoAct, Next;
SourceTransAddress & 255.255 = 0: CountPkt, 0;so the first group of rules checks whether the incoming packet's AdjacentAddresses belong to a flow that has an endpoint at the specified router. anything that doesn't is ignored 🙅♀️ non-IP packets get counted in flows that only have their PeerType pushed, which gives you one flow per non-IP protocol. IP packets with TransTypes other than UDP and TCP get counted at not_wkp, where all their address attributes get pushed 🧚
for UDP and TCP packets, the high-order six bits of SourceTransAddress are compared with zero. if that works out, the source port number is less than 1024, which means it's a well-known port 👑 the port number gets pushed from the rule into the flow's SourceTransAddress and the packet is counted at well_known_port. if that test fails, the same check runs on the packet's DestTransAddress. if the destination is a well-known port, the match gets Retried and succeeds with the well-known port as the flow's source 💖
and if later analysis ever shows that most of the observed flows come from non-well-known ports, you could add more pairs of rules that test each direction for other heavily-used ports 🍓
4 Flow data files
okay so the Architecture document [1] goes super deep on how the Traffic Flow Meter works and how a meter reader should collect flow data from a meter 🤓 but it says nothing about how that collected data should be stored. NeMaC uses a simple, self-documenting file format, and it has proved to be super effective in real use 💅
there are two kinds of records in a flow data file: flow records and information records 🎀 each flow record is just a sequence of attribute values with separators (these can be specified in a NeMaC rule file) or spaces between them, ending with a newline.
information records all start with a cross-hatch. the file's first record begins with ## and identifies the file as a file of data from NeTraMet. it records NeMaC's parameters and the time the collection was started ✨ the file's second record begins with #Format: and is a copy of the Format statement NeMaC used to collect the data.
the rest of the file is a sequence of collected data sets. each one starts with a #Time: record, which gives the time-of-day the collection was started, the meter name, and the range of meter times the collection represents 🦋 those from and to times are meter UpTimes, which means they're times in hundredths of seconds since the meter started operating. most analysis applications have just used the collection start times (which are ASCII time-of-day values), but the from and to times could be used to convert Uptime values to time-of-day 🕰️ the flow records that make up a data set follow the #Time record.
4.1 Sample flow data file
a sample flow data file is below 🌸 most of the flow records have been deleted, but lines of dots show where they used to be.
##NeTraMet v3.2. -c300 -r rules.lan -e rules.default
test_meter -i eth0 4000 flows starting at 12:31:27 Wed 1 Feb 95
#Format: flowruleset flowindex firsttime sourcepeertype
sourcepeeraddress destpeeraddress topdus frompdus
tooctets fromoctets
#Time: 12:31:27 Wed 1 Feb 95 130.216.14.251 Flows
from 1 to 3642
1 2 13 5 31.32.0.0 33.34.0.0 1138 0 121824 0
1 3 13 2 11.12.0.0 13.14.0.0 4215 0 689711 0
1 4 13 7 41.42.0.0 43.34.0.0 1432 0 411712 0
1 5 13 6 21.22.0.0 23.24.0.0 8243 0 4338744 0
3 6 3560 2 130.216.14.0 130.216.3.0 0 10 0 1053
3 7 3560 2 130.216.14.0 130.216.76.0 59 65 4286 3796
3 8 3560 7 0.0.255.0 1.144.200.0 0 4 0 222
3 9 3560 2 130.216.14.0 130.216.14.0 118 1 32060 60
3 10 3560 6 130.216.0.28 130.216.0.192 782 1 344620 66
3 11 3560 7 0.0.255.0 0.128.113.0 0 1 0 73
3 12 3560 5 59.3.13.0 4.1.152.0 1 1 60 60
3 13 3560 7 0.128.94.0 0.129.27.0 2 2 120 158
3 14 3560 5 59.3.40.0 4.1.153.0 2 2 120 120
3 15 3560 5 0.0.0.0 4.1.153.0 0 1 0 60
3 16 3560 5 4.1.152.0 59.2.189.0 2 2 120 120
. . . . . . . . .
3 42 3560 7 0.128.42.0 0.129.34.0 0 1 0 60
3 43 3560 7 0.128.42.0 0.128.43.0 0 1 0 60
3 44 3560 7 0.128.42.0 0.128.41.0 0 1 0 60
3 45 3560 7 0.128.42.0 0.129.2.0 0 1 0 60
3 46 3560 5 4.1.152.0 59.2.208.0 2 2 120 120
3 47 3560 5 59.3.46.0 4.1.150.0 2 2 120 120
3 48 3560 5 4.1.152.0 59.2.198.0 2 2 120 120
3 49 3560 5 0.0.0.0 59.2.120.0 0 1 0 60
3 50 3664 5 4.1.152.0 59.2.214.0 0 1 0 60
3 51 3664 5 0.0.0.0 4.2.142.0 0 1 0 60
3 52 3664 5 4.1.153.0 59.3.45.0 4 4 240 240
#Time: 12:36:25 Wed 1 Feb 95 130.216.14.251 Flows
from 3641 to 33420
3 6 3560 2 130.216.14.0 130.216.3.0 0 21 0 2378
3 7 3560 2 130.216.14.0 130.216.76.0 9586 7148 1111118 565274
3 8 3560 7 0.0.255.0 1.144.200.0 0 26 0 1983
3 9 3560 2 130.216.14.0 130.216.14.0 10596 1 2792846 60
3 10 3560 6 130.216.0.28 130.216.0.192 16589 1 7878902 66
3 11 3560 7 0.0.255.0 0.128.113.0 0 87 0 16848
3 12 3560 5 59.3.13.0 4.1.152.0 20 20 1200 1200
3 13 3560 7 0.128.94.0 0.129.27.0 15 14 900 1144
3 14 3560 5 59.3.40.0 4.1.153.0 38 38 2280 2280
3 15 3560 5 0.0.0.0 4.1.153.0 0 30 0 1800
3 16 3560 5 4.1.152.0 59.2.189.0 20 20 1200 1200
3 17 3560 5 0.0.0.0 59.2.141.0 0 11 0 660
. . . . . . . . .
3 476 26162 7 0.129.113.0 0.128.37.0 0 1 0 82
3 477 27628 7 0.128.41.0 0.128.46.0 1 1 543 543
3 478 27732 7 0.128.211.0 0.128.46.0 1 1 543 543
3 479 31048 7 0.128.47.0 2.38.221.0 1 1 60 60
3 480 32717 2 202.14.100.0 130.216.76.0 0 4 0 240
3 481 32717 2 130.216.76.0 130.216.3.0 0 232 0 16240
#Time: 12:41:25 Wed 1 Feb 95 130.216.14.251 Flows
from 33419 to 63384
3 6 3560 2 130.216.14.0 130.216.3.0 51 180 3079 138195
3 7 3560 2 130.216.14.0 130.216.76.0 21842 18428 2467693 1356570
3 8 3560 7 0.0.255.0 1.144.200.0 0 30 0 2282
3 9 3560 2 130.216.14.0 130.216.14.0 24980 1 5051834 60
3 10 3560 6 130.216.0.28 130.216.0.192 20087 1 8800070 66
3 11 3560 7 0.0.255.0 0.128.113.0 0 164 0 32608
3 12 3560 5 59.3.13.0 4.1.152.0 41 41 2460 2460
3 14 3560 5 59.3.40.0 4.1.153.0 82 82 4920 4920
3 15 3560 5 0.0.0.0 4.1.153.0 0 60 0 3600
. . . . . . . . .4.2. Flow data file features ✨
Okay so NeMaC's flow data files (the sample from above) have some features that are seriously worth knowing about 🧚
- Collection times overlap a little bit between samples 🫶. This means flows created after a collection started still get picked up, so nothing slips through the cracks 💅.
- The rule set can change during a run 🔄. The sample above shows flows from rule set 1 (the default set) in the first collection, followed by the first flows created by rule set 3, which NeMaC had just downloaded.
- The meter can reuse FlowIndexes once their flows have been recovered by the garbage collector ♻️. To identify a flow uniquely, you need the combination of
FlowRuleSet,FlowIndexandStartTime. It's giving detective energy 🔍. - Packet and Byte counters are 32-bit unsigned integers, and the meter never resets them 🔢. To work out the counts within a collection interval, you subtract the count from when the flow was last collected from the newly collected count. Counter wrap-around is handled by doing an unsigned subtraction and ignoring any carry. Slay 👑.
- In the sample flow data file above, I used double spaces as separators between the flow identifiers, peer addresses, pdu counts and packet counts 🎀.
- The format of addresses in the flow data file depends on the type of address 🌈. NeMaC always displays Adjacent addresses as six hex bytes separated by hyphens, and Transport addresses as (16-bit) integers. The format of a Peer address depends on its
PeerType. For example, IP uses dotted decimal. So NeMaC needs to know thePeerTypefor each flow, and the user has to request NeMaC to collect it 📝.
4.3. Terminating and restarting meter reading 🦋
When NeMaC first starts collecting from a meter, it reads the flow data for all active flows 🚀. This gives analysis applications a starting point for computing the counts between successive collections.
From time to time, the user needs to end a flow data file and start a new one 🌙. For example, someone might want a separate file for each day of metering. NeMaC handles this by closing the file after each collection, then opening it again and appending the data from the next collection. To end a file, the user just renames it. Unix will make the name change either right away (if the file is closed) or as soon as the current collection finishes and the file is closed ⏳.
When NeMaC begins its next collection, it notices that the file has disappeared, so it creates a new one and writes the # header records before writing the collected data 📂.
Here's the part that needs a little care 👀. The last data set in a file is not duplicated as the first data set of the next file. So analysis applications need to either look ahead at the first data set of the next file, or start by reading the last data set of the previous file. If they skip this, they'll lose one collection's worth of flow data at every change of file 😭.
5. Analysis applications 💜
Most analysis applications will be one-of-a-kind 💅. They'll take data from a locally-developed rule set and produce reports for specific local needs. The NeTraMet distribution files include three applications that are useful for everyone, as follows:
fd_filtercomputes data rates, which means the differences between successive data sets in a flow data file 📈. It also lets the user assign a 'tag' number to each flow. These are 'computed' attributes, similar toFlowClassandFlowKind. The only difference is that they're computed from the collected data sets 🏷️.fd_extracttakes 'tagged' files fromfd_filterand produces simple 'column list' files for other programs 🗂️. One common use forfd_extractis making time-series data files that you can plot with utilities like GNUPlot 📊.nm_rcis a 'remote console' for a NeTraMet meter 🖥️. It's a slightly simplified version of NeMaC combined withfd_filter. You can use it to monitor any meter, and it displays (as lines of text characters) information about the n busiest flows observed during each collection interval.niftyis a traffic flow analyser that, likenm_rc, displays data from a NeTraMet meter ✨. It's an X/Motif application that makes displays like 'Packet rate (pps) vs Flow lifetime (minutes),' so the 'interesting' flows stand out 🔥.
These apps are super useful on their own, and they're a great starting point if you want to write your own analysis application 🌸.
6. Using NeTraMet in a measurement system 🌸
This section gives a quick summary of the steps for setting up a traffic measurement system with NeTraMet 🗺️. Here they are:
- Decide what you're measuring 🤔. A good way to start is to specify exactly which flows are to be measured, and what reports you'll need. Specifying the flows should make it obvious where meters have to go so the flows can be observed, whether PCs will be good enough for the job, and so on.
- Install meters 🔧. As well as physically placing the meter hosts, this includes making sure they're configured correctly, with appropriate IP addresses, SNMP community strings, and so on.
- Develop the rule set (and a standby rule set) 📜. How hard this is depends on how much you already know about the traffic. One possible approach is to start with the meter default rule set and measure how much traffic there is for each
PeerType. (This is a good way to verify that NeTraMet and NeMaC are working properly 🙌.) You can then add rules to increase the granularity of the flows. This will of course increase the number of flows collected, and force the meter's garbage collector to work harder 🧹. Another approach is to try a rule set with very fine granularity (one thatPushesall the address attributes), then observe how many flows are collected every few minutes ⏱️. - Develop a strategy for controlling the meter reader 🎛️. This means setting the meter's maximum number of flows, the collection interval, how breaks between flow data files will be handled, how often NeMaC should check that the meter is running, and so on.
- Develop application(s) to process the collected flow data and produce the required files and reports 📊.
- Test run 🧪. Monitor the system, then refine the rule sets and meter reading strategy until the overall system performance is satisfactory 💖.
This process can take quite a long time, but the overall result is so worth the effort 🥹.
6.1. Examples of NeTraMet in production use 🌍
At the University of Auckland, we run two sets of meters 🇳🇿. One set measures the traffic entering and leaving our University network, and generates usage reports for all our Internet users. This has been in production since early 1994.
The other set consists of meters distributed at Universities throughout New Zealand 🏫. They provide continuous traffic flow measurements at five-minute intervals for all the links making up the Universities' network (Kawaihiko). This system has been in production since January 1996, and it has already proved very useful in planning the network's development 🚀.
The Kawaihiko Network provides IP connectivity for the New Zealand Universities 💻. They're linked via a Frame Relay cloud, using a partial mesh of permanent virtual circuits. There's a NeTraMet meter at each site, metering inward and outward traffic. All the meters are managed from Auckland, and they all run copies of the same rule set 🎀.
The rule set has about 650 rules, most of which sit in a single subroutine. That subroutine classifies PeerAddresses into three categories: 'Kawaihiko network,' 'other New Zealand network' and 'non-New Zealand network.' Inside New Zealand, IP addresses lie within six CIDR blocks, and there are about four hundred older networks with addresses outside those blocks 🧩. The rules are grouped by subnet size, so all the /24 networks are tested first, then the /23 networks, and so on, finishing with the /16 networks. This means that although there are about 600 networks, any PeerAddress can be classified with only nine tests. Efficiency is everything 😤.
The Kawaihiko rule set classifies flows using computed attributes. These indicate the network 'kind' (Kawaihiko / New Zealand / international) for each flow's SourcePeerAddress and DestPeerAddress, and whether the flow is a 'network news' flow or not 📰.
Flow data is collected from all of the meters every five minutes, and it's used to produce weekly reports, as follows:
- Traffic Plots. Plots of the 5-minute traffic rates for each site, showing international traffic in and out, news traffic in and out, and total traffic in and out of the site 📈.
- Traffic Matrices. Two of these are produced: one for news traffic, the other for total traffic. They show the traffic rates from every site (including 'other New Zealand' and 'international') to every other site. The mean, third quartile and maximum are printed for every cell in the matrices 🧮.
7. Acknowledgments 🫶
This memo documents the implementation work on traffic flow measurement here at the University of Auckland 💖. Many of my University colleagues contributed significantly to this work, especially Russell Fulton, who developed the rule sets, Perl scripts and Cron jobs that produce our traffic usage reports automatically, week after week 👑. Thanks also to John White for his patient help in documenting the project 🌷.
8. References 📚
[1] Brownlee, N., Mills, C., and G. Ruth, "Traffic Flow Measurement: Architecture", RFC 2063, The University of Auckland, Bolt Beranek and Newman Inc., GTE Laboratories, Inc, January 1997.
[2] Brownlee, N., "Traffic Flow Measurement: Meter MIB", RFC 2064, The University of Auckland, January 1997.
[3] CRYNWR Packer Drivers distribution site: http://www.crynwr.com/
[4] Case J., McCloghrie K., Rose M., and Waldbusser S., "Structure of Management Information for version 2 of the Simple Network Managemenet Protocol", RFC 1902, SNMP Research Inc., Hughes LAN Systems, Dover Beach Consulting, Carnegie Mellon University, April 1993.
[5] IBM Corporation, "IBM PC Technical Reference Manual," 1984.
[6] Waterloo TCP distribution site: http://mvmpc9.ciw.uni-karlsruhe.de:80/d:/public/tcp_ip/wattcp
[7] CMU SNMP distribution site: ftp://lancaster.andrew.cmu.edu/pub/snmp-dist
[8] libpcap distribution site: ftp://ftp.ee.lbl.gov/libpcap-*.tar.gz
9. Security Considerations 🔐
Security issues are not discussed in detail in this document. The meter's management and collection protocols are responsible for providing sufficient data integrity and confidentiality 🛡️. Don't skip this part, bestie 🙈.
10. Author's Address 💌
Nevil Brownlee
The University of Auckland
Phone: +64 9 373 7599 x8941
Email: n.brownlee@auckland.ac.nz 🦋
📦 where this page comes from
- ietf: rfc2123 1997-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms