RFC 1187

Bulk Table Retrieval with the SNMP 🦋

okay bestie, this is RFC 1187 and it is a whole vibe 💅 It was written in October 1990 by M. Rose (Performance Systems International, Inc.), K. McCloghrie (Hughes LAN Systems), and J. Davin (MIT Laboratory for Computer Science). It lives in the Network Working Group, and it's an Experimental Protocol 🧪, not a standard. Its header doesn't list anything it obsoletes or updates. It's just a memo that's looking for discussion and suggestions ✨

1. Status of this Memo ✨

This memo shares an interesting family of algorithms for bulk table retrieval using the Simple Network Management Protocol (SNMP). It describes an Experimental Protocol for the Internet community and asks for discussion and suggestions for improvements. It does not specify a standard for the Internet community. For the standardization state and status of this protocol, check the current edition of the "IAB Official Protocol Standards" 📋 Distribution of this memo is unlimited 🌈

Table of Contents 🎀

1. Status of this Memo ..................................    1
2. Abstract .............................................    1
3. Bulk Table Retrieval with the SNMP ...................    2
4. The Pipelined Algorithm ..............................    3
4.1 The Maximum Number of Active Threads ................    4
4.2 Retransmissions .....................................    4
4.3 Some Definitions ....................................    4
4.4 Top-Level ...........................................    5
4.5 Wait for Events .....................................    6
4.6 Finding the Median between two OIDs .................    8
4.7 Experience with the Pipelined Algorithm .............   10
4.8 Dynamic Range of Timeout Values .....................   10
4.9 Incorrect Agent Implementations .....................   10
5. The Parallel Algorithm ...............................   11
5.1 Experience with the Parallel Algorithm ..............   11
6. Acknowledgements .....................................   11
7. References ...........................................   12
Security Considerations..................................   12
Authors' Addresses.......................................   12

2. Abstract 💖

This memo reports an interesting family of algorithms for bulk table retrieval using the Simple Network Management Protocol (RFC 1157) [1].

You're expected to already know SNMP and its powerful get-next operator 🧠 Send your comments to: Marshall T. Rose <mrose@psi.com> 💌

3. Bulk Table Retrieval with the SNMP 🧚

okay so the big question is whether bulk table retrieval can even work with the powerful get-next operator. Empirical evidence shows that get-next is effective for table traversal, especially when the management station only wants well-defined subsets of a particular table. Some people worried it couldn't be done efficiently, but recent experience says otherwise 🙌

The simplest version is to grab one object at a time. To traverse the entire ipRoutingTable, the management station starts with:

get-next (ipRouteDest)

which might return:

ipRouteDest.0.0.0.0

Then it keeps invoking get-next, using the value from the previous response:

get-next (ipRouteDest.0.0.0.0)

As the sequence goes on, each column of the ipRoutingTable can be retrieved, like this:

get-next (ipRouteDest.192.33.4.0)

which might return:

ipRouteIfIndex.0.0.0.0

Eventually you get a response that's outside the table, like this:

get-next (ipRouteMask.192.33.4.0)

which might return:

ipNetToMediaIfIndex.192.33.4.1

So with this scheme, O(rows x columns) management operations are required to retrieve the entire table. Honestly, this is obviously sub-optimal 😭 because get-next can take several operands. So the first step is to grab a whole row with each operation, like this:

get-next (ipRouteDest, ipRouteIfIndex, ..., ipRouteMask)

which might return:

ipRouteDest.0.0.0.0
ipRouteIfIndex.0.0.0.0
ipRouteMask.0.0.0.0

The management station can then keep invoking get-next, using the results of the previous operation as the operands for the next one. With this scheme, O(rows) management operations are required to retrieve the entire table ✨

Some people say this is a weakness of SNMP, since O(rows) serial operations take a lot of time. The problem is that those arguments put an implicit emphasis on the word "serial" 🙄 There's actually nothing stopping a clever management station from invoking get-next several times, each with different operands, to get parallelism and pipelining in the network. And this approach needs no changes in SNMP, nor does it add any significant burden to the agent 💅

4. The Pipelined Algorithm 🎀

Let's get into an algorithm for bulk table retrieval with SNMP. To keep things brief, the "pipelined algorithm" only retrieves a single column from the table. Without loss of generality, the pipelined algorithm can easily be extended to retrieve all columns 🌷

The algorithm uses a multi-threaded approach: each thread generates its own stream of get-next requests and processes the resulting stream of responses. For a given thread, a request corresponds to a different row in the table 🧵

Overall retrieval efficiency gets better when you can keep several transactions in transit, and when the agent and management station process transactions at the same time 🚀

The algorithm adapts itself to changing network conditions and topologies, as well as changing loads on the agent. It does this by varying two things: the number of threads that are active (meaning the number of transactions being processed and in transit), and the retransmission timeout. Both get adjusted based on the transaction round-trip-time and on the loss or timeout of transactions 🦋

4.1. The Maximum Number of Active Threads 🍓

Figuring out how many threads to have active at a time has to be dynamic if you want the best results 💯 With only one thread active, the pipelined algorithm degenerates into the serial algorithm mentioned earlier. With more threads than necessary, there's a danger of overrunning the agent, whose only option is to drop requests, which is wasteful 😤 The ideal number is just enough that the next request arrives at the agent right as it finishes processing the previous one. This depends on the round-trip time, which changes dynamically with network topology and traffic load, and can also be different for different tables in the same agent 🌐

With too few threads active, the round-trip time barely goes up with each new thread. With too many, the round-trip time goes up by the amount of time the agent takes to process one request. So the number is dynamically estimated by calculating the round-trip-time divided by the number of active threads. Whenever this value hits a new minimum, the limit on the number of threads gets adjusted to the number of threads that were active when the corresponding request was sent (plus one to allow for loss of requests) 🫶

4.2. Retransmissions 🌸

When there are no gateways between the manager and agent, the likelihood of requests and responses arriving in order is pretty high ✨ Right now, the decision to retransmit is based solely on the timeout. One possible optimization is for the manager to remember the order in which requests were sent and match that against incoming responses. If one thread gets a response before another thread that sent an earlier request, then lossage could be assumed, and a retransmission could be made immediately 🔥

4.3. Some Definitions 💜

First, let's define a "thread" as some state information kept in the management station that corresponds to a portion of the table to be retrieved. A thread has several bits of information attached to it:

• (1) the range of SNMP request-ids which this thread can use, along with the last request-id used;
• (2) last SNMP message sent, the number of times it has been (re)sent, the time it was (re)sent;
• (3) the inclusive lower-bound and exclusive upper-bound of the object-instance for the portion of the table that this thread will retrieve, along with the current object-instance being used;
• (4) the number of threads which were active at the time it was last sent;

When a thread is created, it automatically sends a get-next message using its inclusive lower-bound OID. After that, it's placed at the end of the "thread queue" 🧵

We also need an OID, which is defined as a concrete representation of an object identifier with two parts:

• (1) the number of sub-identifiers present, "nelem";
• (2) the sub-identifiers themselves in an array, "elems", indexed from 1 up to (and including) "nelem".

---

part 2 is coming up next, so don't go anywhere 🫶💖

4.4. Top-Level ✨

Okay so the top-level is literally just: start three threads, then loop 🔄 As long as threads exist, it waits for events and then reacts to whatever came in 💅 For each thread that got a response, we check whether the OID of the response is greater than or equal to the exclusive upper-bound of that thread. If it is, that thread's chunk of the table is totally retrieved, so the thread gets destroyed 🫶

If it isn't, we store the variable bindings from the response. Then, if a new thread should be created, that thread's portion of the table gets split up accordingly 🔪 Either way, another powerful get-next operator gets issued on behalf of the thread 🌸

The starting positions (column, column.127, and column.192) were picked to make optimal partitions for tables indexed by IP addresses 🧚 The algorithm could easily be modified to pick other partitions, however, it must be stressed that the current choices work for any tabular object 💖

pipelined_algorithm (column)
OID  column;
{

    timeout ::= some initial value;

    start new thread for [column, column.127);
    start new thread for [column.127, column.192);
    start new thread for [column.192, column+1);

    while (threads exist) {
        wait for events;
        foreach (thread that has an incoming message,
                 examined in order from the thread queue) {
            OID     a;

            if (message's OID >= thread's upper-bound) {
                destroy thread;
                continue;
            }

            store variable-bindings from message;

            if (number of simultaneous threads does NOT
                        exceed a maximum number
                     && NOT backoff
                     && (a ::= oid_median (message's OID,
                                           thread's
                                               upper-bound))) {
                start new thread for [a, thread's upper-bound);
                thread's upper-bound ::= a;
                place thread at end of thread queue;
                backoff ::= TRUE;
            }
            do another get-next for thread;
        }
    }
}

---

4.5. Wait for Events ⏰

Waiting for events is basically: wait a small amount of time, OR until at least one message shows up 👀

Any messages that show up get collated with the right thread 🧩 On top of that, we measure the largest round-trip time for request/responses and calculate the maximum number of active threads 📊

Next up, the timeout gets adjusted 🎀 If no responses came back, the timeout gets doubled. Otherwise, a timeout-adjustment is calculated as 1.5 times the largest observed round-trip time. If that timeout-adjustment is greater than the current timeout, the current timeout is set to the timeout-adjustment. Otherwise, the current timeout gets averaged with the timeout-adjustment ⚖️

Finally, if at least one thread didn't get a response, we find the thread that has waited the longest 🥺 If the elapsed time (with noise factor) since the last request (or retransmission) is greater than the current timeout value, another retransmission is attempted 🔁

wait for events ()
{
    backoff ::= TRUE, maxrtt ::= 0;
    find the thread which has been waiting the longest
        for a response;
    timedelta = timeout
                    - time since request was sent for thread;
    wait up to timedelta seconds or until some messages arrive;

    if (least one message arrived) {
        discard any messages which aren't responses;
        foreach (response which corresponds to a thread) {
            if (the response is a duplicate)
                drop it and continue;

            if (this response is for a message that was
                    not retransmitted) {
                if (the round-trip time is larger than maxrtt)
                    set maxrtt to the new round-trip time;
                if (round-trip time / number of active threads
                      < minimum previous round-trip time / number
                           of active threads) {
                    set new minimum round-trip time per number
                        of active threads
                    set new maximum number of threads
                }
                backoff ::= FALSE;
            }
        }
    }
    if (backoff)
        double timeout;
    elsif (maxrtt > 0) {
        timeadjust ::= maxrtt * 3 / 2;
        if (timeadjust > timeout)
            timeout ::= timeadjust; backoff ::= TRUE;
        else
            timeout ::= (timeout + timeadjust) / 2;
    }
    if (timeout exceeds some threshold)
        set timeout to that threshold;
    elsif (timeout is smaller than some threshold)
        set timeout to that threshold;

    if (at least one thread didn't receive a response) {
        find the thread which has been waiting the longest
            for a response,
            and determine the elapsed time since a message
            was sent;
        if (the elapsed time with noise is greater than timeout) {
            if (the number of retransmissions for this thread
                    exceeds a threshold)
                abort the algorithm;
            retransmit the request;
            backoff ::= TRUE;
        }
    }
}

---

4.6. Finding the Median between two OIDs 🧮

The object identifier space is neither uniform nor continuous, which is kind of a vibe-killer 😭 So it's not always possible to pick an object identifier that's lexicographically-between two arbitrary object identifiers. Because of that, the pipelined algorithm makes a best-effort attempt 💅

Starting from the beginning, each sub-identifier of the two OIDs gets scanned until a difference shows up. At that point there are a few possible conditions:

1. The upper OID has run out of sub-identifiers. In this case, either the two OIDs are identical or the lower OID is greater than the upper OID (an interface error), so no OID is returned 🙅
2. The lower OID has run out of sub-identifiers. In this case, the first subsequent non-zero sub-identifier from the upper OID is located. If no such sub-identifier is found, then no OID exists between the lower and upper OIDs, and no OID is returned. Otherwise, a copy of the upper OID is made, but truncated at this non-zero sub-identifier, which is then halved, and the resulting OID is returned ✂️
3. Otherwise, a copy of the lower OID is made, but truncated at the point of difference. This last sub-identifier is then set to the arithmetic mean of the difference. In the case where the difference is only 1 (so the last sub-identifier stays the same), a new sub-identifier is added, taking care to be larger than a possible sub-identifier present in the lower OID. Regardless, the resulting OID is returned 🦋

oid_median (lower, upper)
OID     lower,
        upper;
{
    for (i ::= 1; i < upper:nelem; i++) {
        if (i > lower:nelem) {
            while (upper:elems[i] == 0)
                if (++i > upper:nelem)
                    return NULL;
            median ::= copy of upper;
            median:nelem ::= i;
            median:elems[i] ::= upper:elems[i] / 2;

            return median;
        }

        if (lower:elems[i] == upper:elems[i])
            continue;

        median ::= copy of lower;
        median:nelem ::= i;
        median:elems[i] ::= (lower:elems[i]+upper:elems[i])/2;
        if (median:elems[i] == lower:elems[i]) {
            median:nelem ::= (i + 1);
            if (lower:nelem < i)
                median:elems[median:nelem] ::= 127;
            elsif ((x ::= lower:elems[i + 1]) >= 16383)
                median:elems[median:nelem] ::= x + 16383;
            elsif (x >= 4095)
                median:elems[median:nelem] ::= x + 4095;
            elsif (x >= 1023)
                median:elems[median:nelem] ::= x + 1023;
            elsif (x >= 255)
                median:elems[median:nelem] ::= x + 255;
            else median:elems[median:nelem] ::=
                                        (x / 2) + 128;
        }

        return median;
    }
    return NULL;
}

---

4.7. Experience with the Pipelined Algorithm 🧪

This pipelined algorithm has been implemented, and some experimentation has been done 🎉 It would be premature to share extensive performance figures right now, because the pipelined algorithm is still being tuned and is only implemented in a prototype setting 🛠️ Still, on tables of size O(2500), performance of 121 entries/second has been observed. For comparison, the serial algorithm manages roughly 56 entries/second for the same table 🏎️💨

4.8. Dynamic Range of Timeout Values ⏳

It should be noted that the pipelined algorithm takes a simplistic approach with the timeout value: it does not maintain a history of the value and act accordingly 🙈

For example, if the timeout reaches the maximum timeout limit and then latches for some period of time, this indicates a resource (either the network or the agent) is saturated 🥵 Unfortunately, a solution is difficult. An elegant approach would be to combine two threads, but it is quite possible that no two consecutive threads exist when this determination is made 😩 Another approach might be to delay the transmission for threads which are ready to issue a new get-next operation 🤔

Similarly, if the timeout drops to the minimum value and subsequently latches, more threads should be started 🚀

4.9. Incorrect Agent Implementations 🐛

An interesting result is that many agents do not properly implement the powerful get-next operator 👀 In particular, when a get-next request contains an operand with an arbitrarily-generated suffix, some agent implementations will handle this improperly and ultimately return a result which is lexicographically less than the operand! 💀

A typical cause of this is when the instance-identifier for a columnar object is formed by a MAC or IP address, so each octet of the address forms a sub-identifier of the instance-identifier 🧠 In such circumstances, the incorrect agent implementations compare against only the least significant octet of the sub-identifiers in the operand, instead of the full value of the sub-identifiers. Rude 😤

Upon encountering such an interaction, the pipelined algorithm implementation declares the thread dead (noting a possible gap in the table) and continues 🫡

5. The Parallel Algorithm ✨

Okay bestie, this is where it gets spicy 🌶️. The first optimization is to split the job into two steps. In step one, you walk through one column of the table to figure out the full range of instance identifiers that actually mean something in the table. (Yes, the pipelined algorithm we just talked about grabs a single column, but the prototype can grab multiple columns, so ✨ extra flex ✨.) In step two, you pull the other columns with the SNMP get operation, because the instance identifiers are already known by then 🧠. The manager can also dynamically figure out how many variables to stuff into one SNMP get operation so it sends the fewest requests possible 📦. Here's the lowkey genius part: the agent's get operation is usually way cheaper than the powerful get-next operation, so when you're requesting multiple columns, this two-step process takes less execution time on the agent. It's giving efficiency 💅.

Next up is the "parallel algorithm" 🦋. Right now, each thread gets mapped onto a single SNMP operation. The big insight is to map several threads onto a single SNMP operation instead. The manager has to dynamically figure out how many variables can fit into a single powerful get-next operation 🧩. The payoff is less traffic on the wire 🚚, but the tradeoff is that the agent has to use more resources for each request 😬. Not gonna lie, it's a classic "pick your poison" situation.

Here's how it all connects, and I'm obsessed with this part 😭. Serial retrieval of objects is basically a degenerate case of the pipelined algorithm, where the number of active threads is just one. In the same way, the pipelined algorithm is a special case of the parallel algorithm, where the number of threads per SNMP operation is one. It's like a little family tree of algorithms, and I think that's iconic 🌳💜.

5.1. Experience with the Parallel Algorithm

The parallel algorithm has been implemented, and some experimenting has happened 🧪. It would be premature to share extensive performance figures right now, though, because the algorithm is still being tuned and it only lives in a prototype setting 🙈. That said, on tables of size O(2500), people observed performance of 320 entries/second, which is a 571% performance improvement over the serial algorithm 🚀🔥. Okay, that number is absolutely unhinged in the best way.

6. Acknowledgements 🫶

A lot of the pipelining ideas were motivated by Van Jacobson's work on adaptive timers in TCP 🌸. The parallelization modifications were originally suggested by Jeffrey D. Case 👑.

Finally, the comments of the following individual is acknowledged 💖:

Frank Kastenholz, Racal-Interlan

7. References 📚

[1] Case, J., Fedor, M., Schoffstall, M., and J. Davin, Simple Network Management Protocol (SNMP), RFC 1157, SNMP Research, Performance Systems International, Performance Systems International, MIT Laboratory for Computer Science, May 1990.

---

Security Considerations 🔐

Security issues are not discussed in this memo. 💀 Yes, you read that right. The security side is left out entirely, so don't assume anything is covered here!

Authors' Addresses 💌

Marshall T. Rose
PSI, Inc.
PSI California Office
P.O. Box 391776
Mountain View, CA 94039

Phone: (415) 961-3380
EMail: mrose@PSI.COM


Keith McCloghrie
Hughes LAN Systems
1225 Charleston Road
Mountain View, CA 94043

Phone: (415) 966-7934
EMail: KZM@HLS.COM


James R. Davin
MIT Laboratory for Computer Science, NE43-507
545 Technology Square
Cambridge, MA 02139

Phone:  (617) 253-6020
EMail:  jrd@ptt.lcs.mit.edu