RFC 1224
Techniques for managing asynchronously generated alerts
RFC 1224: Techniques for Managing Asynchronously Generated Alerts 🎀
Okay bestie, header info first 💖 This one comes from the Network Working Group and is numbered RFC 1224. L. Steinberg from IBM Corporation wrote it, and it's dated May 1991 🗓️. It's an Experimental protocol doc (see the status section below), and the title block doesn't list anything it obsoletes or updates 🧾.
Status of this Memo ✨
So first, the vibe check on what this memo is for: it lays out common mechanisms for managing alerts that get produced asynchronously, in a way that's consistent with the network management protocols people use right now 💅
Real talk, this memo specifies an Experimental Protocol for the Internet community 🧪 It's asking for discussion and suggestions for improvement, so bring your ideas 🫶 For the standardization state and status of this protocol, check the current edition of the "IAB Official Protocol Standards" 📚 And distribution of this memo is unlimited 🌈
Abstract 🦋
Okay so this RFC explores ways to stop a remotely managed entity from drowning its manager (or the network) in an unexpected flood of network management info 🌊 while also making sure the "important" stuff actually gets delivered 💌 The focus is on controlling the flow of asynchronously generated information, not on how that information gets generated 🎯
Table of Contents 📑
1. Introduction................................................... 2
2. Problem Definition............................................. 3
2.1 Polling Advantages............................................ 3
(a) Reliable detection of failures............................... 3
(b) Reduced protocol complexity on managed entity................ 3
(c) Reduced performance impact on managed entity................. 3
(d) Reduced configuration requirements to manage remote entity... 4
2.2 Polling Disadvantages......................................... 4
(a) Response time for problem detection.......................... 4
(b) Volume of network management traffic generated............... 4
2.3 Alert Advantages.............................................. 5
(a) Real-time knowledge of problems.............................. 5
(b) Minimal amount of network management traffic................. 5
2.4 Alert Disadvantages........................................... 5
(a) Potential loss of critical information....................... 5
(b) Potential to over-inform a manager........................... 5
3. Specific Goals of this Memo.................................... 6
4. Compatibility with Existing Network Management Protocols....... 6
5. Closed Loop "Feedback" Alert Reporting with a "Pin" Sliding
Window Limit................................................... 6
5.1 Use of Feedback............................................... 7
5.1.1 Example..................................................... 8
5.2 Notes on Feedback/Pin usage................................... 8
6. Polled, Logged Alerts.......................................... 9
6.1 Use of Polled, Logged Alerts.................................. 10
6.1.1 Example..................................................... 12
6.2 Notes on Polled, Logged Alerts................................ 12
7. Compatibility with SNMP and CMOT .............................. 14
7.1 Closed Loop Feedback Alert Reporting.......................... 14
7.1.1 Use of Feedback with SNMP................................... 14
7.1.2 Use of Feedback with CMOT................................... 14
7.2 Polled, Logged Alerts......................................... 14
7.2.1 Use of Polled, Logged Alerts with SNMP...................... 14
7.2.2 Use of Polled, Logged Alerts with CMOT...................... 15
8. Notes on Multiple Manager Environments......................... 15
9. Summary........................................................ 16
10. References.................................................... 16
11. Acknowledgements.............................................. 17
Appendix A. Example of polling costs............................. 17
Appendix B. MIB object definitions............................... 19
Security Considerations........................................... 22
Author's Address.................................................. 221. Introduction 🌸
Okay, intro time 💖 This memo defines ways to keep a remotely managed entity from dumping an unexpected amount of network management information on a manager or the network, while still making sure "important" information gets delivered. Same focus as the abstract: controlling the flow of asynchronously generated information, not how it's generated. Generating that info and controlling its generation can involve protocol-specific stuff 🛠️
So there are two understood ways to move network management information from a managed entity to a manager 🦋: request-response driven polling, and the unsolicited sending of "alerts". An alert is any management info delivered to a manager that isn't the result of a specific query. Both methods have advantages and disadvantages, and section 2 breaks them all down 🧠
Here's the scary part 😭 Alerts in a failing system can get generated so fast that they actually hurt the resources they're running on. They might also fail to get delivered, and critical info could get lost. So we need ways to limit how much alert traffic gets sent AND help a minimum amount of info reach the manager 📉
We believe managed agents that can asynchronously generate alerts should try to adopt mechanisms that meet both of these needs 🫶 Section 2.4 explains why both alert-management requirements are necessary 💅 A complete alert-driven system has to make sure alerts are delivered, or that their loss is detected, with a way to recreate the lost information, AND it must not let itself overburden its manager with an unreasonable amount of information 🎀
2. Problem Definition 🔥
Here's the tea ☕ The following discusses the relative advantages and disadvantages of polled vs. alert-driven management.
2.1 Polling Advantages 💜
(a) Reliable detection of failures.
A manager that polls for all of its information can way more readily figure out machine and network failures 🕵️♀️ If a query gets no response, that means something's wrong with the machine or network. A manager that relies on notification of problems might assume a faulty system is good, if the alert can't reach its destination or the managed system can't correctly generate the alert 😬 Examples include network failures (where an isolated network can't deliver the alert) and power failures (where a failing machine can't generate an alert). More subtle forms of failure in the managed entity might produce an incorrectly generated alert, or no alert at all 🙈
(b) Reduced protocol complexity on managed entity
The request-response system is based on conservative assumptions about the underlying transport protocol 🧱 Timeouts and retransmits (re-requests) can be built into the manager. Plus, this lets the manager directly affect how much network management information flows across the network 🎛️
(c) Reduced performance impact on managed entity
In a purely polled system, there's no danger of having to often test for an alert condition 😌 That testing takes CPU cycles away from the real mission of the managed entity. Honestly, testing a threshold on each packet received could have unwanted performance effects on machines like gateways 🚧 If you want to use thresholds and alerts, you have to choose the parameters being tested with great care, and people should be strongly discouraged from updating statistics and checking values frequently 🙅♀️
(d) Reduced Configuration Requirements to manage remote entity
Remote, managed entities don't need to be configured with one or more destinations for reporting information 🗺️ Instead, the entity just responds to whoever makes a specific request. When you change the network configuration, there's never a need to reconfigure all the remote manageable systems ✨ Also, any number of "authorized" managers (meaning those passing whatever authentication tests the network management protocol imposes) can get information from any managed entity 🔑 This happens without reconfiguring the entity and without hitting an entity-imposed limit on the maximum number of potential managers 💪
2.2 Polling Disadvantages 🥲
(a) Response time for problem detection
Having to poll many MIB [2] variables per machine on a large number of machines is honestly a real problem 😩 The manager's ability to monitor a system like that is limited: if a system fails shortly after being polled, there may be a significant delay before it's polled again. During that time, the manager has to assume a failing system is acceptable 🫠 See Appendix A for a hypothetical example of such a system 📖
It's worth noting that while improving the mean time to detect failures might not greatly improve the time to correct the failure, the problem generally won't get repaired until it's detected 🔧 Plus, most network managers would prefer to at least detect faults before network users start phoning in 📞
(b) Volume of network management traffic
Polling many objects (MIB variables) on many machines greatly increases the amount of network management traffic flowing across the network (see Appendix A) 📈 You can minimize this by using hierarchies (polling a machine for a general status of all the machines it polls), but that makes the response time problem from before even worse 😵💫
2.3 Alert Advantages 🌟
(a) Real-time Knowledge of Problems
Letting the manager be notified of problems gets rid of the delay imposed by polling many objects/ systems in a loop ⚡
(b) Minimal amount of Network Management Traffic
Alerts are only sent due to detected errors 🚨 Since you don't have to transfer big chunks of status info that just proves a system is healthy, network and system (machine processor) resources get freed up to do their primary mission 🏃♀️
2.4 Alert Disadvantages 💀
(a) Potential Loss of Critical Information
Okay bestie, this one is kinda scary 😭 Alerts are most likely not to get delivered when the managed entity fails (power supply fails) or the network is having problems (saturated or isolated). Just remember that failing machines and networks can't be trusted to tell a manager that they're failing. 🙈
(b) Potential to Over-inform the Manager
So picture an "open loop" system, where the flow of alerts to a manager is fully asynchronous 🌀 It can end up sending a flood of alerts (like link up/down messages when lines vacillate). That info piles extra stress onto a network that's already strained, and it could stop the manager from disabling the thing that's generating the alerts. All the bandwidth going into the manager could get saturated with incoming alerts. Not a vibe 💀
Most major network management systems try to use an optimal mix of alerts and polling. Doing that keeps the advantages of each while getting rid of the disadvantages of pure polling. It's giving balance ✨
3. Specific Goals of this Memo ✨
This memo suggests mechanisms to cut down on the disadvantages of using alerts 🦋 An optimal system knows about the problems that come with sending too many alerts, where a manager gets so overwhelmed it becomes ineffective at managing. It also knows about the problems of not using alerts enough, especially given the huge volumes of data that have to be actively monitored, which scale poorly. The author's belief is that this is best handled by allowing alert mechanisms that "close down" automatically when they're over-delivering asynchronous (unexpected) alerts. They should also allow a flow of synchronous alert information through a polled log. The use of "feedback" (with a sliding window "pin") discussed in section 5 handles the first need, while the discussion in section 6 on "polled, logged alerts" handles the second. 💅
This memo does not try to define mechanisms for controlling the asynchronous generation of alerts, since those matters depend on the specifics of the management protocol. It also doesn't try to define what the content of an alert should be. The feedback mechanism does need one single alert type added, but that isn't meant to impact or influence the techniques for generating any other alert. (It can itself be generated from a MIB object or the management protocol.) To make any real use of the alert mechanisms described in this memo, several MIB objects have to be implemented in the relevant managed systems. The location of these objects in the MIB is under an experimental subtree delegated to the Alert-Man working group of the Internet Engineering Task Force (IETF), and it's published in the "Assigned Numbers" RFC [5]. Right now, this subtree is defined as:
alertMan ::= { experimental 24 }4. Compatibility With Existing Network Management Protocols 🎀
The goal of this document is to suggest mechanisms that violate neither the letter nor the spirit of the protocols in CMOT [3] and SNMP [4]. To make that happen, each mechanism described will give an example of how it's used in a conformant way with both SNMP and CMOT. Very respectful, very careful 🫶
5. Closed Loop "Feedback" Alert Reporting with a "Pin" Sliding Window Limit 🌈
One technique for stopping an excess of alerts from being delivered involves required feedback to the managed agent. The name "feedback" describes a required positive response from a potentially "over-reported" manager, which has to come before a remote agent can keep transmitting alerts at a high rate. A sliding window "pin" threshold (named after the metal on the end of a meter 📏) is set up as part of a user-defined SNMP trap, or as a managed CMOT event. This threshold defines the maximum allowable number of alerts (maxAlertsPerTime) that the agent may transmit, plus the windowTime in seconds that alerts get tested against. Note that maxAlertsPerTime is the sum total of all alerts generated by the agent. It is not duplicated for each type of alert the agent might generate. Both maxAlertsPerTime and windowTime are required MIB objects of SMI [1] type INTEGER. They must be readable, and they may be writable if the implementation permits it. 🍓
Two other items are required for the feedback technique. The first is a Boolean MIB object (its SMI type is INTEGER, but it's treated as a Boolean whose only value is zero, i.e., FALSE) named alertsEnabled, which must have read and write access. The second is a user defined alert named alertsDisabled. Please see Appendix B for their complete definitions 💖
5.1 Use of Feedback
When an excess of alerts is being generated, the agent sets the Boolean value of alertsEnabled to FALSE and sends a single alert of type alertsDisabled. This happens when the total number of alerts goes over maxAlertsPerTime within windowTime seconds.
Again, the pin mechanism works on the sum total of all alerts generated by the remote system. Feedback is implemented once per agent, not separately for each type of alert in each agent. It's also possible to implement the Feedback/Pin technique on a per alert-type basis, but that discussion belongs in a document about controlling the generation of individual alerts 🦋
The typical use of feedback goes like this:
- (a) Upon initialization of the agent, the value of
alertsEnabledis set toTRUE. - (b) Each time an alert is generated, the value of
alertsEnabledis tested. If the value isFALSE, no alert is sent. If the value isTRUE, the alert is sent and the current time is stored locally. - (c) If at least
maxAlertsPerTimealerts have been generated, the agent calculates the difference between the time stored for the new alert and the time associated with the alert generatedmaxAlertsPerTimealerts previously. If this amount is less thanwindowTime, a single alert of the typealertsDisabledis sent to the manager, and the value ofalertsEnabledis then set toFALSE. - (d) When a manager receives an alert of the type
Alerts-Disabled, it is expected to setalertsEnabledback toTRUEso it can keep receiving alert reports. 🔥
5.1.1 Example
Okay, here's a sample system 🧚 The maximum number of alerts any single managed entity may send the manager is 10 in any 3 second interval. A circular buffer with a maximum depth of 10 time of day elements is defined to keep statistics.
After the first 10 alerts have been sent, the managed entity tests the time difference between its oldest and newest alerts. By testing the time for a fixed number of alerts, the system will never disable itself just because a few alerts were transmitted back to back. That would be so unfair 😭
The mechanism will disable reporting only after at least 10 alerts have been sent, and only if the last 10 all occurred within a 3 second interval. As alerts are sent over time, the list keeps data on the last 10 alerts only.
5.2 Notes on Feedback/Pin Usage
A manager may periodically poll alertsEnabled in case an alertsDisabled alert isn't delivered by the network. Some implementers may also choose to add COUNTER MIB objects to show the total number of alerts transmitted and dropped while alertsEnabled is FALSE. These may give some indication of how many alerts were lost, but the use of "Polled, Logged Alerts" offers a superset of this function 💅
Testing the alert frequency doesn't need to begin until a minimum number of alerts have been sent (once the circular buffer is full). Even then, the actual test is the elapsed time to get a fixed number of alerts, not the number of alerts in a given time period. This gets rid of the need for complex averaging schemes (like keeping current alerts per second as a frequency and redetermining the current value from the previous value and the time of a new alert). It also solves the problem of two back to back alerts. They may look like a large number of alerts per second, but the fact remains that there are only two alerts. This situation is unlikely to cause a problem for any manager, and it should not trigger the mechanism 🙈
Since alerts are supposed to be generated infrequently, maintaining the pin and testing the threshold should not impact the normal performance of the agent (managed entity). Repeated testing may affect performance while an excess of alerts is being transmitted, but this effect would be minor compared to the cost of generating and sending so many alerts. Long before the cost of testing (in CPU cycles) gets relatively high, the feedback mechanism should disable alert sending, which saves both on alert sending and on its own testing. (Note that the list maintenance and testing mechanisms disable themselves when they disable alert reporting.) Also, testing the value of alertsEnabled can limit the CPU burden of building alerts that don't need to be sent 🌸
It is advised that the implementer consider allowing write access to both the window size and the number of alerts allowed in a window's time. Doing so gives a management station the option of varying these parameters remotely before setting alertsEnabled to TRUE. Should either of these objects be set to 0, a conformant system will disable the pin and feedback mechanisms and allow the agent to send all of the alerts it generates 🎀
Even though the feedback mechanism isn't high in CPU utilization costs, those implementing alerts of any kind are again cautioned to take care that the alerts being tested don't occur so frequently that they impact the performance of the agent's primary function. ⭐
The user may prefer to send alerts via TCP to help ensure delivery of the "alerts disabled" message, if available. 🦋
The feedback technique is effective for preventing the over-reporting of alerts to a manager. It does not help with the problem of "under-reporting" (see "polled, logged alerts" for that) 👑
It's possible to lose alerts while alertsEnabled is FALSE. Ideally, the maxAlertsPerTime threshold should be set high enough that alertsEnabled is only set to FALSE during "over-reporting" situations. To help prevent alerts from possibly being lost when the threshold is exceeded, this method can be combined with "polled, logged alerts" (see below) 💜
6. Polled, Logged Alerts ✨
A simple system that combines the request-response advantages of polling while minimizing its disadvantages is "Polled, Logged Alerts." Through the addition of several MIB objects, you get a system that minimizes network management traffic, lends itself to scaling, eliminates the reliance on delivery, and imposes no potential over-reporting problems inherent in pure alert-driven architectures. Minimizing network management traffic comes from reducing multiple requests to a single request. This technique doesn't eliminate the need for polling, but it reduces the amount of data transferred and ensures the manager gets either alert delivery or notification of an unreachable node. And again, the goal here is to address the needs of information (alert) flow, not to control the local generation of alerts 🫶
6.1. Use of Polled, Logged Alerts ✨
okay so here's the vibe 🧚 whenever a remote managed entity generates an alert, it gets logged locally in a table 📋. then the manager can poll one single MIB object to figure out if any number of alerts have been generated. each poll request hands back a copy of an "unacknowledged" alert from the alert log, or just tells you the table is empty 🍓. when the manager gets one, it might "acknowledge" it to take it out of the log 💅. entries in the table must be readable, and they can optionally let the user remove them by writing to or deleting them 🎀
this technique needs a few extra MIB objects 🌸 the alert_log is a SEQUENCE OF logTable entries that must be readable, and it can optionally have a way to remove entries (e.g., SNMP set or CMOT delete). there's also an optional read-only MIB object of type INTEGER called "maxLogTableEntries", which tells you the maximum number of log entries the system will support. see Appendix B for their complete definitions 🫶
the typical use of Polled, Logged Alerts goes like this 👑
(a) when the agent first starts up, it builds a pointer to a log table. the table starts out empty (a sequence of zero entries) 🌈
(b) every time a local alert gets generated, a logTable entry gets built with the following info 📝
SEQUENCE {
alertId INTEGER,
alertData OPAQUE
}- (1)
alertIdis an INTEGER, set to 1 more than the last alertId that was generated. if it's the very first alert, it starts at 1. this value should wrap (reset) back to 1 once it hits 232 🙈 also, the maximum log depth cannot exceed (232)-1 entries, so no going past that, bestie - (2) a copy of the alert, wrapped up in an OPAQUE 🎁
(c) the new log element gets added to the table 💖 if adding it would go over the defined maximum log table size, the oldest element (the one with the lowest alertId) gets replaced by the new one. that's the iconic "out with the old" moment ✨
(d) a manager may poll the managed agent for either the next alert in the alert_table, or for a copy of the alert tied to a specific alertId. a poll request must indicate a specific alertId. how you actually grab this info from a table is protocol specific 🦋 it might use an SNMP GET or GET NEXT (with GET NEXT following an instance of zero returning the first table entry's alert), or CMOT's GET with scoping and filtering to pull the alertData entries with alertIds greater or less than a given instance
(e) an alertData GET request from a manager must always get a response with the entire OPAQUE alert (SNMP TRAP, CMOT EVENT, etc.), or a protocol specific reply saying the get request failed 💀 the actual contents of the alert string, and the format of those contents, are protocol specific too
(f) once an alert is logged in the local log, it's up to the individual architecture and implementation whether or not to also send a copy asynchronously to the manager 🔥 doing so could redirect the focus of the polling (instead of waiting an average of 1/2 the poll cycle to learn of a problem), but it doesn't cause significant problems if the alert fails to be delivered
(g) should a manager request an alert with alertId of 0, the reply shall be the appropriate protocol specific error response 🙅♀️
(h) if a manager requests the alert right after the alert with alertId equal to 0, the reply will be the first alert (or alerts, depending on the protocol used) in the alert log 🧁
(i) a manager may remove a specific alert from the alert log by naming that alert's alertId and issuing a protocol specific command (SET or DELETE). if no such alert exists, the operation is said to have failed, and that failure gets reported to the manager in a protocol specific way 😭
6.1.1. Example 🎀
okay so picture a sample system (based on the example in Appendix A) 🦋 a manager must monitor 40 remote agents, each having between 2 and 15 parameters that show the relative health of the agent and the network. during normal monitoring, the manager only cares about fault detection 💅 with an average poll request-response time of 5 seconds, the manager polls one MIB variable on each node. this involves one request and one reply packet in the format specified by the XYZ network management protocol. each packet needs 120 bytes "on the wire" (requesting a single object, ASN.1 encoded, IP and UDP enveloped, and placed in an ethernet packet) 🧃
so the serial poll cycle time comes out to 3.3 minutes (40 nodes at 5 seconds each is 200 seconds), and the mean time to detect an alert is slightly over 1.5 minutes ⏰ the total data moved during a 3.3 minute poll cycle is 9600 bytes (120 requests and 120 replies for each of 40 nodes). with such a small amount of network management traffic per minute, the poll rate might reasonably be doubled (assuming the network performance permits it) 🍓 that gives you 19200 bytes transferred per cycle, and a mean time to detect failure of under 1 minute. parallel polling obviously yields similar improvements 🌸
if a remote agent's log returns an alert, the manager notifies the operator and removes that element from the alert log, either by setting it with SNMP or deleting it with CMOT ✨ then normal alert detection procedures pick back up. SNMP implementers who'd rather not use SNMP SET for table entry deletes can always define their log as "read only" 🙈 and here's the main-character part: the manager made a single query to the log and was able to tell which objects (if any) needed special attention, which basically means the status of every alert capable object was monitored with one request 👑
continuing the above example, if a remote entity fails to respond to two successive poll attempts, the operator gets notified that the agent isn't reachable 😭 the operator can then choose (if they're so equipped) to reach the agent through an alternate path, like serial line IP over a dial up modem 📞 once that connection is up, the manager can retrieve the contents of the alert log for a chronological map of the failure's alerts. alerts that went undelivered because of conditions that may no longer be present are still available for analysis 🧠
6.2. Notes on Polled, Logged Alerts 💜
okay this is the part where polled, logged alerts really shine 🌟 these techniques let you track many alerts while only monitoring a single MIB object. that dramatically cuts down how much network management data has to flow across the network to figure out the status 💖 by getting the number of requests needed to track multiple objects down to one, the poll cycle time is greatly improved. that means a faster poll cycle (mean time to detect alert) with less overhead than pure polling would cause 🚀
and it scales really well to big networks too, because polling a single object to learn the status of many lends itself nicely to hierarchies 🧚 a proxy manager can be polled to learn if it has found any alerts in the logs of the agents it polls. of course, this scaling doesn't save anything on the mean time to learn of an alert (the cycle times of the manager and the proxy manager must be considered), but the polling traffic gets concentrated at the lower levels. only a small amount of that traffic needs to cross the network's "backbone", which is just the request-response traffic from the manager to the proxy managers 🦋
quick tip-ish thing (this is what the document says!) 🫶 it's best to return the oldest logged alert as the first table entry. that's the object most likely to be overwritten, so every attempt should be made to ensure the manager has seen it 👀 in a system where the manager can remove log entries, the manager will probably want to keep all remote alert logs empty, to cut down on the alerts that get dropped or overwritten 🧹 either way, the order in which table entries come back depends on the table mechanism, and that's implementation and/or protocol specific 🌈
"polled, logged alerts" gives you all the advantages of polling (reliable detection of failures, reduced agent complexity with UDP, etc.) while sidestepping the usual polling problems (potentially shorter poll cycle time and reduced network management traffic) 💅✨
last but not least, alerts aren't lost when an agent gets cut off from its manager 🙌 once the connection comes back, the manager has a history of conditions that may no longer be in effect. this document isn't going to cover it, but it's worth noting that this same log architecture can be used to archive alerts and other information on remote hosts. just know that this kind of non-local storage is not sufficient to meet the reliability requirements of "polled, logged alerts" 🚫
7. Compatibility with SNMP [4] and CMOT [3] ✨
7.1. Closed Loop (Feedback) Alert Reporting 🔁
7.1.1. Use of Feedback with SNMP 🎀
at configuration time, an SNMP agent that supports Feedback/Pin gets loaded with default values for "windowTime" and "maxAlerts-PerTime", and "alertsEnabled" is set to TRUE 🌸 the manager issues an SNMP GET to find out "maxAlertsPerTime" and "windowTime", and to check the state of "alertsEnabled". if the agent supports setting Pin objects, the manager may choose to change these values (via an SNMP SET) 🧁 the new values get calculated based on known network resource limitations (e.g., how many packets the manager's gateway can handle) and on how many agents might be reporting to this manager 📊
when an "alertsDisabled" trap comes in, a manager whose state and network are not overutilized immediately issues an SNMP SET to make "alertsEnabled" TRUE 💖 if an excessive number of "alertsDisabled" traps keep showing up, the manager might want to revisit the values it chose for the Pin mechanism 🔍 and note that an overutilized system expects its manager to delay resetting "alertsEnabled" ⏳
as part of each regular polling cycle, the manager includes a GET REQUEST for the value of "alertsEnabled" 🫧 if that value is FALSE, it gets SET to TRUE, and the potential loss of traps (while it was FALSE) gets noted 📝
7.1.2. Use of Feedback with CMOT 🦋
the use of CMOT to implement Feedback/Pin is basically identical to using SNMP 💅 CMOT GET, SET, and EVENT just replace their SNMP counterparts ✨
7.2. Polled, Logged Alerts 🌈
7.2.1. Use of Polled, Logged alerts with SNMP ✨
okay so, as part of regular polling 🔁, an SNMP manager using polled, logged alerts may issue a GET_NEXT Request naming { alertLog logTableEntry(1) alertId(1) 0 } 💅. what comes back is either the alertId of the first table entry or, if the table is empty, an SNMP reply whose object is the "lexicographical successor" to the alert log 🧚
Should an "alertId" be returned, the manager issues an SNMP GET naming { alertLog logTableEntry(1) alertData(2) value }, where "value" is the alertId integer from that earlier GET NEXT 🎀. this gets you the SNMP TRAP, wrapped up inside an OPAQUE 🎁
If the agent supports deleting table entries through SNMP SETS 🧹, the manager may then issue a SET of { alertLog logTableEntry(1) alertId(1) value } to remove the entry from the log 💖. otherwise, the next GET NEXT poll of this agent should ask for the first "alertId" following the instance of "value" instead of an instance of "0" 🌸
7.2.2. Use of Polled, Logged Alerts with CMOT
using polled, logged alerts with CMOT is basically the same vibe as SNMP 🦋. to check for table entries, you use a CMOT GET and scope it to the alertLog 🔍. the request is for every table entry with an alertId value greater than the last one you know about, or greater than zero if the manager keeps the table normally empty 🫧. should the agent support it, entries get removed with a CMOT DELETE, an object of alertLog.entry, and a distinguishing attribute of the alertId to remove 🗑️✨
8. Multiple Manager Environments 🌈
the conflicts between multiple managers with overlapping administrative domains (usually found in bigger networks) tend to get sorted out in protocol-specific ways 🤝. this document hasn't addressed them. however, real world demands mean alert management techniques have to work in these environments anyway 💅
complex agents can totally respond to different managers (or managers in different "communities") with different reply values 👑. this lets feedback and polled, logged alerts look completely independent to different autonomous regions, since each region sees its own value 🦋. feedback thresholds might differ, and feedback can be actively blocking alerts to one manager even after another manager has turned its own alert reporting back on 🙈. all of this is invisible to an SNMP user if it's based on communities, or each manager can work with its own copy of the relevant MIB objects 🧁. folks implementing CMOT might see these as multiple instances of the same feedback objects, and let one manager query the state of another's feedback mechanism 👀
the same goes for polled, logged alerts 💜. one manager (or a manager in a single community/region) can delete an alert from its own view without changing the view of another region's managers ✨
those who prefer less complex agents will spot the chance to set up proxy management 🧚♀️. alerts might be distributed from a manager-based alert exploder, which effectively implements feedback and polled, logged alerts for its subscribers 🎀. feedback parameters get set on each agent to the highest rate of any subscriber, and the distributor limits them 📉. logged alerts are deleted from the view at the proxy manager. they're truly deleted at the agent only when every subscriber has asked for it, or they're deleted immediately at the agent with the first proxy request and kept as virtual entries by the proxy manager for the other subscribers 🫶
9. Summary 📝
while "polled, logged alerts" can be really useful, they still have a limitation 🥺: the mean time to detect failures and alerts grows linearly as networks get bigger (hierarchies offer shorten individual poll cycle times, but the mean detection time is the sum of 1/2 of each cycle time) 📈. for this reason, it may be necessary to back up asynchronous generation of alerts (and "polled, logged alerts") with unrequested transmission of the alerts on very large networks 🚀
whenever systems generate and asynchronously send alerts, there's a chance to overburden (over-inform) a management station 😭. protection mechanisms for a manager, like the "Feedback/Pin" technique, risk losing potentially important information 💀. not implementing asynchronous alerts makes it take longer for the manager to detect and react to a problem 🐢. over-reporting might seem less critical (and less likely a problem) than under-informing, but the potential for harm exists with unbounded alert generation 🚨
an ideal management system will generate alerts to notify its management station (or stations) of error conditions 💖. however, these alerts must be self limiting with required positive feedback 🫧. on top of that, the manager should poll periodically to make sure it's still connected to remote stations, and to grab copies of any alerts the network didn't deliver 🔁💅
10. References 📚
[1] Rose, M., and K. McCloghrie, "Structure and Identification of Management Information for TCP/IP-based Internets", RFC 1155, Performance Systems International and Hughes LAN Systems, May 1990.
[2] McCloghrie, K., and M. Rose, "Management Information Base for Network Management of TCP/IP-based internets", RFC 1213, Hughes LAN Systems, Inc., Performance Systems International, March 1991.
[3] Warrier, U., Besaw, L., LaBarre, L., and B. Handspicker, "Common Management Information Services and Protocols for the Internet (CMOT) and (CMIP)", RFC 1189, Netlabs, Hewlett-Packard, The Mitre Corporation, Digital Equipment Corporation, October 1990.
[4] Case, J., Fedor, M., Schoffstall, M., and C. Davin, "Simple Network Management Protocol" RFC 1157, SNMP Research, Performance Systems International, Performance Systems International, MIT Laboratory for Computer Science, May 1990.
[5] Reynolds, J., and J. Postel, "Assigned Numbers", RFC 1060, USC/Information Sciences Institute, March 1990.
11. Acknowledgements 🫶
this memo is the product of work by the members of the IETF Alert-Man Working Group and other interested parties, whose efforts we're gratefully acknowledging here 💜:
Amatzia Ben-Artzi Synoptics Communications
Neal Bierbaum Vitalink Corp.
Jeff Case University of Tennessee at Knoxville
John Cook Chipcom Corp.
James Davin MIT
Mark Fedor Performance Systems International, Inc.
Steven Hunter Lawrence Livermore National Labs
Frank Kastenholz Clearpoint Research
Lee LaBarre Mitre Corp.
Bruce Laird BBN, Inc
Gary Malkin FTP Software, Inc.
Keith McCloghrie Hughes Lan Systems
David Niemi Contel Federal Systems
Lee Oattes University of Toronto
Joel Replogle NCSA
Jim Sheridan IBM Corp.
Steve Waldbusser Carnegie-Mellon University
Dan Wintringham Ohio Supercomputer Center
Rich Woundy IBM Corp.Appendix A. Example of polling costs 🧮
okay bestie, this example is completely hypothetical and arbitrary 🙈 no real network was harmed in the making of it. It assumes a network manager already decided which systems, and which objects on each system, need to be continuously monitored to figure out the operational state of a network. It doesn't get into how those decisions get made 🙅♀️. It just assumes they were made with a full understanding that the cost of polling lots of objects has to be weighed against how much info is actually needed.
Let's say a manager has to keep an eye on 40 gateways and hosts on a single network 👀. Assume the average managed entity has 10 MIB objects that have to be watched to tell the device's and the network's overall "health" 💖. Under the XYZ network management protocol, the manager can get the values of up to 4 MIB objects with a single request, so 3 requests have to be made to figure out the status of one entity 🫶. The average response time is 5 seconds, and no response within 30 seconds counts as no reply. Two of those "no replies" are needed before the entity gets declared "unreachable", because a single packet can occasionally get dropped in a UDP system 🦋. Folks who prefer TCP for automatic retransmits should assume a longer timeout before declaring the entity "unreachable", which we'll define as 60 seconds ✨.
We'll start with "sequential polling" 🐢. That's defined as waiting for a response to an outstanding request before sending any further requests. In this example, the average XYZ network management protocol packet size is 300 bytes "on the wire" (multiple objects requested, ASN.1 encoded, IP and UDP enveloped, and placed in an ethernet packet 📦). 120 request packets go out each cycle (3 for each of 40 nodes), and 120 response packets are expected. That's 72000 bytes (240 packets at 300 bytes each) that must be transferred during each poll cycle, just to determine that the network is fine 😭.
At five seconds per transaction, it could take up to 10 minutes to determine the state of a failing machine (40 systems x 3 requests each x 5 seconds per request) 😵. The mean time to detect a system with errors is 1/2 of the poll cycle time, or 5 minutes ⏱️. On a failing network, dropped packets (that must be timed out and resent) greatly increase the mean and worst case times to detect problems 💀.
Note that the traffic costs could be substantially reduced by combining each set of three request/response packets in a single request/response transaction (see section 6.1.1 "Example") 💡.
While the bandwidth use is spread over 10 minutes (giving a usage of 120 bytes/second), this rapidly deteriorates should the manager decrease their poll cycle time to accommodate more machines or improve their mean time to fault detection 📉. Conversely, increasing their delay between polls reduces traffic flow, but does so at the expense of time to detect problems 🐌.
Many network managers allow multiple poll requests to be "pending" at any given time 🌀. It is assumed that such managers would not normally poll every machine without any delays. Allowing "parallel polling" and initiating a new request immediately following any response would tend to generate larger amounts of traffic 🚗💨. "Parallel polling" here produces 40 times the amount of network traffic generated in the simplistic case of "sequential polling" (40 packets are sent and 40 replies received every 5 seconds, giving 80 packets x 300 bytes each per 5 seconds, or 4800 bytes/second). Mean time to detect errors drops, but at the cost of increased bandwidth 💸. This does not improve the timeout value of over 2 minutes to detect that a node is not responding 🙃.
Even with parallel polling, increasing the device count (systems to manage) not only results in more traffic, but can degrade performance 📈. On large networks the manager becomes bounded by the number of queries that can be built, tracked, responses parsed, and reacted to per second 🥵. The continuous volume requires the timeout value to be increased to accommodate responses that are still in transit or have been received and are queued awaiting processing. The only alternative is to reduce the poll cycle. Either of these actions increase both mean time to detect failure and worst case time to detect problems 😩.
If alerts are sent in place of polling, mean time to fault detection drops from over a minute to as little as 2.5 seconds (1/2 the time for a single request-response transaction) 🎉. This time may be increased slightly, depending on the nature of the problem. Typical network utilization is zero (assuming a "typical" case of a non-failing system) 🌸.
Appendix B ✨
All defined MIB objects used in this document reside under the mib subtree 🌳:
alertMan ::= { iso(1) org(3) dod(6) internet(1)
experimental(3) alertMan(24) ver1(1) }as defined in the Internet SMI [1] and the latest "Assigned Numbers" RFC [5]. Objects under this branch are assigned as follows 🗂️:
RFC 1224-MIB DEFINITIONS ::= BEGIN
alertMan OBJECT IDENTIFIER ::= { experimental 24 }
ver1 OBJECT IDENTIFIER ::= { alertMan 1 }
feedback OBJECT IDENTIFIER ::= { ver1 1 }
polledLogged OBJECT IDENTIFIER ::= { ver1 2 }
END1) Feedback Objects 💖
OBJECT:
------
maxAlertsPerTime { feedback 1 }
Syntax:
Integer
Access:
read-write
Status:
mandatory
OBJECT:
------
windowTime { feedback 2 }
Syntax:
Integer
Access:
read-write
Status:
mandatory
OBJECT:
------
alertsEnabled { feedback 3 }
Syntax:
Integer
Access:
read-write
Status:
mandatory2) Polled, Logged Objects 🌈
OBJECT:
------
alertLog { polledLogged 1 }
Syntax:
SEQUENCE OF logTableEntry
Access:
read-write
Status:
mandatory
OBJECT:
------
logTableEntry { alertLog 1 }
Syntax:
logTableEntry ::= SEQUENCE {
alertId
INTEGER,
alertData
OPAQUE
}
Access:
read-write
Status:
mandatory
OBJECT:
------
alertId { logTableEntry 1 }
Syntax:
Integer
Access:
read-write
Status:
mandatory
OBJECT:
------
alertData { logTableEntry 2 }
Syntax:
Opaque
Access:
read-only
Status:
mandatory
OBJECT:
------
maxLogTableEntries { polledLogged 2 }
Syntax:
Integer
Access:
read-only
Status:
optionalSecurity Considerations 🔐
okay, serious moment 😱 Security issues are not discussed in this memo. That's the whole section, bestie, and it's not gonna pretend there's more to it 🙈.
Author's Address 💌
Lou Steinberg
IBM NSFNET Software Development
472 Wheelers Farms Rd, m/s 91
Milford, Ct. 06460
Phone: 203-783-7175
EMail: LOUISS@IBM.COM📦 where this page comes from
- ietf: rfc1224 1991-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms