RFC 1297

NOC Internal Integrated Trouble Ticket System 🎀

Okay bestie, this one is from RFC 1297 🦋 It's titled NOC Internal Integrated Trouble Ticket System: Functional Specification Wishlist ("NOC TT REQUIREMENTS"), and it was written by D. Johnson at Merit Network, Inc. in January 1992 ✨ It's an Informational doc, which means it just provides information for the Internet community and does not specify an Internet standard 💅 Nothing is listed as obsoleted or updated in the title block, so it's a standalone wishlist, no cap 🌸

---

Status of the Memo 📜

This memo provides information for the Internet community. It does not specify an Internet standard. Distribution of this memo is unlimited. 🫶

Abstract ✨

Okay so, professional-quality handling of network problems needs some kind of problem tracking system, which this doc calls a "trouble ticket" system 🎫 A basic one acts like a hospital chart, coordinating the work of multiple people who might need to work on the same problem 🏥

Once the basic system is in place, though, there are so many extensions that can make Network Operations way more efficient 🔥 Info stored in the tickets can be used to make statistical reports 📊 Operator efficiency and accuracy can go up by automating ticket entry with info from the network Alert system 🤖 The Alert system can also be used to monitor how ticket progress is going 👀 And trouble tickets can be used to share network health info between NOCs, with telcom vendors, and with other internal sales and engineering audiences 🌈

This document explores competing uses, architectures, and desirable features of integrated internal trouble ticket systems for Network and other Operations Centers 💜

Introduction 🧚

This RFC walks through the general functions a Trouble Ticket system could have if it were designed for Network Operations Centers 🖥️ It's being shared with the Internet community to get people talking about new production-oriented, operator-level application tools for network operations 💬 The hope is that this leads to more ideas for improving NOC performance, and more available tools that actually build those ideas in 🙌

PURPOSES OF A NOC TROUBLE TICKET SYSTEM 👑

A good Network Operations Trouble Ticket System should serve many purposes, and here they are, in order of main character energy:

1) SHORT-TERM MEMORY AND COMMUNICATION ("Hospital Chart") 🏥

The big one. The primary purpose of the trouble ticket system is to act as short-term memory about specific problems for the NOC as a whole 🧠 In a multi-operator or multi-shift NOC, calls and problem updates come in without any regard for who worked on a problem last. Problems stretch across shifts, and several different operators can touch the same problem on the same shift 😵‍💫

The ticket (like a hospital chart) keeps a complete history of the problem, so any operator can get up to speed and take the next appropriate step. They don't have to bug other operators who are busy with something else, have gone home, or are on vacation 🏖️ In single-room NOCs, an operator might ask out loud whether someone else knows about or is working on a problem, but the system should allow for more formal communication too 🗣️

2) SCHEDULING and WORK ASSIGNMENT 📅

NOCs usually juggle many simultaneous problems with different priorities, which is a lot 😩 An on-line trouble ticket system can show real-time lists of open problems (or even lists that are constantly displayed and updated), sorted by priority ⭐ That lets operators sort their work at the start of a shift and pick their next task mid-shift 💅

It also lets supervisors and operators keep track of the current NOC workload, and call in and assign extra staff when it's appropriate 🫡

It may be useful to let the current priorities of tickets change based on the time of day, or in response to timer alerts ⏰

3) REFERRALS AND DISPATCHING 📨

If the trouble ticket system is integrated well enough with a mail system, or if Network Engineers use it alongside Network Operators, then some problems can be dispatched just by putting the right Engineer or Operator name in an "assigned to" field of the ticket 🎯 Easy.

4) ALARM CLOCK ⏰

Okay this one is so relatable 😭 Most of the time a trouble ticket is open, it's just waiting for something to happen. There should almost always be a timer attached to every wait ⏳ If a ticket is referred to a phone company, there's an escalation time before which the phone company is supposed to call back with an update on the problem 📞 For tickets referred to remote site personnel, there may be other, more arbitrary timeouts, like "Monday morning" 🗓️ Tickets referred to local engineers or programmers should get timeouts too ("Check in a couple of days if you don't hear back from me") 🙈

A good trouble ticket system will let a timeout be set for each ticket. When the timer goes off, it generates an alert for that ticket at the right time 🔔 Preferably, the system should let text be attached to that timer, a shorthand message about what the alert involves, like Remind Site: TT xxx (the full story can always be found by checking the trouble ticket) 📝 These alerts should feed into the NOC's standard alert system 🚨

The Alarm Clock can also assist with (or enforce!) administrative escalation 💀 An escalation timer could be set automatically based on the type of network, the severity of the problem, and the time the outage occurred ⚡

5) OVERSIGHT BY ENGINEERS AND CUSTOMER/SITE REPRESENTATIVES 🕵️‍♀️

NOCs often run more than one network, or at least have people (engineers, customer representatives, etc.) who are responsible for subsets of the total network 🌐 For these individual representatives, summaries of trouble tickets can be filtered by network or by node, then delivered electronically to the right engineers or site representatives 💌

Each of these reports includes a summary of the previous day's trouble tickets for those sites, a listing of older trouble tickets that are still open, and a section listing recurrent problems 📋 That way the site reps can stay aware of current outages and trends for their particular sites 👀

The trouble ticket system also gives network access to the details of individual tickets, so people who get the general reports can dig into any of their problems by referencing the ticket number 🔍

6) STATISTICAL ANALYSIS 📊

The fixed-form fields of trouble tickets let tickets be categorized, which is super useful for analyzing equipment and NOC performance 🧮 This includes Mean Time Between Failure and Mean Time to Repair reports for specific equipment 🔧 The fields can also help generate statistical quality control reports, which let deteriorating equipment get detected and serviced before it fails completely 🩺

Ticket breakdowns by network can help a NOC apportion its costs appropriately, and they help with developing staffing and funding models 💰 A good trouble ticket system should present this statistical information in a format suitable for spreadsheets and graphics programs 📈✨

7) FILTERING CURRENT ALERTS 🙅‍♀️

It would be possible to use network status information from the trouble ticket system to filter the alerts shown on the alert system 🧹 For instance, if node XXX is known to be down because a trouble ticket is currently open on it, the alert display for that node could be acknowledged automatically 🤫 Trouble tickets could potentially contain a lot more useful information for expert system analysis of current network alert information 🧠💜

8) ACCOUNTABILITY ("CYA"), FACILITATING CUSTOMER FOLLOW-THROUGH, AND NOC IMAGE 💅

Keeping user-complaint tickets enables the kind of follow-through with end users that makes happy clients (and a good NOC image) in normal trouble-fixing situations 🌸 But NOCs also deal with crises by nature 😮‍💨 They occasionally end up with major outages, and with angry users or administrators 🔥 The trouble ticket system documents the NOC's (and the rest of the organization's) efforts to solve problems, so there's a record if complaints come in 📁🫶

FIXED FIELDS, FREE-FORM FIELDS, and TT CONFIGURATION ✨

okay so info in trouble tickets can live in either fixed fields or freeform fields 💅 fixed fields are a total win for searching because the data is structured. a row of fixed fields also acts like a template, either encouraging or requiring operators to fill in certain standard stuff 📋 they're great for data verification too, like making sure an entered name is in an attached contacts database or that a phone number is ten numeric characters 🔢 and they're the move for anything the system fills in automatically, like the operator's login id, the name of the node that got clicked on when the ticket was opened from an alert tool, or names and phone numbers that get auto-filled based on other entries (e.g., filling in a contact name and phone based on a machine name) 🧚

ngl though, fixed fields work best when the problem-debugging environment is uniform, well-understood, and stable 🥹 tickets work best when their fields are really well tailored to the specific problem at hand 🎯 it is SO easy to set up a bunch of fields (or even required fields!) that don't matter for a given problem, and then the operators get slowed down and confused 🙈 adding structure and validity checking to a field makes the data more consistent and reliable, but it also tends to force operators into longer procedures, like menus, just to get the data accepted by the system. it also means more maintenance on those verification systems (adding new entries as they become new legal options), and in some ways it reduces the accuracy of the system because operators are forced to pick "canned" or authorized responses that may not always represent the situation accurately 😭 where statistical operational reports are a primary purpose of the trouble ticket system, several fixed fields may be appropriate. if the main goal is to keep notes for individual problems and help operators talk to each other, then fixed fields may tend to be a hindrance 🚫 one reasonable guideline would be that fixed fields are used ONLY where they are automatically filled in by the larger system, or where the information in that field is explicitly used in a report or standard search procedure.

because the ticket structure and the problem to be solved are so closely related, it is very very useful to be able to define different ticket types for different classes of problems 👑 this matters even more for the many NOCs whose staff are also responsible for other types of operations: mainframe operations, workstation administration, help desk functions, or any of the other real-time response functions. Network operations to justify the expense of an operations center. this kind of operation makes economic sense, and it's becoming more and more common 📈 in these situations it is vital that the same tools used for network operations are also available for the other operations. that means the trouble ticket configurations need to be modifiable by local staff. commercial RDBMS forms builder and report generator packages and "fourth-generation languages" offer a good start at this, although it can be kinda difficult to integrate full trouble ticket functionality through these systems 🌈

---

TROUBLE TICKET STRUCTURE 🧚

1) HEADERS 💖

inevitably, a trouble ticket starts with a bunch of fixed fields. these generally include:

• Time and Date of problem start.
• Initials or signon of the operator opening the ticket.
• Severity of the problem (possibly separating the "customer severity" and the "NOC priority", since these could be different).
• A one-line description of the problem for use in reports.

there can be lots of other fixed fields for specific purposes 🎀 there may also be different kinds of tickets for different problems, where the ticket format differs mainly in its fixed fields. these include:

• Who reported the problem? (Name, organization, phone, email address)
• Machine(s) involved.
• Network involved (for multi-network NOCs).
• User's machine address.
• Destination machine address.
• Next Action.
• Time and date for alarm on this ticket.
• Who should the ticket be dispatched to?
• Ticket "owner" (one person designated to be responsible overall).

2) INCIDENT UPDATES 🫶

the main body of a trouble ticket is usually a series of freeform text fields 📝 ideally, each of these fields is automatically marked with the time and date of the update, and with the signon of the operator making the update. but updates often get recorded sometime after the problem is fixed, so it's useful to let operators override the current time stamp with the time the update was actually made. (in some implementations, both times will be kept internally) ⏰

the first incident update is usually a description of the problem. since the exact nature of the problem usually isn't known when the ticket is first opened, this description can be complex and imprecise 🌀 for problems reported by electronic mail, it's super handy to be able to paste the original message into the ticket, especially if it has cryptic or extensive info (like a user's traceroute output). at least one arbitrarily-long freeform field seems necessary to hold this kind of output, although it's even better to allow arbitrarily long messages at any stage (e.g., so future complex messages can also be archived in the ticket) 🗂️

subsequent update fields can be as simple as "Called site; no answer" 🙃 some systems let these kinds of updates be coded in fixed fields, but most use freeform text.

there should always be an indication of what the next action for this ticket ought to be 🎯 again, this may be implemented as a special fixed field, or by convention of using the last line of text.

advanced systems may also need a facility to allocate the amount of time a ticket is open between multiple sources. a serious NOC will want to use its trouble ticket system to statistically track its performance on responding to problems (e.g., Mean Time Between Failure and Mean Time To Repair reports) 📊 frequently, though, repairs get stopped at the customer's request. ("It's not that important a machine and I don't feel like coming in--can you defer it until Monday Morning?") 💀 in these cases the ticket needs to stay open, but there needs to be a notation that it's now in "customer time" rather than "NOC time". the durations of "customer time" need to be excluded from MTBF and MTTR reports. complicated repairs could move back and forth between "NOC time" and "customer time" repeatedly 🔁 this probably implies that each Incident Update may have a time and date of status change, and that these status changes can be read and aggregated by reporting programs.

3) RESOLUTION DATA 🍓

once a problem is resolved, it's useful to summarize it for future statistical analysis ✨ the following fields have been found to be useful:

• Time and Date of resolution (for outage duration).
• Durations (can be calculated from time of resolution and incident report "customer/NOC time" stamps).
• Resolution (one line of description of what happened, for reports).
• Key component affected (for MTBF and similar reports).
• Checked By -- a field for supervisors to sign off on ticket review.
• Escalated to -- for reports on how many problems require non-NOC help.
• Temp - a database field that can be used to store temporary "check marks" while making statistical investigations.

---

USER, TROUBLE, and ENGINEERING Ticket System(s) 👑

the primary level of a Network Operations trouble ticket is the "problem" or "trouble": a single malfunctioning piece of hardware or software that breaks at some time, has various efforts to fix it, and eventually gets fixed at some given time 🔧

the primary level of a Network Information Center ticket, however, might well be the "user complaint" 📞 a single network failure might well produce a large number of individual user phone calls, and hence "user complaint" tickets. a NIC may want to use tickets to track each one of these calls, e.g., to make sure each user is informed and satisfied about the eventual resolution of the single hardware problem 🥰

in addition, NOCs (or Engineering Staffs) may want to track systematic problems 🧠 the staff may know, for instance, that a particular router is old and fragile, or that a particular section of their network doesn't have enough redundancy. it may be useful to open an "Engineering Ticket" on these known problems, providing a place to record history and notes about the problem, for use in further engineering or funding discussions 💸

even further "Meta" tickets could be described, having to do with issues such as whether the current trouble ticket fields, reports, and operation procedures were sufficient to handle current problems 🪞

it would be very convenient to build all of these systems on the same platform, and to let each type of ticket easily reference other types 🔗 multiple "user complaint" tickets, then, might explicitly point to a single "trouble" ticket. multiple trouble tickets representing independent failures would then point to a single "engineering" ticket, which described the systematic problem. multiple engineering tickets could point to a single "meta" ticket, if appropriate 🌸

---

ASSISTED ENTRY AND DATA VERIFICATION 💅

data (particularly in fixed fields) is only useful for searching if it's entered in consistent formats 🫠 a trouble ticket system needs to help operators fill these fields with the correct format of information. this can be done using assisted entry (menus of acceptable choices), verification routines which check against internal lists or external databases (see next section), or other computer checking ✅

some database systems let a customized "help" screen be associated with each field, which helps new (and experienced!) operators by making context-sensitive trouble ticket system documentation available at every field 💜

very complicated help or operator-guidance systems can be built out of Expert System technology 🤖 this could be as simple as help screens, or help screens with database information inserted (e.g., site contact names and phone numbers). or it could involve hints to the operator based on current network conditions. or it might even ask the operator to run tests and type in the results. (See EXPERT SYSTEMS, below).

INTEGRATION ✨

okay so bestie, a Trouble Ticket system is only maximally efficient and useful if it plays nicely with basically all the other NOC tools 💅 here's the lineup:

1. OPERATOR WINDOW ENVIRONMENT. A NOC Operator needs to see a bunch of info at once, so they're going to be obsessed with a good windowing environment 🪟 The Trouble Ticket system needs to run inside that bigger windowing system so the operator can debug, check databases, use Email, field alerts, and keep an eye out for other emergencies while they're working on a ticket. It's also super useful to run two trouble ticket sessions at the same time. Like, they can search for related tickets while they're in the middle of updating a different one 🙌 And cut and paste between all these screens is mandatory, so technical details can get recorded in the tickets easily 📋
2. ALERT MONITORING SYSTEM. Tickets get opened because of machine alerts all the time, so it should be easy to open a ticket straight from the alert tool 🚨 When that happens, info about the alert and the machine should automatically fill into the ticket ✨ (Now, people have lots of opinions about whether tickets should open automatically with no operator involved. This operator's take is that an operator acknowledgement should be required. But it's a hot debate, so designers of a new system probably should support either option 🤷‍♀️)

The Alarm Clock feature of the trouble ticket system also makes alerts, and those alerts should flow nicely into the Alert Monitor system. That way operators get all their alerts from one single place 🧚

3. DATABASE CONNECTIONS. A good trouble ticketing system will query NOC databases to auto-fill ticket fields wherever it can 🧠 This can be used for:
   • Filling out Network Operator information (e.g., phone number) based on the NetOp's signon id.
   • Filling in contact information based on machine name.
   • Filling in circuit numbers based on link description.
   • Filling in alarm clock or escalation time fields based on the machine or link name and on time of day.
   • Filling in machine serial numbers based on configuration database.
4. MACHINE QUERYABLE INFORMATION. It could also be possible for a trouble ticket system to make standard queries of the network itself when a ticket is opened 🔍 For example, it could request and store the current machine configurations whenever a ticket gets opened for that machine. And on some systems, hardware serial numbers can be pulled by a software query straight from the machine 💜
5. ELECTRONIC MAIL. Problem notifications often show up by email 📧 so it must be possible to easily open a ticket and include the original mail message inside it, as part of the initial problem description. When really technical messages come in from network engineers, it's super helpful to include them verbatim. That way the less technical network operators don't have to rephrase them or squeeze them into predefined formats 🙅‍♀️ Later update messages should be easy to include too. Possibly, tickets could even be opened automatically for mail sent to certain mailboxes. And a response like Your request has been received and assigned ticket number #### might be desirable 💌

Ticket info also needs to be easily available (possibly just through the windowing system) so it can go into emails to engineers and others 📨 And scheduled reports, like daily ones, need to be easy to generate into the Email system too 🗓️

6. DISPATCHING AND NOTIFICATION SYSTEMS. Notifying users, technical contacts, and admins about different classes of problems is a huge real-time part of Network Operations 📣 The rules for who gets told what can be super arbitrary and complicated. They can involve email, notices in computer conferences, automatic beeper pages, and synthesized voice announcements 🎙️ So it would be great if a trouble ticket system could automatically (or operator-initiated) notify the right channels for the current ticket. That would be based on things like network, machine, severity of the problem, how long the problem has lasted, escalation guidelines, and so on ⏰

Databases tied to the trouble ticket system may also have lists of specific people to contact about outages on particular machines. These "who to inform" lists can make customized notification messages super easy to send straight from the trouble ticket system 💅

It may also be possible to dispatch experts directly from the trouble ticket system 🚀 IBM's ECCO system lets customers dispatch Service Engineers straight from machine interactions, and lots of NOCs use a computer hooked up to modems to automatically page engineers. This kind of dispatching should be available from inside the trouble ticket system, along with an automatic note in the ticket that the engineer has been dispatched 📝

7. OTHER TROUBLE TICKET SYSTEMS. When the NOC makes a trouble ticket, it often immediately calls up a telco or another Internet NOC, and they go open their own ticket too 📞 The Internet Engineering Task Force User Connectivity Working Group is also proposing a national trouble ticket tracking system, which would need updates from individual NOC trouble ticket systems 🇺🇸 A state-of-the-art trouble ticket system could have ways to transfer tickets and ticket info in and out of other systems like that 🦋
8. NETWORK ACCESS. Some older trouble ticket systems assumed that anyone who needed the info would get a signon and learn to use that system. But the range of people who need trouble ticket info is now way too big for that to work 🙈 A good system needs to support network queries for tickets and summary reports, plus email delivery of scheduled reports 📬
9. SUBROUTINE INTERFACE. For ad-hoc and not-yet-anticipated needs, the trouble ticket system needs a full-function set of subroutine calls 🧩 These subroutines let people build further trouble ticket functionality that hasn't been specified yet 🌸
10. EXPERT SYSTEMS. Network debugging is a super promising area for expert systems and artificial intelligence 🤖 But an algorithm like that would need access to the alert monitoring system, the configuration and change control systems, the network itself, and the info sitting in the trouble ticket system. So a good future system would make all that info available (probably through the subroutine interface mentioned above). It would also let Network Operators kick off the AI debugging from inside a trouble ticket, including its output as part of the ticket dialogue 🧠✨
11. GRAPHICS/REPORT Capability. Statistical and graphical displays of trouble ticket data need to work with the tools used to make reports, newsletters, and so on 📊

OTHER CONSIDERATIONS ✨

1. INTERACTIVE SPEED. The system must be fast enough to use interactively 🏃‍♀️ NetOps need to answer questions over the phone in real time, and good answers can't happen if every query takes a couple of minutes ☎️ Even more importantly, NetOps need the trouble ticket system to get the info they need to fix the network. If looking up old or currently-open tickets takes more than a few seconds, it simply won't get done 😭 And if updates take forever, they won't get recorded, or they'll get recorded long after the event (which means less accuracy). (Our Operators have asked for a single-line update this ticket with this message utility so they can skip retrieving the ticket for simple updates. Iconic request, honestly 🙌) Any time spent waiting cuts into NetOp productivity and Network reliability ⏳
2. BACKUPS AND RELIABILITY. The trouble ticket system is absolutely crucial for both the immediate and long-term operation of the NOC 🫶 Good systems could back up all their data several times an hour to an auxiliary processor. That processor should be ready for immediate use if the primary system fails 🛟
3. HISTORY AND ARCHIVING. A trouble ticket system is a constantly growing database 📈 Old tickets need to be removed from the system at some interval (a year? several years?) and archived 🗄️ And those archives should be restorable for long-term history processing 🕰️
4. PRIVACY AND SECURITY. Entering, appending, and modifying tickets should be controlled by id and password 🔑 Permissions should be specifiable on a per-field basis. General read access to tickets (or parts of tickets) also needs to be restricted 🙊 Otherwise NetOps will be reluctant to be full and candid in their reporting, and honestly, who can blame them 🥺

---

UTILITY

okay so there are quite a few ideas in this "Wishlist" 📝 Ultimately, what an Operations Center really needs is a totally integrated set of tools that completely models all of its activities and plugs cleanly into all its backup, peer, and vendor NOCs 🌈 It's hard to imagine that this whole system could come out of a shrinkwrapped box, even without the local configuration 📦 But most of these facilities already exist in some system somewhere ✨ Hopefully this document will spark an ongoing conversation about how NOC operator-level tools get used in real operations, and encourage systems implementors and vendors to bring some of this to the aid of real operations 💖 It might even inspire current Operations Centers to add useful features to what they already do 🎀

Security Considerations 🔐

This paper doesn't pose any specific new security issues 🙂 BUT the systems described here would be host database applications, or even distributed host database applications, so all the normal security considerations for that kind of system would apply 🛡️ Multiple classes of user access need to be specified for the different classes of ticket data.

Possible security threats include disclosure of network information, disclosure of confidential material (e.g., circuit numbers or home phone numbers), and denial of service to the Network Operations Center leading to degradation of network service. These are seriously not fun, so please take them seriously 🚨💀

Author's Address 💌

Dale S. Johnson
Merit NOC
1075 Beal Avenue
Ann Arbor, MI 48109-2112

Phone:  (313) 936-2270

Email:  dsj@merit.edu

Discussion and comments can be sent to noc-tt-req@merit.edu. The list is maintained by noc-tt-req-request@merit.edu 🫶