RFC 6057
Comcast's Protocol-Agnostic Congestion Management System
RFC 6057: Comcast's Protocol-Agnostic Congestion Management System ✨
(part 1 of 7 💅)
Okay bestie, this one is a real-world case study from Comcast, a big U.S. cable ISP 🎀. It's RFC 6057, published in December 2010 as an Informational document. Nobody's making it a standard here, it's just documenting how one giant network handled congestion. The authors are C. Bastian, T. Klieber, J. Livingood, J. Mills, and R. Woundy, all from Comcast. The ISSN is 2070-1721. The title block doesn't list anything it obsoletes or updates, so it's a standalone doc 🦋.
Abstract 🌸
So the abstract says this doc describes the congestion management system of Comcast Cable, a huge cable broadband ISP in the U.S. 🇺🇸 Comcast finished rolling this system out on December 31, 2008. Ta-da 💜
Status of This Memo 📜
This one is not an Internet Standards Track spec. It's published just for information 🧚
It's still an IETF product, though, which means it represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Being approved by the IESG doesn't mean it's a candidate for any level of Internet Standard, ngl. Check Section 2 of RFC 5741 if you want the details 🙈
For the current status, errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6057. Bookmark it, periodt 🔖
Copyright Notice ⚖️
Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the publication date. Please actually read those, because they lay out your rights and restrictions for this document 📖
Here's the important bit 👑: any Code Components pulled out of this document need to include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They come with no warranty, same as the Simplified BSD License says.
Table of Contents 🗂️
1. Introduction ....................................................2
2. Applicability to Other Types of Networks ........................3
3. Key Terminology .................................................3
4. Historical Overview .............................................7
5. Summary .........................................................8
6. Relationship between Managing Congestion and Adding Capacity ....9
7. Implementation and Configuration ...............................10
7.1. Thresholds for Determining When a CMTS Port Is in a Near
Congestion State ..........................................14
7.2. Thresholds for Determining When a User Is in an
Extended High Consumption State and for Release from
That Classification .......................................15
7.3. Effect of BE Quality of Service on Users'
Broadband Experience ......................................19
7.4. Equipment/Software Used and Location ......................21
8. Conclusion .....................................................23
9. Exceptional Network Utilization Considerations .................23
10. Limitations of This Congestion Management System ..............24
11. Low Extra Delay Background Transport and Other Possibilities ..24
12. Security Considerations .......................................24
13. Acknowledgements ..............................................25
14. Informative References ........................................261. Introduction 🎀
Comcast Cable is a big broadband ISP based in the U.S., and it mostly serves customers through cable modem tech 📡. Late in 2008, and finishing on December 31, 2008, Comcast rolled out a new congestion management system across its whole network 🚀
Why did they do it? The Internet community was not happy with the old system, and people complained to the U.S. Federal Communications Commission (FCC) about it. The old one targeted specific peer-to-peer (P2P) applications, which was giving major drama 😭. The new system is protocol-agnostic, meaning it doesn't look at or affect specific user applications or network protocols. That's seen as a fairer way to manage network resources during the limited times when congestion might happen 💅
Now, it's really important to get this: congestion can happen in any IP network, and when it does, packets can get delayed or dropped 💀. Bob Briscoe pointed this out on an IETF mailing list. Some packet loss can be normal and/or tolerable. His exact words were: "But a single TCP flow with a round trip time (RTT) of 80 ms can attain 50 Mbps with a loss fraction of 0.0013% (1 in ~74,000 packets) so there's no need to try to achieve loss figures much lower than this. And indeed, if flows aren't bottlenecked elsewhere, TCP will drive the system until it gets such loss levels. If, instead, a customer is downloading five separate 10 Mbps TCP flows still with an 80-ms RTT, TCP will drive losses up to 1 in ~3,000, or 0.03%, and any lower loss rates won't be able to improve performance".
So yeah, apps and protocols have been designed to handle the fact that congestion happens in any IP network. We'll get into the mechanics later in this doc 🌈
The purpose here is to explain how this large-scale congestion management system works. Other ISPs and solution developers asked questions, and they want to learn from it or deploy similar systems on their own networks, so this is partly a response to them 🫶. The authors also hope the doc can help inform new IETF work, so that better systems and protocols might be possible down the road ⭐. Last but not least, the authors wanted to document this system openly and transparently, so there could be no doubt about how it actually worked 🔥
2. Applicability to Other Types of Networks 🌍
A few document reviewers and other IETF folks pointed out something important 💜. Even though this doc talks about functional elements specific to a Data Over Cable Service Interface Specification (DOCSIS)-based network, this kind of congestion management system could be applied to pretty much any type of network 🦋
So readers should keep that in mind. This protocol-agnostic congestion management approach could fit into a wide variety of network types and implementations 🧚
3. Key Terminology 📚
This section defines the key terms used in the doc. Some of them refer to elements of the Comcast network, so it might help to look at Figure 1 (see Section 7) while you go through these 🎀
3.1. Cable Modem 🏠
A device at the customer premise that's used to access the Comcast High Speed Internet (HSI) network. Sometimes the customer owns the cable modem, and sometimes the cable operator owns it 🤷♀️
The device has an interface (basically, a place to plug in a cable) that connects the coaxial cable from the cable company to the modem. It also has one or more interfaces for connecting the modem to a customer's PC or home gateway device (for example, a home gateway, router, firewall, access point, etc.) 🔌
Sometimes the cable modem function, meaning the ability to access the Internet, is built into a home gateway device or an Embedded Multimedia Terminal Adapter (eMTA). Once it's connected, the cable modem links the customer to the HSI network and then to the broader Internet 🌸
3.2. Cable Modem Termination System (CMTS) 🖥️
This is a piece of hardware in a cable operator's local network (usually in a "headend", see Section 3.10). It acts as the gateway to the Internet for cable modems in a particular geographic area 🗺️
Easy way to think about it: the CMTS is like a router. One side has interfaces leading to the Internet, and the other side has interfaces connecting to Optical Nodes and then to customers, in a so-called "last mile" network 🛣️
3.3. Cable Modem Termination System (CMTS) Port 🔗
Also called just a "port". A port is a physical interface on a device, used to connect cables so the device can talk to other devices and transfer information/data 💾. A CMTS port is one example of a physical port 🎀
A CMTS has both upstream and downstream network interfaces to serve the local access network. These are called upstream or downstream ports 📶. A port usually serves a neighborhood of hundreds of homes. Over time, CMTS ports tend to serve fewer and fewer homes, because the network gets segmented for capacity growth 🌱
Before DOCSIS version 3, a single CMTS physical port was used for either transmitting or receiving data downstream or upstream to a given neighborhood. With DOCSIS version 3 and its channel bonding feature, multiple CMTS physical ports can be combined into one virtual port ✨. The CMTS also gets a brief definition in Section 2.6 of [RFC3083] 📖
3.4. Channel Bonding 🔀
This is a technique for combining multiple downstream and/or upstream channels to increase customers' download and/or upload speeds, respectively 🚄
Multiple channels from the Hybrid Fiber Coax (HFC) network (Section 3.11) can get bonded into a single virtual port (called a bonded group). That bonded group acts like one big channel or port, giving customers faster speeds 🔥. Channel bonding is a feature of DOCSIS version 3, as described in [DOCSIS_MULPI] 🧠
3.5. Coaxial Cable (Coax) 📺
This is a type of cable a cable operator uses to connect customer premise equipment (CPE) to the HFC network. CPE includes things like TVs, cable modems (including eMTAs), and Set Top Boxes 📦. Coax can be used inside the home, and also in segments of the "last mile" network running to a home or customer premise location 🏡
There are lots of grades of coaxial cable, and different types are used for different purposes on the network 🌈
3.6. Comcast High Speed Internet (HSI) 🌐
This is a service or product Comcast offers for delivering Internet service over a broadband connection 💖
3.7. Customer Premise Equipment (CPE) 🏘️
Any device that lives at the customer's residence and is connected to the Comcast network. It doesn't matter whether Comcast controls it or not 🫶
3.8. Data Over Cable Service Interface Specification (DOCSIS) 📋
This is a reference standard developed by CableLabs. It specifies how components on cable networks need to be built to enable HSI service over an HFC network, as noted in [DOCSIS_CM2CPE], [DOCSIS_PHY], [DOCSIS_MULPI], [DOCSIS_SEC], and [DOCSIS_OSSI] 🧩. These standards define the specs for the cable modem and the CMTS so that any DOCSIS-certified cable modem works on any DOCSIS-certified CMTS, no matter which vendor made it 🤝
That interoperability is the reason customers can buy a DOCSIS-certified modem at a retail outlet and use it on their cable-networked home 🛒. All the DOCSIS-related standards are public and available at the CableLabs website, http://www.cablelabs.com 🌸
3.9. Downstream ⬇️
okay so downstream is the direction a signal travels, and in this case that means from the network to a user 🌸 It's the traffic you're pulling down when you're watching a web-based video, reading web pages, or downloading software updates. Bestie, you're basically always in downstream mode 💅
3.10. Headend 📺
A headend is a cable facility that receives TV signals so they can be distributed over the HFC network to the end customers. ngl it's giving central hub energy 🏠 It typically also houses one or more CMTSs, and sometimes people call it a "hub" 🧚
3.11. Hybrid Fiber Coax (HFC) 🦋
HFC is a network architecture used mainly by cable companies. It's made of fiber-optic and coaxial cables, and it currently delivers Voice, Video, and Internet services to customers, as defined in Section 1.2 of [DOCSIS_MULPI]. It's a whole combo, and I'm obsessed 😭
3.12. Internet Protocol Detail Record (IPDR) 📊
IPDR is standardized technology for monitoring and/or recording subscribers' upstream and downstream Internet usage data based on their cable modem. The data gets collected from the CMTS and sent to a server for further processing 🫶 You can find more info at http://www.ipdr.org, as well as [IPDR_Standard] and [DOCSIS_IPDR]. Lowkey, the paper trail is iconic ✨
3.13. Optical Node 🔦
An Optical Node is a component of the HFC network that's usually found in customers' local neighborhoods. Its job is to convert the optical signals sent over fiber-optic cables into electrical signals that can travel over coaxial cable to customers' cable modems, or vice versa 💜 A fiber-optic cable connects the Optical Node, through distribution hubs, to the CMTS. Coaxial cable connects the Optical Node to customers' cable modems. It's a whole relay race 🏃♀️
3.14. Provisioned Bandwidth 🎀
Provisioned Bandwidth is the peak speed associated with a tier of service that a customer bought. For example, a customer with a 105 Mbps downstream and 10 Mbps upstream speed tier would be said to be provisioned with 105 Mbps of downstream bandwidth and 10 Mbps of upstream bandwidth. In industry parlance this is often called 105/10 service 💅
The Provisioned Bandwidth is the speed that a customer's modem is configured (and the network is engineered) to deliver on a regular basis. That's not the same as a "Committed Information Rate" or a guaranteed rate, so don't get it twisted 🙈 Internet speeds are generally a best effort service, and they depend on a bunch of variables, many of which are outside the control of an Internet Service Provider (ISP). In general, speeds don't typically exceed a customer's provisioned speed.
Comcast, however, invented a technology called "PowerBoost" [PowerBoost_Specification]. It lets users experience brief boosts above their provisioned speeds while they transfer large files over the Internet, by using excess capacity that may be available in the network at that time. Main character energy, literally 🚀
3.15. Quality of Service (QoS) 🌈
QoS is a set of techniques for managing network resources so that specific data flows get a level of performance, as described in [RFC1633] and [RFC2475]. One method is to differentiate the type of traffic by class or flow and assign priorities to each type. When the network gets congested, data packets marked as higher priority have a higher likelihood of being serviced. It's basically VIP treatment for packets 👑
3.16. Upstream ⬆️
Upstream is the direction a signal travels, in this case from the user to the network. Upstream traffic happens when users are uploading something, like sending email, sending files to another computer, or uploading photos to a digital photo website 🍓
4. Historical Overview 🕰️
Comcast began the engineering project to develop a new congestion management system in March 2008. That's the same month Comcast hosted the 71st meeting of the IETF in Philadelphia, PA, USA ✨ On May 28, 2008, Comcast took part in an IETF Peer-to-Peer Infrastructure Workshop [RFC5594], hosted by the Massachusetts Institute of Technology (MIT) in Cambridge, MA, USA. Such a big moment 😭
To join this workshop, interested attendees had to submit a paper to a technical review team. Comcast did that on May 9, 2008, in [COMCAST_P2PI_PAPER] 📝 Comcast then attended and participated in the workshop, which was super valuable. During the workshop, Comcast outlined the high-level design for a new congestion management system [COMCAST_P2PI_PRES] and asked for comments and other feedback from attendees and other members of the Internet community. Presentations were also posted to the IETF's P2Pi mailing list 💌 The congestion management system outlined in that May 2008 workshop was later tested in trial markets. In essence, it's what Comcast later deployed in 2008 🔥
After an August 2008 FCC document [FCC_Memo_Opinion] about how Comcast managed congestion on its High-Speed Internet ("HSI") network, Comcast disclosed more technical details to the FCC [FCC_Net_Mgmt_Response] and to the public. Those details covered the congestion management system it intended to implement and did implement by the end of 2008 [FCC_Congest_Mgmt_Ltr], including the thresholds involved in this new system 🧾 The way this system is deployed in the Comcast network is necessarily specific to the technologies and designs used in that network. Still, a similar system could be deployed on virtually any large-scale ISP network or other IP network 🌍
5. Summary 🎀
Comcast's HSI network has elements that are shared across many subscribers. That means Comcast's HSI customers share upstream and downstream bandwidth with their neighbors 🏘️ The available bandwidth is substantial, but so is the demand. So when a relatively small number of customers in a neighborhood put disproportionate demands on network resources, it can cause congestion that degrades their neighbors' Internet experience 😤 The goal of Comcast's new congestion management system is to let all users of our network resources access a "fair share" of that bandwidth, so every HSI customer gets a high-quality online experience 💖
Importantly, the new approach is protocol-agnostic. That means it doesn't manage congestion by focusing on the use of specific protocols that place a disproportionate burden on network resources, or on any other protocols 🙅♀️ Instead, it focuses on managing the traffic of the individuals who are using the most bandwidth at times when network congestion threatens to degrade subscribers' broadband experience, and who are contributing disproportionately to that congestion at those points in time. Targeted, not vibes-based 🎯
Specific details about these practices, including relevant threshold information, the type of equipment used, and other particulars, are discussed at length later in this document. At the outset, though, here's a very high-level, simplified overview of how these practices work. Despite all the detail further below, the fundamentals can be summed up succinctly 🧠
- Software installed in the Comcast network continuously examines aggregate traffic usage data for individual segments of Comcast's HSI network. If overall upstream or downstream usage on a particular segment reaches a pre-determined level, the software moves on to step two 👀
- At step two, the software examines bandwidth usage data for subscribers in the affected network segment to determine which subscribers are using a disproportionate share of the bandwidth.
If the software determines that a particular subscriber or subscribers have been the source of high volumes of network traffic during a recent period of minutes, traffic originating from that subscriber or those subscribers will temporarily be assigned a lower priority status 🐢
- While a subscriber's traffic has the lower priority status, their packets will not be delayed or dropped so long as the network segment is not actually congested. If, however, the network segment becomes congested, their packets could be intermittently delayed or dropped 🫣
- The subscriber's traffic returns to normal priority status once his or her bandwidth usage drops below a set threshold over a particular time interval 🌟
Comcast put in a ton of effort over many months to formulate these plans for the congestion management approach. We adjusted the plans and subjected them to real-world trials 🧪 Market trials were conducted in Chambersburg, PA; Warrenton, VA; Lake City, FL; East Orange, FL; and Colorado Springs, CO, between June and September 2008. The trials let us validate the utility of the general approach and collect substantial trial data to test multiple variations and alternative formulations. Science era 🔬
6. Relationship between Managing Congestion and Adding Capacity 📈
Many people have questioned whether congestion should ever exist at all, if an ISP is adding sufficient capacity. There's definitely a relationship between capacity and congestion 🤔 But there are two types of congestion that generally show up in a network.
The first general type is regularly occurring. It results from traffic levels gradually increasing up to a point where typical usage peaks cause congestion on a regular basis. Comcast, like many ISPs, has a set capacity management process where capacity additions are automatically triggered based on certain usage trends 📊 That process is geared toward bringing additional capacity to the network before regularly occurring congestion starts. So capacity gets added when it's needed and before it has noticeable effects. This process exists because capacity additions aren't instantaneous, and in many cases they require significant physical work 🏗️
The second general type is unpredictable congestion, which can happen for a wide range of reasons 😬 One example is current events, where users may all rush to access specific content at the exact same time, and the systems serving that content may not be able to keep up with demand. Another example is a localized disaster, where some network paths have been destroyed or otherwise impaired, and many users are trying to communicate with one another at traffic levels well above normal 🆘
So in both cases, even with continuous upgrades and constant investment in additional capacity, network capacity is still not unlimited 🥲 A congestion management system, absent superior protocol-based solutions that don't currently exist, can therefore help manage the effects of congestion on users and improve their Internet experience 💜
7. Implementation and Configuration ✨
okay so first things first, the implementation details and the overall design of this system are built around the traffic patterns on the Internet today, and the authors think those patterns will still be around soon 🔮. they wanted this thing to be super adaptable and a solid long-term investment, but if traffic changes a lot, the configuration might need a glow-up, or in extreme cases it might need a totally different system altogether 💀
to get how these new congestion management practices work, you kinda need the vibe of how Comcast's HSI network is built 🧚. it's a hybrid fiber-coax network. coaxial cable connects each subscriber's cable modem to an Optical Node, and fiber-optic cables connect the Optical Node, through distribution hubs, to the Cable Modem Termination System (CMTS), which is also known as a "data node" 🎀. the CMTSs connect to higher-level routers, and those hook up to Comcast's Internet backbone facilities 🌈. today Comcast has over 3,200 CMTSs deployed throughout the network, serving over 15 million HSI subscribers 🫶.
each CMTS has multiple "ports" that handle traffic coming in and going out 🦋. every cable modem on the Comcast HSI network is connected to the CMTS through these ports. a port can be "downstream" (it sends info to cable modems) or "upstream" (it receives info from cable modems) 💅. (quick aside 🙈: "port" here generally means single channels on a CMTS, but all of this also applies to virtual channels, aka "bonded groups", in a DOCSIS 3.0 environment.) even without channel bonding, multiple channels usually come out of each physical port, so there's generally a mapping of multiple channels to each physical port 👑.
rn, on average, about 275 cable modems share the same downstream port, and about 100 share the same upstream port 😭. this is constantly changing though (both numbers generally get smaller over time, based on current DOCSIS tech) ✨. both port types can get congested and tank the broadband experience for subscribers, and unlike the previous congestion management practices, both upstream and downstream traffic are subject to management in this new system 🔥.
based on the network design and the traffic patterns they've observed, the most likely spot for congestion is on these CMTS ports 💜. so the congestion management system measures traffic conditions on CMTS ports and applies policy actions to traffic on those ports, not some farther segment of the network 🧚.
to roll out Comcast's new protocol-agnostic congestion management practices, Comcast bought new hardware and software and deployed it near the Regional Network Routers ("RNRs") that sit further upstream 🌸. this new hardware is made up of Internet Protocol Detail Record ("IPDR") servers, Congestion Management servers, and PacketCable Multimedia ("PCMM") servers. more details on each piece are in Section 7.4 below. but here's the key part 👀: even though these servers physically sit at the RNR, they communicate with and individually manage multiple ports on multiple CMTSs 💖. that means bandwidth usage on one CMTS port has no effect on whether the practices in this document get applied to a subscriber on a different CMTS port 🙈.
Figure 1 is a simplified graphical depiction of that architecture 🗺️:
Figure 1: Simplified Network Diagram Showing High-Level Comcast
Network and Servers Relevant to Congestion Management
-------------------------
/ \
| Comcast Internet Backbone |
\ -----
+------------+ --------------------/ \
| Congestion | / \
| Management |<+++GigE++++ +---->| Internet |
| Server | + | | Backbone |
+------------+ + | \ Router /
+ Fiber \ /
+------------+ + | -----
| QoS | + |
| Server |<+++GigE++++ \/
| | + -----
+------------+ + / \
+ / \
+------------+ + | Regional |
| Statistics | +++++++>| Network |
| Collection |<+++GigE++++ | Router |
| Server | \ /
+------------+ +---Fiber------>\ /<------Fiber----+
| ----- |
\/ \/
----- -----
/ \ / \
/ Local \ / Local \
| Market | | Market |
\ Router / \ Router /
+--------->\ /<------------+ \ /
| ----- | ------
| /\ | /\
Fiber | Fiber |
| Fiber | Fiber
| | | |
\/ \/ \/ \/
/------\ /------\ /------\ /------\
| CMTS | | CMTS | | CMTS | | CMTS |
\------/ \------/ \------/ \------/
/\ /\ /\ /\
| | | |
Fiber Fiber Fiber Fiber
| | | |
\/ \/ \/ \/
+---------+ +---------+ +---------+ +---------+
| Optical | | Optical | | Optical | | Optical |
| Node | | Node | | Node | | Node |
+---------+ +---------+ +---------+ +---------+
/\ /\ /\ /\ /\ /\
|| || ||______ || _____|| ||
Coax Coax |__Coax| Coax |Coax__| Coax
|| || || || || ||
\/ \/ \/ \/ \/ \/
+=======+ +=======+ +=======+ +=======+ +=======+ +=======+
= Cable = = Cable = = Cable = = Cable = = Cable = = Cable =
= Modem = = Modem = = Modem = = Modem = = Modem = = Modem =
+=======+ +=======+ +=======+ +=======+ +=======+ +=======+
================================================================
+ Note: This diagram is a simplification of the actual network +
+ and servers, which in actuality includes significant +
+ redundancy and other details too complex to represent here. +
================================================================
Figure 1every Comcast HSI subscriber's cable modem has a "bootfile", which is basically a config file with certain info about the subscriber's service so it keeps working properly 📄. (note: no personal information is in the bootfile, it only has info about the service the subscriber bought 🔒.) for example, it has the maximum speed a modem can hit (what we call "provisioned bandwidth" in this doc) based on the tier the customer picked, like personal/residential, commercial, etc. 💰. bootfiles get reset from time to time to account for network changes and updates, and that's usually done by a command sent from the network that the subscriber never even notices 😴.
to prep for the switch to this new system, Comcast sent new bootfiles to HSI customers' cable modems that created two Quality of Service (QoS) levels for Internet traffic going to and from the modem: (1) "Priority Best Effort" ("PBE") traffic and (2) "Best Effort" ("BE") traffic 🎀. just like with earlier bootfile changes, swapping the old bootfile for the new one needs zero active participation from Comcast customers 🫶.
from then on, all traffic going to or coming from cable modems on the Comcast HSI network is designated as either PBE or BE 💅. PBE is the default status for all Internet traffic coming from or going to a particular cable modem. traffic only gets designated BE for a particular cable modem when both of two conditions are met:
- First, the usage level of a particular upstream or downstream port of a CMTS, as measured over a particular period of time, must be nearing the point where congestion could degrade users' experience 😬. we call this the "Near Congestion State". based on the technical trials we've run (further validated in our full deployment), we've set a threshold, described in more detail below, for when a particular CMTS port enters that state 📊.
- Second, a particular subscriber must be making an extended, high contribution to the bandwidth usage on that port, relative to the service tier they purchased, as measured over a particular period of time 👀. we call this the "Extended High Consumption State". again, based on the technical trials (further validated in our full deployment), we've set a threshold, described in more detail below, for when a particular user enters that state 🧠.
when, and only when, both conditions are met, a user's upstream or downstream traffic (depending on which type of port is in the Near Congestion State) gets designated as BE 🫣. then, to the extent that actual congestion occurs, any delay resulting from it will hit BE traffic before it hits PBE traffic 💀.
we now explain all of this in greater detail in the following sections 💖
7.1. Thresholds for Determining When a CMTS Port Is in a Near Congestion State ✨
okay so bestie, let's talk about when a CMTS port gets flagged as being in a Near Congestion State 💅. It happens when the traffic going to or from that port goes over a set level (the "Port Utilization Threshold") for a set amount of time (the "Port Utilization Duration") 🦋. The Port Utilization Threshold is a percentage of the port's total aggregate upstream or downstream bandwidth for the timeframe we're looking at, and the Port Utilization Duration is measured in minutes ⏱️.
The numbers we use for these thresholds were tentatively picked after a huge process 🧪. We did lab tests, simulations, technical trials, vendor evaluations, customer feedback, and a third-party consulting analysis. Lowkey, it's giving "we did our homework" 📚. But here's the thing: values like these are almost guaranteed to change over time 🌸. Anti-spam and other network management practices get tweaked when new issues show up, and Comcast will keep gathering data and doing more analysis as the technique gets used at scale. Also, big networks and software systems can have bugs or unexpected errors, which might mean software patches or other fixes 🔧. Through all of it, Comcast's decisions are driven by one goal: delivering the best possible experience to our HSI subscribers 💖.
So, the starting points we landed on are a 70 percent Port Utilization Threshold for upstream and an 80 percent Port Utilization Threshold for downstream 🎀. For the Port Utilization Duration, the starting point is approximately 15 minutes 🕒. (Some technical limitations in certain newer CMTSs on Comcast's network might make that time period vary a little, though 🙈.) Putting it all together: over any 15-minute period, if an average of more than 70 percent of a port's upstream bandwidth capacity, or more than 80 percent of a port's downstream bandwidth capacity, is used, that port is determined to be in a Near Congestion State 🚨.
Now for the good news 🥳. Based on the trials and operational experience so far, a typical CMTS port on our HSI network is in a Near Congestion State only for relatively small parts of the day, if at all 🌈. We can't predict which time will be the busiest on a particular port on a particular day, though 🔮. The trial data and operational experience also show that even when a port is in a Near Congestion State, the times the network actually becomes congested during the Port Utilization Duration are few ✨. And managed users whose packets might get delayed or dropped during those congested moments? They perceive little, if any, effect, as discussed below 🫶.
7.2. Thresholds for Determining When a User Is in an Extended High Consumption State and for Release from That Classification ✨
okay so once a CMTS port hits a Near Congestion State 🚨, the software checks whether any cable modems are hogging bandwidth way more than they should 👀. (Quick note bestie: each cable modem is usually tied to one household, but the software can't actually tell who the individual users are, how many people share a modem, or what any particular person is doing online 🙈. So for this doc, "cable modem," "user," and "subscriber" all mean a subscriber or user account, not an actual human 💅.)
To enter an Extended High Consumption State, a user has to eat more than a certain percentage of their provisioned upstream or downstream bandwidth (that's the "User Consumption Threshold") for a certain amount of time (that's the "User Consumption Duration") 🍓. The threshold is measured against the user's total provisioned bandwidth, which is the max speed a particular modem can hit based on the tier they bought, like personal/residential or commercial 💼. For example, if someone buys a plan with 50 Mbps downstream and 10 Mbps upstream, their provisioned downstream speed is 50 Mbps and their provisioned upstream speed is 10 Mbps 🦋. The threshold is a percentage of that provisioned number instead of one fixed value, so people on faster tiers get higher User Consumption Thresholds too 🚀. And the User Consumption Duration is measured in minutes ⏱️.
After lab tests, simulations, technical trials, customer feedback, vendor evaluations, and an independent third-party consulting analysis 🧪, we landed on 70 percent of a subscriber's provisioned upstream or downstream bandwidth as the starting point for the User Consumption Threshold, and 15 minutes as the starting point for the User Consumption Duration 🎯. (Our full deployment has further validated these numbers ✅.) So when a subscriber averages 70 percent or more of their provisioned upstream or downstream bandwidth over a 15-minute window, they're in an Extended High Consumption State 🔥. This is a consumption-based threshold, not a peak-speed-based one 📊. A brief burst above 70 percent doesn't put anyone in this state 🙅♀️. The bitrate has to be exceeded across the entire User Consumption Duration 💜.
The User Consumption Thresholds are set high enough that normal stuff like VoIP, gaming, web surfing, or most streaming video can't push a subscriber on our standard-level HSI service over the threshold all by itself 🎧🎮. For example, one of Comcast's common HSI tiers has 22 Mbps of provisioned downstream bandwidth today 🌈. Streaming video from Hulu (even some HD) uses less than 2.5 Mbps 📺, a Vonage or Skype VoIP call uses less than 131 kbps ☎️, and streaming music uses less than 128 kbps 🎶. (In this example, 70 percent of 22 Mbps is 15.4 Mbps 🧮.) As mentioned above, these values can change as needed 🔄, the same way other anti-spam or network management practices get adjusted when new issues pop up or when unexpected software bugs or other problems show up 🛠️.
Based on data from the trial markets where the new congestion management practices were tested (and further validated in our full deployment) 📈, on average less than one-third of one percent of subscribers get their traffic priority status changed to the BE state on any given day 🥳. For example, in Colorado Springs, CO, the biggest test market, on a typical day in August 2008, an average of 22 users out of 6,016 total subscribers in the trial had their traffic priority changed to BE at some point during the day 🏙️.
A user's traffic gets released from BE when their bandwidth consumption drops below 50 percent of their provisioned upstream or downstream bandwidth for about 15 minutes 🕊️. These release criteria are there to minimize (and hopefully prevent) QoS oscillation, which is when a user keeps bouncing back and forth between BE and PBE 🔁. Without a lower release criteria, we were worried that some users would oscillate between BE and PBE for a long time with no real benefit to the system or to other users, and that it would create an unnecessary signaling burden on the system 😩. NetForecast, Inc., an independent consultant hired to analyze and recommend changes for Comcast's trials and related congestion management work, suggested this approach 🙌. It has worked really well in our trials, lab testing, and the national deployment that followed 💖.
So, simply put, there are four steps for deciding whether a particular cable modem's traffic is designated PBE or BE 📝:
- Determine if the CMTS port is in a Near Congestion State.
- If yes, determine whether any users are in an Extended High Consumption State.
- If yes, change those users' traffic to BE from PBE. If the answer at either step one or step two is no, no action is taken.
- If a user's traffic has been designated BE, check user consumption at the next interval. If user consumption has declined below the predetermined threshold, reassign the user's traffic as PBE. If not, recheck at the next interval.
Now here's the tea ☕. If a CMTS keeps entering a Near Congestion State, and congestion actually happens, but none of the users meet the criteria for an Extended High Consumption State, that might mean the congestion is regularly happening rather than being random and unpredictable 🤔. So it can be one more data point in favor of thinking about whether and when to add capacity 📈✨.
Figure 2 shows how this congestion management process works 🗺️, using an example where upstream port utilization may be reaching a Near Congestion State (the same diagram, with different values in the right places, could show the management process for downstream ports too 💅):
/\
+------------+ / \ +---------+ +---------+
| Start | / \ | | / /
| Congestion | / \ | | / /
| Management +-->+ Question +--YES-->| Result |--THEN-->/ Action /
| Process | \ #1 / | #1 | / #1 /
| | \ / | | / /
+------------+ \ / +---------+ +---------+
\/ |
| THEN
NO |
| \/
\/ /\
+---------+ / \
| | / \
| | / \
| Result |<-------------NO------------+ Question +
| #2 | \ #2 /
| | \ /
+---------+ \ /
\/
|
YES
|
/\ \/
+---------+ / \ +---------+
| | / \ | |
| | / \ THEN, AT | |
| Result |<--YES--+ Question + <---NEXT ANALYSIS------+ Result |
| #4 | \ #3 / POINT /\ | #3 |
| | \ / | | |
+---------+ \ / | +---------+
\/ |
| |
+------------NO-------------+KEY TO FIGURE 2 ABOVE:
Question #1: Is the CMTS Upstream Port Utilization at an average
of OVER 70% for OVER 15 minutes?
Result #1: CMTS marked in a Near Congestion State, indicating
congestion *may* occur soon.
Action #1: Search most recent analysis timeframe (approx. 15 mins.)
of IPDR usage data.
Question #2: Are any users consuming an average of OVER 70% of
provisioned upstream bandwidth for OVER 15 minutes?
Result #2: No action taken.
Result #3: Change user's upstream traffic from Priority Best Effort
(PBE) to Best Effort (BE).
Question #3: Is the user in Best Effort (BE) consuming an average
of LESS THAN 50% of provisioned upstream bandwidth
over a period of 15 minutes?
Result #4: Change user's upstream traffic back to Priority Best
Effort (PBE) from Best Effort (BE).Figure 2 🎀
7.3. Effect of BE Quality of Service on Users' Broadband Experience 🫧
okay bestie, so when a CMTS port is in a Near Congestion State and a cable modem on that port is in an Extended High Consumption State, that modem's traffic gets designated as BE 💅. depending on how much the port is being used, this might or might not actually delay the user's traffic, or in extreme cases drop it, before PBE traffic gets dropped 👀. the reason is how the CMTS handles traffic. CMTS ports have something commonly called a "scheduler" that lines up all the packets coming from or going to the cable modems on that port and handles them in turn 🎀. a certain number of packets can be processed by the scheduler at any given moment. for each time slot, PBE traffic gets priority access to the available capacity, and BE traffic is processed on a space-available basis 🦋.
think of it like buses that empty and fill up at incredibly fast speeds 🚌💨. every two milliseconds in this case, an empty bus pulls up to the figurative "bus stop" and fills up with as many packets as are waiting for "seats," up to the limits of the bus's capacity. during non-congested periods, the bus usually has several empty seats, but during congested periods, the bus fills up and packets have to wait for the next bus 😭. it's during the congested periods that BE packets get affected. if there's no congestion, packets from a user in a BE state should have little trouble getting on the bus when they arrive at the stop ✨. but if there's congestion in a particular instance, the bus might get filled by PBE packets before any BE packets can climb on. in that situation, the BE packets have to wait for the next bus that isn't filled by PBE packets 💀. realistically, all of this happens in two-millisecond increments, so even if the packets miss 50 "busses," the delay will only be about one-tenth of a second 🙈.
during times of actual network congestion, when BE packets might be intermittently delayed, a user whose traffic is delayed could notice different things depending on which apps they're using 📱. typically, a user in a BE state during actual congestion might find that a webpage loads sluggishly, a peer-to-peer upload takes somewhat longer to complete, or a VoIP call sounds choppy 😵. ngl, the same thing could happen to customers on a congested port even without any congestion management. the difference is that the effects of those delays get shifted toward the people who have been placing the greatest burden on the network, instead of being distributed randomly among the users of that port without regard to their consumption levels 💅. as a matter of fact, our studies concluded that the experience of PBE subscribers improves when this congestion management system is enabled 🫶. that conclusion is based on network measurements, such as latency 📊.
NetForecast explored the potential risk of a worst-case scenario for users whose traffic is in a BE state: the possibility of "bandwidth starvation" in the theoretical case where 100 percent of the CMTS bandwidth is taken up by PBE traffic for an extended period of time 😱. in theory, such a condition could mean that a given user whose traffic is designated BE would be unable to do an upload or download (as noted above, both are managed separately) for some period of time 💀. however, when these management techniques were tested, first in company testbeds and then in real-world trials conducted in the five markets (further validated in the full deployment), that theoretical condition did not occur 🎉. also, our experience with the system as fully deployed in the production network shows that these management practices have very modest real-world impacts 🌸. Comcast also did not receive a single customer complaint, in any of the trial markets, that could be traced to this congestion management system, even though the trials were broadly publicized 🙈. in our subsequent national deployment into the production network, we still have yet to find a specific complaint that can be traced back to the effect of this congestion management system 👑.
Comcast continues to monitor how user traffic is affected by these new congestion management techniques, and will make the adjustments necessary to ensure that all Comcast HSI customers have a high-quality Internet experience 🫶💖.
7.4. Equipment/Software Used and Location 🧚
the above-mentioned functions are carried out using three different types of application servers, supplied by three different vendors 🧚. as mentioned above, these servers are installed near Comcast's regional network routers. the exact locations of these servers aren't particularly relevant to this document, since that information doesn't change the fact that the servers manage individual CMTS ports 🗺️.
the first application server is an IPDR server, which collects relevant cable modem volume usage information from the CMTS, like how many aggregate upstream or downstream bytes a subscriber uses over a particular period of time 📥. IPDR has been adopted as a standard by many industry organizations and initiatives, such as CableLabs, the Alliance for Telecommunications Industry Solutions (ATIS), the International Telecommunication Union (ITU), and the Third Generation Partnership Project (3GPP), among others 🌈. the IPDR software deployed was developed by Active Broadband Networks, and it's noted as the Statistics Collection Server in Figure 3 📊.
the second application server is the Congestion Management server, which uses the Simple Network Management Protocol (SNMP) [RFC3410] to measure CMTS port utilization and detect when a port is in a Near Congestion State 🔍. when this happens, the Congestion Management server then queries the relevant IPDR data for a list of cable modems meeting the criteria set forth above for being in an Extended High Consumption State 🍓. the Congestion Management server software deployed was developed by Sandvine 💜.
if one or more users meet the criteria to be managed, then the Congestion Management server notifies a third application server, the PCMM application server, as to which users have been in an Extended High Consumption State and whose traffic should be treated as BE 🦋. the PCMM servers are responsible for signaling a given CMTS to set the traffic for specific cable modems with a BE QoS, and for tracking and managing the state of such CMTS actions 🎀. if no users meet the criteria to be managed, then no users will have their traffic managed ✨. the PCMM software deployed was developed by Camiant, and it's noted as the QoS Server in Figure 3 💅.
Figure 3 graphically depicts the high-level management flows among the congestion management components on Comcast's network, as described above 👑:
Figure 3: Simplified Diagram Showing High-Level Management Flows Relevant to the System
+---------------+ +---------------+
| Congestion | Instruct QoS Server | QoS |
| Management |******to Change QoS for****>| Server |
| Server | a Device | |
+----+---+------+ +-------+-------+
/\ /\ *
| | Relay Selected *
X +---Statistics: Bytes---+ QoS Action:
| Up/Down by Device | Change from PBE
X +-------+-------+ to BE, or from
| | Statistics | BE to PBE
X | Collection | *
Periodic SNMP | Server | *
Requests to +---------------+ *
Check CMTS Port /\ *
Utilization | *
Levels Statistics Sent *
| Periodically From CMTS *
X | *
| +-----------+-----------+ *
+-X-X-X-X-X-X->| CMTS in Headend |<********
+-----------------------+
H /\ /\ H
H Internet Traffic H
H to/from User H
H \/ \/ H
/+---------------------+\
/ | User's +---------+ |\
/ | Home | Cable | | \
| | Modem | |
============ | +---------+ |
= Notes: = +----------------------+
= ========================================================
= 1 - Statistics Collection Servers use IP Detail Records (IPDR). =
= 2 - QoS Servers use PacketCable Multimedia (PCMM) =
= to set QoS gates on the CMTS. =
= 3 - This figure is a simplification of the actual network and =
= servers, which included redundancies and other complexities =
= not necessary to depict the functional design. =
===================================================================
Figure 38. Conclusion ✨
okay bestie, so Comcast started design and development of this new protocol-agnostic congestion management system in March 2008 🌸. we shared the design with the IETF and others in the Internet community, as well as with an independent consultant, and we incorporated the feedback we received into the final design 🫶. following lab testing, the system was tested in Comcast's production network in trial markets between June and September 2008 🧪. Comcast's production network transition to this new protocol-agnostic congestion management system began in October 2008 and was completed on December 31, 2008 🎉.
as described here, the new approach does not manage congestion by focusing on managing the use of specific protocols 🙅. nor does this approach use TCP "reset packets" [RFC3360] 💅. instead, the system acts such that during periods when a CMTS port is in a Near Congestion State, it (1) identifies the subscribers on that port who have consumed a disproportionate amount of bandwidth over the preceding 15 minutes and (2) lowers the priority status of those subscribers' traffic to BE status until those subscribers meet the release criteria 🦋. during periods of actual congestion, the system handles PBE traffic before BE traffic 👑. Comcast's trials and subsequent national deployment indicate that this new congestion management system ensures a quality online experience for all of Comcast's HSI customers 💖🌈
9. Exceptional Network Utilization Considerations 🌪️
Okay so this system was built to handle the pretty normal kind of congestion that can pop up on basically any IP network 💅. But ngl, it could also be super useful for "exceptional network utilization" moments that existing usage models don't or can't accurately predict 👀. Some network operators call these events "surges" in utilization, kind of like sudden spikes in demand on electrical power grids, which lots of people are already familiar with 🔌✨.
For example, during a severe global pandemic, it's totally possible that huge chunks of the population will need to work remotely through their Internet connection 💻. That could mean a level of utilization nobody has ever really seen before 😭. In situations like that, it's giving "we need a flexible congestion management system" 🫶. One that can adapt to the situation and help allocate network resources while more capacity is being brought online, or while a temporary condition is still hanging around 🧚.
10. Limitations of This Congestion Management System 🚧
Let's be real, the system has some main limitations 🥺:
- The system is not an end-to-end congestion management system, nor does it enable one 🙅♀️.
- The system does not signal the presence of congestion to user applications or to all devices on the network path 📡.
- The system does not explicitly enable additional user and/or application responses to congestion 🤷♀️.
- The system does not enable distributed denial-of-service (DDoS) mitigation or other capabilities 💀.
11. Low Extra Delay Background Transport and Other Possibilities 🔮
Okay, so there are several new IETF working group efforts focused on congestion and its effects, avoiding congestion, managing congestion, and communicating congestion information 🤓. That includes the Congestion Exposure (CONEX) working group, the Application Layer Transport Optimization (ALTO) working group, and the Low Extra Delay Background Transport (LEDBAT) working group 🌈. If one or more of these groups actually produce useful work, the design or configuration of the system documented here might need to change 🔄. For example, this congestion management system doesn't currently have a way to account for different classes of data transfer, like the class LEDBAT may specify, which might yield to other traffic better than existing transport protocols do 🦋. CONEX may also specify methods for this system or others to signal congestion state or expected congestion to other parts of the network, and/or to the hosts on either end of a particular network flow 📶. And honestly? It's conceivable that current or future IETF work could make this whole congestion management system unnecessary 😳✨.
12. Security Considerations 🔐
This is SO important, bestie 🚨. It's crucial that an ISP secures access to the Congestion Management servers and the QoS Servers, as well as QoS signaling to the CMTSs, so that unauthorized users and/or hosts can't make unauthorized changes to QoS settings in the network 🙅♀️🔒.
Securing access to the Statistics Collection Server is also a must-do 😤. It holds IPDR-based byte transfer data that end users consider private on an individual basis 🕵️♀️. On an aggregate basis, that same data is considered ISP-proprietary traffic data 🏢. Access to that server should also be locked down so that false usage statistics can't be fed into the system 🚫. Here's the tea: IPDR data contains a count of bytes sent and bytes received, by cable modem MAC address, over a given interval of time 📊. It does not contain things like the source and/or destination Internet address of that data, and it doesn't contain the protocols used, the ports used, etc. 🙌
13. Acknowledgements 🫶
The authors wanted to shout out the hard work of the many people who helped develop and/or review this document, plus the people who helped deploy the system in such a short period of time 👏🌸.
The authors also wanted to acknowledge the following individuals for doing a detailed review of this document and/or providing comments and feedback that helped improve and evolve it 💖:
- Kris Bransom 🎀
- Bob Briscoe 🎀
- Lars Eggert 🎀
- Ari Keranen 🎀
- Tero Kivinen 🎀
- Matt Mathis 🎀
- Stanislav Shalunov 🎀
14. Informative References 📚
Here's the reading list, all the sources this doc points to, with every detail intact 🤓📖:
- [COMCAST_P2PI_PAPER] Livingood, J. and R. Woundy, "Comcast's IETF P2P Infrastructure Workshop Position Paper", FCC Filings Comcast Network Management Proceedings, May 2008,
http://trac.tools.ietf.org/area/rai/trac/raw-attachment/wiki/PeerToPeerInfrastructure/16%20ietf-p2pi-comcast-20080509.pdf. - [COMCAST_P2PI_PRES] Livingood, J. and R. Woundy, "Comcast's IETF P2P Infrastructure Workshop Presentation on May 28, 2008", FCC Filings Comcast Network Management Proceedings, May 2008,
http://trac.tools.ietf.org/area/rai/trac/raw-attachment/wiki/PeerToPeerInfrastructure/02-Comcast-IETF-P2Pi.pdf. - [DOCSIS_CM2CPE] CableLabs, "Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - Cable Modem to Customer Premise Equipment Interface Specification", DOCSIS 3.0 CM-SP-CMCIv3-I01-080320, March 2008,
http://www.cablelabs.com/cablemodem/specifications/specifications30.html. - [DOCSIS_IPDR] Yassini, R., "Data-Over-Cable Service Interface Specifications - DOCSIS 2.0 - Operations Support System Interface Specification", DOCSIS 2.0 CM-SP-OSSIv2.0-C01-081104, November 2008,
http://www.cablelabs.com/cablemodem/specifications/specifications30.html. - [DOCSIS_MULPI] CableLabs, "Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - MAC and Upper Layer Protocols Interface Specification", DOCSIS 3.0 CM-SP-MULPIv3.0-I11-091002, October 2009,
http://www.cablelabs.com/cablemodem/specifications/specifications30.html. - [DOCSIS_OSSI] CableLabs, "Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - Operations Support System Interface Specification", DOCSIS 3.0 CM-SP-OSSIv3.0-I10-091002, October 2009,
http://www.cablelabs.com/cablemodem/specifications/specifications30.html. - [DOCSIS_PHY] CableLabs, "Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - Physical Layer Specification", DOCSIS 3.0 CM-SP-PHYv3.0-I08-090121, January 2009,
http://www.cablelabs.com/cablemodem/specifications/specifications30.html. - [DOCSIS_SEC] CableLabs, "Data-Over-Cable Service Interface Specifications - DOCSIS 3.0 - Security Specification", DOCSIS 3.0 CM-SP-SECv3.0-I11-091002, March 2008,
http://www.cablelabs.com/cablemodem/specifications/specifications30.html. - [FCC_Congest_Mgmt_Ltr] Zachem, K., "Letter to the FCC Advising of Successful Deployment of Comcast's New Congestion Management System", FCC Filings Comcast Network Management Proceedings, January 2009,
http://fjallfoss.fcc.gov/ecfs/document/view?id=6520192582. - [FCC_Memo_Opinion] Martin, K., Copps, M., Adelstein, J., Tate, D., and R. McDowell, "FCC Memorandum and Opinion Regarding Reasonable Network Management", File No. EB-08-IH-1518 WC Docket No. 07-52, August 2008,
http://hraunfoss.fcc.gov/edocs_public/attachmatch/FCC-08-183A1.pdf. - [FCC_Net_Mgmt_Response] Zachem, K., "Letter to the FCC Regarding Comcast's Network Management Practices", FCC Filings Comcast Network Management Proceedings, September 2008,
http://fjallfoss.fcc.gov/ecfs/document/view?id=6520169715. - [IPDR_Standard] Cotton, S., Cockrell, B., Walls, P., and T. Givoly, "Network Data Management - Usage (NDM-U) For IP-Based Services. Service Specification - Cable Labs DOCSIS 2.0 SAMIS", IPDR Service Specifications NDM-U, November 2004,
http://www.ipdr.org/public/Service_Specifications/3.X/DOCSIS(R)3.5-A.0.pdf. - [PowerBoost_Specification] Comcast Cable Communications Management LLC, "Comcast PowerBoost Specification", Website Comcast.com, June 2010,
http://customer.comcast.com/Pages/FAQListViewer.aspx?topic=Internet&folder=8b2fc392-4cde-4750-ba34-051cd5feacf0. - [RFC1633] Braden, B., Clark, D., and S. Shenker, "Integrated Services in the Internet Architecture: an Overview", RFC 1633, June 1994.
- [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998.
- [RFC3083] Woundy, R., "Baseline Privacy Interface Management Information Base for DOCSIS Compliant Cable Modems and Cable Modem Termination Systems", RFC 3083, March 2001.
- [RFC3360] Floyd, S., "Inappropriate TCP Resets Considered Harmful", BCP 60, RFC 3360, August 2002.
- [RFC3410] Case, J., Mundy, R., Partain, D., and B. Stewart, "Introduction and Applicability Statements for Internet-Standard Management Framework", RFC 3410, December 2002.
- [RFC5594] Peterson, J. and A. Cooper, "Report from the IETF Workshop on Peer-to-Peer (P2P) Infrastructure, May 28, 2008", RFC 5594, July 2009.
Authors' Addresses 🎀
Here are the people behind this doc, their deets, verbatim 💅:
Chris Bastian
Comcast Cable Communications
One Comcast Center
1701 John F. Kennedy Boulevard
Philadelphia, PA 19103
US
EMail: chris_bastian@cable.comcast.com
URI: http://www.comcast.com
Tom Klieber
Comcast Cable Communications
1306 Goshen Parkway
West Chester, PA 19380
US
EMail: tom_klieber@cable.comcast.com
URI: http://www.comcast.com
Jason Livingood
Comcast Cable Communications
One Comcast Center
1701 John F. Kennedy Boulevard
Philadelphia, PA 19103
US
EMail: jason_livingood@cable.comcast.com
URI: http://www.comcast.com
Jim Mills
Comcast Cable Communications
One Comcast Center
1800 Bishops Gate Drive
Mount Laurel, NJ 08054
US
EMail: jim_mills@cable.comcast.com
URI: http://www.comcast.com
Richard Woundy
Comcast Cable Communications
27 Industrial Avenue
Chelmsford, MA 01824
US
EMail: richard_woundy@cable.comcast.com
URI: http://www.comcast.com📦 where this page comes from
- ietf: rfc6057 2010-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms