RFC 6983
Models for HTTP-Adaptive-Streaming-Aware Content Distribution Network Interconnection (CDNI)
RFC 6983: Models for HTTP-Adaptive-Streaming-Aware Content Distribution Network Interconnection (CDNI) 🎀
okay so before anything else, here's the paperwork tea ☕
- RFC number and title: RFC 6983, Models for HTTP-Adaptive-Streaming-Aware Content Distribution Network Interconnection (CDNI)
- Authors: R. van Brandenburg and O. van Deventer (both TNO), F. Le Faucheur and K. Leung (all four are listed as Cisco Systems, except the TNO pair 💅)
- Stream and category: Independent Submission, Informational
- Date: July 2013
- ISSN: 2070-1721
- Obsoletes and Updates: the header has no Obsoletes or Updates line, so this doc doesn't replace or update any earlier RFC 🙈
Abstract 🦋
so this doc is the authors sharing their thoughts on what happens when HTTP Adaptive Streaming (HAS) meets Content Distribution Network Interconnection (CDNI) 🎬 they break down the CDNI-HAS problem space and walk through the different options they came up with, plus the ones other people threw out during informal chats, for dealing with HAS in a CDNI setup. it was used as input while the CDNI working group was deciding whether or not to support HAS at all 💜
Status of This Memo 📝
bestie, this is not an Internet Standards Track spec. it's just published for info, so no vibes of "you must implement this" here ✨
this is a contribution to the RFC Series, independent of any other RFC stream. the RFC Editor picked to publish it at their discretion and makes no statement about how useful it is for implementation or deployment. docs the RFC Editor approves aren't a candidate for any level of Internet Standard, see Section 2 of RFC 5741 for the deets 💅
want the current status, any errata, or a way to give feedback? check http://www.rfc-editor.org/info/rfc6983 🌸
Copyright Notice 🔥
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. 👑
this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stood on the publication date. please actually read these, they spell out your rights and restrictions for this doc, no cap 🫶
Table of Contents 💖
1. Introduction ....................................................4
1.1. Terminology ................................................5
2. HTTP Adaptive Streaming Aspects Relevant to CDNI ................6
2.1. Segmentation versus Fragmentation ..........................6
2.2. Addressing Chunks ..........................................7
2.2.1. Relative URLs .......................................8
2.2.2. Absolute URLs with Redirection ......................9
2.2.3. Absolute URLs without Redirection ..................10
2.3. Live Content versus VoD Content ...........................11
2.4. Stream Splicing ...........................................12
3. Possible HAS Optimizations .....................................12
3.1. File Management and Content Collections ...................13
3.1.1. General Remarks ....................................13
3.1.2. Candidate Approaches ...............................13
3.1.2.1. Option 1.1: Do Nothing ....................13
3.1.2.2. Option 1.2: Allow Single-File
Storage of Fragmented Content .............14
3.1.2.3. Option 1.3: Access Correlation Hint .......14
3.1.3. Recommendations ....................................15
3.2. Content Acquisition of Content Collections ................15
3.2.1. General Remarks ....................................15
3.2.2. Candidate Approaches ...............................16
3.2.2.1. Option 2.1: No HAS Awareness ..............16
3.2.2.2. Option 2.2: Allow Single-File
Acquisition of Fragmented Content .........17
3.2.3. Recommendations ....................................17
3.3. Request Routing of HAS Content ............................17
3.3.1. General Remarks ....................................17
3.3.2. Candidate Approaches ...............................18
3.3.2.1. Option 3.1: No HAS Awareness ..............18
3.3.2.2. Option 3.2: Manifest File Rewriting
by uCDN ...................................20
3.3.2.3. Option 3.3: Two-Step Manifest File
Rewriting .................................21
3.3.3. Recommendations ....................................22
3.4. Logging ...................................................23
3.4.1. General Remarks ....................................23
3.4.2. Candidate Approaches ...............................24
3.4.2.1. Option 4.1: Do Nothing ....................24
3.4.2.2. Option 4.2: CDNI Metadata Content
Collection ID .............................26
3.4.2.3. Option 4.3: CDNI Logging Interface
Compression ...............................28
3.4.2.4. Option 4.4: Full HAS
Awareness/Per-Session Logs ................29
3.4.3. Recommendations ....................................30
3.5. URL Signing ...............................................32
3.5.1. HAS Implications ...................................32
3.5.2. CDNI Considerations ................................33
3.5.3. Option 5.1: Do Nothing .............................34
3.5.4. Option 5.2: Flexible URL Signing by CSP ............34
3.5.5. Option 5.3: Flexible URL Signing by uCDN ...........37
3.5.6. Option 5.4: Authorization Group ID and HTTP
Cookie .............................................37
3.5.7. Option 5.5: HAS Awareness with HTTP Cookie in CDN ..38
3.5.8. Option 5.6: HAS Awareness with Manifest
File in CDN ........................................40
3.5.9. Recommendations ....................................41
3.6. Content Purge .............................................41
3.6.1. Option 6.1: No HAS Awareness .......................42
3.6.2. Option 6.2: Purge Identifiers ......................42
3.6.3. Recommendations ....................................43
3.7. Other Issues ..............................................43
4. Security Considerations ........................................43
5. Acknowledgements ...............................................44
6. References .....................................................44
6.1. Normative References ......................................44
6.2. Informative References ....................................441. Introduction ✨
okay so [RFC6707] is the doc that lays out the problem space for Content Distribution Network Interconnection (CDNI) and its interfaces 🧚 it covers getting content to End Users through interconnected Content Delivery Networks (CDNs), using both HTTP progressive download and HTTP Adaptive Streaming (HAS) 💖
HAS is basically an umbrella term for a bunch of HTTP-based streaming techs that let a client hop between multiple bitrates depending on what the network is doing right now 📶 the big thing to know is that because it runs over HTTP, it's pull-based: the client is the one actively requesting content segments, not the server pushing stuff at it. that pull-based energy means media servers doing HAS often act different from ones doing traditional streaming, like Real-time Transport Protocol / Real Time Streaming Protocol (RTP/RTSP), Real Time Messaging Protocol (RTMP), and Multimedia Messaging Service (MMS) 🌈
so this doc talks about how the HAS operation affects the CDNI interfaces, and which HAS-specific optimizations might be needed or just nice to have 🦋
the scope is the authors' analysis of the CDNI-HAS problem space, plus the different options they and others (in informal chats) came up with for handling HAS in CDNI 💅 the doc ends with the authors' recommendations for how the CDNI WG should handle HAS in its initial charter. the focus is on "making it work" rather than adding "nice-to-have" optimizations that could slow down the CDNI WG deliverables listed in that initial charter 🎀
one thing to note: this isn't a WG document. it was used as input during the WG process when they decided whether to support HAS 📝 the authors hope the analysis stays useful later, if and when the WG recharters and wants to revisit how many HAS optimizations to support in CDNI scenarios 🔥
1.1. Terminology 📚
this doc uses the terminology from [RFC6707] and [CDNI-FRAMEWORK] 💜
for convenience, here are the HAS-related term definitions, restated so you don't have to go hunting:
- Content Item: a uniquely addressable content element in a CDN. it's defined by having its own Content Metadata attached. examples are a video file/stream, an audio file/stream, or an image file 🎬
- Chunk: a fixed-length element that comes out of a segmentation or fragmentation operation, and that can be addressed independently 🍓
- Fragment: a specific form of chunk (see Section 2.1). a fragment is stored as part of a larger file that holds all the chunks in the chunk collection 🧩
- Segment: a specific form of chunk (see Section 2.1). a segment is stored as a single file from the file-system point of view 📁
- Original Content: non-chunked content that's the starting point for a segmentation or fragmentation operation. from it, you can derive multiple alternative representations (using different encoding methods, supporting different resolutions, and/or targeting different bitrates), and each of those may be fragmented or segmented 🌸
- Chunk Collection: the set of all chunks produced by one segmentation or fragmentation operation on one representation of the Original Content. a chunk collection is described in a Manifest File 🎀
- Content Collection: the set of all chunk collections derived from the same Original Content. a content collection can have multiple chunk collections, each one matching a single representation of the Original Content. it may be described by one or more Manifest Files ✨
- Manifest File: also called a Media Presentation Description (MPD) file. it lists how the content was chunked (possibly for multiple encodings), plus where the chunks live (for segments) or how they can be addressed (for fragments) 🧚
2. HTTP Adaptive Streaming Aspects Relevant to CDNI ✨
okay so, over the last couple of years a whole bunch of HAS-like protocols have shown up 🎀. Some are proprietary, like Apple's HTTP Live Streaming (HLS), Microsoft's HTTP Smooth Streaming (HSS), and Adobe's HTTP Dynamic Streaming (HDS). Others are standardized, like 3GPP Adaptive HTTP Streaming (AHS) and MPEG Dynamic Adaptive Streaming over HTTP (DASH) 💜. They all share a core set of features, but each one has its own signature moves. This section looks at the shared stuff, the differences, and how all of it matters for CDNI 🦋. Section 2.1 covers the ways HAS content gets stored, and Section 2.2 lists three methods used to address HAS content in a CDN. After those generic basics, there are two special situations to know about: Section 2.3 is about live content vs Video on Demand (VoD) content, and Section 2.4 is about multiple streams getting combined into a single Manifest File (e.g., for ad insertion purposes) 📺.
2.1. Segmentation versus Fragmentation
so every HAS setup is built on something called "chunking" 🧩. That's when a server splits content into a bunch of fixed-duration chunks that can each be decoded on their own. A client grabs the chunks one after another, and that's how it rebuilds and plays the content. The best part is that a client can smoothly switch between different encodings of the same Original Content right at a chunk boundary ✨. Before it asks for a chunk, it can pick from multiple alternative encodings of that same chunk, no matter which encodings it grabbed earlier. Iconic behavior, honestly 💅.
Every HAS implementation uses some form of chunking, but they don't all store the chunks the same way 🙈. Generally speaking, there are two distinct ways to chunk and store things: segmentation and fragmentation.
- With segmentation -- which is, for example, mandatory in all versions of Apple's HLS prior to version 7 -- the chunks (called segments in this case) are stored completely independently of each other. Each segment is its own separate file from the file system's point of view. That means every segment has its own unique URL it can be retrieved from 🍓.
- With fragmentation (or virtual segmentation) -- which is, for example, used in Microsoft's HTTP Smooth Streaming -- all the chunks (fragments), belonging to the same chunk collection are stored together as part of a single file. There are a bunch of container formats that can store this kind of chunked content, but fragmented MP4 is the one that gets used the most. With fragmentation, you address a specific chunk by adding, to the common file URL, an identifier that uniquely points to the chunk you want. That identifier can be a timestamp, a byte range, or something else entirely 🧚.
Ngl, people can argue about which method is better all day 💀. Both have pros and cons in a CDN environment. Fragmentation is often seen as the method with less overhead, both for storage and for processing. Segmentation, on the other hand, is seen as more flexible and easier to cache 🌸. In practice though, current HAS implementations increasingly support both methods 🌈.
2.2. Addressing Chunks
so a client can only request chunks (segments or fragments) if it knows how the content was chunked and where to find those chunks 🗺️. For this, most HAS protocols use something called a Manifest File (also known as a Media Presentation Description, or MPD). It's basically a file listing how the content was chunked, plus where the various chunks live (for segments) or how to address them (for fragments). A Manifest File, or a set of them, can also identify the different representations, and therefore the chunk collections, that are available for the content 📜.
In general, a HAS client first requests and receives a Manifest File. Then, after parsing it, the client goes through the chunks listed in there one after another 🏃♀️. Each HAS implementation has its own Manifest File format, and even inside one particular format there are different ways to specify where a chunk is located 💜.
Managing where files live is a core part of every CDN, and each CDN handles it its own way 🏗️. Some CDNs are purely cache-based, with no higher-level knowledge of where each file sits at any given moment. Other CDNs have dedicated management nodes that do know, at each instant in time, which servers each file is on. The CDNI interfaces designed by the CDNI WG will probably need to stay agnostic to these CDN-internal architecture decisions 🫶. With HAS, though, there's a strict relationship between where the content sits in the CDN (the chunks) and the content itself (the locations written in the Manifest File). So it helps to understand the different methods CDNs use today for specifying chunk locations in Manifest Files. These methods are described in Sections 2.2.1 to 2.2.3 👑.
Segmented content is the main reason these sections matter, since it's inherently distributed. But the methods also apply to fragmented content 🦋. It's also worth knowing that the methods below for specifying locations of content items in Manifest Files aren't only about time-based chunking (segments and fragments). They're also relevant when content is made available in multiple representations (e.g., different qualities, encoding methods, resolutions, and/or bitrates) 🌈. In that case, the content is made up of multiple chunk collections, and those might be described by one single Manifest File or by multiple interrelated Manifest Files. With multiple files, there can be a high-level Manifest File describing the available bitrates, with URLs pointing to separate Manifest Files that describe each specific bitrate. To specify the locations of those other Manifest Files, the same methods used for chunk locations apply too ✨.
One last thing: who delivers the Manifest Files themselves 📦. In most situations, the CDN handles delivery of both the Manifest File and the chunks. But you can imagine scenarios where the Manifest File is delivered by, for example, the Content Service Provider (CSP). In that case the Manifest File isn't visible to the CDN at all 👀.
2.2.1. Relative URLs
One way to specify chunk locations in a Manifest File is with relative URLs 🌷. A relative URL doesn't include the HOST part of a URL. It only includes (part of) the PATH part. In practice, the client treats a relative URL as relative to the location the Manifest File was acquired from. So a relative URL is a string that gets appended to the Manifest File's location to get the location of a specific chunk 🧭. That means that when a Manifest File with relative URLs is used, every chunk is delivered by the same Surrogate that delivered the Manifest File. Which is why a relative URL never includes a hostname 🙅♀️.
Here's an example. Say a Manifest File has been requested (and received) from:
http://surrogate.server.cdn.example.com/content_1/manifest.xmlA relative URL pointing to a specific segment referenced in that Manifest File might be:
segments/segment1_1.tsThe client takes the Manifest File's location and appends the relative URL. So the segment would then be requested from http://surrogate.server.cdn.example.com/content_1/segments/segment1_1.ts 💖.
Relative URLs have a couple of drawbacks, though 😭. First, they force a CDN that relies on HTTP-based request routing to deliver all the segments of a given content item from the same Surrogate that delivered that content item's Manifest File. That's pretty limited in flexibility. Second, relative URLs don't allow for fallback URLs. If the Surrogate that delivered the Manifest File breaks down, the client can't request chunks anymore 💀. The upside is that it's super easy to move content between different Surrogates and even between CDNs 🎀.
2.2.2. Absolute URLs with Redirection
Another way to specify locations of chunks (or other Manifest Files) in a Manifest File is with absolute URLs 🔗. An absolute URL contains a fully formed URL. The client doesn't have to calculate anything the way it does with relative URLs. It can just use the URL from the Manifest File directly ✨.
For Manifest Files, there are two kinds of absolute URLs you can imagine: absolute URLs with redirection, and absolute URLs without redirection 🧐. The difference is whether the URL points to a request routing node that redirects the client to a Surrogate (absolute URLs with redirection), or points directly at a Surrogate that hosts the requested content (absolute URLs without redirection).
With absolute URLs with redirection, a chunk request gets handled by the CDN's Request Routing system, just like a standalone (non-HAS) content request would. That can include looking up the Surrogate (and/or CDN) that's best suited to deliver the requested chunk to that particular user. Then it either sends an HTTP redirect to the user, pointing at the requested chunk on the chosen Surrogate (and/or CDN), or it sends back a DNS response pointing to the specific Surrogate 🛰️.
Here's what an absolute URL with redirection might look like:
http://requestrouting.cdn.example.com/content_request?content=content_1&segment=segment1_1.tsAs you can see from that example, the URL has a pointer to a general CDN Request Routing function, plus some arguments that identify the requested segment 🔍.
The big advantage of absolute URLs with redirection is maximum flexibility. Chunks can be spread across Surrogates and CDNs in any imaginable way, and the Manifest File doesn't have to be changed every time one or more chunks move (which is what happens with absolute URLs without redirection) 🦋. The downside is that it can pile significant load onto a CDN Request Routing system, because it has to do a redirect every single time a client requests a new chunk 😩.
2.2.3. Absolute URLs without Redirection ✨
Okay so with absolute URLs without redirection 💅 the URL goes straight to the exact chunk on the actual Surrogate that's gonna serve it to the client. There's no HTTP redirect happening between the client asking for the chunk and the Surrogate delivering it. Iconic, right? 🦋
Here's an example:
http://surrogate1.cdn.example.com/content_1/segments/segment1_1.tsAs you can see, this URL has both the identifier of the segment you want (segment1_1.ts) and the server that's supposed to deliver it (surrogate1.cdn.example.com). So the client has everything it needs to go request that segment straight from that Surrogate. No detours 🧚
The big win is that this gives you more flexibility than relative URLs, because segments don't have to come from the same server. And it does this without needing a redirect for every single segment, which would pile a ton of load onto whatever node is doing the redirecting 😭 The catch? Every time content gets moved somewhere else (whether that's within one CDN or across CDNs), the Manifest File has to be modified to match. Not gonna lie, that's a lot of admin 🙈
2.3. Live Content versus VoD Content
Okay, the actual formats and addresses of Manifest Files and chunk files usually don't look too different between live and Video-on-Demand (VoD) content. The real difference is when they show up. With live content, chunk files and their Manifest Files get created and delivered in real time, and that creates a few potential issues for HAS optimization 👑
- With live content, chunk files are made available in real time, so bundling for content acquisition purposes doesn't really work well. Pre-positioning can still happen, but if there's a lot of latency in it, it can lose its value. That's because a client might request the chunk before the pre-positioning finishes, or the pre-positioning request might only get serviced after the chunk's playout time has already passed 😬
- In live content, Manifest Files have to be updated for each chunk, which means the client has to retrieve them before every chunk request. So any optimization schemes based on Manifest Files need to be ready to optimize on a per-segment request basis. Manifest Files might also get polled several times before the next chunk is actually available 🔁
- Since live Manifest Files keep getting updated as new chunks show up, they're not super cacheable. Timestamping and reasonable Time-to-Live (TTL) settings can help performance, but timely replication and delivery of updated Manifest Files is critical for keeping playback uninterrupted 🎀
- Manifest Files are usually updated after the matching chunk is available for delivery, so nobody requests chunks that don't exist yet. HAS optimization approaches that use dynamic Manifest File generation need to be synchronized with chunk creation, or playback errors can happen 💀
2.4. Stream Splicing
Stream splicing is how you make media mashups by combining content from multiple sources. A super common example is advertisement insertion, for both VoD and live streams 📺 The thing is, some content lives outside the CDNs entirely. So Manifest Files with absolute URLs with redirection might contain chunk or nested Manifest File URLs pointing to content that isn't delivered through any of the interconnected CDNs. Plot twist 🌀
Also, client and downstream proxy devices might rely on non-URL info in the Manifest File (like comments or custom tags) to do the stream splicing. That often happens outside the scope of the interconnected CDNs, so it's a whole separate situation 🧩 HAS optimization schemes that do dynamic Manifest File generation or rewriting have to be super careful with chunk URLs, nested Manifest File URLs, and any other metadata that shouldn't be changed or removed. If you mess with those, you can cause playback interruptions. And if ads don't get played, CSPs can lose revenue, which is a big no 💸
3. Possible HAS Optimizations
In the last section, we went over some of the unique properties of HAS and some of the CDN-specific design decisions for addressing chunks. Now we're getting into how supporting HAS in CDNI scenarios actually affects things 🌸
There are a bunch of topics, or problem areas, that matter a lot when you combine HAS and CDNI. For each one, there are several different ways the CDNI interfaces can handle it. Generally speaking, each problem area can be solved in one of two ways: either you make the fewest HAS-specific changes to the CDNI interfaces, or you maximize how flexible and efficient the CDNI interfaces are at delivering HAS content. The goal for the CDNI WG should probably be to find the middle ground between those two extremes, and to come up with solutions that balance efficiency against added complexity. Balance is everything, bestie ⚖️
So the WG can make that call, this section briefly describes each of the following problem areas, plus a few different options for dealing with them. Section 3.1 looks at how to handle file management for groups of files, also known as content collections. Section 3.2 is a related topic: how to do content acquisition of content collections between the Upstream CDN (uCDN) and the Downstream CDN (dCDN). After that, Section 3.3 covers the options for request routing of HAS content, especially around Manifest Files. Section 3.4 talks about possible optimizations for logging HAS content, and Section 3.5 is about the options for URL signing. Last but not least, Section 3.6 describes different scenarios for removing HAS content from CDNs 🧁
3.1. File Management and Content Collections
3.1.1. General Remarks
HAS content is special because it doesn't consist of one single file or stream. It's made up of multiple interrelated files (segments, fragments, and/or Manifest Files). In this document, that whole group of files is called a content collection 🗂️ Another key thing is the difference between segments and fragments (see Section 2.1). It's a big deal!
No matter whether you use segments or fragments, different CDNs might handle content collections differently when it comes to file management. For example, some CDNs treat every file in a content collection as an individual file, stored separately from the others. The perk of that is it makes caching individual chunks super easy 🍓 Other CDNs store all the fragments in a content collection in one bundle, as if they were a single file (like using a fragmented MP4 container). The upside there is less file management overhead.
The next subsections look at the different ways the CDNI interfaces might deal with these differences in content collection handling. The options can be sorted by how much HAS awareness they need from the CDNs and the CDNI interfaces 🧠
3.1.2. Candidate Approaches
3.1.2.1. Option 1.1: Do Nothing 🙅♀️
This option assumes zero HAS awareness in both the involved CDNs and the CDNI interfaces. The uCDN uses individual files, and the dCDN isn't explicitly told about the relationship between chunks and doesn't know which files belong to the same content collection. So basically, the file management method the uCDN uses just gets imposed on the dCDN too 🤷♀️
This also means the dCDN can't use any form of file bundling, like the single-file mechanism that stores fragmented content as one file (see Section 2.1). The one exception is when the content is fragmented and the Manifest Files on the uCDN contain byte range requests. In that case, the dCDN might be able to acquire the fragmented content as a single file (see Section 3.2.2.2).
Effect on CDNI interfaces:
- None
Advantages/Drawbacks:
- ➕ No HAS awareness needed in the CDNs, and no changes to the CDNI interfaces needed
- ➖ The dCDN is forced to store chunks as individual files
3.1.2.2. Option 1.2: Allow Single-File Storage of Fragmented Content 📦
Sometimes the dCDN would rather store fragmented content as a single file on its Surrogates, to cut down on file management overhead. To do that, it has to either acquire the content as a single file (see Section 3.2.2.2) or merge the different chunks together and put them in the same container (e.g., fragmented MP4). The catch is that this means the dCDN has to be fully HAS aware, which is a lot 😤
Effect on CDNI interfaces:
- CDNI Metadata interface: Add fields for indicating the particular type of HAS (e.g., MPEG DASH or HLS) that is used and whether segments or fragments are used
- CDNI Metadata interface: Add field for indicating the name and type of the Manifest File(s)
Advantages/Drawbacks:
- ➕ Lets the dCDN store fragmented content as a single file, reducing file management overhead
- ➖ Complex operation, requiring the dCDN to be fully HAS aware
3.1.2.3. Option 1.3: Access Correlation Hint 🪄
This one's kind of a middle ground between the two extremes from the previous two sections. It uses an 'Access Correlation Hint'. The hint gets added to the CDNI Metadata of all chunks in a particular content collection, and it says those files are likely to be requested within a short time window of each other. That helps the dCDN do local file storage optimizations for VoD items (e.g., by bundling all files with the same Access Correlation Hint value into a single bundle/file). The result is fewer files to manage, and no HAS awareness required ✨
Effect on CDNI interfaces:
- CDNI Metadata interface: Add field for indicating Access Correlation Hint
Advantages/Drawbacks:
- ➕ Allows the dCDN to perform file management optimization
- ➕ Does not require any HAS awareness
- ➕ Very small impact on CDNI interfaces
- ➖ Expected benefit compared with Option 1.1 is small
3.1.3. Recommendations
Looking at the pros and cons, the authors recommend that the WG go for Option 1.1 (do nothing) 🙌 The likely benefits of Option 1.3 aren't believed to be significant enough to justify changing the CDNI Metadata interface. Option 1.2 would bring definite benefits for HAS-aware dCDNs, but going for it would require significant CDNI extensions that would hit the WG's milestones. So the authors don't recommend including it in the current work. Instead, they mark it as a possible candidate for rechartering once the initial CDNI solution is done 🗓️
3.2. Content Acquisition of Content Collections
3.2.1. General Remarks ✨
okay so the last section was about file management and HAS in a CDNI setup 💅 this one is about a related topic: content acquisition between two CDNs 🦋
with acquisition, it's giving "dynamic vs pre-positioning" energy 💖 a dynamic CDN only requests a content item when a cache miss happens 🌸 a pre-positioning CDN proactively places content items on the nodes where it expects traffic for that item 🧚 both can benefit from being HAS aware 🔥 for dynamic acquisition, being HAS aware means that after a cache miss for a given chunk, the node might not only grab the requested chunk but also grab some related chunks that are expected to be requested soon 👑 pre-positioning gets similar benefits ✨
3.2.2. Candidate Approaches
3.2.2.1. Option 2.1: No HAS Awareness
this option assumes no HAS awareness in both CDNs and on the CDNI interfaces 🙈 just like Option 1.1 from the file management section, no HAS awareness means the dCDN doesn't know how chunks relate to each other 💀 so every single file in a content collection has to be individually acquired from the uCDN by the dCDN 😭
the exception is fragmented content where the uCDN uses Manifest Files containing byte range requests 🎀 in that case the dCDN can just drop the byte range identifier and acquire the complete file ✨
the advantage? it's highly flexible 💅 if a client only requests a small portion of the chunks in a content collection, the dCDN only has to grab those chunks from the uCDN, which saves both bandwidth and storage 🍓
the downside is that per-chunk acquisition creates more transaction overhead between the dCDN and uCDN, compared to a method where entire content collections can be acquired in one transaction 😩
Effect on CDNI interfaces:
- None
Advantages/Drawbacks:
+ Per-chunk content acquisition allows for a high level of flexibility between the dCDN and uCDN 💖
- Per-chunk content acquisition creates more transaction overhead between the dCDN and uCDN 😤
3.2.2.2. Option 2.2: Allow Single-File Acquisition of Fragmented Content
as we covered in Section 3.2.2.1, there's one (pretty rare) case where fragmented content can be acquired as a single file with zero HAS awareness 🦄 that's when the content is fragmented and the Manifest File specifies byte range requests 🙌 this section is about how to do single-file acquisition in the other (very common) cases 🌈 to do that, the dCDN would need full HAS awareness, at least enough to map between a single file and the individual chunks it needs to serve 🧠
Effect on CDNI interfaces:
- CDNI Metadata interface: Add fields for indicating the particular type of HAS (e.g., MPEG DASH or HLS) that is used and whether segments or fragments are used
- CDNI Metadata interface: Add field for indicating the name and type of the Manifest File(s)
Advantages/Drawbacks:
+ Allows for more efficient content acquisition in all HAS-specific supported forms 🔥
- Requires full HAS awareness on the part of the dCDN 😬
- Requires significant CDNI Metadata interface extensions 🥲
3.2.3. Recommendations
okay so based on the pros and cons, the authors recommend that the WG go for Option 2.1, since it's enough to 'make HAS work' 🫶 Option 2.2 would be great for acquiring large content collections 🙌 but it needs big CDNI extensions that would mess with the WG's milestones 😭 so Option 2.2 might be a candidate for possible rechartering once the initial CDNI solution is done 🦋
3.3. Request Routing of HAS Content
3.3.1. General Remarks
this section is about how HAS content affects request routing 🧚 the big thing to watch here is the different types of Manifest Files that might be used 💜 in Section 2.2 we went over three methods for identifying and addressing chunks inside a Manifest File: relative URLs, absolute URLs with redirection, and absolute URLs without redirection 🌸 not every CDN will use or support all three 🙈 some CDNs might only use one, while others might support two or all three 💅
a super important factor in choosing a chunk-addressing method is the CSP 👑 some CSPs have a strong preference for a particular method and deliver Manifest Files to the CDN in a specific way 🎀 depending on the CDN and its agreement with the CSP, a CDN might either host the Manifest Files exactly as the CSP created them, or modify them to fit its architecture 🛠️ for example, by changing relative URLs into absolute URLs that point to the CDN Request Routing function ✨
3.3.2. Candidate Approaches
3.3.2.1. Option 3.1: No HAS Awareness
this option assumes no HAS awareness in both CDNs and on the CDNI interfaces 🙈 it also assumes that neither the dCDN nor the uCDN can actively manipulate Manifest Files 🚫 just like with file management and content acquisition, no HAS awareness means each file in a content collection is handled individually, and the dCDN doesn't know about any relationships between files 💀
the only chunk-addressing method that works without question here is absolute URLs with redirection 💖 in other words, the CSP that ingested the content into the uCDN created a Manifest File where each chunk location points to the Request Routing function of the uCDN 🦋 or the CSP may have ingested a Manifest File with relative URLs, and the uCDN ingestion function translated them into absolute URLs pointing to the Request Routing function 🔥
in this "absolute URL with redirection" case, the uCDN can just have the dCDN deliver the Manifest File like it's any regular file 📄 once the client parses it, it requests any later chunks from the uCDN Request Routing function 🧚 that function can then decide to outsource delivery of those chunks to the dCDN 👑 depending on whether HTTP-based (recursive or iterative) or DNS-based request routing is used, the uCDN Request Routing function will either directly or indirectly redirect the client to the Request Routing function of the dCDN (assuming it doesn't have the info it needs to send the client straight to a Surrogate in the dCDN) 🌈
the drawback is that this creates a LOT of request routing overhead for both the uCDN and dCDN 😭 for every chunk, the full inter-CDN Request Routing process gets invoked (which can mean two HTTP redirections in the iterative case, or one HTTP redirection plus one CDNI Request Routing Redirection interface request/response) 💀 even with DNS-based redirection, there might be significant overhead, since both Request Routing functions might have to do database lookups and query each other 🥲 DNS's inherent caching could cut that down, but it would seriously hurt how accurate the redirect is 🙃
with no HAS awareness, relative URLs might or might not work, depending on the type of relative URL being used 🤷♀️ when a uCDN hands off delivery of a Manifest File with relative URLs to a dCDN, the client goes straight to the dCDN Surrogate it got the Manifest File from for every later chunk 💖 as long as the relative URL is not path-absolute (see [RFC3986]), this approach works fine ✨
since absolute URLs without redirection inherently need a HAS-aware CDN, they can't be used in this case 😩 the URLs in the Manifest File would point directly to a Surrogate in the uCDN 🙅♀️ and since this scenario assumes no HAS awareness on the part of the dCDN or uCDN, neither of them can rewrite the Manifest File to let the client go to a Surrogate in the dCDN or to a Request Routing function 💔
Effect on CDNI interfaces:
- None
Advantages/Drawbacks:
+ Supports absolute URLs with redirection 💅
+ Supports relative URLs 🫶
+ Does not require HAS awareness and/or changes to the CDNI interfaces 🎉
- Not possible to use absolute URLs without redirection 😭
- Creates significant signaling overhead in cases where absolute URLs with redirection are used (inter-CDN request redirection for each chunk) 💀
3.3.2.2. Option 3.2: Manifest File Rewriting by uCDN ✨
okay so, Option 3.1 does let you use absolute URLs with redirection 🙃 but the way it works makes a ton of request routing overhead for both the dCDN and the uCDN 😭 so this option is the glow-up that cuts that way down 💅
here's the tea: the uCDN can rewrite the Manifest File (or generate a brand new one) so it removes itself from the request routing chain for the chunks listed in that Manifest File 🦋 like we said in Section 3.3.2.1, with no HAS awareness the client goes to the uCDN Request Routing function for every chunk request, and that function then redirects the client to the dCDN Request Routing function. by rewriting the Manifest File (or making a new one), the uCDN skips that first step and the Manifest File points straight at the dCDN Request Routing function 🎀
the best part? this doesn't really touch the CDNI interfaces at all 🫶 it's a CDN-internal thing the uCDN can do on its own, using info it has from regular CDNI setup or info it gets from the dCDN during normal communication over the CDNI Request Routing Redirection interface 🌸
so to rewrite the Manifest File, the bare minimum the uCDN needs is the location of the dCDN Request Routing function (or, alternatively, the location of the dCDN delivering Surrogate) 💖 it can come from configuration, or it can be derived from the regular CDNI Request Routing Redirection interface 💜 for example, the uCDN might ask the dCDN for the location of its request routing node (through the CDNI Request Routing Redirection interface) every single time a request for a Manifest File gets handled by the uCDN Request Routing function 👀 then the uCDN edits the Manifest File and hands it to the client ✨ one big perk of this method is that it's super efficient and flexible, because the dCDN can optionally reply with the locations of its Surrogates instead of its Request Routing function 🧚 (which basically turns the URLs into absolute URLs without redirection, iconic 👑)
there are also a bunch of variations, like only modifying the Manifest File once (or once per period of time) by the uCDN Request Routing function, which happens when the first client for that particular content collection (and redirected to that particular dCDN) sends a Manifest File request 🍓 the upside is the uCDN only has to edit the Manifest File once (or once per time period) 🙌 the downside is the dCDN is no longer in a position to influence the request routing decision across individual content requests 😬
now, it should be noted that there are a bunch of things to think about when changing a Manifest File (see, for example, Sections 2.3 and 2.4 on live HAS content and ad insertion) 🙈 also, some CSPs might have issues with a CDN changing their Manifest Files 💀 BUT in this option only the uCDN is messing with the Manifest File, and the uCDN can be expected to know these limitations if it wants to do Manifest File manipulation, since it's in its own best interest that its customer's content gets delivered the right way 💅 plus there's a direct commercial and technical relationship between the uCDN (the Authoritative CDN in this scenario) and its customer (the CSP) 🤝 and if the CSP wants to limit Manifest File manipulation, they can just work it out with the uCDN bilaterally 🫶
Effect on CDNI interfaces and the pros/cons, in full:
Effect on CDNI interfaces:
o None
Advantages/Drawbacks:
+ Possible to significantly decrease signaling overhead when using
absolute URLs
+ (Optional) Possible to have the uCDN rewrite the Manifest File
with locations of Surrogates in the dCDN (turning absolute URLs
with redirection into absolute URLs without redirection)
+ No changes to CDNI interfaces
+ Does not require HAS awareness in the dCDN
- Requires a high level of HAS awareness in the uCDN (for modifying
Manifest Files)3.3.2.3. Option 3.3: Two-Step Manifest File Rewriting
okay so, with Option 3.2 one of the possibilities is letting the dCDN hand the uCDN the locations of a specific Surrogate 🌈 then the uCDN can fill the Manifest File with absolute URLs without redirection, and the client grabs chunks straight from a dCDN Surrogate 🚀 BUT some dCDNs might not be down to share that info with the uCDN 😤 in that case they can only give the uCDN the location of their Request Routing function, which means no absolute URLs without redirection 🙅♀️
one way to fix that is two-step Manifest File manipulation 🧩 step one: the uCDN makes its own edit and puts the locations of the dCDN Request Routing function in the Manifest File 🎯 step two: once a request for the Manifest File hits the dCDN Request Routing function, it does a second edit where it swaps the URLs in the Manifest Files for URLs of its own Surrogates 💜 this way the dCDN still gets limited request routing traffic WITHOUT having to share sensitive Surrogate info with the uCDN 🔒
the catch? this approach not only assumes HAS awareness in the dCDN, it also needs some HAS-specific additions to the CDNI Metadata interface 🥺 for the dCDN to change the Manifest File, it needs to know the structure of the content 🧠 specifically, it needs to know which chunks make up the content collection 📦
Effect on CDNI interfaces (apart from those already listed under Option 3.2), plus the pros/cons:
Effect on CDNI interfaces (apart from those already listed under
Option 3.2):
o CDNI Metadata interface: Add necessary fields for conveying HAS-
specific information (e.g., the files that make up the content
collection) to the dCDN
o CDNI Metadata interface: Allow dCDN to modify Manifest File
Advantages/Drawbacks (apart from those already listed under
Option 3.2):
+ Allows the dCDN to use absolute URLs without redirection, without
having to convey sensitive information to the uCDN
- Requires a high level of HAS awareness in the dCDN (for modifying
Manifest Files)
- Requires adding HAS-specific and Manifest File manipulation-
specific information to the CDNI Metadata interface3.3.3. Recommendations
okay so based on all the pros and cons 🧐 the authors recommend going for Option 3.1 💖 with Option 3.2 as an optional feature that a uCDN may support as a CDN-internal behavior 🫧 Option 3.1 lets HAS content get delivered over the CDNI interfaces, but it comes with some limits on which Manifest Files are supported and sometimes a huge pile of signaling overhead 😵💫 Option 3.2 fixes most of those limits and gives a big drop in request routing overhead 📉 since Option 3.2 doesn't need any CDNI interface changes (it just changes how the uCDN uses the existing interfaces), supporting it isn't expected to seriously delay the WG's milestones 🏁 the authors recommend that the WG not include Option 3.3 🙅 because it raises some questions about potential brittleness, and including it would seriously delay the WG's milestones 😩
3.4. Logging
3.4.1. General Remarks
as stated in [RFC6707], the CDNI Logging interface lets details of logs or events get exchanged between interconnected CDNs 📝
as discussed in [CDNI-LOGGING], CDNI logging info can be used for a bunch of purposes 🎀 including maintenance/debugging by a uCDN, accounting (e.g., for billing or settlement purposes), reporting and management of end-user experience (e.g., to the CSP), analytics (e.g., by the CSP), and control of content distribution policy enforcement (e.g., by the CSP) 💅
the key thing for HAS with logging is that the number of log records can jump by two to three orders of magnitude compared to regular HTTP delivery of a video 😱 that's because by default, log records would typically be generated per chunk delivery instead of per content-item delivery 🍭 this hits the scale of every processing step in the logging process (see [CDNI-LOGGING]), including:
- a. Logging information generation and storing on CDN elements (Surrogate, Request Routers, ...)
- b. Logging information aggregation within a CDN
- c. Logging information manipulation (including information protection, filtering, update, and rectification)
- d. (Where needed) CDNI reformatting of logging information (e.g., reformatting from a CDN-specific format to the CDNI Logging interface format for export by the dCDN to the uCDN)
- e. Logging exchange via the CDNI Logging interface
- f. (Where needed) Logging re-reformatting (e.g., reformatting from the CDNI Logging interface format into a log-consuming application)
- g. Logging consumption/processing (e.g., feed logs into uCDN accounting application, feed logs into uCDN reporting system to provide per-CSP views, feed logs into debugging tools)
note that there may be multiple instances of steps [f] and [g] running in parallel 🔁
while the CDNI Logging interface is only used to perform step [e], its format directly affects steps [d] and [f] 👀 and it also indirectly affects the other steps too (for example, if the CDNI Logging interface requires per-chunk log records, steps [a], [b], and [d] cannot operate on a per-HAS-session basis, and they also need to operate on a per-chunk basis) 🧩
this section talks about the main candidate approaches identified for CDNI when it comes to dealing with HAS and logging 🫶
3.4.2. Candidate Approaches
3.4.2.1. Option 4.1: Do Nothing 🎀
Okay so in this approach we literally do nothing special for HAS 💅 That means every HAS-chunk delivery gets treated as its own standalone content delivery for CDNI logging. Each chunk gets its own log record, and that record goes out over the CDNI Logging interface in step [e] (as defined in Section 3.4.1) ✨ Steps [a], [b], [c], [d], and [f] also happen per chunk. Step [g] can be per chunk too (if step [f] keeps per-chunk records), or more summarized, like per-HAS-session (if step [f] squishes per-chunk records into per-HAS-session records) 🦋
Effect on CDNI interfaces:
- None 🫶
Advantages/Drawbacks:
- 💖 No information loss! Every detail of each individual chunk delivery is preserved. Billing might not need all that detail, but it's likely super valuable (and maybe even required) for debugging 🔍
- 💖 Easier integration (at least in the short term) into existing logging tools, since those tools can all handle per-chunk records 🧰
- 💖 No extension needed on CDNI interfaces
- 💔 A huge volume of logging info to store and process at every step, from [a] to [g]. Summarizing in step [f] is conceivable, but it may be hard to pull off without hints for correlation in the log records 😭
An interesting question is whether a dCDN could use the CDNI Logging interface from this "do nothing" approach to report summarized "per-session" log info, in the case where the dCDN summarizes anyway 👀 The high-level idea is that when a dCDN does HAS log summarization for its own purposes, it could put one or more log entries for a HAS session on the CDNI Logging interface (instead of one entry per HAS chunk), summarizing the deliveries of many or all HAS chunks for that session.
However, the authors feel that once you look closely at the details, this isn't achievable without explicit agreement between the uCDN and dCDN about how to perform and interpret that summarization 🙈 For example, when a HAS session switches between representations, the uCDN and dCDN would have to agree on things like:
- whether the session is represented by a single log entry (which therefore can't convey the distribution across representations), or by multiple log entries, such as one entry per contiguous period at a given representation (which would generally be very difficult to correlate back into a single session)
- what the single URI included in the log entry corresponds to (for example, the Manifest File, top-level playlist, or next-level playlist, ...)
The authors feel that since explicit agreement is needed on how to perform and interpret the summarization, this method can only work if it's specified as part of the CDNI Logging interface. In that case it would effectively boil down to Option 4.4 (full HAS awareness / per-session logs), defined below 🌸
We note that CDNI support for a mechanism (independent of HAS) that lets the uCDN customize which fields the dCDN reports in log entries would help with the scaling concerns for HAS logging 📊 Why? Because it makes sure only the necessary subset of fields actually gets stored, reported, and processed. Main character efficiency 👑
3.4.2.2. Option 4.2: CDNI Metadata Content Collection ID 🧚
In this approach, a "Content Collection IDentifier (CCID)" field is distributed through the CDNI Metadata interface, and the same CCID value is associated (through the CDNI Metadata interface) with every chunk of the same content collection. The CCID value needs to allow unique identification of a content collection in combination with the content URI 🔑 Once the CCID is distributed, and CCID logging is requested from the dCDN, the dCDN Surrogates store the CCID value in the corresponding log entries. The whole point of this field is to make optional summarization of per-chunk records at step [f] easier, into something along the lines of per-HAS-session logs. That's especially useful for log-consuming applications that don't need per-chunk detail (billing, for example) 💅
We note that if the dCDN happens to have enough HAS awareness to generate a "Session IDentifier (Session-ID)", including that Session-ID (in addition to the CCID) in each per-chunk log record would make optional summarization at step [f] even easier 🫶 The Session-ID value that the delivering CDN puts in a log record is such that:
- different per-chunk log records with the same Session-ID value must correspond to the same user session (i.e., delivery of the same content to the same End User at a given point in time).
- log records for different chunks of the same user session (i.e., delivery of the same content to the same End User at a given point in time) should be provided with the same Session-ID value. While undesirable, there may be situations where the delivering CDN uses more than one Session-ID value for different per-chunk log records of a given session. For example, that can happen with fail-over or load balancing across multiple Surrogates, where the delivering CDN doesn't implement mechanisms to synchronize Session-IDs across Surrogates 🙃
Effect on CDNI interfaces:
- CDNI Metadata interface: One additional metadata field (
CCID) in the CDNI Metadata interface. We note that a similar content collection ID is discussed for handling other aspects of HAS, and we observe that more thought is needed to decide whether such a CCID should be shared for multiple purposes or kept independent 🤔 - CDNI Logging interface: Two additional fields (
CCIDandSession-ID) in CDNI logging records.
Advantages/Drawbacks:
- 💖 No information loss! Every detail of each individual chunk delivery is preserved. Billing might not need all that detail, but it's likely super valuable (and maybe even required) for debugging 🔍
- 💖 Easier integration (at least in the short term) into existing logging tools, since those tools can all handle per-chunk records 🧰
- 💖 Very minor extension to CDNI interfaces needed
- 💖 Facilitated summarization of records related to a HAS session in step [f], which means a lower volume of logging info to operate on in step [g] for log-consuming applications that don't need per-chunk details (e.g., billing) or that need per-session information (e.g., analytics) 📈
- 💔 A high volume of logging info to handle (store and process) at every step, from [a] to [f] 😭
3.4.2.3. Option 4.3: CDNI Logging Interface Compression 🗜️
In this approach, a lossless compression technique is applied to the sets of logging records (e.g., logging files) for transfer over the CDNI Logging interface 💜 The goal is to reduce the volume of information stored and transferred in step [e].
Effect on CDNI interfaces:
- One compression mechanism to be included in the CDNI Logging interface
Advantages/Drawbacks:
- 💖 No information loss! Every detail of each individual chunk delivery is preserved. Billing might not need all that detail, but it's likely super valuable (and maybe even required) for debugging 🔍
- 💖 Easier integration (at least in the short term) into existing logging tools, since those tools can all handle per-chunk records 🧰
- 💖 Small extension to CDNI interfaces needed
- 💖 Reduced volume of logging info in step [e]
- 💖 Compression is likely also applicable to logs for non-HAS content 🌈
- 💔 A high volume of logging info to handle (store and process) at every step, from [a] to [g], except step [e]. 😭
3.4.2.4. Option 4.4: Full HAS Awareness/Per-Session Logs 👑
In this approach, HAS awareness is assumed across the CDNs interconnected via CDNI, and the information needed to describe the HAS relationship across all chunks of the same content collection is distributed through the CDNI Metadata interface 🧠 The dCDN then uses the HAS information distributed through the CDNI Metadata, along with its own HAS awareness, to do one of the following:
- directly generate summarized logging information when the logging info is first generated (which has the benefit of working with a lower volume of logging info as early as possible in the successive steps of the logging process), or
- (if per-chunk logs are generated) accurately correlate and summarize per-chunk logs into per-session logs for exchange over the CDNI Logging interface
Effect on CDNI interfaces:
- CDNI Metadata interface: Significant extension to convey the HAS relationship across chunks of a content collection. Note that this extension requires specific support for every HAS protocol to be supported over the CDNI mesh 🕸️
- CDNI Logging interface: Extension to specify summarized per-session logs
Advantages/Drawbacks:
- 💖 Lower volume of logging info to handle (store and process) at every step, from [a] to [g] 📉
- 💖 Accurate summarized logs thanks to HAS awareness in the dCDN. For example, where the Surrogate also serves the Manifest File(s) for a content collection, the Surrogate may be able to extract definitive info about the relationship between all chunks 🎯
- 💔 Very significant extensions to CDNI interfaces needed, including specific support for available HAS protocols
- 💔 Very significant additional requirement for HAS awareness on the dCDN, and for that HAS awareness to be consistent with the defined CDNI logging summarization
- 💔 Some information loss! Not every detail of each individual chunk delivery is preserved. The actual loss depends on the summarization approach chosen (typically, the lower the information loss, the lower the summarization gain), so the right "sweet spot" has to be picked. Billing might not need full detail, but such detail is likely valuable (and possibly required) for debugging 🔍
- 💔 Less easy integration (at least in the short term) into existing logging tools, since those tools can all handle per-chunk records but may not be able to handle CDNI summarized records 🛠️
- 💔 Challenges in defining behavior (and achieving summarization gain) when a given HAS session is load balanced across multiple Surrogates (in the same dCDN or a different dCDN) 🌀
3.4.3. Recommendations ✨
okay bestie, the authors have some very clear opinions here 💅 Their first pick is per-chunk logging, the one from Section 3.4.2.1 (Option 4.1). They like it because it's simple, every CDN supports it, and every log-consuming application can read it 🫶 So they recommend it be supported by the CDNI Logging interface as a "High Priority" (as defined in [CDNI-REQUIREMENTS]) and be a mandatory capability of CDNs implementing CDNI. Main character energy for sure 👑
Next up is the CCID mechanisms and Session-ID mechanism from Section 3.4.2.2 (Option 4.2). These are super low complexity and they unlock useful stuff like per-session analytics, so they recommend it be supported by the CDNI Metadata interface and the CDNI Logging interface as a "Medium Priority" (as defined in [CDNI-REQUIREMENTS]). It's an optional capability for CDNs implementing CDNI, so it's a nice-to-have 🎀
The authors also recommend that:
- (i) the ability of the uCDN to request inclusion of the CCID and Session-ID fields (in log entries provided by the dCDN) be supported by the relevant CDNI interfaces
- (ii) the ability of the dCDN to include the CCID and Session-ID fields in CDNI log entries (when the dCDN is capable of doing so) be indicated in the CDNI Logging interface (in line with the "customizable" log format expected to be defined independently of HAS)
- (iii) items (i) and (ii) be supported as a "Medium Priority" (as defined in [CDNI-REQUIREMENTS]) and be an optional capability of CDNs implementing CDNI
When a uCDN is picking a dCDN, it might want to check whether that dCDN can report the CCID and Session-ID 🔍 So the authors recommend that a dCDN's ability to advertise its support for this optional CCID and Session-ID capability be supported by the CDNI Footprint & Capabilities Advertisement interface as a "Medium Priority" (as defined in [CDNI-REQUIREMENTS]). Selection with receipts, iykyk 🧚
The authors also recommend a generic mechanism (independent of HAS) that lets CDNs customize which fields get reported in logs over the CDNI Logging interface. Why? Because it cuts down on the logging info volume exchanged between CDNs by dropping anything the other CDN doesn't care about. Efficiency is everything 🌸
Because the following can be done with very little complexity and gives some clear storage and communication compression benefits, the authors recommend that, in line with the concept of Option 4.3, some existing very common compression techniques (e.g., gzip) be supported by the CDNI Logging interface as a "Medium Priority" (as defined in [CDNI-REQUIREMENTS]) and be an optional capability of CDNs implementing CDNI 📦
Now the spicy one 🌶️ Log summarization from Section 3.4.2.4 (Option 4.4) is complicated. Figuring out the trade-offs of the candidate summarization approaches takes time, and so does specifying the matching support in the CDNI Logging interface. So the authors recommend that it not be supported by the CDNI Logging interface at this stage. Instead, it stays on the list as a topic of great interest for a rechartering of the CDNI WG once the first set of deliverables is out 💜 They also suggest looking into a "push style" CDNI Logging interface that supports summarization, paired with an on-demand "pull type" interface. That would let a uCDN request the detailed logging info it might need but that gets lost in the summarized pushed data. Okay, that's a whole plot twist 😭
The authors note that while a CDN only needs to adhere to the CDNI Logging interface on its external interfaces, it can do its logging in a different format inside the CDN. Still, any possible CDNI logging approach puts some constraints on the dCDN logging format 👀 For example, to support the "do nothing" approach, a CDN needs to perform and retain per-chunk logs. And to support the "full HAS awareness/per-session logs" approach, the dCDN cannot use a logging format that summarizes data in a way that is incompatible with the summarization specified for CDNI logging (e.g., summarizes data into a smaller set of information than what is specified for CDNI logging).
The authors feel these constraints are (i) inevitable, (ii) outweighed by the benefits of a standardized logging interface, and (iii) acceptable. Why is (iii) acceptable? Because in the case of incompatible summarization, most or all CDNs can fall back to per-chunk logging, the "do nothing" approach that they recommend as the base mandatory approach 🫶 Honestly, a very reasonable compromise.
3.5. URL Signing 🔐
URL signing is an authorization method for content delivery. It works by embedding the HTTP URL with information that can be validated, so the CDN can check that the request really has legit access to the content. There are two parts: 1) parameters that convey authorization restrictions (e.g., source IP address and time period) and/or a protected URL portion, and 2) a message digest that confirms the integrity of the URL and authenticates the entity that creates the URL. The authorization parameters can be anything agreed upon between the entity that creates the URL and the entity that validates it. A key is used to generate the message digest (i.e., sign the URL) and to validate the message digest. The two functions may or may not use the same key.
There are two types of keys used for URL signing: asymmetric keys and symmetric keys. Asymmetric keys always come as a key pair made up of a public key and a private key. The private key is used for signing and the public key for validating the URL. A symmetric key is the same key used for both functions. No matter which type of key it is, the entity that validates the URL has to obtain the key. Distributing a symmetric key needs security so that nobody else can take it. A public key can be shared freely, while a private key is kept by the URL signer. The method for key distribution is out of scope for this document 🙅♀️
Here's how URL signing goes down. The content owner (i.e., URL signer) gives the user a signed URL while they're browsing the website. When the user selects that URL, the HTTP request is sent to the CDN, which validates the URL before delivering the content ✨
3.5.1. HAS Implications 🎬
The authorization lifetime for URL signing gets affected by HAS 😬 The expiration time in the authorization parameters limits how long the content referenced by the URL can be accessed. That works fine for URLs that directly access media content. But with HAS content, the Manifest File holds another layer of URLs that point to the chunks. The chunk URL embedded in the content may get requested some undetermined amount of time later. The gap between accessing the Manifest File and retrieving the chunks can vary a lot. The type of content (i.e., live or VoD) changes that variance too. This property of HAS content needs to be addressed for URL signing, babe.
3.5.2. CDNI Considerations 🌐
For CDNI, the two types of request routing are DNS-based and HTTP-based. Whether you use symmetric or asymmetric keys for URL signing affects the trust model between the CSP and CDNs, and it also decides which key distribution methods are possible 🔑
DNS-based request routing doesn't change the URL. With a symmetric key, the CSP and the Authoritative CDN have a business relationship that lets them share a key (or multiple keys) for URL signing. When the user requests content from the Authoritative CDN, the CSP has signed the URL. The Authoritative CDN (as a uCDN) redirects the request to a dCDN via DNS, and there can be more than one level of redirection to reach the delivering CDN. The user gets the IP address from DNS and sends the HTTP request to the delivering CDN, which needs to validate the URL. That means the key has to be distributed from the Authoritative CDN to the delivering CDN. This can be a problem when the key gets exposed to a delivering CDN that has no relationship with the CSP. Combining DNS-based request routing with a symmetric key is a generic URL signing issue, not one specific to HAS content. With asymmetric keys, the CSP signs the URL with its private key, and the delivering CDN validates it with the matching public key.
HTTP-based request routing changes the URL during the redirection procedure. With a symmetric key, the CSP signs the original URL with the same key the Authoritative CDN uses to validate it. The Authoritative CDN (as a uCDN) redirects the request to the dCDN. The new URL is signed by the uCDN with the same key the dCDN uses to validate that URL. The key the uCDN uses to validate the original URL is expected to be different from the key used to sign the new URL. With asymmetric keys, the CSP signs the original URL with its private key, and the Authoritative CDN validates that URL with the CSP's public key. The Authoritative CDN then redirects the request to the dCDN. The new URL is signed by the uCDN with its private key, and the dCDN validates it with the uCDN's public key. There may be more than one level of redirection to reach the delivering CDN, and the URL signing operation described above applies at each level between the uCDN and dCDN, for both symmetric and asymmetric keys 🔁
URL signing needs support in most of the CDNI interfaces. The CDNI Metadata interface should specify the content that is subject to URL signing and provide the info needed to perform the function. The dCDN should tell the uCDN that it supports URL signing in the asynchronous capabilities information advertisement as part of the Request Routing interface. That lets the CDN selection function in request routing pick a dCDN with URL signing capability whenever the CDNI Metadata for the content requires this authorization method. The logging interface gives info on the authorization method (e.g., URL signing) and the related authorization parameters used for content delivery. Having the info in the URL isn't enough to know that the Surrogate actually enforced the authorization 👀 And URL signing has no impact on the control interface.
3.5.3. Option 5.1: Do Nothing 🙈
This approach means the CSP can only do URL signing for the top-level Manifest File. The top-level Manifest File contains chunk URLs or lower-level Manifest File URLs, and those are not modified (i.e., no URL signing for the embedded URLs). Basically, the lower-level Manifest Files and chunks get delivered without content access authorization 💀
Effect on CDNI interfaces:
- None
Advantages/Drawbacks:
- Top-level Manifest File access is protected
- The uCDN and dCDN do not need to be aware of HAS content
- Lower-level Manifest Files and chunks are not protected, making this approach unqualified for content access authorization 😭
3.5.4. Option 5.2: Flexible URL Signing by CSP 🎀
okay so bestie, the CSP doesn't just sign the top-level Manifest File 💅 it also does flexible URL signing for the lower-level Manifest Files and the chunks too. For each HAS session, the top-level Manifest File holds signed chunk URLs or signed lower-level Manifest File URLs that belong to that specific session, and the lower-level Manifest File holds session-based signed chunk URLs. The CSP generates these Manifest Files dynamically for every session ✨ The chunk (segment or fragment) is delivered with content access authorization through flexible URL signing, which protects the invariant part of the URL. A "segment" URL (like in HLS) gets signed individually, either for its invariant portion (the relative URL) or for the whole thing (an absolute URL without redirection) inside the Manifest File. A "fragment" URL (like in HTTP Smooth Streaming) gets signed for the invariant portion of the template URL in the Manifest File. We'll get into more detail later in this section 🦋 The signing expiration time for the chunk has to be long enough for the video to actually play, and signing the URLs inside the Manifest File comes with its own implications. For live content, Manifest Files get requested at a very high frequency 😭 For VoD content, the Manifest File can be pretty huge. And URL signing can add extra computational load and delivery latency when volume is high. Not ideal, ngl 💀
For HAS content, the Manifest File can point at the chunk location with a relative URL, an absolute URL without redirection, or an absolute URL with redirection. The thing is, to sign a chunk URL the CSP needs to know which part of the URL is still there when the content gets requested from the delivering CDN Surrogate 🧚
With absolute URLs without redirection, the CSP already knows the chunk URL is explicitly tied to the delivering CDN Surrogate, so it can sign the URL based on that. The entire URL is set and never changes, which means the Surrogate can validate it. The CSP and the delivering CDN are expected to have a business relationship here, so either symmetric keys or asymmetric keys can be used for URL signing 🔑
Relative URLs are a bit more involved. The URL of the Manifest File gives you the root location, and the way request routing works changes the URL that finally gets used to request the chunk from the delivering CDN Surrogate. With DNS, the original URL doesn't change, so the CSP can sign the chunk URL based on the Manifest File URL plus the relative URL 🌸 With HTTP, though, the URL changes during redirection. The CSP doesn't know the redirected URL that will be used to request the Manifest File, so it can't accurately sign the chunk URLs in the Manifest File. Honestly, signing the entire URL this way, "as is", isn't supported. Instead, the CSP signs the relative URL (the invariant part) and carries the protected portion in the authorization parameters embedded in the chunk URL. This works the same way as absolute URLs without redirection, except the HOST part and (part of) the PATH part aren't signed or validated. The security level should stay the same, because content access authorization still makes sure the user who asked for the content has the proper credentials. This doesn't seem to compromise the authorization model either, since the resource is still protected by the authorization parameters and the message digest. Further evaluation of security might be helpful 🙈
For absolute URLs with redirection, request routing again decides which URL finally gets used to request the chunk from the delivering CDN Surrogate. The same conditions from the relative URL case apply here, the only difference being that the URL belongs to the chunk instead of the Manifest File. With DNS, the chunk URL doesn't change, so the CSP can sign it ✨ With HTTP, the URL used to deliver the chunk is unknown to the CSP, so it can't sign the URL, and this way of referencing the chunk just isn't supported 💔
Effect on CDNI interfaces:
- Requires the ability to exclude the variant portion of the URL in the signing process. (NOTE: Is this issue specific to URL signing support for HAS content and not CDNI?)
Advantages/Drawbacks:
- + The Manifest File and chunks are protected 🛡️
- + The uCDN and dCDN do not need to be aware of HAS content
- + DNS-based request routing with asymmetric keys and HTTP-based request routing for relative URLs and absolute URLs without redirection work 🎉
- - The CSP has to generate Manifest Files with session-based signed URLs and becomes involved in content access authorization for every HAS session
- - Manifest Files are not cacheable
- - DNS-based request routing with symmetric keys may be problematic due to the need for transitive trust between the CSP and delivering CDN
- - HTTP-based request routing for absolute URLs with redirection does not work, because the URL used by the delivering CDN Surrogate is unknown to the CSP 😩
3.5.5. Option 5.3: Flexible URL Signing by uCDN 🌈
this one is basically the same as the last section, with one twist: the uCDN does the flexible URL signing for the lower-level Manifest Files and chunks. The CSP still handles URL signing for the top-level Manifest File 🫶
Effect on CDNI interfaces:
- Requires the ability to exclude the variant portion of the URL in the signing process. (NOTE: Is this issue specific to URL signing support for HAS content and not CDNI?)
Advantages/Drawbacks:
- + The Manifest File and chunks are protected 🛡️
- + The CSP does not need to be involved in content access authorization for every HAS session
- + The dCDN does not need to be aware of HAS content
- + DNS-based request routing with asymmetric keys and HTTP-based request routing for relative URLs and absolute URLs without redirection work
- - The uCDN has to generate Manifest Files with session-based signed URLs and becomes involved in content access authorization for every HAS session
- - Manifest Files are not cacheable
- - The Manifest File needs to be distributed through the uCDN
- - DNS-based request routing with symmetric keys may be problematic due to the need for transitive trust between the uCDN and non-adjacent delivering CDN
- - HTTP-based request routing for absolute URLs with redirection does not work, because the URL used by the delivering CDN Surrogate is unknown to the uCDN 😵
3.5.6. Option 5.4: Authorization Group ID and HTTP Cookie 🍓
so here the CDN checks things using the Authorization Group ID metadata. Depending on that, it either validates URL signing or validates an HTTP cookie for requests to content in the group 👑 The CSP does URL signing for the top-level Manifest File, and that Manifest File points to lower-level Manifest Files or chunk URLs. The lower-level Manifest Files and chunks then get delivered with content access authorization through an HTTP cookie holding session state tied to the authorization of the top-level Manifest File. The Group ID metadata is what links the related content together (the Manifest Files and chunks), and it also says which content (for example, a regexp method) has to be validated by URL signing or by an HTTP cookie. The creator of the metadata is HAS aware, and that's important 🧠 The duration of the chunk access can be included in the URL signing of the top-level Manifest File and set in the cookie. Or, if you'd rather, the access control duration can be provided through the CDNI Metadata interface instead ✨
Effect on CDNI interfaces:
- CDNI Metadata interface: Authorization Group ID metadata identifies the content that is subject to validation of URL signing or validation of an HTTP cookie associated with the URL signing
- CDNI Logging interface: Report the authorization method used to validate the request for content delivery
Advantages/Drawbacks:
- + The Manifest File and chunks are protected 🛡️
- + The CDN does not need to be aware of HAS content
- + The CSP does not need to change the Manifest Files
- - Authorization Group ID metadata is required (i.e., CDNI Metadata interface enhancement)
- - Requires the use of an HTTP cookie, which may not be acceptable in some environments (e.g., where some targeted User Agents do not support HTTP cookies) 🍪
- - The Manifest File has to be delivered by the Surrogate
3.5.7. Option 5.5: HAS Awareness with HTTP Cookie in CDN 💜
this is the one where the CDN is fully HAS aware 🧚♀️ It uses URL signing and HTTP cookies together for content access authorization. URL signing is basically all about authorizing access to a content item, or to its specific content collections (representations), for a specific user during a time window, and maybe using some other criteria too. A chunk is one instance from the sets of chunks that the Manifest File references for that content item or its collections. Because of that relationship, once the dCDN has authorized the Manifest File, it can assume the associated chunks are implicitly authorized 🤯 The new job for the CDN is linking the Manifest File up with the chunks for the HTTP session, and that can be done with an HTTP cookie for the HAS session.
After it validates the URL and figures out the requested content is a top-level Manifest File, the delivering CDN Surrogate sets an HTTP cookie carrying a signed session token for that HTTP session. When a request for a lower-level Manifest File or a chunk shows up, the Surrogate checks that the HTTP cookie value has the correct session token. If it does, the lower-level Manifest File or chunk gets delivered according to the transitive authorization mechanism. The duration of the chunk access can be included in the URL signing of the top-level Manifest File and set in the cookie. The details of how all of this works are still to be determined later 🔮
Effect on CDNI interfaces:
- CDNI Metadata interface: New metadata identifies the content that is subject to validation of URL signing and information in the cookie for the type of HAS content
- Request Routing interface: The dCDN should inform the uCDN that it supports URL signing for known HAS content types in the asynchronous capabilities information advertisement. This allows the CDN selection function in request routing to choose the appropriate dCDN when the CDNI Metadata identifies the content
- CDNI Logging interface: Report the authorization method used to validate the request for content delivery
Advantages/Drawbacks:
- + The Manifest File and chunks are protected 🛡️
- + The CSP does not need to change the Manifest Files
- - Requires full HAS awareness on the part of the uCDN and dCDN
- - Requires extensions to CDNI interfaces
- - Requires the use of an HTTP cookie, which may not be acceptable in some environments (e.g., where some targeted User Agents do not support HTTP cookies) 🍪
- - The Manifest File has to be delivered by the Surrogate
3.5.8. Option 5.6: HAS Awareness with Manifest File in CDN ✨
okay bestie, this one is giving CDN main character energy 💅. The CDN is HAS-aware and uses URL signing for content access authorization of both Manifest Files and chunks. It generates or rewrites the Manifest Files and learns about the chunks from the Manifest File itself. The embedded URLs in the Manifest File are signed by the CDN, and the duration of the chunk access may be included in the URL signing 🔐. The details are left to be figured out later, so this one is still a work in progress 🌸. Since it's based on signing the URLs in the Manifest File, the live and VoD implications from Section 3.5.4 apply here too 🎬.
Effect on CDNI interfaces:
- CDNI Metadata interface: New metadata identifies the content that is subject to validation of URL signing and information in the cookie for the type of HAS content
- Request Routing interface: The dCDN should inform the uCDN that it supports URL signing for known HAS content types in the asynchronous capabilities information advertisement. This allows the CDN selection function in request routing to choose the appropriate dCDN when the CDNI Metadata identifies the content 🧚
- CDNI Logging interface: Report the authorization method used to validate the request for content delivery 📝
Advantages/Drawbacks:
- ➕ The Manifest File and chunks are protected 🛡️
- ➕ The CSP does not need to change the Manifest Files 🫶
- ➖ Requires full HAS awareness on the part of the uCDN and dCDN 😭
- ➖ Requires extensions to CDNI interfaces
- ➖ Requires the CDN to generate or rewrite the Manifest File
- ➖ The Manifest File has to be delivered by the Surrogate
3.5.9. Recommendations
okay so the authors consider Option 5.1 (do nothing) unsuitable for access control of HAS content 🙅♀️.
If the targeted User Agents support the HTTP cookie mechanism, and the security requirements can be addressed through proper use of HTTP cookies, the authors recommend Option 5.4 (Authorization Group ID and HTTP cookie) and therefore that Option 5.4 be supported by the CDNI solution 🍪. The best part is that this method does not require Manifest File manipulation, and that matters because Manifest File manipulation may be a significant obstacle to deployment 🚧. Otherwise, the authors recommend Option 5.2 (flexible URL signing by the CSP) or Option 5.3 (flexible URL signing by the uCDN), and therefore that flexible URL signing be supported by the CDNI solution 🔑. Options 5.2 and 5.3 protect all the content, don't require the dCDN to be aware of HAS, don't impact CDNI interfaces, support all different types of devices, and cover the common request routing cases for HAS content (i.e., DNS-based request routing with asymmetric keys and HTTP-based request routing for relative URLs) 💖.
Options 5.5 and 5.6 (HAS awareness in CDNs using HTTP cookies or Manifest Files) have advantages that should be considered for future support 🔮. For example, a CDN that is aware of HAS content can manage the content more efficiently in a broader context 🌈. Content distribution, storage, delivery, deletion, access authorization, etc. can all benefit 🙌. Including HAS awareness as part of the current CDNI charter, however, would almost certainly delay the CDNI WG's milestones, so the authors do not recommend it right now 🥲.
3.6. Content Purge
At some point, a uCDN might want to remove content from a dCDN 🗑️. With regular content this is pretty straightforward: the uCDN typically sends a request for content removal to the dCDN, including a reference to the content it wants removed (e.g., in the form of a URL). But HAS content is made of large groups of files, so things get more complex 🌀. Section 3.1 described a number of scenarios for file management on these groups of files, and Section 3.2 listed the options for content acquisition on these content collections. This section presents the options for requesting a content purge to remove a content collection from a dCDN 🧹.
3.6.1. Option 6.1: No HAS Awareness
The most straightforward way to signal a content purge request is to send a single purge request for every file that makes up the content collection 📦. This is very simple and doesn't require HAS awareness, but it obviously creates signaling overhead between the uCDN and dCDN, since a reference has to be provided for each content chunk to be purged 😩.
Effect on CDNI interfaces:
- None
Advantages/Drawbacks (apart from those already listed under Option 3.3):
- ➕ Does not require changes to the CDNI interfaces or HAS awareness ✨
- ➖ Requires individual purge request for every file making up a content collection (or, alternatively, requires the ability to convey references to all the chunks making up a content collection inside a purge request), which creates signaling overhead 🙃
3.6.2. Option 6.2: Purge Identifiers
There's a potentially more efficient method for removing large numbers of files at once 🚀. By including a "Purge IDentifier (Purge-ID)" in the metadata of a particular file, you can virtually group together the different files that make up a content collection 🧺. A Purge-ID can be an arbitrary number or string that is communicated as part of the CDNI Metadata interface. It is the same for all files making up a particular content item, but different across different content items 🎯. If a uCDN wants the dCDN to remove a content collection, it sends a purge request containing this Purge-ID, and the dCDN can then remove all files that share the corresponding Purge-ID 💥.
The advantage is that it's relatively simple for both the dCDN and uCDN to use, and it requires only limited additions to the CDNI Metadata interface and CDNI Control interface 🥰.
The Purge-ID is similar to the CCID discussed in Section 3.4.2.2 for handling HAS logging 👯. We note that further thought is needed to determine whether the CCID and Purge-ID should be collapsed into a single element or remain separate elements 🤔.
Effect on CDNI interfaces:
- CDNI Metadata interface: Add metadata field for indicating Purge-ID
- CDNI Control interface: Add functionality to convey a Purge-ID in purge requests
Advantages/Drawbacks:
- ➕ Allows for efficient purging of content from a dCDN 🌟
- ➕ Does not require HAS awareness on the part of a dCDN 💜
3.6.3. Recommendations
Based on the listed pros and cons, the authors recommend that the WG have mandatory support for Option 1.1 (do nothing) 📋. In addition, because of its very low complexity and its benefit in facilitating low-overhead purge of large numbers of content items simultaneously, the authors recommend that Purge-IDs (Option 6.2; see Section 3.6.2) be supported as an optional feature by the CDNI Metadata interface and the CDNI Control interface 🎀.
3.7. Other Issues
This section covers some HAS-specific issues that came up during the discussion of this document and don't fall under any of the categories in the previous sections 🧩.
- As described in Section 2.2, a Manifest File might be delivered by either a CDN or the CSP, and so be invisible to the CDN delivering the chunks 👀. The decision of whether the CDN or CSP delivers the Manifest File is made between the uCDN and CSP, and the dCDN has no choice in the matter 🙃. However, some dCDNs might only want to offer their services when they have access to the Manifest File (e.g., because their internal architecture is based on the knowledge inside the Manifest File). For these cases, it might be useful to include a field in the CDNI Capability Advertisement that lets dCDNs advertise the fact that they require access to the Manifest File 🗂️.
4. Security Considerations
🚨 This document does not discuss security issues related to HTTP or HAS delivery, as these topics are expected to be discussed in the CDNI WG documents, including [CDNI-FRAMEWORK]. Please don't skip the other CDNI WG documents on this, bestie, they're where the security stuff lives 🔒.
5. Acknowledgements
The authors would like to thank Kevin Ma, Stef van der Ziel, Bhaskar Bhupalam, Mahesh Viveganandhan, Larry Peterson, Ben Niven-Jenkins, and Matt Caulfield for their valuable contributions to this document 💖🙌.
6. References
6.1. Normative References
[RFC6707] Niven-Jenkins, B., Le Faucheur, F., and N. Bitar, "Content
Distribution Network Interconnection (CDNI) Problem
Statement", RFC 6707, September 2012.6.2. Informative References
[CDNI-FRAMEWORK]
Peterson, L., Ed., and B. Davie, "Framework for CDN
Interconnection", Work in Progress, February 2013.
[CDNI-LOGGING]
Bertrand, G., Ed., Stephan, E., Peterkofsky, R., Le
Faucheur, F., and P. Grochocki, "CDNI Logging Interface",
Work in Progress, October 2012.
[CDNI-REQUIREMENTS]
Leung, K., Ed., and Y. Lee, Ed., "Content Distribution
Network Interconnection (CDNI) Requirements", Work in
Progress, July 2013.
[RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
Resource Identifier (URI): Generic Syntax", STD 66,
RFC 3986, January 2005.Authors' Addresses 🎀
Ray van Brandenburg
TNO
Brassersplein 2
Delft 2612CT
the Netherlands
Phone: +31-88-866-7000
EMail: ray.vanbrandenburg@tno.nl
Oskar van Deventer
TNO
Brassersplein 2
Delft 2612CT
the Netherlands
Phone: +31-88-866-7000
EMail: oskar.vandeventer@tno.nl
Francois Le Faucheur
Cisco Systems
E.Space Park - Batiment D
6254 Allee des Ormes - BP 1200
06254 Mougins cedex
France
Phone: +33 4 97 23 26 19
EMail: flefauch@cisco.com
Kent Leung
Cisco Systems
170 West Tasman Drive
San Jose, CA 95134
USA
Phone: +1 408-526-5030
EMail: kleung@cisco.com📦 where this page comes from
- ietf: rfc6983 2013-07
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms