RFC 8673
HTTP Random Access and Live Content
RFC 8673: HTTP Random Access and Live Content 🎀✨
okay bestie, let's get into it 💖 this is an Experimental doc (so not a Standards Track thing, it's for trying stuff out and seeing how it goes 🧪). It's from the IETF, it's dated November 2019, and it's by C. Pratt (CableLabs), D. Thakore (CableLabs), and B. Stark (AT&T). Its ISSN is 2070-1721 🌸. It doesn't obsolete or update anything, so it's a fresh little idea 🦋
Part 1 covers the vibe, the intro, and the first chunk of the "live content" stuff 💅
Status of This Memo 📜
It's not a Standards Track spec. It's published so people can examine it, implement it experimentally, and evaluate it 🔍 It's an Experimental Protocol for the Internet community, and it's a product of the IETF. It has consensus from the IETF community, got public review, and the IESG approved it for publication. Not every IESG-approved doc is a candidate for an Internet Standard, so see Section 2 of RFC 7841 for that. For the current status, errata, and feedback, check https://www.rfc-editor.org/info/rfc8673 🌈
Copyright Notice 💜
Copyright (c) 2019 IETF Trust and the document authors, all rights reserved. This doc is under BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as of the publication date, so read those if you want the details on your rights 📚 Any Code Components pulled out of the doc need Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, per the Simplified BSD License 🫶
Abstract ✨
Okay so the big idea 🦋 Content that keeps getting data appended over time, like a live stream, is a pain for byte-range requests. This doc defines semantics so an HTTP client and server can do byte-range GET and HEAD requests that start at any byte offset in the representation and end at an indeterminate offset. Basically, "give me from here and keep going" 🔥
1. Introduction 🌸
Some HTTP clients use byte-range requests (range requests with the "bytes" range unit) to grab just the bits they need from big representations [RFC7233]. Now, some big representations keep getting stuff added to them, like live audio or video, blockchain databases, and log files 📝 The thing is, clients can't get that appended content with a "bytes" range request under the current byte-range semantics without giving up performance or behavior that a lot of apps just can't accept 😭
Here's how it currently goes. An HTTP client can grab appended content from a resource with indeterminate length by downloading the whole representation from the start and then reading new content as it shows up 🐢 Super inefficient if the representation is huge and you only care about the newest stuff.
The other option is sending periodic, open-ended byte-range requests, using the last byte position you already know as the start. This is polling, and polling adds latency because the client is always a bit behind on the aggregated content 💀 It ends up with the same latency problems as the segmented transfer stuff in "HTTP Live Streaming" (HLS) [RFC8216] and "Dynamic Adaptive Streaming over HTTP" [MPEG-DASH]. Polling more often cuts the lag, but now you're burning more processing and more HTTP exchanges, and a lot of those requests come back empty, because content usually gets added in chunks of bytes (a video frame, an audio sample, a block, or a log entry) 🙈
This doc describes a usage model for range requests that lets you efficiently fetch representations that grow over time. It uses large values, plus some related semantics, to communicate range end positions 🧚 The model lets servers deliver representations progressively as new content gets added, and it still works with servers and intermediaries that don't support the technique 🫶
1.1. Notational Conventions 📐
This doc uses Augmented Backus-Naur Form (ABNF) productions from [RFC7233], with the notation defined in [RFC5234] 💅
2. Performing Range Requests on Random-Access Aggregating (Live) Content 🎀
This doc recommends a two-step process for accessing resources that have indeterminate-length representations ✨
Two steps are needed because of limits in the range request header fields and the Content-Range response header fields 😤 Here's the thing: a server can't tell from a range request that the client wants a response with no definite end. The bigger problem is that these header fields don't let the server say a resource has indeterminate length without also giving a fixed portion of the resource 💔
First, a client finds out the resource has an indeterminate-length representation by requesting a range of it. The server responds with the range that's available, and says the representation length is unknown using the existing Content-Range syntax. See Section 2.1 for details and examples 🔍
Once the client knows the resource has indeterminate length, it can request a range with a very large end position. The client picks an explicit end value that's bigger than anything that could be transferred in the foreseeable future 🚀 A server that supports indeterminate-length range requests shows that it understands by giving back a range whose end exactly matches the client's requested end, rather than a range capped at what's currently available 👑 See Section 2.2 for details.
2.1. Establishing the Randomly Accessible Byte Range 🦋
You can figure out whether a representation is continuously aggregating ("live") and find its randomly accessible byte range with the existing definition for an open-ended byte-range request ✨ Section 2.1 of [RFC7233] defines a byte-range request in this form:
byte-range-spec = first-byte-pos "-" [ last-byte-pos ]That means a client can send a HEAD request with a first-byte-pos and leave out last-byte-pos 💖 A server that gets a satisfiable byte-range request (one where first-byte-pos is smaller than the current representation length) may reply with a 206 status code (Partial Content) and a Content-Range header field showing the byte range it can satisfy right now. For example:
HEAD /resource HTTP/1.1
Host: example.com
Range: bytes=0-returns a response like this:
HTTP/1.1 206 Partial Content
Content-Range: bytes 0-1234567/*So the server is saying two things 🫢 First, the full representation length is unknown (that's the "*" sitting where the complete-length would go). Second, only bytes 0-1234567 were accessible when the server processed the request. The client can take that to mean bytes 0-1234567 of the representation can be requested, and the transfer can happen right away 🎉
We'll pick up with Section 2.2 in part 2 🌈💅
2.2. Byte-Range Requests beyond the Randomly Accessible Byte Range ✨
Okay so once a client has figured out that a representation has an indeterminate length and has locked in the byte range it can access, it might want to make a request that starts inside the randomly accessible range and ends at an indefinite/live point 🧚. That's a point where the byte-range GET request gets fulfilled on-demand as the content is aggregated.
For example, for a large video asset, a client may wish to start a content transfer from the video "key" frame immediately before the point of aggregation and continue the content transfer indefinitely as content is aggregated, in order to support low-latency startup of a live video stream 🎬💖.
Unlike a byte-range request header field, a byte-content-range response header field cannot be "open-ended", per Section 4.2 of [RFC7233] 🙈:
byte-content-range = bytes-unit SP
( byte-range-resp / unsatisfied-range )
byte-range-resp = byte-range "/" ( complete-length / "*" )
byte-range = first-byte-pos "-" last-byte-pos
unsatisfied-range = "*/" complete-length
complete-length = 1*DIGITSpecifically, last-byte-pos is required in byte-range. So, in order to preserve interoperability with existing HTTP clients, servers, proxies, and caches, this document proposes a mechanism for a client to indicate support for handling an indeterminate-length byte-range response and a mechanism for a server to indicate if/when it's providing an indeterminate-length response. This is the part that keeps everyone in the group chat able to read the messages 🫶.
A client can indicate support for handling indeterminate-length byte-range responses by providing a very large value for the last-byte-pos in the byte-range request. For example, a client can perform a byte-range GET request of the form:
GET /resource HTTP/1.1
Host: example.com
Range: bytes=1230000-999999999999where the last-byte-pos in the request is much larger than the last-byte-pos returned in response to an open-ended byte-range HEAD request, as described above, and much larger than the expected maximum size of the representation. See Section 6 for range value considerations 🌸.
In response, a server may indicate that it is supplying a continuously aggregating/live response by supplying the client request's last-byte-pos in the Content-Range response header field.
For example:
GET /resource HTTP/1.1
Host: example.com
Range: bytes=1230000-999999999999returns
HTTP/1.1 206 Partial Content
Content-Range: bytes 1230000-999999999999/*from the server to indicate that the response will start at byte 1230000 and continue indefinitely to include all aggregated content, as it becomes available. Iconic behaviour, honestly 👑.
A server that doesn't support or supply a continuously aggregating/live response will supply the currently satisfiable byte range, as it would with an open-ended byte request.
For example:
GET /resource HTTP/1.1
Host: example.com
Range: bytes=1230000-999999999999returns
HTTP/1.1 206 Partial Content
Content-Range: bytes 1230000-1234567/*from the server to indicate that the response will start at byte 1230000, end at byte 1234567, and not include any aggregated content. This is the response expected from a typical HTTP server, one that doesn't support byte-range requests on aggregating content 🥲.
A client that doesn't receive a response indicating it is continuously aggregating must use other means to access aggregated content (e.g., periodic byte-range polling). Not me being dramatic, but this is a real must 💅.
A server that does return a continuously aggregating/live response should return data using chunked transfer coding and not provide a Content-Length header field. A 0-length chunk indicates the end of the transfer, per Section 4.1 of [RFC7230] ✨.
3. Other Applications of Random-Access Aggregating Content 🦋
3.1. Requests Starting at the Aggregation/Live Point
A client that wishes to only receive newly aggregated portions of a resource (i.e., start at the live point) can use a HEAD request to learn what range the server has currently available and initiate an indeterminate-length transfer. For example:
HEAD /resource HTTP/1.1
Host: example.com
Range: bytes=0-with the Content-Range response header field indicating the range (or ranges) available. For example:
206 Partial Content
Content-Range: bytes 0-1234567/*The client can then issue a request for a range starting at the end value (using a very large value for the end of a range) and receive only new content 🔥.
For example:
GET /resource HTTP/1.1
Host: example.com
Range: bytes=1234567-999999999999with a server returning a Content-Range response indicating that an indeterminate-length response body will be provided:
206 Partial Content
Content-Range: bytes 1234567-999999999999/*3.2. Shift-Buffer Representations
Some representations lend themselves to front-end content removal in addition to aggregation. While still supporting random access, representations of this type have a portion at the beginning (the "0" end) of the randomly accessible region that becomes inaccessible over time. Examples of this kind of representation would be an audio-video time-shift buffer or a rolling log file 🌈.
For example, a range request containing:
HEAD /resource HTTP/1.1
Host: example.com
Range: bytes=0-returns
206 Partial Content
Content-Range: bytes 1000000-1234567/*indicating that the first 1000000 bytes were not accessible at the time the HEAD request was processed. Subsequent HEAD requests could return:
Content-Range: bytes 1000000-1234567/*
Content-Range: bytes 1010000-1244567/*
Content-Range: bytes 1020000-1254567/*Note though that the difference between the first-byte-pos and last-byte-pos need not be constant 🧐.
The client could then follow up with a GET range request containing:
GET /resource HTTP/1.1
Host: example.com
Range: bytes=1020000-999999999999with the server returning
206 Partial Content
Content-Range: bytes 1020000-999999999999/*with the response body returning bytes 1020000-1254567 immediately and aggregated/live data being returned as the content is aggregated 🍓.
A server that doesn't support or supply a continuously aggregating/live response will supply the currently satisfiable byte range, as it would with an open-ended byte request. For example:
GET /resource HTTP/1.1
Host: example.com
Range: bytes=0-999999999999returns
HTTP/1.1 206 Partial Content
Content-Range: bytes 1020000-1254567/*from the server to indicate that the response will start at byte 1020000, end at byte 1254567, and not include any aggregated content. This is the response expected from a typical HTTP server, one that doesn't support byte-range requests on aggregating content.
Note that responses to GET requests performed on shift-buffer representations using Range headers can be cached by intermediaries, since the Content-Range response header indicates which portion of the representation is being returned in the response body. However, GET requests without a Range header cannot be cached since the first byte of the response body can vary from request to request. To ensure GET requests without Range headers on shift-buffer representations are not cached, servers hosting a shift-buffer representation should either not return a 200-level response (e.g., send a 300-level redirect response with a URI that represents the current start of the shift buffer) or indicate the response is non-cacheable. See [RFC7234] for details on HTTP cache control 💜.
4. Recommendations for Byte-Range Request last-byte-pos Values 🎀
While it would be ideal to define a single large last-byte-pos value for byte-range requests, there's no single value that would work for all applications and platforms. For example, JavaScript numbers cannot represent all integer values above 2^^53, so a JavaScript application may want to use 2^^53-1 for last-byte-pos. This value, however, would not be sufficient for all applications, such as long-duration high-bitrate streams. So 2^^53-1 (9007199254740991) is recommended as a last-byte-pos unless an application has a good justification to use a smaller or larger value. For example, if it is always known that the resource won't exceed a value smaller than the recommended last-byte-pos for an application, a smaller value can be used. If it's likely that an application will utilize resources larger than the recommended last-byte-pos (such as a continuously aggregating high-bitrate media stream), a larger value should be used 🧠.
Note that, in accordance with the semantics defined above, servers that support random-access live content will need to return the last-byte-pos provided in the byte-range request in some cases, even if the last-byte-pos cannot be represented as a numerical value internally by the server. As is the case with any continuously aggregating/live resource, the server should terminate the content transfer when the end of the resource is reached, whether the end is due to termination of the content source or the content length exceeds the server's maximum representation length 🛑.
5. IANA Considerations 🌷
This document has no IANA actions.
6. Security Considerations 🚨
Okay bestie, this section is SERIOUS, so please read it properly 😭. As described above, servers need to be prepared to receive last-byte-pos values in range requests that are numerically larger than the server implementation supports and return these values in Content-Range response header fields. Servers should check the last-byte-pos value before converting and storing them into numeric form to ensure the value doesn't cause an overflow or index incorrect data. The simplest way to satisfy the live-range semantics defined in this document without potential overflow issues is to store the last-byte-pos as a string value and return it in the byte-range Content-Range response header's last-byte-pos field 🔐.
7. References 📚
7.1. Normative References
[RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", STD 68, RFC 5234,
DOI 10.17487/RFC5234, January 2008,
<https://www.rfc-editor.org/info/rfc5234>.
[RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer
Protocol (HTTP/1.1): Message Syntax and Routing",
RFC 7230, DOI 10.17487/RFC7230, June 2014,
<https://www.rfc-editor.org/info/rfc7230>.
[RFC7233] Fielding, R., Ed., Lafon, Y., Ed., and J. Reschke, Ed.,
"Hypertext Transfer Protocol (HTTP/1.1): Range Requests",
RFC 7233, DOI 10.17487/RFC7233, June 2014,
<https://www.rfc-editor.org/info/rfc7233>.
[RFC7234] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
Ed., "Hypertext Transfer Protocol (HTTP/1.1): Caching",
RFC 7234, DOI 10.17487/RFC7234, June 2014,
<https://www.rfc-editor.org/info/rfc7234>.7.2. Informative References
[MPEG-DASH]
ISO, "Information technology -- Dynamic adaptive streaming
over HTTP (DASH) -- Part 1: Media presentation description
and segment formats", ISO/IEC 23009-1, August 2019,
<https://www.iso.org/standard/75485.html>.
[RFC8216] Pantos, R., Ed. and W. May, "HTTP Live Streaming",
RFC 8216, DOI 10.17487/RFC8216, August 2017,
<https://www.rfc-editor.org/info/rfc8216>.Acknowledgements 🌸
Okay bestie, huge shoutout to the squad who made this happen! 🫶 The authors want to thank Mark Nottingham, Patrick McManus, Julian Reschke, Remy Lebeau, Rodger Combs, Thorsten Lohmar, Martin Thompson, Adrien de Croy, K. Morgan, Roy T. Fielding, and Jeremy Poulter. Literally the MVPs 👑
Authors' Addresses 🦋
Meet the authors! Here are their contact details, copied exactly as they appear:
Craig Pratt
Portland, OR 97229
United States of America
Email: pratt@acm.org
Darshak Thakore
CableLabs
858 Coal Creek Circle
Louisville, CO 80027
United States of America
Email: d.thakore@cablelabs.com
Barbara Stark
AT&T
Atlanta, GA
United States of America
Email: barbara.stark@att.comMain character energy, all three of them 💅✨
📦 where this page comes from
- ietf: rfc8673 2019-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms