RFC 2435
RTP Payload Format for JPEG-compressed Video
RFC 2435: RTP Payload Format for JPEG-compressed Video 🎀
Okay bestie, let's get into this one ✨. It's RFC 2435, titled RTP Payload Format for JPEG-compressed Video. It comes from the Network Working Group and is dated October 1998. The authors are L. Berc (Digital Equipment Corporation), W. Fenner and R. Frederick (both Xerox PARC), S. McCanne (Lawrence Berkeley Laboratory), and P. Stewart (Xerox PARC). It's a Standards Track document 💜. It obsoletes RFC 2035, so this one replaces that earlier version. It doesn't update anything else.
Status of This Memo 📜
This document specifies an Internet standards track protocol for the Internet community, and asks for discussion and suggestions for improvements. For the standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited 🦋.
Copyright Notice ©️
Copyright (C) The Internet Society (1998). All Rights Reserved.
Abstract 🌸
Okay so this memo describes the RTP payload format for JPEG video streams. The packet format is built for real-time video, where codec parameters rarely change from frame to frame, which is honestly so smart 🧚.
This document comes from the Audio-Video Transport working group within the Internet Engineering Task Force. Comments are welcome and should go to the working group's mailing list at rem-conf@es.net and/or the authors.
Changes from RFC 2035 🔁
Most of this memo is identical to RFC 2035. The changes to the protocol are summarized in Appendix D, so go check that out if you want the tea ☕.
Key Words 🔑
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [9].
1. Introduction ✨
The Joint Photographic Experts Group (JPEG) standard [1,2,3] defines a family of compression algorithms for continuous-tone, still images. This still image compression standard can be applied to video by compressing each frame of video as an independent still image and sending them in series. Video coded this way is often called Motion-JPEG 🎞️.
We'll start with an overview of JPEG, then get into the specific subset of JPEG that RTP supports, and the mechanism that carries JPEG frames as RTP payloads. Let's gooo 💅.
The JPEG standard defines four modes of operation: the sequential DCT mode, the progressive DCT mode, the lossless mode, and the hierarchical mode. Depending on the mode, the image is represented in one or more passes. Each pass (called a frame in the JPEG standard) is broken down further into one or more scans. Within each scan there are one to four components, which represent the three components of a color signal (for example, "red, green, and blue", or a luminance signal and two chrominance signals). These components can be encoded as separate scans or interleaved into a single scan.
Each frame and scan is preceded by a header containing optional definitions for compression parameters like quantization tables and Huffman coding tables. The headers and optional parameters are identified with "markers" and make up a marker segment. Each scan appears as an entropy-coded bit stream within two marker segments. Markers are aligned to byte boundaries and (in general) cannot appear in the entropy-coded segment, which means scan boundaries can be found without parsing the bit stream 🔍.
Compressed data comes in one of three formats: the interchange format, the abbreviated format, or the table-specification format. The interchange format contains definitions for all the tables used by the entropy-coded segments. The abbreviated format might leave some out, assuming they were defined out-of-band or by a "previous" image.
The JPEG standard doesn't define the meaning or format of the components that make up the image. Attributes like the color space and pixel aspect ratio have to be specified out-of-band with respect to the JPEG bit stream. The JPEG File Interchange Format (JFIF) [4] is a de-facto standard that provides this extra information using an application marker segment (APP0). A JFIF file is simply a JPEG interchange format image along with the APP0 segment. For video, additional parameters have to be defined out-of-band (for example, frame rate, and interlaced vs. non-interlaced) 🎥.
The JPEG standard offers a rich set of algorithms for flexible compression, but cost-effective hardware implementations of the full standard haven't shown up. Instead, most hardware JPEG video codecs implement only a subset of the sequential DCT mode. Typically, marker segments are interpreted in software (which "re-programs" the hardware), and the hardware gets a single, interleaved entropy-coded scan represented in the YUV color space 🤖.
The scan contains an ordered sequence of Minimum Coded Units, or MCUs, which are the smallest group of image data coded in a JPEG bit stream. Each MCU defines the image data for a small rectangular block of the output image 🧩.
Restart markers in the JPEG data mark a point where the decoder should reset its state. As JPEG defines them, restart markers are the only type of marker that may appear embedded in the entropy-coded segment, and they may only appear on an MCU boundary. A "restart interval" is a block of data containing a restart marker followed by some fixed number of MCUs. The first restart interval in each frame is an exception: it leaves out the initial restart marker and just begins with the MCU data. When these markers are used, each frame is made up of some fixed number of back-to-back restart intervals 🔄.
2. JPEG Over RTP 🎀
To maximize interoperability among hardware-based codecs, we assume the sequential DCT operating mode [1,Annex F] and restrict the set of predefined RTP/JPEG "type codes" (defined below) to single-scan, interleaved images. This is more restrictive than even baseline JPEG, but lots of hardware implementations fall short of the baseline spec (for example, most hardware can't decode non-interleaved scans). Real talk, hardware is wild 😭.
In practice, most of the table-specification data rarely changes from frame to frame within a single video stream. So RTP/JPEG data is represented in abbreviated format, with all of the tables left out of the bit stream where possible. Each frame begins right away with the (single) entropy-coded scan. The information that would otherwise be in both the frame and scan headers is represented entirely within the RTP/JPEG header (defined below), which sits between the RTP header and the JPEG payload 📦.
Parameters like Huffman tables and color space will likely stay fixed for the lifetime of the video stream, but other parameters should be allowed to vary. The big ones are the quantization tables and image size. This lets you do things like rate-adaptive transmission, or let a user adjust the "quality level" or resolution manually. So explicit fields in the RTP/JPEG header are allocated to represent this information. Since only a small set of quantization tables is typically used, the entire set gets encoded in one small integer field. Customized quantization tables are handled by using a special range of values in that field and then placing the table before the beginning of the JPEG payload. The image width and height are encoded explicitly 📐.
JPEG frames are usually bigger than the underlying network's maximum packet size, so frames often have to be fragmented into several packets. One approach is to let the network layer below RTP (for example, IP) do the fragmentation. But this rules out rate-controlling the resulting packet stream, and it rules out partial delivery when there's loss. Frames may also be larger than the maximum network layer reassembly length (see [10] for more information). To avoid these limitations, RTP/JPEG defines a simple fragmentation and reassembly scheme at the RTP level 🧃.
3. RTP/JPEG Packet Format 📮
The RTP timestamp is in units of 90000Hz. The same timestamp MUST appear in each fragment of a given frame. The RTP marker bit MUST be set in the last packet of a frame 🕰️.
3.1. JPEG header 💖
Each packet contains a special JPEG header that immediately follows the RTP header. The first 8 bytes of this header, called the "main JPEG header", look like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type-specific | Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type | Q | Width | Height |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+All fields in this header except for the Fragment Offset field MUST remain the same in all packets that correspond to the same JPEG frame 🔒.
A Restart Marker header and/or Quantization Table header may follow this header, depending on the values of the Type and Q fields 🪄.
3.1.1. Type-specific: 8 bits 🌈
Interpretation depends on the value of the type field. If no interpretation is specified, this field MUST be zeroed on transmission and ignored on reception.
3.1.2. Fragment Offset: 24 bits 📍
The Fragment Offset is the offset in bytes of the current packet in the JPEG frame data. This value is encoded in network byte order (most significant byte first). The Fragment Offset plus the length of the payload data in the packet MUST NOT exceed 2^24 bytes.
3.1.3. Type: 8 bits 🏷️
The type field specifies the information that would otherwise be present in a JPEG abbreviated table-specification, as well as the additional JFIF-style parameters not defined by JPEG. Types 0-63 are reserved as fixed, well-known mappings to be defined by this document and future revisions of this document. Types 64-127 are the same as types 0-63, except that restart markers are present in the JPEG data and a Restart Marker header appears immediately after the main JPEG header. Types 128-255 are free to be dynamically defined by a session setup protocol (which is beyond the scope of this document) 🦄.
3.1.4. Q: 8 bits 🎨
The Q field defines the quantization tables for this frame. Q values 0-127 indicate that the quantization tables are computed using an algorithm determined by the Type field (see below). Q values 128-255 indicate that a Quantization Table header appears after the main JPEG header (and the Restart Marker header, if present) in the first packet of the frame (fragment offset 0). This header can be used to explicitly specify the quantization tables in-band 🖌️.
3.1.5. Width: 8 bits 📏
This field encodes the width of the image in 8-pixel multiples (for example, a width of 40 denotes an image 320 pixels wide). The maximum width is 2040 pixels 🖼️.
3.1.6. Height: 8 bits 📐
This field encodes the height of the image in 8-pixel multiples (for example, a height of 30 denotes an image 240 pixels tall). When encoding interlaced video, this is the height of a video field, since fields are individually JPEG encoded. The maximum height is 2040 pixels 🌟.
3.1.7. Restart Marker header 🔁
okay so this header MUST be there right after the main JPEG header whenever you're using types 64-127 💅 It's the extra info a receiver needs to decode a data stream that has restart markers in it 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Restart Interval |F|L| Restart Count |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The Restart Interval field is how many MCUs show up between restart markers. It's the same 16 bit value you'd find in the DRI marker segment of a JFIF header. This value MUST NOT be zero, bestie 🙅♀️
If the restart intervals in a frame aren't guaranteed to line up with packet boundaries, then the F (first) and L (last) bits MUST be set to 1 and the Restart Count MUST be set to 0x3FFF. That's the signal that a receiver MUST reassemble the entire frame before decoding it 🫶
To support partial frame decoding, the frame gets broken into "chunks," and each chunk holds an integral number of restart intervals. The Restart Count field holds the position of the first restart interval in the current "chunk," so receivers know which part of the frame they're looking at 🦋 A Restart Interval value SHOULD be picked so a chunk fits completely inside a single packet. When that happens, both the F and L bits of the packet are set to 1. But if a chunk has to be spread across multiple packets, the F bit is set to 1 in the first packet of the chunk (and only that one), and the L bit is set to 1 in the last packet of the chunk (and only that one) 😭
3.1.8. Quantization Table header 🎀
This header MUST be present after the main JPEG header (and after the Restart Marker header, if there is one) when using Q values 128-255. It's how you send the quantization tables tied to that Q value in-band 🌈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MBZ | Precision | Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Quantization Table Data |
| ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The Length field is the length in bytes of the quantization table data that follows. The Length field MAY be set to zero to say there's no quantization table data in this frame. See section 4.2 for more info. If the Length field in a received packet is bigger than the number of bytes left, the packet MUST be discarded 💀
When table data is included, how many tables you get depends on the JPEG type field. For example, type 0 uses two tables: one for the luminance component and one shared by the chrominance components. Each table is an array of 64 values in zig-zag order, the same format as a JFIF DQT marker segment 🍓
For each quantization table present, a bit in the Precision field tells you the size of that table's coefficients. If the bit is zero, the coefficients are 8 bits, giving a table length of 64 bytes. If the bit is one, the coefficients are 16 bits, giving a table length of 128 bytes. For 16 bit tables, the coefficients are presented in network byte order. The rightmost bit in the Precision field (bit 15 in the diagram above) corresponds to the first table, and each additional table uses the next bit to the left. Bits beyond the ones for the tables the type needs MUST be ignored 🙈
For Q values from 128 to 254, the Q value to quantization table data mapping MUST be static. That means receivers are guaranteed they only need to read the table data once to correctly decode frames sent with that Q value. A Q value of 255 means the mapping is dynamic and can change on every frame. Decoders MUST NOT depend on any previous version of the tables, and they need to reload these tables on every frame. Packets MUST NOT contain Q = 255 and Length = 0 💅
3.1.9. JPEG Payload 🧁
The data after the RTP/JPEG headers is an entropy-coded segment made of a single scan. The scan header isn't there. It's inferred from the RTP/JPEG header instead. The scan ends either implicitly (the point where the image is fully parsed) or explicitly with an EOI marker. The scan may be padded out to any length with undefined bytes. (Some existing hardware codecs add extra lines at the bottom of a video frame, and cutting those out would need a Huffman-decoding pass over the data 🙃)
The type code decides whether restart markers are present. If a type supports restart markers, the packet MUST contain a non-zero Restart Interval value in a Restart Marker header, and restart markers MUST appear on byte aligned boundaries beginning with an 0xFF between MCUs at that interval. Additional 0xFF bytes MAY appear between restart intervals. This can be used in the packetization process to align data to something like a word boundary for more efficient copying ✨ Restart markers MUST NOT appear anywhere else in the JPEG payload. Types that don't support restart markers MUST NOT contain restart markers anywhere in the JPEG payload. All packets MUST contain a "stuffed" 0x00 byte following any true 0xFF byte generated by the entropy coder [1, Sec. B.1.1.5] 🔥
4. Discussion 💬
4.1. The Type Field 🌸
The Type field defines the abbreviated table-specification and the additional JFIF-style parameters that JPEG doesn't define, since they aren't present in the body of the transmitted JPEG data 🎀
Three ranges of the type field are defined right now. Types 0-63 are reserved as fixed, well-known mappings, to be defined by this document and future revisions of it. Types 64-127 are the same as types 0-63, except restart markers are present in the JPEG data and a Restart Marker header appears right after the main JPEG header. Types 128-255 are free to be dynamically defined by a session setup protocol (which is beyond the scope of this document) 🦋
Of the first group of fixed mappings, types 0 and 1 are defined so far, along with the matching types 64 and 65 that say restart markers are present. They correspond to an abbreviated table-specification for the "Baseline DCT sequential" mode, 8-bit samples, square pixels, three components in the YUV color space, standard Huffman tables as defined in [1, Annex K.3], and a single interleaved scan with a scan component selector for components 1, 2, and 3 in that order. The Y, U, and V color planes are component numbers 1, 2, and 3, respectively. Component 1 (the luminance plane) uses Huffman table number 0 and quantization table number 0 (defined below), and components 2 and 3 (the chrominance planes) use Huffman table number 1 and quantization table number 1 (defined below) 🧚
Type numbers 2-5 are reserved and SHOULD NOT be used. Apps based on earlier versions of this document (RFC 2035) should be updated to indicate the presence of restart markers with type 64 or 65 and the Restart Marker header 🙈
The two RTP/JPEG types currently defined are below:
horizontal vertical Quantization
types component samp. fact. samp. fact. table number
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | 1 (Y) | 2 | 1 | 0 |
| 0, 64 | 2 (U) | 1 | 1 | 1 |
| | 3 (V) | 1 | 1 | 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | 1 (Y) | 2 | 2 | 0 |
| 1, 65 | 2 (U) | 1 | 1 | 1 |
| | 3 (V) | 1 | 1 | 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+These sampling factors mean the chrominance components of type 0 video are downsampled horizontally by 2 (often called 4:2:2), while the chrominance components of type 1 video are downsampled both horizontally and vertically by 2 (often called 4:2:0) 📺
Types 0 and 1 can be used to carry both progressively scanned and interlaced image data. That's encoded using the Type-specific field in the main JPEG header. These values are defined:
0: Image is progressively scanned. On a computer monitor, it can be displayed as-is at the specified width and height 🖥️1: Image is an odd field of an interlaced video signal. The height specified in the main JPEG header is half of the height of the entire displayed image. This field should be de-interlaced with the even field following it, such that lines from each of the images alternate. Corresponding lines from the even field should appear just above those same lines from the odd field 🦓2: Image is an even field of an interlaced video signal 🌙3: Image is a single field from an interlaced video signal, but it should be displayed full frame as if it were received as both the odd & even fields of the frame. On a computer monitor, each line in the image should be displayed twice, doubling the height of the image 👯♀️
Appendix B contains C source code for transforming the RTP/JPEG header parameters into the JPEG frame and scan headers that are absent from the data payload 💖
4.2. The Q Field 🎀
For JPEG types 0 and 1 (and their corresponding types 64 and 65), Q values between 1 and 99 inclusive are defined as follows. Other values less than 128 are reserved. Additional types are encouraged to use this definition if applicable 🫶
Both type 0 and type 1 JPEG require two quantization tables. These tables are calculated as follows. For 1 <= Q <= 99, the Independent JPEG Group's formula [5] is used to produce a scale factor S:
S = 5000 / Q for 1 <= Q <= 50
= 200 - 2 * Q for 51 <= Q <= 99This value is then used to scale Tables K.1 and K.2 from [1] (saturating each value to 8 bits) to give quantization table numbers 0 and 1, respectively. C source code is provided in Appendix A to compute these tables 🧮
For Q values 128-255, dynamically defined quantization tables are used. These tables may be specified either in-band or out of band by something like a session setup protocol, but the Quantization Table header MUST be present in the first packet of every frame. When the tables are specified out of band, they may be omitted from the packet by setting the Length field in this header to 0 🌈
When the quantization tables are sent in-band, they need not be sent with every frame. Like the out of band case, frames which do not contain tables will have a Quantization Table header with a Length field of 0. While this does decrease the overhead of including the tables, new receivers will be unable to properly decode frames from the time they start up until they receive the tables 🥺
4.3. Fragmentation and Reassembly 🧩
Okay bestie, JPEG frames can be huge, so they often have to get chopped into pieces 🔪. Frames SHOULD get split into packets in a way that avoids fragmenting again at a lower level. If you want partial frame decoding to be possible, frames SHOULD be split so each packet holds a whole number of restart intervals (more on those below) 💖.
Every packet in one frame MUST carry the same timestamp 🕰️, and the RTP marker bit MUST be set on the last packet of the frame 🏁. Each packet's fragment offset field holds the byte offset of its payload inside the original frame 📍. The packets of a frame SHOULD be sent in order, and the fragments MUST NOT overlap one another 🚫.
An entire frame can be identified as a run of packets that starts with a packet whose fragment offset is zero and ends with a packet that has the RTP marker bit set 🎀. Missing packets can be spotted either with RTP sequence numbers or with the fragment offset and length of each packet 🔍. You could reassemble without the offset field, using only the marker bit and sequence numbers, but then an efficient single-copy implementation wouldn't be possible when packets arrive out of order 😭. Also, if the last packet of the previous frame (the one with the marker bit) got dropped, a receiver couldn't always tell that the current frame is fully intact 💀.
4.4. Restart Markers 🔁
Restart markers mark a spot in the JPEG stream where the Huffman decoder and DC predictors get reset, so decoding can start partway through the frame ✨. To really take advantage of this, though, a decoder has to know which MCUs of a frame a given restart interval encodes 🧠. The original JPEG spec does have a small sequence number field in the restart markers for this, but it isn't big enough to handle losing a whole packet's worth of data at a typical network MTU size 😩. That's where the RTP/JPEG Restart Marker header comes in, carrying the extra info needed to make it work 💅.
The size of restart intervals SHOULD be chosen so that an integral number of restart intervals always fits inside a single packet. This guarantees that packets can be decoded independently of one another 🫶. If a restart interval ends up bigger than a packet, the F and L bits in the Restart Marker header can be used to fragment it, but every packet in that set must be received by the decoder before that restart interval can be decoded properly 🧩.
Once a decoder has either a single packet with both the F and L bits set, or a contiguous run of packets (going by the RTP sequence number) that starts with an F bit and ends with an L bit, it can start decoding 🚀. The MCU position at the start of the data is worked out by multiplying the Restart Count value by the Restart Interval value 🧮. A packet (or a group of packets, as marked by the F and L bits) may hold any number of consecutive restart intervals 🌈.
To support encoders that put restart markers in their frames but can't fragment the data this way, the Restart Count field may be set to 0x3FFF with the F and L bits both set to 1 🦋. This tells decoders that the entire frame has to be reassembled before decoding it 💜.
5. Security Considerations 🔐
Okay, this part is serious, so listen up 👀. RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [6], and any appropriate RTP profile (for example [7]). This means confidentiality of the media streams is achieved by encryption 🔒. Because the data compression used with this payload format is applied end-to-end, encryption may be performed after compression, so there's no conflict between the two operations 🫶.
There's a potential denial-of-service threat for data encodings that use compression techniques with non-uniform receiver-end computational load 😱. An attacker can inject pathological datagrams into the stream that are complex to decode and overload the receiver 💀. However, this encoding does not exhibit any significant non-uniformity, so that's not a big deal here 🙌.
Another potential denial-of-service threat exists around the fragmentation mechanism described here 🚨. Receivers should be prepared to limit the total amount of data associated with assembling received frames, so they don't run out of resources 🧯.
As with any IP-based protocol, a receiver may in some circumstances get overloaded just from receiving too many packets, whether they're wanted or not 📦. Network-layer authentication may be used to discard packets from undesired sources, but the processing cost of the authentication itself may be too high 😮💨. In a multicast environment, pruning of specific sources will be implemented in a future version of IGMP [8] and in multicast routing protocols, so a receiver can choose which sources are allowed to reach it 🦋.
A security review of this payload format found no additional considerations beyond those in the RTP specification ✅.
6. Authors' Addresses 📮
Here are the lovely humans who wrote this, with their addresses. Slay for showing up 👑.
Lance M. Berc
Systems Research Center
Digital Equipment Corporation
130 Lytton Ave
Palo Alto CA 94301
Phone: +1 650 853 2100
EMail: berc@pa.dec.comWilliam C. Fenner
Xerox PARC
3333 Coyote Hill Road
Palo Alto, CA 94304
Phone: +1 650 812 4816
EMail: fenner@parc.xerox.comRon Frederick
Xerox PARC
3333 Coyote Hill Road
Palo Alto, CA 94304
Phone: +1 650 812 4459
EMail: frederick@parc.xerox.comSteven McCanne
University of California at Berkeley
Electrical Engineering and Computer Science
633 Soda Hall
Berkeley, CA 94720
Phone: +1 510 642 0865
EMail: mccanne@cs.berkeley.eduPaul Stewart
Xerox PARC
3333 Coyote Hill Road
Palo Alto, CA 94304
Phone: +1 650 812 4821
EMail: stewart@parc.xerox.com7. References 📚
[1] ISO DIS 10918-1. Digital Compression and Coding of Continuous-tone Still Images (JPEG), CCITT Recommendation T.81.
[2] William B. Pennebaker, Joan L. Mitchell, JPEG: Still Image Data Compression Standard, Van Nostrand Reinhold, 1993.
[3] Gregory K. Wallace, The JPEG Sill Picture Compression Standard, Communications of the ACM, April 1991, Vol 34, No. 1, pp. 31-44.
[4] The JPEG File Interchange Format. Maintained by C-Cube Microsystems, Inc., and available in ftp://ftp.uu.net/graphics/jpeg/jfif.ps.gz.
[5] Tom Lane et. al., The Independent JPEG Group software JPEG codec. Source code available in ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6a.tar.gz.
[6] Schulzrinne, H., Casner, S., Frederick, R. and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", RFC 1889, January 1996.
[7] Schulzrinne, H., "RTP Profile for Audio and Video Conferences with Minimal Control", RFC 1890, January 1996.
[8] Fenner, W., "Internet Group Management Protocol Version 2", RFC 2236, November 1997.
[9] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[10] Kent C., and J. Mogul, "Fragmentation Considered Harmful", Proceedings of the ACM SIGCOMM '87 Workshop on Frontiers in Computer Communications Technology, August 1987.
Appendix A 🎀
The following code can be used to create a quantization table from a Q factor 🧮:
/*
* Table K.1 from JPEG spec.
*/
static const int jpeg_luma_quantizer[64] = {
16, 11, 10, 16, 24, 40, 51, 61,
12, 12, 14, 19, 26, 58, 60, 55,
14, 13, 16, 24, 40, 57, 69, 56,
14, 17, 22, 29, 51, 87, 80, 62,
18, 22, 37, 56, 68, 109, 103, 77,
24, 35, 55, 64, 81, 104, 113, 92,
49, 64, 78, 87, 103, 121, 120, 101,
72, 92, 95, 98, 112, 100, 103, 99
};
/*
* Table K.2 from JPEG spec.
*/
static const int jpeg_chroma_quantizer[64] = {
17, 18, 24, 47, 99, 99, 99, 99,
18, 21, 26, 66, 99, 99, 99, 99,
24, 26, 56, 99, 99, 99, 99, 99,
47, 66, 99, 99, 99, 99, 99, 99,
99, 99, 99, 99, 99, 99, 99, 99,
99, 99, 99, 99, 99, 99, 99, 99,
99, 99, 99, 99, 99, 99, 99, 99,
99, 99, 99, 99, 99, 99, 99, 99
};
/*
* Call MakeTables with the Q factor and two u_char[64] return arrays
*/
void
MakeTables(int q, u_char *lqt, u_char *cqt)
{
int i;
int factor = q;
if (q < 1) factor = 1;
if (q > 99) factor = 99;
if (q < 50)
q = 5000 / factor;
else
q = 200 - factor*2;
for (i=0; i < 64; i++) {
int lq = (jpeg_luma_quantizer[i] * q + 50) / 100;
int cq = (jpeg_chroma_quantizer[i] * q + 50) / 100;
/* Limit the quantizers to 1 <= q <= 255 */
if (lq < 1) lq = 1;
else if (lq > 255) lq = 255;
lqt[i] = lq;
if (cq < 1) cq = 1;
else if (cq > 255) cq = 255;
cqt[i] = cq;
}
}Appendix B 🌸
The following routines can be used to create the JPEG marker segments that correspond to the table-specification data missing from the RTP/JPEG body 🧚.
u_char lum_dc_codelens[] = {
0, 1, 5, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0,
};
u_char lum_dc_symbols[] = {
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
};
u_char lum_ac_codelens[] = {
0, 2, 1, 3, 3, 2, 4, 3, 5, 5, 4, 4, 0, 0, 1, 0x7d,
};
u_char lum_ac_symbols[] = {
0x01, 0x02, 0x03, 0x00, 0x04, 0x11, 0x05, 0x12,
0x21, 0x31, 0x41, 0x06, 0x13, 0x51, 0x61, 0x07,
0x22, 0x71, 0x14, 0x32, 0x81, 0x91, 0xa1, 0x08,
0x23, 0x42, 0xb1, 0xc1, 0x15, 0x52, 0xd1, 0xf0,
0x24, 0x33, 0x62, 0x72, 0x82, 0x09, 0x0a, 0x16,
0x17, 0x18, 0x19, 0x1a, 0x25, 0x26, 0x27, 0x28,
0x29, 0x2a, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39,
0x3a, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49,
0x4a, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59,
0x5a, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68, 0x69,
0x6a, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79,
0x7a, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89,
0x8a, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98,
0x99, 0x9a, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7,
0xa8, 0xa9, 0xaa, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6,
0xb7, 0xb8, 0xb9, 0xba, 0xc2, 0xc3, 0xc4, 0xc5,
0xc6, 0xc7, 0xc8, 0xc9, 0xca, 0xd2, 0xd3, 0xd4,
0xd5, 0xd6, 0xd7, 0xd8, 0xd9, 0xda, 0xe1, 0xe2,
0xe3, 0xe4, 0xe5, 0xe6, 0xe7, 0xe8, 0xe9, 0xea,
0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, 0xf8,
0xf9, 0xfa,
};
u_char chm_dc_codelens[] = {
0, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0,
};
u_char chm_dc_symbols[] = {
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
};
u_char chm_ac_codelens[] = {
0, 2, 1, 2, 4, 4, 3, 4, 7, 5, 4, 4, 0, 1, 2, 0x77,
};u_char chm_ac_symbols[] = { 0x00, 0x01, 0x02, 0x03, 0x11, 0x04, 0x05, 0x21, 0x31, 0x06, 0x12, 0x41, 0x51, 0x07, 0x61, 0x71, 0x13, 0x22, 0x32, 0x81, 0x08, 0x14, 0x42, 0x91, 0xa1, 0xb1, 0xc1, 0x09, 0x23, 0x33, 0x52, 0xf0, 0x15, 0x62, 0x72, 0xd1, 0x0a, 0x16, 0x24, 0x34, 0xe1, 0x25, 0xf1, 0x17, 0x18, 0x19, 0x1a, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68, 0x69, 0x6a, 0x73, 0x74, 0x75, 0x76, 0x77, 0x78, 0x79, 0x7a, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89, 0x8a, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9a, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, 0xa8, 0xa9, 0xaa, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, 0xb8, 0xb9, 0xba, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, 0xc8, 0xc9, 0xca, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, 0xd8, 0xd9, 0xda, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, 0xe8, 0xe9, 0xea, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, 0xf8, 0xf9, 0xfa, };
u_char *
MakeQuantHeader(u_char *p, u_char *qt, int tableNo)
{
*p++ = 0xff;
*p++ = 0xdb; /* DQT */
*p++ = 0; /* length msb */
*p++ = 67; /* length lsb */
*p++ = tableNo;
memcpy(p, qt, 64);
return (p + 64);
}
u_char *
MakeHuffmanHeader(u_char *p, u_char *codelens, int ncodes,
u_char *symbols, int nsymbols, int tableNo,
int tableClass)
{
*p++ = 0xff;
*p++ = 0xc4; /* DHT */
*p++ = 0; /* length msb */
*p++ = 3 + ncodes + nsymbols; /* length lsb */
*p++ = (tableClass << 4) | tableNo;
memcpy(p, codelens, ncodes);
p += ncodes;
memcpy(p, symbols, nsymbols);
p += nsymbols;
return (p);
}
u_char *
MakeDRIHeader(u_char *p, u_short dri) {
*p++ = 0xff;
*p++ = 0xdd; /* DRI */
*p++ = 0x0; /* length msb */
*p++ = 4; /* length lsb */
*p++ = dri >> 8; /* dri msb */
*p++ = dri & 0xff; /* dri lsb */
return (p);
}
/*
* Arguments:
* type, width, height: as supplied in RTP/JPEG header
* lqt, cqt: quantization tables as either derived from
* the Q field using MakeTables() or as specified
* in section 4.2.
* dri: restart interval in MCUs, or 0 if no restarts.
*
* p: pointer to return area
*
* Return value:
* The length of the generated headers.
*
* Generate a frame and scan headers that can be prepended to the
* RTP/JPEG data payload to produce a JPEG compressed image in
* interchange format (except for possible trailing garbage and
* absence of an EOI marker to terminate the scan).
*/
int MakeHeaders(u_char *p, int type, int w, int h, u_char *lqt,
u_char *cqt, u_short dri)
{
u_char *start = p;
/* convert from blocks to pixels */
w <<= 3;
h <<= 3;
*p++ = 0xff;
*p++ = 0xd8; /* SOI */
p = MakeQuantHeader(p, lqt, 0);
p = MakeQuantHeader(p, cqt, 1);
if (dri != 0)
p = MakeDRIHeader(p, dri);
*p++ = 0xff;
*p++ = 0xc0; /* SOF */
*p++ = 0; /* length msb */
*p++ = 17; /* length lsb */
*p++ = 8; /* 8-bit precision */
*p++ = h >> 8; /* height msb */
*p++ = h; /* height lsb */
*p++ = w >> 8; /* width msb */
*p++ = w; /* wudth lsb */
*p++ = 3; /* number of components */
*p++ = 0; /* comp 0 */
if (type == 0)
*p++ = 0x21; /* hsamp = 2, vsamp = 1 */
else
*p++ = 0x22; /* hsamp = 2, vsamp = 2 */
*p++ = 0; /* quant table 0 */
*p++ = 1; /* comp 1 */
*p++ = 0x11; /* hsamp = 1, vsamp = 1 */
*p++ = 1; /* quant table 1 */
*p++ = 2; /* comp 2 */
*p++ = 0x11; /* hsamp = 1, vsamp = 1 */
*p++ = 1; /* quant table 1 */
p = MakeHuffmanHeader(p, lum_dc_codelens,
sizeof(lum_dc_codelens),
lum_dc_symbols,
sizeof(lum_dc_symbols), 0, 0);
p = MakeHuffmanHeader(p, lum_ac_codelens,
sizeof(lum_ac_codelens),
lum_ac_symbols,
sizeof(lum_ac_symbols), 0, 1);
p = MakeHuffmanHeader(p, chm_dc_codelens,
sizeof(chm_dc_codelens),
chm_dc_symbols,
sizeof(chm_dc_symbols), 1, 0);
p = MakeHuffmanHeader(p, chm_ac_codelens,
sizeof(chm_ac_codelens),
chm_ac_symbols,
sizeof(chm_ac_symbols), 1, 1);
*p++ = 0xff;
*p++ = 0xda; /* SOS */
*p++ = 0; /* length msb */
*p++ = 12; /* length lsb */
*p++ = 3; /* 3 components */
*p++ = 0; /* comp 0 */
*p++ = 0; /* huffman table 0 */
*p++ = 1; /* comp 1 */
*p++ = 0x11; /* huffman table 1 */
*p++ = 2; /* comp 2 */
*p++ = 0x11; /* huffman table 1 */
*p++ = 0; /* first DCT coeff */
*p++ = 63; /* last DCT coeff */
*p++ = 0; /* sucessive approx. */
return (p - start);
};Appendix C ✨
Okay so this bit is where the spec gets its hands dirty 💅 It walks through a routine that shows how RTP/JPEG packets get fragmented and how their headers get built, step by step. Super useful, bestie 🎀
Quick heads up before you dive in: the structs below only work on 32-bit big-endian machines (most significant byte first) 🙈 Bit fields are packed tightly in big-endian bit order with zero extra padding. If you want something portable, you'll need to make changes, so don't just copy-paste and pray 💀
/*
* RTP data header from RFC1889
*/
typedef struct {
unsigned int version:2; /* protocol version */
unsigned int p:1; /* padding flag */
unsigned int x:1; /* header extension flag */
unsigned int cc:4; /* CSRC count */
unsigned int m:1; /* marker bit */
unsigned int pt:7; /* payload type */
u_int16 seq; /* sequence number */
u_int32 ts; /* timestamp */
u_int32 ssrc; /* synchronization source */
u_int32 csrc[1]; /* optional CSRC list */
} rtp_hdr_t;
#define RTP_HDR_SZ 12
/* The following definition is from RFC1890 */
#define RTP_PT_JPEG 26
struct jpeghdr {
unsigned int tspec:8; /* type-specific field */
unsigned int off:24; /* fragment byte offset */
u_int8 type; /* id of jpeg decoder params */
u_int8 q; /* quantization factor (or table id) */
u_int8 width; /* frame width in 8 pixel blocks */
u_int8 height; /* frame height in 8 pixel blocks */
};
struct jpeghdr_rst {
u_int16 dri;
unsigned int f:1;
unsigned int l:1;
unsigned int count:14;
};
struct jpeghdr_qtable {
u_int8 mbz;
u_int8 precision;
u_int16 length;
};
#define RTP_JPEG_RESTART 0x40
/* Procedure SendFrame:
*
* Arguments:
* start_seq: The sequence number for the first packet of the current
* frame.
* ts: RTP timestamp for the current frame
* ssrc: RTP SSRC value
* jpeg_data: Huffman encoded JPEG scan data
* len: Length of the JPEG scan data
* type: The value the RTP/JPEG type field should be set to
* typespec: The value the RTP/JPEG type-specific field should be set
* to
* width: The width in pixels of the JPEG image
* height: The height in pixels of the JPEG image
* dri: The number of MCUs between restart markers (or 0 if there
* are no restart markers in the data
* q: The Q factor of the data, to be specified using the Independent
* JPEG group's algorithm if 1 <= q <= 99, specified explicitly
* with lqt and cqt if q >= 128, or undefined otherwise.
* lqt: The quantization table for the luminance channel if q >= 128
* cqt: The quantization table for the chrominance channels if
* q >= 128
*
* Return value:
* the sequence number to be sent for the first packet of the next
* frame.
*
* The following are assumed to be defined:
*
* PACKET_SIZE - The size of the outgoing packet
* send_packet(u_int8 *data, int len) - Sends the packet to the network
*/
u_int16 SendFrame(u_int16 start_seq, u_int32 ts, u_int32 ssrc,
u_int8 *jpeg_data, int len, u_int8 type,
u_int8 typespec, int width, int height, int dri,
u_int8 q, u_int8 *lqt, u_int8 *cqt) {
rtp_hdr_t rtphdr;
struct jpeghdr jpghdr;
struct jpeghdr_rst rsthdr;
struct jpeghdr_qtable qtblhdr;
u_int8 packet_buf[PACKET_SIZE];
u_int8 *ptr;
int bytes_left = len;
int seq = start_seq;
int pkt_len, data_len;
/* Initialize RTP header
*/
rtphdr.version = 2;
rtphdr.p = 0;
rtphdr.x = 0;
rtphdr.cc = 0;
rtphdr.m = 0;
rtphdr.pt = RTP_PT_JPEG;
rtphdr.seq = start_seq;
rtphdr.ts = ts;
rtphdr.ssrc = ssrc;
/* Initialize JPEG header
*/
jpghdr.tspec = typespec;
jpghdr.off = 0;
jpghdr.type = type | ((dri != 0) ? RTP_JPEG_RESTART : 0);
jpghdr.q = q;
jpghdr.width = width / 8;
jpghdr.height = height / 8;
/* Initialize DRI header
*/
if (dri != 0) {
rsthdr.dri = dri;
rsthdr.f = 1; /* This code does not align RIs */
rsthdr.l = 1;
rsthdr.count = 0x3fff;
}
/* Initialize quantization table header
*/
if (q >= 128) {
qtblhdr.mbz = 0;
qtblhdr.precision = 0; /* This code uses 8 bit tables only */
qtblhdr.length = 128; /* 2 64-byte tables */
}
while (bytes_left > 0) {
ptr = packet_buf + RTP_HDR_SZ;
memcpy(ptr, &jpghdr, sizeof(jpghdr));
ptr += sizeof(jpghdr);
if (dri != 0) {
memcpy(ptr, &rsthdr, sizeof(rsthdr));
ptr += sizeof(rsthdr);
}
if (q >= 128 && jpghdr.off == 0) {
memcpy(ptr, &qtblhdr, sizeof(qtblhdr));
ptr += sizeof(qtblhdr);
memcpy(ptr, lqt, 64);
ptr += 64;
memcpy(ptr, cqt, 64);
ptr += 64;
}
data_len = PACKET_SIZE - (ptr - packet_buf);
if (data_len >= bytes_left) {
data_len = bytes_left;
rtphdr.m = 1;
}
memcpy(packet_buf, &rtphdr, RTP_HDR_SZ);
memcpy(ptr, jpeg_data + jpghdr.off, data_len);
send_packet(packet_buf, (ptr - packet_buf) + data_len);
jpghdr.off += data_len;
bytes_left -= data_len;
rtphdr.seq++;
}
return rtphdr.seq;
}Appendix D ✨
okay so this appendix is the "what changed since last time" recap 💅 it's comparing this doc to its predecessor, RFC 2035. The big goal was to avoid breaking existing implementations, so newer code can still decode a lot of the old conventions. Still, sticking with the old stuff in new code is strongly discouraged 🙈
- Interlaced video is in now 🎞️ Types 0 and 1 got upgraded so they can encode interlaced video images, using 2 bits of the type-specific field. Details are in section 4.1 ✨
- Quantization tables are more flexible 🌸 The working group wanted more say over the JPEG quantization tables, so this memo lets you spell out the table coefficients explicitly with an optional Quantization Table header. Check sections 3.1.8 and 4.2 for the full tea ☕
- Restart markers got a glow-up 👑 In RFC 2035, restart marker info was crammed into the Type field, which made adding new types a nightmare. This memo moves the restart marker indication to one particular bit in the Type field, and adds an optional header for the extra info. That frees up the type-specific field to do its actual job. Partial frame decoding also got sturdier against packet loss, so see sections 3.1.7 and 4.4 🦋
Full Copyright Statement ✨
Copyright (C) The Internet Society (1998). All Rights Reserved. 🌈
You're allowed to copy and share this document and its translations with others. You can also write derivative works that comment on or explain it, or that help people implement it, and publish and distribute those however you like, in whole or in part, with no restrictions. The one catch is that the copyright notice and this paragraph have to stay on every copy and derivative work. The document itself can't be modified in any way, like removing the copyright notice or the references to the Internet Society or other Internet organizations. The exceptions are when you need it for developing Internet standards (then you follow the copyright procedures in the Internet Standards process), or when you're translating it into languages other than English.
These permissions are perpetual, and the Internet Society or its successors or assigns will never revoke them 🫶
This document and the information in it are provided "AS IS". THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT USING THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS, OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 💀
📦 where this page comes from
- ietf: rfc2435 1998-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms