RFC 2190
RTP Payload Format for H.263 Video Streams
RFC 2190: RTP Payload Format for H.263 Video Streams 🎀
okay bestie, this one's the spec for squishing H.263 video into RTP packets 🦋 It's a Standards Track doc from September 1997, written by C. Zhu at Intel Corp. 💖 The header doesn't say whether it obsoletes or updates anything, so I'm not going to guess 🙈
Status of This Memo ✨
okay so this doc is an Internet standards track protocol for the Internet community, and it's asking for discussion and suggestions for improvements 💬 If you want the current standardization state of this protocol, check the latest edition of the "Internet Official Protocol Standards" (STD 1) 📚 And yes, the memo can be distributed freely, no limits 🌸
Abstract 🦋
so the big idea is how to encapsulate an H.263 bitstream inside the Real-Time Transport Protocol (RTP) 🎀 There are three modes for the H.263 payload header, and you pick one based on the network packet size you want and which H.263 encoding options you're using 💅 The shortest header, mode A, can fragment at Group of Block (GOB) boundaries. The longer headers, mode B and mode C, can fragment at Macroblock (MB) boundaries 🧚
1. Introduction ✨
this section is the "what is this even for" part 💖 It describes a way to packetize an H.263 video stream so it can be transported using RTP [1]. The H.263 video stream is defined by ITU-T Recommendation H.263 (referred to as H.263 in this document) [4], and it's built for video coding at very low data rates 🐣 RTP is defined by the Internet Engineering Task Force (IETF) to give end-to-end network transport functions for apps sending real-time data over multicast or unicast network services 🌈
2. Definitions 📖
the following definitions apply in this document, so let's get the vocab down 💜
- CIF: Common Intermediate Format. For H.263, a CIF picture has 352 x 288 pixels for luminance, and 176 x 144 pixels for chrominance. 🎀
- QCIF: Quarter CIF source format with 176 x 144 pixels for luminance and 88 x 72 pixels for chrominance. 🌸
- Sub-QCIF: picture source format with 128 x 96 pixels for luminance and 64 x 48 pixels for chrominance. 🐥
- 4CIF: Picture source format with 704 x 576 pixels for luminance and 352 x 288 pixels for chrominance. 🔥
- 16CIF: Picture source format with 1408 x 1152 pixels for luminance and 704 x 576 pixels for chrominance. 👑
- GOB: For H.263, a Group of Blocks (GOB) consists of k*16 lines, where k depends on the picture format (k=1 for QCIF, CIF and sub-QCIF; k=2 for 4CIF and k=4 for 16CIF). 🧚
- MB: A macroblock (MB) contains four blocks of luminance and the spatially corresponding two blocks of chrominance. Each block consists of 8x8 pixels. For example, there are eleven MBs in a GOB in QCIF format and twenty two MBs in a GOB in CIF format. 🍓
3. Design Issues for Packetizing H.263 Bitstreams 💅
okay so H.263 is based on ITU-T Recommendation H.261 [2] (referred to as H.261 in this document) 🦋 Compared to H.261, H.263 uses similar techniques to cut down on both temporal and spatial redundancy, BUT there are a few major differences between the two algorithms that seriously change how you design a packetization scheme 😭 This section sums those up 💖
3.1. Optional Features of H.263 🌈
in addition to the basic source coding algorithms, H.263 supports four negotiable coding options to improve performance: Advanced Prediction, PB-frames, Syntax-based Arithmetic Coding, and Unrestricted Motion Vectors. They can be used in any combination. 🎀
- Advanced Prediction (AP): one or four motion vectors can be used for some macroblocks in a frame. This makes recovering from packet loss harder, because more redundant information has to be kept at the start of a packet when you fragment at a macroblock boundary. 💀
- PB-frames: two frames (a P frame and a B frame) are coded into one bitstream, with macroblocks from the two frames interleaved. From a packetization point of view, a MB from the P frame and a MB from the B frame have to be treated together, because each MB for the B frame is coded based on the corresponding MB for the P frame. There must be a way to make sure both frames render in the right order. Also, if part of this combined bitstream gets lost, it affects both frames, and possibly more. 🫶
- Syntax-based Arithmetic Coding (SAC): when SAC is used, the run-value pair you get after quantizing Discrete Cosine Transform (DCT) coefficients is coded differently from Huffman codes, but the macroblock hierarchy stays the same. Context variables only sync up after fixed length codes in the bitstream, so any fragmentation that starts at a variable length code will make decoding hard when packets get lost, unless you carry the values of all the context variables in each H.263 payload header. 🙈
- Unrestricted Motion Vectors: this feature allows a large range of motion vectors, which helps motion compensation for inter-coded pictures. It also affects packetization, because it uses a larger range of motion vectors than normal. 🔥
so, to decode received packets properly without depending on previous packets, the use of these optional features gets signaled in the H.263 payload header, as described in Section 5 ✨
3.2. GOB Numbering 🔢
in H.263, each picture is split into groups of blocks (GOB) 🧩 GOBs are numbered in a vertical scan of the picture, starting from the top GOB and ending at the bottom GOB. In H.261, a GOB is three rows of 16x16 MB for QCIF, and three half-rows of MBs for CIF. In H.263, a GOB is divided into macroblocks too, and the macroblock definitions are the same as in H.261 🌸
Each GOB in H.263 can have a fixed GOB header, but using the header is optional. If the GOB header is present, it may or may not start on a byte boundary. Byte alignment can be achieved by proper bit stuffing by the encoder, but the H.263 bitstream specification [4] doesn't require it 🐥
in summary, a GOB in H.263 is defined and coded with finer granularity but with the same source format, which gives you more flexibility for packetization than H.261 does 💜
3.3. Motion Vector Encoding 🏃♀️
differential coding is used to code motion vectors as variable length codes. Unlike H.261, where each motion vector is predicted from the previous MB in the GOB, H.263 uses a more flexible prediction scheme, where one or three candidate predictors could be used depending on whether GOB headers are present 🎀
if the GOB header is present in a GOB, motion vectors are coded with reference to MBs in the current GOB only. If a GOB header is not present in the current GOB, three motion vectors must be available to decode one macroblock, and two of them might come from the previous GOB. To correctly decode a whole inter-coded GOB, all the motion vectors for MBs in the previous GOB must be available to compute the predictors, or the predictors themselves must be present. The optional use of three motion vector predictors can be a major problem for a packetization scheme like the one defined for H.261 when you packetize at MB boundaries [5] 😭
Consider the case where a packet starts with a MB but the GOB header is not present. If the previous packet is lost, then all the motion vectors needed to predict the motion vectors for the MBs in the current GOB are gone. To decode the received MBs correctly, all the motion vectors for the previous GOB, or the motion vector predictors, would have to be duplicated at the start of the packet. That duplication would be very expensive and unacceptable in terms of bandwidth overhead 💸
the encoding strategy of each H.263 CODEC (CODer and DECoder) implementation is beyond the scope of this document, even though it has a significant effect on visual quality when packets are lost. However, we strongly recommend using the GOB header for every GOB at the beginning of a packet to address this problem 🫶
there are similar problems from cross-GOB data dependency related to motion vectors, but the GOB header can't fix those 🙅♀️ For 16CIF and 4CIF pictures, a GOB contains more than one row of MBs. If a GOB can't fit in one RTP packet, and the first packet containing the GOB header is lost, then MBs in the second packet can't compute motion vectors correctly, because they're coded relative to data in the lost packet. Similarly, when OBMC (Overlapped Block Motion Compensation) [4] in Advanced Prediction mode is used, motion compensation for some MBs in one GOB could use motion vectors of MBs in a previous GOB, regardless of whether a GOB header is present. When the MBs used to decode received MBs are lost, those received MBs can't be decoded correctly. Each implementation of the method described in this document should take these limitations into account 🧠
3.4. Macroblock Address 📍
as H.261 specifies, a macroblock address (MBA) is encoded with a variable length code to show where a macroblock sits within a group of MBs in H.261 bitstreams. H.263 doesn't code the MBA explicitly, but the macroblock address within a GOB is needed to recover from packet loss when you fragment at MB boundaries. So this information must be included in the H.263 payload header for the modes that allow packetization at MB boundaries (mode B and mode C, as described in Section 5) 🌟
4. Usage of RTP 🎬
when you transmit H.263 video streams over the Internet, the encoder's output can be packetized directly 💖 For every video frame, the H.263 bitstream itself is carried in the RTP payload without alteration, including the picture start code and the entire picture header, plus any fixed length codes and variable length codes 🎀 The encoder's output is packetized without adding the framing information specified by H.223 [6], so multiplexing audio and video signals in the same packet isn't accommodated, because UDP and RTP already give a much more efficient way to multiplex 🦋
RTP doesn't guarantee reliable and orderly data delivery, so a packet might get lost in the network 😭 To get a best-effort recovery from packet loss, the decoder needs help to keep decoding the other packets it receives. So it's desirable to be able to process each packet independently of the others 🧩 Some frame level information is included in each packet, like source format and flags for optional features, to help the decoder operate correctly and efficiently when packets are lost 🫶 The flags for H.263 optional features also tell session management tools which coding options the H.263 video bitstreams use 🌈
H.263 video bitstreams will be carried as payload data within RTP packets. A new H.263 payload header is defined in section 5 on the H.263 payload header. This section defines how the RTP fixed header and H.263 video packet structure get used 👑
4.1 RTP Header Usage 🎀
okay so every RTP packet starts with a fixed RTP header [1], and only a few of its fields matter for H.263 video streams 💅
Marker bit (M bit): this one is the "end of the frame" flag. It gets set to 1 when the current packet carries the end of the current frame, and it's 0 otherwise 🦋
Payload Type (PT): the PT has to name the H.263 video payload format, using whatever value the RTP profile in use says. For example, RFC 1890 [3] 🌸
Timestamp: this encodes the sampling instant of the video frame inside the packet. If one frame is split across several packets, the timestamp can be the same on each of them ✨ For H.263 video streams the timestamp runs on a 90 kHz clock, same as H.261 video streams [5] 🧚
4.2 Video Packet Structure 🌸
each RTP packet is basically a sandwich: RTP header first, then the H.263 payload header, then the standard H.263 compressed bitstream [4] 🥪
the payload header's size changes depending on the mode, which we'll get into in the next section. Here's the layout:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| H.263 payload header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| H.263 bitstream |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+5. H.263 Payload Header 💖
5.1 Mode A 🌈
okay bestie, this is the simplest one. In mode A the bitstream gets packetized on a GOB boundary or a picture boundary. Mode A packets always start with the H.263 picture start code [4] or a GOB, but they don't have to contain complete GOBs. The mode A payload header is four bytes, and here's the layout with F=0:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|P|SBIT |EBIT | SRC |I|U|S|A|R |DBQ| TRB | TR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Mode A packets are allowed to start at a GOB boundary even when there's no GOB header for that GOB in the bitstream. Honestly though? Doing that is discouraged, because it creates dependencies across GOB boundaries, as described in Section 3.3 😬
F: 1 bit. This flag tells you the mode of the payload header. F=0 means mode A. F=1 means mode B or mode C, and the P bit decides which one 🔑
P: 1 bit. This is the optional PB-frames mode from H.263 [4]. "0" means a normal I or P frame, and "1" means PB-frames. When F=1, P also picks the mode: mode B if P=0, mode C if P=1 ✨
SBIT: 3 bits. The start bit position says how many of the most significant bits in the first data byte get ignored 🫶
EBIT: 3 bits. The end bit position says how many of the least significant bits in the last data byte get ignored 🫶
SRC: 3 bits. This is the source format. It's bits 6, 7 and 8 in PTYPE, as defined by H.263 [4], and it tells you the resolution of the current picture 📺
I: 1 bit. This is the picture coding type, bit 9 in PTYPE from H.263 [4]. "0" means intra-coded and "1" means inter-coded 🎨
U: 1 bit. This gets set to 1 if the Unrestricted Motion Vector option (bit 10 in PTYPE, H.263 [4]) was set to 1 in the current picture header. Otherwise it's 0 🏃♀️
S: 1 bit. This gets set to 1 if the Syntax-based Arithmetic Coding option (bit 11 in PTYPE, defined by H.263 [4]) was set to 1 for the current picture header. Otherwise it's 0 🧮
A: 1 bit. This gets set to 1 if the Advanced Prediction option (bit 12 in PTYPE, defined by H.263 [4]) was set to 1 for the current picture header. Otherwise it's 0 🔮
R: 4 bits. Reserved. It must be set to zero 🚫
DBQ: 2 bits. This is the differential quantization parameter. It's used to work out the quantizer for the B frame from the quantizer for the P frame, when the PB-frames option is used. The value should match DBQUANT from H.263 [4]. If the PB-frames option isn't used, set it to zero 🍓
TRB: 3 bits. This is the Temporal Reference for the B frame, as defined by H.263 [4]. Set it to zero if the PB-frames option isn't used 🕰️
TR: 8 bits. This is the Temporal Reference for the P frame, as defined by H.263 [4]. Set it to zero if the PB-frames option isn't used ⏱️
5.2 Mode B 🦋
in mode B the H.263 bitstream can be fragmented at MB boundaries. Whenever a packet starts at a MB boundary, this mode shall be used without the PB-frames option. Mode B is meant for GOBs that are bigger than the max packet size the underlying protocol allows, so you can't squeeze one or more complete GOBs into a packet 📦 This mode only works without the PB-frames option. If you need to fragment at MB boundaries with PB-frames, mode C (next section) is your friend 💜 The mode B payload header is shown below with F=1 and P=0:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|P|SBIT |EBIT | SRC | QUANT | GOBN | MBA |R |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|I|U|S|A| HMV1 | VMV1 | HMV2 | VMV2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+F, P, SBIT, EBIT, SRC, I, U, S and A all mean the same thing as they do in mode A. Here's the rest:
QUANT: 5 bits. This is the quantization value for the first MB coded at the start of the packet. Set it to 0 if the packet begins with a GOB header. It's the equivalent of GQUANT from H.263 [4] 🎚️
GOBN: 5 bits. This is the GOB number in effect at the start of the packet. GOB numbers work differently for different resolutions, so check H.263 [4] for the details 🗺️
MBA: 9 bits. This is the address of the first MB in the packet, counted from zero in scan order, within the GOB. For example, the third MB in any GOB gets MBA = 2 🔢
HMV1, VMV1: 7 bits each. These are the horizontal and vertical motion vector predictors for the first MB in this packet [4]. When four motion vectors are used for the current MB with the advanced prediction option, these are the motion vector predictors for block number 1 in the MB. Each 7-bit field encodes a motion vector predictor in half pixel resolution as a 2's complement number 🏹
HMV2, VMV2: 7 bits each. These are the horizontal and vertical motion vector predictors for block number 3 in the first MB in this packet, when four motion vectors are used with the advanced prediction option. Block number 3 needs different predictors than the other blocks in the MB, which is why these exist. These two fields aren't used when the MB only has one motion vector. Check H.263 [4] for how blocks are organized in a macroblock. Each 7-bit field encodes a motion vector predictor in half pixel resolution as a 2's complement number 🧩
R: 2 bits. Reserved. It must be set to zero 🚫
5.3 Mode C 💅
mode C is for fragmenting an H.263 bitstream at MB boundaries of P frames that use the PB-frames option. It's for GOBs that are larger than the max packet size the underlying protocol allows when PB-frames is in use 🎬 The mode C payload header is shown below with F=1 and P=1:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|P|SBIT |EBIT | SRC | QUANT | GOBN | MBA |R |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|I|U|S|A| HMV1 | VMV1 | HMV2 | VMV2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RR |DBQ| TRB | TR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The fields F, P, SBIT, EBIT, SRC, QUANT, GOBN, MBA, R, I, U, S, A, HMV1, VMV1, HMV2 and VMV2 are all defined the same as in mode B. The rest (TR, DBQ, TRB) are defined the same as in mode A, except for the RR field. RR takes 19 bits and is currently reserved, so it must be set to zero 🔒
5.4 Selection of Modes for the H.263 Payload Header 🧠
packets carrying H.263 video streams can have different modes mixed together. You've gotta pick modes carefully, based on network packet size, H.263 coding options and the underlying network protocols 🧐 Mode A must be used for packets that start with a GOB or the H.263 picture start code [4]. Mode B or C must be used whenever a packet has to start at a MB boundary. And mode B or C are needed for GOBs bigger than the network packet size 📏
we strongly recommend mode A whenever possible 💖 The big win is simplicity. The mode A payload header is smaller than the mode B and C ones, so transmission overhead drops, and those savings can be huge at very low data rates or with pretty small packet sizes 🪶
another perk of mode A is that error recovery is easier when packets get lost. A decoder's internal state can be recovered at GOB boundaries, instead of having to resync with MBs like in mode B and C. GOB headers and the picture start code are easy to spot, and their presence will normally make an H.263 decoder re-synchronize its internal states 🔄
and finally, we really need to stress this: recovering from packet loss depends on the decoder being able to use the information in the H.263 payload header inside the RTP packets 🚨
6. Limitations 🚧
the packetization method in this document applies to the 1996 version of H.263. It may not be applicable to bitstreams with features added after that 📜
Security Considerations 🔐
security issues are handled by RTP [1]. This memo doesn't bring up any additional security issues 🫶
7. Acknowledgments 💖
Huge thank yous to the people who gave the author their valuable comments 🫶: Linda S. Cline, Christian Maciocco, Mojy Mirashrafi, Phillip Lantz, Steve Casner, Gary Sullivan, and Sassan Pejhan. You're all literally the best 👑✨
8. References 📚
[1] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", RFC 1889, January 1996. This is the one that brings the RTP basics 🚀
[2] International Telecommunication Union. Video Codec for Audiovisual Services at p x 64 kbits/s, ITU-T Recommendation H.261, 1993. The OG video codec 🎀
[3] Schulzrinne, H., "RTP Profile for Audio and Video Conference with Minimal Control", RFC 1890, January 1996. The profile that sets the vibe for audio and video conferences 🦋
[4] International Telecommunication Union. Video Coding for Low Bitrate Communication, ITU-T Recommendation H.263, 1996. The H.263 bible, obsessed 🌸
[5] Turletti, T., and C. Huitema, "RTP Payload Format for H.261 Video Streams", RFC 2032, October 1996. The payload format that makes H.261 work over RTP 💅
[6] International Telecommunication Union. Multiplexing Protocol for Low Bitrate Multimedia Communication, ITU-T Recommendation H.223, 1995. Multiplexing, but make it iconic 🧚
7. Author's Address 🌈
Here's where you can find the author, Chad Zhu, at Intel 💜:
C. "Chad" Zhu
Mail Stop: JF3-202
Intel Corporation
2111 N.E. 25th Avenue
Hillsboro, OR 97124
USA
EMail: czhu@ibeam.intel.com
Phone: (503) 264-6008
Fax: (503) 264-1805That's the whole thing, bestie! 🦋💅
📦 where this page comes from
- ietf: rfc2190 1997-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms