RFC 2032
RTP Payload Format for H.261 Video Streams
RFC 2032: RTP Payload Format for H.261 Video Streams 🎀✨
Okay bestie, this is RFC 2032, and it's giving main character for video over RTP 🦋. It's a Standards Track document from the Network Working Group (dated October 1996). The authors are T. Turletti from MIT and C. Huitema from Bellcore 💜. It doesn't have any "Obsoletes" or "Updates" line in its header, so it doesn't replace or change any other RFC 🌸.
Here's the whole ToC, copied exactly so we can find our way around 🗺️:
Table of Contents
1. Abstract ............................................. 1
2. Purpose of this document ............................. 2
3. Structure of the packet stream ....................... 2
3.1 Overview of the ITU-T recommendation H.261 .......... 2
3.2 Considerations for packetization .................... 3
4. Specification of the packetization scheme ............ 4
4.1 Usage of RTP ........................................ 4
4.2 Recommendations for operation with hardware codecs .. 6
5. Packet loss issues ................................... 7
5.1 Use of optional H.261-specific control packets ...... 8
5.2 H.261 control packets definition .................... 9
5.2.1 Full INTRA-frame Request (FIR) packet ............. 9
5.2.2 Negative ACKnowledgements (NACK) packet ........... 9
6. Security Considerations .............................. 10
Authors' Addresses ..................................... 10
Acknowledgements ....................................... 10
References ............................................. 11Status of this Memo
This doc is a standards track protocol spec for the Internet community 🫶. It's basically asking for feedback and suggestions to make it better 💬. If you want to know where it stands officially, go check the current edition of the "Internet Official Protocol Standards" (STD 1) 📚. Also, distribution is totally unlimited, so share it with whoever 🌈.
1. Abstract ✨
So the big idea: this memo explains how to take an H.261 video stream and packetize it so it can go over the Real-time Transport Protocol, RTP, no matter which underlying protocols are carrying the RTP 🦋.
This spec came out of the Audio/Video Transport working group at the Internet Engineering Task Force 💅. They want comments, so send them to the working group's mailing list at rem-conf@es.net and/or to the authors 💌.
2. Purpose of this document 🎀
The ITU-T recommendation H.261 [6] is the one that defines how ITU-T compliant video-conference codecs encode stuff 📹. Originally these encodings were built for fixed-rate ISDN circuits, but experiments [3],[8] showed they also work over packet-switched networks like the Internet, which is iconic 🔥.
So the purpose of this memo is to define the RTP payload format for wrapping H.261 video streams in RTP [1] 🎁.
3. Structure of the packet stream 🧩
3.1. Overview of the ITU-T recommendation H.261 💜
Okay so H.261 is organized like a hierarchy of groupings 🪆. A video stream is a sequence of images (called frames), and each frame is made of Groups of Blocks (GOB) 🧱. Quick note: in this doc, H.261 "pictures" are called "frames" 🙈.
Each GOB holds 3 lines of 11 macro blocks (MB) 💖. Each MB covers a 16x16 pixel area. The luminance (brightness) gets 4 blocks of 8x8 pixels, and the chrominance (color) is two "red" and "blue" color difference components at only 8x8 pixels resolution 🌈. These components, and the codes for their sampled values, follow ITU-R Recommendation 601 [7] 🍓.
This grouping sets up info at each level of the hierarchy:
- At the frame level, you specify stuff like the delay from the previous frame, the image format, and various indicators ✨.
- At the GOB level, you specify the GOB number and the default quantifier that the MBs will use 🎀.
- At the MB level, you specify which blocks are present and which didn't change, and optionally a quantifier and motion vectors 🦋.
Blocks that have changed get encoded by computing the discrete cosine transform (DCT) of their coefficients. Then they're quantized and Huffman encoded (Variable Length Codes) 🧠.
🚨 The H.261 Huffman encoding has a special "GOB start" pattern: 15 zeroes followed by a single 1. No other code words can imitate it. This pattern shows up at the beginning of each GOB header (and at the beginning of each frame header too) to mark where one GOB ends and the next begins. It's literally used as the signal that the current GOB is terminated 😭.
The encoding also has a stuffing pattern: seven zeroes followed by four ones. That stuffing pattern can only be entered between the encoding of MBs, or just before the GOB separator 🧃.
3.2. Considerations for packetization 📦
So H.261 codecs built for ISDN circuits spit out a bit stream with several layers of encoding, defined by H.261 and its companion recommendations 🪄. The bits from the Huffman encoding get arranged into 512-bit frames: 2 bits of synchronization, 492 bits of data, and 18 bits of error correcting code 🔐. Those 512-bit frames then get interlaced with an audio stream and sent over px64 kbps circuits following H.221 [5] 📞.
When we're sending over the Internet, we're going to look directly at the output of the Huffman encoding 💅. All the bits from that stage get included in the packet. We're not carrying the 512-bit frames, because protection against bit errors can be handled by other means 🙅♀️. We also won't try to multiplex audio and video in the same packets, since UDP and RTP already do multiplexing way more efficiently 🫶.
Sending the Huffman output straight over an unreliable stream of UDP datagrams would have poor error resistance though 😬. Because of the hierarchical structure of the H.261 bit stream, you need the info from the frame header to decode the GOBs, and the info from the GOB header to decode the MBs 🪆. Without precautions, that means you'd have to receive all the packets carrying an image just to decode its components 💀.
If each image could fit in a single packet, that wouldn't be a problem at all 🙌. But a whole image, or even a single GOB, can sometimes be too big for one packet 📏. So the MB is the unit of fragmentation 💖. Packets must start and end on an MB boundary, which means an MB can't be split across multiple packets 🚫. When several MBs fit within the maximal packet size allowed, they may be carried together in one packet 🎁. This practice is recommended because it lowers the packet send rate and the packet overhead 📉.
To let each packet be processed on its own for quick resynchronization when packets get lost 🔄, some state info from the frame header and GOB header gets carried with each packet 🧚. This lets the MBs in that packet be decoded. The state info includes:
- the GOB number in effect at the start of the packet,
- the macroblock address predictor (i.e. the last MBA encoded in the previous packet),
- the quantizer value in effect prior to the start of this packet (
GQUANT,MQUANTor zero in case of a beginning of GOB), and - the reference motion vector data (MVD) for computing the true MVDs contained within this packet 💜.
The bit stream cannot be fragmented between a GOB header and MB 1 of that GOB 🙅♀️.
Also, since a compressed MB might not fill up a whole number of octets, the data header has two three-bit integers, SBIT and EBIT 🔢. They tell you how many unused bits are in the first and last octets of the H.261 data, respectively 🍰.
4. Specification of the packetization scheme 🌸
4.1. Usage of RTP 💖
okay so the H.261 info rides inside RTP as payload data 🎀 These are the RTP header fields that get specified:
- The payload type should specify H.261 payload format (see the companion RTP profile document RFC 1890) 🧚
- The RTP timestamp encodes the sampling instant of the first video image in the RTP data packet ⏰ If one video image takes up more than one packet, the timestamp is the same on all of them. Packets from different video images must have different timestamps so frames can be told apart ✨ For H.261 streams the timestamp is based on a 90kHz clock. That clock rate is a multiple of the natural H.261 frame rate (i.e. 30000/1001 or approx. 29.97 Hz), so for each frame time you just bump the clock by that multiple, and there's no inaccuracy in the timestamp math 🫶 The starting timestamp is random (unpredictable) so known-plaintext attacks on encryption are harder, see RTP [1] 🔐 Also, if multiple frames get packed into one packet (e.g. when there are very few changes between two images), you have to work out the display times for the frames after the first one using the timing info in the H.261 frame header 🙈 This is required because the RTP timestamp only gives the display time of the first frame in the packet.
- The marker bit of the RTP header is set to one in the last packet of a video frame, and otherwise, must be zero 🍓 So you don't have to wait for the next packet (the one with the start code that ends the current frame) to figure out it's time to show a new frame 👑
The H.261 data follows the RTP header, like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. .
. RTP header .
. .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| H.261 header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| H.261 stream ... .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The H.261 header is defined as following:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|SBIT |EBIT |I|V| GOBN | MBAP | QUANT | HMVD | VMVD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Here's what each field in the H.261 header means 🦋
Start bit position (SBIT): 3 bits Number of most significant bits that should be ignored in the first data octet.
End bit position (EBIT): 3 bits Number of least significant bits that should be ignored in the last data octet.
INTRA-frame encoded data (I): 1 bit Set to 1 if this stream contains only INTRA-frame coded blocks. Set to 0 if this stream may or may not contain INTRA-frame coded blocks. The sense of this bit may not change during the course of the RTP session.
Motion Vector flag (V): 1 bit Set to 0 if motion vectors are not used in this stream. Set to 1 if motion vectors may or may not be used in this stream. The sense of this bit may not change during the course of the session.
GOB number (GOBN): 4 bits Encodes the GOB number in effect at the start of the packet. Set to 0 if the packet begins with a GOB header.
Macroblock address predictor (MBAP): 5 bits Encodes the macroblock address predictor (i.e. the last MBA encoded in the previous packet). This predictor ranges from 0-32 (to predict the valid MBAs 1-33), but because the bit stream cannot be fragmented between a GOB header and MB 1, the predictor at the start of the packet can never be 0. Therefore, the range is 1-32, which is biased by -1 to fit in 5 bits. For example, if MBAP is 0, the value of the MBA predictor is 1. Set to 0 if the packet begins with a GOB header.
Quantizer (QUANT): 5 bits Quantizer value (MQUANT or GQUANT) in effect prior to the start of this packet. Set to 0 if the packet begins with a GOB header.
Horizontal motion vector data (HMVD): 5 bits Reference horizontal motion vector data (MVD). Set to 0 if V flag is 0 or if the packet begins with a GOB header, or when the MTYPE of the last MB encoded in the previous packet was not MC. HMVD is encoded as a 2's complement number, and `10000' corresponding to the value -16 is forbidden (motion vector fields range from +/-15).
Vertical motion vector data (VMVD): 5 bits Reference vertical motion vector data (MVD). Set to 0 if V flag is 0 or if the packet begins with a GOB header, or when the MTYPE of the last MB encoded in the previous packet was not MC. VMVD is encoded as a 2's complement number, and `10000' corresponding to the value -16 is forbidden (motion vector fields range from +/-15).
Note that the I and V flags are hint flags, i.e. they can be inferred from the bit stream. They're included so decoders can make optimizations that wouldn't be possible if these hints weren't provided before the bit stream gets decoded 🌈 That's why these bits can't change for the duration of the stream. A conformant implementation can always set V=1 and I=0 💅
4.2. Recommendations for operation with hardware codecs 🔥
Packetizers for hardware codecs can trivially figure out GOB boundaries using the GOB-start pattern included in the H.261 data. (Note that software encoders already know the boundaries.) The cheapest packetization implementation is to packetize at the GOB level all the GOBs that fit in a packet. But when a GOB is too large, the packetizer has to parse it to do MB fragmentation. (Note that only the Huffman encoding must be parsed and that it is not necessary to fully decompress the stream, so this requires relatively little processing; example implementations can be found in some public H.261 codecs such as IVS [4] and VIC [9].) It is recommended that MB level fragmentation be used when feasible in order to obtain more efficient packetization. Using this fragmentation scheme reduces the output packet rate and therefore reduces the overhead 💜
At the receiver, the data stream can be depacketized and directed to a hardware codec's input. If the hardware decoder operates at a fixed bit rate, synchronization may be maintained by inserting the stuffing pattern between MBs (i.e., between packets) when the packet arrival rate is slower than the bit rate ✨
5. Packet loss issues 🌸
On the Internet, most packet losses are due to network congestion rather than transmission errors. Using UDP, no mechanism is available at the sender to know if a packet has been successfully received. It is up to the application, i.e. coder and decoder, to handle the packet loss. Each RTP packet includes a sequence number field which can be used to detect packet loss 😭
H.261 uses the temporal redundancy of video to perform compression. This differential coding (or INTER-frame coding) is sensitive to packet loss. After a packet loss, parts of the image may remain corrupt until all corresponding MBs have been encoded in INTRA-frame mode (i.e. encoded independently of past frames). There are several ways to mitigate packet loss 🧚
(1) One way is to use only INTRA-frame encoding and MB level conditional replenishment. That is, only MBs that change (beyond some threshold) are transmitted.
(2) Another way is to adjust the INTRA-frame encoding refreshment rate according to the packet loss observed by the receivers. The H.261 recommendation specifies that a MB is INTRA-frame encoded at least every 132 times it is transmitted. However, the INTRA-frame refreshment rate can be raised in order to speed the recovery when the measured loss rate is significant 🔁
(3) The fastest way to repair a corrupted image is to request an INTRA-frame coded image refreshment after a packet loss is detected. One means to accomplish this is for the decoder to send to the coder a list of packets lost. The coder can decide to encode every MB of every GOB of the following video frame in INTRA-frame mode (i.e. Full INTRA-frame encoded), or if the coder can deduce from the packet sequence numbers which MBs were affected by the loss, it can save bandwidth by sending only those MBs in INTRA-frame mode. This mode is particularly efficient in point-to-point connection or when the number of decoders is low. The next section specifies how the refresh function may be implemented 🦋
Note that the method (1) is currently implemented in the VIC videoconferencing software [9]. Methods (2) and (3) are currently implemented in the IVS videoconferencing software [4].
5.1. Use of optional H.261-specific control packets 🎀
okay so this section is about two H.261-specific RTCP control packets, "Full INTRA-frame Request" and "Negative Acknowledgement" (we'll get to them in the next section 💖). Their whole job is to speed up refreshing the video when that's actually doable. Supporting them on the H.261 sender is optional 🫶, and bestie, early experiments showed that using them could have very negative effects when there are a huge number of sites 💀. So these control packets should be used with caution, no cap.
Here's the tea 🍵. These packets are NOT sent to the normal RTCP destination transport address for the RTP session (which is often a multicast address). Instead they go straight from the decoder to the coder via unicast ✨. The destination port is the same port the coder uses as its source port for sending RTP (data) packets. That's why they can be called "reverse" control packets 🔁.
Now the catch 🙈. These control packets only work when there are no RTP mixers or translators in the path from the coder to the decoder. If one of those middlemen shows up, the coder's address is no longer the network-level source address in the packets the decoder gets, and the decoder might not even be able to send packets directly back to the coder. Not iconic 😭.
Other reliable multicast protocols use similar NACK control packets over the normal multicast channel, but they usually add random delays to avoid a NACK implosion problem [2]. Their goal is reliable multicast delivery even though it costs delay, which makes sense for something like a shared whiteboard 📝.
Video is different, though 🎥. Interactive video cares more about delay and doesn't need full reliability. So for video it's better to send NACK control packets ASAP, meaning as soon as a loss is detected, with no random delays added ⚡. Multicasting those NACKs would just create useless traffic between receivers, since only the coder will actually use them 🙄. But this only works well when the number of receivers is small. For example, in IVS [4] the H.261-specific control packets are only used in point-to-point connections, or in point-to-multipoint connections with fewer than 10 participants in the conference 🧚.
5.2. H.261 control packets definition 💅
5.2.1. Full INTRA-frame Request (FIR) packet 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P| MBZ | PT=RTCP_FIR | length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+so the FIR packet is basically a receiver saying "hey, I need a FULL encoded image" 🖼️. That's for starting a decode from scratch, or for recovering fast after a burst of lost packets. The receiver asks the source to force the next image into full "INTRA-frame" coding mode, meaning no differential coding 💜. The various fields are defined in the RTP specification [1]. SSRC is the synchronization source identifier for the sender of this packet 🪪. And the packet type (PT) is the constant RTCP_FIR (192) ✨.
5.2.2. Negative ACKnowledgements (NACK) packet 🌸
The NACK packet looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P| MBZ | PT=RTCP_NACK | length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SSRC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FSN | BLP |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The fields T, P, PT, length and SSRC all come from the RTP specification [1], which is the same family of fields we saw above 🔗. The packet type (PT) is the constant RTCP_NACK (193), and SSRC is again the sender's synchronization source identifier 🫶.
The last two fields are where the magic is 🪄:
First Sequence Number (FSN): 16 bits 🔢 This one identifies the first sequence number that was lost.
Bitmask of following lost packets (BLP): 16 bits 🎭 Each bit is set to 1 if the matching packet was lost, and to 0 if it wasn't. BLP is set to 0 only when no packet other than the one being NACKed (the one named in the FSN field) has been lost. If the packet for the FSN and the packet right after it were both lost, BLP is set to 0x00001, and so on 🧮.
6. Security Considerations 🔒
Security issues are not discussed in this memo. 😬 Yep, that's the whole section, so keep it in mind when you're deploying this stuff 👀.
Authors' Addresses 🖊️
Thierry Turletti
INRIA - RODEO Project
2004 route des Lucioles
BP 93, 06902 Sophia Antipolis
FRANCE
EMail: turletti@sophia.inria.fr
Christian Huitema
MCC 1J236B Bellcore
445 South Street
Morristown, NJ 07960-6438
EMail: huitema@bellcore.comAcknowledgements 💜
This memo is based on discussion within the AVT working group chaired by Stephen Casner. Steve McCanne, Stephen Casner, Ronan Flood, Mark Handley, Van Jacobson, Henning G. Schulzrinne and John Wroclawski provided valuable comments 🫶. Stephen Casner and Steve McCanne also helped greatly with getting this document into readable form ✨.
References 📚
[1] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", RFC 1889, January 1996.
[2] Sridhar Pingali, Don Towsley and James F. Kurose, A comparison of sender-initiated and receiver-initiated reliable multicast protocols, IEEE GLOBECOM '94.
[3] Thierry Turletti, H.261 software codec for videoconferencing over the Internet INRIA Research Report no 1834, January 1993.
[4] Thierry Turletti, INRIA Videoconferencing tool (IVS), available by anonymous ftp from zenon.inria.fr in the "rodeo/ivs/last_version" directory. See also URL <http://www.inria.fr/rodeo/ivs.html>.
[5] Frame structure for Audiovisual Services for a 64 to 1920 kbps Channel in Audiovisual Services ITU-T (International Telecommunication Union - Telecommunication Standardisation Sector) Recommendation H.221, 1990.
[6] Video codec for audiovisual services at p x 64 kbit/s ITU-T (International Telecommunication Union - Telecommunication Standardisation Sector) Recommendation H.261, 1993.
[7] Digital Methods of Transmitting Television Information ITU-R (International Telecommunication Union - Radiocommunication Standardisation Sector) Recommendation 601, 1986.
[8] M.A Sasse, U. Bilting, C-D Schulz, T. Turletti, Remote Seminars through MultiMedia Conferencing: Experiences from the MICE project, Proc. INET'94/JENC5, Prague, June 1994, pp. 251/1-251/8.
[9] Steve MacCanne, Van Jacobson, VIC Videoconferencing tool, available by anonymous ftp from ee.lbl.gov in the "conferencing/vic" directory. 🎬
📦 where this page comes from
- ietf: rfc2032 1996-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms