RFC 3016

RTP Payload Format for MPEG-4 Audio/Visual Streams

📅 published November 2000, from ietf

RTP Payload Format for MPEG-4 Audio/Visual Streams 🎀

Okay bestie, here's the title block tea ☕✨

  • RFC: 3016
  • Working group: Network Working Group
  • Category: Standards Track 👑
  • Date: November 2000 📅
  • Authors: Y. Kikuchi (Toshiba), T. Nomura (NEC), S. Fukunaga (Oki), Y. Matsui (Matsushita), H. Kimata (NTT) 💅
  • Obsoletes / Updates: the title block doesn't list anything this one obsoletes or updates 🦋

Status of this Memo ✨

So this is a Standards Track document for the Internet community, and it's literally asking for discussion and suggestions for improvements 🫶 If you want the current standardization state and status of this protocol, check the "Internet Official Protocol Standards" (STD 1) 📚 And distribution of this memo is unlimited, so share it with everyone 🌈

Copyright (C) The Internet Society (2000). All Rights Reserved.

Abstract

Real-Time Transport Protocol (RTP) payload formats for MPEG-4 Audio and MPEG-4 Visual bitstreams, used without MPEG-4 Systems 🎀 The whole point is mapping MPEG-4 Audio/Visual bitstreams straight onto RTP packets, so this doc defines how RTP header fields get used and sets fragmentation rules 🧚 It also covers MIME type registrations and how Session Description Protocol (SDP) gets used. It's giving complete package, no cap 💜

1. Introduction ✨

Okay so the RTP payload formats in this doc spell out how MPEG-4 Audio [3][5] and MPEG-4 Visual streams [2][4] get fragmented and mapped directly onto RTP packets 🦋

These formats let you transport MPEG-4 Audio/Visual streams without the synchronization and stream management stuff that MPEG-4 Systems [6] normally handles 🫶 They're meant for systems that already have their own stream management, so they don't need it from MPEG-4 Systems. H.323 terminals are a great example, since there the MPEG-4 Audio/Visual streams aren't managed by MPEG-4 Systems Object Descriptors but by H.245 💅 The streams just get mapped straight onto RTP packets with no MPEG-4 Systems Sync Layer involved. SIP and RTSP are other examples, where MIME and SDP do the work 🌸 The MIME types and SDP usages in this doc directly specify the attributes of Audio/Visual streams (like media type, packetization format, and codec configuration) without MPEG-4 Systems 🧚

The obvious benefit is that these MPEG-4 Audio/Visual RTP payload formats can be handled in a unified way alongside the formats for non-MPEG-4 codecs, which is so iconic 👑 The downside is that interoperability with environments using MPEG-4 Systems may be difficult, and other payload formats might be better suited to those applications 💀

The semantics of RTP headers in these cases need to be clearly defined, including how they connect to MPEG-4 Audio/Visual data elements 🍓 It's also beneficial to define fragmentation rules for RTP packets carrying MPEG-4 Video streams, so you can boost error resiliency by using the error resilience tools already inside the MPEG-4 Video stream 🔥

1.1 MPEG-4 Visual RTP payload format

MPEG-4 Visual is a visual coding standard with a bunch of new features: high coding efficiency, high error resiliency, multiple arbitrary shape object-based coding, and more [2] 💖 It covers a huge range of bitrates, from scores of Kbps all the way up to several Mbps 🌈 It also covers a wide variety of networks, from ones that are basically error-free to mobile networks with high error rates 📱

For the fragmentation rules in this doc, MPEG-4 Visual gets used on tons of different networks, so it's desirable not to apply too much restriction on fragmentation 🙈 A rule like "a single video packet shall always be mapped on a single RTP packet" may be inappropriate. On the flip side, careless, media unaware fragmentation can cause degradation in error resiliency and bandwidth efficiency 😭 The fragmentation rules here are flexible, but they still manage to define the minimum rules needed to prevent meaningless fragmentation while using the error resilience functionalities of MPEG-4 Visual ✨

The fragmentation rule recommends not mapping more than one VOP in an RTP packet, so the RTP timestamp uniquely indicates the VOP time framing 🦋 But here's the twist: MPEG-4 video can generate VOPs of very small size, like an empty VOP (vop_coded=0) with only a VOP header, or an arbitrary shaped VOP with a small number of coding blocks 🥺 To cut down the overhead for those cases, the fragmentation rule permits concatenating multiple VOPs in one RTP packet. (See fragmentation rule (4) in section 3.2 and marker bit and timestamp in section 3.1.) 💅

Extra media specific RTP headers, like the ones defined for video coding tools such as H.261 or MPEG-1/2, are effective at helping recover picture headers corrupted by packet losses 🧚 But MPEG-4 Visual already has error resilience functionalities for recovering corrupt headers, and those work on RTP/IP networks as well as other networks (H.223/mobile, MPEG-2/TS, etc.) 🫶 So no extra RTP header fields are defined in this MPEG-4 Visual RTP payload format 🎀

1.2 MPEG-4 Audio RTP payload format

MPEG-4 Audio is a brand new kind of audio standard that integrates a ton of different audio coding tools 🎧 Low-overhead MPEG-4 Audio Transport Multiplex (LATM) manages sequences of audio data with relatively small overhead, which is so efficient 🔥 So in audio-only applications, it's desirable for LATM-based MPEG-4 Audio bitstreams to get mapped directly onto RTP packets without MPEG-4 Systems 💖

LATM has several multiplexing features, though:

  • Carrying configuration information with audio data,
  • Concatenation of multiple audio frames in one audio stream,
  • Multiplexing multiple objects (programs),
  • Multiplexing scalable layers,

but in RTP transmission there's no need for the last two features 💀 So these two features MUST NOT be used in applications based on RTP packetization specified by this document. LATM was only developed for natural audio coding tools, not synthesis tools, so it seems difficult to transmit Structured Audio (SA) data and Text to Speech Interface (TTSI) data by LATM 🙈 That's why SA data and TTSI data MUST NOT be transported by the RTP packetization in this document. Not for them, bestie 🍓

For transmission of scalable streams, audio data of each layer SHOULD be packetized onto different RTP packets 🌈 That way the layers can get treated differently at the IP level, for example through some means of differentiated service ✨ Meanwhile, all configuration data of the scalable streams lives in one LATM configuration data "StreamMuxConfig," and every scalable layer shares that StreamMuxConfig 🧚 The mapping between each layer and its configuration data is handled by LATM header information attached to the audio data. To indicate the dependency information of the scalable streams, there's a restriction on the dynamic assignment rule of payload type (PT) values (see section 4.2) 🦋

For MPEG-4 Audio coding tools, like other audio coders, if the payload is a single audio frame, packet loss will not impair the decodability of adjacent packets 💖 So the additional media specific header for recovering errors won't be required for MPEG-4 Audio 🫶 Existing RTP protection mechanisms, like Generic Forward Error Correction (RFC 2733) and Redundant Audio Data (RFC 2198), MAY be applied to improve error resiliency 🌸

2. Conventions used in this document 💅

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [7] 🎀

3. RTP Packetization of MPEG-4 Visual bitstream 🌈

This section is where the RTP packetization rules for MPEG-4 Visual content live 🧚 An MPEG-4 Visual bitstream gets mapped directly onto RTP packets, with no extra header fields added and no Visual syntax elements removed 🦋 The Combined Configuration/Elementary stream mode MUST be used, so configuration information gets carried to the same RTP port as the elementary stream (see 6.2.1 "Start codes" of ISO/IEC 14496-2 [2][9][4]) ✨ The configuration information MAY additionally be specified by some out-of-band means. If an H.323 terminal needs it, the H.245 codepoint "decoderConfigurationInformation" MUST be used for this purpose 💅 If systems using MIME content type and SDP parameters need it, like SIP and RTSP, the optional parameter "config" MUST be used to specify the configuration information (see 5.1 and 5.2) 🫶

When the short video header mode is used, the RTP payload format for H.263 SHOULD be used 🔥 The format defined in RFC 2429 is RECOMMENDED, but the RFC 2190 format MAY be used for compatibility with older implementations 🥹

Here's the packet layout, Figure 1, in all its glory 🎀

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X|  CC   |M|     PT      |       sequence number         | RTP
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           timestamp                           | Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           synchronization source (SSRC) identifier            |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|            contributing source (CSRC) identifiers             |
|                             ....                              |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|                                                               | RTP
|       MPEG-4 Visual stream (byte aligned)                     | Pay-
|                                                               | load
|                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               :...OPTIONAL RTP padding        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 1 - An RTP packet for MPEG-4 Visual stream 💖

3.1 Use of RTP header fields for MPEG-4 Visual 💅

okay so the Payload Type (PT) is giving "not my problem," bestie 🙈 Assigning a payload type for this new packet format is outside the scope of this doc, so it's not specified here. The RTP profile for a particular class of apps is expected to assign a payload type for this encoding. If that doesn't happen, then a payload type in the dynamic range SHALL be chosen through an out of band signaling protocol (e.g., H.245, SIP, etc) 🦋

The Extension (X) bit is defined by whichever RTP profile you're using. Totally their vibe 💜

The Sequence Number goes up by one for every RTP data packet sent. For security reasons it starts at a random initial value 🔐✨

The Marker (M) bit gets set to one to flag the last RTP packet (or the only RTP packet) of a VOP. When multiple VOPs are carried in the same RTP packet, the marker bit is set to one too 🎀

The Timestamp tells you the sampling instance of the VOP inside the RTP packet. A constant offset gets added for security reasons, and that offset is random 🌸 The timestamp rules get a little spicy:

  • When multiple VOPs are carried in the same RTP packet, the timestamp indicates the earliest of the VOP times in that packet. The timestamp info for the rest of the VOPs is derived from the timestamp fields in the VOP header (modulo_time_base and vop_time_increment) 🧚
  • If the RTP packet contains only configuration information and/or Group_of_VideoObjectPlane() fields, the timestamp of the next VOP in the coding order is used.
  • If the RTP packet contains only visual_object_sequence_end_code information, the timestamp of the immediately preceding VOP in the coding order is used.

The timestamp resolution is set to its default value of 90kHz, unless an out-of-band means says otherwise (e.g., an SDP parameter or a MIME parameter, as defined in section 5) ⭐

Other header fields are used as described in RFC 1889 [8]. Easy peasy 🫶

3.2 Fragmentation of MPEG-4 Visual bitstream 🌈

Okay this part is the rulebook, and it's lowkey important 📖 A fragmented MPEG-4 Visual bitstream gets mapped directly onto the RTP payload. No extra header fields get added and no Visual syntax elements get removed. The Combined Configuration/Elementary streams mode is used, and these rules apply to the fragmentation.

So here's what counts as a header in this context:

  • Configuration information (Visual Object Sequence Header, Visual Object Header and Video Object Layer Header)
  • visual_object_sequence_end_code
  • The header of the entry point function for an elementary stream (Group_of_VideoObjectPlane() or the header of VideoObjectPlane(), video_plane_with_short_header(), MeshObject() or FaceObject())
  • The video packet header (video_packet_header() excluding next_resync_marker())
  • The header of gob_layer()

See 6.2.1 "Start codes" of ISO/IEC 14496-2 [2][9][4] for the definition of the configuration information and the entry point functions 🔍

Now the rules, bestie 💖

(1) Configuration information and Group_of_VideoObjectPlane() fields SHALL be placed at the beginning of the RTP payload (just after the RTP header), or just after the header of the syntactically upper layer function.

(2) If one or more headers exist in the RTP payload, the RTP payload SHALL begin with the header of the syntactically highest function. Note: The visual_object_sequence_end_code is regarded as the lowest function. 👑

(3) A header SHALL NOT be split into a plurality of RTP packets. No splitting headers, period 🙅‍♀️

(4) Different VOPs SHOULD be fragmented into different RTP packets, so that one RTP packet consists of the data bytes associated with a unique VOP time instance (indicated in the timestamp field in the RTP packet header). The exception is that multiple consecutive VOPs MAY be carried within one RTP packet, in decoding order, if the size of the VOPs is small 🦋

Note: When multiple VOPs are carried in one RTP payload, the timestamp of the VOPs after the first one may be calculated by the decoder. This operation is necessary only for RTP packets in which the marker bit equals to one and the beginning of RTP payload corresponds to a start code. (See timestamp and marker bit in section 3.1.) 🧠

(5) It is RECOMMENDED that a single video packet is sent as a single RTP packet. The size of a video packet SHOULD be adjusted so the resulting RTP packet is not larger than the path-MTU.

Note: Rule (5) does not apply when the video packet is disabled by the coder configuration (by setting resync_marker_disable in the VOL header to 1), or in coding tools where the video packet is not supported. In this case, a VOP MAY be split at arbitrary byte-positions. 🌸

The video packet starts with the VOP header or the video packet header, followed by motion_shape_texture(), and ends with next_resync_marker() or next_start_code(). That's the whole vibe ✨

3.3 Examples of packetized MPEG-4 Visual bitstream 🎀

Figure 2 shows examples of RTP packets generated based on the criteria in 3.2. Let's go through them, I'm obsessed 😭

(a) is an example of the first RTP packet, or the random access point, of an MPEG-4 Visual bitstream containing the configuration information. Under criterion (1), the Visual Object Sequence Header (VS header) goes at the beginning of the RTP payload, before the Visual Object Header and the Video Object Layer Header (VO header, VOL header). Since the fragmentation rule in 3.2 guarantees that the configuration information, starting with visual_object_sequence_start_code, is always at the beginning of the RTP payload, RTP receivers can detect the random access point by checking whether the first 32-bit field of the RTP payload is visual_object_sequence_start_code. Main character energy 🧚

(b) is another example of an RTP packet containing configuration information. It differs from example (a) because the RTP packet also contains a video packet in the VOP that follows the configuration information. The configuration information is relatively short (typically scores of bytes), and an RTP packet containing only that could bump up the overhead. So the configuration information and the immediately following GOV and/or (a part of) VOP can be packetized into a single RTP packet, like in this example 🍓

(c) is an example of an RTP packet that contains Group_of_VideoObjectPlane(GOV). Following criterion (1), the GOV goes at the beginning of the RTP payload. It would be a total waste of RTP/IP header overhead to make a packet with only a GOV that's 7 bytes long 💀 So (a part of) the following VOP can go in the same RTP packet, as shown in (c).

(d) is an example of one video packet packetized into one RTP packet. When the packet-loss rate of the underlying network is high, this kind of packetization is recommended. Even when the RTP packet containing the VOP header is discarded by a packet loss, the other RTP packets can still be decoded using the HEC(Header Extension Code) information in the video packet header. No extra RTP header field is necessary 🛡️

(e) is an example of more than one video packet packetized into one RTP packet. This kind of packetization is effective at saving RTP/IP header overhead when the bit-rate of the underlying network is low. However, it decreases packet-loss resiliency, because multiple video packets are discarded by a single RTP packet loss. The optimal number of video packets in an RTP packet, and the length of the RTP packet, can be determined by considering the packet-loss rate and the bit-rate of the underlying network 📉

(f) is an example of the case when the video packet is disabled by setting resync_marker_disable in the VOL header to 1. In this case, a VOP may be split into a plurality of RTP packets at arbitrary byte-positions. For example, a VOP can be split into fixed-length packets. This kind of coder configuration and RTP packet fragmentation may be used when the underlying network is guaranteed to be error-free. On the other hand, it is not recommended in an error-prone environment, since it provides only poor packet loss resiliency 🙃

Figure 3 shows examples of RTP packets prohibited by the criteria of 3.2. Ooh, the "don'ts" 🚫

Fragmenting a header into multiple RTP packets, as in (a), will not only increase RTP/IP header overhead but also decrease error resiliency. That's why it's prohibited by criterion (3) 🙅‍♀️

When concatenating more than one video packet into an RTP packet, the VOP header or video_packet_header() shall not be placed in the middle of the RTP payload. The packetization in (b) is not allowed by criterion (2), due to error resiliency. Compare this with Figure 2(d): although two video packets are mapped onto two RTP packets in both cases, the packet-loss resiliency is not identical. If the second RTP packet is lost, both video packets 1 and 2 are lost in the case of Figure 3(b), whereas only video packet 2 is lost in the case of Figure 2(d). Spicy difference 🌶️

+------+------+------------+ +------+------------+ (f) | RTP | VOP |VOP fragment| | RTP |VOP fragment|

headerheader(1)header(2)___

Oops, that last line belongs to Figure 2, so here's the full Figure 2 exactly as written:

    +------+------+------+------+
(a) | RTP  |  VS  |  VO  | VOL  |
    |header|header|header|header|
    +------+------+------+------+

    +------+------+------+------+------------+
(b) | RTP  |  VS  |  VO  | VOL  |Video Packet|
    |header|header|header|header|            |
    +------+------+------+------+------------+

    +------+-----+------------------+
(c) | RTP  | GOV |Video Object Plane|
    |header|     |                  |
    +------+-----+------------------+

    +------+------+------------+  +------+------+------------+
(d) | RTP  | VOP  |Video Packet|  | RTP  |  VP  |Video Packet|
    |header|header|    (1)     |  |header|header|    (2)     |
    +------+------+------------+  +------+------+------------+

    +------+------+------------+------+------------+------+------------+
(e) | RTP  |  VP  |Video Packet|  VP  |Video Packet|  VP  |Video Packet|
    |header|header|     (1)    |header|    (2)     |header|    (3)     |
    +------+------+------------+------+------------+------+------------+

    +------+------+------------+  +------+------------+
(f) | RTP  | VOP  |VOP fragment|  | RTP  |VOP fragment|
    |header|header|    (1)     |  |header|    (2)     | ___
    +------+------+------------+  +------+------------+

     Figure 2 - Examples of RTP packetized MPEG-4 Visual bitstream

And Figure 3, the prohibited layouts:

    +------+-------------+  +------+------------+------------+
(a) | RTP  |First half of|  | RTP  |Last half of|Video Packet|
    |header|  VP header  |  |header|  VP header |            |
    +------+-------------+  +------+------------+------------+

    +------+------+----------+  +------+---------+------+------------+
(b) | RTP  | VOP  |First half|  | RTP  |Last half|  VP  |Video Packet|
    |header|header| of VP(1) |  |header| of VP(1)|header|    (2)     |
    +------+------+----------+  +------+---------+------+------------+

   Figure 3 - Examples of prohibited RTP packetization for MPEG-4 Visual
   bitstream

4. RTP Packetization of MPEG-4 Audio bitstream 🎧

This section specifies RTP packetization rules for MPEG-4 Audio bitstreams. MPEG-4 Audio streams MUST be formatted by the LATM (Low-overhead MPEG-4 Audio Transport Multiplex) tool [5], and the LATM-based streams are then mapped onto RTP packets as described the three sections below. Audio era 🎶

4.1 RTP Packet Format ✨

Okay so LATM-based streams are basically a sequence of audioMuxElements, and each one holds one or more audio frames 🎶. A complete audioMuxElement, or a piece of one, SHALL go straight into an RTP payload with no audioMuxElement syntax elements stripped out 💅 (see Figure 4). And the very first byte of each audioMuxElement SHALL sit at the first payload location in an RTP packet 🫶.

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X|  CC   |M|     PT      |       sequence number         |RTP
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           timestamp                           |Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           synchronization source (SSRC) identifier            |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|            contributing source (CSRC) identifiers             |
|                             ....                              |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|                                                               |RTP
:                 audioMuxElement (byte aligned)                :Payload
|                                                               |
|                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               :...OPTIONAL RTP padding        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

             Figure 4 - An RTP packet for MPEG-4 Audio

To decode the audioMuxElement, you need the muxConfigPresent info 🕵️, and it has to be indicated by an out-of-band means. When SDP is the thing doing that job, the MIME parameter "cpresent" is what carries the muxConfigPresent info (see section 5.3).

  • muxConfigPresent set to 1 (in-band mode) 🌸: the audioMuxElement SHALL include an indication bit "useSameStreamMux", and it MAY include the configuration information for audio compression "StreamMuxConfig". The useSameStreamMux bit says whether the StreamMuxConfig element from the previous frame gets applied to the current frame. Now here's the catch 😭: if it says to reuse the StreamMuxConfig from the previous frame but that previous frame got lost, the current frame may not be decodable. So for in-band mode, the StreamMuxConfig element SHOULD be transmitted repeatedly, depending on network conditions.
  • muxConfigPresent set to 0 (out-band mode) 🧚: the StreamMuxConfig element is required to be transmitted by an out-of-band means. For SDP, the MIME parameter "config" is used (see section 5.3).

4.2 Use of RTP Header Fields for MPEG-4 Audio 🎀

Payload Type (PT): Assigning an RTP payload type for this new packet format is outside the scope of this document, so it won't be specified here 🙈. The expectation is that the RTP profile for a particular class of applications assigns a payload type for this encoding. If that doesn't happen, a payload type in the dynamic range shall be chosen by means of an out of band signaling protocol (e.g., H.245, SIP, etc). For dynamic assignment of RTP payload types for scalable streams, a different value SHOULD be assigned to each layer. The assigned values SHOULD follow enhance layer dependency order, where the base layer has the smallest value 📚.

Marker (M) bit: The marker bit marks audioMuxElement boundaries. It gets set to one to say the RTP packet holds a complete audioMuxElement or the last fragment of one 🏁.

Timestamp: The timestamp shows the sampling instance of the first audio frame in the RTP packet ⏰. Timestamps are recommended to start at a random value for security reasons 🔐.

Unless an out-of-band means specifies otherwise, the timestamp resolution is set to its default value of 90 kHz.

Sequence Number: Goes up by one for every RTP packet sent, starting, for security reasons, at a random value 🎲.

Other header fields are used as described in RFC 1889 [8].

4.3 Fragmentation of MPEG-4 Audio bitstream 🦋

It is RECOMMENDED to put one audioMuxElement in each RTP packet 💖. If an audioMuxElement is small enough that the RTP packet holding it doesn't go over the path-MTU, you're totally fine ✨. If it can't fit, the audioMuxElement MAY be fragmented and spread across multiple packets 🧩.

5. MIME type registration for MPEG-4 Audio/Visual streams 👑

Here's the lineup for MPEG-4 Audio/Visual streams 🌈. MIME type registration and SDP usage for the MPEG-4 Visual stream are in Sections 5.1 and 5.2, and MIME type registration and SDP usage for the MPEG-4 Audio stream are in Sections 5.3 and 5.4.

5.1 MIME type registration for MPEG-4 Visual 💖

MIME media type name: video

MIME subtype name: MP4V-ES

Required parameters: none

Optional parameters:

  • rate: This parameter is only used for RTP transport 🚀. It says what the resolution of the timestamp field in the RTP header is. If it isn't specified, the default value of 90000 (90kHz) gets used.
  • profile-level-id: This is a decimal representation of the MPEG-4 Visual Profile and Level indication value (profile_and_level_indication) defined in Table G-1 of ISO/IEC 14496-2 [2][4]. It MAY be used in the capability exchange or session setup procedure to show which MPEG-4 Visual Profile and Level combination the MPEG-4 Visual codec is capable of 🧠. If the procedure doesn't specify it, the default value of 1 (Simple Profile/Level 1) is used.
  • config: This parameter SHALL be used to indicate the configuration of the corresponding MPEG-4 Visual bitstream 🎯. It SHALL NOT be used to indicate codec capability in the capability exchange procedure 🙅‍♀️. It's a hexadecimal representation of an octet string that expresses the MPEG-4 Visual configuration information, as defined in subclause 6.2.1 Start codes of ISO/IEC14496-2 [2][4][9]. The configuration information gets mapped onto the octet string on an MSB-first basis. The first bit of the configuration information SHALL be located at the MSB of the first octet. The configuration information indicated by this parameter SHALL be the same as the configuration information in the corresponding MPEG-4 Visual stream, except for first_half_vbv_occupancy and latter_half_vbv_occupancy, if they exist. Those two can vary in the repeated configuration information inside an MPEG-4 Visual stream (See 6.2.1 Start codes of ISO/IEC14496-2) 🌷.

Example usages for these parameters are:

MPEG-4 Visual Simple Profile/Level 1:
    Content-type: video/mp4v-es; profile-level-id=1

MPEG-4 Visual Core Profile/Level 2:
    Content-type: video/mp4v-es; profile-level-id=34

MPEG-4 Visual Advanced Real Time Simple Profile/Level 1:
    Content-type: video/mp4v-es; profile-level-id=145

Published specification: The specifications for MPEG-4 Visual streams are presented in ISO/IEC 14469-2 [2][4][9]. The RTP payload format is described in RFC 3016 📖.

Encoding considerations: Video bitstreams MUST be generated according to the MPEG-4 Visual specifications (ISO/IEC 14496-2). A video bitstream is binary data, so it MUST be encoded for non-binary transport (for Email, Base64 encoding is sufficient) 📧. This type is also defined for transfer via RTP. The RTP packets MUST be packetized according to the MPEG-4 Visual RTP payload format defined in RFC 3016 📦.

Security considerations: See section 6 of RFC 3016 🔒.

Interoperability considerations: MPEG-4 Visual has a huge, rich set of tools for coding visual objects 🎬. To make it practical to implement, subsets of the MPEG-4 Visual tool sets have been provided for specific applications. These subsets, called 'Profiles', limit how big the tool set is that a decoder has to implement. To restrict computational complexity, one or more Levels are set for each Profile. A Profile@Level combination allows:

  • a codec builder to implement only the subset of the standard they need, while still interworking with other MPEG-4 devices in the same combination, and
  • checking whether MPEG-4 devices comply with the standard ('conformance testing').

The visual stream SHALL be compliant with the MPEG-4 Visual Profile@Level specified by the parameter "profile-level-id". Interoperability between a sender and a receiver may be achieved by specifying the parameter "profile-level-id" in the MIME content, or by arranging, in the capability exchange/announcement procedure, to set this parameter mutually to the same value 🤝.

Applications which use this media type: Audio and visual streaming and conferencing tools, Internet messaging, and Email applications 💬.

Additional information: none

Person & email address to contact for further information: The authors of RFC 3016 (See section 8.)

Intended usage: COMMON

Author/Change controller: The authors of RFC 3016 (See section 8.)

5.2 SDP usage of MPEG-4 Visual 🧚

The MIME media type video/MP4V-ES string maps to fields in the Session Description Protocol (SDP), RFC 2327, like this 🗺️:

  • The MIME type (video) goes in SDP "m=" as the media name.
  • The MIME subtype (MP4V-ES) goes in SDP "a=rtpmap" as the encoding name.
  • The optional parameter "rate" goes in "a=rtpmap" as the clock rate.
  • The optional parameters "profile-level-id" and "config" go in the "a=fmtp" line to indicate the coder capability and configuration, respectively. These parameters are written as a MIME media type string, in the form of a semicolon separated list of parameter=value pairs.

Here are some examples of media representation in SDP 🎀:

Simple Profile/Level 1, rate=90000(90kHz), "profile-level-id" and "config" are present in "a=fmtp" line:

m=video 49170/2 RTP/AVP 98
a=rtpmap:98 MP4V-ES/90000
a=fmtp:98 profile-level-id=1;config=000001B001000001B509000001000000012
     0008440FA282C2090A21F

Core Profile/Level 2, rate=90000(90kHz), "profile-level-id" is present in "a=fmtp" line:

m=video 49170/2 RTP/AVP 98
a=rtpmap:98 MP4V-ES/90000
a=fmtp:98 profile-level-id=34

Advance Real Time Simple Profile/Level 1, rate=90000(90kHz), "profile-level-id" is present in "a=fmtp" line:

m=video 49170/2 RTP/AVP 98
a=rtpmap:98 MP4V-ES/90000
a=fmtp:98 profile-level-id=145

5.3. MIME type registration of MPEG-4 Audio ✨

Okay bestie, time for the audio side of the MIME registration 🎀 It's giving very official, but the vibe is still iconic.

  • MIME media type name: audio
  • MIME subtype name: MP4A-LATM
  • Required parameters:
    • rate: this is the RTP time stamp clock rate 🕰️ The default is 90000. Other rates MAY be set, but only if they match the audio sampling rate (the number of samples per second) 💅
  • Optional parameters:
    • profile-level-id: a decimal value for the MPEG-4 Audio Profile Level indication, defined in ISO/IEC 14496-1 ([6] and its amendments) 🧚 It tells you which MPEG-4 Audio tool subsets the decoder can actually use. If nobody specifies it during capability exchange or session setup, the default is 30 (Natural Audio Profile/Level 1).
    • object: a decimal value for the MPEG-4 Audio Object Type, defined in ISO/IEC 14496-3 [5] 🔧 It picks the tool the coder should use, and it can be used to limit the capability inside the specified profile-level-id. Very main character energy 👑
    • bitrate: the data rate for the audio bit stream 📈
    • cpresent: a boolean that tells you whether the audio payload configuration data got multiplexed into the RTP payload (see section 4.1) 🍓 0 means it wasn't multiplexed, 1 means it was. If you leave it out, the default is 1.
    • config: a hexadecimal octet string holding the audio payload configuration data "StreamMuxConfig", as defined in ISO/IEC 14496-3 [5] (see section 4.1) 🌸 It's mapped MSB-first. The first bit of the configuration data SHALL sit at the MSB of the first octet. In the last octet, any zero-padding bits (if needed) SHALL come after the configuration data.
    • ptime: the RECOMMENDED duration of each packet, in milliseconds ⏱️
  • Published specification: the payload format is described in this document, and the encoding specs live in ISO/IEC 14496-3 [3][5] 📚
  • Encoding considerations: this type is only defined for transfer via RTP 🚚
  • Security considerations: see Section 6 of RFC 3016 🔐
  • Interoperability considerations: MPEG-4 Audio has a huge, rich set of tools for coding audio objects 🎶 To make implementing it less overwhelming, there are subsets of the tool sets, kind of like the ones used in MPEG-4 Visual (see section 5.1) ✨

The audio stream SHALL be compliant with the MPEG-4 Audio Profile@Level given by the profile-level-id parameter. Two sides can get interoperability by setting profile-level-id in the MIME content, or by setting it to the same value during capability exchange. The object parameter can also be used to limit the capability within the specified Profile@Level during capability exchange 🫶

  • Applications which use this media type: audio and video streaming and conferencing tools 📞
  • Additional information: none
  • Personal & email address to contact for further information: see Section 8 of RFC 3016 💌
  • Intended usage: COMMON
  • Author/Change controller: see Section 8 of RFC 3016 🎀

5.4. SDP usage of MPEG-4 Audio 🎧

The MIME media type audio/MP4A-LATM gets mapped onto fields in the Session Description Protocol (SDP), RFC 2327, like this:

  • The MIME type (audio) goes in the SDP m= line as the media name.
  • The MIME subtype (MP4A-LATM) goes in the SDP a=rtpmap line as the encoding name.
  • The required rate parameter goes in a=rtpmap as the clock rate.
  • The optional ptime parameter goes in the SDP a=ptime attribute.
  • The optional profile-level-id parameter goes in the a=fmtp line to show the coder capability. The object parameter goes in the a=fmtp attribute too. The payload-format-specific parameters bitrate, cpresent and config also go in the a=fmtp line. They're written as a MIME media type string, a semicolon-separated list of parameter=value pairs 🧁

Here are some examples of the media representation in SDP, bestie 💖

For 6 kb/s CELP bitstreams (with an audio sampling rate of 8 kHz):

m=audio 49230 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/8000
a=fmtp:96 profile-level-id=9;object=8;cpresent=0;config=9128B1071070
a=ptime:20

For 64 kb/s AAC LC stereo bitstreams (with an audio sampling rate of 24 kHz):

m=audio 49230 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/24000
a=fmtp:96 profile-level-id=1; bitrate=64000; cpresent=0;
config=9122620000

In the two examples above, the audio configuration data is not multiplexed into the RTP payload, so it only shows up in SDP. The clock rate is also set to the audio sampling rate 🎯

If the clock rate is left at its default value and you need the audio sampling rate, you can get it by parsing the config parameter. Check out this example 👀

m=audio 49230 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/90000
a=fmtp:96 object=8; cpresent=0; config=9128B1071070

And this one shows the audio configuration data sitting in the RTP payload 🌈

m=audio 49230 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/90000
a=fmtp:96 object=2; cpresent=1

6. Security Considerations 🔐

RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [8]. That means confidentiality of the media streams is achieved by encryption 🔒 Because the data compression used with this payload format is applied end-to-end, encryption may be performed on the compressed data, so the two operations don't clash 🫶

⚠️ Heads up, this one's dramatic: the complete MPEG-4 system can carry a wide range of content, including Java applets (MPEG-J) and scripts 😭 Since this payload format is restricted to audio and video streams, there's no way to transport that active content through it. Phew!

7. References 📚

  1. Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996.
  2. ISO/IEC 14496-2:1999, "Information technology - Coding of audio-visual objects - Part2: Visual".
  3. ISO/IEC 14496-3:1999, "Information technology - Coding of audio-visual objects - Part3: Audio".
  4. ISO/IEC 14496-2:1999/Amd.1:2000, "Information technology - Coding of audio-visual objects - Part 2: Visual, Amendment 1: Visual extensions".
  5. ISO/IEC 14496-3:1999/Amd.1:2000, "Information technology - Coding of audio-visual objects - Part3: Audio, Amendment 1: Audio extensions".
  6. ISO/IEC 14496-1:1999, "Information technology - Coding of audio-visual objects - Part1: Systems".
  7. Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  8. Schulzrinne, H., Casner, S., Frederick, R. and V. Jacobson "RTP: A Transport Protocol for Real Time Applications", RFC 1889, January 1996.
  9. ISO/IEC 14496-2:1999/Cor.1:2000, "Information technology - Coding of audio-visual objects - Part2: Visual, Technical corrigendum 1".

8. Authors' Addresses 💜

Yoshihiro Kikuchi
Toshiba corporation
1, Komukai Toshiba-cho, Saiwai-ku, Kawasaki, 212-8582, Japan

EMail: yoshihiro.kikuchi@toshiba.co.jp


Yoshinori Matsui
Matsushita Electric Industrial Co., LTD.
1006, Kadoma, Kadoma-shi, Osaka, Japan

EMail: matsui@drl.mei.co.jp


Toshiyuki Nomura
NEC Corporation
4-1-1,Miyazaki,Miyamae-ku,Kawasaki,JAPAN

EMail: t-nomura@ccm.cl.nec.co.jp


Shigeru Fukunaga
Oki Electric Industry Co., Ltd.
1-2-27 Shiromi, Chuo-ku, Osaka 540-6025 Japan.

EMail: fukunaga444@oki.co.jp


Hideaki Kimata
Nippon Telegraph and Telephone Corporation
1-1, Hikari-no-oka, Yokosuka-shi, Kanagawa, Japan

EMail: kimata@nttvdt.hil.ntt.co.jp

Copyright (C) The Internet Society (2000). All Rights Reserved. 🌸

This document and translations of it may be copied and furnished to others. Derivative works that comment on or otherwise explain it, or help with its implementation, may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind. The one condition is that the above copyright notice and this paragraph are included on all such copies and derivative works 📝 However, this document itself may not be modified in any way, such as by removing the copyright notice or the references to the Internet Society or other Internet organizations. The only exceptions are when it's needed for developing Internet standards (in which case the copyright procedures defined in the Internet Standards process must be followed), or when it's translated into languages other than English 🌍

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. Forever and ever 💅

⚠️ This document and the information in it are provided on an "AS IS" basis. THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 🫶

Funding for the RFC Editor function is currently provided by the Internet Society. Thank you, Internet Society 🦋✨

📦 where this page comes from