RFC 4396
RTP Payload Format for 3rd Generation Partnership Project (3GPP) Timed Text
RFC 4396: RTP Payload Format for 3rd Generation Partnership Project (3GPP) Timed Text 🎀
- Series: Network Working Group 💌
- RFC number: 4396
- Authors: J. Rey and Y. Matsui, both from Panasonic 👑
- Category: Standards Track ✨
- Date: February 2006 🌸
- Obsoletes / Updates: the title block doesn't list any RFCs it obsoletes or updates 🙈
Part 1 of 15 🦋
Status of This Memo ✨
Okay so this one is an Internet standards track protocol for the Internet community 💜. It's basically asking everyone to discuss it and send ideas for making it better 🫶. If you want to know where this protocol stands in the official standardization state, check the current edition of the "Internet Official Protocol Standards" (STD 1) 📚. And yes, distribution of this memo is unlimited 🌈.
Copyright Notice 💅
Copyright (C) The Internet Society (2006). That's the legal bit, bestie 🧚
Abstract 🦋
Okay, the main vibe here: this document lays out an RTP payload format for sending 3GPP (3rd Generation Partnership Project) timed text around ✨. 3GPP timed text is text that's lined up with time and comes with decoration, and it has a defined way of being stored in a 3GP file 🎬. It can sync up with audio and video and is used for captioning, titling, and multimedia presentations 💖. The later sections cover the problems with streaming timed text and then spell out a payload format for streaming 3GPP timed text over RTP 🔥.
Table of Contents 🗂️
1. Introduction ....................................................3
2. Motivation, Requirements, and Design Rationale ..................3
2.1. Motivation .................................................3
2.2. Basic Components of the 3GPP Timed Text Media Format .......4
2.3. Requirements ...............................................5
2.4. Limitations ................................................6
2.5. Design Rationale ...........................................7
3. Terminology ....................................................10
4. RTP Payload Format for 3GPP Timed Text .........................12
4.1. Payload Header Definitions ................................13
4.1.1. Common Payload Header Fields .......................15
4.1.2. TYPE 1 Header ......................................17
4.1.3. TYPE 2 Header ......................................20
4.1.4. TYPE 3 Header ......................................23
4.1.5. TYPE 4 Header ......................................24
4.1.6. TYPE 5 Header ......................................25
4.2. Buffering of Sample Descriptions ..........................25
4.2.1. Dynamic SIDX Wraparound Mechanism ..................26
4.3. Finding Payload Header Values in 3GP Files ................28
4.4. Fragmentation of Timed Text Samples .......................31
4.5. Reassembling Text Samples at the Receiver .................33
4.6. On Aggregate Payloads .....................................35
4.7. Payload Examples ..........................................39
4.8. Relation to RFC 3640 ......................................43
4.9. Relation to RFC 2793 ......................................44
5. Resilient Transport ............................................45
6. Congestion Control .............................................46
7. Scene Description ..............................................47
7.1. Text Rendering Position and Composition ...................47
7.2. SMIL Usage ................................................48
7.3. Finding Layout Values in a 3GP File .......................48
8. 3GPP Timed Text Media Type .....................................49
9. SDP Usage ......................................................53
9.1. Mapping to SDP ............................................53
9.2. Parameter Usage in the SDP Offer/Answer Model .............53
9.2.1. Unicast Usage ......................................54
9.2.2. Multicast Usage ....................................57
9.3. Offer/Answer Examples .....................................58
9.4. Parameter Usage outside of Offer/Answer ...................60
10. IANA Considerations ...........................................60
11. Security Considerations .......................................60
12. References ....................................................61
12.1. Normative References .....................................61
12.2. Informative References ...................................61
13. Basics of the 3GP File Structure ..............................64
14. Acknowledgements ..............................................651. Introduction 🎀
3GPP timed text is a media format for time-lined, decorated text, and it's specified in the 3GPP Technical Specification TS 26.245, "Transparent end-to-end packet switched streaming service (PSS); Timed Text Format (Release 6)" [1]. Okay, the iconic part is that it's not just plain text 💅. It can do decorated text too, like karaoke, scrolling newscast text, or hyperlinked text ✨. This text may or may not be synced up with other media like audio or video 🎶.
The purpose of this document is to give you a way to stream 3GPP timed text using RTP [3] 🦋. That covers streaming timed text read out of a (3GP) file, and also streaming timed text that's generated in real time, aka live streaming 🔴.
Here's the tour 🗺️: Section 2 has the motivation, an overview of the media format, the requirements, and the design rationale 💭. Section 3 defines the terminology 📖. Section 4 specifies the payload headers, the fragmentation and re-assembly rules for text samples, the rules for payload aggregation, and how this document relates to RFC 3640 [12] and RFC 2793 [22] 🔗. Section 5 covers some simple schemes for resilient transport and points you to other possible mechanisms 🛡️. Section 6 handles congestion control 🚦. Section 7 specifies scene description 🎬. Section 8 defines the media type 🏷️. Section 9 specifies SDP for unicast and multicast sessions, including usage in the Offer/Answer model [13] 📡. Sections 10 and 11 cover IANA and security considerations 🔐. Section 12 lists references 📚. And the Basics of the 3GP File Structure are in Section 13 🧚.
2. Motivation, Requirements, and Design Rationale 💖
2.1. Motivation 🌸
The 3GPP timed text format was built for the services in the 3GPP Transparent End-to-end Packet-switched Streaming Services (3GPP PSS) specification [16] 📱.
Right now PSS lets you download 3GPP timed text contents stored in 3GP files 📥. But since there's no RTP payload format for it, you literally can't stream 3GPP timed text contents over RTP 😭. That's the gap this document fills ✨.
2.2. Basic Components of the 3GPP Timed Text Media Format 🦋
Before we get into the design details, you need to know how the media format is built 🧩. There are four functional components: layout information, default formatting, text strings, and decoration 👑. Here's each one and where it lives in a 3GP file (the 3GP file names are capitalized so they're easy to spot) 💅:
- Initial spatial layout information for the text strings: this is the height and width of the text region where text shows up, the position of that region on the display, and the layer or how close the text sits to the user 📐. In 3GP files, this lives in the Track Header Box 📦.
- Default settings for formatting and positioning text: style (font, size, color, and so on), background color, horizontal and vertical justification, line width, scrolling, etc. 🎨. For 3GP files, this maps to the Sample Descriptions 🗂️.
- The actual text strings: characters encoded with either UTF-8 [18] or UTF-16 [19] 🔤.
- The decoration: if some characters need a different style, delay, blink, etc., that has to be flagged 🌈. The decoration only shows up in the text samples when it's actually needed. Otherwise the default settings above apply 💫. In 3GP files, inside each Text Sample, the decoration (i.e., Modifier Boxes) gets tacked onto the text strings when needed 🎀. When this payload format was written, the 3GPP timed text media format specification [1] specified these modifiers:
- text highlight 🖍️
- highlight color 🎨
- blinking text ✨
- karaoke feature 🎤
- hyperlink 🔗
- text delay ⏳
- text style 💄
- positioning of the text box 📍
- text wrap indication 🧵
Heads up, bestie: this part is just the groundwork. Sections 2.3 onward get into the actual requirements and design choices 👀
2.3. Requirements
okay so once we know the basic components, we need to figure out what requirements the payload format shall fulfill 💅
- It shall enable both live streaming and streaming from a 3GP file.
Informative note: For the purpose of this document, the term "live streaming" refers to those scenarios where the timed text stream is sent from a live encoder. Upon reception, the content may or may not be stored in a 3GP file. Typically, in live streaming applications, the sender encapsulates the timed text content in RTP packets following the guidelines given in this document. At the receiving side, a buffer is used to cancel the network delay and delay jitter. If receiver and sender support packet loss resilience mechanisms (see Section 5), it may also be possible to recover from packet losses. Note that how sender and receiver actually manage and dimension the buffers is an implementation design choice. 🎀
- Furthermore, it shall be possible for an RTP receiver using this payload format, and capable of storing in 3GP format, to obtain all necessary information from the RTP packets for storing the received text contents according to the 3GP file format. This file may or may not be the same as the original file. ✨
Informative note: The 3GP file format itself is based on the ISO Base Media File Format recommendation [2]. Section 13.1 gives some insight into the 3GP file structure. Further, Sections 4.3 and 7.3 specify where the information needed for filling in payload headers is found in a 3GP file. For live streaming, appropriate values complying with the format and units described in [1] shall be used. Where needed, clarifications on appropriate values are given in this document. 🧚
- It shall enable efficient and resilient transport of timed text contents over RTP. In particular:
- a. Enable the transmission of the sample descriptions by both out-of-band and in-band means. Sample descriptions are important information, which potentially apply to several text samples. These default formatting settings are typically transmitted out-of-band (reliably) once at the initialization phase. If additional sample descriptions are needed in the course of a session, these may also be sent out-of-band or in-band. In-band transmission, although unreliable, may be more appropriate for sending sample descriptions if these should be sent frequently, as opposed to establishing an additional communication channel for SDP, for example. It is also useful in cases where an out-of-band channel may not be available and for live streaming, where contents are not known a priori. Thus, the payload format shall enable out-of-band and in-band transmission of sample descriptions. Section 4.1.6 specifies a payload header for transmitting sample descriptions in-band. Section 9 specifies how sample descriptions are mapped to SDP. 📦
- b. Enable the fragmentation of a text sample into several RTP packets in order to cover a wide range of applications and network environments. In general, fragmentation should be a rare event, given the low bit rates and relatively small text sample sizes. However, the 3GPP Timed Text media format does allow for larger text samples. Therefore, the payload format shall take this into account and provide a means for coping with fragmentation and reassembly. Section 4.4 deals with fragmentation. 🦋
- c. Enable the aggregation of units into an RTP packet for making the transport more efficient. In a mobile communication environment, a typical text sample size is around 100-200 bytes. If the available bit rate and the packet size allow it, units should be aggregated into one RTP packet. Section 4.6 deals with aggregation. 🌸
- d. Enable the use of resilient transport mechanisms, such as repetition, retransmission [11], and FEC [7] (see Section 5). For a more general discussion, refer to RFC 2354 [8], which discusses available mechanisms for stream repair. 🛡️
2.4. Limitations
okay the payload headers were optimized in size for RTP, and that's a whole vibe 🌈 Instead of using 32-bit (S)LEN, SDUR, and SIDX header fields, which would carry many unused bits much of the time, it was a design choice to reduce the size of these fields. As a consequence, this payload format has reduced maximum values with respect to sizes and durations of (text) samples and sample descriptions. These maximum values differ from those allowed in 3GP files, where they are expressed using 32-bit (unsigned) integers. In some cases, extension mechanisms are provided to deal with larger values. However, it is noted that the values used here should be enough for the streaming applications targeted. 💜
The following limitations apply:
- The maximum size of text samples carried in RTP packets is restricted to be a 16-bit (unsigned) integer (this includes the text strings and modifiers). This means a maximum size for the unit would be about 64 Kbytes. No extension mechanism is provided. 📏
- The sample description index values are restricted to be an 8-bit (unsigned) integer. An extension mechanism is given in Section 4.3. 🔢
- The text sample duration is restricted to be a 24-bit (unsigned) integer. This yields a maximum duration at a timestamp clockrate of 1000 Hz of about 4.6 hours. Nevertheless, an extension mechanism is provided in Section 4.3. ⏳
- Sample descriptions are also restricted in size: If the size cannot be expressed as a 16-bit (unsigned) integer, the sample description shall not be conveyed. As in the case of the sample size, no extension mechanism is provided. 🙈
- A further limitation concerns the UTF-16 encodings supported: Only transport of text strings following big endian byte order is supported. See Section 4.1.1 for details. 🔤
2.5. Design Rationale
the following design choices were made, and honestly they're iconic 👑
- 'Unit' approach: The payload formats specified in this document follow a simple scheme: a 3-byte common header (Common Payload Header) followed by a specific header for each text sample (fragment) type. Following these headers, the text sample contents are placed (Section 4.1.1 and following). This structure is called a 'unit'.
The following units have been devised to comply with the requirements mentioned in Section 2.3:
- a. A TYPE 1 unit that contains one complete text sample,
- b. A TYPE 2 unit that contains a complete text string or a fragment thereof,
- c. A TYPE 3 unit that contains the complete modifiers or only the first fragment thereof,
- d. A TYPE 4 unit that contains one modifier fragment other than the first, and
- e. A TYPE 5 unit that contains one sample description.
This 'unit' approach was motivated by the following reasons:
- Allows a simple classification of the text samples and text sample fragments that can be conveyed by the payload format.
- Enables easy interoperability with RFC 3640 [12]. During the development of this payload format, interest was shown from MPEG-4 standardization participants in developing a common payload structure for the transport of 3GPP Timed Text. While interoperability is not strictly necessary for this payload format to work, it has been pursued in this payload format. Section 4.8 explains how this is done. 🤝
- Character count is not implemented. This payload format does detect lost text samples fragments, but it does not enable an RTP receiver to find out the exact number of text characters lost. In fact, the fragment size included in the payload headers does not help in finding the number of lost characters because the UTF-8/UTF-16 [18][19] encodings used yield a variable number of bytes per character. 🤔
For finding the exact number of lost characters, an additional field reflecting the character count (and possibly the character offset) upon fragmentation would be required. This would additionally require that the entity performing fragmentation count the characters included in each text fragment.
One benefit of having a character count would be that the display application would be able to replace missing characters through some other character representing character loss. For example:
If we take the "Some text is lost now" and assume the loss of a packet containing the text in the middle, this could be displayed (with a character count):
"Some ############now"As opposed to:
"Some #now"which is what this payload format enables ("#" indicates a missing character or packet, respectively).
However, it is the consensus of the working group that for applications such as subtitling applications and multimedia presentations that use this payload format, such partial error correction is not worth the cost of including two additional fields; namely, character count and character offset. Instead, it is recommended that some more overhead be invested to provide full error correction by protecting the less text sample fragments using the measures outlined in Section 5. 🛠️
- Fragment re-assembly: In order to re-assemble the text samples, offset information is needed. Instead of a character or byte offset, a single byte, TOTAL/THIS, is used. These two values indicate the total number and current index of fragments of a text sample. This is simpler than having a character offset field in each fragment. Details in Section 4.1.3. 🧩
- A length field, LEN, is present in the common header fields. While the length in the RTP payload format is not needed by most RTP applications (typically lower layers, like UDP, provide this information), it does ease interoperability with RFC 3640. This is because the Access Units (AUs) used for carriage of data in RFC 3640 must include a length indication. Details are in Section 4.8. 📏
- The header fields in the specific payload headers (TYPE headers in Sections 4.1.2 to 4.1.6) have been arranged for easy processing on 32-bit machines. For this reason, the fields SIDX and SDUR are swapped in TYPE 1 unit, compared to the other units. 💻
3. Terminology 🎀
Okay bestie, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this doc mean exactly what RFC 2119 [5] says they mean 💅 no cap.
Okay, and these terms have very specific meanings in this doc, so pay attention 🤓
text sample or whole text sample 🌸 In the 3GPP Timed Text media format [1], a text sample is one unit of timed text data in the source (3GP) file. It includes the text string byte count, maybe a Byte Order Mark, the text string, and any modifiers that come after. The audio/video version of this is a frame 🎬
Here though, a text sample is just the text strings plus zero or more modifiers. So this definition leaves out the 16-bit text string byte count and the 16-bit Byte Order Mark (BOM) that show up in 3GP file text samples (see Section 4.3 and Figure 9). The 16-bit BOM doesn't get sent over RTP, as Section 4.1.1 explains 🙈
text strings 💬 The actual text characters, encoded as UTF-8 or UTF-16. When you use this payload format, the text string does not contain any byte order mark (BOM). See Figure 9 for details.
fragment or text sample fragment 🧩 A fraction of a text sample. A fragment may contain either text strings or modifier (decoration) contents, but not both at the same time.
sample contents 📦 A general term for the timed text data carried with this payload format. Sample contents may be one or several text samples, sample descriptions, and sample fragments. Note that, as per Section 4.6, there is only one case in which more than one fragment may be included in a payload.
decoration or modifiers ✨ These are used interchangeably throughout the doc to mean the contents of a text sample that change the default text formatting. Modifiers can, for example, specify a different font size for a particular run of characters, or define karaoke timing for the sample.
sample description 📝 Info that could be shared by more than one text sample. In a 3GP file, a sample description is stored somewhere it can be shared. It holds setup and default info like scrolling direction, text box position, delay value, default font, background color, etc.
units or transport units 🚚 The payload headers in this doc wrap text samples, fragments of them, and sample descriptions. They do this by putting a common header and a specific payload header (Sections 4.1.1 to 4.1.6) in front of them, which builds what this doc calls a (transport) unit.
aggregation or aggregate packet 🧺 The payload of an aggregate (RTP) packet is made up of several (transport) units.
track or stream 🎞️ 3GP files contain audio/video and text tracks. This doc lets you stream text tracks using RTP. So in the context of 3GP files, these terms are used interchangeably.
Media Header Box / Track Header Box / ... 📦 The 3GP file format uses these structures, which are defined in the ISO Base File Format [2]. When this doc refers to them, the initials are capitalized for clarity.
4. RTP Payload Format for 3GPP Timed Text ✨
Here's what an RTP packet carrying 3GPP timed text looks like 👀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
/+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |U| R | TYPE| LEN | :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ :
U| : (variable header fields depending on TYPE :
N| : :
I< +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
T| | |
| : SAMPLE CONTENTS :
| | +-+-+-+-+-+-+-+-+
| | |
\+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1. 3GPP Timed Text RTP Packet FormatMarker bit (M): The marker bit SHALL be set to 1 if the RTP packet includes one or more whole text samples or the last fragment of a text sample. Otherwise, it is set to zero (0) 🚩
Timestamp: The timestamp MUST indicate the sampling instant of the earliest (or only) unit contained in the RTP packet. The initial value SHOULD be randomly determined, as specified in RTP [3] 🎲
The timestamp value should provide enough timing resolution for expressing the duration of text samples, for synchronizing text with other media, and for performing RTP Control Protocol (RTCP) measurements such as the interarrival delay jitter or the RTCP Packet Receipt Times Report Block (Section 4.3 of RFC 3611 [20]). This is compliant to RTP, Section 5.1:
"The resolution of the clock MUST be sufficient for the desired synchronization accuracy and for measuring packet arrival jitter (one tick per video frame is typically not sufficient)".
The above observation applies to both timed text tracks included in a 3GP file and live streaming sessions. For a 3GP timed text track, the timestamp clockrate is the value of the "timescale" parameter in the Media Header Box for that text track. Each track in a 3GP file MAY have its own clockrate, as specified in the Media Header Box. Likewise, live streaming applications SHALL use an appropriate timestamp clockrate. A default value of 1000 Hz is RECOMMENDED. Other timestamp clockrates MAY be used. In that case, the typical behavior here is to match the 3GPP timed text clockrate to the one used by an associated audio or video stream ⏱️
In an aggregate payload, units MUST be placed in play-out order, i.e., earliest first in the payload. If TYPE 1 units are aggregated, the timestamp of the subsequent units MUST be obtained by adding the timed text sample duration of previous samples to the RTP timestamp value. There are two exceptions to this rule: TYPE 5 units and an aggregate payload containing two fragments of the same text sample. The details of the timestamp calculation are given in Section 4.6 🧮
Finally, timestamp clockrates MUST be signaled by out-of-band means at session setup, e.g., using the media type "rate" parameter in SDP. See Section 9 for details 📡
Payload Type (PT): The payload type is set dynamically and sent by out-of-band means 🫶
The usage of the remaining RTP header fields (namely, V, P, X, CC, SN and SSRC) follows the rules of RTP and the profile in use.
4.1. Payload Header Definitions 🧱
The (transport) units specified in this doc are built from a set of common fields (U, R, TYPE, LEN), followed by specific header fields (TYPES 1-5) and text sample contents. See Figure 1 and Figure 2.
In Figure 2, two example RTP packets are shown. The first contains an aggregate RTP payload with two complete text samples, and the second contains one text sample fragment. After each unit header is explained, detailed payload examples follow in Section 4.7 🎀
+----------------------+
| |
| RTP Header |
| |
---------+----------------------+
| | |
| |COMMON + TYPE 1 Header|
| ........................
UNIT 1 - | |
| | Text Sample |
| | |
|-------\........................
-------/| |
| |COMMON + TYPE 1 Header|
| ........................
UNIT 2 - | |
| | Text Sample |
| | |
| | |
---------+----------------------+
+----------------------+
| |
| RTP Header |
| |
---------+----------------------+
| | COMMON + TYPE 2 |
| | (or 3 or 4) Hdr |
| ........................
UNIT 3 - | |
| | Text Sample Fragment |
| | |
| | |
---------+----------------------+
Figure 2. Example RTP packets4.1.1. Common Payload Header Fields ✨
okay so every payload header starts with the same little crew of fields, bestie 🎀 here's the layout:
0 1 2
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3. Common payload header fields 🦋
okay so here's what each piece does 💅:
- U (1 bit) "UTF Transformation flag" 💖: this tells RTP receivers whether the text string was encoded in UTF-8 (
U=0) or UTF-16 (U=1). UTF-16 text strings sent with this payload format MUST be serialized in big endian order, a.k.a. network byte order. 🌊
Informative note 🤓: timed text clients that follow the 3GPP Timed Text format [1] are only required to understand the big endian serialization. Thus, in order to ease interoperability, the reverse serialization (little endian) is not supported by this payload format. so iconic of them to keep it simple 🙈
For the payload formats defined in this document, the U bit is only used in TYPE 1 and TYPE 2 headers. Senders MUST set the U bit to zero in TYPE 3, TYPE 4, and TYPE 5 headers. Consequently, receivers MUST ignore the U bit in TYPE 3, TYPE 4, and TYPE 5 headers. not me being like "wait the U bit just chills here" 🫶
- R (4 bits) "Reserved bits" 🧚: these are saved for future extensions. This field MUST be set to zero (
0x0) and MUST be ignored by receivers. hands off, babe 🚫 - TYPE (3 bits) "Type Field" 👑: this one is the boss, bc it says which specific header fields come next. The TYPE values defined so far are:
- TYPE 1, for a whole text sample. ✨
- TYPE 2, for a text string fragment (without modifiers). 🍓
- TYPE 3, for a whole modifier box or the first fragment of a modifier box. 💜
- TYPE 4, for a modifier fragment other than first. 🌸
- TYPE 5, for a sample description. Exactly one header per sample description. 🎀
- TYPE 0, 6, and 7 are reserved for future extensions. Note that future extensions are possible, e.g., a unit that explicitly signals the number of characters present in a fragment (see Section 2.5). In order to guarantee backwards-compatibility, it SHALL be possible that older clients ignore (newer) units they do not understand, without invalidating the timestamp calculation mechanisms or otherwise preventing them from decoding the other units. 🔮
- Finally, the LEN (16 bits) "Length Field" 📏: this indicates the size (in bytes) of this header field and all the fields following it, i.e., the LEN field followed by the unit payload: text strings and modifiers (if any). This definition only excludes the initial U/R/TYPE byte of the common header. The LEN field follows network byte order. 🌈
The way LEN gets calculated when you stream out of a 3GP file depends on the unit type. each unit explains its own version in the sections below 📖
For live streaming, both sample length and the LEN value for the current fragment MUST be calculated during the sampling process or during fragmentation. bestie, this part is non-negotiable 💀
In general, LEN may take the following values:
TYPE = 1, LEN >= 8💖TYPE = 2, LEN > 9🦋TYPE = 3, LEN > 6🌸TYPE = 4, LEN > 6✨TYPE = 5, LEN > 3🧚
Receivers MUST discard units that do not comply with these values. However, the RTP header fields and the rest of the units in the payload (if any) are still useful, as guaranteed by the requirement for future extensions above. so even if one unit gets tossed, the rest of the payload still slaps 🫧
In the following subsections, the different payload headers for each TYPE value are specified. let's gooo 💜
4.1.2. TYPE 1 Header 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN (always >=8) | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | TLEN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TLEN |
+-+-+-+-+-+-+-+-+Figure 4. TYPE 1 Header Format
okay so this header type is the one for whole text samples 💖 It's the most common case by far, because the text sample should usually be small enough to ride along in one unit without having to split the text strings from the modifiers. And when an aggregate (RTP packet) payload holds several text samples, every single sample gets its own TYPE 1 header (see Figure 12) 🧚
Informative note: okay, quick refresher from Section 3, "Terminology"! A text sample is the text strings followed by the modifiers (if there are any). That's also how text samples live in 3GP files 📂 Splitting a sample into strings and modifiers is only needed for big samples (or when the available IP MTU is small; see Section 4.4), and that's what the TYPE 2 and TYPE 3 headers are for, as you'll see in the sections below 🦋
Also, empty text samples count as whole text samples, even though they carry no sample contents 🙈 They're super handy for clearing the display or ending a sample of unknown duration, for example. Units without sample contents SHALL have a LEN field value of 8 (0x0008).
The fields, bestie:
- U, R, and TYPE, as defined in Section 4.1.1.
- LEN is the length of the complete text sample plus eight (8) bytes of headers 💅 To find the sample length in the Sample Size Box of 3GP files, see Section 4.3.
- SIDX (8 bits), "Text Sample Entry Index": this is the index that identifies the sample descriptions 🌈
- The SIDX field tells you which sample description goes with the unit's payload. There are two flavours of SIDX values: static and dynamic.
- Static SIDX values identify sample descriptions that MUST be sent out-of-band and MUST remain active for the whole session 🔒 Each static SIDX is permanently tied to one specific sample description for the entire session. Carrying lots of sample descriptions out-of-band SHOULD be avoided, since they can get big, and transport isn't really the job of the out-of-band channel 🙅♀️ So this feature is RECOMMENDED for the sample descriptions that give a set of minimum default format settings. Static SIDX values MUST fall in the (closed) interval [129,254].
- Dynamic SIDX values are for sample descriptions sent in-band ✨ They MAY be sent in-band for a few reasons: they're generated in real time, for transport resiliency, or both. A dynamic SIDX is tied to one particular sample description for as long as it's active in the session, and it SHALL NOT be modified during that period 😤 That period MAY be smaller than or equal to the session duration, and it isn't known a priori. At most 64 dynamic SIDX values can be simultaneously active at any moment. Dynamic SIDX values MUST fall in the closed interval [0,127]. That should be plenty for both recorded content and live streaming. Still, Section 4.2.1 gives you a wraparound mechanism for streaming sessions that might need more than 64 SIDX values, so you're covered 🫶 Servers MAY make use of dynamic sample descriptions, and clients MUST be able to receive and interpret them 💜
- Finally, SIDX values 128 and 255 are reserved for future use 🚧
- SDUR (24 bits), "Text Sample Duration": this tells you the sample duration in RTP timestamp units of the text sample ⏱️ For this field, 3 bytes is preferred over 2 bytes, and here's the tea 🍵 At a typical clockrate of 1000 Hz, 16 bits would only stretch to about 65 seconds, which could be way too short for some streams. But 24 bits at 1000 Hz gives you a max of about 4.6 hours, and at 90 KHz it's about 3 minutes. Those numbers should be enough for streaming applications. If you ever need something longer, though, the extension mechanism in Section 4.3 SHALL be used 🎀
- Besides saying how long the text stays on screen, the duration also helps find the timestamp of the later units inside an aggregate RTP packet payload (if there are any). That's explained in Section 4.6 🧩
- Text samples usually have a known duration when they're sent 📅 But sometimes, like in live streaming, nobody knows up front how long a text piece should be shown. So the value zero, SDUR=0 (0x000000), is reserved to signal unknown duration 🌸 A sample of unknown duration is shown until the next text sample that should be displayed becomes active, as marked by its timestamp. Text samples of unknown duration SHALL be displayed until the next text sample becomes active, as indicated by its timestamp.
- The next example shows how units of unknown duration MUST be presented 📺 If no following text sample is available, what gets displayed is an implementation issue. For example, a server could send an empty sample to clear the text box.
Example: Picture this, you're at the airport watching the latest news while you wait for your flight ✈️ Airports are loud, so the news report gets transcribed in the lower part of the screen. That area shows two lines: the headlines and the words the speaker is saying. The headlines stay up way longer than the rest, obviously 📰 The stream server doesn't really know how long each one should stay, because it's streaming live. So a headline just gets swapped out when the next headline arrives 💅
- However, when a text sample with SDUR=0 gets stored in a 3GP file, its SDUR value MUST be changed to the effective duration of the text sample, and that duration MUST always be greater than zero ⚠️ (The ISO file format [2] explicitly forbids a sample duration of zero, so no zeros allowed, periodt 🙅♀️) The effective duration MUST be calculated as the timestamp difference between the current sample (the one with unknown duration) and the next text sample that gets displayed 🧮
- Samples of unknown duration SHALL NOT use features that need to know the sample's duration up front 🙈 Those features are scrolling and karaoke in [1]. The same goes for future extensions of the Timed Text format. Also, only sample descriptions (TYPE 5 units) MAY follow units of unknown duration in the same aggregate payload. Otherwise there'd be no way to calculate the timestamps of those other units, and that's a whole mess 😭
- For text contents stored in 3GP files, check Section 4.3 to see how to pull out the duration value. For live streaming, live encoders SHALL assign appropriate values and units according to [1] and later releases 🎥
- TLEN (16 bits), "Text String Length", is a byte count of the text string 🔢 The decoder needs this length to know where the modifiers in the payload start. TLEN is not present in text string fragments (TYPE 2), since it can be worked out from the LEN values of each fragment 🧠
- The TLEN value comes from the text samples as they're contained in 3GP files, so refer to Section 4.3. For live content, the TLEN MUST be obtained during the sampling process 📸
- Last but not least, the actual text sample sits right after the TLEN field 💖 As defined in Section 3, a text sample is a string of characters encoded in either UTF-8 or UTF-16, followed by zero or more modifiers. Also note that the strings carried in the payload have no BOM and no byte count (unlike text samples stored in 3GP files [1]) ✨
4.1.3. TYPE 2 Header ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN( always >9) | TOTAL | THIS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SLEN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5. TYPE 2 Header FormatOkay, so TYPE 2 headers are for carrying either a whole text string or just a fragment of one 💖 Big rule first: TYPE 2 units SHALL NOT contain modifiers. No modifiers, no exceptions, periodt 💅
Here's the breakdown:
- U, R, and TYPE work exactly like they do in Section 4.1.1 🎀
- SIDX and SDUR work exactly like they do in Section 4.1.2 🦋
Note that the U, SIDX, and SDUR fields are meaningful since partial text strings can also be displayed. So even when you only get a piece of the text, these fields still matter ✨
- LEN is a 16-bit field that tells you the length of the text string fragment plus nine (9) bytes of headers 🌸 It gets calculated when the fragmentation happens. LEN MUST always be greater than nine (0x0009). If it isn't, the unit MUST be discarded. No second chances 💀
Now, according to the guidelines in Section 4.4, text strings MUST be split at character boundaries so that text fragments can be displayed 🧚 That means every text fragment MUST contain at least one character in either UTF-8 or UTF-16. Honestly that's kind of a formality, because if you follow the guidelines you'll end up making way bigger fragments anyway 🙈
Also, TYPE 2 units don't have an explicit text string length (TLEN) field the way TYPE 1 does (see TYPE 1). That's because TYPE 2 units never carry modifiers after the text string. If you do need the length of what you received, you can work it out from the LEN values of the TYPE 2 units 🍓
- SLEN is a 16-bit field with the size (in bytes) of the original, whole text sample this fragment belongs to 💜 That size covers the text string plus any modifier boxes present, and it does NOT include the byte order mark or the text string length mentioned in Section 3, "Terminology".
Real talk about sample length: timed text samples aren't generated at regular intervals, and there's no default sample size either 😭 If 3GP files are being streamed, the sample lengths get calculated ahead of time and put into the track itself. For live encoding, it's the real-time encoder that SHALL choose an appropriate size for each text sample. The amount of text that gets "captured" in one sample depends on the text source and the application (see examples below). Samples can be tailored to match the packet MTU as closely as possible, or to give a certain amount of redundancy for the bit rate you have available 🌈 The encoding application MUST also take the delay constraints of the real-time session into account, and check whether FEC, retransmission, or other similar techniques make sense for repairing the stream 🔥
The examples below show how a real-time encoder might pick its settings for different scenarios. The examples themselves are copied exactly as written:
Example: Imagine a newscast scenario, where the spoken news is transcribed and synchronized with the image and voice of the reporter. We assume that the news speaker talks at an average speed of 5 words per second with an average word length of 5 characters plus one space per word, i.e., 30 characters per second. We assume an available IP MTU of 576 bytes and an available bitrate of 576*8 bits per second = 4.6 Kbps. We assume each character can be encoded using 2 bytes in UTF-16. In this scenario, several constraints may apply; for example: available IP MTU, available bandwidth, allowable delay, and required redundancy. If the target were to minimize the packet overhead, a text sample covering 8 seconds of text would be closest to the IP MTU:
IP/UDP/RTP/TYPE1 Header + (8-second text sample)
= 20 + 8 + 12 + 8 + (~6 chars/word * 5 word/s * 8 s * 2 chars/word)
= 528 bytes < 576 bytesFor other scenarios, like lossy networks, it may happen that just one packet per sample is too low a redundancy. In this case, a choice could be that the encoder 'collects' text every second, thus yielding text samples (TYPE 1 units) of 68 bytes, TYPE 1 header included. We can, e.g., include three contiguous text samples in one RTP payload: the current and last two text samples (see below). This accounts to a total IP packet size of 20 + 8 + 12 + 3*(8 + 60) = 244 bytes. Now, with the same available bitrate of 4.6 Kbps, these 244-byte packets can be sent redundantly up two times per second:
RTP payload (1,2,3)(1,2,3) (2,3,4)(2,3,4) (3,4,5)(3,4,5) ...
Time: <----1s------> <----1s------> <-----1s-----> ...This means that each text sample is sent at least six times, which should provide enough redundancy. Although not as bandwidth efficient (488*8 < 528*8 < 576*8 bps) as the previous packetization, this option increases the stream redundancy while still meeting the delay and bandwidth constraints.
Another example would be a user sending timed text from a type-in area in the display. In this case, the text sample is created as soon as the user clicks the 'send' button. Depending on the packet length, fragmentation may be needed.
In a video conferencing application, text is synchronized with audio and video. Thus, the text samples shall be displayed long enough to be read by a human, shall fit in the video screen, and shall 'capture' the audio contents rendered during the time the corresponding video and audio is rendered.
For stored content, see Section 4.3 for details on how to find the SLEN value in a 3GP file. For live content, the SLEN MUST be obtained during the sampling process.
Last but not least, clients MAY use SLEN to buffer space for the remaining fragments of a text sample 🧺
- TOTAL (4 bits) and THIS (4 bits) tell you how many fragments the original text sample (the text string and its modifiers) got chopped into, and which one of those fragments you're looking at right now 🎀 A sequence number alone can't do what THIS does, because packets (and fragments) may get repeated, for example in repeated transmission (see Section 5). So you need some kind of "fragment offset" indication.
The usual "byte offset" field isn't used here, for two reasons: a) it would take one more byte, and b) it doesn't tell you anything about the character offset. UTF-8/UTF-16 text strings have, in general, a variable character length ranging from 1 to 6 bytes, so the TOTAL/THIS solution is preferred 💅 People could argue the LEN and SLEN fields should do this job instead. They do tell you about how complete the text sample is, but they don't tell you what order the fragments go in, so no 🙅♀️
In all cases (TYPEs 2, 3 and 4), if the value of THIS is greater than TOTAL or if TOTAL equals zero (0x0), the fragment SHALL be discarded. Bye bye 👋
- Finally, the sample contents that come after the SLEN field are a fragment of the UTF-8/UTF-16 character string, and no modifiers follow 🫶
4.1.4. TYPE 3 Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN( always >6) |TOTAL | THIS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6. TYPE 3 Header FormatTYPE 3 headers are the modifier crew 🦋 They carry either the entire modifier contents of a text sample or just the first fragment of them. Which one you get depends on whether the modifier boxes fit in the current RTP payload 🌸
If a text sample containing modifiers gets fragmented, this header MUST be used to transport the first fragment or, if possible, the complete modifiers 🧚
Here's the detail:
- The U, R, and TYPE fields are defined as in Section 4.1.1 🎀
- LEN tells you the length of the modifier contents. Its value gets obtained upon fragmentation. The LEN field MUST be greater than six (0x0006). Otherwise, the unit MUST be discarded 💀
- The TOTAL/THIS field means the same thing it does for TYPE 2 🍓
For TYPE 3 units that contain the last (trailing) modifier fragment, the value of TOTAL MUST be equal to that of THIS (TOTAL=THIS). On top of that, TOTAL=THIS MUST be greater than one, because the total number of fragments of a text sample is logically always larger than one 🙈
If TOTAL is different from THIS in a TYPE 3 unit, that means the unit is carrying the first fragment of the modifiers 🫧
- The SDUR has the same definition as it does for TYPE 1. Since the fragments are always transported in their own RTP packets, this field is only needed to figure out how long this fragment is valid 🌈 It might, for example, be used to decide how long the fragment should stay in the display buffer ✨
Note that the SLEN and SIDX fields are NOT present in TYPE 3 unit headers 🙅♀️ That's because a) these fragments don't contain text strings, and b) these fragments get applied over text string fragments, which already carry that information 🍓
4.1.5. TYPE 4 Header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN( always >6) |TOTAL | THIS |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7. TYPE 4 Header FormatTYPE 4 is the header you use for modifier fragments that aren't the first one 🦋 Think of it as the sequel to TYPE 3 👑
- The U, R, and TYPE fields are used as per Section 4.1.1 🎀
- LEN indicates the length of the modifier contents, same as in TYPE 3, and it SHALL also be obtained upon fragmentation. The LEN field MUST be greater than six (0x0006). Otherwise, the unit MUST be discarded 💀
- TOTAL/THIS gets used the same way as in TYPE 2 🌸
- The SDUR field is defined as in TYPE 1 ✨
The reason SLEN and SIDX are absent here is the same as for TYPE 3 units 🧚
4.1.6. TYPE 5 Header ✨
Here's the bit diagram, bestie 🦋 Keep an eye on it, it's literally the whole vibe:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE | LEN( always >3) | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 8. TYPE 5 Header Format 🎀
Okay so this header is used to carry (dynamic) sample descriptions 💜 Every sample description MUST have its own TYPE 5 header, no exceptions, periodt 💅
The U, R, and TYPE fields work exactly like they do in Section 4.1.1, so no new drama there 🌸
The LEN field tells you the length of the sample description, plus three units to cover the SIDX and LEN field itself. So this field MUST be greater than three (0x0003). If it isn't, the unit MUST be discarded 🙈
If the sample is being streamed from a 3GP file, the length of the sample description contents (aka whatever comes after SIDX in the unit) gets pulled from the file. See Section 4.3 for that 🧚
The SIDX field holds a dynamic SIDX value that's assigned to the sample description riding along as the content of this unit. Since only dynamic sample descriptions go in TYPE 5, the possible SIDX values are in the (closed) interval [0,127] ✨
Now the fun part: Senders MAY make use of TYPE 5 units 🦋 But all receivers MUST implement support for TYPE 5 units, because it adds minimum complexity and may make the streaming session more robust. Iconic behaviour from the spec, honestly 👑
The next section explains how SIDX values get calculated, so stay tuned 🫶
4.2. Buffering of Sample Descriptions 🌈
Buffering sample descriptions is up to the client's timed text codec implementation 💅 Still, for this payload format to work properly, it requires that:
- Static sample descriptions MUST be buffered at the client, at least, for the duration of the session. 🍓
- If dynamic sample descriptions are used, their buffering and update of the SIDX values MUST follow the mechanism described in the next section. 🧚
4.2.1. Dynamic SIDX Wraparound Mechanism 💀
Using dynamic sample descriptions is OPTIONAL for senders 🌸 Here's the catch though: if they do use them, senders MUST implement this mechanism. Receivers MUST always implement it, no cap 😭
Dynamic SIDX values stay active either for the entire session (if used just once) or in different intervals of it (if used once or more). 🦋
Note: In the following, SIDX means dynamic SIDX.
So here's the rationale for picking the wraparound mechanism 🎀 There are 128 dynamic SIDX values possible, [0..127]. If you let a maximum of 127 be used as dynamic SIDXs, then any reordered packet with a new sample description would make the mechanism fail, and that's giving chaos 💀 For example, if the last packet received is SIDX=5, then all 127 values except SIDX=6 would be "active". Now if a reordered packet shows up with a new description, SIDX=9, it gets mistakenly discarded, because SIDX=9 is marked as "active" at that moment, and active sample descriptions shall not be re-written. Not it 🙅♀️
So the spec adds a "guard interval" 🛡️ It cuts the number of active SIDXs at any point in time down to 64. Most timed text apps will probably need fewer than 64 sample descriptions in total during a session, but a wraparound mechanism is described here for when they need more ✨
So basically, you get a sliding window of 64 active SIDX values 🪟 Values inside the window are "active", and everything else is marked "inactive". An SIDX value becomes active once at least one sample description identified by that SIDX has been received. Since sample descriptions MAY be sent redundantly, a client might receive a given SIDX several times, and that's normal 💖 Still, active sample descriptions SHALL NOT be overwritten. The receiver SHALL ignore redundant sample descriptions and it MUST use the already cached copy 🫶 The "guard interval" of (64) inactive values is what makes sure the correct association between SIDX and sample description is always used. Iykyk 😌
Informative note: As for the "guard interval" value itself, 64 as 128/2 was considered simple enough while still meeting the expected maximum number of sample descriptions. Besides that, there's no other motivation for choosing 64 or a different value. 🌸
Here's the algorithm for buffering dynamic sample descriptions and maintaining the dynamic SIDX values 🧠 Quick setup first: let X be the last SIDX received that updated the range of active sample descriptions. Let Y be a value within the allowed range for dynamic SIDX, [0,127], and different from X. Let Z be the SIDX of the last received sample description. Then:
- Initialize all dynamic SIDX values as inactive. For stored contents, read the sample description index in the Sample to Chunk box (
"stsc") for that sample. For live streaming, the first value MAY be zero or any other value in the interval above. Go to step 2. 🍓 - First, an in-band sample description with SIDX=Z is received and stored; set X=Z. Go to step 3. ✨
- Any SIDX within the interval
[X+1 modulo(128), X+64 modulo(128)]is marked as inactive, and any corresponding sample description is deleted. Any SIDX within the interval[X+65 modulo(128), X]is set active. Go to step 4 (wait state). 🪟 - Wait for the next sample description. Once the client is initialized, the interval of active SIDX values MUST change whenever a sample description with an SIDX value in the inactive set is received. That is, upon reception of a sample description with SIDX=Z, do the following: 🦋 a. If Z is in the (closed) interval
[X+1 modulo(128), X+64 modulo(128)], then set X=Z, store the sample description, and go to step 3. b. Else, Z must be in the interval[X+65 modulo(128), X], thus: i. If SIDX=Z is not stored, then store the sample description. Go to the beginning of step 4 (wait state). ii. Else, go to the beginning of step 4 (wait state).
Informative note: It is allowed that any value of SIDX=X be sent in the interval[0,127]. For example, if[64..127]is the current active set and SIDX=0 is sent, a new sample description is defined (0) and an old one deleted (64), so[65..127]and[0]are active. Similarly, one could now send SIDX=64, thus inverting the active and inactive sets. 🙃
Example: If X=4, any SIDX in the interval [5,68] is inactive. Active SIDX values are in the complementary interval [69,127] plus [0,4]. For example, if the client receives a SIDX=6, then the active interval is now different: [0,6] plus [71,127]. If the received SIDX is in the current active interval, no change SHALL be applied. Slay 💅🎀
4.3. Finding Payload Header Values in 3GP Files ✨
Okay so, when we stream timed text, some values sitting inside the boxes of a 3GP file get mapped onto fields of this payload header 🎀. This section is basically the treasure map showing where to find them 🗺️
Also, for the duration and sample description indexes, there are extension mechanisms. All senders MUST implement the extension mechanisms described herein. No skipping, bestie 💅
If the file is streamed out of a 3GP file, the following guidelines SHALL be followed. 🫶
Note: All fields in the objects (boxes) of a 3GP file are found in network byte order. 🙈
Information obtained from the Sample Table Box (stbl):
- Sample Descriptions and Sample Description length: The Sample Description box (stsd, inside the stbl) holds the sample descriptions. For timed text media, each element of stsd is a timed text sample entry (type "tx3g") 💖
The (unsigned) 32 bits of the "size" field in the stsd box represent the length (in bytes) of the sample description, as carried in TYPE 5 units. But the LEN field of TYPE 5 units is restricted to 16 bits 😭 So if the value of "size" is greater than (2^16-1-3)[bytes], then the sample description SHALL NOT be streamed with this payload format. There's no extension mechanism defined for this case, since fragmentation of sample descriptions is not defined (sample descriptions are typically up to some 200 bytes in size). Note: The three (3) accounts for the TYPE 5 header fields included in the LEN value. 🧚
- SDUR from the Decoding Time to Sample Box (stts): The (unsigned) 32 bits of the "sample delta" field are used for calculating SDUR. However, since the SDUR field is only 3 bytes long, text samples with duration values larger than (2^24-1)/(timestamp clockrate)[seconds] cannot be streamed directly 🥺 The solution is simple: copies of the corresponding text sample SHALL be sent. The timestamp and duration values SHALL be adjusted so that a continuous display is guaranteed, as if just one sample had been sent. So a sample with timestamp TS and duration SDUR can be sent as two samples having timestamps TS1 and TS2 and durations SDUR1 and SDUR2, such that TS1=TS, TS2=TS1+SDUR1, and SDUR=SDUR1+SDUR2 ✨
- Text sample length from the Sample Size Box (stsz): The (unsigned) 32 bits of the "sample size" or "entry size" (one of them, depending on whether the sample size is fixed or variable) indicate the length (in bytes) of the 3GP text sample. To get the length of the (actual) streamed text sample, the text string byte count (2 bytes) and, for UTF-16 strings, the BOM (also 2 bytes) SHALL be deducted. This is illustrated in Figure 9 🎨
Text Sample according to 3GPP TS 26.245
TEXT SAMPLE (length=stsz)
.--------------------------------------------------.
/ \
TEXT STRING (length=TBC)
.------------------------------------.
/ \
TBC BOM MODIFIERS
+---+---+----------------------------------+-----------+
||
|| TBC BOM -> TLEN field
|| +---+---+ U bit
||
\/
Text Sample according to this Payload Format
TEXT SAMPLE (length=SLEN w/o TBC,BOM)
.--------------------------------------------.
/ \
TEXT STRING (length=TLEN)
.--------------------------------.
/ \
TEXT STRING MODIFIERS
+----------------------------------+-----------+
KEY:
TBC = Text string Byte Count
BOM = Byte Order Mark
Figure 9. Text sample compositionMoreover, since the LEN field in the TYPE 1 unit header is 16 bits long, text sample sizes larger than (2^16-1-8) [bytes] SHALL NOT be streamed. Also, in this case, no extension mechanism is defined. This is because this maximum is considered enough for the targeted streaming applications. (Note: The eight (8) accounts for the TYPE 1 header fields included in the LEN value.) 🌸
- SIDX from the Sample to Chunk Box (stsc): The stsc Box is used to find samples and their corresponding sample descriptions. These are referenced by the "sample description index", a 32-bit (unsigned) integer. If possible, these indices may be directly mapped to the SIDX field. However, there are several cases where this may not be possible:
- a) The total number of indices used is greater than the number of indices available, i.e., if the static sample descriptions are more than 127 or the dynamic ones are more than 64.
- b) The original SIDX value ranges do not fit in the allowed ranges for static (129-254) or dynamic (0-127) values.
So, when assigning SIDX values to the sample descriptions, the following guidelines are provided:
- Static sample descriptions can simply be assigned consecutive values within the range 129-254 (closed interval). This range should be well enough for static sample descriptions. 🍓
- As for dynamic sample descriptions:
- a) Streams that use less than 64 dynamic sample descriptions SHOULD use consecutive values for SIDX anywhere in the range 0-127 (closed interval).
- b) For streams with more than 64 sample descriptions, the SIDX values MUST be assigned in usage order. If any sample description shall be used after it has been set inactive, it will need to be re-sent and assigned a new SIDX value (according to the algorithm in Section 4.2.1). 👑
Information obtained from the Media Data Box:
- Text strings, TLEN, U bit, and modifiers from the Media Data Box (mdat): Text strings, the 16-bit text string byte count, the Byte Order Mark (BOM, indicating UTF encoding), and modifier boxes can be found here 🦋
For TYPE 1 units, the value of TLEN is extracted from the text string byte count that precedes the text string in the text sample, as stored in the 3GP file. If UTF-16 encoding is used, two (2) more bytes have to be deducted from this byte count beforehand, in order to exclude the BOM. See Figure 9. 💜
4.4. Fragmentation of Timed Text Samples 🧩
Okay, so this section explains why text samples might have to be fragmented and talks through some of the possible ways to do it. A solution is proposed, along with rules and recommendations for fragmenting and transporting text samples. It's giving very organized 💅
3GPP Timed Text applications are expected to operate at low bitrates. Add the small size of timed text samples (typically one or two hundred bytes), and fragmentation of text samples becomes a rare event 🌈 Samples should usually fit into the MTU size of the used network path.
Nevertheless, some text strings (e.g., an ending roll in a movie) and some modifier boxes (i.e., for hyperlinks, for karaoke, or for styles) may become large 😭 This may also apply to future modifier boxes. In such cases, the first option to consider is whether it's possible to adjust the encoding (e.g., the size of the sample) so that fragmentation is avoided. If it is, that's preferred to fragmentation and SHOULD be done ✨
Otherwise, if this isn't possible or other constraints prevent it, fragmentation MAY be used, and the basic guidelines given in this document MUST be followed:
- It is RECOMMENDED that text samples be fragmented as seldom as possible, i.e., the least possible number of fragments is created out of a text sample. 🍰
- If there is some bitrate and free space in the payload available, sample descriptions (if at hand) SHOULD be aggregated.
- Text strings MUST split at character boundaries; see TYPE 2 header. Otherwise, it's not possible to display the text contents of a fragment if a previous fragment was lost. As a consequence, text string fragmentation requires knowledge of the UTF-8/UTF-16 encoding formats to determine character boundaries 🫣
- Unlike text strings, the modifier boxes are NOT REQUIRED to be split at meaningful boundaries. However, it is RECOMMENDED that this be done whenever possible, because it decreases the effects of packet loss. This payload format does not ensure that partially received modifiers are applied to text strings. If only part of the modifiers is received, it's an application issue how to deal with them, i.e., whether or not to use them 🤷♀️
Informative note: Ensuring that partially received modifiers can be applied to text strings in all cases (for all modifier types and for all fragment loss constellations) would place additional requirements on the payload format. In particular, this would require that: a) senders understand the semantics of the modifier boxes and b) specific fragment headers for each of the modifier boxes are defined, in addition to the payload formats defined below. Understanding the modifier semantics means knowing, e.g., where each modifier starts and ends, which text fragments are affected, which modifiers may or may not be split, or what the fields indicate. This is necessary to be able to split the modifiers in such a way that each fragment can be applied independently of previous packet losses. This would require a more intelligent fragmentation entity and more complex headers. Given the low probability of fragmentation and the desire to keep the requirements low, it does not seem reasonable to specify such modifier box specific headers. 🧠
- Modifier and text string fragments SHOULD be protected against packet losses, i.e., using FEC [7], retransmission [11], repetition (Section 5), or an equivalent technique. This minimizes the effects of packet loss 🛡️
- An additional requirement when fragmenting text samples is that the start of the modifiers MUST be indicated using the payload header defined for that purpose, i.e., a TYPE 3 unit MUST be used (see Section 4.1.4). This enables a receiver to detect the start of the modifiers as long as there are not two or more consecutive packet losses 💫
- Finally, sample descriptions SHALL NOT be fragmented, because they contain important information that may affect several text samples 🎀
4.5. Reassembling Text Samples at the Receiver 🧩
Okay bestie, the payload headers in this doc let you stitch fragmented text samples back together ✨ To do that, you use the standard RTP timestamp, the duration field (SDUR), and the TOTAL/THIS fields in the payload headers 💅
Units that belong to the same text sample MUST have the same timestamp. TYPE 5 units don't follow this rule because they aren't part of any particular text sample 🙈
Here's how the collecting process goes 🎀
- Search for units with the same timestamp value, i.e., units that belong to the same text sample or sample descriptions that shall become available at that time instant. If several units of the same sample are repeated, only one of them SHALL be used. Repeated units are the ones with the same timestamp and the same values for TOTAL/THIS.
Note that, as mentioned in Section 4.1.1, the receiver SHALL ignore units with unrecognized TYPE value. However, the RTP header fields and the rest of the units (if any) in the payload are still useful 🦋
- Check within this set whether any of the units from the text sample is missing. This is done using the TOTAL and THIS fields. The TOTAL field indicates how many fragments were created out of the text sample, and the THIS field indicates the position of this fragment in the text sample. As a result of this operation, two outcomes are possible:
a. No fragment is missing. Then, the THIS field SHALL be used to order the fragments and reassemble the text sample before forwarding it to the decoding application. Special care SHALL be taken when reassembling the text string as indicated in bullet 4 below ✨
b. One or more fragments are missing. Check whether this fragment belongs to the text string or to the modifiers. TYPE 2 units identify text string fragments, and TYPE 3 and 4 identify modifier fragments 🌸
i. If the fragment or fragments missing belong to the text string and the modifiers were received complete, then the received text characters may, at least, be displayed as plain text. Some modifiers may only be applied as long as it is possible to identify the character numbers, e.g., if only the last text string fragment is lost. This is the case for modifiers defining specific font styles (styl), highlighted characters (hlit), karaoke feature (krok), and blinking characters (blnk). Other modifiers such as dlay or tbox can be applied without the knowledge of the character number. It is an application issue to decide whether or not to apply the modifiers 💜
ii. If the fragment missing belongs to the modifiers and the text strings were received complete, then the incomplete modifiers may be used. The text string SHOULD at least be displayed as plain text. As mentioned in Section 4.4, modifiers may split without observing meaningful boundaries. Hence, it may not always be possible to make use of partially received modifiers. However, to avoid this, it is RECOMMENDED that the modifiers do split at meaningful boundaries 🍓
iii. A third possibility is that it is not possible to discern whether modifiers or text strings were received complete. For example, if the TYPE 3 unit of a sample plus the following or preceding packet is lost, there is no way for the RTP receiver to know if one or both packets lost belong to the modifiers or if there are also some missing text strings. Repetition, FEC, retransmission, or other protection mechanisms as per section 4.6 are RECOMMENDED to avoid this situation 🛟
iv. Finally, if it is sure that neither text strings nor modifiers were received complete, then the text strings and the modifiers may be rendered partially or may be discarded. This is an application choice 🤷♀️
- Sample descriptions can be directly associated with the reassembled text samples, via the sample description index (SIDX) 🧚
- Reassembling of text strings: since the text strings transported in RTP packets MUST NOT include any byte order mark (BOM), the receiver MUST prepend it to the reassembled UTF-16 string before handing it to the timed text decoder (see Figure 9). The value of the BOM is
0xFEFF, because only big endian serialization of UTF-16 strings is supported by this payload format 👑
4.6. On Aggregate Payloads 📦
Units SHOULD be aggregated to avoid overhead, whenever possible. The aggregate payloads MUST comply with one of the following ordered configurations 💖
- Zero or more sample descriptions (TYPE 5) followed by zero or more whole text samples (TYPE 1 units). At least one unit of either type MUST be present.
- Zero or more sample descriptions followed by zero or one modifier fragment, either TYPE 3 or TYPE 4. At least one unit MUST be present.
- Zero or more sample descriptions, followed by zero or one text string fragment (TYPE 2), followed by zero or one TYPE 3 unit. If a TYPE 2 unit and a TYPE 3 unit are present, then they MUST belong to the same text sample. At least one unit MUST be present.
Some observations 👀
- Different aggregates than the ones listed above SHALL NOT be used.
- Sample descriptions MUST be placed in the aggregate payload before the occurrence of any non-TYPE 5 units.
- Correct reception of TYPE 5 units is important since their contents may be referenced by several other units in the stream.
Receivers are unable to use text samples until their corresponding sample descriptions are received. Accordingly, a sender SHOULD send multiple copies of a sample description to ensure reliability (see Section 5). Receivers MAY use payload-specific feedback messages [21] to tell a sender that they have received a particular sample description 📬
- Regarding timestamp calculation: in general, the rules for calculating the timestamp of units in an aggregate payload depend on the type of unit. Based on the possible constellations for aggregate payloads, as above, we have:
- Sample descriptions MUST receive the RTP timestamp of the packet in which they are included.
Note that for TYPE 5 units, the timestamp actually does not represent the instant when they are played out, but instead the instant at which they become available for use.
- For the first configuration: the first TYPE 1 unit receives the RTP timestamp. The timestamp of any subsequent TYPE 1 unit MUST be obtained by adding sample duration and timestamp, both of the preceding TYPE 1 unit.
- For the second and third configuration, all units, TYPE 2, 3, and 4, MUST receive the RTP timestamp.
Refer to detailed examples on the timestamp calculation below ⏱️
- As per configuration 3 above, a payload MAY contain several fragments of one (and only one) text sample. If it does, then exactly one TYPE 2 unit followed by exactly one TYPE 3 unit is allowed in the same payload. This is in line with RFC 3640 [12], Section 2.4, which explicitly disallows combining fragments of different samples in the same RTP payload. Note that, in this special case, no timestamp calculation is needed. That is, the RTP timestamp of both units is equal to the timestamp in the packet's RTP header 🌈
- Finally, note that the use of empty text samples allows for aggregating non-consecutive TYPE 1 units in the same payload. Two text samples, with timestamps TS1 and TS3 and durations SDUR1 and SDUR3, are not consecutive if it holds
TS1+SDUR1 < TS3. A solution for this is to include an empty TYPE 1 unit with duration SDUR2 between them, such thatTS2+SDUR2 = TS1+SDUR1+SDUR2 = TS3🧁
Some examples of aggregate payloads are illustrated in Figure 10. (Note: The figure is not scaled.) 🎨
N/A TS1 TS2 TS3
+------+-----+------+-----+
|TYPE5 |TYPE1|TYPE1 |TYPE1|
+------+-----+------+-----+
N/A sdur1 sdur2 sdur3
N/A TS4
+-----+-------+
|TYPE5| TYPE 1| a)
+-----+-------+
N/A sdur4
TS4 TS4 TS4
+--------------+ +--------------+
| TYPE2 | |TYPE2 |TYPE 3 | b)
+--------------+ +--------------+
sdur4 sdur4 sdur4
TS4 TS4
+--------------+ +--------------+
| TYPE2| TYPE 3| | TYPE4 | c)
+--------------+ +--------------+
sdur4 sdur4 sdur4
|----------PAYLOAD 1------| |--PAYLOAD 2---| |--PAYLOAD 3---|
rtpts1 rtpts2 rtpts3
KEY:
TSx = Text Sample x
rtptsy = the standard RTP timestamp for PAYLOAD y
sdurx = the duration of Text Sample x
N/A = not applicable
Figure 10. Example aggregate payloadsIn Figure 10, four text samples (TS1 through TS4) are sent using three RTP packets. These configurations have been chosen to show how the 5 TYPE headers are used. Additionally, three different possibilities for the last text sample, TS4, are depicted: a), b), and c) 🦋
In Figure 11, option b) from Figure 10 is chosen to illustrate how the timestamp for each unit is found 🔍
N/A TS1 TS2 TS3 TS4 TS4 TS4
+------+-----+------+-----+ +--------------+ +--------------+
|TYPE5 |TYPE1|TYPE1 |TYPE1| | TYPE2 | |TYPE2 |TYPE 3 |
+------+-----+------+-----+ +--------------+ +--------------+
N/A sdur1 sdur2 sdur3 sdur4 sdur4 sdur4
(#1) (#2) (#3) (#4) (#5) (#6) (#7)
|----------PAYLOAD 1------| |--PAYLOAD 2---| |--PAYLOAD 3---|
rtpts1 rtpts2 rtpts3
Figure 11. Selected payloads from Figure 10Assuming TSx means Text Sample x, rtptsy represents the standard RTP timestamp for PAYLOAD y, and sdurx is the duration of Text Sample x, the timestamp for unit #z, ts(#z), can be found as the sum of rtptsy and the cumulative sum of the durations of preceding units in that payload (except in the case of PAYLOAD 3 as per rule 3 above). Thus, we have:
- okay so for the units in the first aggregate payload, PAYLOAD 1:
ts(#1) = rtpts1
ts(#2) = rtpts1
ts(#3) = rtpts1 + sdur1
ts(#4) = rtpts1 + sdur1 + sdur2Note that the TYPE 5 and the first TYPE 1 unit have both the RTP timestamp. 🎀
- next up, PAYLOAD 2:
ts(#5) = rtpts2- and for PAYLOAD 3:
ts(#6) = ts(#7) = rtpsts2 = rtpts3bestie, according to configuration 3 above, the TYPE2 and the TYPE 3 units shall belong to the same sample 🧚 So rtpts3 must be equal to rtpts2. For that same reason, the value of SDUR is not be used to calculate the timestamp of the next unit. 💅
4.7. Payload Examples ✨
Okay, time for the fun part! 🦋 Here are some payloads that use the headers we just defined, so you can actually see what they look like on the wire.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE1| LEN (always >=8) | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | TLEN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TLEN | |
+---------------+ |
| text string (no.bytes=TLEN) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| modifiers (no.bytes=LEN - 8 - TLEN) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE1| LEN (always >=8) | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | TLEN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TLEN | |
+-+-+-+-+-+-+-+-+ |
| text string fragment (no.bytes=TLEN) |
| |
| |
| +-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+I'm obsessed with Figure 12 🙈 It's an RTP packet carrying two TYPE 1 units. The LEN and TLEN fields are the MVPs here: LEN finds the start of the next unit, TLEN finds the start of the modifiers, and LEN-TLEN gives you the length of the modifiers. Iconic. ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE5| LEN( always >3) | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| sample description (no.bytes=LEN - 3) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE1| LEN (always >=8) | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | TLEN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TLEN | |
+-+-+-+-+-+-+-+-+ |
| text string fragment (no.bytes=TLEN) |
| |
| |
| +-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 13 is giving "aggregate everything" 🌸 It shows a sample description and a TYPE 1 unit packed together. The TYPE 1 unit only has text strings and is pretty small, so a TYPE 5 unit tags along to make use of the leftover bits in the packet. Efficiency, bestie! 💖
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE2| LEN( always >9) |TOTAL=4|THIS=1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SLEN | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| text string fragment (no.bytes=LEN - 9) |
| |
: :
: :
| +-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 14. Payload with first text string fragment of a sample
Figures 14, 15, and 16 are one storyline 🎬 A text sample gets split across three RTP packets. In Figure 14 the text string is so big it takes up the whole packet length. In Figure 15, the only possibility for carrying two fragments of the same text sample is shown (see configuration 3 in Section 4.6). And the last packet, Figure 16, carries the last modifier fragment, which is a TYPE 4. Plot twist: it's a sample split three ways! 😭
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE2| LEN( always >9) |TOTAL=4|THIS=2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | SIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SLEN | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| text string fragment (no.bytes=LEN - 9) |
| |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE3| LEN( always >6) |TOTAL=4|THIS=3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| modifiers (no.bytes=LEN - 6) |
| +-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 15. An RTP packet carrying a TYPE 2 unit and a TYPE 3 unit
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|U| R |TYPE4| LEN( always >6) |TOTAL=4|THIS=4 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SDUR | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| modifiers (no.bytes=LEN - 6) |
| +-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 16. An RTP packet carrying last modifiers fragment (TYPE 4)
4.8. Relation to RFC 3640 🎀
Okay so RFC 3640 [12] is the payload format for carrying any non-multiplexed MPEG-4 elementary stream 💖. One of those stream types is MPEG-4 timed text, specified in MPEG-4 part 17 [26], also known as ISO/IEC 14496-17. MPEG-4 timed text streams can carry 3GPP timed text data, as specified in 3GPP TS 26.245 [1].
MPEG-4 timed text streams were deliberately built to guarantee interoperability between RFC 3640 and this payload format ✨. So the way you construct the RTP packets carrying timed text is the same in both. The MPEG-4 timed text elementary stream from ISO/IEC 14496-17 is identical to the (aggregate) payloads built with this payload format. It's giving perfectly matched 🫶
Figure 17 shows how an RTP packet containing timed text gets built. In the partition block you can see that the (transport) units used here are identical to the Timed Text Units (TTUs) defined in ISO/IEC 14496-17 🧚. The payload aggregation rules in Section 4.6 are also identical to those in ISO/IEC 14496-17, and they're compliant with RFC 3640. Which means an RTP packet using this payload format is identical to an RTP packet using RFC 3640 that carries TTUs according to ISO/IEC 14496-17 👑. Also, MPEG-4 Part 17 says that when RFC 3640 is used to transport timed text streams, the "streamType" parameter is set to 0x0D, and the "objectTypeIndication" value in "config" is 0x08 💅
+--------------------------------------+
Text samples | +--------------+ +--------------+ |
as per 3GPP | |Text Sample 1 | |Text Sample N | |
TS 26245 | +--------------+ +--------------+ |
+--------------------------------------+
\/
+-------------------------------------------------------------------+
| Partition Text Samples into units. TTU[i]= TYPE i units. |
| |
|[U R TYPE LEN][{TOTAL,THIS}SIDX{SDUR}{TLEN}{SLEN}][SampleContents] |
|{..} means present if applicable, [..] means always present |
+-------------------------------------------------------------------+
\/ \/
+-------------------------------------------------------------------+
| Aggregation (if possible) |
+-------------------------------------------------------------------+
\/ \/
+-------------------------------------------------------------------+
| RTP Entity adds and fills RTP header and Sends RTP packet, where |
| RTP packets according to this Payload Format = |
| RTP packets carrying MPEG-4 Timed Text ES over RFC 3640 |
+-------------------------------------------------------------------+Figure 17. Relation to RFC 3640
Note: The use of RFC 3640 for transport of ISO/IEC 14496-17 data does not require any new SDP parameters or any new mode definition. 🌸
4.9. Relation to RFC 2793 🦋
RFC 2793 [22] and its revision, RFC 4103 [23], specify a protocol for enabling text conversation 💬. The typical apps for this payload format are text communication terminals and text conferencing tools. Text session contents are specified in ITU-T Recommendation T.140 [24]. T.140 text is UTF-8 coded as specified in T.140 [24] with no extra framing. The T140block holds one or more T.140 code elements as specified in T.140. Code elements are control sequences like "New Line", "Interrupt", "String Terminator", or "Start of String". Most T.140 code elements are single ISO 10646 [25] characters, but some are multiple character sequences. Each character is UTF-8 encoded [18] into one or more octets.
This payload format may also be used for conversational applications (even for instant messaging) 📱. However, that isn't its main target. The thing that makes 3GPP Timed Text special is that it allows text decoration ✨. That's super useful for multimedia presentations, karaoke, commercial banners, news tickers, clickable text strings, and captions 🍓. T.140 text used in RFC 2793 doesn't allow text decoration, so it's a whole different vibe.
The conversational text RTP payload format also recommends a method to include redundant text from already transmitted packets, which lowers the risk of text loss caused by packet loss 🛡️. Each payload would include a redundant copy of the last payload sent. This payload format doesn't describe such a method, but it's also applicable here. As explained in Section 5, packet redundancy SHOULD be used, whenever possible. The aggregation guidelines in Section 4.6 allow redundant payloads 🔥.
5. Resilient Transport 🛡️
Apart from the basic fragmentation guidelines in the section above, the simplest option for packet-loss-resilient transport is packet repetition 🔁. This mechanism may consist of a strict window-based repetition mechanism or, simply, a repetition mechanism in a wider sense, where new and old packets are mixed, for example.
A server MAY decide to use repetition as a measure for packet loss resilience 💜. So a server MAY send the same RTP payloads, or just some of the units from the payloads.
As with complete payloads, single repeated units MUST exactly match the same units sent in the first transmission. So if fragmentation is needed, it SHALL be performed only once for each text sample 🎯. Only then can a receiver use the already received units and the repeated units to rebuild the original text samples. The RTP timestamp is used to group the fragments of a sample together, so care must be taken to preserve the timing of units when constructing new RTP packets ⏰.
For example, if a text sample was originally sent as a single non-fragmented text sample (one TYPE 1 unit), a repetition of that sample MUST be sent also as a single non-fragmented text sample in one unit. Likewise, if the original text sample was fragmented and spread over several RTP packets (say, a total of 3 units), then the repeated fragments SHALL also have the same byte boundaries and use the same unit headers and bytes per fragment. 🙈
With repetition, repeated units resolve to the same timestamp as their originals. Where redundant units are available, only one of them SHALL be used.
Regarding the RTP header fields:
- If the whole RTP payload is repeated, all payload-specific fields in the RTP header (the M, TS and PT fields) MUST keep their original values, except the sequence number, which MUST be incremented to comply with RTP. The TOTAL/THIS fields make it possible to re-assemble fragments with different sequence numbers 🧩.
- In packets containing single repeated units, the general rules in Section 3 for assigning values to the RTP header fields apply. Keeping the value of the RTP timestamp to preserve the timing of the units is especially relevant here ✨.
Apart from repetition, other mechanisms such as FEC [7], retransmission [11], or similar techniques could be used to cope with packet losses 🌈.
6. Congestion Control 🚦
Congestion control for RTP SHALL be implemented in accordance with RTP [3] and the applicable RTP profile, e.g., RTP/AVP [17].
When using this payload format, mainly two factors may affect congestion control:
- The use of (unit) aggregation may make the payload format more bandwidth efficient, by avoiding header overhead and thus reducing the used bitrate 📉.
- The use of resilient transport mechanisms. Timed text applications typically operate at low bitrates, but the increase due to resilient transport shall be considered for congestion control mechanisms. This applies to all mechanisms, but especially to less efficient ones like repetition 💅.
7. Scene Description 🎬
7.1. Text Rendering Position and Composition 🌟
To set up a timed text session, whether the stream is stored in a 3GP file or streamed live, the communicating peers need some initial layout information 🗺️.
+-------------------------------------------+
| <-> tx | +-------------+
| +-------------------------------+ |<---|Display Area |
| ^ | | | +-------------+
| : | | |
| :ty| | | +-------------+
| : | |<---------|Video track |
| : | | | +-------------+
| : | | |
| : | | |
| : | | |
| v | | |
| - | x-------------------------+ | | +-------------+
|h ^ | | |<-----------|Text Track |
|e : +---|-------------------------|-+ | +-------------+
|i : | +---------------------+ | |
|g : | | | | | +-------------+
|h : | | |<------------ |Text Box |
|t v | +---------------------+ | | +-------------+
| - +-------------------------+ |
+-------------------------------------------+
<........................>
w i d t hFigure 18. Illustration of text rendering position and composition
Figure 18 shows the parameters used to negotiate the position and size of the text track in the display area 📐. These are the "width" and "height" of the text track, its translation values "tx" and "ty", and its "layer", meaning how close it sits to the user 👀.
At the same time, the sender needs to know the receiver's capabilities 🤝. That means the maximum allowable values for the text track height and width, "max-h" and "max-w", for the stream the receiver shall display.
This layout information MUST be conveyed in a reliable form before the start of the session, e.g., during session announcement or in an Offer/Answer (O/A) exchange 📣. An example of a reliable transport may be the out-of-band channel used for SDP. Sections 8 and 9 provide details on mapping these parameters to SDP descriptions and on their usage in O/A 🧾.
For stored content, the layout values expressing stream properties MUST be obtained from the Track Header Box. See Section 7.3 💖.
For live streaming, appropriate values negotiated during session setup shall be used ✨.
7.2. SMIL Usage ✨
okay so the attributes in the Track Header Boxes of a 3GP file only tell you where each track sits inside that one file 📦 that's it, that's the whole job
but if you send multiple 3GP files, they need spatial synchronization too 🧚 like, for a text and video stream, you have to figure out where the text track and the video track go in Figure 18. for that, SMIL [9] MAY be used 💅
SMIL hands each file its own region on the display and then places the tracks inside those regions. the big difference is that in SMIL, one track's position is usually written relative to another track. in a 3GP file, though, the upper left corner is the reference point for every translation offset 🎀 so the numbers only match up if the SMIL position is measured from the video track's origin. then the translation offset has the same value as (tx, ty) in the 3GP file 🫶
also, the original track header info only applies to each track within its region, the one SMIL assigns. so even if you're using a SMIL scene description, the track header information pieces SHOULD be sent anyway, because they describe the intrinsic media properties 🌸 for the full details, see 3GPP SMIL Language Profile in [27].
7.3. Finding Layout Values in a 3GP File 🦋
inside the Track Header Box (tkhd) of a 3GP file, here's where to look:
tx,ty: these give the translation offset of the (text) track relative to the upper left corner of the video track, if there is one. they're the second-to-last and third-to-last values in the unity matrix. the values are fixed-point 16.16 values, restricted to be (signed) integers (i.e., the lower 16 bits of each value shall be all zeros). so only the first 16 bits end up being used to get the value of the media type parameters 🔥width,height: these have the same name in the tkhd box. all (unsigned) 32 bits are meaningful ✨layer: all (signed) 16 bits are used 💜
8. 3GPP Timed Text Media Type 🍓
the media subtype for the 3GPP Timed Text codec comes from the standards tree 👑 the top-level media type this payload format gets registered under is video. the registration uses the template defined in [29], and it follows RFC 3555 [28].
the receiver MUST ignore any unrecognized parameter 🙈 no drama, just skip it.
here's the registration template, verbatim:
Media type: video
Media subtype: 3gpp-tt
Required parameters
rate:
Refer to Section 3 in RFC 4396.
sver:
The parameter "sver" contains a list of supported
backwards-compatible versions of the timed text format
specification (3GPP TS 26.245) that the sender accepts
to receive (and that are the same that it would be
willing to send). The first value is the value
preferred to receive (or preferred to send). The first
value MAY be followed by a comma-separated list of
versions that SHOULD be used as alternatives. The order
is meaningful, being first the most preferred and last
the least preferred. Each entry has the format
Zi(xi*256+yi), where "Zi" is the number of the Release
and "xi" and "yi" are taken from the 3GPP specification
version (i.e., vZi.xi.yi). For example, for 3GPP TS
26.245 v6.0.0, Zi(xi*256+yi)=6(0), the version value is
"60". (Note that "60" is the concatenation of the
values Zi=6 and (xi*256+yi)=0 and not their product.)
If no "sver" value is available, for example, when
streaming out of a 3GP file, the default value "60",
corresponding to the 3GPP Release 6 version of 3GPP TS
26.245, SHALL be used.okay so about sver 💅 it's a list of the backwards-compatible versions of the timed text format spec (3GPP TS 26.245) that the sender accepts to receive, and the ones it'd be willing to send too. the first value is the one it prefers. that first value MAY be followed by a comma-separated list of versions that SHOULD be used as alternatives. order matters: first is most preferred, last is least preferred 🎀
each entry looks like Zi(xi*256+yi), where Zi is the Release number, and xi and yi come from the 3GPP version, vZi.xi.yi. so for 3GPP TS 26.245 v6.0.0, that's Zi(xi*256+yi)=6(0), and the version value is 60. (not the product, babe, it's the concatenation of Zi=6 and (xi*256+yi)=0 🙃)
and if there's no sver value available, like when you're streaming out of a 3GP file, the default 60 SHALL be used. that's the 3GPP Release 6 version of TS 26.245 ✨
now the optional ones:
Optional parameters:
tx:
This parameter indicates the horizontal translation
offset in pixels of the text track with respect to the
origin of the video track. This value is the decimal
representation of a 16-bit signed integer. Refer to TS
3GPP 26.245 for an illustration of this parameter.
ty:
This parameter indicates the vertical translation offset
in pixels of the text track with respect to the origin
of the video track. This value is the decimal
representation of a 16-bit signed integer. Refer to TS
3GPP 26.245 for an illustration of this parameter.
layer:
This parameter indicates the proximity of the text track
to the viewer. More negative values mean closer to the
viewer. This parameter has no units. This value is the
decimal representation of a 16-bit signed integer.
tx3g:
This parameter MUST be used for conveying sample
descriptions out-of-band. It contains a comma-separated
list of base64-encoded entries. The entries of this
list MAY follow any particular order and the list SHALL
NOT be empty. Each entry is the result of running
base64 encoding over the concatenation of the (static)
SIDX value as an 8-bit unsigned integer and the (static)
sample description for that SIDX, in that order. The
format of a sample description entry can be found in
3GPP TS 26.245 Release 6 and later releases. All
servers and clients MUST understand this parameter and
MUST be capable of using the sample description(s)
contained in it. Please refer to RFC 3548 [6] for
details on the base64 encoding.
width:
This parameter indicates the width in pixels of the text
track or area of the text being sent. This value is the
decimal representation of a 32-bit unsigned integer.
Refer to TS 3GPP 26.245 for an illustration of this
parameter.
height:
This parameter indicates the height in pixels of the
text track being sent. This value is the decimal
representation of a 32-bit unsigned integer. Refer to
TS 3GPP 26.245 for an illustration of this parameter.
max-w:
This parameter indicates display capabilities. This is
the maximum "width" value that the sender of this
parameter supports. This value is the decimal
representation of a 32-bit unsigned integer.
max-h:
This parameter indicates display capabilities. This is
the maximum "height" value that the sender of this
parameter supports. This value is the decimal
representation of a 32-bit unsigned integer.tx3g is the big one 🌈 it MUST be used for sending sample descriptions out-of-band. it's a comma-separated list of base64-encoded entries. the entries MAY come in any order, and the list SHALL NOT be empty. each entry is base64 encoding of the (static) SIDX value as an 8-bit unsigned integer, concatenated with the (static) sample description for that SIDX, in that order. the sample description format lives in 3GPP TS 26.245 Release 6 and later 🧚 and every server and client MUST understand this parameter and MUST be capable of using the sample descriptions in it. for base64 details, see RFC 3548 [6].
the rest are pretty chill 🦋 tx and ty are pixel offsets of the text track from the video track's origin, both 16-bit signed integers. layer says how close the text track sits to the viewer, where more negative means closer. it has no units, and it's a 16-bit signed integer too. width and height are the pixel size of the text track (or the area of text being sent), as 32-bit unsigned integers. max-w and max-h tell you the display capabilities: the biggest width and height the sender can handle, also 32-bit unsigned integers 💖
Encoding considerations:
This media type is framed (see Section 4.8 in [29]) and
partially contains binary data.
Restrictions on usage:
This media type depends on RTP framing, and hence is only
defined for transfer via RTP [3]. Transport within other
framing protocols is not defined at this time.
Security considerations:
Please refer to Section 11 of RFC 4396.
Interoperability considerations:
The 3GPP Timed Text media format and its file storage is
specified in Release 6 of 3GPP TS 26.245, "Transparent end-to-
end packet switched streaming service (PSS); Timed Text Format
(Release 6)". Note also that 3GPP may in future releases
specify extensions or updates to the timed text media format in
a backwards-compatible way, e.g., new modifier boxes or
extensions to the sample descriptions. The payload format
defined in RFC 4396 allows for such extensions. For future 3GPP
Releases of the Timed Text Format, the parameter "sver" is used
to identify the exact specification used.
The defined storage format for 3GPP Timed Text format is the
3GPP File Format (3GP) [30]. 3GP files may be transferred using
the media type video/3gpp as registered by RFC 3839 [31]. The
3GPP File Format is a container file that may contain, e.g.,
audio and video that may be synchronized with the 3GPP Timed
Text.
Published specification: RFC 4396
Applications which use this media type:
Multimedia streaming applications.
Additional information:
The 3GPP Timed Text media format is specified in 3GPP TS 26.245,
"Transparent end-to-end packet switched streaming service (PSS);
Timed Text Format (Release 6)". This document and future
extensions to the 3GPP Timed Text format are publicly available
at http://www.3gpp.org.
Magic number(s): None.
File extension(s): None.
Macintosh File Type Code(s): None.
Person & email address to contact for further information:
Jose Rey, jose.rey@eu.panasonic.com
Yoshinori Matsui, matsui.yoshinori@jp.panasonic.com
Audio/Video Transport Working Group.
Intended usage: COMMON
Authors:
Jose Rey
Yoshinori Matsui
Change controller: IETF Audio/Video Transport Working Group delegated
from the IESG.9. SDP Usage 🎀
9.1. Mapping to SDP 🦋
the info in the media type spec maps onto specific SDP [4] fields. if you use SDP to set up sessions with this payload format, here's how it goes:
- the media type (
video) goes in the SDPm=line as the media name:
m=video <port number> RTP/<RTP profile> <dynamic payload type>- the media subtype (
3gpp-tt) and the timestamp clockraterate(the RECOMMENDED 1000 Hz or another value) go in the SDPa=rtpmapline as the encoding name and rate, respectively:
a=rtpmap:<payload type> 3gpp-tt/1000- the REQUIRED parameter
svergoes in the SDPa=fmtpattribute, copied directly from the media type string as a semicolon-separated parameter=value pair 💅 - the OPTIONAL parameters
tx,ty,layer,tx3g,width,height,max-wandmax-halso go in the SDPa=fmtpattribute, copied directly from the media type string as a semicolon-separated list of parameter=value(s) pairs:
a=fmtp:<dynamic payload type> <parameter
name>=<value>[,<value>][; <parameter name>=<value>]- any parameter the device doesn't know SHALL be ignored. for example, parameters added to the media format in later specifications MAY be copied into the SDP, and they SHALL be ignored by receivers that don't understand them 🌸
9.2. Parameter Usage in the SDP Offer/Answer Model 💜
in this section, we explain what the SDP parameters defined in this document mean inside the Offer/Answer [13] context.
in unicast, the sender and receiver usually negotiate the streams, like which codecs and parameter values get used in the session. multicast can do this too, just to a lesser extent ✨
on top of that, what the parameters mean MAY change depending on which direction is used. in the sections that follow, a <directionality> offer means an offer that contains a stream set to <directionality>. <directionality> can be sendrecv, sendonly, or recvonly. answers work the same way. for example, an answer to a sendonly offer is a recvonly answer 🫶
9.2.1. Unicast Usage ✨
okay so bestie, there are a few kinds of parameters in this payload format and they each have their own rules 💅
- Declarative parameters: the offerer and answerer each declare the values they'll use for the incoming (sendrecv/recvonly) or outgoing (sendonly) stream. The offerer and answerer MAY use different values. 🦋
- a. "tx", "ty", and "layer": these describe where the received text track gets placed. It depends on the directionality:
- i. They MUST appear in all sendrecv offers and answers and in all recvonly offers and answers, so they apply to the incoming stream. For sendrecv offers and answers and for recvonly offers, these values SHOULD be used by the sender of the stream unless it has a particular preference. If it does, it MUST make sure those different values don't corrupt the presentation 🫶. For recvonly answers, the answerer MAY accept the proposed values for the incoming stream (in a sendonly offer; see ii. below) or respond with different ones. The offerer MUST use the returned values.
- ii. They MAY appear in sendonly offers and MUST appear in sendonly answers. In sendonly offers, they specify the values the offerer proposes for sending (see the example in Section 9.3) 🌸. In sendonly answers, these values SHOULD be copied from the corresponding recvonly offer when the stream is accepted, unless the receiver of the stream has a particular preference, as explained in the previous point.
- a. "tx", "ty", and "layer": these describe where the received text track gets placed. It depends on the directionality:
- Display capabilities, "max-h" and "max-w": these show the maximum dimensions of the text track (the text display area) for the incoming stream's "tx" and "ty" values (see Figure 18) 👑. "max-h" and "max-w" MUST be included in all offers and answers where "tx" and "ty" refer to the incoming stream. That means sendonly offers and answers are excluded (see the example in Section 9.3), and in those, they SHALL NOT be present 💀.
- Sent stream properties: the sender of the stream decides these values 🍓.
- a. "width" and "height" specify the text track dimensions. They SHALL ALWAYS be present in sendrecv and sendonly offers and answers. For recvonly answers, the answerer MUST include the offered parameter values (if any) verbatim in the answer when accepting the stream 🎀.
- b. "tx3g" contains static sample descriptions. It MAY only be present in sendrecv and sendonly offers and answers. This parameter applies to the stream that the offerer or answerer sends 🧚.
- Negotiable parameters: these MUST be agreed on. "sver" is the case here. This parameter MUST be present in every offer and answer. The answerer SHALL choose one supported value from the offerer's list. If it can't, it MUST remove the stream or reject the session 😭.
- Symmetric parameters: "rate" and the timestamp clockrate belong to this class 🌈. Symmetric parameters MUST be echoed verbatim in the answer. Otherwise, the stream MUST be removed or the session rejected.
The following table summarizes all of the options 📋:
+..---------------------------+----------+----------+----------+
| ``--..__ Directionality/ | sendrecv | recvonly | sendonly |
+ Type of ``--..__ O or A +----------+----------+----------+
| Parameter ``--..__ | O/A | O/A | O/A |
+--------------+------------``+----------+----------+----------+
| Declarative |tx, ty, layer | M/M | M/M | m/M |
| | | | | |
+--------------+--------------+----------+----------+----------+
| Display |max-h, max-w | M/M | M/M | -/- |
| Capabilities | | | | |
+--------------+--------------+----------+----------+----------+
| Stream |height, width | M/M | -/(M) | M/M |
| properties |tx3g | m/m | -/- | m/m |
| | | | | |
+--------------+--------------+----------+----------+----------+
| Negotiable |sver | M/M | M/M | M/M |
| | | | | |
+--------------+--------------+----------+----------+----------+
| Symmetric |rate | M/M | M/M | M/M |
+--------------+--------------+----------+----------+----------+
Table 1. Parameter usage in Unicast Offer / Answer.KEY, so you can read the table:
- M means MUST be present.
- m means MAY be present (such as proposed values).
- (M) or (m) means MUST or MAY, if applicable.
- A hyphen ("-") means the parameter MUST NOT be present.
Other observations about parameter usage 👀:
- Translation and transparency values: in sendonly offers, "tx", "ty", and "layer" indicate proposed values. This is useful for visually composed sessions where the different streams occupy different parts of the display, e.g., a video stream and the captions 🎬. These are just suggested values. The peer rendering the text ultimately decides where to place the text track.
- Text track (area) dimensions, "height" and "width": for sendonly offers, an answerer accepting the offer MUST be prepared to render the stream using these values. If any of these conditions are not met, the stream MUST be removed or the session rejected.
- Display capabilities, "max-h" and "max-w": an answerer sending a stream SHALL ensure that the "height" and "width" values in the answer are compatible with the offerer's signaled capabilities 💖.
- Version handling via "sver": the idea is that the offerer and answerer communicate using the same version. This is achieved by letting the answerer choose from a list of supported versions, "sver" 🔥. For recvonly streams, the first value in the list is the preferred version to receive. So for sendonly (and sendrecv) streams, the first value is the one preferred for sending (and receiving). The answerer MUST choose one value and return it in the answer. When the offerer gets the answer, the offerer SHALL be prepared to send (sendonly and sendrecv) and receive (recvonly and sendrecv) a stream using that version. If none of the versions in the list is supported, the stream MUST be removed or the session rejected. Note that if alternative non-compatible versions are offered, then this SHALL be done using different payload types ⭐.
9.2.2. Multicast Usage 📡
In multicast, the parameter usage is similar to the unicast case, except for these things 🌸:
- The parameters "tx", "ty", and "layer" in multicast offers only have meaning for sendrecv and recvonly streams. So all clients have the same vision of the session, they MUST be used symmetrically 🫶.
- For "height", "width", and "tx3g" (for sendrecv and sendonly), multicast offers specify which values of these parameters the participants MUST use for sending. So if the stream is accepted, the answerer MUST also include them verbatim in the answer (also "tx3g", if present) 🎀.
- The capability parameters, "max-h" and "max-w", SHALL NOT be used in multicast. If the offered text track should change in size, a new offer SHALL be used instead 🙈.
- Regarding version handling: for multicast offers, an answerer MAY accept a multicast offer as long as one of the versions listed in "sver" is supported. So if the stream is accepted, the answerer MUST choose its preferred version. Unlike unicast, though, the offerer SHALL NOT change the offered stream to this chosen version, because other session participants may support the newer extensions 🧚. As a result, different session participants may end up using different backwards-compatible media format versions. It is RECOMMENDED that the multicast offer contains a limited number of versions, so all participants have the same view of the session. This is the responsibility of the session creator. If none of the offered versions is supported, the stream SHALL be removed or the session rejected. Also in this case, if alternative non-compatible versions are offered, then this SHALL be done using different payload types 💅.
9.3. Offer/Answer Examples ✨
okay so these unicast O/A examples are a bit messy on the page 🙈 because the long lines got wrapped around, and the static sample descriptions got shortened so they're easier to read 💖
For sendrecv:
O -> A
m=video <port> RTP/AVP 98
a=rtpmap:98 3gpp-tt/1000
a=fmtp:98 tx=100; ty=100; layer=0; height=80; width=100; max-h=120;
max-w=160; sver=6256,60; tx3g=81...
a=sendrecvA -> O
m=video <port> RTP/AVP 98..
a=rtpmap:98 3gpp-tt/1000
a=fmtp:98 tx=100; ty=95; layer=0; height=90; width=100; max-h=100;
max-w=160; sver=60; tx3g=82...
a=sendrecvin this one the offerer is basically saying "hey, this is where I'll put the stream I receive, and this is the biggest height and width I can handle" 🦋 it also tells the answerer the size of the text track for the stream it sends, plus which sample description to use. it offers two versions, 6256 and 60. the answerer comes back with an equivalent set of params for the stream it receives. here the answerer's "max-h" and "max-w" are compatible with the offerer's "height" and "width" 🎀
if they weren't, the answerer would have to remove this stream, and the offerer would have to send a new offer that takes the answerer's capabilities into account. that only works if the first offer has multiple payload types, so at least one of them matches what the answerer can do, as expressed by "max-h" and "max-w" in the negative answer ✨ also the answerer's text box dimensions fit inside the maximum values signaled in the offer. and finally the answerer picks version 60 of the timed text format 👑
For recvonly:
Offerer -> Answerer
m=video <port> RTP/AVP 98
a=rtpmap:98 3gpp-tt/1000
a=fmtp:98 tx=100; ty=100; layer=0; max-h=120; max-w=160; sver=6256,60
a=recvonlyA -> O
m=video <port> RTP/AVP 98..
a=rtpmap:98 3gpp-tt/1000
a=fmtp:98 tx=100; ty=100; layer=0; height=90; width=100; sver=60;
tx3g=82...
a=sendonlythis case is different because the offer doesn't include the stream properties "height", "width", and "tx3g" 🌸 the answerer copies the "tx", "ty", and "layer" values, which is how it acknowledges them. "max-h" and "max-w" aren't in the answer, because in this special case the "tx" and "ty" (and "layer") don't apply to the received stream. they apply to the sent stream 🧚 also, if the offerer and answerer had very different display sizes, there would be no way to express the answerer's capabilities. in the example above, with an answerer that has a 50x50 display, the translation values are already out of range 😭
For sendonly:
O -> A
m=video <port> RTP/AVP 98
a=rtpmap:98 3gpp-tt/1000
a=fmtp:98 tx=100; ty=100; layer=0; height=80; width=100;
sver=6256,60; tx3g=81...
a=sendonlyA -> O
m=video <port> RTP/AVP 98..
a=rtpmap:98 3gpp-tt/1000
a=fmtp:98 tx=100; ty=100; layer=0; height=80; width=100; max-h=100;
max-w=160; sver=60
a=recvonlynote that "max-h" and "max-w" aren't in the offer 💅 with this answer, the answerer accepts the offer as is (echoing "tx", "ty", "height", "width", and "layer") and also lets the offerer know its capabilities with "max-h" and "max-w" 🫶
another possible answer for this case would be:
A -> O
m=video <port> RTP/AVP 98..
a=rtpmap:98 3gpp-tt/1000
a=fmtp:98 tx=120; ty=105; layer=0; max-h=95; max-w=150; sver=60
a=recvonlyhere the answerer does not accept the values offered 🥺 the offerer MUST use these values or else remove the stream.
9.4. Parameter Usage outside of Offer/Answer 🎀
SDP can also show up outside of the Offer/Answer context 🌈 for example, for multimedia sessions that get announced through the Session Announcement Protocol (SAP) [14] or streamed through the Real Time Streaming Protocol (RTSP) [15].
in that case, the receiver of a session description is required to support the parameters and given values for the streams, or else it MUST reject the session 🚫 it's the responsibility of the sender (or creator) of the session descriptions to define the session parameters so the probability of unsuccessful session setup is minimized. that part is out of the scope of this document, so it's not on us, bestie 💁♀️
10. IANA Considerations 🍓
IANA has registered the media subtype name 3gpp-tt for the media type video, as specified in Section 8 of this document. ✨ official stamp of approval ✨
11. Security Considerations 🙈
okay this section is serious, so listen up 🚨
RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [3] and any applicable RTP profile, e.g., AVP [17].
in particular, an attacker can mess with the current set of active sample descriptions at the client by repeating a packet with an old sample description. that's a replay attack 💀 the display of the text would get corrupted, if it gets displayed at all. another kind of attack is sending redundant fragments whose boundaries don't match the exact boundaries of the originals (as indicated by LEN), or fragments that carry different sample lengths (SLEN). that can make a decoder crash 😱
these types of attack can easily be avoided by using source authentication and integrity protection 🔐
peers in a timed text session might also want to keep their communication private, which is confidentiality 🤫
this payload format does not provide any mechanisms for achieving these. confidentiality, integrity protection, and authentication have to be solved by a mechanism external to this payload format, e.g., SRTP [10] 💜
12. References 📚
12.1. Normative References
[1] Transparent end-to-end packet switched streaming service (PSS); Timed Text Format (Release 6), TS 26.245 v 6.0.0, June 2004.
[2] ISO/IEC 14496-12:2004 Information technology - Coding of audio-visual objects - Part 12: ISO base media file format.
[3] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.
[4] Handley, M. and V. Jacobson, "SDP: Session Description Protocol", RFC 2327, April 1998.
[5] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[6] Josefsson, S., "The Base16, Base32, and Base64 Data Encodings", RFC 3548, July 2003.
12.2. Informative References
[7] Rosenberg, J. and H. Schulzrinne, "An RTP Payload Format for Generic Forward Error Correction", RFC 2733, December 1999.
[8] Perkins, C. and O. Hodson, "Options for Repair of Streaming Media", RFC 2354, June 1998.
[9] W3C, "Synchronised Multimedia Integration Language (SMIL 2.0)", August, 2001.
[10] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004.
[11] Rey, J., Leon, D., Miyazaki, A., Varsa, V., and R. Hakenberg, "RTP Retransmission Payload Format", Work in Progress, September 2005.
[12] van der Meer, J., Mackie, D., Swaminathan, V., Singer, D., and P. Gentric, "RTP Payload Format for Transport of MPEG-4 Elementary Streams", RFC 3640, November 2003.
[13] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, June 2002.
[14] Handley, M., Perkins, C., and E. Whelan, "Session Announcement Protocol", RFC 2974, October 2000.
[15] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time Streaming Protocol (RTSP)", RFC 2326, April 1998.
[16] Transparent end-to-end packet switched streaming service (PSS); Protocols and codecs (Release 6), TS 26.234 v 6.1.0, September 2004.
[17] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video Conferences with Minimal Control", STD 65, RFC 3551, July 2003.
[18] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003.
[19] Hoffman, P. and F. Yergeau, "UTF-16, an encoding of ISO 10646", RFC 2781, February 2000.
[20] Friedman, T., Caceres, R., and A. Clark, "RTP Control Protocol Extended Reports (RTCP XR)", RFC 3611, November 2003.
[21] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, "Extended RTP Profile for RTCP-based Feedback (RTP/AVPF)", Work in Progress, August 2004.
[22] Hellstrom, G., "RTP Payload for Text Conversation", RFC 2793, May 2000.
[23] Hellstrom, G. and P. Jones, "RTP Payload for Text Conversation", RFC 4103, June 2005.
[24] ITU-T Recommendation T.140 (1998) - Text conversation protocol for multimedia application, with amendment 1, (2000).
[25] ISO/IEC 10646-1: (1993), Universal Multiple Octet Coded Character Set.
[26] ISO/IEC FCD 14496-17 Information technology - Coding of audio-visual objects - Part 17: Streaming text format, Work in progress, June 2004.
[27] Transparent end-to-end Packet-switched Streaming Service (PSS); 3GPP SMIL language profile, (Release 6), TS 26.246 v 6.0.0, June 2004.
[28] Casner, S. and P. Hoschka, "MIME Type Registration of RTP Payload Formats", RFC 3555, July 2003.
[29] Freed, N. and J. Klensin, "Media Type Specifications and Registration Procedures", BCP 13, RFC 4288, December 2005.
[30] Transparent end-to-end packet switched streaming service (PSS); 3GPP file format (3GP) (Release 6), TS 26.244 V6.3. March 2005.
[31] Castagno, R. and D. Singer, "MIME Type Registrations for 3rd Generation Partnership Project (3GPP) Multimedia files", RFC 3839, July 2004.
13. Basics of the 3GP File Structure ✨
Okay bestie, this section is a quick tour of the 3GP file structure 🎀, which follows the ISO Base Media file Format [2]. Let's go 💅
Every 3GP file is made of "Boxes" 📦. Generally, a 3GP file holds the File Type Box (ftyp), the Movie Box (moov), and the Media Data Box (mdat). The File Type Box tells you the type and properties of the 3GP file itself. The Movie Box and the Media Data Box are containers, so they hold their own boxes for each media. Every box starts with a header that gives its size and type (those fields are literally called "size" and "type" 🙈). And each box type can hold a bunch of other boxes too.
We're only going to talk about the boxes that matter for this payload format 🦋.
The Movie Box (moov) holds one or more Track Boxes (trak), and those have info about each track. A Track Box has, among others, the Track Header Box (tkhd), the Media Header Box (mdhd), and the Media Information Box (minf).
The Track Header Box describes a single track, and in this case a track is basically the streamed text during a session 💜. There's exactly one Track Header Box per track. It holds stuff like the spatial layout (width and height), the video transformation matrix, and the layer number. These bits are essential and static (i.e., constant) for the whole session, so they must be sent before any text samples go out. No cap, the order matters 😭
The Media Header Box has the "timescale", which is the number of time units that pass in one second, i.e., cycles per second or Hertz ⏱️. The Media Information Box includes the Sample Table Box (stbl), which has all the time and data indexing for the media samples in a track. With it you can locate samples in time and figure out their type, size, container, and offset into that container 🗺️. Inside the Sample Table Box you'll find:
- the Sample Description Box (
stsd, for finding sample descriptions) - the Decoding Time to Sample Box (
stts, for finding sample duration) - the Sample Size Box (
stsz) - the Sample to Chunk Box (
stsc, for finding the sample description index)
Finally, the Media Data Box holds the actual media data 🎬. In timed text tracks, this box contains text samples. Over in audio and video, the matching things are audio frames and video frames, respectively.
A text sample has the text length, the text string, and one or several Modifier Boxes. The text length is the size of the text in bytes.
The text string is plain text to render. The Modifier Box is extra info for rendering alongside the text, like color, font, etc. 🌈
14. Acknowledgements 🫶
The authors want to thank Dave Singer, Jan van der Meer, Magnus Westerlund, and Colin Perkins for their comments and suggestions about this document 💖.
The authors also want to thank Markus Gebhard for the free and publicly available JavE ASCII Editor (used for the ASCII drawings in this document), and Henrik Levkowetz for the Idnits web service 🌸.
Authors' Addresses 🦋
Here's who wrote it, bestie 💜:
Jose Rey
Panasonic R&D Center Germany GmbH
Monzastr. 4c
D-63225 Langen, Germany
EMail: jose.rey@eu.panasonic.com
Phone: +49-6103-766-134
Fax: +49-6103-766-166
Yoshinori Matsui
Matsushita Electric Industrial Co., LTD.
1006 Kadoma
Kadoma-shi, Osaka, Japan
EMail: matsui.yoshinori@jp.panasonic.com
Phone: +81 6 6900 9689
Fax: +81 6 6900 9699Full Copyright Statement ⭐
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions in BCP 78, and except as set forth there, the authors keep all of their rights.
Now the legal bit, but make it cute 💅. The disclaimer says:
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 🔐
The IETF takes no position on the validity or scope of any Intellectual Property Rights or other rights that might be claimed to apply to the implementation or use of the technology in this document, or on whether any license under those rights might or might not be available. It also doesn't claim to have made any independent effort to find those rights ✨. Info on the procedures for rights in RFC documents is in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the results of an attempt to get a general license or permission for implementers or users to use those proprietary rights, can be found in the IETF online IPR repository at http://www.ietf.org/ipr. The IETF invites anyone to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology needed to implement this standard. Send that info to the IETF at ietf-ipr@ietf.org 💌.
Acknowledgement 💖
Funding for the RFC Editor function comes from the IETF Administrative Support Activity (IASA) 🌈
📦 where this page comes from
- ietf: rfc4396 2006-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms