RFC 4696
An Implementation Guide for RTP MIDI
An Implementation Guide for RTP MIDI 🎀
okay bestie, this one is RFC 4696 🦋 It was written in November 2006 by J. Lazzaro and J. Wawrzynek at UC Berkeley, and it's filed under the Network Working Group. It's Category: Informational, which means it's guidance and not a standard (more on that in the next bit). The title block doesn't list anything it obsoletes or updates, so it isn't replacing or patching any other RFC 💅
Status of This Memo ✨
ngl this part matters: this memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. 🌈
Copyright Notice
Copyright (C) The IETF Trust (2006). 💜
Abstract 🫶
okay so this memo is non-normative implementation guidance for the Real-time Protocol (RTP) MIDI (Musical Instrument Digital Interface) payload format 🎹 It explains its advice through a network musical performance app, where two musicians in different physical locations interact over a network and perform as if they were in the same room 😭 Underneath the performances are RTP MIDI sessions over unicast UDP. The memo also goes into detail on how to send and receive recovery journals, which are the resiliency structure for the payload format 🔥 Even though the focus is network musical performance, the advice is relevant to other RTP MIDI apps too, iykyk 🎶
Table of Contents 📖
1. Introduction ....................................................2
2. Starting the Session ............................................3
3. Session Management: Session Housekeeping ........................6
4. Sending Streams: General Considerations .........................7
4.1. Queuing and Coding Incoming MIDI Data .....................11
4.2. Sending Packets with Empty MIDI Lists .....................12
4.3. Congestion Control and Bandwidth Management ...............13
5. Sending Streams: The Recovery Journal ..........................14
5.1. Initializing the RJSS .....................................16
5.2. Traversing the RJSS .......................................19
5.3. Updating the RJSS .........................................19
5.4. Trimming the RJSS .........................................20
5.5. Implementation Notes ......................................21
6. Receiving Streams: General Considerations ......................21
6.1 The NMP Receiver Design ....................................22
6.2 High-Jitter Networks, Local Area Networks ..................24
7. Receiving Streams: The Recovery Journal ........................25
7.1. Chapter W: MIDI Pitch Wheel (0xE) .........................30
7.2. Chapter N: MIDI NoteOn (0x8) and NoteOff (0x9) ............30
7.3. Chapter C: MIDI Control Change (0xB) ......................32
7.4. Chapter P: MIDI Program Change (0xC) ......................34
8. Security Considerations ........................................35
9. IANA Considerations ............................................35
10. Acknowledgements ..............................................35
11. References ....................................................35
11.1. Normative References .....................................35
11.2. Informative References ...................................361. Introduction ✨
okay so [RFC4695] normatively defines a Real-time Transport Protocol (RTP, [RFC3550]) payload format for the MIDI (Musical Instrument Digital Interface) command language [MIDI] 🎹 It's meant to be used under any applicable RTP profile, like the Audio/Visual Profile (AVP, [RFC3551]) 🎀
BUT (and this is the tea) [RFC4695] does not define algorithms for sending and receiving MIDI streams 🫣 So implementors are free to use any sending or receiving algorithm, as long as it conforms to the normative text in [RFC4695], [RFC3550], [RFC3551], and [MIDI] 💅
This memo is where we give implementation guidance on sending and receiving MIDI RTP streams 🧚 Unlike [RFC4695], this memo is not normative, so think of it as helpful advice and not a rulebook 🌸
RTP is a mature protocol, and there are excellent RTP reference materials out there [RTPBOOK] 📚 This memo doesn't try to repeat them. It focuses on the issues that are specific to the MIDI payload format, which is the whole point, bestie 💜
The memo is about one app: two-party network musical performance over wide-area networks, following the interoperability guidelines in Appendix C.7.2 of [RFC4695] 🎶 Underneath the performances are RTP MIDI sessions over unicast UDP transport. Resiliency comes from the recovery journal system [RFC4695] ✨ The app also uses the RTP Control Protocol (RTCP, [RFC3550]) 🦋
The app is built for a network with a particular set of characteristics: low nominal jitter, low packet loss, and occasional outlier packets that arrive very late 😭 That said, in Section 6.2 of this memo, we talk about adapting the app to other network environments 🌈
As defined in [NMP], a network musical performance happens when musicians at different physical locations interact over a network to perform as if they were in the same room 🎹🫶
Here's the layout: Sections 2-3 cover session startup and maintenance. Sections 4-5 cover sending MIDI streams, and Sections 6-7 cover receiving MIDI streams 🗺️
2. Starting the Session 🎀
okay so in this section we describe how the app starts a two-player session 💖 We assume the two parties have already agreed on a session configuration, which is embodied by a pair of Session Description Protocol (SDP, [RFC4566]) session descriptions 📝
One session description (Figure 1) defines how the first party wishes to receive its stream. The other one (Figure 2) defines how the second party wishes to receive its stream 🌸
The session description in Figure 1 codes that the first party intends to receive a MIDI stream on IP4 number 192.0.2.94 (coded in the c= line) at UDP port 16112 (coded in the m= line) ✨ Implicit in the SDP m= line syntax [RFC4566] is that the first party also intends to receive an RTCP stream on 192.0.2.94 at UDP port 16113 (16112 + 1) 🫶 The receiver expects that the PT field of each RTP header in the received stream will be set to 96 (coded in the m= line) 🔥
Likewise, the session description in Figure 2 codes that the second party intends to receive a MIDI stream on IP4 number 192.0.2.105 at UDP port 5004, and intends to receive an RTCP stream on 192.0.2.105 at UDP port 5005 (5004 + 1) 💜 The second party expects that the PT RTP header field of the received stream will be set to 101 🦋
v=0
o=first 2520644554 2838152170 IN IP4 first.example.net
s=Example
t=0 0
c=IN IP4 192.0.2.94
m=audio 16112 RTP/AVP 96
b=AS:20
b=RS:0
b=RR:400
a=rtpmap:96 mpeg4-generic/44100
a=fmtp:96 streamtype=5; mode=rtp-midi; config=""; profile-level-id=12;
cm_unused=ABFGHJKMQTVXYZ; cm_unused=C120-127; ch_never=ADEFMQTVX;
tsmode=buffer; linerate=320000; octpos=last; mperiod=44; rtp_ptime=0;
rtp_maxptime=0; guardtime=44100; render=synthetic; rinit="audio/asc";
url="http://example.net/sa.asc";
cid="xjflsoeiurvpa09itnvlduihgnvet98pa3w9utnuighbuk"(The a=fmtp line has been wrapped to fit the page to accommodate memo formatting restrictions; it constitutes a single line in SDP.)
Figure 1. Session description for first participant 🎀
v=0
o=second 2520644554 2838152170 IN IP4 second.example.net
s=Example
t=0 0
c=IN IP4 192.0.2.105
m=audio 5004 RTP/AVP 101
b=AS:20
b=RS:0
b=RR:400
a=rtpmap:101 mpeg4-generic/44100
a=fmtp:101 streamtype=5; mode=rtp-midi; config=""; profile-level-id=12;
cm_unused=ABFGHJKMQTVXYZ; cm_unused=C120-127; ch_never=ADEFMQTVX;
tsmode=buffer; linerate=320000;octpos=last;mperiod=44; guardtime=44100;
rtp_ptime=0; rtp_maxptime=0; render=synthetic; rinit="audio/asc";
url="http://example.net/sa.asc";cid="xjflsoeiurvpa09itnvlduihgnvet98pa3w9utnuighbuk"
(The a=fmtp line has been wrapped to fit the page to accommodate memo formatting restrictions; it constitutes a single line in SDP.)
Figure 2. Session description for second participant 🎀
okay so these session descriptions use the mpeg4-generic media type (coded in the a=rtpmap line) to say we're using the MPEG 4 Structured Audio renderer [MPEGSA]. 💖 They also use parameters to customize the stream (Appendix C of [RFC4695]). The parameter values are identical for both parties, so both client hosts get identical rendering environments. It's giving matching outfits 👯♀️✨
The bandwidth (b=) AS parameter [RFC4566] [RFC3550] says the total RTP session bandwidth is 20 kbs. 🦋 That assumes the two players each send a 10 kbs stream at the same time. To get the 10 kbs number, we start with the RTP MIDI payload bandwidth analysis in Appendix A.4 of [NMP], then add RTP and IP4 packet overhead plus a small safety factor. Math with a little cushion, iconic 🧮
The bandwidth RR parameter [RFC3556] says the shared RTCP session bandwidth for the two parties is 400 bps. 🌸 We set the bandwidth SR parameter to 0 bps, which signals that sending parties and non-sending parties equally share the 400 bps of RTCP bandwidth. (Note that in this particular example, the guardtime parameter value of 44100 makes sure both parties are sending for the whole session 🙈) The 400 bps RTCP bandwidth supports one RTCP packet per 5 seconds from each party, containing a Sender Report and CNAME information [RFC3550]. Tiny budget, big vibes 💅
We're about to look at some code that does what the parties do during the session. 🧚 It's written in C and uses the standard network programming techniques described in [STEVENS]. We only show the first party's code, because the second party does a symmetric set of actions. 🫶
Figure 3 shows how the first party sets up a pair of socket descriptors (rtp_fd and rtcp_fd) to send and receive UDP packets. Once the code in Figure 3 has run, the first party can check for new RTP or RTCP packets by calling recv() on rtp_fd or rtcp_fd. ✨
Applications can use recv() to receive UDP packets on a socket in one of two general ways: "blocking" or "non-blocking". 🎀
A call to recv() on a blocking UDP socket puts the calling thread to sleep until a new packet arrives. 😴
A call to recv() on a non-blocking socket polls the device instead. The recv() call returns right away, with a return value that tells you the polling result. A positive return value means the size of a new received packet, and a negative return value (paired with an errno value of EAGAIN) means no new packet was available. Basically it checks and moves on, no drama 💅
Picking blocking or non-blocking sockets is a really critical choice for the app. 👑 Blocking sockets give the lowest potential latency, since the OS wakes the caller as soon as a packet arrives. BUT audio apps that use blocking sockets have to go multi-threaded, so audio samples can be generated on a "rendering thread" while the "network thread" sleeps until the next packet shows up. The architecture also has to support a thread communication mechanism, so the network thread has a way to send MIDI commands to the rendering thread. It's a lot of coordination, not gonna lie 😭
In contrast, audio apps that use non-blocking sockets can be coded with a single thread that alternates between audio sample generation and network polling. 🦋 That trades off some extra network latency (a packet may arrive between polls) for a simpler program architecture. For simplicity, our example uses non-blocking sockets and assumes a single run loop. Figure 4 shows how the example configures its sockets to be non-blocking. Simple is serving 🍓
Figure 5 shows how to use recv() to check a non-blocking socket for new packets. 🌈
The first party also uses rtp_fd and rtcp_fd to send RTP and RTCP packets to the second party. In Figure 6, we show how to initialize socket structures that address the second party. In Figure 7, we show how to use one of those structures in a sendto() call to send an RTP packet to the second party. 💜
Heads up: the code in Figures 3-7 assumes a clear network path between the participants. ⚠️ It may not work if firewalls or Network Address Translation (NAT) devices are in the network path.
3. Session Management: Session Housekeeping ✨
Once the two-party interactive session is set up, the parties start sending and receiving RTP packets. 🌸 In Sections 4-7, we talk about the RTP MIDI sending and receiving algorithms. In this section, we describe the session "housekeeping" tasks the participants also perform. Not the flashiest part, but so important 🧹
One housekeeping task is maintaining the 32-bit Synchronization Source (SSRC) value that uniquely identifies each party. Section 8 of [RFC3550] describes SSRC issues in detail, as does Section 2.1 in [RFC4695]. 🆔 Another housekeeping task is sending and receiving RTCP. Section 6 of [RFC3550] describes RTCP in detail.
Another housekeeping task is security, and this one is serious bestie 🔒 As detailed in the Security Considerations section of [RFC4695], per-packet authentication is strongly recommended for use with MIDI streams, because accepting rogue packets may lead to the execution of arbitrary MIDI commands. > 🚨 Accepting a fake packet could mean somebody else gets to run arbitrary MIDI commands. Please don't skip the auth step, periodt 💀
A final housekeeping task is the termination of the session. 🌙 In our two-party example, the session ends when one of the participants exits. A clean termination may take some active effort from a receiver, because a MIDI stream stopped at an arbitrary point may cause stuck notes and other indefinite artifacts in the MIDI renderer. Nobody wants ghost notes haunting their speakers 👻
The exit of a party may be signalled in several ways. 📣 Session management tools may offer a reliable signal for termination (such as the SIP BYE method [RFC3261]). The (unreliable) RTCP BYE packet [RFC3550] may also signal a party's exit. Receivers may also notice the lack of RTCP activity and time out a party, or they may use transport methods to detect an exit. Lots of options, pick your fighter 🥊💖
4. Sending Streams: General Considerations 🚀
Okay bestie, this section is all about what the sender side actually does. 💅
The sender is literally a real-time, data-driven queen 👑 It's constantly checking whether the local player has cooked up new MIDI data. At any time, the sender may transmit a new RTP packet to the remote player, and it's for one of these two reasons:
- New MIDI data has been generated by the local player, and the sender decides that it is time to issue a packet coding the data.
- The local player has not generated new MIDI data, but the sender decides that too much time has elapsed since the last RTP packet transmission. The sender transmits a packet in order to relay updated header and recovery journal data.
In both cases, the sender builds a packet made of an RTP header, a MIDI command section, and a recovery journal 🎀 In the first case, the MIDI list of the MIDI command section codes the new MIDI data. In the second case, the MIDI list is empty.
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
int rtp_fd, rtcp_fd; /* socket descriptors */
struct sockaddr_in addr; /* for bind address */
/*********************************/
/* create the socket descriptors */
/*********************************/
if ((rtp_fd = socket(AF_INET, SOCK_DGRAM, 0)) < 0)
ERROR_RETURN("Couldn't create Internet RTP socket");
if ((rtcp_fd = socket(AF_INET, SOCK_DGRAM, 0)) < 0)
ERROR_RETURN("Couldn't create Internet RTCP socket");
/**********************************/
/* bind the RTP socket descriptor */
/**********************************/
memset(&(addr.sin_zero), 0, 8);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons(16112); /* port 16112, from SDP */
if (bind(rtp_fd, (struct sockaddr *)&addr,
sizeof(struct sockaddr)) < 0)
ERROR_RETURN("Couldn't bind Internet RTP socket");
/***********************************/
/* bind the RTCP socket descriptor */
/***********************************/
memset(&(addr.sin_zero), 0, 8);
addr.sin_family = AF_INET;
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_port = htons(16113); /* port 16113, from SDP */
if (bind(rtcp_fd, (struct sockaddr *)&addr,
sizeof(struct sockaddr)) < 0)
ERROR_RETURN("Couldn't bind Internet RTCP socket");Figure 3. Setup code for listening for RTP/RTCP packets
#include <unistd.h>
#include <fcntl.h>
/***************************/
/* set non-blocking status */
/***************************/
if (fcntl(rtp_fd, F_SETFL, O_NONBLOCK))
ERROR_RETURN("Couldn't unblock Internet RTP socket");
if (fcntl(rtcp_fd, F_SETFL, O_NONBLOCK))
ERROR_RETURN("Couldn't unblock Internet RTCP socket");Figure 4. Code to set socket descriptors to be non-blocking
#include <errno.h>
#define UDPMAXSIZE 1472 /* based on Ethernet MTU of 1500 */
unsigned char packet[UDPMAXSIZE+1];
int len, normal;
while ((len = recv(rtp_fd, packet, UDPMAXSIZE + 1, 0)) > 0)
{
/* process packet[]. If (len == UDPMAXSIZE + 1), recv()
* may be returning a truncated packet -- process with care
*/
}
/* line below sets "normal" to 1 if the recv() return */
/* status indicates no packets are left to process */
normal = (len < 0) && (errno == EAGAIN);
if (!normal)
{
/*
* recv() return status indicates an empty UDP payload
* (len == 0) or an error condition (coded by (len < 0)
* and (errno != EAGAIN)). Examine len and errno, and
* take appropriate recovery action.
*/
}Figure 5. Code to check rtp_fd for new RTP packets
#include <arpa/inet.h>
#include <netinet/in.h>
struct sockaddr_in * rtp_addr; /* RTP destination IP/port */
struct sockaddr_in * rtcp_addr; /* RTCP destination IP/port */
/* set RTP address, as coded in Figure 2's SDP */
rtp_addr = calloc(1, sizeof(struct sockaddr_in));
rtp_addr->sin_family = AF_INET;
rtp_addr->sin_port = htons(5004);
rtp_addr->sin_addr.s_addr = inet_addr("192.0.2.105");
/* set RTCP address, as coded in Figure 2's SDP */
rtcp_addr = calloc(1, sizeof(struct sockaddr_in));
rtcp_addr->sin_family = AF_INET;
rtcp_addr->sin_port = htons(5005); /* 5004 + 1 */
rtcp_addr->sin_addr.s_addr = rtp_addr->sin_addr.s_addr;Figure 6. Initializing destination addresses for RTP and RTCP
unsigned char packet[UDPMAXSIZE]; /* RTP packet to send */
int size; /* length of RTP packet */
/* first fill packet[] and set size ... then: */
if (sendto(rtp_fd, packet, size, 0, rtp_addr,
sizeof(struct sockaddr)) == -1)
{
/*
* try again later if errno == EAGAIN or EINTR
*
* other errno values --> an operational error
*/
}Figure 7. Using sendto() to send an RTP packet
Figure 8 lays out the 5 steps a sender takes to issue a packet 🦋 This algorithm matches the code fragment for sending RTP packets shown in Figure 7 of Section 2. Steps 1, 2, and 3 happen before the sendto() call in that code fragment. Step 4 is the sendto() call itself. Step 5 may happen once Step 3 completes.
The algorithm for Sending a Packet is as follows:
- Generate the RTP header for the new packet. See Section 2.1 of [RFC4695] for details.
- Generate the MIDI command section for the new packet. See Section 3 of [RFC4695] for details.
- Generate the recovery journal for the new packet. We discuss this process in Section 5.2. The generation algorithm examines the Recovery Journal Sending Structure (RJSS), a stateful coding of a history of the stream.
- Send the new packet to the receiver.
- Update the RJSS to include the data coded in the MIDI command section of the packet sent in step 4. We discuss the update procedure in Section 5.3.
Figure 8. A 5 step algorithm for sending a packet
Next up, we get into the specific sender implementation issues in detail ✨
4.1. Queuing and Coding Incoming MIDI Data 🎀
Simple senders transmit a new packet as soon as the local player generates a complete MIDI command. The system described in [NMP] uses this algorithm. This algorithm minimizes the sender queuing latency, as the sender never delays the transmission of a new MIDI command.
In a relative sense, this algorithm uses bandwidth inefficiently, as it does not amortize the overhead of a packet over several commands. This inefficiency may be acceptable for sparse MIDI streams (see Appendix A.4 of [NMP]). More sophisticated sending algorithms [GRAME] improve efficiency by coding small groups of commands into a single packet, at the expense of increasing the sender queuing latency. 🧚
Senders assign a timestamp value to each command issued by the local player (Appendix C.3 of [RFC4695]). Senders may code the timestamp value of the first MIDI list command in two ways. The most efficient method is to set the RTP timestamp of the packet to the timestamp value of the first command. In this method, the Z bit of the MIDI command section header (Figure 2 of [RFC4695]) is set to 0, and the RTP timestamps increment at a non-uniform rate.
However, in some applications, senders may wish to generate a stream whose RTP timestamps increment at a uniform rate. To do so, senders may use the Delta Time MIDI list field to code a timestamp for the first command in the list. In this case, the Z bit of the MIDI command section header is set to 1.
Senders should strive to maintain a constant relationship between the RTP packet timestamp and the packet sending time: if two packets have RTP timestamps that differ by 1 second, the second packet should be sent 1 second after the first packet. To the receiver, variance in this relationship is indistinguishable from network jitter. Latency issues are discussed in detail in Section 6. 🔥
Senders may alter the running status coding of the first command in the MIDI list, in order to comply with the coding rules defined in Section 3.2 of [RFC4695]. The P header bit (Figure 2 of [RFC4695]) codes this alteration of the source command stream. 💜
4.2. Sending Packets with Empty MIDI Lists 💤
Okay so, during a session musicians can totally chill and not generate MIDI data for ages, like seconds or even minutes 🫠. If an RTP stream copied that silent MIDI source and just stopped sending RTP packets, the whole system could get kinda messy:
- The receiver's model of network performance may fall out of date. 💀
- Network middleboxes (such as Network Address Translators) may "time-out" the silent stream and drop the port and IP association state. 🙈
- If the session does not use RTCP, receivers may misinterpret the silent stream as a dropped network connection. 😭
Senders avoid all this drama by sending "keep-alive" RTP packets during periods of network inactivity ✨. Keep-alive packets have empty MIDI lists, so they're basically the stream saying "hey, still here" 🫶.
Session participants may specify how often keep-alive packets go out during session configuration with the MIME parameter "guardtime" (Appendix C.4.2 of [RFC4695]). The session descriptions in Figures 1-2 use guardtime to set a keep-alive sending interval of 1 second. Iconic, honestly 💅.
Senders may also send empty packets to make the recovery journal system work better 🎀. As we describe in Section 6, the recovery process begins when a receiver detects a break in the RTP sequence number pattern of the stream. The receiver uses the recovery journal of the break packet to guide its corrective rendering actions, like ending stuck notes and updating out-of-date controller values 🔥.
Here's the situation: the local player produces a MIDI NoteOff command (which the sender promptly transmits in a packet), but then 5 seconds pass before the player produces another MIDI command (which the sender transmits in a second packet) ⏳. If the packet coding the NoteOff gets lost, the receiver isn't aware of the packet loss for 5 seconds, and the rendered MIDI performance contains a note that sounds for 5 seconds too long. Not a vibe 💀.
To handle this, senders may transmit empty packets to "guard" the stream during silent sections 🛡️. The guard packet algorithm defined in Section 7.3 of [NMP], as applied to the situation above, sends a guard packet after 100 ms of player inactivity, then sends a second guard packet 100 ms later. Subsequent guard packets go out with an exponential backoff, with a limiting period of 1 second (set by the "guardtime" parameter in Figures 1-2). The algorithm terminates once MIDI activity resumes, or once RTCP receiver reports indicate that the receiver is up to date 🌈.
The perceptual quality of guard packet-sending algorithms is a quality of implementation issue for RTP MIDI applications 💜. Sophisticated implementations may tailor the guard packet sending rate to the nature of the MIDI commands recently sent in the stream, so the perceptual impact of moderate packet loss stays minimal. Main character engineering 👑.
As an example of this sort of specialization, the guard packet algorithm described in [NMP] protects against the transient artifacts that occur when NoteOn commands are lost 🦋. The algorithm sends a guard packet 1 ms after every packet whose MIDI list contains a NoteOn command. The Y bit in Chapter N note logs (Appendix A.6 of [RFC4695]) supports this use of guard packets 🍓.
Congestion control and bandwidth management are key issues in guard packet algorithms. We'll discuss these in the next section, so stay tuned 🧚.
4.3. Congestion Control and Bandwidth Management 🚦
The congestion control section of [RFC4695] talks about why congestion control matters for RTP MIDI streams, and it references the normative text in [RFC3550] and [RFC3551] that covers congestion control. To comply with the requirements in those normative documents, RTP MIDI senders may use several methods to control the sending rate 🌸:
- As described in Section 4.1, senders may pack several MIDI commands into a single packet, which reduces stream bandwidth (at the expense of increasing sender queuing latency) 📦.
- Guard packet algorithms (Section 4.2) may be designed in a parametric way, so the tradeoff between artifact reduction and stream bandwidth can be tuned dynamically 🎚️.
- The recovery journal size may be reduced by adapting the techniques described in Section 5 of this memo. Note that in all cases, the recovery journal sender must conform to the normative text in Section 4 of [RFC4695].
- The incoming MIDI stream may be modified to reduce the number of MIDI commands without significantly altering the performance. Lossy "MIDI filtering" algorithms are well developed in the MIDI community and may be directly applied to RTP MIDI rate management 🧃.
RTP MIDI senders fold these rate control methods into feedback systems to implement congestion control and bandwidth management ✨. Sections 10 and 6.4.4 of [RFC3550] and Section 2 in [RFC3551] describe feedback systems for congestion control in RTP, and Section 6 of [RFC4566] describes bandwidth management in media sessions 📚.
5. Sending Streams: The Recovery Journal 🧾
In this section, we get into how senders actually implement the recovery journal system 💖. The implementation we describe uses the default "closed-loop" recovery journal semantics (Appendix C.2.2.2 of [RFC4695]).
We start with the Recovery Journal Sending Structure (RJSS) 🏗️. Senders use the RJSS to generate the recovery journal section for RTP MIDI packets.
The RJSS is a hierarchical representation of the checkpoint history of the stream. The checkpoint history holds the MIDI commands that are at risk to packet loss (Appendix A.1 of [RFC4695] precisely defines the checkpoint history). The layout of the RJSS mirrors the hierarchical structure of the recovery journal bitfields 🪆.
Figure 9 shows an RJSS implementation for a simple sender. The leaf level of the RJSS hierarchy (the jsend_chapter structures) corresponds to channel chapters (Appendices A.2-9 in [RFC4695]). The second level of the hierarchy (jsend_channel) corresponds to the channel journal header (Figure 9 in [RFC4695]). The top level of the hierarchy (jsend_journal) corresponds to the recovery journal header (Figure 8 in [RFC4695]).
Each RJSS data structure may code several items:
- The current contents of the recovery journal bitfield associated with the RJSS structure (
jheader[],cheader[], or a chapter bitfield). - A
seqnumvariable. Seqnum codes the extended RTP sequence number of the most recent packet that added information to the RJSS structure. If the seqnum of a structure is updated, the seqnums of all structures above it in the recovery journal hierarchy are also updated. So a packet that caused an update to a specificjsend_chapterstructure would update the seqnum values of this structure and of thejsend_channelandjsend_journalstructures that contain it 🔗. - Ancillary variables used by the sending algorithm.
A seqnum variable for a level is set to zero if the checkpoint history contains no information at that level, and no information at any level below it. This coding scheme assumes that the first sequence number of a stream is normalized to 1, and limits the total number of stream packets to 2^32 - 1 🔢.
The cm_unused and ch_never parameters in Figures 1-2 define the subset of MIDI commands supported by the sender (see Appendix C.2.3 of [RFC4695] for details). The sender transmits most voice commands but does not transmit system commands. The sender assumes the MIDI source uses note commands in the typical way, so the sender does not use the Chapter E note resiliency tools (Appendix A.7 of [RFC4695]). The sender does not support Control Change commands for controller numbers with All Notes Off (123-127), All Sound Off (120), and Reset All Controllers (121) semantics, and it does not support enhanced Chapter C encoding (Appendix A.3.3 of [RFC4695]) 🙅♀️.
We chose this subset of MIDI commands to keep the example simple 🧸. In particular, the command restrictions ensure that all commands are active, that all note commands are N-active, and that all Control Change commands are C-active (see Appendix A.1 of [RFC4695] for definitions of active, N-active, and C-active) 💯.
In the sections that follow, we walk through the tasks a sender performs to manage the recovery journal system. Let's gooo 🚀.
5.1. Initializing the RJSS
okay bestie, starting a stream is a whole vibe 🌸 the sender sets up the RJSS (recovery journal sending structure) from scratch ✨ all the seqnum variables go to zero, and that includes every single element of note_seqnum[] and control_seqnum[] 🫶
the sender then sets jheader[] up so it codes an empty journal 💅 the S bit of the header is set to 1, while the A, Y, R, and TOTCHAN header fields are all set to zero. the checkpoint packet sequence number field gets the sequence number of the upcoming first RTP packet (per Appendix A.1 of [RFC4695]) 🦋
typedef unsigned char uint8; /* must be 1 octet */
typedef unsigned short uint16; /* must be 2 octet */
typedef unsigned long uint32; /* must be 4 octets */
/**************************************************************/
/* leaf level hierarchy: Chapter W, Appendix A.5 of [RFC4695] */
/**************************************************************/
typedef struct jsend_chapterw { /* Pitch Wheel (0xE) */
uint8 chapterw[2]; /* bitfield Figure A.5.1 [RFC4695] */
uint32 seqnum; /* extended sequence number, or 0 */
} jsend_chapterw;
/**************************************************************/
/* leaf level hierarchy: Chapter N, Appendix A.6 of [RFC4695] */
/**************************************************************/
typedef struct jsend_chaptern { /* Note commands (0x8, 0x9) */
/* chapter N maximum size is 274 octets: a 2 octet header, */
/* and a maximum of 128 2-octet logs and 16 OFFBIT octets */
uint8 chaptern[274]; /* bitfield Figure A.6.1 [RFC4695] */
uint16 size; /* actual size of chaptern[] */
uint32 seqnum; /* extended seq number, or 0 */
uint32 note_seqnum[128]; /* most recent note seqnum, or 0 */
uint32 note_tstamp[128]; /* NoteOn execution timestamp */
uint32 bitfield_ptr[128]; /* points to a chapter log, or 0 */
} jsend_chaptern;
/**************************************************************/
/* leaf level hierarchy: Chapter C, Appendix A.3 of [RFC4695] */
/**************************************************************/
typedef struct jsend_chapterc { /* Control Change (0xB) */
/* chapter C maximum size is 257 octets: a 1 octet header */
/* and a maximum of 128 2-octet logs */
uint8 chapterc[257]; /* bitfield Figure A.3.1 [RFC4695] */
uint16 size; /* actual size of chapterc[] */
uint32 seqnum; /* extended sequence number, or 0 */
uint32 control_seqnum[128]; /* most recent seqnum, or 0 */
uint32 bitfield_ptr[128]; /* points to a chapter log, or 0 */
} jsend_chapterc;Figure 9. Recovery Journal Sending Structure (part 1)
/**************************************************************/
/* leaf level hierarchy: Chapter P, Appendix A.2 of [RFC4695] */
/**************************************************************/
typedef struct jsend_chapterp { /* MIDI Program Change (0xC) */
uint8 chapterp[3]; /* bitfield Figure A.2.1 [RFC4695] */
uint32 seqnum; /* extended sequence number, or 0 */
} jsend_chapterp;
/***************************************************/
/* second-level of hierarchy, for channel journals */
/***************************************************/
typedef struct jsend_channel {
uint8 cheader[3]; /* header Figure 9 [RFC4695]) */
uint32 seqnum; /* extended sequence number, or 0 */
jsend_chapterp chapterp; /* chapter P info */
jsend_chapterc chapterc; /* chapter C info */
jsend_chapterw chapterw; /* chapter W info */
jsend_chaptern chaptern; /* chapter N info */
} jsend_channel;
/*******************************************************/
/* top level of hierarchy, for recovery journal header */
/*******************************************************/
typedef struct jsend_journal {
uint8 jheader[3]; /* header Figure 8, [RFC4695] */
/* Note: Empty journal has a header */
uint32 seqnum; /* extended sequence number, or 0 */
/* seqnum = 0 codes empty journal */
jsend_channel channels[16]; /* channel journal state */
/* index is MIDI channel */
} jsend_journal;Figure 9. Recovery Journal Sending Structure (part 2)
in jsend_chaptern, every element of note_tstamp[] gets set to zero 🙈 and in both jsend_chaptern and jsend_chapterc, the elements of bitfield_ptr[] get set to the null pointer index value. (fyi bitfield_ptr[] is an array whose elements point to the first octet of the note or control log that goes with that array index 🧚)
5.2. Traversing the RJSS
okay this is the part where the sender walks the whole structure every single time it builds a packet 🎀 whenever an RTP packet is created (Step 3 of the algorithm defined in Figure 8), the sender traverses the RJSS to make the recovery journal for that packet. the traversal starts at the top level of the RJSS. the sender copies jheader[] into the packet, and then sets the S bit of jheader[] to 1 ✨
the traversal keeps going depth-first 🌈 it visits every jsend_channel whose seqnum variable is non-zero. the sender copies the cheader[] array into the packet and then sets the S bit of cheader[] to 1. after each cheader[] copy, the sender visits each leaf-level chapter, in the order they show up in the chapter journal Table of Contents (first P, then C, then W, then N, as shown in Figure 9 of [RFC4695]) 👑
if a chapter has a non-zero seqnum, the sender copies the chapter bitfield array into the packet and then sets the S bit of the RJSS array to 1. for chaptern[], the B bit is also set to 1 🔥 for the variable-length chapters (chaptern[] and chapterc[]), the sender checks the size variable to figure out how long the bitfield is.
before copying chaptern[], the sender updates the Y bit of each note log to code the onset of the associated NoteOn command (Figure A.6.3 in [RFC4695]). to work out the Y bit value, the sender checks the note_tstamp[] array for the note timing info 💜
5.3. Updating the RJSS
after an RTP packet is sent, the sender updates the RJSS to refresh the checkpoint history (Step 5 of the sending algorithm defined in Figure 8) 🦋 for each command in the MIDI list of the sent packet, the sender runs the update procedure we're about to talk about.
the update procedure starts at the leaf level 🍓 the sender generates a new bitfield array for the chapter that goes with the MIDI command, using the chapter-specific semantics defined in Appendix A of [RFC4695].
for Chapter N and Chapter C, the sender uses the bitfield_ptr[] array to find and update an existing log for a note or controller 🔍 if a log doesn't exist yet, the sender adds one to the end of the chaptern[] or chapterc[] bitfield and changes the bitfield_ptr[] value so it points to that log. for Chapter N, the sender also updates note_tstamp[].
the sender also clears the S bit of the chapterp[], chapterw[], or chapterc[] bitfield. for chaptern[], the sender clears the S bit or the B bit of the bitfield, as described in Appendix A.6 of [RFC4695] 💅
next, the sender refreshes the upper levels of the RJSS hierarchy ✨ at the second level, it updates the cheader[] bitfield of the channel that goes with the command, and sets the S bit of cheader[] to 0. if the new command forced the addition of a new chapter or channel journal, the sender may also update other cheader[] fields. at the top level, the sender updates the top-level jheader[] bitfield in a similar way 🫶
finally, the sender updates the seqnum variables that go with the changed bitfield arrays. it sets those seqnum variables to the extended sequence number of the packet 🎀
5.4. Trimming the RJSS
this is the cleanup crew, and it's lowkey so important 🧹 at regular intervals, receivers send RTCP receiver reports to the sender (as described in Section 6.4.2 of [RFC3550]). these reports include the extended highest sequence number received (EHSNR) field. that field codes the highest sequence number the receiver has seen from the sender, extended so rollover can't mess up the count 🌸
when the sender gets an RTCP receiver report, it runs the RJSS trimming algorithm. the trimming algorithm uses the EHSNR to trim away parts of the RJSS, so the recovery journals in later RTP packets get smaller 🦋 the algorithm follows the closed-loop sending policy defined in Appendix C.2.2.2 of [RFC4695] 👑
the trimming algorithm is built on one observation: if the EHSNR says a packet with sequence number K has been received, then MIDI commands sent in packets with sequence numbers J <= K can be removed from the RJSS without breaking the closed-loop policy 🧚
to kick off the trimming algorithm, the sender pulls the EHSNR field out of the receiver report and adjusts it to reflect the sender's sequence number extension prefix. then it compares the adjusted EHSNR value with the seqnum fields at each level of the RJSS, starting at the top level 💖
levels whose seqnum is less than or equal to the adjusted EHSNR get trimmed, by setting their seqnum to zero 🙈 if needed, the jheader[] and cheader[] arrays above the trimmed level get adjusted to match the new journal layout. the checkpoint packet sequence number field of jheader[] gets updated to match the EHSNR.
at the leaf level, the sender trims the size of the variable-length chaptern[] and chapterc[] bitfields 🔥 it loops through the note_seqnum[] or control_seqnum[] array and removes the chaptern[] or chapterc[] logs whose seqnum value is less than or equal to the adjusted EHSNR. it sets the associated bitfield_ptr[] to null and updates the LENGTH field of the associated cheader[] bitfield 🍓
the trimming algorithm does NOT add anything to the checkpoint history 👀 so it doesn't clear the S bit (and for chaptern[], the B bit) of any recovery journal bitfield. it also doesn't set the RJSS seqnum variables to the EHSNR value. (stay with me, it matters 💀)
5.5. Implementation Notes
real talk, the recovery journal sender in this doc is simplified on purpose 🎓 it's for teaching, not for shipping as-is. in practice, an implementation would use enhanced versions of the traversing, updating, and trimming algorithms presented in Sections 5.2-5.4 ✨
6. Receiving Streams: General Considerations ✨
okay bestie, now we're on the receiver side 🎧 this part is all about what happens when the packets actually land on the other end. we're talking receiver implementation issues, so pay attention 💅
to start, we pretend the network is perfect 🦋 packets never get lost or reordered, and the end-to-end latency stays constant. we also assume that all commands coded in the MIDI list of a packet share the same timestamp. that assumption is coded by the "rtp_ptime" and "rtp_maxptime" values in Figures 1-2 (see Appendix C.4.1 of [RFC4695] for the deets).
under those conditions, a simple algorithm can give you a high-quality performance 🔥 when an RTP packet arrives, the receiver immediately executes the commands coded in the MIDI command section of the payload. the commands run in the order they appear in the MIDI list, and the command timestamps get ignored 🙈
but this simple algorithm falls apart once we stop assuming a perfect network and a perfect MIDI list 💀
- if we let lost and reordered packets happen in the network, the algorithm may produce rendering artifacts that can't be recovered from, which goes against the mandate defined in Section 4 of [RFC4695] 😭
- if we let the network have variable latency, the algorithm modulates the network jitter onto the rendered MIDI command stream 🌀
- if we let a MIDI list code commands with different timestamps, the algorithm adds temporal jitter to the rendered performance, because it ignores MIDI list timestamps 🎀
in this section, we look at interactive receiver design techniques for these messier conditions ✨ Section 6.1 covers a receiver design for high-performance Wide Area Networks (WANs), and Section 6.2 talks about design issues for other kinds of networks.
6.1. The NMP Receiver Design
the Network Musical Performance (NMP) system [NMP] is an interactive performance app that uses an early version of the RTP MIDI payload format 🎹 it was designed for universities within the State of California, which run on the high-performance CalREN2 network. so iconic, right? 🌈
in the NMP system, network artifacts can change how a musician hears the performances of remote players. however, the network does not affect how a musician hears their own performance 🫶
several aspects of CalREN2 network behavior guided the NMP system design. these were measured in the 2001 timeframe and documented in [NMP] 📊
- the median symmetric latency (1/2 the round-trip time) of packets sent between network sites is comparable to the acoustic latency between two musicians in the same room. for example, the latency between Berkeley and Stanford is 2.1 ms, which corresponds to an acoustic distance of 2.4 feet (0.72 meters). those campuses are 40 miles (64 km) apart. keeping the benefits of the underlying network latency at the application level was a key NMP design goal 🦋
- for most times of day, the nominal temporal jitter is pretty short. for Berkeley-Stanford, the standard deviation of the round-trip time was under 200 microseconds 🌸
- for most times of day, a few percent (0-4%) of packets arrive significantly late (> 40 ms), probably because of a queuing transient somewhere in the network path. more rarely (< 0.1%), a packet gets lost during the transient 💀
- at predictable times during the day (before lunchtime, at the end of the workday, etc.), network performance gets worse (10-20% late packets), in a way that makes the network unsuitable for low-latency interactive use 😭
- CalREN2 has deeply over-provisioned bandwidth, relative to MIDI bandwidth usage 👑
the NMP sender freely uses network bandwidth to make the performance experience better 💅 as soon as a musician generates a MIDI command, an RTP packet coding that command is sent to the other players. this sending algorithm reduces latency at the cost of bandwidth 💖 on top of that, guard packets (described in Section 4.2) are sent at frequent intervals to minimize the impact of packet loss 🛡️
the NMP receiver keeps a model of the stream and uses that model as the basis of its resiliency system 🧠 when a packet arrives, the receiver predicts the RTP sequence number and the RTP timestamp (with error bars). under normal network conditions, about 95% of received packets fit those predictions [NMP]. in this common case, the receiver immediately executes the MIDI command coded in the packet 🎉
note: the NMP receiver does not use a playout buffer. the design is optimized for the lowest latency, at the expense of command jitter. so the NMP receiver design does not completely satisfy the interoperability text in Appendix C.7.2 of [RFC4695], which requires that receivers in network musical performance applications be capable of using a playout buffer 🙈
sometimes an incoming packet fits the sequence number prediction but falls outside the timestamp prediction error bars (see Appendix B of [NMP] for the timestamp model details) 🫣 in most cases, the receiver still executes the command coded in the packet. the one exception is that the receiver discards NoteOn commands with non-zero velocity. by tossing late commands that sound notes, the receiver stops "straggler notes" from messing up a performance 🚫 by executing all the other late commands, the receiver quiets "soft stuck notes" right away and updates the state of the MIDI system ✨
more rarely, an incoming packet doesn't fit the sequence number prediction at all 😳 the receiver keeps track of the highest sequence number received in the stream, and predicts that an incoming packet will have a sequence number one greater than that value. if an incoming packet's sequence number is greater than the prediction, a packet loss has happened. if it's less than the prediction, the packet arrived out of order. all sequence number calculations are modulo 2^16, and they use the standard methods (described in [RFC3550]) to avoid tracking errors during rollover 🔄
if a packet loss has happened, the receiver checks the journal section of the received packet and uses it to recover gracefully from the loss episode 🩹 we describe this recovery procedure in Section 7 of this memo. the recovery process may result in executing one or more MIDI commands. after running the recovery commands, the receiver processes the MIDI command encoded in the packet using the timestamp model test described above 💜
if a packet arrives out of order, the receiver ignores it 🙅♀️ the reason is that an out-of-order packet is always preceded by a packet that signalled a loss event. that loss event triggered the recovery process, which may already have executed recovery commands. the MIDI command coded in the out-of-order packet might duplicate those recovery commands if it ran, and that duplication could endanger the integrity of the stream. so ignoring the out-of-order packet is the safe move 🫶
6.2. High-Jitter Networks, Local Area Networks
the NMP receiver targets a network with a particular set of traits: low nominal jitter, low packet loss, and the occasional outlier packet that shows up very late 🌸 in this section, we look at how networks with different traits affect receiver design 🧐
networks with significant nominal jitter can't use the buffer-free receiver design from Section 6.1. for example, the NMP system performs poorly for musicians on dial-up modem connections, because the buffer-free design modulates modem jitter onto the performances 📞 receivers built for high-jitter networks should use a substantial playout buffer. References [GRAME] and [CCRMA] describe how to use playout buffers in latency-critical applications 📚
receivers intended for Local Area Networks (LANs) face a different set of issues 🏠 a dedicated LAN fabric built with modern hardware is predictable in many ways. the network problems the NMP receiver design addresses (packet loss and outlier late packets) might only happen under extreme network overload conditions 🤯
systems designed for this environment may choose to configure streams without the recovery journal system (Appendix C.2.1 of [RFC4695]). receivers may also wish to forgo or simplify the detection of outlier late packets. receivers should monitor the RTP sequence numbers of incoming packets to detect network unreliability 👀
however, in some ways LAN applications can be more demanding than WAN applications 😤 in LAN applications, musicians may be getting performance feedback from audio that is rendered from the stream. the tolerance a musician has for latency and jitter in this context may be quite low 🥺
to reduce the perceived jitter, receivers may use a small playout buffer (in the range of 100us to 2ms) 🎧 the buffer adds a little latency to the system, which may annoy some players. receiver designs should include buffer tuning parameters so musicians can adjust the tradeoff between latency and jitter 🎛️💖
7. Receiving Streams: The Recovery Journal ✨
okay bestie, this section is where we get into the recovery algorithm the NMP receiver uses [NMP] 💅 Most of it is pretty general, so it works for interactive receiver design in lots of places 🦋, but a few parts are custom-built for NMP 🫶
- The recovery algorithm covers a subset of the MIDI command set. MIDI Systems (
0xF), Poly Aftertouch (0xA), and Channel Aftertouch (0xD) commands are not protected, and Control Change (0xB) command protection is simplified. Note commands for a particular note number are assumed to follow the typicalNoteOn->NoteOff->NoteOn->NoteOffpattern. Thecm_unusedandch_neverparameters in Figures 1-2 specify this coverage. - The NMP system does not use a playout buffer. Therefore, the recovery algorithm does not address interactions with a playout buffer.
So at a high level, the receiver works like this 🔍 When it detects a packet loss, it looks at the recovery journal of the packet that ends the loss event. If it's needed, the receiver runs one or more MIDI commands to recover from the loss.
To get ready for recovery, a receiver keeps a data structure called the Recovery Journal Receiver Structure (RJRS) 🧠 The RJRS tracks info about the MIDI commands the receiver runs, both the incoming stream commands and the recovery commands it generates itself. At the start of the stream, the RJRS is set to say that no commands have been executed yet. Right after the receiver executes a MIDI command, it updates the RJRS with info about that command ⭐
Now let's get into the details 💜 We start with two definitions that sort loss events into types. These definitions assume that the packet that ends the loss event has RTP sequence number I.
- Single-packet loss. A single-packet loss occurs if the last packet received before the loss event (excluding out-of-order packets) has the sequence number
I-2(modulo2^16). - Multi-packet loss. A multi-packet loss occurs if the last packet received before the loss event (excluding out-of-order packets) has a sequence number less than
I-2(modulo2^16).
When a packet loss is detected, the recovery algorithm checks the recovery journal header (Figure 8 of [RFC4695]) for special cases 👀
- If the header field
Ais0, the recovery journal has no channel journals, so no action is taken. - If a single-packet loss has occurred, and if the header
Sbit is1, the lost packet has a MIDI command section with an empty MIDI list. No action is taken.
If none of those special cases apply, the algorithm parses the recovery journal body 🎀 For each channel journal (Figure 9 in [RFC4695]) in the recovery journal, the receiver compares the data in each chapter journal (Appendix A of [RFC4695]) to the RJRS data for that chapter. If the data don't match, the algorithm infers that the MIDI commands related to that chapter journal were lost. The recovery algorithm then runs MIDI commands to fix the loss and updates the RJRS to reflect the fix 🔧
For single-packet losses, the receiver skips channel and chapter journals whose S bits are set to 1. For multi-packet losses, the receiver parses each channel and chapter journal and checks for inconsistency ✨
In the sections that follow, we go through the recovery steps for each chapter journal type 💖 We cover 4 types: P (Program Change, 0xC), C (Control Change, 0xB), W (Pitch Wheel, 0xE), and N (Note, 0x8 and 0x9). Chapters are parsed in the order they show up in the channel journal (P, then W, then N, then C) 🦋
The sections below use the C implementation of the RJRS shown in Figure 10 💅 The structure is hierarchical, which matches the recovery journal architecture. At the leaf level, specialized data structures (jrec_chapterw, jrec_chaptern, jrec_chapterc, and jrec_chapterp) hold the state variables for a single chapter journal type. A mid-level structure (jrec_channel) represents one MIDI channel, and the top-level structure (jrec_stream) represents the entire MIDI stream 🌈
typedef unsigned char uint8; /* must be 1 octet */
typedef unsigned short uint16; /* must be 2 octets */
typedef unsigned long uint32; /* must be 4 octets */
/*****************************************************************/
/* leaf level of hierarchy: Chapter W, Appendix A.5 of [RFC4695] */
/*****************************************************************/
typedef struct jrec_chapterw { /* MIDI Pitch Wheel (0xE) */
uint16 val; /* most recent 14-bit wheel value */
} jrec_chapterw;
/*****************************************************************/
/* leaf level of hierarchy: Chapter N, Appendix A.6 of [RFC4695] */
/*****************************************************************/
typedef struct jrec_chaptern { /* Note commands (0x8, 0x9) */
/* arrays of length 128 --> one for each MIDI Note number */
uint32 time[128]; /* exec time of most recent NoteOn */
uint32 extseq[128]; /* extended seqnum for that NoteOn */
uint8 vel[128]; /* NoteOn velocity (0 for NoteOff) */
} jrec_chaptern;
/*****************************************************************/
/* leaf level of hierarchy: Chapter C, Appendix A.3 of [RFC4695] */
/*****************************************************************/
typedef struct jrec_chapterc { /* Control Change (0xB) */
/* array of length 128 --> one for each controller number */
uint8 value[128]; /* Chapter C value tool state */
uint8 count[128]; /* Chapter C count tool state */
uint8 toggle[128]; /* Chapter C toggle tool state */
} jrec_chapterc;
Figure 10. Recovery Journal Receiving Structure (part 1)
/*****************************************************************/
/* leaf level of hierarchy: Chapter P, Appendix A.2 of [RFC4695] */
/*****************************************************************/
typedef struct jrec_chapterp { /* MIDI Program Change (0xC) */
uint8 prognum; /* most recent 7-bit program value */
uint8 prognum_qual; /* 1 once first 0xC command arrives */
uint8 bank_msb; /* most recent Bank Select MSB value */
uint8 bank_msb_qual; /* 1 once first 0xBn 0x00 arrives */
uint8 bank_lsb; /* most recent Bank Select LSB value */
uint8 bank_lsb_qual; /* 1 once first 0xBn 0x20 arrives */
} jrec_chapterp;
/***************************************************/
/* second-level of hierarchy, for MIDI channels */
/***************************************************/
typedef struct jrec_channel {
jrec_chapterp chapterp; /* Program Change (0xC) info */
jrec_chapterc chapterc; /* Control Change (0xB) info */
jrec_chapterw chapterw; /* Pitch Wheel (0xE) info */
jrec_chaptern chaptern; /* Note (0x8, 0x9) info */
} jrec_channel;
/***********************************************/
/* top level of hierarchy, for the MIDI stream */
/***********************************************/
typedef struct jrec_stream {
jrec_channel channels[16]; /* index is MIDI channel */
} jrec_stream;
Figure 10. Recovery Journal Receiving Structure (part 2)7.1. Chapter W: MIDI Pitch Wheel (0xE) 🌸
Chapter W of the recovery journal protects against losing MIDI Pitch Wheel (0xE) commands 🎡 A common use of the Pitch Wheel command is to send the current position of a rotary "pitch wheel" controller on the side of MIDI piano controllers. Players use the pitch wheel to change the pitch of all the depressed keys on the fly 🎹
The NMP receiver keeps the jrec_chapterw structure (Figure 10) for each voice channel in jrec_stream, so it can track the pitch wheel state 💅 In jrec_chapterw, val holds the 14-bit data value of the most recent Pitch Wheel command that has arrived on a channel. At the start of the stream, val is set to the default pitch wheel value (0x2000) 🧚
When a loss event ends, a receiver may find a Chapter W (Appendix A.5 in [RFC4695]) bitfield in a channel journal. This chapter codes the 14-bit data value of the most recent MIDI Pitch Wheel command in the checkpoint history. If the Chapter W value and the jrec_chapterw pitch wheel value don't match, then one or more commands have been lost 😭
To recover from that loss, the NMP receiver immediately runs a MIDI Pitch Wheel command on the channel, using the data value coded in the recovery journal 🙌 After that, the receiver updates the jrec_chapterw variables to reflect the command it just ran 💖
7.2. Chapter N: MIDI NoteOn (0x8) and NoteOff (0x9) 🎹✨
Okay bestie, Chapter N of the recovery journal protects against the loss of MIDI NoteOn (0x9) and NoteOff (0x8) commands 🎹. If a NoteOn gets lost, a note just gets skipped 💀. If a NoteOff gets lost, a note may sound forever, which is so not it 😭. Also, ngl, a NoteOn with a velocity of 0 has the same semantics as a NoteOff 🙈.
The recovery algorithms in this section only work for MIDI sources that produce NoteOn->NoteOff->NoteOn->NoteOff patterns for a note number. Piano keyboards and drum pads produce these patterns 🎹🥁. MIDI sources that use NoteOn->NoteOn->NoteOff->NoteOff patterns for legato repeated notes, like guitar and wind controllers, need more sophisticated recovery strategies 🎸. Chapter E (not used in this example) supports recovery algorithms for atypical note command patterns (see Appendix A.7 of [RFC4695] for details) 🧠.
The NMP receiver keeps a jrec_chaptern structure (Figure 10) for each voice channel in jrec_stream to code note-related state information 📦. State is kept for each of the 128 note numbers on a channel, using three arrays of length 128: vel[], seq[], and time[]. The arrays are initialized to zero at the start of a stream 🌸.
The vel[n] array element holds info about the most recent note command for note number n 💜. If that command is a NoteOn command, vel[n] holds the velocity data for the command. If that command is a NoteOff command, vel[n] is set to 0 ✨.
The time[n] and extseq[n] array elements code info about the most recently executed NoteOn command 🕰️. The time[n] element holds the execution time of the command, referenced to the local timebase of the receiver. The extseq[n] element holds the RTP extended sequence number of the packet associated with the command. For incoming stream commands, extseq[n] codes the packet of the associated MIDI list. For commands executed to perform loss recovery, extseq[n] codes the packet of the associated recovery journal 📬.
The Chapter N recovery journal bitfield (Figure A.6.1 in [RFC4695]) has two data structures. First, there's a bit array coding recently sent NoteOff commands that are vulnerable to packet loss. Second, there's a note log list coding recently sent NoteOn commands that are vulnerable to packet loss 🗂️.
At the end of a loss event, Chapter N recovery processing begins with the NoteOff bit array 🔍. For each set bit in the array, the receiver checks the corresponding vel[n] element in jrec_chaptern. If vel[n] is non-zero, a NoteOff command or a NoteOff->NoteOn->NoteOff command sequence has been lost. To recover, the receiver immediately executes a NoteOff command for that note number on the channel and sets vel[n] to 0 🛠️.
The receiver then parses the note log list, using the S bit to skip over "safe" logs in the single-packet loss case. For each at-risk note log, the receiver checks the corresponding vel[n] element 👀.
If vel[n] is zero, a NoteOn command or a NoteOn->NoteOff->NoteOn command sequence has been lost. The receiver may execute the most recent lost NoteOn (to play the note) or may take no action (to skip the note), based on criteria we describe at the end of this section 🤔. Whether the note is played or skipped, the receiver updates the vel[n], time[n], and extseq[n] elements as if the NoteOn executed 💅.
If vel[n] is non-zero, the receiver runs several checks to test if a NoteOff->NoteOn sequence has been lost 🧐:
- If
vel[n]does not match the note log velocity, the note log must code a different NoteOn command, so aNoteOff->NoteOnsequence has been lost. - If
extseq[n]is less than the (extended) checkpoint packet sequence numbered coded in the recovery journal header (Figure 8 of [RFC4695]), thevel[n]NoteOn command is not in the checkpoint history, so aNoteOff->NoteOnsequence has been lost. - If the Y bit is set to 1, the NoteOn is musically "simultaneous" with the RTP timestamp of the packet. If
time[n]codes a time value that is clearly not recent, aNoteOff->NoteOnsequence has been lost.
If these tests indicate a lost NoteOff->NoteOn sequence, the receiver immediately executes a NoteOff command 🛑. The receiver then decides if the most graceful action is to play or to skip the lost NoteOn, using the criteria we describe at the end of this section. Whether or not the receiver issues a NoteOn command, the vel[n], time[n], and extseq[n] arrays are updated as if it did 🫶.
Note that the tests above do not catch every lost NoteOff->NoteOn command 🙃. If a fast NoteOn->NoteOff->NoteOn sequence occurs on a note number with identical velocity values for both NoteOn commands, a lost NoteOff->NoteOn does not result in the recovery algorithm generating a NoteOff command. Instead, the first NoteOn keeps sounding, to be terminated by the future NoteOff command. In practice, this (rare) outcome is not musically objectionable 🎶.
The number of tests in this resiliency algorithm may seem excessive, but in some common cases a subset of the tests isn't useful 🤷♀️. For example, MIDI streams that assign the same velocity value to all note events are often produced by inexpensive keyboards. The vel[n] tests are not useful for these streams 💸.
Finally, here's how the receiver decides whether to play or skip a lost NoteOn command 🎯. The note log Y bit is set if the NoteOn is "simultaneous" with the RTP timestamp of the packet holding the note log. If Y is 0, the receiver does not execute a NoteOn command. If Y is 1, and the packet has not arrived late, the receiver immediately executes a NoteOn command for the note number, using the velocity coded in the note log 🔥.
7.3. Chapter C: MIDI Control Change (0xB) 🎛️
Chapter C (Appendix A.3 in [RFC4695]) protects against the loss of MIDI Control Change commands 🛡️. A Control Change command alters the 7-bit value of one of the 128 MIDI controllers 🎚️.
Chapter C offers three tools for protecting a Control Change command: the value tool (for graded controllers such as sliders), the toggle tool (for on/off switches), and the count tool (for momentary-contact switches) 🧰. Senders choose a tool to encode recovery info for a controller, then encode the tool type along with the data in the journal (Figures A.3.2 and A.3.3 in [RFC4695]) 📝.
A few uses of Control Change commands aren't solely protected by Chapter C. The protection of controllers 0 and 32 (Bank Select MSB and Bank Select LSB) is shared between Chapter C and Chapter P (Section 7.4) 🤝.
Chapter M (Appendix A.4 of [RFC4695]) also protects the Control Change command. However, the NMP system doesn't use this chapter, because MPEG 4 Structured Audio [MPEGSA] does not use the controllers protected by this chapter 🙅♀️.
The Chapter C bitfield is a list of controller logs. Each log codes the controller number, the tool type, and the state value for the tool 📋.
The NMP receiver keeps the jrec_chapterc structure (Figure 10) for each voice channel in jrec_stream to code Control Change state info 🧠. The value[] array holds the most recent data values for each controller number. At the start of the stream, value[] is initialized to the default controller data values specified in [MPEGSA] 🌱.
The count[] and toggle[] arrays hold the count tool and toggle tool state values. At the start of a stream, these arrays are initialized to zero. Whenever a Control Command executes, the receiver updates the count[] and toggle[] state values, using the algorithms defined in Appendix A.3 of [RFC4695] ⚙️.
At the end of a loss event, the receiver parses the Chapter C controller log list, using the S bit to skip over "safe" logs in the single-packet loss case. For each at-risk controller number n, the receiver determines the tool type in use (value, toggle, or count) and compares the data in the log to the associated jrec_chapterc array element (value[n], toggle[n], or count[n]). If the data don't match, one or more Control Change commands have been lost 🕵️♀️.
The method the receiver uses to recover depends on the tool type and the controller number. For graded controllers protected by the value tool, the receiver executes a Control Change command using the new data value 🎉.
For the toggle and count tools, the recovery action is more complex 😵💫. For example, the Damper Pedal (Sustain) controller (number 64) is typically used as a sustain pedal for piano-like sounds and is typically coded using the toggle tool. If Damper Pedal (Sustain) Control Change commands are lost, the receiver takes different actions depending on the starting and ending state of the lost sequence, so that "ringing" piano notes get "damped" to silence 🎹🤫.
After recovering from the loss, the receiver updates the value[], toggle[], and count[] arrays to reflect the Chapter C data and the executed commands 📈.
7.4. Chapter P: MIDI Program Change (0xC) 🎀
Chapter P of the recovery journal protects against the loss of MIDI Program Change (0xC) commands 🎀.
The 7-bit data value of the Program Change command selects one of 128 possible timbres for the channel 🎷. To increase the number of possible timbres, Control Change (0xB) commands may be issued before the Program Change command to select a "program bank" 🏦. The Bank Select MSB (number 0) and Bank Select LSB (number 32) controllers specify the 14-bit bank number that subsequent Program Change commands reference 🔢.
The NMP receiver keeps the jrec_chapterp structure (Figure 10) for each voice channel in jrec_stream to code Program Change state info 🧩.
The prognum variable of jrec_chapterp holds the data value for the most recent Program Change command that has arrived on the stream. The bank_msb and bank_lsb variables of jrec_chapterp code the Bank Select MSB and Bank Select LSB controller data values that were in effect when that Program Change command arrived. The prognum_qual, bank_msb_qual, and bank_lsb_qual variables are initialized to 0 and are set to 1 to qualify the associated data values 🏷️.
Chapter P fields code the data value for the most recent Program Change command, and the MSB and LSB bank values in effect for that command 💜.
At the end of a loss event, the receiver checks Chapter P to see if the recovery journal fields match the data stored in jrec_chapterp. If these checks fail, one or more Program Change commands have been lost 🚨.
To recover, the receiver takes the following steps 🪄. If the B bit in Chapter P is set (Figure A.2.1 in [RFC4695]), Control Change bank commands have preceded the Program Change command. The receiver compares the bank data coded by Chapter P with the current bank data for the channel (coded in jrec_channelc).
If the bank data don't agree, the receiver issues Control Change commands to align the stream with Chapter P. The receiver then updates jrec_channelp and jrec_channelc variables to reflect the executed command(s). Finally, the receiver issues a Program Change command that reflects the data in Chapter P, and updates the prognum and qual_prognum fields in jrec_channelp ✅.
Note that this method relies on Chapter P recovery to precede Chapter C recovery during channel journal processing. This ordering ensures that lost Bank Select Control Change commands that occur after a lost Program Change command in a stream are handled correctly 🫶.
8. Security Considerations 🔐
okay so the security stuff for the RTP MIDI payload format lives in the Security Considerations section of [RFC4695]. go read that one, bestie, it's the source of truth and it's not something to skip 🫶
9. IANA Considerations 🧚
same deal for IANA! the IANA considerations for the RTP MIDI payload format are in the IANA Considerations section of [RFC4695] ✨
10. Acknowledgements 💖
this memo was written in conjunction with [RFC4695], so the Acknowledgements section of [RFC4695] also applies here. everyone who helped get this across the finish line is credited there 🌸
11. References 📚
11.1. Normative References
[RFC4695] Lazzaro, J. and J. Wawrzynek, "RTP Payload Format for
MIDI", RFC 4695, November 2006.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, July 2003.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551,
July 2003.
[RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session
Description Protocol", RFC 4566, July 2006.
[MIDI] MIDI Manufacturers Association. "The Complete MIDI 1.0
Detailed Specification", 1996.
[MPEGSA] International Standards Organization. "ISO/IEC 14496
MPEG-4", Part 3 (Audio), Subpart 5 (Structured Audio),
2001.
[RFC3556] Casner, S., "Session Description Protocol (SDP) Bandwidth
Modifiers for RTP Control Protocol (RTCP) Bandwidth", RFC
3556, July 2003.11.2. Informative References
[NMP] Lazzaro, J. and J. Wawrzynek. "A Case for Network Musical
Performance", 11th International Workshop on Network and
Operating Systems Support for Digital Audio and Video
(NOSSDAV 2001) June 25-26, 2001, Port Jefferson, New York.
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E. Schooler,
"SIP: Session Initiation Protocol", RFC 3261, June 2002.
[GRAME] Fober, D., Orlarey, Y. and S. Letz. "Real Time Musical
Events Streaming over Internet", Proceedings of the
International Conference on WEB Delivering of Music 2001,
pages 147-154.
[CCRMA] Chafe C., Wilson S., Leistikow R., Chisholm D., and G.
Scavone. "A simplified approach to high quality music and
sound over IP", COST-G6 Conference on Digital Audio Effects
(DAFx-00), Verona, Italy, December 2000.
[RTPBOOK] Perkins, C. "RTP: Audio and Video for the Internet",
Addison-Wesley, ISBN 0-672-32249-8, 2003.
[STEVENS] Stevens, R. W, Fenner, B., and A. Rudoff. "Unix Network
Programming: The Sockets Networking API", Addison-Wesley,
2003.Authors' Addresses 🎀
the two authors, and where to find them:
John Lazzaro (corresponding author)
UC Berkeley
CS Division
315 Soda Hall
Berkeley CA 94720-1776
EMail: lazzaro@cs.berkeley.edu
John Wawrzynek
UC Berkeley
CS Division
631 Soda Hall
Berkeley CA 94720-1776
EMail: johnw@cs.berkeley.eduFull Copyright Statement 🌈
the copyright belongs to The IETF Trust, Copyright (C) The IETF Trust (2006). this document is subject to the rights, licenses and restrictions in BCP 78, and except as set forth there, the authors keep all their rights 💜
the document and the info in it are provided "AS IS", and the warranty disclaimer is written in full, all caps, here:
THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST,
AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT
THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY
IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR
PURPOSE.Intellectual Property 💅
okay this part is about who owns what. the IETF doesn't take a position on whether any IPR claims that might cover the tech in this document are valid or in scope, and it doesn't claim it has checked for any of them either. if you want the details on how rights work for RFCs, check out BCP 78 and BCP 79 ✨
IPR disclosures sent to the IETF Secretariat, plus any licence assurances, or the results of trying to get a general licence or permission for implementers and users, can all be found in the IETF online IPR repository at http://www.ietf.org/ipr.
and if you're an interested party who knows about any copyrights, patents, patent applications, or other proprietary rights that might cover technology needed to implement this standard, the IETF wants to hear from you. send it to ietf-ipr@ietf.org 🙌
Acknowledgement 🫶
funding for the RFC Editor function is currently provided by the Internet Society. thank you to them for keeping the lights on 🌟
📦 where this page comes from
- ietf: rfc4696 2006-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms