RFC 6787
Media Resource Control Protocol Version 2 (MRCPv2)
Media Resource Control Protocol Version 2 (MRCPv2) 🎀
Okay bestie, this is RFC 6787 💖, and it's all about the Media Resource Control Protocol Version 2 (MRCPv2). It's an IETF Standards Track document, dated November 2012 🧚 D. Burnett from Voxeo and S. Shanmugham from Cisco Systems, Inc. wrote it, and it's published with ISSN 2070-1721. The title block doesn't list any Obsoletes or Updates entries, so this RFC isn't replacing or amending anything else 💅
Abstract ✨
MRCPv2 lets client hosts control media service resources, like speech synthesizers, recognizers, verifiers, and identifiers, that live on servers across the network 🎙️ It is not a "stand-alone" protocol 💀 It leans on the Session Initiation Protocol (SIP) to coordinate clients and servers and manage sessions between them, and on the Session Description Protocol (SDP) to describe, discover, and exchange capabilities 🔥
SIP and SDP also set up the media sessions and their parameters between the media source or sink and the media server. Once that's done, the MRCPv2 exchange runs over the control session, and that's how the client gets to control the media processing resources on the speech resource server 👑
Status of This Memo 📜
This is an Internet Standards Track document 🌸
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). More info on Internet Standards is in Section 2 of RFC 5741 🦋
Info about the current status of this document, any errata, and how to send feedback lives at http://www.rfc-editor.org/info/rfc6787 ✨
Copyright Notice 🌈
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved 💖
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document 🙈 Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 🍓
This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The people controlling the copyright in some of that material may not have granted the IETF Trust the right to allow modifications outside the IETF Standards Process. Without an adequate license from those copyright holders, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English 🔥
Table of Contents 🦋
The table of contents is coming up in the next part, so stay tuned 💜🧚
Bestie, I'm so sorry 😭 the text you sent is the same Table of Contents we already covered in part 1, so there's no new content for part 2 yet 🎀
Could you paste the actual body of the RFC starting from 1. Introduction (the text after the TOC)? Once I have it, I'll keep every section number, every BCP 14 keyword, and every code block exactly as written ✨
okay so the table of contents keeps going, bestie 🎀 we're in the Recorder section now, then Speaker Verification, Security, IANA, Examples, and the ABNF and XML schemas at the end 🧚
10.2. Recorder Methods . . . . . . . . . . . . . . . . . . . . 130
10.3. Recorder Events . . . . . . . . . . . . . . . . . . . . 130
10.4. Recorder Header Fields . . . . . . . . . . . . . . . . . 130
10.4.1. Sensitivity-Level . . . . . . . . . . . . . . . . . 130
10.4.2. No-Input-Timeout . . . . . . . . . . . . . . . . . . 131
10.4.3. Completion-Cause . . . . . . . . . . . . . . . . . . 131
10.4.4. Completion-Reason . . . . . . . . . . . . . . . . . 132
10.4.5. Failed-URI . . . . . . . . . . . . . . . . . . . . . 132
10.4.6. Failed-URI-Cause . . . . . . . . . . . . . . . . . . 132
10.4.7. Record-URI . . . . . . . . . . . . . . . . . . . . . 132
10.4.8. Media-Type . . . . . . . . . . . . . . . . . . . . . 133
10.4.9. Max-Time . . . . . . . . . . . . . . . . . . . . . . 133
10.4.10. Trim-Length . . . . . . . . . . . . . . . . . . . . 134
10.4.11. Final-Silence . . . . . . . . . . . . . . . . . . . 134
10.4.12. Capture-On-Speech . . . . . . . . . . . . . . . . . 134
10.4.13. Ver-Buffer-Utterance . . . . . . . . . . . . . . . . 134
10.4.14. Start-Input-Timers . . . . . . . . . . . . . . . . . 135
10.4.15. New-Audio-Channel . . . . . . . . . . . . . . . . . 135
10.5. Recorder Message Body . . . . . . . . . . . . . . . . . 135
10.6. RECORD . . . . . . . . . . . . . . . . . . . . . . . . . 135
10.7. STOP . . . . . . . . . . . . . . . . . . . . . . . . . . 136
10.8. RECORD-COMPLETE . . . . . . . . . . . . . . . . . . . . 137
10.9. START-INPUT-TIMERS . . . . . . . . . . . . . . . . . . . 138
10.10. START-OF-INPUT . . . . . . . . . . . . . . . . . . . . . 138
11. Speaker Verification and Identification . . . . . . . . . . . 139
11.1. Speaker Verification State Machine . . . . . . . . . . . 140
11.2. Speaker Verification Methods . . . . . . . . . . . . . . 142
11.3. Verification Events . . . . . . . . . . . . . . . . . . 144
11.4. Verification Header Fields . . . . . . . . . . . . . . . 144
11.4.1. Repository-URI . . . . . . . . . . . . . . . . . . . 144
11.4.2. Voiceprint-Identifier . . . . . . . . . . . . . . . 145
11.4.3. Verification-Mode . . . . . . . . . . . . . . . . . 145
11.4.4. Adapt-Model . . . . . . . . . . . . . . . . . . . . 146
11.4.5. Abort-Model . . . . . . . . . . . . . . . . . . . . 146
11.4.6. Min-Verification-Score . . . . . . . . . . . . . . . 147
11.4.7. Num-Min-Verification-Phrases . . . . . . . . . . . . 147
11.4.8. Num-Max-Verification-Phrases . . . . . . . . . . . . 147
11.4.9. No-Input-Timeout . . . . . . . . . . . . . . . . . . 148
11.4.10. Save-Waveform . . . . . . . . . . . . . . . . . . . 148
11.4.11. Media-Type . . . . . . . . . . . . . . . . . . . . . 148
11.4.12. Waveform-URI . . . . . . . . . . . . . . . . . . . . 148
11.4.13. Voiceprint-Exists . . . . . . . . . . . . . . . . . 149
11.4.14. Ver-Buffer-Utterance . . . . . . . . . . . . . . . . 149
11.4.15. Input-Waveform-URI . . . . . . . . . . . . . . . . . 149
11.4.16. Completion-Cause . . . . . . . . . . . . . . . . . . 150
11.4.17. Completion-Reason . . . . . . . . . . . . . . . . . 151
11.4.18. Speech-Complete-Timeout . . . . . . . . . . . . . . 151
11.4.19. New-Audio-Channel . . . . . . . . . . . . . . . . . 152
11.4.20. Abort-Verification . . . . . . . . . . . . . . . . . 152
11.4.21. Start-Input-Timers . . . . . . . . . . . . . . . . . 152
11.5. Verification Message Body . . . . . . . . . . . . . . . 152
11.5.1. Verification Result Data . . . . . . . . . . . . . . 152
11.5.2. Verification Result Elements . . . . . . . . . . . . 153
11.6. START-SESSION . . . . . . . . . . . . . . . . . . . . . 157
11.7. END-SESSION . . . . . . . . . . . . . . . . . . . . . . 158
11.8. QUERY-VOICEPRINT . . . . . . . . . . . . . . . . . . . . 159
11.9. DELETE-VOICEPRINT . . . . . . . . . . . . . . . . . . . 160
11.10. VERIFY . . . . . . . . . . . . . . . . . . . . . . . . . 160
11.11. VERIFY-FROM-BUFFER . . . . . . . . . . . . . . . . . . . 160
11.12. VERIFY-ROLLBACK . . . . . . . . . . . . . . . . . . . . 164
11.13. STOP . . . . . . . . . . . . . . . . . . . . . . . . . . 164
11.14. START-INPUT-TIMERS . . . . . . . . . . . . . . . . . . . 165
11.15. VERIFICATION-COMPLETE . . . . . . . . . . . . . . . . . 165
11.16. START-OF-INPUT . . . . . . . . . . . . . . . . . . . . . 166
11.17. CLEAR-BUFFER . . . . . . . . . . . . . . . . . . . . . . 166
11.18. GET-INTERMEDIATE-RESULT . . . . . . . . . . . . . . . . 167
12. Security Considerations . . . . . . . . . . . . . . . . . . . 168
12.1. Rendezvous and Session Establishment . . . . . . . . . . 168
12.2. Control Channel Protection . . . . . . . . . . . . . . . 168
12.3. Media Session Protection . . . . . . . . . . . . . . . . 169
12.4. Indirect Content Access . . . . . . . . . . . . . . . . 169
12.5. Protection of Stored Media . . . . . . . . . . . . . . . 170
12.6. DTMF and Recognition Buffers . . . . . . . . . . . . . . 171
12.7. Client-Set Server Parameters . . . . . . . . . . . . . . 171
12.8. DELETE-VOICEPRINT and Authorization . . . . . . . . . . 171
13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 171
13.1. New Registries . . . . . . . . . . . . . . . . . . . . . 171
13.1.1. MRCPv2 Resource Types . . . . . . . . . . . . . . . 171
13.1.2. MRCPv2 Methods and Events . . . . . . . . . . . . . 172
13.1.3. MRCPv2 Header Fields . . . . . . . . . . . . . . . . 173
13.1.4. MRCPv2 Status Codes . . . . . . . . . . . . . . . . 176
13.1.5. Grammar Reference List Parameters . . . . . . . . . 176
13.1.6. MRCPv2 Vendor-Specific Parameters . . . . . . . . . 176
13.2. NLSML-Related Registrations . . . . . . . . . . . . . . 177
13.2.1. 'application/nlsml+xml' Media Type Registration . . 177
13.3. NLSML XML Schema Registration . . . . . . . . . . . . . 178
13.4. MRCPv2 XML Namespace Registration . . . . . . . . . . . 178
13.5. Text Media Type Registrations . . . . . . . . . . . . . 178
13.5.1. text/grammar-ref-list . . . . . . . . . . . . . . . 178
13.6. 'session' URI Scheme Registration . . . . . . . . . . . 180
13.7. SDP Parameter Registrations . . . . . . . . . . . . . . 181
13.7.1. Sub-Registry "proto" . . . . . . . . . . . . . . . . 181
13.7.2. Sub-Registry "att-field (media-level)" . . . . . . . 182
14. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 183
14.1. Message Flow . . . . . . . . . . . . . . . . . . . . . . 183
14.2. Recognition Result Examples . . . . . . . . . . . . . . 192
14.2.1. Simple ASR Ambiguity . . . . . . . . . . . . . . . . 192
14.2.2. Mixed Initiative . . . . . . . . . . . . . . . . . . 192
14.2.3. DTMF Input . . . . . . . . . . . . . . . . . . . . . 193
14.2.4. Interpreting Meta-Dialog and Meta-Task Utterances . 194
14.2.5. Anaphora and Deixis . . . . . . . . . . . . . . . . 195
14.2.6. Distinguishing Individual Items from Sets with
One Member . . . . . . . . . . . . . . . . . . . . . 195
14.2.7. Extensibility . . . . . . . . . . . . . . . . . . . 196
15. ABNF Normative Definition . . . . . . . . . . . . . . . . . . 196
16. XML Schemas . . . . . . . . . . . . . . . . . . . . . . . . . 211
16.1. NLSML Schema Definition . . . . . . . . . . . . . . . . 211
16.2. Enrollment Results Schema Definition . . . . . . . . . . 213
16.3. Verification Results Schema Definition . . . . . . . . . 214
17. References . . . . . . . . . . . . . . . . . . . . . . . . . 218
17.1. Normative References . . . . . . . . . . . . . . . . . . 218
17.2. Informative References . . . . . . . . . . . . . . . . . 220
Appendix A. Contributors . . . . . . . . . . . . . . . . . . . . 223
Appendix B. Acknowledgements . . . . . . . . . . . . . . . . . . 2231. Introduction ✨
okay so MRCPv2 is the protocol that lets a client device boss around media processing resources sitting on the network 💅 those resources can be speech recognition engines, speech synthesis engines, speaker verification engines, and speaker identification engines 🎀 the big payoff is that you can build a distributed Interactive Voice Response platform using VoiceXML [W3C.REC-voicexml20-20040316] browsers or other client apps, while the heavy speech processing lives on specialized speech processing servers in the back end. it's giving separation of concerns, honestly 🧚
MRCPv2 is built on the earlier Media Resource Control Protocol (MRCP) [RFC4463], which was developed jointly by Cisco Systems, Inc., Nuance Communications, and Speechworks, Inc. 🌸 some of the method names look the same, but the way those methods get communicated is totally different. there are also more resources and more methods per resource now, so it's a whole glow-up 🦋 the first version of MRCP was basically just used as input while this protocol was being developed. there's no expectation that an MRCPv2 client will work with an MRCPv1 server, or the other way around. there's also no migration plan or gateway definition between the two, so don't go looking for one 🙈
The protocol requirements of Speech Services Control (SPEECHSC) [RFC4313] say the solution has to be able to reach a media processing server, set up communication channels to the media resources, and send and receive control messages and media streams to and from the server. the Session Initiation Protocol (SIP) [RFC3261] checks all of those boxes 👑
The proprietary version of MRCP ran over the Real Time Streaming Protocol (RTSP) [RFC2326]. ngl, that's where the drama started 💀 when work on MRCPv2 began, the consensus was that this use of RTSP would break the RTSP protocol or cause backward-compatibility problems, and Section 3.2 of [RFC4313] forbids that. so that's the reason MRCPv2 does not run over RTSP 🙅♀️
MRCPv2 leverages these capabilities by building on SIP and the Session Description Protocol (SDP) [RFC4566]. it uses SIP to set up and tear down media and control sessions with the server 💖 the client can also use a SIP re-INVITE method (an INVITE dialog sent within an existing SIP session) to change the characteristics of those media and control sessions, and the SIP dialog between the client and server stays intact while it does that ✨ SDP is used to describe the parameters of the media sessions associated with that dialog. SIP support is mandatory as the session establishment protocol to keep everyone interoperable 🫶 other protocols can be used for session establishment by prior agreement, but this document only describes the use of SIP and SDP.
MRCPv2 uses SIP and SDP to create the speech client/server dialog and set up the media channels to the server 🦋 it also uses SIP and SDP to establish MRCPv2 control sessions between the client and the server, one for each media processing resource needed for that dialog. the MRCPv2 protocol exchange between the client and the media resource rides on that control session. here's the important bit: MRCPv2 exchanges do not change the state of the SIP dialog, the media sessions, or other parameters of the dialog that was started via SIP. instead, they control and affect the state of the media processing resource tied to the MRCPv2 session(s) 🎀
MRCPv2 defines the messages used to control the different media processing resources, plus the state machines that guide how those resources operate 💅 it also describes how these messages are carried over a transport-layer protocol such as the Transmission Control Protocol (TCP) [RFC0793] or the Transport Layer Security (TLS) Protocol [RFC5246]. (Note: the Stream Control Transmission Protocol (SCTP) [RFC4960] is a viable transport for MRCPv2 as well, but the mapping onto SCTP is not described in this specification.) 🌈
2. Document Conventions ✨
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 📖
Since so many of the definitions and syntax are identical to those for the Hypertext Transfer Protocol -- HTTP/1.1 [RFC2616], this specification points to the section where they're defined rather than copying them. for brevity, [HX.Y] means Section X.Y of RFC 2616. no copy-paste drama here 🙈
All the mechanisms in this document are described in both prose and augmented Backus-Naur form (ABNF [RFC5234]). 🧚
The complete message format in ABNF form is provided in Section 15, and it's the normative format definition. productions may be duplicated within the main body of the document for reading convenience, but if a production in the body of the text conflicts with one in the normative definition, the latter rules. so Section 15 wins, periodt 👑
2.1. Definitions
- Media Resource: An entity on the speech processing server that can be controlled through MRCPv2. 🎤
- MRCP Server: Aggregate of one or more "Media Resource" entities on a server, exposed through MRCPv2. Often, 'server' in this document refers to an MRCP server. 🖥️
- MRCP Client: An entity controlling one or more Media Resources through MRCPv2 ("Client" for short). 💖
- DTMF: Dual-Tone Multi-Frequency; a method of transmitting key presses in-band, either as actual tones (Q.23 [Q.23]) or as named tone events (RFC 4733 [RFC4733]). 🌸
- Endpointing: The process of automatically detecting the beginning and end of speech in an audio stream. This is critical both for speech recognition and for automated recording as one would find in voice mail systems. 🎙️
- Hotword Mode: A mode of speech recognition where a stream of utterances is evaluated for match against a small set of command words. This is generally employed either to trigger some action or to control the subsequent grammar to be used for further recognition. 🔥
2.2. State-Machine Diagrams
the state-machine diagrams in this document don't show every possible method call 🙈 instead, they reflect the state of the resource based on the methods that have moved to IN-PROGRESS or COMPLETE states (see Section 5.3). PENDING requests basically haven't affected the resource yet, since they're just chilling in the queue waiting to be processed, so they aren't reflected in the state-machine diagrams 🦋
2.3. URI Schemes
This document defines many protocol headers that contain URIs (Uniform Resource Identifiers [RFC3986]) or lists of URIs for referencing media 🔗 the entire document, including the Security Considerations section (Section 12), assumes that HTTP or HTTP over TLS (HTTPS) [RFC2818] will be used as the URI addressing scheme unless otherwise stated. however, implementations MAY support other schemes (such as 'file'), provided they have addressed any security considerations described in this document and any others particular to the specific scheme. for example, implementations where the client and server both reside on the same physical hardware, and the file system is secured by traditional user-level file access controls, could be reasonable candidates for supporting the 'file' scheme. ⚠️
3. Architecture ✨
A system using MRCPv2 has a client that needs to generate and/or consume media streams, plus a media resource server that has the resources or "engines" to process those streams as input or generate them as output 🏗️ the client uses SIP and SDP to establish an MRCPv2 control channel with the server so it can use the server's media processing resources. MRCPv2 servers are addressed using SIP URIs 💅
SIP uses SDP with the offer/answer model described in RFC 3264 [RFC3264] to set up the MRCPv2 control channels and describe their characteristics. a separate MRCPv2 session is needed to control each of the media processing resources associated with the SIP dialog between the client and server 🎀 within a SIP dialog, the individual resource control channels for the different resources get added or removed through SDP offer/answer carried in a SIP re-INVITE transaction.
the server, through the SDP exchange, gives the client a difficult-to-guess, unambiguous channel identifier and a TCP port number (see Section 4.2) 🔐 the client MAY then open a new TCP connection with the server on this port number. multiple MRCPv2 channels can share a TCP connection between the client and the server 🧚 all MRCPv2 messages exchanged between the client and the server carry the specified channel identifier, and the server MUST ensure that identifier is unambiguous among all MRCPv2 control channels active on that server. the client uses this channel identifier to indicate the media processing resource associated with that channel. for information on message framing, see Section 5. 📦
SIP also establishes the media sessions between the client (or other source/sink of media) and the MRCPv2 server using SDP "m=" lines 🎶
One or more media processing resources may share a media session under a SIP session, or each media processing resource may have its own media session. your call, bestie 👑
The following diagram shows the general architecture of a system that uses MRCPv2. to keep things simple, only a few resources are shown 🌈
MRCPv2 client MRCPv2 Media Resource Server
|--------------------| |------------------------------------|
||------------------|| ||----------------------------------||
|| Application Layer|| ||Synthesis|Recognition|Verification||
||------------------|| || Engine | Engine | Engine ||
||Media Resource API|| || || | || | || ||
||------------------|| ||Synthesis|Recognizer | Verifier ||
|| SIP | MRCPv2 || ||Resource | Resource | Resource ||
||Stack | || || Media Resource Management ||
|| | || ||----------------------------------||
||------------------|| || SIP | MRCPv2 ||
|| TCP/IP Stack ||---MRCPv2---|| Stack | ||
|| || ||----------------------------------||
||------------------||----SIP-----|| TCP/IP Stack ||
|--------------------| || ||
| ||----------------------------------||
SIP |------------------------------------|
| /
|-------------------| RTP
| | /
| Media Source/Sink |------------/
| |
|-------------------|
Figure 1: Architectural Diagram3.1. MRCPv2 Media Resource Types 🎀
Okay bestie, an MRCPv2 server can offer one or more of these media processing resources to its clients 💖
Basic Synthesizer 🗣️ is a speech synthesizer with very limited capabilities. It makes its media stream only from concatenated audio clips. The speech data is written in a limited subset of Speech Synthesis Markup Language (SSML) [W3C.REC-speech-synthesis-20040907] elements. A basic synthesizer MUST support the SSML tags <speak>, <audio>, <say-as>, and <mark> 🧚
Speech Synthesizer 👑 is the full-capability one. It renders speech from text, and it MUST have full SSML [W3C.REC-speech-synthesis-20040907] support. Main character energy, no cap ✨
Recorder 🎙️ records audio and gives you a URI pointing to the recording. A recorder MUST provide endpointing so it can suppress silence at the beginning and end of a recording. It MAY also suppress silence in the middle. If it does that middle suppression, the recorder MUST keep timing metadata showing the actual timestamps of the recorded media. Iconic of it to keep receipts 🙈
DTMF Recognizer 📞 extracts and interprets Dual-Tone Multi-Frequency (DTMF) [Q.23] digits in a media stream, then matches them against a supplied digit grammar. It could also do semantic interpretation using the semantic tags in the grammar 🦋
Speech Recognizer 💜 is the full speech recognition resource. It takes a media stream of audio and interprets it into recognition results. It also has a natural language semantic interpreter that post-processes the recognized data using the semantic data in the grammar, then hands back semantic results along with the recognized input. The recognizer MAY also support enrolled grammars, which lets the client enroll and create brand new personal grammars for future recognition operations. Okay that's giving personalization 😭
Speaker Verifier 🔐 checks whether a claimed identity is authentic by matching a media stream of spoken input against a pre-existing voiceprint. This can also involve matching the caller's voice against more than one voiceprint, which is called multi-verification or speaker identification 🌸
3.2. Server and Resource Addressing
The MRCPv2 server is a generic SIP server, so it gets addressed with a SIP URI (RFC 3261 [RFC3261]) 📮
For example:
sip:mrcpv2@example.net or
sips:mrcpv2@example.net4. MRCPv2 Basics ✨
MRCPv2 needs a connection-oriented transport-layer protocol, like TCP, to guarantee reliable sequencing and delivery of MRCPv2 control messages between the client and the server 🎀 To meet the security requirements listed in the SPEECHSC requirements [RFC4313], clients and servers MUST implement TLS too 🔒 One or more connections between the client and the server can be shared among different MRCPv2 channels to the server. Each individual message carries a channel identifier so it can tell messages on different channels apart 💅
MRCPv2 encoding is text based, but it has mechanisms to carry embedded binary data. This means arbitrary data like recognition grammars, recognition results, synthesizer speech markup, and so on can ride along inside MRCPv2 messages 🦋 For information on message framing, see Section 5 📖
4.1. Connecting to the Server
MRCPv2 uses SIP together with SDP as its session establishment and management protocol 🌈 The client reaches an MRCPv2 server with the usual INVITE and other SIP requests for establishing, maintaining, and terminating SIP dialogs. The SDP offer/answer exchange model over SIP is used to set up a resource control channel for each resource. That same SDP offer/answer exchange also sets up the media sessions between the server and the source or sink of audio ✨
4.2. Managing Resource Control Channels
The client needs its own separate MRCPv2 resource control channel to control each media processing resource under the SIP dialog 💖 A unique channel identifier string identifies these channels. It's a difficult-to-guess, unambiguous string, followed by an "@", then a string token naming the type of resource. The server generates the channel identifier and MUST make sure it does not clash with the identifier of any other MRCP channel that server has currently allocated 🫶 MRCPv2 defines these IANA-registered types of media processing resources. More resource types, along with their methods/events and state machines, can be added later as described in Section 13 🌸
+---------------+----------------------+--------------+
| Resource Type | Resource Description | Described in |
+---------------+----------------------+--------------+
| speechrecog | Speech Recognizer | Section 9 |
| dtmfrecog | DTMF Recognizer | Section 9 |
| speechsynth | Speech Synthesizer | Section 8 |
| basicsynth | Basic Synthesizer | Section 8 |
| speakverify | Speaker Verification | Section 11 |
| recorder | Speech Recorder | Section 10 |
+---------------+----------------------+--------------+
Table 1: Resource TypesThe SIP INVITE or re-INVITE transaction and the SDP offer/answer exchange it carries include "m=" lines describing the resource control channel to be allocated. There MUST be exactly one SDP "m=" line for each MRCPv2 resource used in the session 🍓 That "m=" line MUST have a media type field of "application" and a transport type field of either "TCP/MRCPv2" or "TCP/TLS/MRCPv2". In the SDP offer from the client, the port number field of the "m=" line MUST contain the "discard" port of the transport protocol (port 9 for TCP). In the SDP answer, it MUST contain the TCP listen port on the server. The client can then either set up a TCP or TLS connection to that server port, or share an already established connection to that port. Since MRCPv2 lets multiple sessions share the same TCP connection, multiple "m=" lines in a single SDP document MAY share the same port field value. Even so, MRCPv2 servers MUST NOT assume any relationship between resources that use the same port, other than sharing the communication channel 🙅♀️
MRCPv2 resources don't use the port or format field of the "m=" line to tell themselves apart from other resources on the same channel. The client MUST specify the resource type identifier in the resource attribute attached to the control "m=" line of the SDP offer. The server MUST respond with the full Channel-Identifier (which has the resource type identifier plus a difficult-to-guess, unambiguous string) in the "channel" attribute attached to the control "m=" line of the SDP answer. To stay backwards compatible with conventional SDP usage, the format field of the "m=" line MUST have the arbitrarily chosen value of "1" 💜
When the client wants to add a media processing resource to the session, it issues a new SDP offer, following the procedures of RFC 3264 [RFC3264], in a SIP re-INVITE request. That SDP offer/answer exchange contains one or more additional control "m=" lines for the new resources to be allocated to the session. When the server sees the new "m=" line, it allocates the resources (if they're available) and responds with a matching control "m=" line in the SDP answer carried in the SIP response. If the new resources aren't available, the re-INVITE gets an error message, and any media processing already going on before the re-INVITE keeps running exactly as before 😤 It is not possible to allocate more than one resource of each type. If a client requests more than one resource of any type, the server MUST behave as if the resources of that type beyond the first one are not available 🚫
MRCPv2 clients and servers using TCP as a transport protocol MUST use the procedures in RFC 4145 [RFC4145] to set up the TCP connection, with the considerations described here. Likewise, MRCPv2 clients and servers using TCP/TLS as a transport protocol MUST use the procedures in RFC 4572 [RFC4572] to set up the TLS connection, with the considerations described here. The a=setup attribute, described in RFC 4145 [RFC4145], MUST be "active" in the offer from the client and MUST be "passive" in the answer from the MRCPv2 server. The a=connection attribute MUST have a value of "new" on the very first control "m=" line offer from the client to an MRCPv2 server. Later control "m=" line offers from the client to the MRCP server MAY contain "new" or "existing", depending on whether the client wants a new connection or wants to share an existing one. If the client specifies "new", the server MUST respond with "new". If the client specifies "existing", the server MUST respond. The legal values in that response are "existing" if the server prefers to share an existing connection, or "new" if not. In the latter case, the client MUST initiate a new transport connection 🔗
When the client wants to deallocate a resource from this session, it issues a new SDP offer, following RFC 3264 [RFC3264], where the control "m=" line port MUST be set to 0. That SDP offer is sent in a SIP re-INVITE request, and it deallocates the associated MRCPv2 identifier and resource 🧹 The server MUST NOT close the TCP or TLS connection while it's being shared among multiple MRCP channels. Once all the MRCP channels that might be sharing the connection are released, and/or the associated SIP dialog is terminated, the client or server terminates the connection 💀
When the client wants to tear down the whole session and all of its resources, it MUST issue a SIP BYE request to close the SIP session. That deallocates every control channel and resource allocated under the session 👋
All servers MUST support TLS 🔐 Servers MAY use TCP without TLS in controlled environments (for example, not on the public Internet) where both nodes are inside a protected perimeter, such as when remote nodes outside that perimeter are kept away from the MRCP server. It's up to the client, through the SDP offer, to choose which transport it wants for an MRCPv2 session. Apart from the exceptions above, when using TCP, the "m=" lines MUST conform to RFC4145 [RFC4145], which describes how SDP is used for connection-oriented transport. When using TLS, the SDP "m=" line for the control stream MUST conform to Connection-Oriented Media (COMEDIA) over TLS [RFC4572], which specifies how SDP is used to establish a secure connection-oriented transport over TLS 🌟
4.3. SIP Session Example ✨
okay so this first example is the ultimate proof that SIP is a total main character when it comes to routing 💅 Look at the INVITE to mresources@example.com. That's just a request to a domain's speech server service, and the SIP routing machinery in the domain finds the real server, mresources@server.example.com, which comes back in the 200 OK 🧚 Also, cmid gets defined in Section 4.4, so keep that in mind 👀
This exchange adds a resource control channel for a synthesizer 🎀 Since a synthesizer also generates audio, this interaction creates a receive-only Real-Time Protocol (RTP) [RFC3550] media session so the server has somewhere to send audio 🦋 The SIP dialog with the media source/sink is totally separate from MRCP, so it isn't shown here 🙈
C->S: INVITE sip:mresources@example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf1
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314161 INVITE
Contact:<sip:sarvi@client.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=sarvi 2890844526 2890844526 IN IP4 192.0.2.12
s=-
c=IN IP4 192.0.2.12
t=0 0
m=application 9 TCP/MRCPv2 1
a=setup:active
a=connection:new
a=resource:speechsynth
a=cmid:1
m=audio 49170 RTP/AVP 0
a=rtpmap:0 pcmu/8000
a=recvonly
a=mid:1
S->C: SIP/2.0 200 OK
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf1;received=192.0.32.10
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314161 INVITE
Contact:<sip:mresources@server.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=- 2890842808 2890842808 IN IP4 192.0.2.11
s=-
c=IN IP4 192.0.2.11
t=0 0
m=application 32416 TCP/MRCPv2 1
a=setup:passive
a=connection:new
a=channel:32AECB234338@speechsynth
a=cmid:1
m=audio 48260 RTP/AVP 0
a=rtpmap:0 pcmu/8000
a=sendonly
a=mid:1
C->S: ACK sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf2
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314161 ACK
Content-Length:0Example: Add Synthesizer Control Channel
This next exchange picks up right where the last one left off and gives us an extra resource control channel for a recognizer 🎤 A recognizer needs to receive audio to do its job, so this interaction also flips the audio stream to sendrecv. Now it's a two-way RTP media session, and honestly it's giving full duplex energy 🔥
C->S: INVITE sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf3
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314162 INVITE
Contact:<sip:sarvi@client.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=sarvi 2890844526 2890844527 IN IP4 192.0.2.12
s=-
c=IN IP4 192.0.2.12
t=0 0
m=application 9 TCP/MRCPv2 1
a=setup:active
a=connection:existing
a=resource:speechsynth
a=cmid:1
m=audio 49170 RTP/AVP 0 96
a=rtpmap:0 pcmu/8000
a=rtpmap:96 telephone-event/8000
a=fmtp:96 0-15
a=sendrecv
a=mid:1
m=application 9 TCP/MRCPv2 1
a=setup:active
a=connection:existing
a=resource:speechrecog
a=cmid:1
S->C: SIP/2.0 200 OK
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf3;received=192.0.32.10
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314162 INVITE
Contact:<sip:mresources@server.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=- 2890842808 2890842809 IN IP4 192.0.2.11
s=-
c=IN IP4 192.0.2.11
t=0 0
m=application 32416 TCP/MRCPv2 1
a=setup:passive
a=connection:existing
a=channel:32AECB234338@speechsynth
a=cmid:1
m=audio 48260 RTP/AVP 0 96
a=rtpmap:0 pcmu/8000
a=rtpmap:96 telephone-event/8000
a=fmtp:96 0-15
a=sendrecv
a=mid:1
m=application 32416 TCP/MRCPv2 1
a=setup:passive
a=connection:existing
a=channel:32AECB234338@speechrecog
a=cmid:1
C->S: ACK sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf4
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314162 ACK
Content-Length:0Example: Add Recognizer
Last but not least, we're letting go of the recognizer channel 🫶 Since the recognizer doesn't need audio anymore, this interaction also drops the RTP media session back down to recvonly. Clean exit, no drama ✨
C->S: INVITE sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf5
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314163 INVITE
Contact:<sip:sarvi@client.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=sarvi 2890844526 2890844528 IN IP4 192.0.2.12
s=-
c=IN IP4 192.0.2.12
t=0 0
m=application 9 TCP/MRCPv2 1
a=resource:speechsynth
a=cmid:1
m=audio 49170 RTP/AVP 0
a=rtpmap:0 pcmu/8000
a=recvonly
a=mid:1
m=application 0 TCP/MRCPv2 1
a=resource:speechrecog
a=cmid:1
S->C: SIP/2.0 200 OK
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf5;received=192.0.32.10
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314163 INVITE
Contact:<sip:mresources@server.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=- 2890842808 2890842810 IN IP4 192.0.2.11
s=-
c=IN IP4 192.0.2.11
t=0 0
m=application 32416 TCP/MRCPv2 1
a=channel:32AECB234338@speechsynth
a=cmid:1
m=audio 48260 RTP/AVP 0
a=rtpmap:0 pcmu/8000
a=sendonly
a=mid:1
m=application 0 TCP/MRCPv2 1
a=channel:32AECB234338@speechrecog
a=cmid:1
C->S: ACK sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf6
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:314163 ACK
Content-Length:0Example: Deallocate Recognizer
4.4. Media Streams and RTP Ports 🎀
okay so MRCPv2 resources either make media streams or eat them 🎧, which means the client or the server has to link each media session to the right resource (or resources) 💅. One media session can serve more than one resource, or each resource can get its own. You can also hook up more than one media session to a single resource 🤔, but ngl that setup isn't really useful for the resources we have right now. For example, a synthesizer and a recognizer can share the same media session (the m=audio line) if it's opened as "sendrecv" 🎀. Or the recognizer can have its own "sendonly" audio session while the synthesizer gets its own "recvonly" one 🦋.
The link between a control channel and its media session is made with a brand-new media-level attribute called "resource channel media identifier", or "cmid" for short ✨. Its values are the values of the "mid" attribute defined in RFC 5888 [RFC5888]. If there's more than one audio "m=" line, then each audio "m=" line MUST have a "mid" attribute 🫶. Each control "m=" line MAY have one or more "cmid" attributes that match the resource control channel to the "mid" attributes of the audio "m=" lines it's associated with. Heads up 💀: if a control "m=" line doesn't have a "cmid" attribute, it won't be associated with any media, so its operations get limited. Like, if it's a recognizer, the RECOGNIZE method needs an associated media to process, but the INTERPRET method doesn't. The formatting of "cmid" is described by this ABNF:
cmid-attribute = "a=cmid:" identification-tag
identification-tag = tokenThis flexible mapping lets a single audio "m=" line serve multiple MRCPv2 control channels, or lets each resource have its own audio "m=" line 🌈. Say the client wants a recognizer and a synthesizer on one two-way audio stream. Then the SDP offer has two control "m=" lines and a single audio "m=" line with the attribute "sendrecv", and each control "m=" line has a "cmid" attribute whose value matches the "mid" of that audio line 😭. But if the client wants the recognizer and the synthesizer each on their own separate audio stream, the SDP offer carries two control "m=" lines (one for the recognizer, one for the synthesizer) and two audio "m=" lines (one "sendonly" and one "recvonly"). The recognizer's control "m=" line has a "cmid" that matches the "mid" of the "sendonly" audio "m=" line, and the synthesizer's control "m=" line has a "cmid" that matches the "mid" of the "recvonly" one 🍓.
When a server receives media (like audio) on a media session that's linked to more than one media processing resource, it's the server's job to receive the media and fork it to the resources that need to consume it 🧚. If multiple resources in an MRCPv2 session are generating audio (or other media) for a single associated media session, the server is responsible for either multiplexing those streams onto the single RTP session, or using an embedded RTP mixer (see RFC 3550 [RFC3550]) to combine them into one 🔥. With multiplexing, the media stream contains RTP packets from different sources, so the packets have different Synchronization Source Identifiers (SSRCs) 👑. With mixing, the RTP packets contain multiple Contributing Source Identifiers (CSRCs) for the original streams before the mixer combined them ✨. If an MRCPv2 server implementation neither multiplexes nor mixes, it MUST disallow the client from associating multiple such resources with a single audio stream, by rejecting the SDP offer with a SIP 488 "Not Acceptable" error 🙅♀️. There's a large installed base that returns a SIP 501 "Not Implemented" error in this case instead. To keep things interoperable with that installed base, new implementations SHOULD treat a 501 in this context as a 488 when it comes from an element known to be a legacy implementation 🫶.
4.5. MRCPv2 Message Transport 🦋
the MRCPv2 messages in this document get carried over a TCP or TLS connection between the client and the server 💖. The steps for setting up that transport connection and the resource control channel are covered in Sections 4.1 and 4.2. Multiple resource control channels between a client and a server that belong to different SIP dialogs can share one or more TLS or TCP connections between them, and the server and client MUST support this mode of operation 🤝. Clients and servers MUST use the MRCPv2 channel identifier, which goes in the Channel-Identifier header field of each MRCPv2 message, to tell apart messages from different resource channels (see Section 6.2.1 for details) 🎯. All MRCPv2 servers MUST support TLS 🔐. Servers MAY use TCP without TLS in controlled environments (for example, not on the public Internet) where both nodes sit inside a protected perimeter, like when you want to stop remote nodes outside that perimeter from reaching the MRCP server 🌸. It's up to the client to pick which transport mode it wants for an MRCPv2 session 💅.
Most examples from here on show only the MRCPv2 messages and skip the SIP messages that might have been used to set up the MRCPv2 control channel 🙈.
4.6. MRCPv2 Session Termination 💀
If an MRCP client notices that the underlying connection has closed for one of its MRCP channels, and it hasn't previously sent a re-INVITE to close that channel, it MUST send a BYE to close down the SIP dialog and all of its other MRCP channels 📞. If an MRCP server notices that the underlying connection has closed for one of its MRCP channels, and it hasn't previously received and accepted a re-INVITE closing that channel, then it MUST send a BYE to close down the SIP dialog and all other MRCP channels 😭.
5. MRCPv2 Specification 🌈
Except where otherwise indicated, MRCPv2 messages are Unicode encoded in UTF-8 (RFC 3629 [RFC3629]) so that lots of different languages can be represented 🌍. DEFINE-GRAMMAR (Section 9.8) is one such exception, since its body can contain arbitrary XML in arbitrary encodings (as long as the XML specifies them) 🧾. MRCPv2 also lets message bodies use other character sets (for example, ISO 8859-1 [ISO.8859-1.1987]), because in some locales those character sets are already widely used 🌸. The MRCPv2 headers (the first line of an MRCP message) and header field names only use the US-ASCII subset of UTF-8 💜.
Lines end with CRLF (carriage return, then line feed) ✨. Some parameters in the message may also contain binary data or a record that spans multiple lines 🫧. Those fields come with a length value that tells you how many octets immediately follow the parameter 🔢.
5.1. Common Protocol Elements ✨
okay so the MRCPv2 message set has three flavours 💅: requests from the client to the server, responses from the server back to the client, and asynchronous events from the server to the client. They're all built the same way. There's a start-line, one or more header fields, an empty line (a line with nothing before the CRLF) that says "headers are done," and then an optional message body 🎀
generic-message = start-line
message-header
CRLF
[ message-body ]
message-body = *OCTET
start-line = request-line / response-line / event-line
message-header = 1*(generic-header / resource-header / generic-field)
resource-header = synthesizer-header
/ recognizer-header
/ recorder-header
/ verifier-headerThe message-body holds resource-specific and message-specific data 🌸 The actual media types used to carry that data are defined in the sections for each individual message. Generic header fields are described in Section 6.2 🫶
If a message contains a message body, the message MUST contain content-headers indicating the media type and encoding of the data in the message body. no skipping this one bestie 😭
Request, response and event messages (we'll get to them in the next sections) include the version of MRCP they conform to. Version compatibility rules follow [H3.1] when it comes to version ordering, compliance requirements, and upgrading version numbers. The version info is written as "MRCP" (instead of "HTTP" in [H3.1]) or "MRCP/2.0" (instead of "HTTP/1.1" in [H3.1]). To be compliant with this spec, clients and servers sending MRCPv2 messages MUST indicate an mrcp-version of "MRCP/2.0". ABNF productions using mrcp-version can be found in Sections 5.2, 5.3, and 5.5 🦋
mrcp-version = "MRCP" "/" 1*2DIGIT "." 1*2DIGITThe message-length field gives the length of the message in octets, including the start-line, and it MUST be the second token from the beginning of the message. This is what makes framing and parsing the message simpler, which is honestly so iconic of it 🧚 The field counts the whole message, including any data encoded into the body. Note that this value MAY be given as a fixed-length integer that is zero-padded (with leading zeros), so you don't get inefficiency when the message-length value would change just because of how long the message-length token itself is 🍓 This value, like all lengths in MRCP, is interpreted as a base-10 number. Leading zeros do NOT mean it's base-8. (They don't mean that, I promise 🙈)
message-length = 1*19DIGITThe following sample MRCP exchange shows proper message-length values. The values for message-length have been removed from all the other examples in the spec and replaced by '...' so there's less confusion if there are minor message-length computation errors in those examples 💜
C->S: MRCP/2.0 877 INTERPRET 543266
Channel-Identifier:32AECB23433801@speechrecog
Interpret-Text:may I speak to Andre Roy
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:661
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0" root="request">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
</grammar>
S->C: MRCP/2.0 82 543266 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
S->C: MRCP/2.0 634 INTERPRETATION-COMPLETE 543266 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
Content-Type:application/nlsml+xml
Content-Length:441
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="session:request1@form-level.store">
<interpretation>
<instance name="Person">
<ex:Person>
<ex:Name> Andre Roy </ex:Name>
</ex:Person>
</instance>
<input> may I speak to Andre Roy </input>
</interpretation>
</result>All MRCPv2 messages, responses and events MUST carry the Channel-Identifier header field, so the server or client can tell apart messages from different control channels that might share the same transport connection 🔥
In the resource-specific header field descriptions in Sections 8-11, a header field is disallowed on a method (request, response, or event) for that resource unless it's specifically listed as allowed. Also, the phrasing "This header field MAY occur on method X" means the header field is allowed on that method, but it isn't required in every instance of that method 👑
5.2. Request
An MRCPv2 request is a Request line, followed by the message header section and an optional message body with data specific to the request message 🌈
The Request message from a client to the server has, in its first line, the method to be applied, a method tag for that request, and the version of the protocol in use ✨
request-line = mrcp-version SP message-length SP method-name
SP request-id CRLFThe mrcp-version field is the MRCP protocol version the client is using 🦋
The message-length field gives the length of the message, including the start-line.
Details about the mrcp-version and message-length fields are in Section 5.1.
The method-name field identifies the specific request the client is making to the server. Each resource supports a subset of the MRCPv2 methods, and that subset is defined in the section of the spec for the corresponding resource 💅
method-name = generic-method
/ synthesizer-method
/ recognizer-method
/ recorder-method
/ verifier-methodThe request-id field is a unique identifier, representable as an unsigned 32-bit integer, that the client creates and sends to the server. Clients MUST utilize monotonically increasing request-ids for consecutive requests within an MRCP session. The request-id space is linear (not mod(32)), so it does not wrap, and validity can be checked with a simple unsigned comparison operation 🧮 The client may choose any initial value for its first request, but a small integer is RECOMMENDED to avoid exhausting the space in long sessions. If the server receives duplicate or out-of-order requests, the server MUST reject the request with a response code of 410 😭 Since request-ids are scoped to the MRCP session, they are unique across all TCP connections and all resource channels in the session 🫶
The server resource MUST use the client-assigned identifier in its response to the request. If the request doesn't complete synchronously, future asynchronous events associated with this request MUST carry the client-assigned request-id 🎀
request-id = 1*10DIGIT5.3. Response
After receiving and interpreting the request message for a method, the server resource responds with an MRCPv2 response message. The response is a response line, followed by the message header section and an optional message body with data specific to the method 🌸
response-line = mrcp-version SP message-length SP request-id
SP status-code SP request-state CRLFThe mrcp-version field MUST contain the version of the request if it's supported. Otherwise, it MUST contain the highest version of MRCP supported by the server 🍓
The message-length field gives the length of the message, including the start-line.
Details about the mrcp-version and message-length fields are in Section 5.1.
The request-id used in the response MUST match the one sent in the corresponding request message 💜
The status-code field is a 3-digit code representing the success, failure, or other status of the request.
status-code = 3DIGITThe request-state field indicates whether the action started by the Request is PENDING, IN-PROGRESS, or COMPLETE. COMPLETE means the request was processed to completion, and there won't be any more events or other messages from that resource to the client with that request-id 🏁 PENDING means the request has been placed in a queue and will be processed in first-in-first-out order ⏳ IN-PROGRESS means the request is being processed and isn't finished yet 🔄 A PENDING or IN-PROGRESS status tells you that further Event messages may still be delivered with that request-id 🧚
request-state = "COMPLETE"
/ "IN-PROGRESS"
/ "PENDING"5.4. Status Codes ✨
okay so status codes are sorted into three buckets: Success (2xx) 💖, Client Failure (4xx) 🙈, and Server Failure (5xx) 💀. Here's the full tea:
+------------+--------------------------------------------------+
| Code | Meaning |
+------------+--------------------------------------------------+
| 200 | Success |
| 201 | Success with some optional header fields ignored |
+------------+--------------------------------------------------+
Success (2xx)
+--------+----------------------------------------------------------+
| Code | Meaning |
+--------+----------------------------------------------------------+
| 401 | Method not allowed |
| 402 | Method not valid in this state |
| 403 | Unsupported header field |
| 404 | Illegal value for header field. This is the error for a |
| | syntax violation. |
| 405 | Resource not allocated for this session or does not |
| | exist |
| 406 | Mandatory Header Field Missing |
| 407 | Method or Operation Failed (e.g., Grammar compilation |
| | failed in the recognizer. Detailed cause codes might be |
| | available through a resource-specific header.) |
| 408 | Unrecognized or unsupported message entity |
| 409 | Unsupported Header Field Value. This is a value that is |
| | syntactically legal but exceeds the implementation's |
| | capabilities or expectations. |
| 410 | Non-Monotonic or Out-of-order sequence number in request.|
| 411-420| Reserved for future assignment |
+--------+----------------------------------------------------------+
Client Failure (4xx)
+------------+--------------------------------+
| Code | Meaning |
+------------+--------------------------------+
| 501 | Server Internal Error |
| 502 | Protocol Version not supported |
| 503 | Reserved for future assignment |
| 504 | Message too large |
+------------+--------------------------------+
Server Failure (5xx)5.5. Events 🦋
okay this one is so good bestie. sometimes the media resource needs to tell the client that something changed or something happened 👀. this comes up when a request doesn't finish right away and the response comes back as PENDING or IN-PROGRESS. the in-between results and events get sent to the client in an event message from the server. an event message is an event header line, then the message header section, then an optional message body with data specific to that event 💅. the header line carries the request-id of the request it belongs to, plus a status value. the request-state is COMPLETE if the request is done and this was the last event, otherwise it's IN-PROGRESS ✨.
event-line = mrcp-version SP message-length SP event-name
SP request-id SP request-state CRLFthe mrcp-version here is the same one used in the Request/Response line, and it tells you the highest MRCP version running on the server 🌸.
message-length is the total length of the message, start-line included 🧮.
more on mrcp-version and message-length in Section 5.1 🎀.
event-name says what kind of event the media resource generated 🔥. which names are valid depends on the resource, so check the resource-specific section of the doc for the good stuff 🧚.
event-name = synthesizer-event
/ recognizer-event
/ recorder-event
/ verifier-eventthe request-id in an event MUST match the one sent in the request that caused the event. no exceptions, periodt 💅.
request-state tells you whether the Request/Command that caused this event is finished or still going, and whether it's the same one mentioned in Section 5.3. the final event for a request is marked COMPLETE, which means the request is officially done 🎉.
6. MRCPv2 Generic Methods, Headers, and Result Structure 👑
MRCPv2 has a set of methods and header fields that every resource shares 🫶. we're covering those here. the resource-specific methods and header fields live in their own sections of the document, so go find them there 🍓.
6.1. Generic Methods 🌈
MRCPv2 has two generic methods for reading and writing the state tied to a resource ✨.
generic-method = "SET-PARAMS"
/ "GET-PARAMS"here's the breakdown 👇.
6.1.1. SET-PARAMS 🎀
SET-PARAMS goes from client to server, and it's how you set session parameters like voice characteristics and prosody on synthesizers, or recognition timers on recognizers, and so on 💜. if the server accepts and sets every parameter, it MUST return a response status-code of 200. if it decides to ignore some optional header fields that can be safely skipped without messing with how the server works, it MUST return 201 🔥.
if one or more of the header fields you're sending is wrong, then error 403, 404, or 409 MUST be returned, like this 🙈:
o If one or more of the header fields being set has an illegal value, the server MUST reject the request with a 404 Illegal Value for Header Field. o If one or more of the header fields being set is unsupported for the resource, the server MUST reject the request with a 403 Unsupported Header Field, except as described in the next paragraph. o If one or more of the header fields being set has an unsupported value, the server MUST reject the request with a 409 Unsupported Header Field Value, except as described in the next paragraph.
if you get error 404 along with another error, only 404 MUST be returned 💀. if you get both 403 and 409 but no 404, only 403 MUST be returned 💅.
if error 403, 404, or 409 comes back, the response MUST include the bad or unsupported header fields with their values exactly as the client sent them 🫶. and session parameters changed with SET-PARAMS don't override parameters set explicitly on individual requests, or on requests that are IN-PROGRESS ✨.
C->S: MRCP/2.0 ... SET-PARAMS 543256
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:female
Voice-variant:3
S->C: MRCP/2.0 ... 543256 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth6.1.2. GET-PARAMS 🧚
GET-PARAMS also goes from client to server, and it asks the MRCPv2 resource what its current session parameters are, like voice characteristics and prosody on synthesizers, or recognition timers on recognizers 🦋. for every header field the client sends without a value, the server MUST include that header field and its value in the response. if the client doesn't specify any parameter header fields, the server MUST return all the settable parameters and their values in the response's header section, vendor-specific ones included 🌸. heads up though, these wildcard requests can be really processing-intensive, since there can be a LOT of settable parameters depending on the implementation 🥵. so it is RECOMMENDED that the client not use the wildcard GET-PARAMS operation very often 🙏. also, GET-PARAMS only returns values that apply to the whole session, not values with request-level scope. for example, Input-Waveform-URI is a request-level header field, so GET-PARAMS won't return it 🍓.
if every requested header field is supported, the server MUST return a response status-code of 200 ✅. if some of the requested header fields aren't supported for the resource, the server MUST reject the request with a 403 Unsupported Header Field. that response MUST include the unsupported header fields exactly as the client sent them, without values 💅.
C->S: MRCP/2.0 ... GET-PARAMS 543256
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:
Voice-variant:
Vendor-Specific-Parameters:com.example.param1;
com.example.param2
S->C: MRCP/2.0 ... 543256 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:female
Voice-variant:3
Vendor-Specific-Parameters:com.example.param1="Company Name";
com.example.param2="124324234@example.com"6.2. Generic Message Headers ✨
Okay bestie, every MRCPv2 header field follows the same generic format, and that includes the generic-headers in the subsections below and the resource-specific ones coming later 🎀. The format is the one from Section 3.1 of RFC 5322 [RFC5322]. So you get a name, then a colon ":", then the value. Header field names are case-insensitive, so Accept and accept are literally the same thing 🙈. The value MAY be preceded by any amount of LWS (linear white space), though a single SP (space) is the fave 💅. If a header gets too long, you can spread it over several lines by starting each extra line with at least one SP or HT (horizontal tab) 🦋.
generic-field = field-name ":" [ field-value ]
field-name = token
field-value = *LWS field-content *( CRLF 1*LWS field-content)
field-content = <the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string>The field-content doesn't include any leading or trailing LWS, meaning the whitespace before the first non-whitespace character of the field-value or after the last non-whitespace character of it. That extra LWS MAY be removed without changing the semantics of the field value. Any LWS sitting between field-content MAY be swapped for a single SP before the field value gets interpreted or forwarded downstream ✨.
Real talk on ordering: MRCPv2 servers and clients MUST NOT depend on header field order 🙅. It is RECOMMENDED to send general-header fields first, then request-header or response-header fields, and to end with the entity-header fields. BUT even if the order is all over the place, MRCPv2 servers and clients MUST be prepared to process the header fields in any order. The only exception to this rule is when there are multiple header fields with the same name in one message.
Multiple header fields with the same name MAY be present in a message if and only if the entire value for that header field is defined as a comma-separated list [i.e., #(values)] 🍓.
Since vendor-specific parameters may be order-dependent, it MUST be possible to smoosh multiple header fields of the same name into one "name:value" pair without changing the semantics of the message. You do this by appending each later value to the first one, with a comma between them. The order those same-name header fields arrive in is therefore a big deal for how the combined value gets read, so an intermediary MUST NOT change the order of these values when a message is forwarded 👑.
generic-header = channel-identifier
/ accept
/ active-request-id-list
/ proxy-sync-id
/ accept-charset
/ content-type
/ content-id
/ content-base
/ content-encoding
/ content-location
/ content-length
/ fetch-timeout
/ cache-control
/ logging-tag
/ set-cookie
/ vendor-specific6.2.1. Channel-Identifier 🧚
All MRCPv2 requests, responses, and events MUST contain the Channel-Identifier header field. The value is allocated by the server when a control channel is added to the session, and the server tells the client about it through the "a=channel" attribute in the SDP answer. The header field value is made of 2 parts separated by the '@' symbol. The first part is an unambiguous string identifying the MRCPv2 session. The second part is a string token that specifies one of the media processing resource types listed in Section 3.1. The unambiguous string (first part) MUST be difficult to guess, unique among the resource instances managed by the server, and common to all resource channels with that server established through a single SIP dialog 🔐.
channel-identifier = "Channel-Identifier" ":" channel-id CRLF
channel-id = 1*alphanum "@" 1*alphanum6.2.2. Accept 🌈
The Accept header field follows the syntax defined in [H14.1]. The semantics are identical too, with one exception: if no Accept header field is present, the server MUST assume a default value that is specific to the resource type being controlled. That default can be changed for a resource on a session by sending this header field in a SET-PARAMS method. To check what the current default is for a resource in a session, you use a GET-PARAMS method. This header field MAY occur on any request 💖.
6.2.3. Active-Request-Id-List 🦋
In a request, this header field indicates the list of request-ids that the request applies to. This is super useful when there are multiple requests that are PENDING or IN-PROGRESS and the client wants this request to hit one or more of them specifically 🎯.
In a response, this header field returns the list of request-ids that the method modified or affected. There could be one or more requests in a request-state of PENDING or IN-PROGRESS. When a method affecting one or more PENDING or IN-PROGRESS requests gets sent from the client to the server, the response MUST contain the list of request-ids that were affected or modified by that command in its header section 📋.
The Active-Request-Id-List is only used in requests and responses, never in events 🙅.
For example, say a STOP request with no Active-Request-Id-List gets sent to a synthesizer resource that has one or more SPEAK requests in the PENDING or IN-PROGRESS state. Then all SPEAK requests MUST be cancelled, including the one that's IN-PROGRESS 💀. The response to the STOP request contains the request-ids of all the SPEAK requests that were terminated in its Active-Request-Id-List value. After sending the STOP response, the server MUST NOT send any SPEAK-COMPLETE or RECOGNITION-COMPLETE events for the terminated requests 🛑.
active-request-id-list = "Active-Request-Id-List" ":"
request-id *("," request-id) CRLF6.2.4. Proxy-Sync-Id 🔥
When any server resource generates a "barge-in-able" event, it also generates a unique tag. That tag is sent as this header field's value in an event to the client. The client then acts as an intermediary among the server resources and sends a BARGE-IN-OCCURRED method to the synthesizer server resource, using the Proxy-Sync-Id it got from the server resource. When the recognizer and synthesizer resources are part of the same session, they may choose to work together for quicker interaction and response. In that case, the Proxy-Sync-Id helps the resource that receives the event (which goes through the client) figure out whether the event was already handled through a direct interaction between the resources. This header field MAY occur only on events and the BARGE-IN-OCCURRED method. Also, the 'proxy' in the name is only there for historical reasons, so no, there's no proxy server involved 🙃.
proxy-sync-id = "Proxy-Sync-Id" ":" 1*VCHAR CRLF6.2.5. Accept-Charset 🎀
See [H14.2]. This specifies the acceptable character sets for entities returned in the response or events associated with this request. It's handy for picking the character set used in the Natural Language Semantic Markup Language (NLSML) results of a RECOGNITION-COMPLETE event 🧠. This header field is only used on requests.
6.2.6. Content-Type 🌸
See [H14.17]. MRCPv2 supports a restricted set of registered media types for content, covering speech markup, grammar, and recognition results. The content types that apply to each MRCPv2 resource-type are specified in the corresponding section of the document, and they're registered in the MIME Media Types registry maintained by IANA 📚. The multipart content type "multipart/mixed" is supported for communicating several of the contents mentioned above at once. When it's used that way, the body parts MUST NOT contain any MRCPv2-specific header fields. This header field MAY occur on all messages ✨.
content-type = "Content-Type" ":" media-type-value CRLF
media-type-value = type "/" subtype *( ";" parameter )
type = token
subtype = token
parameter = attribute "=" value
attribute = token
value = token / quoted-string6.2.7. Content-ID 🫶
This header field holds an ID or name for the content so it can be referenced later. It follows the specification in RFC 2392 [RFC2392] and is required for content disambiguation in multipart messages. In MRCPv2, whenever the associated content is stored by either the client or the server, it MUST be retrievable using this ID. Content like that can be referenced later in a session by addressing it with the 'session' URI scheme described in Section 13.6. This header field MAY occur on all messages 🔗.
6.2.8. Content-Base 🍓
The Content-Base entity-header MAY be used to specify the base URI for resolving relative URIs within the entity.
content-base = "Content-Base" ":" absoluteURI CRLFNote, however, that the base URI of the contents within the entity-body may be redefined within that entity-body. Multipart media is a classic example, since it can contain multiple entities inside it 🧩. This header field MAY occur on all messages.
6.2.9. Content-Encoding 💜
The Content-Encoding entity-header works as a modifier to the Content-Type. When it's present, its value tells you what additional content encoding has been applied to the entity-body, and therefore what decoding mechanisms must be applied to get the Media Type referenced by the Content-Type header field. Content-Encoding is mainly used to let a document be compressed without losing the identity of its underlying media type 🗜️. Note that the SIP session can be used to determine accepted encodings (see Section 7). This header field MAY occur on all messages.
content-encoding = "Content-Encoding" ":"
*WSP content-coding
*(*WSP "," *WSP content-coding *WSP )
CRLFContent codings are defined in [H3.5]. An example of its use is Content-Encoding:gzip 🎁.
If multiple encodings have been applied to an entity, the content encodings MUST be listed in the order in which they were applied. Order matters, no cap 😭.
6.2.10. Content-Location
Okay so the Content-Location header is bestie's way of saying "hey, this entity lives over HERE too" 🦋 It MAY be used to supply the resource location for the entity enclosed in the message when that entity is accessible from a location separate from the requested resource's URI. Refer to [H14.14]. ✨
content-location = "Content-Location" ":"
( absoluteURI / relativeURI ) CRLFThe Content-Location value is a statement of the location of the resource corresponding to this particular entity at the time of the request. 💅 This header field is provided for optimization purposes only, so it's purely a speed thing 🏎️ The receiver of this header field MAY assume that the entity being sent is identical to what would have been retrieved or might already have been retrieved from the Content-Location URI. 🫶
For example, if the client provided a grammar markup inline, and it had previously retrieved it from a certain URI, that URI can be provided as part of the entity, using the Content-Location header field. 🧚 This lets a resource like the recognizer peek into its cache to see if this grammar was previously retrieved, compiled, and cached. 👑 In this case, it might optimize by using the previously compiled grammar object. It's giving main character energy for caching 😭
If the Content-Location is a relative URI, the relative URI is interpreted relative to the Content-Base URI. 🌸 This header field MAY occur on all messages.
6.2.11. Content-Length
This header field contains the length of the content of the message body (i.e., after the double CRLF following the last header field). 💜 Unlike in HTTP, it MUST be included in all messages that carry content beyond the header section. 🔥 If it is missing, a default value of zero is assumed. Otherwise, it is interpreted according to [H14.13]. 🍓 When a message having no use for a message body contains one, i.e., the Content-Length is non-zero, the receiver MUST ignore the content of the message body. 💀 This header field MAY occur on all messages.
content-length = "Content-Length" ":" 1*19DIGIT CRLF6.2.12. Fetch Timeout
When the recognizer or synthesizer needs to fetch documents or other resources, this header field controls the corresponding URI access properties. ✨ This defines the timeout for content that the server may need to fetch over the network. The value is interpreted to be in milliseconds and ranges from 0 to an implementation-specific maximum value. 🌈 It is RECOMMENDED that servers be cautious about accepting long timeout values. 🙈 The default value for this header field is implementation specific. This header field MAY occur in DEFINE-GRAMMAR, RECOGNIZE, SPEAK, SET-PARAMS, or GET-PARAMS. 🎀
fetch-timeout = "Fetch-Timeout" ":" 1*19DIGIT CRLF6.2.13. Cache-Control
If the server implements content caching, it MUST adhere to the cache correctness rules of HTTP 1.1 [RFC2616] when accessing and caching stored content. 🦋 In particular, the "expires" and "cache-control" header fields of the cached URI or document MUST be honored and take precedence over the Cache-Control defaults set by this header field. The Cache-Control directives are used to define the default caching algorithms on the server for the session or request. The scope of the directive is based on the method it is sent on. ✨ If the directive is sent on a SET-PARAMS method, it applies for all requests for external documents the server makes during that session, unless it is overridden by a Cache-Control header field on an individual request. If the directives are sent on any other requests, they apply only to external document requests the server makes for that request. 💖 An empty Cache-Control header field on the GET-PARAMS method is a request for the server to return the current Cache-Control directives setting on the server. This header field MAY occur only on requests. 🫶
cache-control = "Cache-Control" ":"
[*WSP cache-directive
*( *WSP "," *WSP cache-directive *WSP )]
CRLF
cache-directive = "max-age" "=" delta-seconds
/ "max-stale" [ "=" delta-seconds ]
/ "min-fresh" "=" delta-seconds
delta-seconds = 1*19DIGITHere, delta-seconds is a decimal time value specifying the number of seconds since the instant the message response or data was received by the server. ⭐
The different cache-directive options allow the client to ask the server to override the default cache expiration mechanisms: 💅
max-age Indicates that the client can tolerate the server
using content whose age is no greater than the
specified time in seconds. Unless a "max-stale"
directive is also included, the client is not willing
to accept a response based on stale data.
min-fresh Indicates that the client is willing to accept a
server response with cached data whose expiration is
no less than its current age plus the specified time
in seconds. If the server's cache time-to-live
exceeds the client-supplied min-fresh value, the
server MUST NOT utilize cached content.
max-stale Indicates that the client is willing to allow a server
to utilize cached data that has exceeded its
expiration time. If "max-stale" is assigned a value,
then the client is willing to allow the server to use
cached data that has exceeded its expiration time by
no more than the specified number of seconds. If no
value is assigned to "max-stale", then the client is
willing to allow the server to use stale data of any
age.If the server cache is requested to use stale response/data without validation, it MAY do so only if this does not conflict with any "MUST"-level requirements concerning cache validation (e.g., a "must-revalidate" Cache-Control directive in the HTTP 1.1 specification pertaining to the corresponding URI). 🙈 If both the MRCPv2 Cache-Control directive and the cached entry on the server include "max-age" directives, then the lesser of the two values is used for determining the freshness of the cached entry for that request. 🌈
6.2.14. Logging-Tag
This header field MAY be sent as part of a SET-PARAMS/GET-PARAMS method to set or retrieve the logging tag for logs generated by the server. 🧚 Once set, the value persists until a new value is set or the session ends. The MRCPv2 server MAY provide a mechanism to create subsets of its output logs so that system administrators can examine or extract only the log file portion during which the logging tag was set to a certain value. 🍓
It is RECOMMENDED that clients include in the logging tag information to identify the MRCPv2 client User Agent, so that one can determine which MRCPv2 client request generated a given log message at the server. 🎀 It is also RECOMMENDED that MRCPv2 clients not log personally identifiable information such as credit card numbers and national identification numbers. 🔥 No cap, privacy is everything 🫶
logging-tag = "Logging-Tag" ":" 1*UTFCHAR CRLF6.2.15. Set-Cookie 🍪
okay so, the MRCPv2 server's HTTP client fetches documents on behalf of the MRCPv2 client, which means its cookie store is basically an extension of the client's cookie store 🧁. bestie, that means the two have to be able to sync their cookie jar whenever needed 🫶. so the Set-Cookie entity-header field MAY be included in MRCPv2 requests to update the cookie store on a server, and it can come back in final MRCPv2 responses or events so the client can update its own store 🦋. the cookies on the server stick around for the whole MRCPv2 session and MUST be destroyed when the session ends 💅. and to make sure cookies are supported at all, MRCPv2 clients and servers MUST support the Set-Cookie entity-header field 👑.
ngl the client is the one in charge of which cookies get sent to the server 💖. nobody has to share everything, and it is RECOMMENDED that MRCPv2 clients only communicate the cookies the MRCPv2 server actually needs to process its requests 🌸. main character energy, minimal oversharing ✨.
set-cookie = "Set-Cookie:" cookies CRLF
cookies = cookie *("," *LWS cookie)
cookie = attribute "=" value *(";" cookie-av)
cookie-av = "Comment" "=" value
/ "Domain" "=" value
/ "Max-Age" "=" value
/ "Path" "=" value
/ "Secure"
/ "Version" "=" 1*19DIGIT
/ "Age" "=" delta-seconds
set-cookie = "Set-Cookie:" SP set-cookie-string
set-cookie-string = cookie-pair *( ";" SP cookie-av )
cookie-pair = cookie-name "=" cookie-value
cookie-name = token
cookie-value = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
cookie-octet = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
token = <token, defined in [RFC2616], Section 2.2>
cookie-av = expires-av / max-age-av / domain-av /
path-av / secure-av / httponly-av /
extension-av / age-av
expires-av = "Expires=" sane-cookie-date
sane-cookie-date = <rfc1123-date, defined in [RFC2616], Section 3.3.1>
max-age-av = "Max-Age=" non-zero-digit *DIGIT
non-zero-digit = %x31-39
domain-av = "Domain=" domain-value
domain-value = <subdomain>
path-av = "Path=" path-value
path-value = <any CHAR except CTLs or ";">
secure-av = "Secure"
httponly-av = "HttpOnly"
extension-av = <any CHAR except CTLs or ";">
age-av = "Age=" delta-secondsthe Set-Cookie header field itself is specified in RFC 6265 [RFC6265] 📜. this spec adds an "Age" attribute to show how old the cookie is, and it is OPTIONAL 🕰️. an MRCPv2 client or server MUST work out the age of the cookie using the age calculation rules in the HTTP/1.1 specification [RFC2616] and then append the "Age" attribute 🧮. why? because time may have passed since the client got the cookie from an HTTP server ⏳. instead of the client subtracting the actual age from Max-Age, it passes Max-Age through verbatim and tacks on "Age" 🎀. the cookie stays exactly as it was received, and the age still gets accounted for. it's giving precision 💅.
if the HTTP origin server leaves out the "Domain" or "Path" attributes, the MRCPv2 client or server MUST supply defaults for them, as RFC 6265 says 🌈. fun detail: in that case there's no leading dot on the "Domain" value 🙈. an explicitly specified "Domain" value that came in over HTTP can get a leading dot added, but an MRCPv2 client or server MUST NOT modify the "Domain" value when it arrives over MRCPv2 🚫. hands off, no edits 🫶.
an MRCPv2 client or server MAY combine multiple cookie header fields of the same type into one "field-name:field-value" pair, as described in Section 6.2 🧃.
the Set-Cookie header field MAY be specified in any request that ends up making the server do an HTTP access ✨. when a server gets new cookie info from an HTTP origin server, and the cookie store changes according to RFC 6265, the server MUST send that new cookie info back in the MRCPv2 COMPLETE response or event, whichever one fits, so the client can update its own store 🍓. ate, no crumbs.
the SET-PARAMS request MAY specify the Set-Cookie header field to update the cookie store on a server 🍒. and the GET-PARAMS request MAY be used to return the entire cookie store of "Set-Cookie" type cookies to the client 💜.
6.2.16. Vendor-Specific Parameters 🧚
this set of header fields lets the client set or retrieve vendor-specific parameters 🎀.
vendor-specific = "Vendor-Specific-Parameters" ":"
[vendor-specific-av-pair
*(";" vendor-specific-av-pair)] CRLF
vendor-specific-av-pair = vendor-av-pair-name "="
value
vendor-av-pair-name = 1*UTFCHARheader fields in this form MAY be sent in any method (request) and are used to manage implementation-specific parameters on the server side 🦋. the vendor-av-pair-name follows the reverse Internet Domain Name convention (see Section 13.1.6 for syntax and registration information) 🌍. the value of the vendor attribute comes after the "=" symbol and MAY be quoted 💅. for example:
com.example.companyA.paramxyz=256
com.example.companyA.paramabc=High
com.example.companyB.paramxyz=Lowwhen GET-PARAMS is used to get the current values of these parameters from the server, this header field value MAY contain a semicolon-separated list of implementation-specific attribute names 🌸.
6.3. Generic Result Structure 🏆
okay so the result data from the server for the Recognizer and Verifier resources gets carried as a typed media entity in the MRCPv2 message body of various events 💌. the Natural Language Semantics Markup Language (NLSML), which is an XML markup based on an early draft from the W3C, is the default standard for sending results back to the client 👑. so all servers implementing these resource types MUST support the media type 'application/nlsml+xml' 🫶. the Extensible MultiModal Annotation (EMMA) [W3C.REC-emma-20090210] format can be used to return results too 🦋. you negotiate that at session setup time with SDP (a=resultformat:application/emma+xml) or with SIP (Allow/Accept) ✨. with SIP, for example, if a client wants EMMA results, an MRCPv2 server can route the request to another server that supports EMMA just by inspecting the SIP header fields, instead of having to dig into the SDP 🧚.
MRCPv2 uses this representation to pass content between the clients and servers that make and use the markup 💖. specifically, MRCPv2 uses NSLML to carry recognition, enrollment, and verification results between the matching resource on the MRCPv2 server and the MRCPv2 client 🔥. the full details of this result format are in Section 6.3.1 🍓.
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="http://theYesNoGrammar">
<interpretation>
<instance>
<ex:response>yes</ex:response>
</instance>
<input>OK</input>
</interpretation>
</result>Result Example 🎀
6.3.1. Natural Language Semantics Markup Language 📝
the Natural Language Semantics Markup Language (NLSML) is an XML data structure with elements and attributes built to carry result info from recognizer (including enrollment) and verifier resources 🧠. the normative definition of NLSML is the RelaxNG schema in Section 16.1 📐. note that the elements and attributes of this format are defined in the MRCPv2 namespace 🌈. inside the result structure, they must either have a namespace prefix declared within the result, or be children of an element identified as belonging to the respective namespace 💅. for details on how to use XML Namespaces, see [W3C.REC-xml-names11-20040204], and Section 2 of [W3C.REC-xml-names11-20040204] explains how to declare namespaces and namespace prefixes 🤓.
the root element of NLSML is <result> 👑. the optional child elements are <interpretation>, <enrollment-result>, and <verification-result>, and at least one of them must be present ✨. a single <result> MAY contain any or all of the optional child elements 🫧. details about the <result> and <interpretation> elements, along with their subelements and attributes, are in Section 9.6 🍒. details about the <enrollment-result> element and its subelements are in Section 9.7 🌸. and details about the <verification-result> element and its subelements are in Section 11.5.2 🦋.
7. Resource Discovery 🔎
server resources can be discovered, and their capabilities learned, by clients through standard SIP machinery 📡. the client MAY send a SIP OPTIONS transaction to a server, which basically asks for the server's capabilities 🙋♀️. the server MUST respond to that request with an SDP-encoded description of its capabilities, according to RFC 3264 [RFC3264] 💌. the MRCPv2 capabilities are described by a single "m=" line with the media type "application" and the transport type "TCP/TLS/MRCPv2" or "TCP/MRCPv2" 🎧. there MUST be one "resource" attribute for each media resource the server supports, and its value is the resource type identifier 🏷️.
the SDP description MUST also contain "m=" lines describing the server's audio capabilities and the coders it supports 🎶.
in this example, the client uses the SIP OPTIONS method to query the capabilities of the MRCPv2 server 🧁.
C->S:
OPTIONS sip:mrcp@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf7
Max-Forwards:6
To:<sip:mrcp@example.com>
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:63104 OPTIONS
Contact:<sip:sarvi@client.example.com>
Accept:application/sdp
Content-Length:0
S->C:
SIP/2.0 200 OK
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bf7;received=192.0.32.10
To:<sip:mrcp@example.com>;tag=62784
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:63104 OPTIONS
Contact:<sip:mrcp@server.example.com>
Allow:INVITE, ACK, CANCEL, OPTIONS, BYE
Accept:application/sdp
Accept-Encoding:gzip
Accept-Language:en
Supported:foo
Content-Type:application/sdp
Content-Length:...
v=0
o=sarvi 2890844536 2890842811 IN IP4 192.0.2.12
s=-
i=MRCPv2 server capabilities
c=IN IP4 192.0.2.12/127
t=0 0
m=application 0 TCP/TLS/MRCPv2 1
a=resource:speechsynth
a=resource:speechrecog
a=resource:speakverify
m=audio 0 RTP/AVP 0 3
a=rtpmap:0 PCMU/8000
a=rtpmap:3 GSM/8000Using SIP OPTIONS for MRCPv2 Server Capability Discovery 🦋
8. Speech Synthesizer Resource ✨
okay so this resource is the one that takes text markup from the client and turns it into a stream of synthesized speech in real time 🎀 Depending on what the server can do, the client can also dictate things like voice characteristics and speaker speed 🗣️💖
The synthesizer is controlled by MRCPv2 requests from the client 💅 It can respond to those requests, or send asynchronous events to the client to flag conditions the client cares about while the speech is being generated 🔥
This section applies for the following resource types:
- speechsynth
- basicsynth
The capabilities of these resources are defined in Section 3.1 🦋
8.1. Synthesizer State Machine
The synthesizer runs a state machine to process MRCPv2 requests from the client 🧚 The transitions below show the synthesizer's states and reflect the state of the request sitting at the head of the synthesizer resource queue. A SPEAK request in the PENDING state can be deleted or stopped by a STOP request without affecting the state of the resource 🌸
Idle Speaking Paused
State State State
| | |
|----------SPEAK-------->| |--------|
|<------STOP-------------| CONTROL |
|<----SPEAK-COMPLETE-----| |------->|
|<----BARGE-IN-OCCURRED--| |
| |---------| |
| CONTROL |-----------PAUSE--------->|
| |-------->|<----------RESUME---------|
| | |----------|
|----------| | PAUSE |
| BARGE-IN-OCCURRED | |--------->|
|<---------| |----------| |
| | SPEECH-MARKER |
| |<---------| |
|----------| |----------| |
| STOP | RESUME |
| | |<---------| |
|<---------| | |
|<---------------------STOP-------------------------|
|----------| | |
| DEFINE-LEXICON | |
| | | |
|<---------| | |
|<---------------BARGE-IN-OCCURRED------------------|
Synthesizer State Machine8.2. Synthesizer Methods
the synthesizer supports the following methods 💜
synthesizer-method = "SPEAK"
/ "STOP"
/ "PAUSE"
/ "RESUME"
/ "BARGE-IN-OCCURRED"
/ "CONTROL"
/ "DEFINE-LEXICON"8.3. Synthesizer Events
the synthesizer can generate the following events ✨
synthesizer-event = "SPEECH-MARKER"
/ "SPEAK-COMPLETE"8.4. Synthesizer Header Fields
a synthesizer method can carry header fields with request options and extra info that augment the Request, Response, or Event it's attached to 🍓
synthesizer-header = jump-size
/ kill-on-barge-in
/ speaker-profile
/ completion-cause
/ completion-reason
/ voice-parameter
/ prosody-parameter
/ speech-marker
/ speech-language
/ fetch-hint
/ audio-fetch-hint
/ failed-uri
/ failed-uri-cause
/ speak-restart
/ speak-length
/ load-lexicon
/ lexicon-search-order8.4.1. Jump-Size
this header field MAY be specified in a CONTROL method, and it controls how far forward or backward to jump in an active SPEAK request 🦋 A '+' or '-' means the value is relative to what's currently playing. this header field MAY also be specified in a SPEAK request as a desired offset into the synthesized speech. In that case, the synthesizer MUST begin speaking from this amount of time into the speech markup. Heads up: an offset that runs past the end of the produced speech will result in audio of length zero 🙈 The different speech length units supported depend on the synthesizer implementation. If the synthesizer resource doesn't support a unit for the operation, the resource MUST respond with a status-code of 409 "Unsupported Header Field Value" 💀
jump-size = "Jump-Size" ":" speech-length-value CRLF
speech-length-value = numeric-speech-length
/ text-speech-length
text-speech-length = 1*UTFCHAR SP "Tag"
numeric-speech-length = ("+" / "-") positive-speech-length
positive-speech-length = 1*19DIGIT SP numeric-speech-unit
numeric-speech-unit = "Second"
/ "Word"
/ "Sentence"
/ "Paragraph"8.4.2. Kill-On-Barge-In
this header field MAY be sent as part of the SPEAK method to turn on "kill-on-barge-in" support 🔥 When it's enabled, the SPEAK method gets interrupted by DTMF input detected by a signal detector resource, or by the start of speech sensed or recognized by the speech recognizer resource 🎧
kill-on-barge-in = "Kill-On-Barge-In" ":" BOOLEAN CRLFThe client MUST send a BARGE-IN-OCCURRED method to the synthesizer resource when it receives a barge-in-able event from any source 🫶 That source could be a synthesizer resource or a signal detector resource, and it MAY be either local or distributed. If this header field isn't specified in a SPEAK request or explicitly set by a SET-PARAMS, the default value is "true" ✨
If the recognizer or signal detector resource is on the same server as the synthesizer, and both are part of the same session, the server MAY work with both to provide internal notification to the synthesizer. That way, audio can be stopped without having to wait for the client's BARGE-IN-OCCURRED event 🚀
It is generally RECOMMENDED when playing a prompt to the user with Kill-On-Barge-In and asking for input, that the client issue the RECOGNIZE request ahead of the SPEAK request for optimum performance and user experience 💅 That way, it's guaranteed the recognizer is online before the prompt starts playing, so the user's speech won't get truncated at the beginning (especially for power users) 😭
8.4.3. Speaker-Profile
this header field MAY be part of the SET-PARAMS/GET-PARAMS or SPEAK request from the client to the server, and it specifies a URI that references the profile of the speaker 🎤 Speaker profiles are collections of voice parameters like gender, accent, etc. 🌈
speaker-profile = "Speaker-Profile" ":" uri CRLF8.4.4. Completion-Cause
this header field MUST be specified in a SPEAK-COMPLETE event coming from the synthesizer resource to the client 📣 It tells you why the SPEAK request completed 👑
completion-cause = "Completion-Cause" ":" 3DIGIT SP
1*VCHAR CRLF
+------------+-----------------------+------------------------------+
| Cause-Code | Cause-Name | Description |
+------------+-----------------------+------------------------------+
| 000 | normal | SPEAK completed normally. |
| 001 | barge-in | SPEAK request was terminated |
| | | because of barge-in. |
| 002 | parse-failure | SPEAK request terminated |
| | | because of a failure to |
| | | parse the speech markup |
| | | text. |
| 003 | uri-failure | SPEAK request terminated |
| | | because access to one of the |
| | | URIs failed. |
| 004 | error | SPEAK request terminated |
| | | prematurely due to |
| | | synthesizer error. |
| 005 | language-unsupported | Language not supported. |
| 006 | lexicon-load-failure | Lexicon loading failed. |
| 007 | cancelled | A prior SPEAK request failed |
| | | while this one was still in |
| | | the queue. |
+------------+-----------------------+------------------------------+
Synthesizer Resource Completion Cause Codes8.4.5. Completion-Reason
this header field MAY be specified in a SPEAK-COMPLETE event coming from the synthesizer resource to the client 📝 It carries the reason text behind the SPEAK request completion. It communicates text describing the reason for a failure, like an error in parsing the speech markup text 🥺
completion-reason = "Completion-Reason" ":"
quoted-string CRLFThe completion reason text is there for client use in logs and for debugging and instrumentation 🔍 Clients MUST NOT interpret the completion reason text, so hands off 🙅♀️💖
8.4.6. Voice-Parameter
this set of header fields defines the voice of the speaker 🎀
voice-parameter = voice-gender
/ voice-age
/ voice-variant
/ voice-name
voice-gender = "Voice-Gender:" voice-gender-value CRLF
voice-gender-value = "male"
/ "female"
/ "neutral"
voice-age = "Voice-Age:" 1*3DIGIT CRLF
voice-variant = "Voice-Variant:" 1*19DIGIT CRLF
voice-name = "Voice-Name:"
1*UTFCHAR *(1*WSP 1*UTFCHAR) CRLFthe "Voice-" parameters are derived from the similarly named attributes of the voice element in W3C's Speech Synthesis Markup Language Specification (SSML) [W3C.REC-speech-synthesis-20040907]. Legal values for these parameters are whatever that specification says 📚✨
these header fields MAY be sent in SET-PARAMS or GET-PARAMS requests to define or get default values for the entire session, or they MAY be sent in the SPEAK request to define default values for that one SPEAK request 🧁 Note that SSML content can set these values internally inside the SSML document too, of course 💅
Voice parameter header fields MAY also be sent in a CONTROL method to affect a SPEAK request in progress and change its behavior on the fly 🦋 If the synthesizer resource doesn't support this operation, it MUST reject the request with a status-code of 403 "Unsupported Header Field" 🚫💀
8.4.7. Prosody-Parameters ✨
This set of header fields defines the prosody of the speech. 🎀
prosody-parameter = "Prosody-" prosody-param-name ":"
prosody-param-value CRLF
prosody-param-name = 1*VCHAR
prosody-param-value = 1*VCHARThe prosody-param-name is any one of the attribute names under the prosody element specified in W3C's Speech Synthesis Markup Language Specification [W3C.REC-speech-synthesis-20040907]. The prosody-param-value is any one of the value choices of the corresponding prosody element attribute from that specification. 💅
These header fields MAY be sent in SET-PARAMS or GET-PARAMS requests to define or get default values for the entire session, or MAY be sent in the SPEAK request to define default values for that SPEAK request. 🦋 Furthermore, these attributes can be part of the speech text marked up in SSML. 🌸
The prosody parameter header fields in the SET-PARAMS or SPEAK request only apply if the speech data is of type 'text/plain' and does not use a speech markup format. 🙈
These prosody parameter header fields MAY also be sent in a CONTROL method to affect a SPEAK request in progress and change its behavior on the fly. ⚡ If the synthesizer resource does not support this operation, it MUST respond back to the client with a status-code of 403 "Unsupported Header Field". 💀
8.4.8. Speech-Marker ⏰
This header field contains timestamp information in a "timestamp" field. It's a Network Time Protocol (NTP) [RFC5905] timestamp, a 64-bit number in decimal form. It MUST be synced with the Real-Time Protocol (RTP) [RFC3550] timestamp of the media stream through the Real-Time Control Protocol (RTCP) [RFC3550]. 🕰️✨
Markers are bookmarks that are defined within the markup. Most speech markup formats provide mechanisms to embed marker fields within speech texts. 🔖 The synthesizer generates SPEECH-MARKER events when it reaches these marker fields. This header field MUST be part of the SPEECH-MARKER event and contain the marker tag value after the timestamp, separated by a semicolon. In these events, the timestamp marks the time the text corresponding to the marker was emitted as speech by the synthesizer. 🎤
This header field MUST also be returned in responses to STOP, CONTROL, and BARGE-IN-OCCURRED methods, in the SPEAK-COMPLETE event, and in an IN-PROGRESS SPEAK response. In these messages, if any markers have been encountered for the current SPEAK, the marker tag value MUST be the last embedded marker encountered. If no markers have yet been encountered for the current SPEAK, only the timestamp is REQUIRED. Note that in these events, the purpose of this header field is to provide timestamp information associated with important events within the lifecycle of a request (start of SPEAK processing, end of SPEAK processing, receipt of CONTROL/STOP/BARGE-IN-OCCURRED). 👑
timestamp = "timestamp" "=" time-stamp-value
time-stamp-value = 1*20DIGIT
speech-marker = "Speech-Marker" ":"
timestamp
[";" 1*(UTFCHAR / %x20)] CRLF8.4.9. Speech-Language 🌍
This header field specifies the default language of the speech data if the language is not specified in the markup. The value of this header field MUST follow RFC 5646 [RFC5646] for its values. The header field MAY occur in SPEAK, SET-PARAMS, or GET-PARAMS requests. 🫶
speech-language = "Speech-Language" ":" 1*VCHAR CRLF8.4.10. Fetch-Hint 📥
When the synthesizer needs to fetch documents or other resources like speech markup or audio files, this header field controls the corresponding URI access properties. It gives the client a say in when the synthesizer should retrieve content from the server. 🧚 A value of "prefetch" indicates the content MAY be downloaded when the request is received, whereas "safe" indicates that content MUST NOT be downloaded until actually referenced. The default value is "prefetch". This header field MAY occur in SPEAK, SET-PARAMS, or GET-PARAMS requests. 💜
fetch-hint = "Fetch-Hint" ":" ("prefetch" / "safe") CRLF8.4.11. Audio-Fetch-Hint 🔊
When the synthesizer needs to fetch documents or other resources like speech audio files, this header field controls the corresponding URI access properties. It's the client's policy on whether or not the synthesizer is permitted to attempt to optimize speech by pre-fetching audio. 🎧 The value is either "safe" to say that audio is only fetched when it is referenced, never before; "prefetch" to permit, but not require the implementation to pre-fetch the audio; or "stream" to allow it to stream the audio fetches. The default value is "prefetch". This header field MAY occur in SPEAK, SET-PARAMS, or GET-PARAMS requests. 🌈
audio-fetch-hint = "Audio-Fetch-Hint" ":"
("prefetch" / "safe" / "stream") CRLF8.4.12. Failed-URI 🚨
When a synthesizer method needs a synthesizer to fetch or access a URI and the access fails, the server SHOULD provide the failed URI in this header field in the method response, unless there are multiple URI failures, in which case the server MUST provide one of the failed URIs in this header field in the method response. 😭
failed-uri = "Failed-URI" ":" absoluteURI CRLF8.4.13. Failed-URI-Cause 🔍
When a synthesizer method needs a synthesizer to fetch or access a URI and the access fails, the server MUST provide the URI-specific or protocol-specific response code for the URI in the Failed-URI header field in the method response through this header field. The value encoding is UTF-8 (RFC 3629 [RFC3629]) to accommodate any access protocol. Some access protocols might have a response string instead of a numeric response code, so this keeps everything flexible. 🫧
failed-uri-cause = "Failed-URI-Cause" ":" 1*UTFCHAR CRLF8.4.14. Speak-Restart 🔁
When a client issues a CONTROL request to a currently speaking synthesizer resource to jump backward, and the target jump point is before the start of the current SPEAK request, the current SPEAK request MUST restart from the beginning of its speech data, and the server's response to the CONTROL request MUST contain this header field with a value of "true" indicating a restart. It's giving "back to the top" energy. 🔝✨
speak-restart = "Speak-Restart" ":" BOOLEAN CRLF8.4.15. Speak-Length 📏
This header field MAY be specified in a CONTROL method to control the maximum length of speech to speak, relative to the current speaking point in the currently active SPEAK request. If numeric, the value MUST be a positive integer. If a header field with a Tag unit is specified, then the speech output continues until the tag is reached or the SPEAK request is completed, whichever comes first. 🏁 This header field MAY be specified in a SPEAK request to indicate the length to speak from the speech data, and is relative to the point in speech that the SPEAK request starts. The different speech length units supported are synthesizer implementation dependent. If a server does not support the specified unit, the server MUST respond with a status-code of 409 "Unsupported Header Field Value". 🙅♀️
speak-length = "Speak-Length" ":" positive-length-value
CRLF
positive-length-value = positive-speech-length
/ text-speech-length
text-speech-length = 1*UTFCHAR SP "Tag"
positive-speech-length = 1*19DIGIT SP numeric-speech-unit
numeric-speech-unit = "Second"
/ "Word"
/ "Sentence"
/ "Paragraph"8.4.16. Load-Lexicon 📖
This header field is used to indicate whether a lexicon has to be loaded or unloaded. The value "true" means to load the lexicon if not already loaded, and the value "false" means to unload the lexicon if it is loaded. The default value for this header field is "true". This header field MAY be specified in a DEFINE-LEXICON method. 💖
load-lexicon = "Load-Lexicon" ":" BOOLEAN CRLF8.4.17. Lexicon-Search-Order 🗂️
This header field is used to specify a list of active pronunciation lexicon URIs and the search order among the active lexicons. Lexicons specified within the SSML document take precedence over the lexicons specified in this header field. This header field MAY be specified in the SPEAK, SET-PARAMS, and GET-PARAMS methods. 🌟
lexicon-search-order = "Lexicon-Search-Order" ":"
"<" absoluteURI ">" *(" " "<" absoluteURI ">") CRLF8.5. Synthesizer Message Body 🎀
A synthesizer message can contain additional information associated with the Request, Response, or Event in its message body. 💌
8.5.1. Synthesizer Speech Data ✨
Okay so the marked-up text you want the synthesizer to speak goes in the message body as a typed media entity 🎀 You can send the speech data inline, embedded right in the body, or by reference, with a URI the server goes and fetches 💅 Either way, the data and the markup format need to be a content type the server supports, periodt.
Every MRCPv2 server with synthesizer resources MUST support plain text speech data and W3C's Speech Synthesis Markup Language [W3C.REC-speech-synthesis-20040907], so it MUST support the media types 'text/plain' and 'application/ssml+xml' 🌸 Other formats MAY be supported 🦋
If the speech data is fetched by URI reference, the media type 'text/uri-list' (see RFC 2483 [RFC2483]) is used to list one or more URIs that, once dereferenced, contain the content to be spoken. If you give a list of speech URIs, the resource MUST speak the speech data from each URI in the order the URIs appear in the content 🦋
MRCPv2 clients and servers MUST support the 'multipart/mixed' media type. This is the go-to when you're mixing URIs and inline speech data together 🫶 Inside the multipart content block, you MAY include content for 'text/uri-list', 'application/ssml+xml', and/or 'text/plain'. The character set and encoding of the speech data follow the standard media type definitions 🌈 The multipart content MAY also contain actual audio data. Clients might have recorded audio clips sitting in memory or on a local device that they want to play as part of the SPEAK request, and the audio portions MAY be sent by the client as part of the multipart content block 🎧 That audio gets referenced from the speech markup data, which is another part of the multipart content block, following the 'multipart/mixed' media type specification 💜
Here's a URI list:
Content-Type:text/uri-list
Content-Length:...
http://www.example.com/ASR-Introduction.ssml
http://www.example.com/ASR-Document-Part1.ssml
http://www.example.com/ASR-Document-Part2.ssml
http://www.example.com/ASR-Conclusion.ssmlURI List Example
And here's an SSML one:
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Aldine Turnbet
and arrived at <break/>
<say-as interpret-as="vxml:time">0345p</say-as>.</s>
<s>The subject is <prosody
rate="-20%">ski trip</prosody></s>
</p>
</speak>SSML Example
And finally, a multipart one that mixes both:
Content-Type:multipart/mixed; boundary="break"
--break
Content-Type:text/uri-list
Content-Length:...
http://www.example.com/ASR-Introduction.ssml
http://www.example.com/ASR-Document-Part1.ssml
http://www.example.com/ASR-Document-Part2.ssml
http://www.example.com/ASR-Conclusion.ssml
--break
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams
and arrived at <break/>
<say-as interpret-as="vxml:time">0342p</say-as>.</s>
<s>The subject is <prosody
rate="-20%">ski trip</prosody></s>
</p>
</speak>
--break--Multipart Example
8.5.2. Lexicon Data 📖
Synthesizer lexicon data going from the client to the server can be sent inline or by reference. Either way, it travels as typed media in the message body of the MRCPv2 request message (see Section 8.14) 💖
When you send a lexicon inline, the client MUST give it a Content-ID as part of the content header fields. The server MUST store the lexicon tied to that Content-ID for the duration of the session 🧚 You can overwrite a stored lexicon by defining a new one with the same Content-ID ✨
Lexicons that have a Content-ID can be referenced through the 'session' URI scheme (see Section 13.6) 🦋
If lexicon data is specified by external URI reference, the media type 'text/uri-list' (see RFC 2483 [RFC2483] ) lists the one or more URIs that may be dereferenced to get the lexicon data. All MRCPv2 servers MUST support the "http" and "https" URI access mechanisms, and MAY support other mechanisms 🌐
If the message body mixes URIs and inline lexicon data, use the 'multipart/mixed' media type. The character set and encoding of the lexicon data may be specified according to standard media type definitions 🌸
8.6. SPEAK Method 🗣️
The SPEAK request hands the synthesizer resource the speech text and kicks off speech synthesis and streaming 🔥 The SPEAK method MAY carry voice and prosody header fields that change how the synthesized voice behaves, along with a typed media message body holding the marked-up text to be spoken 💅
The SPEAK method implementation MUST fetch all external URIs that are part of that operation. If caching is implemented, this URI fetching MUST conform to the cache-control hints and parameter header fields that go with the method when it decides whether to fetch from cache or from the external server 👀 If those hints or parameters aren't specified in the method, the values set for the session using SET-PARAMS/GET-PARAMS apply. If they weren't set for the session either, their default values apply 💜
When voice parameters get applied, there are three levels of precedence 👑 The highest is whatever's specified within the speech markup text. Next come the header fields of the SPEAK request, which only apply to that one SPEAK request. After those are the session default values, which you can set with the SET-PARAMS request and which apply to later methods during the session 🫶
If the resource was idle when the SPEAK request arrived at the server and the SPEAK method is being actively processed, the resource responds right away with a success status code and a request-state of IN-PROGRESS 🌈
If the resource is in the speaking or paused state when the SPEAK method arrives, meaning it's in the middle of processing a previous SPEAK request, the status comes back as success with a request-state of PENDING 🎀 The server puts the SPEAK request into the synthesizer resource request queue. That queue operates strictly FIFO, so requests are processed serially in the order they were received 💅 If the current SPEAK fails, every SPEAK method in the pending queue gets cancelled, and each one generates a SPEAK-COMPLETE event with a Completion-Cause of "cancelled" 😭
For the synthesizer resource, SPEAK is the only method that can return a request-state of IN-PROGRESS or PENDING 💖 Once the text has been synthesized and played into the media stream, the resource issues a SPEAK-COMPLETE event with the request-id of the SPEAK request and a request-state of COMPLETE 🦋
Here's the flow:
C->S: MRCP/2.0 ... SPEAK 543257
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:neutral
Voice-Age:25
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams and arrived at
<break/>
<say-as interpret-as="vxml:time">0342p</say-as>.
</s>
<s>The subject is
<prosody rate="-20%">ski trip</prosody>
</s>
</p>
</speak>
S->C: MRCP/2.0 ... 543257 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857206027059
S->C: MRCP/2.0 ... SPEAK-COMPLETE 543257 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Completion-Cause:000 normal
Speech-Marker:timestamp=857206027059SPEAK Example
8.7. STOP 🛑
The STOP method, sent from the client to the server, tells the synthesizer resource to stop speaking if it's speaking something 🙈
A STOP request can include an Active-Request-Id-List header field to stop zero or more specific SPEAK requests that may be queued, and it gets a response status-code of 200 "Success" back ✨ If you don't send an Active-Request-Id-List header field in the STOP request, the server terminates all outstanding SPEAK requests 💀
If a STOP request successfully terminated one or more PENDING or IN-PROGRESS SPEAK requests, the response MUST contain an Active-Request-Id-List header field listing the SPEAK request-ids that were terminated. Otherwise, the response has no Active-Request-Id-List header field. No SPEAK-COMPLETE events get sent for those terminated requests 🫠
If a SPEAK request that was IN-PROGRESS and speaking gets stopped, the next pending SPEAK request, if there is one, becomes IN-PROGRESS at the resource and enters the speaking state 🎶
If a SPEAK request that was IN-PROGRESS and paused gets stopped, the next pending SPEAK request, if there is one, becomes IN-PROGRESS and enters the paused state 🌙
Here's how that looks:
C->S: MRCP/2.0 ... SPEAK 543258
Channel-Identifier:32AECB23433802@speechsynth
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams and arrived at
<break/>
<say-as interpret-as="vxml:time">0342p</say-as>.</s>
<s>The subject is
<prosody rate="-20%">ski trip</prosody></s>
</p>
</speak>
S->C: MRCP/2.0 ... 543258 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857206027059
C->S: MRCP/2.0 ... STOP 543259
Channel-Identifier:32AECB23433802@speechsynth
S->C: MRCP/2.0 ... 543259 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Active-Request-Id-List:543258
Speech-Marker:timestamp=857206039059STOP Example
8.8. BARGE-IN-OCCURRED ✨
okay bestie, BARGE-IN-OCCURRED is the synthesizer-resource method a client uses to tell the synthesizer that a barge-in-able event happened 👀 so the synthesizer can react too 🦋 it's giving teamwork 🫶
This method is useful in two scenarios:
- The client spotted DTMF digits in the input media (or some other barge-in-able event) and wants the synthesizer resource to know 📞
- The recognizer resource and the synthesizer resource live on different servers 🌐 so the client becomes the middle-person 💅 it gets an event from the recognition resource and then sends a BARGE-IN-OCCURRED request to the synthesizer. In this case, the BARGE-IN-OCCURRED method would also carry a
Proxy-Sync-Idheader field, taken from the resource that generated the original event 🧚
Now the main drama 🔥 If a SPEAK request is active with kill-on-barge-in enabled (see Section 8.4.2) and a BARGE-IN-OCCURRED event comes in, the synthesizer MUST immediately stop streaming out audio. It MUST also terminate any speech requests queued behind the current active one, irrespective of whether or not they have barge-in enabled. If a barge-in-able SPEAK request was playing and got terminated, the response MUST contain an Active-Request-Id-List header field listing the request-ids of every SPEAK request that was terminated. The server generates no SPEAK-COMPLETE events for these requests 💀
If the synthesizer resource didn't terminate any SPEAK requests because of the BARGE-IN-OCCURRED method, the server MUST respond with a status-code of 200 "Success", and the response MUST NOT contain an Active-Request-Id-List header field. Clean and simple 🌸
If the synthesizer and recognizer resources are part of the same MRCPv2 session, they can be optimized for a quicker kill-on-barge-in response when the recognizer and synthesizer talk directly ⚡ Even so, the client MUST still react to a START-OF-INPUT event from the recognizer by invoking the BARGE-IN-OCCURRED method to the synthesizer. The client MUST invoke BARGE-IN-OCCURRED if it has any outstanding requests to the synthesizer resource in either the PENDING or IN-PROGRESS state 🎀
C->S: MRCP/2.0 ... SPEAK 543258
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:neutral
Voice-Age:25
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams and arrived at
<break/>
<say-as interpret-as="vxml:time">0342p</say-as>.</s>
<s>The subject is
<prosody rate="-20%">ski trip</prosody></s>
</p>
</speak>
S->C: MRCP/2.0 ... 543258 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857206027059
C->S: MRCP/2.0 ... BARGE-IN-OCCURRED 543259
Channel-Identifier:32AECB23433802@speechsynth
Proxy-Sync-Id:987654321
S->C:MRCP/2.0 ... 543259 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Active-Request-Id-List:543258
Speech-Marker:timestamp=857206039059
BARGE-IN-OCCURRED Example8.9. PAUSE ⏸️
okay so PAUSE is the client's way of telling the synthesizer to pause speech output while it's talking 🎤 If a PAUSE method shows up when no SPEAK is active, the server MUST respond with a status-code of 402 "Method not valid in this state" 🙈 If a PAUSE method shows up while a SPEAK is active and already paused, the server MUST respond with a status-code of 200 "Success" ✨ And if a SPEAK request was active, the server MUST return an Active-Request-Id-List header field whose value holds the request-id of the SPEAK request that got paused 💜
C->S: MRCP/2.0 ... SPEAK 543258
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:neutral
Voice-Age:25
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams and arrived at
<break/>
<say-as interpret-as="vxml:time">0342p</say-as>.</s>
<s>The subject is
<prosody rate="-20%">ski trip</prosody></s>
</p>
</speak>
S->C: MRCP/2.0 ... 543258 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857206027059
C->S: MRCP/2.0 ... PAUSE 543259
Channel-Identifier:32AECB23433802@speechsynth
S->C: MRCP/2.0 ... 543259 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Active-Request-Id-List:543258
PAUSE Example8.10. RESUME ▶️
RESUME is the vibe check for a paused synthesizer 💖 it tells the synthesizer to pick up where it left off and start talking again 🗣️ If a RESUME request shows up on a session with no active SPEAK request, the server MUST respond with a status-code of 402 "Method not valid in this state" 😭 If a RESUME request lands on a session with an active SPEAK request that's already speaking (so, not paused), the server MUST respond with a status-code of 200 "Success" 👑 And if a SPEAK request was paused, the server MUST return an Active-Request-Id-List header field whose value contains the request-id of the SPEAK request that got resumed 🌈
C->S: MRCP/2.0 ... SPEAK 543258
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:neutral
Voice-age:25
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams and arrived at
<break/>
<say-as interpret-as="vxml:time">0342p</say-as>.</s>
<s>The subject is
<prosody rate="-20%">ski trip</prosody></s>
</p>
</speak>
S->C: MRCP/2.0 ... 543258 200 IN-PROGRESS@speechsynth
Channel-Identifier:32AECB23433802
Speech-Marker:timestamp=857206027059
C->S: MRCP/2.0 ... PAUSE 543259
Channel-Identifier:32AECB23433802@speechsynth
S->C: MRCP/2.0 ... 543259 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Active-Request-Id-List:543258
C->S: MRCP/2.0 ... RESUME 543260
Channel-Identifier:32AECB23433802@speechsynth
S->C: MRCP/2.0 ... 543260 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Active-Request-Id-List:543258
RESUME Example8.11. CONTROL 🎛️
okay the CONTROL method is literally the remote for a speaking synthesizer 📺 The client uses it to change what the synthesizer is saying on the fly ✨ It can jump forward or backward in the speech, change the speaking rate, change speaker parameters, and so on 🏃♀️ It only affects the SPEAK request that's currently IN-PROGRESS 🎯 And depending on the implementation and capability of the synthesizer resource, it may or may not support the different modifications named in the header fields of the CONTROL request 🤷♀️
When a client uses CONTROL to jump forward and the jump goes past the end of the active SPEAK request's text, the CONTROL request still succeeds 🥳 The active SPEAK request finishes and sends a SPEAK-COMPLETE event right after the response to the CONTROL method. If there are more SPEAK requests in the queue, the synthesizer resource starts at the beginning of the next one 🍓
When a client uses CONTROL to jump backward and the jump lands at the beginning (or goes even further back than the beginning) of the active SPEAK request's speech data, the CONTROL request still succeeds 🙌 The response to the CONTROL request contains the speak-restart header field, and the active SPEAK request restarts from the beginning of its speech data 🔁
Those two behaviors are super handy for rewinding or fast-forwarding across multiple speech requests 🦋 which is useful if the client chopped a speech markup text into several SPEAK requests 🧩
If a SPEAK request was active when the CONTROL method was received, the server MUST return an Active-Request-Id-List header field containing the request-id of that active SPEAK request 💅
C->S: MRCP/2.0 ... SPEAK 543258
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:neutral
Voice-age:25
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams
and arrived at <break/>
<say-as interpret-as="vxml:time">0342p</say-as>.</s>
<s>The subject is <prosody
rate="-20%">ski trip</prosody></s>
</p>
</speak>
S->C: MRCP/2.0 ... 543258 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857205016059
C->S: MRCP/2.0 ... CONTROL 543259
Channel-Identifier:32AECB23433802@speechsynth
Prosody-rate:fast
S->C: MRCP/2.0 ... 543259 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Active-Request-Id-List:543258
Speech-Marker:timestamp=857206027059
C->S: MRCP/2.0 ... CONTROL 543260
Channel-Identifier:32AECB23433802@speechsynth
Jump-Size:-15 Words
S->C: MRCP/2.0 ... 543260 200 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Active-Request-Id-List:543258
Speech-Marker:timestamp=857206039059
CONTROL Example8.12. SPEAK-COMPLETE
okay so this is the Event message the synthesizer sends the client to say "bestie, that SPEAK request is DONE" 💅✨ The request-id matches the request-id of the SPEAK that started the speech that just wrapped up 🦋 The server sets request-state to COMPLETE, which means this is the very last event for that request-id 💖 And the Completion-Cause header field tells you why it finished: normal, an error, kill-on-barge-in, or whatever else applies 🔥
C->S: MRCP/2.0 ... SPEAK 543260
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:neutral
Voice-age:25
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams
and arrived at <break/>
<say-as interpret-as="vxml:time">0342p</say-as>.</s>
<s>The subject is
<prosody rate="-20%">ski trip</prosody></s>
</p>
</speak>
S->C: MRCP/2.0 ... 543260 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857206027059
S->C: MRCP/2.0 ... SPEAK-COMPLETE 543260 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Completion-Cause:000 normal
Speech-Marker:timestamp=857206039059
SPEAK-COMPLETE Example8.13. SPEECH-MARKER
this one is the synthesizer's "I just passed a marker tag in the markup I'm reading!" event 🧚 The request-id value MUST match the one on the matching SPEAK request 💜 The request-state MUST be "IN-PROGRESS", since the speech is still going and not done yet ✨ The marker tag it hit, which shows where the synthesizer is in the markup, MUST come back in the Speech-Marker header field, along with an NTP timestamp for the moment that marker showed up in the output speech stream 🌸 Also, a SPEECH-MARKER event MUST be generated with a null marker value and output NTP timestamp when a SPEAK request in Pending-State (aka it's waiting in the queue) switches to IN-PROGRESS and starts talking 🎀 And the NTP timestamp MUST be synchronized with the RTP timestamp used to make the speech stream, using the standard RTCP machinery 🔥
C->S: MRCP/2.0 ... SPEAK 543261
Channel-Identifier:32AECB23433802@speechsynth
Voice-gender:neutral
Voice-age:25
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams
and arrived at <break/>
<say-as interpret-as="vxml:time">0342p</say-as>.</s>
<mark name="here"/>
<s>The subject is
<prosody rate="-20%">ski trip</prosody>
</s>
<mark name="ANSWER"/>
</p>
</speak>
S->C: MRCP/2.0 ... 543261 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857205015059
S->C: MRCP/2.0 ... SPEECH-MARKER 543261 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857206027059;here
S->C: MRCP/2.0 ... SPEECH-MARKER 543261 IN-PROGRESS
Channel-Identifier:32AECB23433802@speechsynth
Speech-Marker:timestamp=857206039059;ANSWER
S->C: MRCP/2.0 ... SPEAK-COMPLETE 543261 COMPLETE
Channel-Identifier:32AECB23433802@speechsynth
Completion-Cause:000 normal
Speech-Marker:timestamp=857207689259;ANSWER
SPEECH-MARKER Example8.14. DEFINE-LEXICON
the DEFINE-LEXICON method is the client's way of handing the server a lexicon and telling it to load or unload that lexicon (check out Section 8.4.16 for the details) 📚💖 The lexicon's media type goes in the Content-Type header (see Section 8.5.2) 🎀 One example of a media type is "application/pls+xml", which is for the Pronunciation Lexicon Specification (PLS) [W3C.REC-pronunciation-lexicon-20081014] [RFC4267] 🧚
If the server resource is in the speaking or paused state, the server MUST respond with a failure status-code of 402 "Method not valid in this state" 🙈 That's a no, bestie 💀
If the resource is in the idle state and it can load or unload the lexicon successfully, the status MUST return a 200 "Success" status-code, and the request-state MUST be COMPLETE ✨🫶
If the synthesizer can't define the lexicon for some reason (like the download failed, or the lexicon was in a form it doesn't support), the server MUST respond with a failure status-code of 407 and a Completion-Cause header field describing what went wrong 😭
9. Speech Recognizer Resource 🎀
the speech recognizer resource takes in an incoming voice stream and gives the client back an interpretation of what was said, in text form 🦋 It's giving "listening ear" energy 👂✨
The recognizer resource is controlled by MRCPv2 requests from the client 💅 It can respond to those requests, and it can also send asynchronous events back to the client to flag anything important that happens while a method is running 🔥
This section covers the following resource types:
- speechrecog
- dtmfrecog
The big difference between those two is how much they support recognition grammars 💜 The "dtmfrecog" resource type can only recognize DTMF digits, so it only accepts DTMF grammars 🌸 It only generates barge-in for DTMF input and ignores speech 🙈 The "speechrecog" resource type can recognize regular speech as well as DTMF digits, so it MUST support grammars describing either speech or DTMF 💖 This resource generates barge-in events for speech and/or DTMF 🔥 It figures out whether a barge-in should happen for speech and/or DTMF by looking at the grammars that the RECOGNIZE method activates 🧚 When the recognizer decides it needs to generate a barge-in, it also sends a START-OF-INPUT event to the client ✨ The recognizer resource MAY support recognition in the normal or hotword modes, or both (but a single "speechrecog" resource doesn't run normal and hotword recognition at the same time, so keep that in mind) 🦋 If an implementation's single recognizer resource doesn't support both modes, or if simultaneous normal and hotword recognition is wanted, the two modes can be invoked through separate resources allocated to the same SIP dialog (each with its own MRCP session identifier), and they share the RTP audio feed 🫶
The capabilities of the recognizer resource are listed below:
- Normal Mode Recognition: Normal mode tries to match all of the speech or DTMF against the grammar. It returns a no-match status if the input doesn't match or if the method times out 💅
- Hotword Mode Recognition: In hotword mode, the recognizer looks for a match against a specific speech grammar or DTMF sequence and ignores any speech or DTMF that doesn't match 👀 The recognition completes only when the grammar matches successfully, when the client cancels the request, or when there's a non-input or recognition timeout 💀
- Voice Enrolled Grammars: A recognizer resource MAY optionally support Voice Enrolled Grammars 🎤 With this feature, enrollment is done using a person's voice. For example, you could build and keep up a list of contacts by recording each person's name in the caller's voice 🌈 This is sometimes called speaker-dependent recognition 🧚
- Interpretation: A recognizer resource MAY be used purely for its natural language interpretation abilities, by giving it a text string as input instead of speech 🍓 In this mode, the resource takes text as input and produces an "interpretation" of that input according to the supplied grammar ✨
Voice enrollment has the concept of an enrollment session 💜 To add a new phrase to a personal grammar, you start with the initial enrollment and then repeat enough utterances before committing the new phrase to the personal grammar 🌸 Every time an utterance is recorded, it gets compared for similarity with the other samples, and a clash test runs against the other entries in the personal grammar so there aren't any similar, confusable entries 😭
Enrollment is done using a recognizer resource 🎀 You control which utterances count toward enrolling a new phrase by setting a header field (see Section 9.4.39) in the Recognize request 💖 Interpretation happens through the INTERPRET method (Section 9.20) and the Interpret-Text header field (Section 9.4.30) 🔥
9.1. Recognizer State Machine 🦋
the recognizer resource keeps a state machine to process the MRCPv2 requests from the client 💅
Idle Recognizing Recognized
State State State
| | |
|---------RECOGNIZE---->|---RECOGNITION-COMPLETE-->|
|<------STOP------------|<-----RECOGNIZE-----------|
| | |
| |--------| |-----------|
| START-OF-INPUT | GET-RESULT |
| |------->| |---------->|
|------------| | |
| DEFINE-GRAMMAR |----------| |
|<-----------| | START-INPUT-TIMERS |
| |<---------| |
|------| | |
| INTERPRET | |
|<-----| |------| |
| | RECOGNIZE |
|-------| |<-----| |
| STOP |
|<------| |
|<-------------------STOP--------------------------|
|<-------------------DEFINE-GRAMMAR----------------|
Recognizer State MachineIf a recognizer resource supports voice enrolled grammars, starting an enrollment session does not change the recognizer's state ✨ Once the enrollment session has started, utterances get enrolled by calling the RECOGNIZE method over and over 🫶 Each time RECOGNIZE is called, the speech recognizer resource's state moves from IDLE to RECOGNIZING 🧚
9.2. Recognizer Methods 🎀
Okay bestie, let's talk about what the recognizer can actually do 💅 The recognizer supports the following methods.
recognizer-method = recog-only-method
/ enrollment-method
recog-only-method = "DEFINE-GRAMMAR"
/ "RECOGNIZE"
/ "INTERPRET"
/ "GET-RESULT"
/ "START-INPUT-TIMERS"
/ "STOP"Supporting voice enrolled grammars is OPTIONAL for a recognizer resource, so no pressure 🫶 But if it does support them, it MUST support the following methods 👑
enrollment-method = "START-PHRASE-ENROLLMENT"
/ "ENROLLMENT-ROLLBACK"
/ "END-PHRASE-ENROLLMENT"
/ "MODIFY-PHRASE"
/ "DELETE-PHRASE"9.3. Recognizer Events ⭐
The recognizer can generate the following events 🦋
recognizer-event = "START-OF-INPUT"
/ "RECOGNITION-COMPLETE"
/ "INTERPRETATION-COMPLETE"9.4. Recognizer Header Fields 🌈
okay so a recognizer message can carry header fields with request options and extra info that augments whichever Method, Response, or Event message it's attached to 🧚
recognizer-header = recog-only-header
/ enrollment-header
recog-only-header = confidence-threshold
/ sensitivity-level
/ speed-vs-accuracy
/ n-best-list-length
/ no-input-timeout
/ input-type
/ recognition-timeout
/ waveform-uri
/ input-waveform-uri
/ completion-cause
/ completion-reason
/ recognizer-context-block
/ start-input-timers
/ speech-complete-timeout
/ speech-incomplete-timeout
/ dtmf-interdigit-timeout
/ dtmf-term-timeout
/ dtmf-term-char
/ failed-uri
/ failed-uri-cause
/ save-waveform
/ media-type
/ new-audio-channel
/ speech-language
/ ver-buffer-utterance
/ recognition-mode
/ cancel-if-queue
/ hotword-max-duration
/ hotword-min-duration
/ interpret-text
/ dtmf-buffer-time
/ clear-dtmf-buffer
/ early-no-matchIf a recognizer resource supports voice enrolled grammars, the following header fields are also used too 💖
enrollment-header = num-min-consistent-pronunciations
/ consistency-threshold
/ clash-threshold
/ personal-grammar-uri
/ enroll-utterance
/ phrase-id
/ phrase-nl
/ weight
/ save-best-waveform
/ new-phrase-id
/ confusable-phrases-uri
/ abort-phrase-enrollmentFor enrollment-specific header fields that can show up as part of SET-PARAMS or GET-PARAMS methods, the following general rule applies: the START-PHRASE-ENROLLMENT method MUST be invoked before these header fields may be set through the SET-PARAMS method or retrieved through the GET-PARAMS method. this is the rule, babe, don't skip it 🙈
Also, the Waveform-URI header field of the Recognizer resource can also appear in the response to the END-PHRASE-ENROLLMENT method ✨
9.4.1. Confidence-Threshold 💅
When a recognizer resource recognizes or matches a spoken phrase with some portion of the grammar, it attaches a confidence level to that match. The Confidence-Threshold header field tells the recognizer resource what confidence level the client considers a successful match. It's a float value between 0.0-1.0 showing the recognizer's confidence in the recognition. If the recognizer determines that there is no candidate match with a confidence that is greater than the confidence threshold, then it MUST return no-match as the recognition result. This header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. The default value for this header field is implementation specific, as is the interpretation of any specific value for this header field. Although values for servers from different vendors are not comparable, it is expected that clients will tune this value over time for a given server. 🎯
confidence-threshold = "Confidence-Threshold" ":" FLOAT CRLF9.4.2. Sensitivity-Level 🔊
To filter out background noise and not mistake it for speech, the recognizer resource supports a variable level of sound sensitivity. The Sensitivity-Level header field is a float value between 0.0 and 1.0 and lets the client set the sensitivity level for the recognizer. This header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. A higher value for this header field means higher sensitivity. The default value for this header field is implementation specific, as is the interpretation of any specific value for this header field. Although values for servers from different vendors are not comparable, it is expected that clients will tune this value over time for a given server. 🎧
sensitivity-level = "Sensitivity-Level" ":" FLOAT CRLF9.4.3. Speed-Vs-Accuracy 🏎️
Depending on the implementation and capability of the recognizer resource, it may be tunable towards Performance or Accuracy. Higher accuracy may mean more processing and higher CPU utilization, meaning fewer active sessions per server, and vice versa. The value is a float between 0.0 and 1.0. A value of 0.0 means fastest recognition. A value of 1.0 means best accuracy. This header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. The default value for this header field is implementation specific. Although values for servers from different vendors are not comparable, it is expected that clients will tune this value over time for a given server. ⚖️
speed-vs-accuracy = "Speed-Vs-Accuracy" ":" FLOAT CRLF9.4.4. N-Best-List-Length 🍓
When the recognizer matches an incoming stream with the grammar, it may come up with more than one alternative match because of confidence levels in certain words or conversation paths. If this header field isn't specified, by default the recognizer resource returns only the best match above the confidence threshold. The client, by setting this header field, can ask the recognition resource to send it more than one alternative. All alternatives must still be above the Confidence-Threshold. A value greater than one does not guarantee that the recognizer will provide the requested number of alternatives. This header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. The minimum value for this header field is 1. The default value for this header field is 1. 🌸
n-best-list-length = "N-Best-List-Length" ":" 1*19DIGIT CRLF9.4.5. Input-Type 🎤
When the recognizer detects barge-in-able input and generates a START-OF-INPUT event, that event MUST carry this header field to specify whether the input that caused the barge-in was DTMF or speech. 🫣
input-type = "Input-Type" ":" inputs CRLF
inputs = "speech" / "dtmf"9.4.6. No-Input-Timeout ⏰
When recognition starts and there's no speech detected for a certain period of time, the recognizer can send a RECOGNITION-COMPLETE event to the client with a Completion-Cause of "no-input-timeout" and terminate the recognition operation. The client can use the No-Input-Timeout header field to set this timeout. The value is in milliseconds and can range from 0 to an implementation-specific maximum value. This header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. The default value is implementation specific. 💀
no-input-timeout = "No-Input-Timeout" ":" 1*19DIGIT CRLF9.4.7. Recognition-Timeout ⌛
When recognition starts and there's no match for a certain period of time, the recognizer can send a RECOGNITION-COMPLETE event to the client and terminate the recognition operation. The Recognition-Timeout header field lets the client set this timeout value. The value is in milliseconds. It ranges from 0 to an implementation-specific maximum value. The default value is 10 seconds. This header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. 🧁
recognition-timeout = "Recognition-Timeout" ":" 1*19DIGIT CRLF9.4.8. Waveform-URI 🎙️
If the Save-Waveform header field is set to "true", the recognizer MUST record the incoming audio stream of the recognition into a stored form and provide a URI for the client to access it. This header field MUST be present in the RECOGNITION-COMPLETE event if the Save-Waveform header field was set to "true". The value of the header field MUST be empty if there was some error condition preventing the server from recording. Otherwise, the URI generated by the server MUST be unambiguous across the server and all its recognition sessions. The content associated with the URI MUST be available to the client until the MRCPv2 session terminates. 🗂️
Similarly, if the Save-Best-Waveform header field is set to "true", the recognizer MUST save the audio stream for the best repetition of the phrase that was used during the enrollment session. The recognizer MUST then record the recognized audio and make it available to the client by returning a URI in the Waveform-URI header field in the response to the END-PHRASE-ENROLLMENT method. The value of the header field MUST be empty if there was some error condition preventing the server from recording. Otherwise, the URI generated by the server MUST be unambiguous across the server and all its recognition sessions. The content associated with the URI MUST be available to the client until the MRCPv2 session terminates. See the discussion on the sensitivity of saved waveforms in Section 12.
⚠️ bestie, the sensitivity of saved waveforms is a whole topic. Go read Section 12 before you save anything spicy 🙈
The server MUST also return the size in octets and the duration in milliseconds of the recorded audio waveform as parameters associated with the header field. 📏
waveform-uri = "Waveform-URI" ":" ["<" uri ">"
";" "size" "=" 1*19DIGIT
";" "duration" "=" 1*19DIGIT] CRLF9.4.9. Media-Type 📼
This header field MAY be specified in the SET-PARAMS, GET-PARAMS, or the RECOGNIZE methods, and it tells the server resource the media type to store captured audio or video in, such as the one captured and returned by the Waveform-URI header field. 🎬
media-type = "Media-Type" ":" media-type-value
CRLF9.4.10. Input-Waveform-URI 🔗
This optional header field specifies a URI pointing to audio content to be processed by the RECOGNIZE operation. It lets the client request recognition from a specified buffer or audio file. 🥰
input-waveform-uri = "Input-Waveform-URI" ":" uri CRLF9.4.11. Completion-Cause 💅
Okay so this header field MUST be part of a RECOGNITION-COMPLETE event that the recognizer resource sends to the client 💖. It tells you why the RECOGNIZE method wrapped up, which is literally the tea ☕. It also MUST be sent in the DEFINE-GRAMMAR and RECOGNIZE responses whenever they come back with a failure status and a COMPLETE state 💀. In the ABNF below, the cause-code is a 3-digit number picked from the Cause-Code column of the table, and the cause-name is the matching token from the Cause-Name column 🎀.
completion-cause = "Completion-Cause" ":" cause-code SP
cause-name CRLF
cause-code = 3DIGIT
cause-name = *VCHAR
+------------+-----------------------+------------------------------+
| Cause-Code | Cause-Name | Description |
+------------+-----------------------+------------------------------+
| 000 | success | RECOGNIZE completed with a |
| | | match or DEFINE-GRAMMAR |
| | | succeeded in downloading and |
| | | compiling the grammar. |
| | | |
| 001 | no-match | RECOGNIZE completed, but no |
| | | match was found. |
| | | |
| 002 | no-input-timeout | RECOGNIZE completed without |
| | | a match due to a |
| | | no-input-timeout. |
| | | |
| 003 | hotword-maxtime | RECOGNIZE in hotword mode |
| | | completed without a match |
| | | due to a |
| | | recognition-timeout. |
| | | |
| 004 | grammar-load-failure | RECOGNIZE failed due to |
| | | grammar load failure. |
| | | |
| 005 | grammar-compilation- | RECOGNIZE failed due to |
| | failure | grammar compilation failure. |
| | | |
| 006 | recognizer-error | RECOGNIZE request terminated |
| | | prematurely due to a |
| | | recognizer error. |
| | | |
| 007 | speech-too-early | RECOGNIZE request terminated |
| | | because speech was too |
| | | early. This happens when the |
| | | audio stream is already |
| | | "in-speech" when the |
| | | RECOGNIZE request was |
| | | received. |
| | | |
| 008 | success-maxtime | RECOGNIZE request terminated |
| | | because speech was too long |
| | | but whatever was spoken till |
| | | that point was a full match. |
| | | |
| 009 | uri-failure | Failure accessing a URI. |
| | | |
| 010 | language-unsupported | Language not supported. |
| | | |
| 011 | cancelled | A new RECOGNIZE cancelled |
| | | this one, or a prior |
| | | RECOGNIZE failed while this |
| | | one was still in the queue. |
| | | |
| 012 | semantics-failure | Recognition succeeded, but |
| | | semantic interpretation of |
| | | the recognized input failed. |
| | | The RECOGNITION-COMPLETE |
| | | event MUST contain the |
| | | Recognition result with only |
| | | input text and no |
| | | interpretation. |
| | | |
| 013 | partial-match | Speech Incomplete Timeout |
| | | expired before there was a |
| | | full match. But whatever was |
| | | spoken till that point was a |
| | | partial match to one or more |
| | | grammars. |
| | | |
| 014 | partial-match-maxtime | The Recognition-Timeout |
| | | expired before full match |
| | | was achieved. But whatever |
| | | was spoken till that point |
| | | was a partial match to one |
| | | or more grammars. |
| | | |
| 015 | no-match-maxtime | The Recognition-Timeout |
| | | expired. Whatever was spoken |
| | | till that point did not |
| | | match any of the grammars. |
| | | This cause could also be |
| | | returned if the recognizer |
| | | does not support detecting |
| | | partial grammar matches. |
| | | |
| 016 | grammar-definition- | Any DEFINE-GRAMMAR error |
| | failure | other than |
| | | grammar-load-failure and |
| | | grammar-compilation-failure. |
+------------+-----------------------+------------------------------+9.4.12. Completion-Reason 🧚
This header field MAY be specified in a RECOGNITION-COMPLETE event coming from the recognizer resource to the client 🦋. It carries the reason text behind the RECOGNIZE request completion. The server uses it to spill the details on a failure, like the specific error hit while parsing a grammar markup 🔍.
The completion reason text is there for client logs, debugging, and instrumentation 📝. Clients MUST NOT interpret the completion reason text 🙅♀️.
completion-reason = "Completion-Reason" ":"
quoted-string CRLF9.4.13. Recognizer-Context-Block 🌈
This header field MAY be sent as part of the SET-PARAMS or GET-PARAMS request ✨. If GET-PARAMS includes this header field with no value, that's a request asking the recognizer to return its recognizer context block. The response to that message MAY contain a recognizer context block as a typed media message body. If the server does return a recognizer context block, the response MUST contain this header field, and its value MUST match the Content-ID of the corresponding media block.
If SET-PARAMS includes this header field, it MUST also contain a message body with the recognizer context data and a Content-ID matching this header field value 🎀. That Content-ID MUST match the Content-ID that came with the context data during the GET-PARAMS operation.
An implementation that uses this mechanism to hand off recognizer context data between servers MUST distinguish its implementation-specific block of data by using an IANA-registered content type in the IANA Media Type vendor tree. 🚨
recognizer-context-block = "Recognizer-Context-Block" ":"
[1*VCHAR] CRLF9.4.14. Start-Input-Timers ⏱️
This header field MAY be sent as part of the RECOGNIZE request 💜. A value of false tells the recognizer to start recognition but not start the no-input timer yet. The recognizer MUST NOT start the timers until the client sends a START-INPUT-TIMERS request to the recognizer. This is useful when the recognizer and synthesizer engines aren't part of the same session 🫶. In those setups, when a kill-on-barge-in prompt is playing (see Section 8.4.2), the client wants the RECOGNIZE request to be active at the same time so it can detect and implement kill-on-barge-in. However, the recognizer SHOULD NOT start the no-input timers until the prompt is finished 🙈. The default value is "true".
start-input-timers = "Start-Input-Timers" ":" BOOLEAN CRLF9.4.15. Speech-Complete-Timeout 🍓
This header field sets how much silence is needed after user speech before the speech recognizer finalizes a result (either accepting it or generating a no-match result) 🕰️. The Speech-Complete-Timeout value applies when the recognizer currently has a complete match against an active grammar, and it says how long the recognizer MUST wait for more input before declaring a match. Speech-Incomplete-Timeout is the one used when the speech is an incomplete match to an active grammar instead 💅. The value is in milliseconds.
speech-complete-timeout = "Speech-Complete-Timeout" ":" 1*19DIGIT CRLFA long Speech-Complete-Timeout value delays the result for the client, so the app's response feels slow 🐢. A short one may chop an utterance up in weird places 😭. Reasonable speech complete timeout values are typically in the range of 0.3 seconds to 1.0 seconds ✨. The value for this header field ranges from 0 to an implementation-specific maximum value. The default value for this header field is implementation specific. This header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS 🌸.
9.4.16. Speech-Incomplete-Timeout ✨
okay so this one is the "how long do I wait after you go quiet" header 🤫💖 it sets how much silence the recognizer waits for after user speech before it finalizes a result. the incomplete timeout kicks in when what was just said is only a partial match for all the active grammars. once it fires, the partial result gets tossed, with a Completion-Cause of "partial-match" 🙅♀️ the value is in milliseconds, ranges from 0 up to an implementation-specific max, and the default is implementation specific too 🎀
speech-incomplete-timeout = "Speech-Incomplete-Timeout" ":" 1*19DIGIT
CRLFit also covers the case where the speech is a complete match of an active grammar, but you could still keep talking and still match the grammar 🧠 compare that with Speech-Complete-Timeout, which is for when the speech is a full match and no more words could extend it 💅
a long Speech-Incomplete-Timeout means the result takes forever to reach the client, so the app feels slooow 🐢 a short one can chop an utterance into pieces in the wrong spots 😭
the Speech-Incomplete-Timeout is usually longer than the Speech-Complete-Timeout so people can pause mid-sentence, like to take a breath 🌸 this header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. 🦋
9.4.17. DTMF-Interdigit-Timeout 🎹
this header sets the inter-digit timeout used when recognizing DTMF input (aka keypad presses 📱). value is in milliseconds, goes from 0 up to an implementation-specific max, and the default is 5 seconds ⏱️ this header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. 💜
dtmf-interdigit-timeout = "DTMF-Interdigit-Timeout" ":" 1*19DIGIT CRLF9.4.18. DTMF-Term-Timeout 🔚
this is the terminating timeout for DTMF recognition, and it only matters when the grammar doesn't allow any more input 🙌 otherwise the DTMF-Interdigit-Timeout is the one in charge. value is in milliseconds, ranges from 0 to an implementation-specific max, and the default is 10 seconds 🍓 this header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. ✨
dtmf-term-timeout = "DTMF-Term-Timeout" ":" 1*19DIGIT CRLF9.4.19. DTMF-Term-Char 🔤
this one picks the terminating DTMF character for recognition 🎯 the default is NULL, which you'll see as an empty header field value 🫧 this header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. 🧚
dtmf-term-char = "DTMF-Term-Char" ":" VCHAR CRLF9.4.20. Failed-URI 🚨
so when a recognizer tries to fetch or access a URI and it fails, the server SHOULD put the failed URI in this header in the method response 💌 the exception is when there are multiple URI failures, because then one of the failed URIs MUST be provided in this header field in the method response. 🥺
failed-uri = "Failed-URI" ":" absoluteURI CRLF9.4.21. Failed-URI-Cause 📋
same situation as Failed-URI, but this one carries the URI-specific or protocol-specific response code 🙈 the server MUST provide that code for the URI in this header in the method response. the value is UTF-8 (RFC 3629 [RFC3629]) so it can handle any access protocol, since some of them send back a response string instead of a numeric code 🌈
failed-uri-cause = "Failed-URI-Cause" ":" 1*UTFCHAR CRLF9.4.22. Save-Waveform 🎙️
this header lets the client ask the recognizer to save the audio input 💾 the recognizer MUST then try to record the recognized audio without endpointing, and hand it back as a URI in the Waveform-URI header field in the RECOGNITION-COMPLETE event 🦋 if recording broke or the audio just isn't available, the recognizer MUST return an empty Waveform-URI header field. default is "false" 🙅♀️ this header field MAY occur in RECOGNIZE, SET-PARAMS, or GET-PARAMS. also, check out the sensitivity-of-saved-waveforms talk in Section 12, it's important 🔐
save-waveform = "Save-Waveform" ":" BOOLEAN CRLF9.4.23. New-Audio-Channel 🎧
this header MAY be specified in a RECOGNIZE request, and it tells the server that the audio coming in from now on is from a different source, channel, or speaker 🗣️ if the recognizer had collected input statistics or adaptation state, it MUST do whatever is appropriate for its recognition tech, which includes but isn't limited to throwing out that collected data before the RECOGNIZE request starts 🧹
heads up: if several resources share a media stream and they're collecting or using this data, and the client sends this header to just one of them, the reset applies to every resource using that shared stream 🫶 this is super useful when the client wants to reuse an open recognition session with an existing media session for multiple phone calls 📞✨
new-audio-channel = "New-Audio-Channel" ":" BOOLEAN
CRLF9.4.24. Speech-Language 🌍
this one sets the language of the recognition grammar data for a session or request, but only if the data doesn't already say what language it's in 🗺️ the value MUST follow RFC 5646 [RFC5646] 💅 this MAY occur in DEFINE-GRAMMAR, RECOGNIZE, SET-PARAMS, or GET-PARAMS requests. 🌸
speech-language = "Speech-Language" ":" 1*VCHAR CRLF9.4.25. Ver-Buffer-Utterance 🪣
this lets the client ask the server to buffer the utterance for this recognition request into a buffer that a co-resident verifier resource can use 🧺 the buffer is shared across resources in a session and gets allocated when a verifier resource is added to that session. the client MUST NOT send this header field unless a verifier resource is instantiated for the session 🚫 the buffer is released when that verifier resource is released from the session. bye buffer 👋
9.4.26. Recognition-Mode 🔁
this sets what mode RECOGNIZE runs in: "normal" or "hotword" 🎀 in "normal" mode, RECOGNIZE starts matching speech and DTMF against the grammars in the request. if any part of the speech doesn't match the grammar, RECOGNIZE finishes with a no-match status ❌ timers might be active to detect speech (see Section 9.4.14), so RECOGNIZE can also finish because it timed out waiting for speech ⏰
in "hotword" mode it only looks for the specific keywords or DTMF sequences in the grammar, and it ignores silence or other speech in the audio stream 🔥 the default is "normal" 💖 this header field MAY occur on the RECOGNIZE method.
recognition-mode = "Recognition-Mode" ":"
"normal" / "hotword" CRLF9.4.27. Cancel-If-Queue 🚦
this one decides what happens when the client tries to start another RECOGNIZE while this one is already running on the resource 👀 it's a Boolean. "true" means the server MUST terminate this RECOGNIZE request, with a Completion-Cause of "cancelled", if the client sends another RECOGNIZE for the same resource 💀 "false" means this RECOGNIZE keeps going to completion, and any more RECOGNIZE requests for that resource get queued 📚
when the active RECOGNIZE stops or finishes with a successful match, the first one in the queue becomes active 🏃♀️ but if the current RECOGNIZE fails, every RECOGNIZE in the pending queue gets cancelled, and each one sends a RECOGNITION-COMPLETE event with a Completion-Cause of "cancelled" 😭 this header field MUST be present in every RECOGNIZE request. there is no default value, so you always have to say it 🫠
cancel-if-queue = "Cancel-If-Queue" ":" BOOLEAN CRLF9.4.28. Hotword-Max-Duration ⏫
this MAY be sent in a hotword mode RECOGNIZE request 🔥 it sets the max length of an utterance (in seconds) that gets considered for hotword recognition. together with Hotword-Min-Duration, it helps tune performance by keeping the recognizer from checking utterances that are too short or too long to be one of the hotwords in the grammar(s) 🎯 the value is in milliseconds, the default is implementation dependent, and if it shows up in a RECOGNIZE request for a mode other than "hotword", it just gets ignored 🙈
hotword-max-duration = "Hotword-Max-Duration" ":" 1*19DIGIT
CRLF9.4.29. Hotword-Min-Duration ⏬
the little sibling of the one above 💕 this MAY be sent in a hotword mode RECOGNIZE request, and it sets the minimum length of an utterance (in seconds) that gets considered for hotword recognition. same deal: it works with Hotword-Max-Duration to keep things efficient 🧚♀️ the value is in milliseconds, the default is implementation dependent, and it's ignored if the RECOGNIZE request is for a mode other than "hotword" 🌷
hotword-min-duration = "Hotword-Min-Duration" ":" 1*19DIGIT CRLF9.4.30. Interpret-Text 💬
this header points to the text you want natural language interpretation for 🧠 the value is either a URI or plain text. if it's a URI, it MUST be a Content-ID that refers to an entity of type 'text/plain' in the body of the message 📎 otherwise the server MUST treat the value as the text to interpret. this header field MUST be used when invoking the INTERPRET method. non-negotiable, bestie 🫡
interpret-text = "Interpret-Text" ":" 1*VCHAR CRLF9.4.31. DTMF-Buffer-Time 🧺
this is the type-ahead buffer for the recognizer, and it's measured in milliseconds 🕰️ it MAY be specified in a GET-PARAMS or SET-PARAMS method. the buffer collects DTMF digits as they get pressed, even when no RECOGNIZE command is active 🎹 when a RECOGNIZE method comes in, it MUST look at this buffer to match the request. if the digits in there aren't enough, it can keep listening for more digits to match the grammar. the default size is platform specific, so it depends on the platform 🌈
dtmf-buffer-time = "DTMF-Buffer-Time" ":" 1*19DIGIT CRLF9.4.32. Clear-DTMF-Buffer ✨
okay so this header field is allowed in a RECOGNIZE method 💅 and its job is to tell the recognizer to wipe the DTMF type-ahead buffer before it starts the RECOGNIZE. the default is false, which means the buffer stays as it is. if you set it to true, the RECOGNIZE clears the DTMF buffer first, so any digits the caller pressed before the RECOGNIZE command came in get tossed 🗑️ no cap, that's the whole point.
clear-dtmf-buffer = "Clear-DTMF-Buffer" ":" BOOLEAN CRLF9.4.33. Early-No-Match 🦋
this one is allowed in a RECOGNIZE method too, and it tells the recognizer that it MUST NOT wait for the end of speech before it processes the collected speech to match against the active grammars 🙈 when the value is true, the recognizer MUST do early matching. the default is false if you don't set it. and if the recognizer can't process the collected audio before the end of speech, it's totally fine to just ignore this header field 💖
early-no-match = "Early-No-Match" ":" BOOLEAN CRLF9.4.34. Num-Min-Consistent-Pronunciations 🎀
this header field MAY show up in START-PHRASE-ENROLLMENT, SET-PARAMS, or GET-PARAMS. it sets the minimum number of consistent pronunciations you need before a new phrase counts as voice enrolled 🗣️ the minimum is 1, and the default is implementation specific and MAY be greater than 1. so it's not always just one, babe 🌸
num-min-consistent-pronunciations =
"Num-Min-Consistent-Pronunciations" ":" 1*19DIGIT CRLF9.4.35. Consistency-Threshold 🌈
also MAY be sent with START-PHRASE-ENROLLMENT, SET-PARAMS, or GET-PARAMS ✨ during enrollment, this one decides how close an utterance has to sound to an earlier pronunciation of the same phrase before it gets called "consistent" 👑 the higher the threshold, the tighter the match has to be. the range is a float from 0.0 to 1.0, and the default is implementation specific. main character energy for accuracy, honestly 😭
consistency-threshold = "Consistency-Threshold" ":" FLOAT CRLF9.4.36. Clash-Threshold 💀
this one MAY be sent in START-PHRASE-ENROLLMENT, SET-PARAMS, or GET-PARAMS too 🫶 during voice enrollment, it sets how similar two different phrases can sound before they count as clashing. like "John Smith" and "Jon Smits" can sound so alike that they're hard to tell apart, iykyk 🙃 a smaller threshold means fewer clashes get flagged. the range is a float from 0.0 to 1.0, and the default is implementation specific. if you set it to 0, clash testing is turned off completely 🍓
clash-threshold = "Clash-Threshold" ":" FLOAT CRLF9.4.37. Personal-Grammar-URI 🧚
this header field picks the speaker-trained grammar that gets used or referenced during enrollment, and new phrases get added to it while enrolling 📇 for example, a contact list for a user named "Jeff" could live at http://myserver.example.com/myenrollmentdb/jeff-list. the generated grammar syntax MAY be implementation specific. there's no default value. this header field MAY be sent as part of the START-PHRASE-ENROLLMENT, SET-PARAMS, or GET-PARAMS method 💜
personal-grammar-uri = "Personal-Grammar-URI" ":" uri CRLF9.4.38. Enroll-Utterance ⭐
this header field MAY be specified in the RECOGNIZE method. if it's set to true and an Enrollment is active, the RECOGNIZE command MUST add the collected utterance to the personal grammar being enrolled 🎯 how that happens is engine specific and may be an area of future standardization, so stay tuned 👀 the default is false.
enroll-utterance = "Enroll-Utterance" ":" BOOLEAN CRLF9.4.39. Phrase-Id 🍓
this header field, in a request, identifies a phrase in an existing personal grammar that you want to enroll 💅 it also comes back to the client in the RECOGNIZE complete event. it MAY occur in START-PHRASE-ENROLLMENT, MODIFY-PHRASE, or DELETE-PHRASE requests. there's no default value.
phrase-id = "Phrase-ID" ":" 1*VCHAR CRLF9.4.40. Phrase-NL 🌷
this string is the interpreted text that gets returned when the phrase is recognized ✨ it MAY occur in START-PHRASE-ENROLLMENT and MODIFY-PHRASE requests. there's no default value here either, bestie 🙈
phrase-nl = "Phrase-NL" ":" 1*UTFCHAR CRLF9.4.41. Weight 🏋️
okay this one is giving math-nerd realness 🤓 the value is the occurrence likelihood of a phrase in an enrolled grammar. when you enroll, the system basically builds a grammar segment that's a list of possible match phrases, kind of like a dynamic <one-of> tag from the W3C grammar spec. each enrolled phrase becomes an <item> in that list. this header field lets you give a weight to that phrase's <item> entry. grammar weights get normalized so they add up to one at compile time, which means a weight of 1 for every phrase just means they're all equally likely 🌈 it MAY occur in START-PHRASE-ENROLLMENT and MODIFY-PHRASE requests, and the default is implementation specific.
weight = "Weight" ":" FLOAT CRLF9.4.42. Save-Best-Waveform 🎧
this lets the client ask the recognizer to save the audio for the best repetition of the phrase from the enrollment session 💖 the recognizer MUST attempt to record the recognized audio and make it available as a URI in the Waveform-URI header field of the END-PHRASE-ENROLLMENT response. if the recording broke or the audio just isn't available, the recognizer MUST return an empty Waveform-URI header field 😭 this header field MAY occur in START-PHRASE-ENROLLMENT, SET-PARAMS, and GET-PARAMS.
save-best-waveform = "Save-Best-Waveform" ":" BOOLEAN CRLF9.4.43. New-Phrase-Id 🔁
this one replaces the ID that identifies the phrase in a personal grammar ✨ the recognizer sends back the new ID when you're using an enrollment grammar. it MAY occur in MODIFY-PHRASE requests.
new-phrase-id = "New-Phrase-ID" ":" 1*VCHAR CRLF9.4.44. Confusable-Phrases-URI 🚫
this header field points to a grammar that defines invalid phrases for enrollment 🙅 for example, typical apps don't let you enroll a phrase that's also a command word, which is valid tea 🫖 it MAY occur in RECOGNIZE requests that are part of an enrollment session.
confusable-phrases-uri = "Confusable-Phrases-URI" ":" uri CRLF9.4.45. Abort-Phrase-Enrollment 🛑
this one MAY be specified in the END-PHRASE-ENROLLMENT method to abort the enrollment instead of committing the phrase to the personal grammar 🥲 basically, you're hitting undo 💅
abort-phrase-enrollment = "Abort-Phrase-Enrollment" ":"
BOOLEAN CRLF9.5. Recognizer Message Body 💜
a recognizer message can carry extra data tied to the request, response, or event 📦 the client MAY provide the grammar to be recognized in DEFINE-GRAMMAR or RECOGNIZE requests. when you send one or more grammars with DEFINE-GRAMMAR, the server MUST attempt to fetch, compile, and optimize the grammar before it sends back its response to DEFINE-GRAMMAR 🔥 a RECOGNIZE request MUST completely specify the grammars that are active during the recognition operation, except when RECOGNIZE is being used to enroll a grammar. during grammar enrollment, those grammars are OPTIONAL 🫧 the server resource sends recognition results in the RECOGNITION-COMPLETE event and the GET-RESULT response. grammars and recognition results both live in the message body of the matching MRCPv2 messages, and that's the tea ☕✨
9.5.1. Recognizer Grammar Data 🧚
Okay bestie, grammar data going from client to server can be sent inline or by reference 💖 Either way it rides along as typed media entities in the message body of a RECOGNIZE or DEFINE-GRAMMAR request. All MRCPv2 servers MUST accept grammars in the XML form (media type 'application/srgs+xml') of the W3C's XML-based Speech Grammar Markup Format (SRGS) [W3C.REC-speech-grammar-20040316] and MAY accept grammars in other formats. Examples include but are not limited to:
- the ABNF form (media type
'application/srgs') of SRGS - Sun's Java Speech Grammar Format (JSGF) [refs.javaSpeechGrammarFormat]
Additionally, MRCPv2 servers MAY support the Semantic Interpretation for Speech Recognition (SISR) [W3C.REC-semantic-interpretation-20070405] specification ✨
When a grammar is specified inline in the request, the client MUST provide a Content-ID for that grammar as part of the content header fields. If there is no space on the server to store the inline grammar, the request MUST return with a Completion-Cause code of 016 "grammar-definition-failure". Otherwise, the server MUST associate the inline grammar block with that Content-ID and MUST store it on the server for the duration of the session. However, if the Content-ID is redefined later in the session through a subsequent DEFINE-GRAMMAR, the inline grammar previously associated with the Content-ID MUST be freed. If the Content-ID is redefined through a subsequent DEFINE-GRAMMAR with an empty message body (i.e., no grammar definition), then in addition to freeing any grammar previously associated with the Content-ID, the server MUST clear all bindings and associations to the Content-ID. Unless and until subsequently redefined, this URI MUST be interpreted by the server as one that has never been set. 🙈
Grammars that have been associated with a Content-ID can be referenced through the 'session' URI scheme (see Section 13.6). For example:
session:help@root-level.storeGrammar data MAY be specified using external URI references. To do so, the client uses a body of media type 'text/uri-list' (see RFC 2483 [RFC2483]) to list the one or more URIs that point to the grammar data. The client can use a body of media type 'text/grammar-ref-list' (see Section 13.5.1) if it wants to assign weights to the list of grammar URI. All MRCPv2 servers MUST support grammar access using the 'http' and 'https' URI schemes. 🦋
If the grammar data the client wishes to be used on a request consists of a mix of URI and inline grammar data, the client uses the 'multipart/mixed' media type to enclose the 'text/uri-list', 'application/srgs', or 'application/srgs+xml' content entities. The character set and encoding used in the grammar data are specified using to standard media type definitions.
When more than one grammar URI or inline grammar block is specified in a message body of the RECOGNIZE request, the server interprets this as a list of grammar alternatives to match against. 💅
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0" root="request">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
<!-- multiple language attachment to a token -->
<rule id="people1">
<token lexicon="en-US,fr-CA"> Robert </token>
</rule>
<!-- the equivalent single-language attachment expansion -->
<rule id="people2">
<one-of>
<item xml:lang="en-US">Robert</item>
<item xml:lang="fr-CA">Robert</item>
</one-of>
</rule>
</grammar>SRGS Grammar Example 🌸
Content-Type:text/uri-list
Content-Length:...
session:help@root-level.store
http://www.example.com/Directory-Name-List.grxml
http://www.example.com/Department-List.grxml
http://www.example.com/TAC-Contact-List.grxml
session:menu1@menu-level.storeGrammar Reference Example 🎀
Content-Type:multipart/mixed; boundary="break"
--break
Content-Type:text/uri-list
Content-Length:...
http://www.example.com/Directory-Name-List.grxml
http://www.example.com/Department-List.grxml
http://www.example.com/TAC-Contact-List.grxml
--break
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
<!-- multiple language attachment to a token -->
<rule id="people1">
<token lexicon="en-US,fr-CA"> Robert </token>
</rule>
<!-- the equivalent single-language attachment expansion -->
<rule id="people2">
<one-of>
<item xml:lang="en-US">Robert</item>
<item xml:lang="fr-CA">Robert</item>
</one-of>
</rule>
</grammar>
--break--Mixed Grammar Reference Example 🌈
9.5.2. Recognizer Result Data 💜
Recognition results come back to the client in the message body of either the RECOGNITION-COMPLETE event or the GET-RESULT response message, as described in Section 6.3. The element and attribute descriptions for the recognition portion of the NLSML format are in Section 9.6, and the normative schema definition is in Section 16.1 ✨
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="http://www.example.com/theYesNoGrammar">
<interpretation>
<instance>
<ex:response>yes</ex:response>
</instance>
<input>OK</input>
</interpretation>
</result>Result Example 💖
9.5.3. Enrollment Result Data 🍓
Enrollment results come back to the client in the message body of the RECOGNITION-COMPLETE event, as described in Section 6.3. The element and attribute descriptions for the enrollment portion of the NLSML format are in Section 9.7, and the normative schema definition is in Section 16.2. 🦋
9.5.4. Recognizer Context Block 🔥
Okay so when a client switches servers while working on the same incoming communication session, this header field lets it collect a block of opaque data from one server and hand it to another. This is super useful if the client needs different language support or because the server issued a redirect. The first recognizer resource may have collected acoustic and other data while running recognition methods, and after a server switch, passing that data along can help the recognizer resource on the new server do better recognition. This block of data is implementation specific and MUST be carried as media type 'application/octets' in the body of the message. 🧠
This block of data is communicated in the SET-PARAMS and GET-PARAMS method/response messages. In the GET-PARAMS method, if an empty Recognizer-Context-Block header field is present, then the recognizer SHOULD return its vendor-specific context block, if any, in the message body as an entity of media type 'application/octets' with a specific Content-ID. The Content-ID value MUST also be specified in the Recognizer-Context-Block header field in the GET-PARAMS response. The SET-PARAMS request wishing to provide this vendor-specific data MUST send it in the message body as a typed entity with the same Content-ID that it received from the GET-PARAMS. The Content-ID MUST also be sent in the Recognizer-Context-Block header field of the SET-PARAMS message. 👑
Each speech recognition implementation choosing to use this mechanism to hand off recognizer context data among servers MUST distinguish its implementation-specific block of data from other implementations by choosing a Content-ID that is recognizable among the participating servers and unlikely to collide with values chosen by another implementation. 🫶
9.6. Recognizer Results ⭐
The recognizer portion of NLSML (see Section 6.3.1) represents information automatically extracted from a user's utterances by a semantic interpretation component. Here "utterance" is meant in the general sense of a meaningful user input in any modality supported by the MRCPv2 implementation. 💅
9.6.1. Markup Functions 🌸
MRCPv2 recognizer resources use the Natural Language Semantics Markup Language (NLSML) to interpret natural language speech input and to format that interpretation for an MRCPv2 client to consume. ✨
The elements of the markup fall into three general functional categories: interpretation, side information, and multi-modal integration. 🎀
9.6.1.1. Interpretation 🧚
Elements and attributes represent the semantics of a user's utterance, including the <result>, <interpretation>, and <instance> elements. The <result> element contains the full result of processing one utterance. It MAY contain multiple <interpretation> elements if the interpretation of the utterance results in multiple alternative meanings due to uncertainty in speech recognition or natural language understanding. There are at least two reasons for providing multiple interpretations:
- The client application might have additional information, for example, information from a database, that would allow it to select a preferred interpretation from among the possible interpretations returned from the semantic interpreter.
- A client-based dialog manager (e.g., VoiceXML [W3C.REC-voicexml20-20040316]) that was unable to select between several competing interpretations could use this information to go back to the user and find out what was intended. For example, it could issue a SPEAK request to a synthesizer resource to emit "Did you say 'Boston' or 'Austin'?" 😭
9.6.1.2. Side Information
okay so side information is all the extra stuff that comes along with an interpretation, on top of the interpretation itself 💅✨ it covers three things:
- Whether an interpretation was achieved at all (the
<nomatch>element), plus how confident the system is in an interpretation (the "confidence" attribute of<interpretation>) 🎀 - Alternative interpretations (
<interpretation>) 🦋 - Input formats and Automatic Speech Recognition (ASR) information, which is the
<input>element, representing the input that went to the semantic interpreter 🫶
9.6.1.3. Multi-Modal Integration
bestie, when more than one modality is available for input, the interpretations of those inputs need to be coordinated 🌈 the "mode" attribute of <input> handles that by saying whether the utterance came in through speech, DTMF, pointing, and so on 🎀 the "timestamp-start" and "timestamp-end" attributes of <input> add the temporal coordination too, by saying when each input happened ⏰💖
9.6.2. Overview of Recognizer Result Elements and Their Relationships
the recognizer elements in NLSML sort into two categories, and I'm obsessed with how clean this is 😭🧚
- a description of the input that was processed, and
- a description of the meaning that was pulled out of the input.
every element has its attributes listed right next to it 🌸 some elements can also hold multiple instances of other elements. for example, a <result> can contain multiple <interpretation> elements, and each one counts as an alternative 💜 similarly, <input> can contain multiple child <input> elements, and those are treated as cumulative 🍓
here's a super simple example to show the basic usage. the utterance is "OK", and it gets interpreted as "yes" ✨ the example shows how that utterance and its interpretation look in NLSML markup 💅
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="http://www.example.com/theYesNoGrammar">
<interpretation>
<instance>
<ex:response>yes</ex:response>
</instance>
<input>OK</input>
</interpretation>
</result>so this example has only the minimum required info 🥺 there's an overall <result> element holding one interpretation and one input element. the interpretation has the application-specific element "<response>", which is the semantically interpreted result 🔥
9.6.3. Elements and Attributes
9.6.3.1. <result> Root Element
the root element of the markup is <result> 👑 it includes one or more <interpretation> elements. multiple interpretations can show up because of ambiguities in the input or in the semantic interpretation 🌈 if the "grammar" attribute doesn't apply to all the interpretations in the result, you can override it for individual interpretations at the <interpretation> level 🦋
Attributes:
- grammar: the grammar or recognition rule that this result matched 💖 the format of the grammar attribute matches the rule reference semantics defined in the grammar specification. specifically, the rule reference is in the external XML form for grammar rule references 🎀 the markup interpreter needs to know which grammar rule the utterance matched, because multiple rules may be active at the same time 😭 the value is the grammar URI that the markup interpreter used to specify the grammar. the grammar can be overridden by a grammar attribute in the
<interpretation>element if the input was ambiguous about which grammar it matched 🙈 and if every interpretation element inside the result contains its own grammar attribute, the attribute can be dropped from the result element 🫶
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
grammar="http://www.example.com/grammar">
<interpretation>
....
</interpretation>
</result>9.6.3.2. <interpretation> Element
an <interpretation> element contains a single semantic interpretation 🦋
Attributes:
- confidence: a float value from 0.0-1.0 showing the semantic analyzer's confidence in this interpretation 💅 a value of 1.0 means maximum confidence 💯 the values are implementation dependent, but they're meant to line up with the value interpretation for the confidence MRCPv2 header field defined in Section 9.4.1 ✨ this attribute is OPTIONAL 🌸
- grammar: the grammar or recognition rule matched by this interpretation, if you need to override the grammar specification at the
<interpretation>level 🍓 this attribute is only needed under<interpretation>if it's necessary to override a grammar that was defined at the<result>level 👑 the grammar attribute on the interpretation element is optional if and only if the grammar attribute is specified in the<result>element 🫶
Interpretations MUST be sorted best-first by some measure of "goodness" 🔥 the goodness measure is "confidence" if it's present. otherwise, it's some implementation-specific indication of quality 🌈
the grammar is expected to be specified most often at the <result> level 💖 but it can be overridden at the <interpretation> level, because different interpretations may match different grammar rules 🎀
the <interpretation> element includes an optional <input> element containing the input being analyzed, and at least one <instance> element containing the interpretation of the utterance 🧚
<interpretation confidence="0.75"
grammar="http://www.example.com/grammar">
...
</interpretation>9.6.3.3. <instance> Element
the <instance> element holds the interpretation of the utterance 💜 when the Semantic Interpretation for Speech Recognition format is used, the <instance> element contains the XML serialization of the result, using the approach defined in that specification ✨ when the grammar has semantic markup that doesn't create semantic objects, but only does a semantic translation of part of the input (like turning "coke" into "coca-cola"), the instance contains the whole input with the translation applied 🦋 the NLSML looks like the markup in Figure 2 below 🎀 if there are no semantic objects created and no semantic translation, the instance value is the same as the input value 🌸
Attributes:
- confidence: each element of the instance MAY have a confidence attribute, defined in the NLSML namespace 💅 the confidence attribute contains a float value in the range from 0.0-1.0, reflecting the system's confidence in the analysis of that slot 💖 a value of 1.0 means maximum confidence 💯 the values are implementation dependent, but are intended to align with the value interpretation for the MRCPv2 header field Confidence-Threshold defined in Section 9.4.1 🔥 this attribute is OPTIONAL ✨
<instance>
<nameAddress>
<street confidence="0.75">123 Maple Street</street>
<city>Mill Valley</city>
<state>CA</state>
<zip>90952</zip>
</nameAddress>
</instance>
<input>
My address is 123 Maple Street,
Mill Valley, California, 90952
</input>
<instance>
I would like to buy a coca-cola
</instance>
<input>
I would like to buy a coke
</input>Figure 2: NSLML Example 🍓
9.6.3.4. <input> Element
the <input> element is the text representation of a user's input 🌈 it includes an optional "confidence" attribute, which tells you the recognizer's confidence in the recognition result. this is different from the confidence in the interpretation, which lives in the "confidence" attribute of <interpretation> 🎀 optional "timestamp-start" and "timestamp-end" attributes show the start and end times of a spoken utterance, in ISO 8601 format [ISO.8601.1988] ⏰✨
Attributes:
- timestamp-start: the time at which the input began. (optional) 🦋
- timestamp-end: the time at which the input ended. (optional) 🦋
- mode: the modality of the input, for example speech, DTMF, etc. (optional) 💖
- confidence: the confidence of the recognizer in the correctness of the input, in the range 0.0 to 1.0. (optional) 🫶
note that it may not make sense for temporally overlapping inputs to have the same mode 🙈 however, this constraint is not expected to be enforced by implementations 💅
when there's no time zone designator, ISO 8601 time representations default to local time 🌸
there are three possible formats for the <input> element 👑
- the
<input>element can contain simple text:
<input>onions</input>a future possibility is for <input> to contain not only text but also additional markup that represents prosodic information that was in the original utterance and extracted by the speech recognizer 🎀 this depends on the availability of ASRs that can produce prosodic information. MRCPv2 clients MUST be prepared to receive such markup and MAY make use of it 🔥
- an
<input>tag can also contain additional<input>tags 🧚 having additional input elements lets the representation support future multi-modal inputs, as well as finer-grained speech information, like timestamps for individual words and word-level confidences ✨
<input>
<input mode="speech" confidence="0.5"
timestamp-start="2000-04-03T0:00:00"
timestamp-end="2000-04-03T0:00:00.2">fried</input>
<input mode="speech" confidence="1.0"
timestamp-start="2000-04-03T0:00:00.25"
timestamp-end="2000-04-03T0:00:00.6">onions</input>
</input>- finally, the
<input>element can contain<nomatch>and<noinput>elements 💜 these describe situations where the speech recognizer received input it couldn't process, or didn't receive any input at all, respectively 😭
9.6.3.5. <nomatch> Element
the <nomatch> element under <input> is used to indicate that the semantic interpreter couldn't successfully match any input with confidence above the threshold 🙈 it can optionally contain the text of the best of the (rejected) matches 🍓
<interpretation>
<instance/>
<input confidence="0.1">
<nomatch/>
</input>
</interpretation>
<interpretation>
<instance/>
<input mode="speech" confidence="0.1">
<nomatch>I want to go to New York</nomatch>
</input>
</interpretation>9.6.3.6. <noinput> Element 🙈
okay so <noinput> is the "...hello? anyone there?" element 😭 It tells you there was no input at all, which means the speech recognizer hit a timeout because it was silent 🔇 Here's the shape:
<interpretation>
<instance/>
<input>
<noinput/>
</input>
</interpretation>If there are multiple levels of inputs, the most natural place for <nomatch> and <noinput> elements to appear is under the highest level of <input> for <noinput>, and under the appropriate level of <interpretation> for <nomatch>. So <noinput> means "no input at all" 🫥 and <nomatch> means "no match in speech modality" or "no match in DTMF modality" 💅 For example, to represent garbled speech combined with DTMF "1 2 3 4", the markup would be:
<input>
<input mode="speech"><nomatch/></input>
<input mode="dtmf">1 2 3 4</input>
</input>Note: while<noinput>could be represented as an attribute of input,<nomatch>cannot, since it could potentially include PCDATA content with the best match. For parallelism,<noinput>is also an element. 🎀
9.7. Enrollment Results 🌸
okay this whole section is about enrollment, which is basically when the user teaches the system a brand new phrase 🦋 Every enrollment element lives inside one <enrollment-result> element under <result>. The elements have the schema defined in Section 16.2, and these are the ones you get:
- num-clashes
- num-good-repetitions
- num-repetitions-still-needed
- consistency-status
- clash-phrase-ids
- transcriptions
- confusable-phrases
9.7.1. <num-clashes> Element 💜
<num-clashes> is the number of clashes this pronunciation has with other pronunciations in an active enrollment session 👀 The Clash-Threshold header field decides how sensitive that clash measurement is. Also, clash testing can be turned off completely by setting Clash-Threshold to 0 🙅♀️
9.7.2. <num-good-repetitions> Element ✨
<num-good-repetitions> holds the number of consistent pronunciations collected so far in an active enrollment session 🎉 bestie is making progress!
9.7.3. <num-repetitions-still-needed> Element 🍓
<num-repetitions-still-needed> is how many consistent pronunciations still have to be collected before the new phrase can be added to the enrollment grammar 😤 The number of consistent pronunciations required is set by the client in the request header field Num-Min-Consistent-Pronunciations. the returned value must be 0 before the client can successfully commit a phrase to the grammar by ending the enrollment session 💖
9.7.4. <consistency-status> Element 🌈
<consistency-status> shows how consistent the repetitions are while the system is learning a new phrase 🧐 It can be consistent, inconsistent, or undecided ✨
9.7.5. <clash-phrase-ids> Element 🦋
<clash-phrase-ids> contains the phrase IDs of any clashing pronunciation(s) 💅 If there are no clashes, this element is absent 🫶
9.7.6. <transcriptions> Element 🎀
<transcriptions> holds the transcriptions returned in the last repetition of the phrase being enrolled 📝
9.7.7. <confusable-phrases> Element 👑
<confusable-phrases> is a list of phrases from a command grammar that sound confusable with the phrase being added to the personal grammar 🙈 This element MAY be absent if there are no confusable phrases. Not me being so relieved about that 💀
9.8. DEFINE-GRAMMAR 🧚
okay so DEFINE-GRAMMAR is the client-to-server method that hands over one or more grammars and asks the server to access, fetch, and compile them as needed 🌸 The DEFINE-GRAMMAR method implementation MUST do a fetch of all external URIs that are part of that operation. If caching is implemented, this URI fetching MUST conform to the cache control hints and parameter header fields associated with the method, and that's how it decides whether the URIs get fetched from cache or from the external server 🧠 If those hints or parameters aren't specified in the method, the values set for the session using SET-PARAMS/GET-PARAMS apply. And if it wasn't set for the session either, the default values apply 💅
If the server resource is in the recognition state, the DEFINE-GRAMMAR request MUST respond with a failure status. Not it 😭
If the resource is in the idle state and is able to successfully process the supplied grammars, the server MUST return a success code status and the request-state MUST be COMPLETE ✨ iconic behavior, honestly.
If the recognizer resource could not define the grammar for some reason (for example, the download failed, the grammar failed to compile, or the grammar was in an unsupported form), the MRCPv2 response for the DEFINE-GRAMMAR method MUST contain a failure status-code of 407 and contain a Completion-Cause header field describing the failure reason 🥲
Here's a full example with all the messages:
C->S:MRCP/2.0 ... DEFINE-GRAMMAR 543257
Channel-Identifier:32AECB23433801@speechrecog
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
</grammar>
S->C:MRCP/2.0 ... 543257 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
C->S:MRCP/2.0 ... DEFINE-GRAMMAR 543258
Channel-Identifier:32AECB23433801@speechrecog
Content-Type:application/srgs+xml
Content-ID:<helpgrammar@root-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0">
<rule id="request">
I need help
</rule>
S->C:MRCP/2.0 ... 543258 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
C->S:MRCP/2.0 ... DEFINE-GRAMMAR 543259
Channel-Identifier:32AECB23433801@speechrecog
Content-Type:application/srgs+xml
Content-ID:<request2@field-level.store>
Content-Length:...
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE grammar PUBLIC "-//W3C//DTD GRAMMAR 1.0//EN"
"http://www.w3.org/TR/speech-grammar/grammar.dtd">
<grammar xmlns="http://www.w3.org/2001/06/grammar" xml:lang="en"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/06/grammar
http://www.w3.org/TR/speech-grammar/grammar.xsd"
version="1.0" mode="voice" root="basicCmd">
<meta name="author" content="Stephanie Williams"/>
<rule id="basicCmd" scope="public">
<example> please move the window </example>
<example> open a file </example>
<ruleref
uri="http://grammar.example.com/politeness.grxml#startPolite"/>
<ruleref uri="#command"/>
<ruleref
uri="http://grammar.example.com/politeness.grxml#endPolite"/>
</rule>
<rule id="command">
<ruleref uri="#action"/> <ruleref uri="#object"/>
</rule>
<rule id="action">
<one-of>
<item weight="10"> open <tag>open</tag> </item>
<item weight="2"> close <tag>close</tag> </item>
<item weight="1"> delete <tag>delete</tag> </item>
<item weight="1"> move <tag>move</tag> </item>
</one-of>
</rule>
<rule id="object">
<item repeat="0-1">
<one-of>
<item> the </item>
<item> a </item>
</one-of>
</item>
<one-of>
<item> window </item>
<item> file </item>
<item> menu </item>
</one-of>
</rule>
</grammar>
S->C:MRCP/2.0 ... 543259 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 successDefine Grammar Example
9.9. RECOGNIZE 🎤
okay so RECOGNIZE is the client-to-server method that tells the recognizer to start recognition and hands it one or more grammar references to match against the input media 🦋 It can carry header fields that control sensitivity, confidence level, and how much detail shows up in the results. Those header field values override whatever a previous SET-PARAMS method set 💅
The RECOGNIZE method can ask the recognizer resource to run in normal or hotword mode, as set by the Recognition-Mode header field 🔥 The default value is "normal". If the resource could not start a recognition, the server MUST respond with a failure status-code of 407 and a Completion-Cause header field in the response describing the cause of failure 😭
The RECOGNIZE request uses the message body to specify the grammars that apply to the request 🌈 There are three ways to specify the active grammar(s) for the request. If the client needs to explicitly control grammar weights for the recognition operation, it MUST employ method 3 below. The order of these grammars sets their precedence, which matters when more than one grammar in the list matches the speech: the grammar with the higher precedence is returned as the match. This precedence capability is super useful in applications like VoiceXML browsers, where you order grammars specified at the dialog, document, and root level of a VoiceXML application 👑
- The grammar MAY be placed directly in the message body as typed content. If more than one grammar is included in the body, the order of inclusion controls the corresponding precedence for the grammars during recognition, with earlier grammars in the body having a higher precedence than later ones.
- The body MAY contain a list of grammar URIs specified in content of media type 'text/uri-list' [RFC2483]. The order of the URIs determines the corresponding precedence for the grammars during recognition, with highest precedence first and decreasing for each URI thereafter. 🫶
- okay so the body MAY contain a list of grammar URIs in the content of the media type
text/grammar-ref-list📜 This type lets you list grammar URIs and give each one a weight 💅 That weight means the same thing as the weights in Section 2.4.1 of the Speech Grammar Markup Format (SRGS) [W3C.REC-speech-grammar-20040316] ✨
Okay, so on top of doing the recognition itself, the recognizer MUST also enroll the collected utterance in a personal grammar 🎀 That happens if the Enroll-Utterance header field is set to true AND an Enrollment is active (which was started by an earlier START-PHRASE-ENROLLMENT method). If both of those are true, and the RECOGNIZE request has a Content-ID header field, then the resulting grammar (the personal grammar tucked inside as a sub-grammar) can be referenced through the 'session' URI scheme (see Section 13.6) 🦋
If the resource manages to successfully start the recognition, the server MUST return a success status-code and a request-state of IN-PROGRESS ✨ This basically means the recognizer is active, so the client MUST be ready to receive more events with this request-id 🫶
If the resource is able to queue the request instead, the server MUST return a success code and a request-state of PENDING 🌸 That means the recognizer is already busy with another request, and this one is waiting its turn in the queue ⏳
If the resource can't start a recognition at all, the server MUST respond with a failure status-code of 407 and a Completion-Cause header field in the response that explains what went wrong 💀
Here's the vibe check: for the recognizer resource, RECOGNIZE and INTERPRET are the ONLY requests that return a request-state of IN-PROGRESS 👑 When recognition finishes (by matching one of the grammar alternatives, timing out without a match, or for some other reason), the recognizer resource MUST send the client a RECOGNITION-COMPLETE event (or INTERPRETATION-COMPLETE if INTERPRET was the request) with the result and a request-state of COMPLETE 🎉
Big grammars can take forever for the server to compile 😭 So for grammars you're going to use over and over, the client can speed things up by issuing a DEFINE-GRAMMAR request with the grammar ahead of time 🚀 Then the client can send the RECOGNIZE request and point at the grammar through the 'session' URI scheme (see Section 13.6). This also works in general if the client wants to repeat recognition with a previous inline grammar 🔁
The RECOGNIZE method implementation MUST fetch all of the external URIs that are part of that operation 📥 If caching is implemented, this fetching MUST follow the cache control hints and parameter header fields associated with the method when it decides whether to grab things from the cache or from the external server 🧠 If those hints or parameters aren't specified in the method, the values set for the session using SET-PARAMS/GET-PARAMS apply. And if those weren't set for the session either, their default values apply 💖
Here's a tricky bit, bestie 🙈 Since the audio and the messages travel over separate communication paths, there can be a race condition between the start of the audio flow and the receipt of the RECOGNIZE method. For example, if the client starts an audio flow at the exact same time it sends RECOGNIZE, either the audio or the RECOGNIZE could show up at the recognizer first. Another example: the client might keep sending audio nonstop and then signal the server to recognize with RECOGNIZE. How to resolve this is outside the scope of this specification, so not my problem 💅 The recognizer can expect the media to start flowing when it gets the RECOGNIZE request, but it MUST NOT buffer anything it receives beforehand, so the input timers behave the way application authors expect 🌈
When a RECOGNIZE method has been received, recognition starts on the stream ✨ The No-Input-Timer MUST be started at this point if the Start-Input-Timers header field is specified as "true" ⏱️ If that header field is set to "false", the No-Input-Timer MUST be started when the START-INPUT-TIMERS method arrives from the client 🫠 The Recognition-Timeout MUST be started when the recognition resource detects speech or a DTMF digit in the media stream 🎵
For recognition when not in hotword mode:
When the recognizer resource detects speech or a DTMF digit in the media stream, it MUST send the START-OF-INPUT event 📣 Once enough speech has been collected for the server to process, the recognizer can try to match it with the active grammars. If the speech collected so far fully matches any of the active grammars, the Speech-Complete-Timer is started 🏁 If it only matches partially with one or more of the active grammars, and more speech is needed before a full match happens, then the Speech-Incomplete-Timer is started ⏳
- When the No-Input-Timer expires, the recognizer MUST complete with a Completion-Cause code of "no-input-timeout" 🙉
- The recognizer MUST support detecting a no-match condition upon detecting end of speech. The recognizer MAY support detecting a no-match condition before waiting for end-of-speech. If it does, that capability is turned on by setting the
Early-No-Matchheader field to "true". Upon detecting a no-match condition, the RECOGNIZE MUST return with "no-match" 🙅♀️ - When the Speech-Incomplete-Timer expires, the recognizer SHOULD complete with a Completion-Cause code of "partial-match", unless the recognizer cannot tell a partial-match apart, in which case it MUST return a Completion-Cause code of "no-match". The recognizer MAY return results for the partially matched grammar 🧩
- When the Speech-Complete-Timer expires, the recognizer MUST complete with a Completion-Cause code of "success" 🥳
- When the Recognition-Timeout expires, one of the following MUST happen:
- 5.1. If there was a partial-match, the recognizer SHOULD complete with a Completion-Cause code of "partial-match-maxtime", unless the recognizer cannot differentiate a partial-match, in which case it MUST complete with a Completion-Cause code of "no-match-maxtime". The recognizer MAY return results for the partially matched grammar 🌷
- 5.2. If there was a full-match, the recognizer MUST complete with a Completion-Cause code of "success-maxtime" 🏆
- 5.3. If there was no match, the recognizer MUST complete with a Completion-Cause code of "no-match-maxtime" 🫥
For recognition in hotword mode:
Quick heads up 🚨 for recognition in hotword mode, the START-OF-INPUT event is NOT generated when speech or a DTMF digit is detected 🙃
- When the No-Input-Timer expires, the recognizer MUST complete with a Completion-Cause code of "no-input-timeout" 😴
- If a match happens at any point, the RECOGNIZE MUST complete with a Completion-Cause code of "success" 🎯
- When the Recognition-Timeout expires and there is NOT a match, the RECOGNIZE MUST complete with a Completion-Cause code of "hotword-maxtime" 🥀
- When the Recognition-Timeout expires and there IS a match, the RECOGNIZE MUST complete with a Completion-Cause code of "success-maxtime" 💫
- When the Recognition-Timeout is running but the detected speech/DTMF has not resulted in a match, the Recognition-Timeout MUST be stopped and reset. It MUST then be restarted when speech/DTMF is detected again 🔄
Below is a complete example of using RECOGNIZE 🎀 It shows the call to RECOGNIZE, the IN-PROGRESS and START-OF-INPUT status messages, and the final RECOGNITION-COMPLETE message with the result 💅
C->S:MRCP/2.0 ... RECOGNIZE 543257
Channel-Identifier:32AECB23433801@speechrecog
Confidence-Threshold:0.9
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0" root="request">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
</grammar>
S->C: MRCP/2.0 ... 543257 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
S->C:MRCP/2.0 ... START-OF-INPUT 543257 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
S->C:MRCP/2.0 ... RECOGNITION-COMPLETE 543257 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
Waveform-URI:<http://web.media.com/session123/audio.wav>;
size=424252;duration=2543
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="session:request1@form-level.store">
<interpretation>
<instance name="Person">
<ex:Person>
<ex:Name> Andre Roy </ex:Name>
</ex:Person>
</instance>
<input> may I speak to Andre Roy </input>
</interpretation>
</result>Below is an example of calling RECOGNIZE with a different grammar 🌈 No status or completion messages are shown in this one, although they would totally happen in normal usage ✨
C->S: MRCP/2.0 ... RECOGNIZE 543257
Channel-Identifier:32AECB23433801@speechrecog
Confidence-Threshold:0.9
Fetch-Timeout:20
Content-Type:application/srgs+xml
Content-Length:...
<?xml version="1.0"? Version="1.0" mode="voice"
root="Basic md">
<rule id="rule_list" scope="public">
<one-of>
<item weight=10>
<ruleref uri=
"http://grammar.example.com/world-cities.grxml#canada"/>
</item>
<item weight=1.5>
<ruleref uri=
"http://grammar.example.com/world-cities.grxml#america"/>
</item>
<item weight=0.5>
<ruleref uri=
"http://grammar.example.com/world-cities.grxml#india"/>
</item>
</one-of>
</rule>9.10. STOP 🛑
STOP is how the client tells the resource to halt recognition, but only if something is actually running 💅 If a RECOGNIZE request is active and the STOP successfully ends it, the response header section includes an Active-Request-Id-List header field holding the request-id of that RECOGNIZE. No RECOGNITION-COMPLETE event gets sent for the terminated request 🙈 If nothing was being recognized, the response MUST NOT include an Active-Request-Id-List header field. Either way, the response MUST include a status-code of 200 "Success" ✨
C->S: MRCP/2.0 ... RECOGNIZE 543257
Channel-Identifier:32AECB23433801@speechrecog
Confidence-Threshold:0.9
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0" root="request">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
</grammar>
S->C: MRCP/2.0 ... 543257 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
C->S: MRCP/2.0 ... STOP 543258 200
Channel-Identifier:32AECB23433801@speechrecog
S->C: MRCP/2.0 ... 543258 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Active-Request-Id-List:5432579.11. GET-RESULT 🔍
The client can send GET-RESULT MAY when the recognizer resource is in the recognized state. It lets the client pull back the results of a finished recognition, which is perfect if they want more alternatives or extra info 🧠 When the server gets it, the server re-computes the results and returns them according to the recognition constraints in that GET-RESULT request.
GET-RESULT can ask for stuff like a different confidence-threshold or n-best-list-length. This capability is OPTIONAL for MRCPv2 servers, and if the automatic speech recognition engine in the server doesn't support it, the server MUST return a status of unsupported feature 🙃
C->S: MRCP/2.0 ... GET-RESULT 543257
Channel-Identifier:32AECB23433801@speechrecog
Confidence-Threshold:0.9
S->C: MRCP/2.0 ... 543257 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="session:request1@form-level.store">
<interpretation>
<instance name="Person">
<ex:Person>
<ex:Name> Andre Roy </ex:Name>
</ex:Person>
</instance>
<input> may I speak to Andre Roy </input>
</interpretation>
</result>9.12. START-OF-INPUT 👀
This is an event from the server to the client saying the recognizer has spotted speech or a DTMF digit in the media stream 🎤 It's super useful for kill-on-barge-in when the synthesizer lives in a different session from the recognizer, so it has no clue there's incoming audio (see Section 8.4.2). In those cases the client has to play middleman 🫶 and respond to this event by issuing a BARGE-IN-OCCURRED event to the synthesizer. The recognizer resource also MUST send a Proxy-Sync-Id header field with a unique value for this event.
This event MUST be generated by the server, whether or not the synthesizer and recognizer are on the same server 💜
9.13. START-INPUT-TIMERS ⏱️
The client sends this request to the recognizer resource once it knows a kill-on-barge-in prompt has finished playing (see Section 8.4.2). It's handy when the recognition and synthesizer engines aren't in the same session 🎀 While a kill-on-barge-in prompt is playing, the client may want a RECOGNIZE request running at the same time so it can catch the barge-in. But it doesn't want the no-input timers to start until the prompt is done. The Start-Input-Timers header field in the RECOGNIZE request lets the client choose whether the timers start right away. If not, the recognizer resource MUST NOT start the timers until the client sends a START-INPUT-TIMERS method to it 🌸
9.14. RECOGNITION-COMPLETE 🎉
This is an event from the recognizer resource to the client saying recognition is done 🥳 The result rides along in the body of the MRCPv2 message. The request-state field MUST be COMPLETE, which means this is the last event for that request-id and that request is officially finished. The server MUST maintain the recognizer context, meaning the results and the audio waveform input, until the next RECOGNIZE request for that resource is issued or the session terminates 📼 If the server returns a URI to the audio waveform, it MUST do so in a Waveform-URI header field in the RECOGNITION-COMPLETE event. The client can use that URI to retrieve or play back the audio 🔊
Note, if an enrollment session was active, the RECOGNITION-COMPLETE event can contain either recognition or enrollment results depending on what was spoken 👀 Here's a full exchange with a recognition result first.
C->S: MRCP/2.0 ... RECOGNIZE 543257
Channel-Identifier:32AECB23433801@speechrecog
Confidence-Threshold:0.9
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0" root="request">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
</grammar>
S->C: MRCP/2.0 ... 543257 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
S->C: MRCP/2.0 ... START-OF-INPUT 543257 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
S->C: MRCP/2.0 ... RECOGNITION-COMPLETE 543257 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
Waveform-URI:<http://web.media.com/session123/audio.wav>;
size=342456;duration=25435
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="session:request1@form-level.store">
<interpretation>
<instance name="Person">
<ex:Person>
<ex:Name> Andre Roy </ex:Name>
</ex:Person>
</instance>
<input> may I speak to Andre Roy </input>
</interpretation>
</result>If the result were an enrollment result instead, the final message from the server above could have looked like this 🧸
S->C: MRCP/2.0 ... RECOGNITION-COMPLETE 543257 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version= "1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
grammar="Personal-Grammar-URI">
<enrollment-result>
<num-clashes> 2 </num-clashes>
<num-good-repetitions> 1 </num-good-repetitions>
<num-repetitions-still-needed>
1
</num-repetitions-still-needed>
<consistency-status> consistent </consistency-status>
<clash-phrase-ids>
<item> Jeff </item> <item> Andre </item>
</clash-phrase-ids>
<transcriptions>
<item> m ay b r ow k er </item>
<item> m ax r aa k ah </item>
</transcriptions>
<confusable-phrases>
<item>
<phrase> call </phrase>
<confusion-level> 10 </confusion-level>
</item>
</confusable-phrases>
</enrollment-result>
</result>9.15. START-PHRASE-ENROLLMENT ✨
okay so START-PHRASE-ENROLLMENT is the client's way of saying "let's start a brand new phrase enrollment session" 💅 During that session the client can call RECOGNIZE over and over so the system can "learn" a new utterance for a grammar. The caller says the phrase a bunch of times, and then it gets saved into a personal grammar (aka a speaker-trained grammar) so the system can recognize it later 🎀
Only one enrollment session can be active per resource at a time 🙈 The Personal-Grammar-URI points at the grammar that gets used during enrollment to store the personal list of phrases. Once RECOGNIZE is called, the result comes back in a RECOGNITION-COMPLETE event, and it'll be either an enrollment result OR a recognition result for a regular recognition 🔥
Then END-PHRASE-ENROLLMENT ends the session. That's usually done after a run of successful RECOGNIZE calls ✨ You can use it to commit the new phrase to the personal grammar, or to abort the whole enrollment session if you change your mind 🦋
The grammar that'll hold the new phrase, named by Personal-Grammar-URI, gets created if it doesn't already exist. Also, the personal grammar MUST ONLY contain phrases added via a phrase enrollment session. Strict, but iconic 👑
The Phrase-ID you pass in is how the phrase is identified inside the grammar, and it's what comes back as the speech input when you do a RECOGNIZE on that grammar 🧚 The Phrase-NL gets returned in a RECOGNITION-COMPLETE event, the same way other Natural Language (NL) in a grammar does. The tag-format of this NL is implementation specific 💜
If the client has set Save-Best-Waveform to true, then the response after ending the phrase enrollment session MUST contain the location/URI of a recording of the best repetition of the learned phrase. Keep that recording in your back pocket 🌸
C->S: MRCP/2.0 ... START-PHRASE-ENROLLMENT 543258
Channel-Identifier:32AECB23433801@speechrecog
Num-Min-Consistent-Pronunciations:2
Consistency-Threshold:30
Clash-Threshold:12
Personal-Grammar-URI:<personal grammar uri>
Phrase-Id:<phrase id>
Phrase-NL:<NL phrase>
Weight:1
Save-Best-Waveform:true
S->C: MRCP/2.0 ... 543258 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog9.16. ENROLLMENT-ROLLBACK 🙃
ENROLLMENT-ROLLBACK throws out the last live utterance from the RECOGNIZE operation. Super handy when the caller says something messy, like non-speech noises, side-speech, commands, or an utterance from the RECOGNIZE grammar 🙈 Just know it doesn't keep a stack of rollback states, so there's no going back further than one step. Also, running ENROLLMENT-ROLLBACK twice in a row without a recognition in between does nothing the second time. Not me hitting undo and nothing happening 💀
C->S: MRCP/2.0 ... ENROLLMENT-ROLLBACK 543261
Channel-Identifier:32AECB23433801@speechrecog
S->C: MRCP/2.0 ... 543261 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog9.17. END-PHRASE-ENROLLMENT 🎬
The client MAY call END-PHRASE-ENROLLMENT ONLY during an active phrase enrollment session. It MUST NOT be called during an ongoing RECOGNIZE operation, so don't interrupt the RECOGNIZE mid-song 🎶 If you want to commit the new phrase, the client MAY call this method once successive calls to RECOGNIZE have succeeded and Num-Repetitions-Still-Needed has come back as 0 in the RECOGNITION-COMPLETE event. Or, if you'd rather bail, the client MAY abort the phrase enrollment session by calling this method with the Abort-Phrase-Enrollment header field 🦋
If the client set Save-Best-Waveform to "true" in the START-PHRASE-ENROLLMENT request, then the response MUST contain a Waveform-URI header whose value is the location/URI of a recording of the best repetition of the learned phrase 🎀
C->S: MRCP/2.0 ... END-PHRASE-ENROLLMENT 543262
Channel-Identifier:32AECB23433801@speechrecog
S->C: MRCP/2.0 ... 543262 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Waveform-URI:<http://mediaserver.com/recordings/file1324.wav>;
size=242453;duration=254329.18. MODIFY-PHRASE ✏️
MODIFY-PHRASE is the client's glow-up button 💖 It's sent from the client to the server to change the phrase ID, NL phrase, and/or weight for a given phrase in a personal grammar. If no fields are supplied, then calling this method has no effect. Nothing in, nothing out 🌈
C->S: MRCP/2.0 ... MODIFY-PHRASE 543265
Channel-Identifier:32AECB23433801@speechrecog
Personal-Grammar-URI:<personal grammar uri>
Phrase-Id:<phrase id>
New-Phrase-Id:<new phrase id>
Phrase-NL:<NL phrase>
Weight:1
S->C: MRCP/2.0 ... 543265 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog9.19. DELETE-PHRASE 🗑️
DELETE-PHRASE is sent from the client to the server to delete a phase that lives in a personal grammar and was added through voice enrollment or text enrollment. If the specified phrase doesn't exist, this method has no effect, so no drama 💅
C->S: MRCP/2.0 ... DELETE-PHRASE 543266
Channel-Identifier:32AECB23433801@speechrecog
Personal-Grammar-URI:<personal grammar uri>
Phrase-Id:<phrase id>
S->C: MRCP/2.0 ... 543266 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog9.20. INTERPRET 🧠
INTERPRET is the client sending the server an Interpret-Text header field with the text it wants semantically interpreted 💜 The result comes back via INTERPRETATION-COMPLETE, and it's very similar to what a RECOGNIZE method call returns. The only bits left out are the parts that are relevant to acoustic matching. The Interpret-Text header field MUST be included in the INTERPRET request, so don't forget it 🫶
Recognizer grammar data is treated the same way it is when you issue a RECOGNIZE method call. Same vibes, same treatment ✨
If a RECOGNIZE, RECORD, or another INTERPRET operation is already in progress for the resource, the server MUST reject the request with a response having a status-code of 402 "Method not valid in this state", and a COMPLETE request state. Double-booking is a no-go 🙅♀️
C->S: MRCP/2.0 ... INTERPRET 543266
Channel-Identifier:32AECB23433801@speechrecog
Interpret-Text:may I speak to Andre Roy
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0" root="request">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
</grammar>
S->C: MRCP/2.0 ... 543266 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
S->C: MRCP/2.0 ... INTERPRETATION-COMPLETE 543266 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="session:request1@form-level.store">
<interpretation>
<instance name="Person">
<ex:Person>
<ex:Name> Andre Roy </ex:Name>
</ex:Person>
</instance>
<input> may I speak to Andre Roy </input>
</interpretation>
</result>9.21. INTERPRETATION-COMPLETE 🎉
This event comes from the recognizer resource to the client and says the INTERPRET operation is complete 🌸 The interpretation result rides along in the body of the MRCP message. The request state MUST be set to COMPLETE. The Completion-Cause header field MUST be included in this event, and it MUST be set to an appropriate value from the list of cause codes. Every event needs its receipts 🧾
C->S: MRCP/2.0 ... INTERPRET 543266
Channel-Identifier:32AECB23433801@speechrecog
Interpret-Text:may I speak to Andre Roy
Content-Type:application/srgs+xml
Content-ID:<request1@form-level.store>
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0" root="request">
<!-- single language attachment to tokens -->
<rule id="yes">
<one-of>
<item xml:lang="fr-CA">oui</item>
<item xml:lang="en-US">yes</item>
</one-of>
</rule>
<!-- single language attachment to a rule expansion -->
<rule id="request">
may I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
</grammar>
S->C: MRCP/2.0 ... 543266 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
S->C: MRCP/2.0 ... INTERPRETATION-COMPLETE 543266 200 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="session:request1@form-level.store">
<interpretation>
<instance name="Person">
<ex:Person>
<ex:Name> Andre Roy </ex:Name>
</ex:Person>
</instance>
<input> may I speak to Andre Roy </input>
</interpretation>
</result>9.22. DTMF Detection 🔢
Digits that arrive as DTMF tones get delivered to the recognition resource in the MRCPv2 server through the RTP stream, according to RFC 4733 [RFC4733] 🎹 The Automatic Speech Recognizer (ASR) MUST support RFC 4733 to recognize digits, and it MAY support recognizing DTMF tones [Q.23] in the audio. So the baseline is digits via RFC 4733, and recognizing tones in the audio is a bonus 💖
10. Recorder Resource ✨
okay so the recorder resource is the one that captures received audio and video and stores it at a URI 🎙️🎀 the two main jobs are
- grabbing speech audio so it can be recognized later on, and
- recording voice or video mails 💌
both of these need extra powers beyond what something like RTSP [RFC2326] gives you, and that includes audio endpointing (basically figuring out when someone is talking and when they're not) 👂✨ video is OPTIONAL, mostly for video mails that need that same speech or audio processing 🎥
so a recorder MUST be able to do endpointing to cut out silence at the start and end of a recording 🧹 it MAY also trim silence from the middle too 🌸 but if it does that, the recorder MUST keep timing metadata so everyone knows the real timestamps of the recorded media ⏱️💖
and make sure you read Section 12 about how sensitive saved waveforms can be 🔐👀
10.1. Recorder State Machine 🦋
Idle Recording
State State
| |
|---------RECORD------->|
| |
|<------STOP------------|
| |
|<--RECORD-COMPLETE-----|
| |
| |--------|
| START-OF-INPUT |
| |------->|
| |
| |--------|
| START-INPUT-TIMERS |
| |------->|
| |
Recorder State Machine10.2. Recorder Methods 🧚
the recorder resource supports these methods, nothing more and nothing less 💅
recorder-method = "RECORD"
/ "STOP"
/ "START-INPUT-TIMERS"10.3. Recorder Events 🔔
the recorder resource can throw these events at you 🎉
recorder-event = "START-OF-INPUT"
/ "RECORD-COMPLETE"10.4. Recorder Header Fields 🎀
method invocations for the recorder can carry resource-specific header fields 💌 these hold request options and extra info that goes along with the Method, Response, or Event message they belong to ✨
recorder-header = sensitivity-level
/ no-input-timeout
/ completion-cause
/ completion-reason
/ failed-uri
/ failed-uri-cause
/ record-uri
/ media-type
/ max-time
/ trim-length
/ final-silence
/ capture-on-speech
/ ver-buffer-utterance
/ start-input-timers
/ new-audio-channel10.4.1. Sensitivity-Level 🔊
so background noise is a whole mess, and you don't want the recorder mistaking it for speech 🙅♀️ that's where a variable sound sensitivity level comes in 🎚️ the Sensitivity-Level header field is a float between 0.0 and 1.0, and the client uses it to set how sensitive the recorder is 💖 this header field MAY occur in RECORD, SET-PARAMS, or GET-PARAMS 📝 higher number means higher sensitivity 📈 and the default is implementation specific, so it's up to the implementation 🤷♀️
sensitivity-level = "Sensitivity-Level" ":" FLOAT CRLF10.4.2. No-Input-Timeout ⏳
okay so if recording starts and nobody says anything for a while, the recorder can send a RECORD-COMPLETE event to the client and end the record operation 😭 the No-Input-Timeout header field is what sets that timeout 🕰️ the value is in milliseconds 💜 this header field MAY occur in RECORD, SET-PARAMS, or GET-PARAMS 📝 the value goes from 0 up to an implementation-specific maximum 🌈 and the default is also implementation specific 🤍
no-input-timeout = "No-Input-Timeout" ":" 1*19DIGIT CRLF10.4.3. Completion-Cause 🏁
this one MUST be part of a RECORD-COMPLETE event that goes from the recorder resource to the client 💅 it tells you why the RECORD method finished 🎯 it MUST also be sent in the RECORD responses when they come back with a failure status and a COMPLETE state 🚨 in the ABNF, the 'cause-code' is a number picked from the Cause-Code column of the table below, and the 'cause-name' is the matching token from the Cause-Name column 🫶
completion-cause = "Completion-Cause" ":" cause-code SP
cause-name CRLF
cause-code = 3DIGIT
cause-name = *VCHAR
+------------+-----------------------+------------------------------+
| Cause-Code | Cause-Name | Description |
+------------+-----------------------+------------------------------+
| 000 | success-silence | RECORD completed with a |
| | | silence at the end. |
| 001 | success-maxtime | RECORD completed after |
| | | reaching maximum recording |
| | | time specified in record |
| | | method. |
| 002 | no-input-timeout | RECORD failed due to no |
| | | input. |
| 003 | uri-failure | Failure accessing the record |
| | | URI. |
| 004 | error | RECORD request terminated |
| | | prematurely due to a |
| | | recorder error. |
+------------+-----------------------+------------------------------+10.4.4. Completion-Reason 📝
this header field MAY show up in a RECORD-COMPLETE event coming from the recorder to the client 🌸 it has the reason text for why the RECORD request ended 💬 basically it's the failure explanation in plain words 🙈
the completion reason text is there for the client to use in logs and for debugging and instrumentation 🧪 and clients MUST NOT interpret it 🛑 hands off, bestie 🫶
completion-reason = "Completion-Reason" ":"
quoted-string CRLF10.4.5. Failed-URI 🚧
when a recorder method needs to post the audio to a URI and getting to that URI fails, the server MUST put the failed URI in this header field in the method response 😤
failed-uri = "Failed-URI" ":" absoluteURI CRLF10.4.6. Failed-URI-Cause 🔍
same situation as above, when posting to a URI fails 💔 this time the server MAY send along the URI-specific or protocol-specific response code in this header field 🔢 the value is encoded in UTF-8 (RFC 3629 [RFC3629]) so it works for any access protocol 🌍 some protocols don't use a numeric code and use a string instead, and that's exactly why 🙌
failed-uri-cause = "Failed-URI-Cause" ":" 1*UTFCHAR
CRLF10.4.7. Record-URI 🔗
when a recorder method includes this header field, the server MUST capture the audio and store it 💾 if the header field shows up with no value at all, the server MUST store the content locally and generate a URI pointing to it ✨ that URI comes back in either the STOP response or the RECORD-COMPLETE event 🎁 if the RECORD method's header field has a URI, the server MUST try to capture and store the audio at that location 📍 and if the header field isn't in the RECORD request at all, the server MUST capture the audio, MUST encode it, and MUST send it back in the STOP response or the RECORD-COMPLETE event as a message body 📦 in that case, the response carrying the audio MUST include a Content ID (cid) [RFC2392] value in this header pointing to the Content-ID in the message body 🫧
the server MUST also return the size in octets and the duration in milliseconds of the recorded audio waveform, as parameters on the header field 📏⏱️
implementations MUST support 'http' [RFC2616], 'https' [RFC2818], 'file' [RFC3986], and 'cid' [RFC2392] schemes in the URI 🚨 and yep, other schemes already exist in real implementations too 🌈
record-uri = "Record-URI" ":" ["<" uri ">"
";" "size" "=" 1*19DIGIT
";" "duration" "=" 1*19DIGIT] CRLF10.4.8. Media-Type 🎞️
a RECORD method MUST contain this header field, and it tells the server what media type the captured audio or video is 📼
media-type = "Media-Type" ":" media-type-value
CRLF10.4.9. Max-Time ⌛
when recording starts, this sets the longest the recording can be, in milliseconds 🕛 the clock starts when the actual capture and store begins, which is not necessarily when the RECORD method arrives 📬 it's measured before any silence suppression the recorder may have done 🤏 once that time runs out, the recording stops and the server MUST send a RECORD-COMPLETE event to the client with a request-state of COMPLETE 🏁 this header field MAY occur in RECORD, SET-PARAMS, or GET-PARAMS 📝 the value goes from 0 up to an implementation-specific maximum 🌈 and 0 means infinity, so the recording keeps going until one of the other stop conditions kicks in ♾️ the default is 0 🤍
max-time = "Max-Time" ":" 1*19DIGIT CRLF10.4.10. Trim-Length ✂️
this header field MAY be sent on a STOP method, and it says how much audio to trim off the end of the recording after the stop 🫧 the length is in milliseconds 🕐 the default is 0 💫
trim-length = "Trim-Length" ":" 1*19DIGIT CRLF10.4.11. Final-Silence 🤫
when the recorder starts and capture actually begins, this header field says how much silence in the audio counts as the end of the recording 🔚 this header field MAY occur in RECORD, SET-PARAMS, or GET-PARAMS 📝 the value goes from 0 up to an implementation-specific maximum 🌈 and it's in milliseconds ⏱️ 0 means infinity, so the recording keeps going until one of the other stop conditions is met ♾️ the default is implementation specific 🤷♀️
final-silence = "Final-Silence" ":" 1*19DIGIT CRLF10.4.12. Capture-On-Speech 🗣️
if it's "false", the recorder MUST start capturing right away once it's started 🏃♀️ if it's "true", the recorder MUST wait for the endpointing functionality to detect speech before it starts capturing 👂💕 this header field MAY occur in RECORD, SET-PARAMS, or GET-PARAMS 📝 the value is a Boolean 🔘 and the default is "false" 🤍
capture-on-speech = "Capture-On-Speech " ":" BOOLEAN CRLF10.4.13. Ver-Buffer-Utterance 🎀
okay so this one is giving "shared storage" energy 🧺 It's the same header field as the one you'll find on the verifier resource (see Section 11.4.14). It tells the server to stash the utterance from this recording request into the verification buffer. You can only send it if the verification buffer belongs to the session. That buffer is shared across resources in a session, so it's basically the group closet 👗 It gets created when a verifier resource joins the session and gets released when that verifier resource leaves. Iconic teamwork 🫶
10.4.14. Start-Input-Timers ⏱️
bestie, this one is so clever 🧠 You MAY send this header field as part of the RECORD request. When it's set to "false", the recorder starts the operation but holds off on starting the no-input timer until the client sends a START-INPUT-TIMERS request to it. This comes in clutch when the recorder and synthesizer aren't in the same session 🦋 Picture this: a kill-on-barge-in prompt is playing, and the client wants the RECORD request already running so it can catch the barge-in (see Section 8.4.2). But it also doesn't want the no-input timers ticking until the prompt is done. Both wants, one request, no drama 💅 The default value is "true".
start-input-timers = "Start-Input-Timers" ":"
BOOLEAN CRLF10.4.15. New-Audio-Channel 🌈
this header field is the exact same vibe as the one on the recognizer resource (see Section 9.4.23). Copy and paste energy, but make it cute 🥰
10.5. Recorder Message Body 📼
if the RECORD request didn't include a Record-URI header field, then the STOP response or the RECORD-COMPLETE event MUST contain a message body carrying the captured audio 🎧 In that case, the message carrying the audio has a Record-URI header field with a Content ID value that points at the message body entity holding the recording. See Section 10.4.7 for the details 💜
10.6. RECORD 🔴
the RECORD request puts the recorder resource into the recording state 🎬 Depending on which header fields you send, the resource might start recording right away, or it might wait for endpointing to detect speech first. Then the audio is handed to the client, either in the message body or wherever Record-URI says 🍓
The server MUST support the 'https' URI scheme and MAY support other schemes. Heads up, because voice recordings are seriously sensitive 🥺 any protocols used to dereference them SHOULD employ integrity and confidentiality, unless some other means is in place, like a controlled environment (see Section 4.2). Protect your bestie's voice 🔐
If a RECORD operation is already in progress, invoking this method causes the server to issue a response having a status-code of 402 "Method not valid in this state" and a request-state of COMPLETE. oop, double-booked 😭
If the Record-URI isn't valid, the response comes back with a status-code of 404 "Illegal Value for Header Field". If the server can't create the requested stored content, you get a status-code of 407 "Method or Operation Failed" 💀
If the type in the Media-Type header field isn't supported, the server MUST respond with a status-code of 409 "Unsupported Header Field Value" and include the Media-Type header field in its response. Not supported, periodt 🙅♀️
When the recording starts, the response shows an IN-PROGRESS request state. The server MAY generate a START-OF-INPUT event once it detects speech. When the recording wraps up, the server sends a RECORD-COMPLETE event ✨
C->S: MRCP/2.0 ... RECORD 543257
Channel-Identifier:32AECB23433802@recorder
Record-URI:<file://mediaserver/recordings/myfile.wav>
Media-Type:audio/wav
Capture-On-Speech:true
Final-Silence:300
Max-Time:6000
S->C: MRCP/2.0 ... 543257 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@recorder
S->C: MRCP/2.0 ... START-OF-INPUT 543257 IN-PROGRESS
Channel-Identifier:32AECB23433802@recorder
S->C: MRCP/2.0 ... RECORD-COMPLETE 543257 COMPLETE
Channel-Identifier:32AECB23433802@recorder
Completion-Cause:000 success-silence
Record-URI:<file://mediaserver/recordings/myfile.wav>;
size=242552;duration=25645RECORD Example 🎀
10.7. STOP 🛑
the STOP method moves the recorder from the recording state back to idle 🧘♀️ If a RECORD request is active and STOP successfully ends it, then the STOP response MUST contain an Active-Request-Id-List header field holding the RECORD request-id that got terminated. No RECORD-COMPLETE event gets sent for that terminated request. If nothing was being recorded, the response MUST NOT contain an Active-Request-Id-List header field, so don't sneak one in 🙈 If the recording was a success, the STOP response MUST contain a Record-URI header field pointing to the recorded audio content, or to a typed entity in the body of the STOP response holding the recording. The STOP method MAY have a Trim-Length header field, and if it does, that amount of audio gets chopped off the end of the recording after the stop ✂️ Either way, the response MUST contain a status-code of 200 "Success" ✅
C->S: MRCP/2.0 ... RECORD 543257
Channel-Identifier:32AECB23433802@recorder
Record-URI:<file://mediaserver/recordings/myfile.wav>
Capture-On-Speech:true
Final-Silence:300
Max-Time:6000
S->C: MRCP/2.0 ... 543257 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@recorder
S->C: MRCP/2.0 ... START-OF-INPUT 543257 IN-PROGRESS
Channel-Identifier:32AECB23433802@recorder
C->S: MRCP/2.0 ... STOP 543257
Channel-Identifier:32AECB23433802@recorder
Trim-Length:200
S->C: MRCP/2.0 ... 543257 200 COMPLETE
Channel-Identifier:32AECB23433802@recorder
Record-URI:<file://mediaserver/recordings/myfile.wav>;
size=324253;duration=24561
Active-Request-Id-List:543257STOP Example 🎬
10.8. RECORD-COMPLETE 🏁
if the recording finishes because of no input, silence after speech, or hitting the max-time, the server MUST generate the RECORD-COMPLETE event for the client with a request-state of COMPLETE 🥳 If the recording was a success, the RECORD-COMPLETE event carries a Record-URI header field pointing at the recorded audio file on the server, or at a typed entity in the message body with the recording inside 🎧
C->S: MRCP/2.0 ... RECORD 543257
Channel-Identifier:32AECB23433802@recorder
Record-URI:<file://mediaserver/recordings/myfile.wav>
Capture-On-Speech:true
Final-Silence:300
Max-Time:6000
S->C: MRCP/2.0 ... 543257 200 IN-PROGRESS
Channel-Identifier:32AECB23433802@recorder
S->C: MRCP/2.0 ... START-OF-INPUT 543257 IN-PROGRESS
Channel-Identifier:32AECB23433802@recorder
S->C: MRCP/2.0 ... RECORD-COMPLETE 543257 COMPLETE
Channel-Identifier:32AECB23433802@recorder
Completion-Cause:000 success
Record-URI:<file://mediaserver/recordings/myfile.wav>;
size=325325;duration=24652RECORD-COMPLETE Example 💖
10.9. START-INPUT-TIMERS ⏰
the client sends this request to the recorder resource once it notices that a kill-on-barge-in prompt has finished playing (see Section 8.4.2). It's useful when the recorder and synthesizer aren't in the same MRCPv2 session 🦋 While a kill-on-barge-in prompt plays, the client wants RECORD running so it can catch the barge-in, but it doesn't want the no-input timers starting yet. The Start-Input-Timers header field in the RECORD request is how the client says whether the timers should start. In that case, the recorder doesn't start the timers until the client sends a START-INPUT-TIMERS method to it 🌸
10.10. START-OF-INPUT 👂
the START-OF-INPUT event comes from the server to the client once it has detected speech. The recorder resource always returns this event when speech is detected 🎤 The recorder resource also MUST send a Proxy-Sync-Id header field with a unique value for this event 🔑
S->C: MRCP/2.0 ... START-OF-INPUT 543259 IN-PROGRESS
Channel-Identifier:32AECB23433801@recorder
Proxy-Sync-Id:98765432111. Speaker Verification and Identification ✨
okay so this section is all about the methods, responses and events MRCPv2 uses for speaker verification and identification 🎀
Speaker verification is a voice authentication method 🔐 that can identify the speaker so they can get access to sensitive info and transactions. Speech is a biometric, which means a bunch of important security stuff about biometric authentication applies here 😭 Implementers should read Section 12 of this document and the matching section of the SPEECHSC requirements [RFC4313] super carefully. Implementers and deployers are strongly encouraged to check the state of the art for new risks and solutions that might have popped up 🫶
In verification, a recorded utterance gets compared to a voiceprint that was stored earlier, and that voiceprint is tied to a claimed identity for that user 💅 Verification usually has two phases: a designation phase, where the claimed identity of the caller gets established, and an execution phase, where a voiceprint is either created (training) or used to check the claimed identity (verification) 🦋
Speaker identification is figuring out which member of a population an unknown speaker is 🌸 It does not use a claimed identity. When someone claims to belong to a group (like one of the owners of a joint bank account), a group authentication happens instead. This is usually a kind of verification that compares against more than one voice model, and it's sometimes called 'multi-verification' 👑 If the individual can be picked out from the group, that can be useful for apps where several users share the same access to some data or application ✨ Speaker identification and group authentication also run in two phases, a designation phase and an execution phase 💜
Here's the thing that's kind of iconic 🔥 From a functionality standpoint, identification can be thought of as a special case of group authentication (if the individual is identified) where the group is the entire population. The implementation can still be different from how group authentication is done, though 🙈 To handle single-voiceprint verification, verification against multiple voiceprints, group authentication, and identification all together, this spec gives you one set of methods. They take a list of identifiers, called "voiceprint identifiers", and return a list of identifiers, each with a score showing how well the input speech matched that identifier 🧚 The input and output lists don't have to match, so a vendor-specific group identifier can be used as input to say identification should happen 🌈 In this spec, "identification" and "multi-verification" mean the input represents a group (possibly the entire population) and results for multiple voiceprints may come back 💖
It's possible for a verifier resource to share the same session with a recognizer resource, or to work independently ⭐ To share the same session, the verifier and recognizer resources MUST be allocated from within the same SIP dialog. Otherwise, an independent verifier resource gets set up, either on the same physical server or a separate one 🍓 You can allocate both resources in the same INVITE, or allocate one first and the other later with a re-INVITE. Not me being obsessed with how flexible this is 😭
Some of the speaker verification methods, described below, only apply to a specific mode of operation 🦋
The verifier resource has a verification buffer associated with it (see Section 11.4.14) 🫶 This lets speech utterances get stored for verification, identification, or training later on. The buffer is owned by the verifier resource, but other input resources (like the recognizer resource or recorder resource) may write to it too. So speech received during a recognition or recording operation can be used later for verification, identification, or training 💅 Access to the buffer is limited to one operation at a time. So when the resource is doing read, write, or delete operations (like a RECOGNIZE with ver-buffer-utterance turned on), any other operation touching the buffer fails with a status-code of 402 💀 The verification buffer can be cleared by a CLEAR-BUFFER request from the client, and it gets freed when the verifier resource is deallocated or the session with the server ends 🌸
The verification buffer is different from collecting waveforms and processing them with either the real-time audio stream or stored audio, because this buffering doesn't just pile up speech in a buffer 👑 The verification buffer MAY contain additional info gathered by the recognizer resource that helps improve verification performance ✨
11.1. Speaker Verification State Machine 🎀
Speaker verification can run in a training or a verification session 💖 Starting one of these sessions does NOT change the state of the verifier resource, so it stays idle. Once a verification or training session starts, utterances get trained or verified by calling the VERIFY or VERIFY-FROM-BUFFER method. The verifier resource goes from IDLE to VERIFYING each time VERIFY or VERIFY-FROM-BUFFER is called 🔥
Idle Session Opened Verifying/Training
State State State
| | |
|--START-SESSION--->| |
| | |
| |----------| |
| | START-SESSION |
| |<---------| |
| | |
|<--END-SESSION-----| |
| | |
| |---------VERIFY--------->|
| | |
| |---VERIFY-FROM-BUFFER--->|
| | |
| |----------| |
| | VERIFY-ROLLBACK |
| |<---------| |
| | |
| | |--------|
| | GET-INTERMEDIATE-RESULT |
| | |------->|
| | |
| | |--------|
| | START-INPUT-TIMERS |
| | |------->|
| | |
| | |--------|
| | START-OF-INPUT |
| | |------->|
| | |
| |<-VERIFICATION-COMPLETE--|
| | |
| |<--------STOP------------|
| | |
| |----------| |
| | STOP |
| |<---------| |
| | |
|----------| | |
| STOP | |
|<---------| | |
| |----------| |
| | CLEAR-BUFFER |
| |<---------| |
| | |
|----------| | |
| CLEAR-BUFFER | |
|<---------| | |
| | |
| |----------| |
| | QUERY-VOICEPRINT |
| |<---------| |
| | |
|----------| | |
| QUERY-VOICEPRINT | |
|<---------| | |
| | |
| |----------| |
| | DELETE-VOICEPRINT |
| |<---------| |
| | |
|----------| | |
| DELETE-VOICEPRINT | |
|<---------| | |Verifier Resource State Machine
11.2. Speaker Verification Methods 💅
The verifier resource supports the following methods 🦋
verifier-method = "START-SESSION"
/ "END-SESSION"
/ "QUERY-VOICEPRINT"
/ "DELETE-VOICEPRINT"
/ "VERIFY"
/ "VERIFY-FROM-BUFFER"
/ "VERIFY-ROLLBACK"
/ "STOP"
/ "CLEAR-BUFFER"
/ "START-INPUT-TIMERS"
/ "GET-INTERMEDIATE-RESULT"These methods let the client control the mode and target of verification or identification operations within a session 🌈 All the verification input operations that happen within a session can be used to create, update, or validate against the voiceprint specified during the session 💜 At the start of each session, the verifier resource gets reset to the state it had before any previous verification session ✨
Verification and identification can run against live or buffered audio 🎀 The verifier resource gives you methods for collecting and evaluating live audio, and methods for controlling the verifier resource and adjusting its configured behavior 🫶
There are no dedicated methods for collecting buffered audio 🙈 Instead, you call VERIFY, RECOGNIZE, or RECORD (whichever fits the resource) with the header field Ver-Buffer-Utterance. Then, when the following method is called, verification runs on the set of buffered audio 🍓
- VERIFY-FROM-BUFFER
For verification of live audio utterances, use these methods:
- VERIFY
- START-INPUT-TIMERS
For configuring the verifier resource and establishing resource states, use these methods:
- START-SESSION
- END-SESSION
- QUERY-VOICEPRINT
- DELETE-VOICEPRINT
- VERIFY-ROLLBACK
- STOP
- CLEAR-BUFFER
The following method lets you poll a verification in progress for intermediate results 🧚
- GET-INTERMEDIATE-RESULT
11.3. Verification Events 🌸
The verifier resource generates the following events ⭐
verifier-event = "VERIFICATION-COMPLETE"
/ "START-OF-INPUT"11.4. Verification Header Fields 💖
okay so a verifier resource message can carry header fields with request options and extra info that boosts whatever Request, Response, or Event message it's attached to 💅
verification-header = repository-uri
/ voiceprint-identifier
/ verification-mode
/ adapt-model
/ abort-model
/ min-verification-score
/ num-min-verification-phrases
/ num-max-verification-phrases
/ no-input-timeout
/ save-waveform
/ media-type
/ waveform-uri
/ voiceprint-exists
/ ver-buffer-utterance
/ input-waveform-uri
/ completion-cause
/ completion-reason
/ speech-complete-timeout
/ new-audio-channel
/ abort-verification
/ start-input-timers11.4.1. Repository-URI 🗂️
this one picks the voiceprint repository you want to use or reference during speaker verification or identification ops. it is required in START-SESSION, QUERY-VOICEPRINT, and DELETE-VOICEPRINT, so don't forget it 🙈
repository-uri = "Repository-URI" ":" uri CRLF11.4.2. Voiceprint-Identifier 🪪
this is the claimed identity for verification apps, and it's giving main character energy ✨. The claimed identity MAY be used to specify an existing voiceprint or to establish a new voiceprint. This header field MUST be present in the QUERY-VOICEPRINT and DELETE-VOICEPRINT methods. The Voiceprint-Identifier MUST be present in the START-SESSION method for verification operations. For identification or multi-verification operations, this header field MAY contain a list of voiceprint identifiers separated by semicolons. For identification operations, the client MAY also specify a voiceprint group identifier instead of a list of voiceprint identifiers.
voiceprint-identifier = "Voiceprint-Identifier" ":"
vid *[";" vid] CRLF
vid = 1*VCHAR ["." 1*VCHAR]11.4.3. Verification-Mode 🎀
this header field sets the mode of the verifier resource, and START-SESSION is the method that sets it. The acceptable values tell you whether the session will train a voiceprint ("train") or verify/identify using an existing voiceprint ("verify").
training and verification sessions both need the voiceprint Repository-URI specified in START-SESSION. ngl though, in lots of setups the system doesn't know the speaker's claimed identity until a recognition operation has done its thing, like recognizing an account number the user wants to get into 🔐. so the first few utterances of a dialog can be recognized and verified, the verifier resource on the MRCPv2 server keeps a buffer 🧺. the MRCPv2 server piles up recognized utterances in that buffer, and the client can later run a verification method and apply the buffered utterances to the current verification session.
some voice user interfaces need extra user input that should NOT get verified 🙅♀️. like, if the caller's input was recognized with low confidence, there might be a confirmation cycle. in those cases, the client SHOULD NOT execute the VERIFY or VERIFY-FROM-BUFFER methods to collect and analyze the caller's input. a separate recognizer resource can analyze the caller's response with zero help from the verifier resource 💅.
once both of these are true:
- the voiceprint identity has been successfully established through the Voiceprint-Identifier header fields of the START-SESSION method, and
- the verification mode has been set to one of "train" or "verify",
the verifier resource can start giving verification info during verification operations. If the verifier resource does not reach one of the two major states ("train" or "verify") , it MUST report an error condition in the MRCPv2 status code to indicate why the verifier resource is not ready for the corresponding usage. 😭
the value of verification-mode is persistent within a verification session. If the client tries to change the mode during a verification session, the verifier resource reports an error and the mode keeps its current value. no switching mid-session, bestie 🚫
verification-mode = "Verification-Mode" ":"
verification-mode-string
verification-mode-string = "train"
/ "verify"11.4.4. Adapt-Model 🦋
this one is about how the verifier resource acts after a successful verification operation. If the value is "true", the server SHOULD use audio collected during the verification session to update the voiceprint, so it keeps up with the speaker's changing speech characteristics, unless local policy says no to updating the voiceprint. If the value is "false" (the default), the server MUST NOT update the voiceprint. This header field MAY occur in the START-SESSION method.
adapt-model = "Adapt-Model" ":" BOOLEAN CRLF11.4.5. Abort-Model 🌸
the Abort-Model header field sets what the verifier resource does when the session ends. If the value is "true", the server MUST discard any pending changes to a voiceprint from verification training or verification adaptation. If the value is "false" (the default), the server MUST commit any pending changes for a training session or a successful verification session to the voiceprint repository. a "true" for Abort-Model overrides a "true" for the Adapt-Model header field, so it wins every time 👑. This header field MAY occur in the END-SESSION method.
abort-model = "Abort-Model" ":" BOOLEAN CRLF11.4.6. Min-Verification-Score 🌈
the Min-Verification-Score header field, when used with a verifier resource through a SET-PARAMS, GET-PARAMS, or START-SESSION method, sets the minimum verification score that the server can use to declare a verification decision of "accepted". it's a float between -1.0 and 1.0. The default value for this header field is implementation specific.
min-verification-score = "Min-Verification-Score" ":"
[ %x2D ] FLOAT CRLF11.4.7. Num-Min-Verification-Phrases 🍓
this header field says how many valid utterances you need before a positive decision is given for verification. It's an integer, and the default value is 1. The verifier resource MUST NOT declare a verification 'accepted' unless Num-Min-Verification-Phrases valid utterances have been received. The minimum value is 1. This header field MAY occur in START-SESSION, SET-PARAMS, or GET-PARAMS.
num-min-verification-phrases = "Num-Min-Verification-Phrases" ":"
1*19DIGIT CRLF11.4.8. Num-Max-Verification-Phrases ⭐
this one is the number of valid utterances needed before a decision gets forced for verification. It's an integer with a minimum value of 1, and the default is implementation specific. The verifier resource MUST NOT return a decision of 'undecided' once Num-Max-Verification-Phrases have been collected and used to work out a verification score. This header field MAY occur in START-SESSION, SET-PARAMS, or GET-PARAMS. no more 'undecided' vibes after that 💀
num-max-verification-phrases = "Num-Max-Verification-Phrases" ":"
1*19DIGIT CRLF11.4.9. No-Input-Timeout ⏳
the No-Input-Timeout header field sets how long it takes, counting from when the verification timers start (see START-INPUT-TIMERS), until the VERIFICATION-COMPLETE server event message says no input was received, meaning it has a Completion-Cause of no-input-timeout. The value is in milliseconds. This header field MAY occur in VERIFY, SET-PARAMS, or GET-PARAMS. The value ranges from 0 to an implementation-specific maximum value, and the default value is implementation specific too 🧚
no-input-timeout = "No-Input-Timeout" ":" 1*19DIGIT CRLF11.4.10. Save-Waveform 🎙️
this header field lets the client ask the verifier resource to save the audio stream used for verification or identification. The verifier resource MUST attempt to record the audio and make it available to the client as a URI returned in the Waveform-URI header field in the VERIFICATION-COMPLETE event. If there was an error recording the stream, or the audio content isn't available some other way, the verifier resource MUST return an empty Waveform-URI header field. The default value for this header field is "false". This header field MAY appear in the VERIFY method. Note that this header field does NOT appear in the VERIFY-FROM-BUFFER method, since it only controls whether the waveform gets saved for live verification or identification operations 🫶.
save-waveform = "Save-Waveform" ":" BOOLEAN CRLF11.4.11. Media-Type 🎧
This header field MAY be specified in the SET-PARAMS, GET-PARAMS, or the VERIFY methods, and it tells the server resource the media type of the captured audio or video, like the stuff returned by the Waveform-URI header field.
media-type = "Media-Type" ":" media-type-value
CRLF11.4.12. Waveform-URI 🌊
if the Save-Waveform header field is set to "true", the verifier resource MUST attempt to record the incoming audio stream of the verification into a file and provide a URI so the client can access it. This header field MUST be present in the VERIFICATION-COMPLETE event if the client set Save-Waveform to true. The value of the header field MUST be empty if some error condition stopped the server from recording. Otherwise, the URI the server generates MUST be globally unique across the server and all its verification sessions. The content MUST be available via the URI until the verification session ends. Since the Save-Waveform header field only applies to live verification or identification operations, the server can return the Waveform-URI only in the VERIFICATION-COMPLETE event for live verification or identification operations.
the server MUST also return the size in octets and the duration in milliseconds of the recorded audio waveform as parameters attached to the header field. no skipping those, they're part of the deal 🎀
waveform-uri = "Waveform-URI" ":" ["<" uri ">"
";" "size" "=" 1*19DIGIT
";" "duration" "=" 1*19DIGIT] CRLF11.4.13. Voiceprint-Exists 🔍
this header field MUST be returned in QUERY-VOICEPRINT and DELETE-VOICEPRINT responses. It shows the status of the voiceprint named in the QUERY-VOICEPRINT method. For the DELETE-VOICEPRINT method, it shows the status of the voiceprint at the moment the method started running 💜.
voiceprint-exists = "Voiceprint-Exists" ":" BOOLEAN CRLF11.4.14. Ver-Buffer-Utterance ✨
Okay bestie, this header is giving "save this for later" 🦋 It marks an utterance as something that could be used for speaker verification down the road. So a client can ask the server to buffer utterances during normal recognition or verification, then request speaker verification on those buffered utterances later 💖 This header is optional in the RECOGNIZE, VERIFY, and RECORD methods. The default value is "false" 🌸
ver-buffer-utterance = "Ver-Buffer-Utterance" ":" BOOLEAN
CRLF11.4.15. Input-Waveform-URI 🎧
This one points to stored audio 🎀 The client asks the server to fetch and process that audio using the current verification mode, either to train a voiceprint or to verify a claimed identity. It's the move for the buffering use case where the recognizer and verifier resources live in different sessions and the verification buffer technique can't be used 🫶 It MAY be specified on the VERIFY request.
input-waveform-uri = "Input-Waveform-URI" ":" uri CRLF11.4.16. Completion-Cause 💅
This header MUST be part of a VERIFICATION-COMPLETE event going from the verifier resource to the client. It tells you why a VERIFY or VERIFY-FROM-BUFFER method finished 😭 It MUST be sent in the VERIFY, VERIFY-FROM-BUFFER, and QUERY-VOICEPRINT responses if they come back with a failure status and a COMPLETE state. In the ABNF below, cause-code is a number picked from the Cause-Code column of the table, and cause-name is the matching token from the Cause-Name column 🌈
completion-cause = "Completion-Cause" ":" cause-code SP
cause-name CRLF
cause-code = 3DIGIT
cause-name = *VCHAR
+------------+--------------------------+---------------------------+
| Cause-Code | Cause-Name | Description |
+------------+--------------------------+---------------------------+
| 000 | success | VERIFY or |
| | | VERIFY-FROM-BUFFER |
| | | request completed |
| | | successfully. The verify |
| | | decision can be |
| | | "accepted", "rejected", |
| | | or "undecided". |
| 001 | error | VERIFY or |
| | | VERIFY-FROM-BUFFER |
| | | request terminated |
| | | prematurely due to a |
| | | verifier resource or |
| | | system error. |
| 002 | no-input-timeout | VERIFY request completed |
| | | with no result due to a |
| | | no-input-timeout. |
| 003 | too-much-speech-timeout | VERIFY request completed |
| | | with no result due to too |
| | | much speech. |
| 004 | speech-too-early | VERIFY request completed |
| | | with no result due to |
| | | speech too soon. |
| 005 | buffer-empty | VERIFY-FROM-BUFFER |
| | | request completed with no |
| | | result due to empty |
| | | buffer. |
| 006 | out-of-sequence | Verification operation |
| | | failed due to |
| | | out-of-sequence method |
| | | invocations, for example, |
| | | calling VERIFY before |
| | | QUERY-VOICEPRINT. |
| 007 | repository-uri-failure | Failure accessing |
| | | Repository URI. |
| 008 | repository-uri-missing | Repository-URI is not |
| | | specified. |
| 009 | voiceprint-id-missing | Voiceprint-Identifier is |
| | | not specified. |
| 010 | voiceprint-id-not-exist | Voiceprint-Identifier |
| | | does not exist in the |
| | | voiceprint repository. |
| 011 | speech-not-usable | VERIFY request completed |
| | | with no result because |
| | | the speech was not usable |
| | | (too noisy, too short, |
| | | etc.) |
+------------+--------------------------+---------------------------+11.4.17. Completion-Reason 📝
This header MAY be specified in a VERIFICATION-COMPLETE event coming from the verifier resource to the client. It carries the reason text behind how the VERIFY request ended, so you can see what went wrong when it failed 🔍 The completion reason text is there for client logs, debugging, and instrumentation. Clients MUST NOT interpret the completion reason text 🙈
completion-reason = "Completion-Reason" ":"
quoted-string CRLF11.4.18. Speech-Complete-Timeout ⏱️
This header is the same as the one described for the Recognizer resource, so go check Section 9.4.15 for the full tea ☕ This header MAY occur in VERIFY, SET-PARAMS, or GET-PARAMS. 🍓
11.4.19. New-Audio-Channel 🎙️
Same deal here! This header is the same as the one for the Recognizer resource, and you can find it in Section 9.4.23 💅 This header MAY be specified in a VERIFY request.
11.4.20. Abort-Verification 🛑
This header MUST be sent in a STOP request to say whether a VERIFY method in progress should be aborted 😤 A value of "true" tells the server to discard the results. A value of "false" tells the server to hand back, in the STOP response, whatever verification results it has gathered up to the moment it got the STOP request 💜
abort-verification = "Abort-Verification " ":" BOOLEAN CRLF11.4.21. Start-Input-Timers ⏰
This header MAY be sent as part of a VERIFY request 🧚 A value of "false" tells the verifier resource to start the VERIFY operation but not to start the no-input timer yet. The verifier resource MUST NOT start the timers until the client sends a START-INPUT-TIMERS request to the resource. This is super useful when the verifier and synthesizer resources aren't in the same session. Say a kill-on-barge-in prompt is playing. The client might want the VERIFY request active at the same time so it can detect and handle kill-on-barge-in (see Section 8.4.2) 🦋 But it doesn't want the verifier to start the no-input timers until the prompt is done 🫶 The default value is "true" 🌸
start-input-timers = "Start-Input-Timers" ":"
BOOLEAN CRLF11.5. Verification Message Body 💜
A verification response or event message can carry extra data, as described in the following subsection 🎀
11.5.1. Verification Result Data 📊
Verification results go back to the client in the message body of the VERIFICATION-COMPLETE event or the GET-INTERMEDIATE-RESULT response message, as described in Section 6.3 ✨ The element and attribute descriptions for the verification part of the NLSML format are in Section 11.5.2, and the normative schema definition is in Section 16.3 🧠
11.5.2. Verification Result Elements 🌷
All verification elements live inside a single <verification-result> element under <result> 💖 They're described below and follow the schema defined in Section 16.2. The following elements are defined:
<voiceprint><incremental><cumulative><decision><utterance-length><device><gender><adapted><verification-score><vendor-specific-results>
11.5.2.1. <voiceprint> Element 🎤
This element shows how the speech data matched a single voiceprint 🔍 The result data returned MAY have more than one of these in the case of identification or multi-verification. Each <voiceprint> element, and the XML data inside it, describes how well the speech matched that particular voiceprint. The <voiceprint> element data is ordered by cumulative verification match scores, with the highest score first 👑
11.5.2.2. <cumulative> Element 📈
Inside each <voiceprint> element there MUST be a <cumulative> element 💅 It holds the cumulative scores for how well multiple utterances matched the voiceprint.
11.5.2.3. <incremental> Element 🌱
The first <voiceprint> element MAY contain an <incremental> element with the incremental scores for how well the last utterance matched the voiceprint ✨
11.5.2.4. <Decision> Element ⚖️
This element lives inside the <incremental> or <cumulative> element in the verification results 🦋 Its value is the verification decision, which can be "accepted", "rejected", or "undecided" 💀
11.5.2.5. <utterance-length> Element ⏳
This element MAY occur inside either the <incremental> or <cumulative> elements within the first <voiceprint> element 🧚 Its value is the size in milliseconds of the last utterance or of the cumulated set of utterances, respectively 🍓
11.5.2.6. <device> Element 📱
This element lives inside the <incremental> or <cumulative> element in the verification results. Its value is the apparent type of device the caller used, as the verifier resource figured it out. It can be "cellular-phone", "electret-phone", "carbon-button-phone", or "unknown" 📞
11.5.2.7. <gender> Element 🌈
This element lives inside the <incremental> or <cumulative> element in the verification results. Its value is the apparent gender of the speaker, as the verifier resource determined it. It can be "male", "female", or "unknown" 💜
11.5.2.8. <adapted> Element 🔄
This element lives inside the first <voiceprint> element in the verification results. When verification is trying to confirm the voiceprint, this tells you whether the voiceprint got adapted because of analyzing the source utterances 🧩 It is not returned during verification training. The value can be "true" or "false" 🙌
11.5.2.9. <verification-score> Element
okay bestie, this one lives inside the <incremental> or <cumulative> element within the verification results 💅 Its value tells you the score of the last utterance as determined by verification 🎀
During verification, the higher the score, the more likely it is that the speaker is the same one who spoke the voiceprint utterances 💖 During training, the higher the score, the more likely the speaker is to have spoken all of the analyzed utterances ✨ The value is a floating point between -1.0 and 1.0. If there are no such utterances, the score is -1 🥺 Note that the verification score is not a probability value, so don't treat it like one 🙈
11.5.2.10. <vendor-specific-results> Element
MRCPv2 servers MAY send verification results that contain implementation-specific data that augment the information provided by the MRCPv2-defined elements 🧚 Such data might be useful to clients who have private knowledge of how to interpret these schema extensions 🔥 Implementation-specific additions to the verification results schema MUST belong to the vendor's own namespace 👑 In the result structure, either they MUST be indicated by a namespace prefix declared within the result, or they MUST be children of an element identified as belonging to the respective namespace 💅
The following example shows the results of three voiceprints. Note that the first one has crossed the verification score threshold, and the speaker has been accepted. The voiceprint was also adapted with the most recent utterance 🦋
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
grammar="What-Grammar-URI">
<verification-result>
<voiceprint id="johnsmith">
<adapted> true </adapted>
<incremental>
<utterance-length> 500 </utterance-length>
<device> cellular-phone </device>
<gender> male </gender>
<decision> accepted </decision>
<verification-score> 0.98514 </verification-score>
</incremental>
<cumulative>
<utterance-length> 10000 </utterance-length>
<device> cellular-phone </device>
<gender> male </gender>
<decision> accepted </decision>
<verification-score> 0.96725</verification-score>
</cumulative>
</voiceprint>
<voiceprint id="marysmith">
<cumulative>
<verification-score> 0.93410 </verification-score>
</cumulative>
</voiceprint>
<voiceprint uri="juniorsmith">
<cumulative>
<verification-score> 0.74209 </verification-score>
</cumulative>
</voiceprint>
</verification-result>
</result>Verification Results Example 1
In this next example, the verifier has enough information to decide to reject the speaker 🙅♀️
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:xmpl="http://www.example.org/2003/12/mrcpv2"
grammar="What-Grammar-URI">
<verification-result>
<voiceprint id="johnsmith">
<incremental>
<utterance-length> 500 </utterance-length>
<device> cellular-phone </device>
<gender> male </gender>
<verification-score> 0.88514 </verification-score>
<xmpl:raspiness> high </xmpl:raspiness>
<xmpl:emotion> sadness </xmpl:emotion>
</incremental>
<cumulative>
<utterance-length> 10000 </utterance-length>
<device> cellular-phone </device>
<gender> male </gender>
<decision> rejected </decision>
<verification-score> 0.9345 </verification-score>
</cumulative>
</voiceprint>
</verification-result>
</result>Verification Results Example 2
11.6. START-SESSION
The START-SESSION method starts a speaker verification or speaker identification session 🎬 Running it places the verifier resource into its initial state. If this method is called during an ongoing verification session, the previous session is implicitly aborted 😤 If this method is invoked when VERIFY or VERIFY-FROM-BUFFER is active, the method fails and the server returns a status-code of 402 💀
Once START-SESSION completes, the verifier resource MUST have terminated any ongoing verification session and cleared any voiceprint designation 🧹
A verification session is associated with the voiceprint repository to be used during the session. This is specified through the Repository-URI header field (see Section 11.4.1) 🗂️
START-SESSION also establishes, through the Voiceprint-Identifier header field, which voiceprints are to be matched or trained during the verification session 🎯 If this is an Identification session or if the client wants to do Multi-Verification, the Voiceprint-Identifier header field contains a list of semicolon-separated voiceprint identifiers 🌈
The Adapt-Model header field MAY also be present in the START-SESSION request to indicate whether or not to adapt a voiceprint based on data collected during the session (if the voiceprint verification phase succeeds) ✨ By default, the voiceprint model MUST NOT be adapted with data from a verification session 🔒
START-SESSION also determines whether the session is for a train or verify of a voiceprint 🧠 Hence, the Verification-Mode header field MUST be sent in every START-SESSION request. The value of the Verification-Mode header field MUST be one of either "train" or "verify" 🫶
Before a verification/identification session is started, the client may only request that VERIFY-ROLLBACK and generic SET-PARAMS and GET-PARAMS operations be performed on the verifier resource. The server MUST return status-code 402 "Method not valid in this state" for all other verification operations 🚫
A verifier resource MUST NOT have more than a single session active at one time 🙅♀️
C->S: MRCP/2.0 ... START-SESSION 314161
Channel-Identifier:32AECB23433801@speakverify
Repository-URI:http://www.example.com/voiceprintdbase/
Voiceprint-Mode:verify
Voiceprint-Identifier:johnsmith.voiceprint
Adapt-Model:true
S->C: MRCP/2.0 ... 314161 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify11.7. END-SESSION
The END-SESSION method terminates an ongoing verification session and releases the verification voiceprint resources 🌸 The session may terminate in one of three ways:
- abort - the voiceprint adaptation or creation may be aborted so that the voiceprint remains unchanged (or is not created).
- commit - when terminating a voiceprint training session, the new voiceprint is committed to the repository.
- adapt - an existing voiceprint is modified using a successful verification.
The Abort-Model header field MAY be included in the END-SESSION to control whether or not to abort any pending changes to the voiceprint 🍓 The default behavior is to commit (not abort) any pending changes to the designated voiceprint.
The END-SESSION method may be safely executed multiple times without first executing the START-SESSION method. Any additional executions of this method without an intervening use of the START-SESSION method have no effect on the verifier resource 🧘♀️
The following example assumes there is either a training session or a verification session in progress.
C->S: MRCP/2.0 ... END-SESSION 314174
Channel-Identifier:32AECB23433801@speakverify
Abort-Model:true
S->C: MRCP/2.0 ... 314174 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify11.8. QUERY-VOICEPRINT
The QUERY-VOICEPRINT method gets status information on a particular voiceprint 🔍 The client can use it to check whether a voiceprint or repository exists and whether it contains trained voiceprints 🎀
The response to the QUERY-VOICEPRINT request contains an indication of the status of the designated voiceprint in the Voiceprint-Exists header field. This lets the client decide whether to use the current voiceprint for verification, train a new voiceprint, or choose a different voiceprint 🧩
A voiceprint is completely specified by providing a repository location and a voiceprint identifier. The particular voiceprint or identity within the repository is specified by a string identifier that is unique within the repository. The Voiceprint-Identifier header field carries this unique voiceprint identifier within a given repository 🆔
The following example assumes a verification session is in progress and the voiceprint exists in the voiceprint repository.
C->S: MRCP/2.0 ... QUERY-VOICEPRINT 314168
Channel-Identifier:32AECB23433801@speakverify
Repository-URI:http://www.example.com/voiceprints/
Voiceprint-Identifier:johnsmith.voiceprint
S->C: MRCP/2.0 ... 314168 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
Repository-URI:http://www.example.com/voiceprints/
Voiceprint-Identifier:johnsmith.voiceprint
Voiceprint-Exists:trueThe following example assumes that the URI provided in the Repository-URI header field is a bad URI 🙈
C->S: MRCP/2.0 ... QUERY-VOICEPRINT 314168
Channel-Identifier:32AECB23433801@speakverify
Repository-URI:http://www.example.com/bad-uri/
Voiceprint-Identifier:johnsmith.voiceprint
S->C: MRCP/2.0 ... 314168 405 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
Repository-URI:http://www.example.com/bad-uri/
Voiceprint-Identifier:johnsmith.voiceprint
Completion-Cause:007 repository-uri-failure11.9. DELETE-VOICEPRINT
The DELETE-VOICEPRINT method removes a voiceprint from a repository 🗑️ This method MUST carry the Repository-URI and Voiceprint-Identifier header fields.
An MRCPv2 server MUST reject a DELETE-VOICEPRINT request with a 401 status code unless the MRCPv2 client has been authenticated and authorized. Note that MRCPv2 does not have a standard mechanism for this. See Section 12.8 🔐
If the corresponding voiceprint does not exist, the DELETE-VOICEPRINT method MUST return a 200 status code 🤷♀️
The following example demonstrates a DELETE-VOICEPRINT operation to remove a specific voiceprint.
C->S: MRCP/2.0 ... DELETE-VOICEPRINT 314168
Channel-Identifier:32AECB23433801@speakverify
Repository-URI:http://www.example.com/bad-uri/
Voiceprint-Identifier:johnsmith.voiceprint
S->C: MRCP/2.0 ... 314168 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify11.10. VERIFY
The VERIFY method asks the verifier resource to either train/adapt the voiceprint or verify/identify a claimed identity 🎤 If the voiceprint is new or was deleted by a previous DELETE-VOICEPRINT method, VERIFY trains the voiceprint 🌱 If the voiceprint already exists, it is adapted and not retrained by the VERIFY command 🔄
C->S: MRCP/2.0 ... VERIFY 543260
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 543260 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speakverifyWhen the VERIFY request completes, the MRCPv2 server MUST send a VERIFICATION-COMPLETE event to the client 🥳
11.11. VERIFY-FROM-BUFFER 🪣
okay so VERIFY-FROM-BUFFER is the method that tells the verifier resource to check buffered audio against a voiceprint 🎤✨ only one VERIFY or VERIFY-FROM-BUFFER method may be active for a verifier resource at a time, so no double-booking, bestie 💅
the buffered audio doesn't get used up by this method 🫶 which means VERIFY-FROM-BUFFER may be invoked multiple times by the client, so you can try the same audio against different voiceprints. iconic 🦋
for the VERIFY-FROM-BUFFER method, the server MAY optionally return an IN-PROGRESS response before the VERIFICATION-COMPLETE event.
now the sus part 🙈 when the verification buffer is already in use by another resource sharing it, the server MUST return an IN-PROGRESS response and wait until the buffer is free for it. the verification buffer belongs to the verifier resource, but other input resources on the same session can write to it too. so it counts as "in use" when there's a read or write going on, like a RECORD or RECOGNIZE with the Ver-Buffer-Utterance header field set to "true" on a resource that shares this buffer 👀 and if a RECORD or RECOGNIZE method returns with a failure cause code, the VERIFY-FROM-BUFFER request that was waiting on that buffer MUST also fail with a Completion-Cause of 005 (buffer-empty) 💀
here's a example scenario 🌸 the client first did a live verification, but the utterance got rejected. meanwhile, that same utterance also got saved to the audio buffer. then another voiceprint gets checked against the buffer, and this time the utterance is accepted ✨ for the example, we assume both Num-Min-Verification-Phrases and Num-Max-Verification-Phrases are 1.
C->S: MRCP/2.0 ... START-SESSION 314161
Channel-Identifier:32AECB23433801@speakverify
Verification-Mode:verify
Adapt-Model:true
Repository-URI:http://www.example.com/voiceprints
Voiceprint-Identifier:johnsmith.voiceprint
S->C: MRCP/2.0 ... 314161 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
C->S: MRCP/2.0 ... VERIFY 314162
Channel-Identifier:32AECB23433801@speakverify
Ver-buffer-utterance:true
S->C: MRCP/2.0 ... 314162 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... VERIFICATION-COMPLETE 314162 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
Completion-Cause:000 success
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
grammar="What-Grammar-URI">
<verification-result>
<voiceprint id="johnsmith">
<incremental>
<utterance-length> 500 </utterance-length>
<device> cellular-phone </device>
<gender> female </gender>
<decision> rejected </decision>
<verification-score> 0.05465 </verification-score>
</incremental>
<cumulative>
<utterance-length> 500 </utterance-length>
<device> cellular-phone </device>
<gender> female </gender>
<decision> rejected </decision>
<verification-score> 0.05465 </verification-score>
</cumulative>
</voiceprint>
</verification-result>
</result>
C->S: MRCP/2.0 ... QUERY-VOICEPRINT 314163
Channel-Identifier:32AECB23433801@speakverify
Repository-URI:http://www.example.com/voiceprints/
Voiceprint-Identifier:johnsmith
S->C: MRCP/2.0 ... 314163 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
Repository-URI:http://www.example.com/voiceprints/
Voiceprint-Identifier:johnsmith.voiceprint
Voiceprint-Exists:true
C->S: MRCP/2.0 ... START-SESSION 314164
Channel-Identifier:32AECB23433801@speakverify
Verification-Mode:verify
Adapt-Model:true
Repository-URI:http://www.example.com/voiceprints
Voiceprint-Identifier:marysmith.voiceprint
S->C: MRCP/2.0 ... 314164 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
C->S: MRCP/2.0 ... VERIFY-FROM-BUFFER 314165
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 314165 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... VERIFICATION-COMPLETE 314165 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
Completion-Cause:000 success
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
grammar="What-Grammar-URI">
<verification-result>
<voiceprint id="marysmith">
<incremental>
<utterance-length> 1000 </utterance-length>
<device> cellular-phone </device>
<gender> female </gender>
<decision> accepted </decision>
<verification-score> 0.98 </verification-score>
</incremental>
<cumulative>
<utterance-length> 1000 </utterance-length>
<device> cellular-phone </device>
<gender> female </gender>
<decision> accepted </decision>
<verification-score> 0.98 </verification-score>
</cumulative>
</voiceprint>
</verification-result>
</result>
C->S: MRCP/2.0 ... END-SESSION 314166
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 314166 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverifyVERIFY-FROM-BUFFER Example
11.12. VERIFY-ROLLBACK ↩️
VERIFY-ROLLBACK throws away the last buffered utterance, or the last live utterances when the mode is "train" or "verify" 🗑️ the client will probably want this when the user gives undesirable input, like non-speech noises, side-speech, out-of-grammar utterances, commands, and so on 🙃 just know it doesn't keep a stack of rollback states 📚 so running VERIFY-ROLLBACK twice in a row, without a recognition operation in between, does nothing the second time 💀
C->S: MRCP/2.0 ... VERIFY-ROLLBACK 314165
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 314165 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverifyVERIFY-ROLLBACK Example
11.13. STOP 🛑
the STOP method, sent from the client to the server, tells the verifier resource to stop the VERIFY or VERIFY-FROM-BUFFER request if one is active 🙅♀️ if there's an active request and STOP successfully ends it, then the response header section carries an Active-Request-Id-List header field with the request-id of the VERIFY or VERIFY-FROM-BUFFER request that got terminated. in that case, no VERIFICATION-COMPLETE event gets sent for the terminated request 🚫 if there wasn't a verify request active, then the response MUST NOT contain an Active-Request-Id-List header field. either way, the response MUST contain a status-code of 200 "Success" ✨
the STOP method can also carry an Abort-Verification header field, which decides whether the verification result up to that point gets thrown out or returned 🎀 if the header field is missing, or its value is "true", the verification result is discarded and the STOP response doesn't contain any result data 💔 if the header field is present and its value is "false", the STOP response MUST contain a Completion-Cause header field and carry the Verification result data in its body 🫶
an aborted VERIFY request does an automatic rollback, so it doesn't affect the cumulative score 🧼 but a VERIFY request that was stopped with no Abort-Verification header field, or with Abort-Verification set to "false", does affect the cumulative scores. if the client doesn't want that result counted, it would need to be explicitly rolled back 👀
the example below assumes a voiceprint identity has already been set up 🌷
C->S: MRCP/2.0 ... VERIFY 314177
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 314177 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speakverify
C->S: MRCP/2.0 ... STOP 314178
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 314178 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
Active-Request-Id-List:314177STOP Verification Example
11.14. START-INPUT-TIMERS ⏱️
this request is sent from the client to the verifier resource to start the no-input timer 🕰️ usually the client does this once it's confirmed that any audio prompts to the user have finished playing 🎶
C->S: MRCP/2.0 ... START-INPUT-TIMERS 543260
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 543260 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify11.15. VERIFICATION-COMPLETE 🏁
the VERIFICATION-COMPLETE event comes after a call to VERIFY or VERIFY-FROM-BUFFER, and it's how the verification results get passed back to the client 📬 the event message body contains only verification results, nothing else 🫧
S->C: MRCP/2.0 ... VERIFICATION-COMPLETE 543259 COMPLETE
Completion-Cause:000 success
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
grammar="What-Grammar-URI">
<verification-result>
<voiceprint id="johnsmith">
<incremental>
<utterance-length> 500 </utterance-length>
<device> cellular-phone </device>
<gender> male </gender>
<decision> accepted </decision>
<verification-score> 0.85 </verification-score>
</incremental>
<cumulative>
<utterance-length> 1500 </utterance-length>
<device> cellular-phone </device>
<gender> male </gender>
<decision> accepted </decision>
<verification-score> 0.75 </verification-score>
</cumulative>
</voiceprint>
</verification-result>
</result>11.16. START-OF-INPUT 🎙️
the START-OF-INPUT event is sent from the server to the client once the server has detected speech 👂✨ the verifier resource always returns this when speech is detected 💯 and that's true whether or not the recognizer and verifier resources share the same session, so no exceptions, periodt 💅
S->C: MRCP/2.0 ... START-OF-INPUT 543259 IN-PROGRESS
Channel-Identifier:32AECB23433801@speakverify11.17. CLEAR-BUFFER 🧹
Okay so CLEAR-BUFFER is the method that wipes the verification buffer 🫧. That buffer holds on to speech captured during recognize, record, or verify stuff, so VERIFY-FROM-BUFFER can reuse it later. Here's the tea: the buffer is shared with the recognizer and recorder resources too, so a CLEAR-BUFFER request fails if the buffer is busy. That happens when any of those input resources is in the middle of a read or write, like RECORD, RECOGNIZE, or VERIFY with Ver-Buffer-Utterance set to "true". Not me being picky, just the rules 💅
C->S: MRCP/2.0 ... CLEAR-BUFFER 543260
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 543260 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify11.18. GET-INTERMEDIATE-RESULT 👀
Okay this one is so useful ✨ A client can use GET-INTERMEDIATE-RESULT to peek at the intermediate results of a verification that's still running. Invoking it does not change the resource's state, so it's a totally safe check-in 🦋 The verifier just grabs the accumulated results and sends them back in the response. The message body of that response has only verification results in it. The method response MUST NOT contain a Completion-Cause header field, because the request isn't complete yet, so no final verdict is allowed 🙅♀️ And if there's no verification in progress, you get a 402 failure status-code and no result in the body. Rough, but clear.
C->S: MRCP/2.0 ... GET-INTERMEDIATE-RESULT 543260
Channel-Identifier:32AECB23433801@speakverify
S->C: MRCP/2.0 ... 543260 200 COMPLETE
Channel-Identifier:32AECB23433801@speakverify
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
grammar="What-Grammar-URI">
<verification-result>
<voiceprint id="marysmith">
<incremental>
<utterance-length> 50 </utterance-length>
<device> cellular-phone </device>
<gender> female </gender>
<decision> undecided </decision>
<verification-score> 0.85 </verification-score>
</incremental>
<cumulative>
<utterance-length> 150 </utterance-length>
<device> cellular-phone </device>
<gender> female </gender>
<decision> undecided </decision>
<verification-score> 0.65 </verification-score>
</cumulative>
</voiceprint>
</verification-result>
</result>12. Security Considerations 🔐
Okay, security is the part where I get very serious, bestie 😤 MRCPv2 is designed to comply with the security-related requirements written up in the SPEECHSC requirements [RFC4313]. Implementers and users are strongly encouraged to read the Security Considerations section of [RFC4313], because it covers a bunch of big issues: using speech as biometric authentication, the threats against systems that store recorded speech, hold large corpora of voiceprints, and send or receive sensitive stuff based on voice input to a recognizer or speech output from a synthesizer. The specific measures MRCPv2 uses are summarized in the subsections below. Also, check the matching sections of this spec to see how each operation calls the security machinery 🧠
12.1. Rendezvous and Session Establishment 🤝
MRCPv2 control sessions are set up as media sessions described by SDP, inside a SIP dialog. To make sure the rendezvous between MRCPv2 clients and servers is secure, here's what's required:
- The SIP implementation in MRCPv2 clients and servers MUST support SIP digest authentication [RFC3261] and SHOULD employ it.
- The SIP implementation in MRCPv2 clients and servers MUST support 'sips' URIs and SHOULD employ 'sips' URIs; this includes that clients and servers SHOULD set up TLS [RFC5246] connections.
- If media stream cryptographic keying is done through SDP (e.g. using [RFC4568]), the MRCPv2 clients and servers MUST employ the 'sips' URI.
- When TLS is used for SIP, the client MUST verify the identity of the server to which it connects, following the rules and guidelines defined in [RFC5922].
12.2. Control Channel Protection 🛡️
Sensitive data travels over the MRCPv2 control channel, and that's a lot of spicy stuff 🌶️ Think speech recognition output, speaker verification results, input to text-to-speech, personally identifying grammars, all of it. So MRCPv2 servers must be properly authenticated, and the control channel must allow both confidentiality and integrity for the data. To protect the control channel, MRCPv2 clients and servers MUST support TLS and SHOULD use it by default, unless some other control channel protection is in use. When TLS is used, the client MUST verify the identity of the server it connects to, following the rules and guidelines defined in [RFC4572]. If there are multiple TLS-protected channels between the client and the server, the server MUST NOT send a response to the client over a channel where the TLS identities of the server or client differ from the channel the server got the matching request on. Alternative control-channel protection MAY be used if desired (e.g., Security Architecture for the Internet Protocol (IPsec) [RFC4301]).
12.3. Media Session Protection 🎧
Sensitive data also shows up on media sessions that end on MRCPv2 servers. (The other end of a media channel may or may not be on the MRCPv2 client, so keep that in mind 👀) This includes the user's spoken utterances and the output of text-to-speech operations. MRCPv2 servers MUST support a security mechanism for protecting audio media sessions. MRCPv2 clients that originate or consume audio MUST similarly support a security mechanism for protecting the audio. One such mechanism is the Secure Real-time Transport Protocol (SRTP) [RFC3711].
12.4. Indirect Content Access 🔗
MRCPv2 uses content indirection a LOT. Content may be fetched and/or stored based on URI addressing on systems other than the MRCPv2 client or server. Not all of that stored content is sensitive (XML schemas, for example, are fine 🙃), but most of it generally needs protection. Some indirect content, like voice recordings and voiceprints, is extremely sensitive and must always be protected. MRCPv2 clients and servers MUST implement HTTPS for indirect content access and SHOULD employ secure access for all sensitive indirect content. Other secure URI schemes, such as Secure FTP (FTPS) [RFC4217], MAY also be used. See Section 6.2.15 for the header fields used to transfer cookie information between the MRCPv2 client and server, if they're needed for authentication.
Accessing URIs that servers hand out comes with risks, so keep your eyes open 👁️ Although RFC 6454 [RFC6454] focuses on a same-origin policy, which MRCPv2 doesn't restrict URIs to, it still does a great job explaining the pitfalls of blindly following server-provided URIs in Section 3 of the RFC. Servers also have to remember that clients could hand them URIs pointing at sites built to tie the server up in super long or otherwise problematic document fetches 😤 MRCPv2 servers, and the services they access, MUST always be prepared for the possibility of such a denial-of-service attack.
MRCPv2 makes no built-in assumptions about the lifetime and access controls of a URI. For example, if neither authentication nor scheme-specific access controls are used, leaking the URI is the same as leaking the content 🫠 MRCPv2 also doesn't make any specific demands on how long a URI lives. If a server offers a URI and the client takes a long, long time to access it, the server may have removed the resource in the meantime. MRCPv2 handles this with the URI access scheme's 'resource not found' error, like 404 for HTTPS. How long a server should keep a dynamic resource around is very application and context dependent. Still, the server SHOULD keep the resource available for a reasonable amount of time, so the client is likely to have it when it needs it. On the flip side, to help mitigate state exhaustion attacks, MRCPv2 servers aren't obligated to keep resources and resource state forever. The server SHOULD delete dynamically generated resources associated with an MRCPv2 session when that session ends 🧹
One way to avoid resource leakage is for the server to use difficult-to-guess, one-time resource URIs. That means there can only be a single access to the underlying resource through the given URI. The catch is that if an attacker uses the URI before the client does, the client gets denied the resource 😬 Other options would be a mechanism similar to the URLAUTH IMAP extension [RFC4467], where the server sets cryptographic checks on URI usage and also has capabilities for expiration, revocation, and so on. Specifying such a mechanism is beyond the scope of this document.
12.5. Protection of Stored Media 💾
MRCPv2 applications often need stored media. Voice recordings get stored (for diagnosis and system tuning, for example) and fetched (for replaying utterances into multiple MRCPv2 resources). Voiceprints are fundamental to the speaker identification and verification functions. This data can be extremely sensitive, and if it gets stolen, it can cause serious privacy and impersonation risks 😭 Systems that use MRCPv2 SHOULD be deployed in ways that minimize these risks. The SPEECHSC requirements RFC [RFC4313] has a much more detailed discussion of these risks and how they might be mitigated.
12.6. DTMF and Recognition Buffers 🔢
DTMF buffers and recognition buffers can grow big enough to exceed what a server can handle, so the server MUST be ready to handle resource consumption gracefully 🫶 A server MAY respond with the appropriate recognition incomplete if it's in danger of running out of resources.
12.7. Client-Set Server Parameters ⚙️
In MRCPv2, some tasks, like URI resource fetches, are done by the server on behalf of the client. To control this behavior, MRCPv2 has a number of server parameters a client can configure. One of them, Fetch-Timeout (Section 6.2.12), is a little concerning: a malicious client could set a very large value and then ask the server to fetch a non-existent document 🙃 It is RECOMMENDED that servers be cautious about accepting long timeout values, or abnormally large values, for other client-set parameters. Be careful out there, bestie 💖
12.8. DELETE-VOICEPRINT and Authorization 🔐
okay bestie, this spec doesn't mandate one specific way to handle authentication and authorization when you ask for DELETE-VOICEPRINT (Section 11.9) 😬. So there's a risk that an MRCPv2 server might not do that check at all. ngl, that's a little scary. In practice though, every provider of voice biometric solutions insists on its own authentication and authorization mechanism outside of this spec, so it's not likely to be a major problem 💅. And if voice biometric providers ever standardize on one mechanism, a future version of MRCP can make it mandatory 🫶.
13. IANA Considerations ✨
13.1. New Registries 📚
this section covers the name spaces (registries) for MRCPv2 that IANA created and now maintains 🦋. The assignment and registration policies are described in RFC 5226 [RFC5226].
13.1.1. MRCPv2 Resource Types 🎀
IANA made a brand new name space called "MRCPv2 Resource Types" 🌸. All maintenance in it, and every addition to its contents, MUST follow the "Standards Action" registration policy. The starting contents of the registry, defined in Section 4.2, look like this:
Resource type Resource description Reference
------------- -------------------- ---------
speechrecog Speech Recognizer [RFC6787]
dtmfrecog DTMF Recognizer [RFC6787]
speechsynth Speech Synthesizer [RFC6787]
basicsynth Basic Synthesizer [RFC6787]
speakverify Speaker Verifier [RFC6787]
recorder Speech Recorder [RFC6787]13.1.2. MRCPv2 Methods and Events 🧚
IANA also made a new name space called "MRCPv2 Methods and Events" 💜. All maintenance in it, and every addition to its contents, MUST follow the "Standards Action" registration policy. The starting contents are defined by the "method-name" and "event-name" BNF in Section 15 and explained in Sections 5.2 and 5.5, and they're listed below. obsessed with how organized this is 😭.
Name Resource type Method/Event Reference
---- ------------- ------------ ---------
SET-PARAMS Generic Method [RFC6787]
GET-PARAMS Generic Method [RFC6787]
SPEAK Synthesizer Method [RFC6787]
STOP Synthesizer Method [RFC6787]
PAUSE Synthesizer Method [RFC6787]
RESUME Synthesizer Method [RFC6787]
BARGE-IN-OCCURRED Synthesizer Method [RFC6787]
CONTROL Synthesizer Method [RFC6787]
DEFINE-LEXICON Synthesizer Method [RFC6787]
DEFINE-GRAMMAR Recognizer Method [RFC6787]
RECOGNIZE Recognizer Method [RFC6787]
INTERPRET Recognizer Method [RFC6787]
GET-RESULT Recognizer Method [RFC6787]
START-INPUT-TIMERS Recognizer Method [RFC6787]
STOP Recognizer Method [RFC6787]
START-PHRASE-ENROLLMENT Recognizer Method [RFC6787]
ENROLLMENT-ROLLBACK Recognizer Method [RFC6787]
END-PHRASE-ENROLLMENT Recognizer Method [RFC6787]
MODIFY-PHRASE Recognizer Method [RFC6787]
DELETE-PHRASE Recognizer Method [RFC6787]
RECORD Recorder Method [RFC6787]
STOP Recorder Method [RFC6787]
START-INPUT-TIMERS Recorder Method [RFC6787]
START-SESSION Verifier Method [RFC6787]
END-SESSION Verifier Method [RFC6787]
QUERY-VOICEPRINT Verifier Method [RFC6787]
DELETE-VOICEPRINT Verifier Method [RFC6787]
VERIFY Verifier Method [RFC6787]
VERIFY-FROM-BUFFER Verifier Method [RFC6787]
VERIFY-ROLLBACK Verifier Method [RFC6787]
STOP Verifier Method [RFC6787]
START-INPUT-TIMERS Verifier Method [RFC6787]
GET-INTERMEDIATE-RESULT Verifier Method [RFC6787]
SPEECH-MARKER Synthesizer Event [RFC6787]
SPEAK-COMPLETE Synthesizer Event [RFC6787]
START-OF-INPUT Recognizer Event [RFC6787]
RECOGNITION-COMPLETE Recognizer Event [RFC6787]
INTERPRETATION-COMPLETE Recognizer Event [RFC6787]
START-OF-INPUT Recorder Event [RFC6787]
RECORD-COMPLETE Recorder Event [RFC6787]
VERIFICATION-COMPLETE Verifier Event [RFC6787]
START-OF-INPUT Verifier Event [RFC6787]13.1.3. MRCPv2 Header Fields ✨
okay so IANA made a brand new name space called "MRCPv2 Header Fields" 🎀 and bestie, every change to it MUST go through the "Standards Action" registration policy 💅 The starting list comes from the "message-header" BNF in Section 15 and gets explained in Section 5.1 🧚 Heads up that the values allowed for "Vendor-Specific-Parameters" follow a totally different policy, so check out Section 13.1.6 for that drama 🔥
Name Resource type Reference
---- ------------- ---------
Channel-Identifier Generic [RFC6787]
Accept Generic [RFC2616]
Active-Request-Id-List Generic [RFC6787]
Proxy-Sync-Id Generic [RFC6787]
Accept-Charset Generic [RFC2616]
Content-Type Generic [RFC6787]
Content-ID Generic
[RFC2392], [RFC2046], and [RFC5322]
Content-Base Generic [RFC6787]
Content-Encoding Generic [RFC6787]
Content-Location Generic [RFC6787]
Content-Length Generic [RFC6787]
Fetch-Timeout Generic [RFC6787]
Cache-Control Generic [RFC6787]
Logging-Tag Generic [RFC6787]
Set-Cookie Generic [RFC6787]
Vendor-Specific Generic [RFC6787]
Jump-Size Synthesizer [RFC6787]
Kill-On-Barge-In Synthesizer [RFC6787]
Speaker-Profile Synthesizer [RFC6787]
Completion-Cause Synthesizer [RFC6787]
Completion-Reason Synthesizer [RFC6787]
Voice-Parameter Synthesizer [RFC6787]
Prosody-Parameter Synthesizer [RFC6787]
Speech-Marker Synthesizer [RFC6787]
Speech-Language Synthesizer [RFC6787]
Fetch-Hint Synthesizer [RFC6787]
Audio-Fetch-Hint Synthesizer [RFC6787]
Failed-URI Synthesizer [RFC6787]
Failed-URI-Cause Synthesizer [RFC6787]
Speak-Restart Synthesizer [RFC6787]
Speak-Length Synthesizer [RFC6787]
Load-Lexicon Synthesizer [RFC6787]
Lexicon-Search-Order Synthesizer [RFC6787]
Confidence-Threshold Recognizer [RFC6787]
Sensitivity-Level Recognizer [RFC6787]
Speed-Vs-Accuracy Recognizer [RFC6787]
N-Best-List-Length Recognizer [RFC6787]
Input-Type Recognizer [RFC6787]
No-Input-Timeout Recognizer [RFC6787]
Recognition-Timeout Recognizer [RFC6787]
Waveform-URI Recognizer [RFC6787]
Input-Waveform-URI Recognizer [RFC6787]
Completion-Cause Recognizer [RFC6787]
Completion-Reason Recognizer [RFC6787]
Recognizer-Context-Block Recognizer [RFC6787]
Start-Input-Timers Recognizer [RFC6787]
Speech-Complete-Timeout Recognizer [RFC6787]
Speech-Incomplete-Timeout Recognizer [RFC6787]
Dtmf-Interdigit-Timeout Recognizer [RFC6787]
Dtmf-Term-Timeout Recognizer [RFC6787]
Dtmf-Term-Char Recognizer [RFC6787]
Failed-URI Recognizer [RFC6787]
Failed-URI-Cause Recognizer [RFC6787]
Save-Waveform Recognizer [RFC6787]
Media-Type Recognizer [RFC6787]
New-Audio-Channel Recognizer [RFC6787]
Speech-Language Recognizer [RFC6787]
Ver-Buffer-Utterance Recognizer [RFC6787]
Recognition-Mode Recognizer [RFC6787]
Cancel-If-Queue Recognizer [RFC6787]
Hotword-Max-Duration Recognizer [RFC6787]
Hotword-Min-Duration Recognizer [RFC6787]
Interpret-Text Recognizer [RFC6787]
Dtmf-Buffer-Time Recognizer [RFC6787]
Clear-Dtmf-Buffer Recognizer [RFC6787]
Early-No-Match Recognizer [RFC6787]
Num-Min-Consistent-Pronunciations Recognizer [RFC6787]
Consistency-Threshold Recognizer [RFC6787]
Clash-Threshold Recognizer [RFC6787]
Personal-Grammar-URI Recognizer [RFC6787]
Enroll-Utterance Recognizer [RFC6787]
Phrase-ID Recognizer [RFC6787]
Phrase-NL Recognizer [RFC6787]
Weight Recognizer [RFC6787]
Save-Best-Waveform Recognizer [RFC6787]
New-Phrase-ID Recognizer [RFC6787]
Confusable-Phrases-URI Recognizer [RFC6787]
Abort-Phrase-Enrollment Recognizer [RFC6787]
Sensitivity-Level Recorder [RFC6787]
No-Input-Timeout Recorder [RFC6787]
Completion-Cause Recorder [RFC6787]
Completion-Reason Recorder [RFC6787]
Failed-URI Recorder [RFC6787]
Failed-URI-Cause Recorder [RFC6787]
Record-URI Recorder [RFC6787]
Media-Type Recorder [RFC6787]
Max-Time Recorder [RFC6787]
Trim-Length Recorder [RFC6787]
Final-Silence Recorder [RFC6787]
Capture-On-Speech Recorder [RFC6787]
Ver-Buffer-Utterance Recorder [RFC6787]
Start-Input-Timers Recorder [RFC6787]
New-Audio-Channel Recorder [RFC6787]
Repository-URI Verifier [RFC6787]
Voiceprint-Identifier Verifier [RFC6787]
Verification-Mode Verifier [RFC6787]
Adapt-Model Verifier [RFC6787]
Abort-Model Verifier [RFC6787]
Min-Verification-Score Verifier [RFC6787]
Num-Min-Verification-Phrases Verifier [RFC6787]
Num-Max-Verification-Phrases Verifier [RFC6787]
No-Input-Timeout Verifier [RFC6787]
Save-Waveform Verifier [RFC6787]
Media-Type Verifier [RFC6787]
Waveform-URI Verifier [RFC6787]
Voiceprint-Exists Verifier [RFC6787]
Ver-Buffer-Utterance Verifier [RFC6787]
Input-Waveform-URI Verifier [RFC6787]
Completion-Cause Verifier [RFC6787]
Completion-Reason Verifier [RFC6787]
Speech-Complete-Timeout Verifier [RFC6787]
New-Audio-Channel Verifier [RFC6787]
Abort-Verification Verifier [RFC6787]
Start-Input-Timers Verifier [RFC6787]
Input-Type Verifier [RFC6787]13.1.4. MRCPv2 Status Codes 🌸
IANA has spun up a new name space called "MRCPv2 Status Codes" 🦋 It starts with the values defined in Section 5.4. Any maintenance or new additions MUST follow the "Specification Required with Expert Review" registration policy, so no sneaking things in 🙈
13.1.5. Grammar Reference List Parameters 💜
IANA made a new name space called "Grammar Reference List Parameters" 🍓 Changes here MUST go through the "Specification Required with Expert Review" policy too. There's only one starter parameter, and it's giving minimalist energy ✨
Name Reference
---- -------------
weight [RFC6787]13.1.6. MRCPv2 Vendor-Specific Parameters 👑
IANA created the "MRCPv2 Vendor-Specific Parameters" name space 🌈 Every maintenance task and addition MUST follow the "Hierarchical Allocation" registration policy, and here's how it goes. Each name (matching the "vendor-av-pair-name" ABNF production) MUST satisfy the syntax rules for Internet Domain Names from Section 2.3.1 of RFC 1035 [RFC1035] (and whatever later RFCs update or obsolete that), with one twist: the domain name order gets reversed 🔄 So a vendor-specific parameter "foo" from example.com would look like com.example.foo 💖 The top-level domain has to be one of exactly the Top-Level Internet Domains IANA defines, and IANA only updates that list when the set actually changes 🫶 The second-level domain and every subdomain inside the parameter name MUST be allocated under the "First Come First Served" policy. It's RECOMMENDED that requests line up with how Internet domain names are already handed out to organizations, institutions, corporations, and so on 🎀
The registry lists vendor-registered parameters. Each parameter is tied to a contact person and can optionally include a reference to where it's defined, ideally an RFC 📚 The registry starts out completely empty 🫧
13.2. NLSML-Related Registrations 🦋
13.2.1. 'application/nlsml+xml' Media Type Registration
okay so IANA registered this media type through the process in RFC 4288 [RFC4288] 💅 Here's the whole registration, bestie:
- To: ietf-types@iana.org
- Subject: Registration of media type application/nlsml+xml
- MIME media type name: application
- MIME subtype name: nlsml+xml
- Required parameters: none
- Optional parameters:
- Encoding considerations: All of the considerations described in RFC 3023 also apply to the 'application/nlsml+xml' media type.
- Security considerations: 🔐 Just like HTML, NLSML documents link out to other data stores (grammars, verifier resources, etc.). Unlike HTML though, those data stores are NOT treated as media to be rendered. Still, linked files can have their own security considerations, and those are the ones belonging to each individual registered type. On top of that, this media type has all of the security considerations described in RFC 3023.
- Interoperability considerations: 🧩 Even though an NLSML document is a complete XML document all on its own, a receiver that wants the fuller meaning of the content may want to access the resources the document links to. If an NLSML processor can't access or process those linked resources, the ultimate consumer of the data could end up behaving differently.
- Published specification: RFC 6787
- Applications that use this media type: MRCPv2 clients and servers
- Additional information: none
- Magic number(s): There is no single initial octet sequence that is always present for NLSML files.
- Person & email address to contact for further information: Sarvi Shanmugham, sarvi@cisco.com
- Intended usage: This media type is expected to be used only in conjunction with MRCPv2.
13.3. NLSML XML Schema Registration
IANA has registered and now maintains the following XML Schema 🌸 The info follows the template in RFC 3688 [RFC3688].
- XML element type: schema
- URI: urn:ietf:params:xml:schema:nlsml
- Registrant Contact: IESG
- XML: See Section 16.1.
13.4. MRCPv2 XML Namespace Registration
IANA has registered and now maintains the following XML Name space ✨ Again, the info follows the template in RFC 3688 [RFC3688].
- XML element type: ns
- URI: urn:ietf:params:xml:ns:mrcpv2
- Registrant Contact: IESG
- XML: RFC 6787
13.5. Text Media Type Registrations
IANA registered the following text media type through the process defined in RFC 4288 [RFC4288] 🦋
13.5.1. text/grammar-ref-list
- To: ietf-types@iana.org
- Subject: Registration of media type text/grammar-ref-list
- MIME media type name: text
- MIME subtype name: text/grammar-ref-list
- Required parameters: none
- Optional parameters: none
- Encoding considerations: 🧵 Depending on the transfer protocol, a transfer encoding may be necessary to deal with very long lines.
- Security considerations: 🛡️ This media type contains URIs that might point to external resources. Since those resources are assumed to be speech recognition grammars, similar considerations as for the media types 'application/srgs' and 'application/srgs+xml' apply.
- Interoperability considerations: '>' must be percent encoded in URIs according to RFC 3986 [RFC3986].
- Published specification: 🎤 The RECOGNIZE method of the MRCP protocol performs a recognition operation that matches input against a set of grammars. When matching against more than one grammar, it's sometimes necessary to use different weights for the individual grammars. These weights aren't a property of the grammar resource itself. Instead, they qualify the reference to that grammar for the particular recognition operation started by the RECOGNIZE method. The format of the proposed 'text/grammar-ref-list' media type looks like this:
body = *reference
reference = "<" uri ">" [parameters] CRLF
parameters = ";" parameter *(";" parameter)
parameter = attribute "=" valueThis specification currently only defines a 'weight' parameter, but new parameters MAY be added through the "Grammar Reference List Parameters" IANA registry established through this specification. Here's an example 👀
<http://example.com/grammars/field1.gram>
<http://example.com/grammars/field2.gram>;weight="0.85"
<session:field3@form-level.store>;weight="0.9"
<http://example.com/grammars/universals.gram>;weight="0.75"- Applications that use this media type: MRCPv2 clients and servers
- Additional information: none
- Magic number(s): none
- Person & email address to contact for further information: Sarvi Shanmugham, sarvi@cisco.com
- Intended usage: This media type is expected to be used only in conjunction with MRCPv2.
13.6. 'session' URI Scheme Registration
IANA registered the following new URI scheme 🔗 The info follows the template given in RFC 4395 [RFC4395].
- URI scheme name: session
- Status: Permanent
- URI scheme syntax: The syntax of this scheme is identical to the one defined for the "cid" scheme in Section 2 of RFC 2392 [RFC2392].
- URI scheme semantics: The URI identifies a data resource previously given to the network computing resource. The point is to allow access to that specific resource for the lifetime of the session with the entity storing it. The media type of the resource CAN vary, and there's no explicit mechanism for communicating the media type. This scheme is already widely used internally by existing implementations, and this registration is meant to provide info in the rare (and unfortunate) case that the scheme shows up somewhere else. The scheme SHOULD NOT be used for open Internet protocols. 🙈
- Encoding considerations: There are no other encoding considerations for the 'session' URIs not described in RFC 3986 [RFC3986]
- Applications/protocols that use this URI scheme name: This scheme name is used by MRCPv2 clients and servers.
- Interoperability considerations: ⏳ Heads up: none of the resources are accessible after the MCRPv2 session ends, which is where the name of the scheme comes from. If a client sets up one MRCPv2 session for the whole speech application, that's sufficient. But clients who create, terminate, and recreate MRCP sessions for performance or scalability reasons will lose access to resources established in the earlier session(s).
- Security considerations: 🔒 Generic security considerations for URIs described in RFC 3986 [RFC3986] apply to this scheme as well. The URIs defined here are an identification mechanism only. As long as the communication channel between client and server is secure, the server correctly accesses the resource associated with the URI, and the server ensures session-only lifetime and access for each URI, the only additional security issues are the ones tied to the types of media the URI refers to.
- Contact: Sarvi Shanmugham, sarvi@cisco.com
- Author/Change controller: IESG, iesg@ietf.org
- References: This specification, particularly Sections 6.2.7, 8.5.2, 9.5.1, and 9.9.
13.7. SDP Parameter Registrations
IANA has registered the following SDP parameter values 🎀 Each one follows the template given in RFC 4566 [RFC4566], Appendix B.
13.7.1. Sub-Registry "proto"
"TCP/MRCPv2" value of the "proto" parameter 🧚
- Contact name, email address, and telephone number: Sarvi Shanmugham, sarvi@cisco.com, +1.408.902.3875
- Name being registered (as it will appear in SDP): TCP/MRCPv2
- Long-form name in English: MCRPv2 over TCP
- Type of name: proto
- Explanation of name: This name represents the MCRPv2 protocol carried over TCP.
- Reference to specification of name: RFC 6787
"TCP/TLS/MRCPv2" value of the "proto" parameter 🔐
- Contact name, email address, and telephone number: Sarvi Shanmugham, sarvi@cisco.com, +1.408.902.3875
- Name being registered (as it will appear in SDP): TCP/TLS/MRCPv2
- Long-form name in English: MCRPv2 over TLS over TCP
- Type of name: proto
- Explanation of name: This name represents the MCRPv2 protocol carried over TLS over TCP.
- Reference to specification of name: RFC 6787
13.7.2. Sub-Registry "att-field (media-level)"
"resource" value of the "att-field" parameter 📦
- Contact name, email address, and telephone number: Sarvi Shanmugham, sarvi@cisco.com, +1.408.902.3875
- Attribute name (as it will appear in SDP): resource
- Long-form attribute name in English: MRCPv2 resource type
- Type of attribute: media-level
- Subject to charset attribute? no
- Explanation of attribute: See Section 4.2 of RFC 6787 for description and examples.
- Specification of appropriate attribute values: See section Section 13.1.1 of RFC 6787.
"channel" value of the "att-field" parameter 📡
- Contact name, email address, and telephone number: Sarvi Shanmugham, sarvi@cisco.com, +1.408.902.3875
- Attribute name (as it will appear in SDP): channel
- Long-form attribute name in English: MRCPv2 resource channel identifier
- Type of attribute: media-level
- Subject to charset attribute? no
- Explanation of attribute: See Section 4.2 of RFC 6787 for description and examples.
- Specification of appropriate attribute values: See Section 4.2 and the "channel-id" ABNF production rules of RFC 6787.
"cmid" value of the "att-field" parameter 💜
- Contact name, email address, and telephone number: Sarvi Shanmugham, sarvi@cisco.com, +1.408.902.3875
- Attribute name (as it will appear in SDP): cmid
- Long-form attribute name in English: MRCPv2 resource channel media identifier
- Type of attribute: media-level
- Subject to charset attribute? no
- Explanation of attribute: See Section 4.4 of RFC 6787 for description and examples.
- Specification of appropriate attribute values: See Section 4.4 and the "cmid-attribute" ABNF production rules of RFC 6787.
14. Examples ✨
14.1. Message Flow
okay this part is the fun one 🎬 The following is an example of a typical MRCPv2 session of speech synthesis and recognition between a client and a server. The SDP "s=" attribute in these examples has a text description value to help you follow along, but keep in mind that RFC 3264 [RFC3264] recommends that messages actually put on the wire use a space or a dash.
The figure below shows opening a session to the MRCPv2 server. This exchange doesn't allocate a resource or set up media. It just establishes a SIP session with the MRCPv2 server 🦋
C->S:
INVITE sip:mresources@example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg1
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323123 INVITE
Contact:<sip:sarvi@client.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=sarvi 2614933546 2614933546 IN IP4 192.0.2.12
s=Set up MRCPv2 control and audio
i=Initial contact
c=IN IP4 192.0.2.12The client asks the server to create a synthesizer resource control channel for speech synthesis 💖 This also adds a media stream so the generated speech can come back to the client 🎀 In this example, the client asks for a brand-new MRCPv2 TCP stream between client and server. In the requests after this, the client asks to reuse that existing connection ✨
S->C:
SIP/2.0 200 OK
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg1;received=192.0.32.10
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323123 INVITE
Contact:<sip:mresources@server.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=- 3000000001 3000000001 IN IP4 192.0.2.11
s=Set up MRCPv2 control and audio
i=Initial contact
c=IN IP4 192.0.2.11
C->S:
ACK sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg2
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323123 ACK
Content-Length:0
C->S:
INVITE sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg3
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323124 INVITE
Contact:<sip:sarvi@client.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=sarvi 2614933546 2614933547 IN IP4 192.0.2.12
s=Set up MRCPv2 control and audio
i=Add TCP channel, synthesizer and one-way audio
c=IN IP4 192.0.2.12
t=0 0
m=application 9 TCP/MRCPv2 1
a=setup:active
a=connection:new
a=resource:speechsynth
a=cmid:1
m=audio 49170 RTP/AVP 0 96
a=rtpmap:0 pcmu/8000
a=rtpmap:96 telephone-event/8000
a=fmtp:96 0-15
a=recvonly
a=mid:1
S->C:
SIP/2.0 200 OK
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg3;received=192.0.32.10
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323124 INVITE
Contact:<sip:mresources@server.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=- 3000000001 3000000002 IN IP4 192.0.2.11
s=Set up MRCPv2 control and audio
i=Add TCP channel, synthesizer and one-way audio
c=IN IP4 192.0.2.11
t=0 0
m=application 32416 TCP/MRCPv2 1
a=setup:passive
a=connection:new
a=channel:32AECB23433801@speechsynth
a=cmid:1
m=audio 48260 RTP/AVP 0
a=rtpmap:0 pcmu/8000
a=sendonly
a=mid:1
C->S:
ACK sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg4
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323124 ACK
Content-Length:0This exchange allocates an additional resource control channel for a recognizer. Since a recognizer needs an audio stream to listen to, this step also flips the audio stream to sendrecv, so it becomes a two-way audio stream 🦋
C->S:
INVITE sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg5
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323125 INVITE
Contact:<sip:sarvi@client.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=sarvi 2614933546 2614933548 IN IP4 192.0.2.12
s=Set up MRCPv2 control and audio
i=Add recognizer and duplex the audio
c=IN IP4 192.0.2.12
t=0 0
m=application 9 TCP/MRCPv2 1
a=setup:active
a=connection:existing
a=resource:speechsynth
a=cmid:1
m=audio 49170 RTP/AVP 0 96
a=rtpmap:0 pcmu/8000
a=rtpmap:96 telephone-event/8000
a=fmtp:96 0-15
a=recvonly
a=mid:1
m=application 9 TCP/MRCPv2 1
a=setup:active
a=connection:existing
a=resource:speechrecog
a=cmid:2
m=audio 49180 RTP/AVP 0 96
a=rtpmap:0 pcmu/8000
a=rtpmap:96 telephone-event/8000
a=fmtp:96 0-15
a=sendonly
a=mid:2
S->C:
SIP/2.0 200 OK
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg5;received=192.0.32.10
To:MediaServer <sip:mresources@example.com>;tag=62784
From:sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323125 INVITE
Contact:<sip:mresources@server.example.com>
Content-Type:application/sdp
Content-Length:...
v=0
o=- 3000000001 3000000003 IN IP4 192.0.2.11
s=Set up MRCPv2 control and audio
i=Add recognizer and duplex the audio
c=IN IP4 192.0.2.11
t=0 0
m=application 32416 TCP/MRCPv2 1
a=channel:32AECB23433801@speechsynth
a=cmid:1
m=audio 48260 RTP/AVP 0
a=rtpmap:0 pcmu/8000
a=sendonly
a=mid:1
m=application 32416 TCP/MRCPv2 1
a=channel:32AECB23433801@speechrecog
a=cmid:2
m=audio 48260 RTP/AVP 0
a=rtpmap:0 pcmu/8000
a=rtpmap:96 telephone-event/8000
a=fmtp:96 0-15
a=recvonly
a=mid:2
C->S:
ACK sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg6
Max-Forwards:6
To:MediaServer <sip:mresources@example.com>;tag=62784
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
Call-ID:a84b4c76e66710
CSeq:323125 ACK
Content-Length:0A MRCPv2 SPEAK request kicks off the speech 🗣️✨
C->S:
MRCP/2.0 ... SPEAK 543257
Channel-Identifier:32AECB23433801@speechsynth
Kill-On-Barge-In:false
Voice-gender:neutral
Voice-age:25
Prosody-volume:medium
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>You have 4 new messages.</s>
<s>The first is from Stephanie Williams
<mark name="Stephanie"/>
and arrived at <break/>
<say-as interpret-as="vxml:time">0345p</say-as>.</s>
<s>The subject is <prosody
rate="-20%">ski trip</prosody></s>
</p>
</speak>
S->C:
MRCP/2.0 ... 543257 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechsynth
Speech-Marker:timestamp=857205015059The synthesizer hits the special marker in the message being spoken, and it dutifully lets the client know 🫶
S->C: MRCP/2.0 ... SPEECH-MARKER 543257 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechsynth
Speech-Marker:timestamp=857206027059;StephanieThe synthesizer wraps up the SPEAK request 🎉
S->C: MRCP/2.0 ... SPEAK-COMPLETE 543257 COMPLETE
Channel-Identifier:32AECB23433801@speechsynth
Speech-Marker:timestamp=857207685213;StephanieNow the recognizer gets asked to listen for the customer's choices 👂💅
C->S: MRCP/2.0 ... RECOGNIZE 543258
Channel-Identifier:32AECB23433801@speechrecog
Content-Type:application/srgs+xml
Content-Length:...
<?xml version="1.0"?>
<!-- the default grammar language is US English -->
<grammar xmlns="http://www.w3.org/2001/06/grammar"
xml:lang="en-US" version="1.0" root="request">
<!-- single language attachment to a rule expansion -->
<rule id="request">
Can I speak to
<one-of xml:lang="fr-CA">
<item>Michel Tremblay</item>
<item>Andre Roy</item>
</one-of>
</rule>
</grammar>
S->C: MRCP/2.0 ... 543258 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecogThe client sends the next MRCPv2 SPEAK method 🍓
C->S: MRCP/2.0 ... SPEAK 543259
Channel-Identifier:32AECB23433801@speechsynth
Kill-On-Barge-In:true
Content-Type:application/ssml+xml
Content-Length:...
<?xml version="1.0"?>
<speak version="1.0"
xmlns="http://www.w3.org/2001/10/synthesis"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.w3.org/2001/10/synthesis
http://www.w3.org/TR/speech-synthesis/synthesis.xsd"
xml:lang="en-US">
<p>
<s>Welcome to ABC corporation.</s>
<s>Who would you like to talk to?</s>
</p>
</speak>
S->C: MRCP/2.0 ... 543259 200 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechsynth
Speech-Marker:timestamp=857207696314Okay, this next bit of the story shows kill-on-barge-in in action 🌈💥 Because this last SPEAK request had Kill-On-Barge-In set to "true", the moment the recognizer (the server) fired off a START-OF-INPUT event while a SPEAK was still going, the client immediately sent a BARGE-IN-OCCURRED method to the synthesizer resource 🙈 The speech synthesizer then cut off playback and told the client about it. The completion-cause code is what showed it was a kill-on-barge-in interruption and not a normal completion 💜
S->C: MRCP/2.0 ... START-OF-INPUT 543258 IN-PROGRESS
Channel-Identifier:32AECB23433801@speechrecog
Proxy-Sync-Id:987654321
C->S: MRCP/2.0 ... BARGE-IN-OCCURRED 543259
Channel-Identifier:32AECB23433801@speechsynth
Proxy-Sync-Id:987654321
S->C: MRCP/2.0 ... 543259 200 COMPLETE
Channel-Identifier:32AECB23433801@speechsynth
Active-Request-Id-List:543258
Speech-Marker:timestamp=857206096314Heads up, bestie: the recognition and synthesizer resources are in the same session on the same server, so to get a faster response, the server might have relayed the start-of-input condition to the synthesizer directly, before the expected BARGE-IN-OCCURRED event showed up 👀 However, any such communication is outside the scope of MRCPv2 🦋
The recognizer resource matched what was said to a grammar and cooked up some results 🎤✨ Then the server sent the result back inside the RECOGNITION-COMPLETE event, periodt 💅
S->C: MRCP/2.0 ... SPEAK-COMPLETE 543259 COMPLETE
Channel-Identifier:32AECB23433801@speechsynth
Completion-Cause:001 barge-in
Speech-Marker:timestamp=857207685213S->C: MRCP/2.0 ... RECOGNITION-COMPLETE 543258 COMPLETE
Channel-Identifier:32AECB23433801@speechrecog
Completion-Cause:000 success
Waveform-URI:<http://web.media.com/session123/audio.wav>;
size=423523;duration=25432
Content-Type:application/nlsml+xml
Content-Length:...
<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="session:request1@form-level.store">
<interpretation>
<instance name="Person">
<ex:Person>
<ex:Name> Andre Roy </ex:Name>
</ex:Person>
</instance>
<input> may I speak to Andre Roy </input>
</interpretation>
</result>The client was all done with the session and its resources 🧹, so it sent a SIP BYE to close the SIP session 👋 That took down every control channel and freed up every resource that was allocated under the session. Main character exit energy 🦋
C->S: BYE sip:mresources@server.example.com SIP/2.0
Via:SIP/2.0/TCP client.atlanta.example.com:5060;
branch=z9hG4bK74bg7
Max-Forwards:6
From:Sarvi <sip:sarvi@example.com>;tag=1928301774
To:MediaServer <sip:mresources@example.com>;tag=62784
Call-ID:a84b4c76e66710
CSeq:323126 BYE
Content-Length:014.2. Recognition Result Examples
Okay, now for the juicy part: what recognition results actually look like 🍓 These examples are the vibe check for everything above 💖
14.2.1. Simple ASR Ambiguity
System: To which city will you be traveling?
User: I want to go to Pittsburgh.<?xml version="1.0"?>
<result xmlns="urn:ietf:params:xml:ns:mrcpv2"
xmlns:ex="http://www.example.com/example"
grammar="http://www.example.com/flight">
<interpretation confidence="0.6">
<instance>
<ex:airline>
<ex:to_city>Pittsburgh</ex:to_city>
<ex:airline>
<instance>
<input mode="speech">
I want to go to Pittsburgh
</input>
</interpretation>
<interpretation confidence="0.4"
<instance>
<ex:airline>
<ex:to_city>Stockholm</ex:to_city>
</ex:airline>
</instance>
<input>I want to go to Stockholm</input>
</interpretation>
</result>14.2.2. Mixed Initiative
System: What would you like?
User: I would like 2 pizzas, one with pepperoni and cheese,
one with sausage and a bottle of coke, to go.Bestie, this one's a whole order 🍕 It has an order object holding "food_item", "drink_item", and "delivery_method" objects. The representation assumes there are no ambiguities in the speech or natural language processing, which is a big assumption 🙈 It also assumes some level of intra-sentential anaphora resolution, i.e., it resolves both "one"s as "pizza". Iconic resolving skills 👑
<?xml version="1.0"?>
<nl:result xmlns:nl="urn:ietf:params:xml:ns:mrcpv2"
xmlns="http://www.example.com/example"
grammar="http://www.example.com/foodorder">
<nl:interpretation confidence="1.0" >
<nl:instance>
<order>
<food_item confidence="1.0">
<pizza>
<ingredients confidence="1.0">
pepperoni
</ingredients>
<ingredients confidence="1.0">
cheese
</ingredients>
</pizza>
<pizza>
<ingredients>sausage</ingredients>
</pizza>
</food_item>
<drink_item confidence="1.0">
<size>2-liter</size>
</drink_item>
<delivery_method>to go</delivery_method>
</order>
</nl:instance>
<nl:input mode="speech">I would like 2 pizzas,
one with pepperoni and cheese, one with sausage
and a bottle of coke, to go.
</nl:input>
</nl:interpretation>
</nl:result>14.2.3. DTMF Input
Mixing DTMF and speech is represented with nested input elements 🎹 Like this:
User: My pin is (dtmf 1 2 3 4)<input>
<input mode="speech" confidence ="1.0"
timestamp-start="2000-04-03T0:00:00"
timestamp-end="2000-04-03T0:00:01.5">My pin is
</input>
<input mode="dtmf" confidence ="1.0"
timestamp-start="2000-04-03T0:00:01.5"
timestamp-end="2000-04-03T0:00:02.0">1 2 3 4
</input>
</input>Heads up: grammars that recognize mixtures of speech and DTMF are not currently possible in SRGS 🙃 Still, this representation might be needed for other NLSML applications, and this mixture capability might show up in future versions of SRGS. Watch this space ✨
14.2.4. Interpreting Meta-Dialog and Meta-Task Utterances
Real talk, people do meta-dialog and meta-task stuff all the time, like asking about the conversation itself 🙄 This spec is flexible enough that these meta-utterances can be represented on an application-specific basis, with no extra standard markup needed. So chill 🧚
Here are two examples of how meta-task and meta-dialog utterances might be represented.
System: What toppings do you want on your pizza?
User: What toppings do you have?<interpretation grammar="http://www.example.com/toppings">
<instance>
<question>
<questioned_item>toppings<questioned_item>
<questioned_property>
availability
</questioned_property>
</question>
</instance>
<input mode="speech">
what toppings do you have?
</input>
</interpretation>User: slow down.<interpretation grammar="http://www.example.com/generalCommandsGrammar">
<instance>
<command>
<action>reduce speech rate</action>
<doer>system</doer>
</command>
</instance>
<input mode="speech">slow down</input>
</interpretation>14.2.5. Anaphora and Deixis
Okay, this one's about the tricky words, like pronouns and "this" 👀 Anaphoric references point back to something mentioned earlier in the conversation, like pronouns and definite noun phrases. Deictic references point at something in the non-linguistic context, like a thing on screen. Both can leave the interpretation a bit fuzzy, because there may not be enough unambiguous linguistic context to pin down their exact role. To represent unresolved anaphora and deixis with this spec, one strategy is for the developer to define a more surface-oriented representation that leaves the details of the reference open. (This assumes a later component is responsible for actually resolving the reference, so somebody's gotta do it 💅)
Example: (ignoring the issue of representing the input from the pointing gesture.)
System: What do you want to drink?
User: I want this. (clicks on picture of large root beer.)<?xml version="1.0"?>
<nl:result xmlns:nl="urn:ietf:params:xml:ns:mrcpv2"
xmlns="http://www.example.com/example"
grammar="http://www.example.com/beverages.grxml">
<nl:interpretation>
<nl:instance>
<doer>I</doer>
<action>want</action>
<object>this</object>
</nl:instance>
<nl:input mode="speech">I want this</nl:input>
</nl:interpretation>
</nl:result>14.2.6. Distinguishing Individual Items from Sets with One Member
For programming convenience, it helps to tell apart an individual item from a set that only has one member, in the XML representation of semantic results 🧠 Like, a pizza order might have exactly one pizza, but a pizza can have zero or more toppings. There's no standard way to mark that difference directly in XML, so in the current framework the developer is free to pick any convention that gets the info across in the markup. One strategy is wrapping the set of items in a grouping element, like this:
<order>
<pizza>
<topping-group>
<topping>mushrooms</topping>
</topping-group>
</pizza>
<drink>coke</drink>
</order>In this example, the programmer can assume there's supposed to be exactly one pizza and one drink in the order 🍓 The fact that there's only one topping is just an accident of this particular pizza order.
Note that the client controls both the grammar and the semantics returned upon grammar matches, so the MRCPv2 user is fully empowered to get results back in NLSML in whatever way makes the interpretation clear to them 💖
14.2.7. Extensibility
Extensibility in NLSML comes from result content flexibility, like we saw with the meta-utterances and anaphora stuff 🦋 Sophisticated systems can use NLSML to carry application-specific info that more basic systems wouldn't use, like defining speech acts 🌈
15. ABNF Normative Definition ✨
okay bestie, this is the grammar section 💅 everything below is written in ABNF (the little rule language from the standards world), and it leans on the core rules from Section B.1 of RFC 5234 [RFC5234]. think of it as the blueprint that every MRCP message has to follow 🎀
the grammar starts with the basics: whitespace, UTF-8, quoted strings, tokens, and the URI pieces 🌸
LWS = [*WSP CRLF] 1*WSP ; linear whitespace
SWS = [LWS] ; sep whitespace
UTF8-NONASCII = %xC0-DF 1UTF8-CONT
/ %xE0-EF 2UTF8-CONT
/ %xF0-F7 3UTF8-CONT
/ %xF8-FB 4UTF8-CONT
/ %xFC-FD 5UTF8-CONT
UTF8-CONT = %x80-BF
UTFCHAR = %x21-7E
/ UTF8-NONASCII
param = *pchar
quoted-string = SWS DQUOTE *(qdtext / quoted-pair )
DQUOTE
qdtext = LWS / %x21 / %x23-5B / %x5D-7E
/ UTF8-NONASCII
quoted-pair = "\" (%x00-09 / %x0B-0C / %x0E-7F)
token = 1*(alphanum / "-" / "." / "!" / "%" / "*"
/ "_" / "+" / "`" / "'" / "~" )
reserved = ";" / "/" / "?" / ":" / "@" / "&" / "="
/ "+" / "$" / ","
mark = "-" / "_" / "." / "!" / "~" / "*" / "'"
/ "(" / ")"
unreserved = alphanum / mark
pchar = unreserved / escaped
/ ":" / "@" / "&" / "=" / "+" / "$" / ","
alphanum = ALPHA / DIGIT
BOOLEAN = "true" / "false"
FLOAT = *DIGIT ["." *DIGIT]
escaped = "%" HEXDIG HEXDIG
fragment = *uric
uri = [ absoluteURI / relativeURI ]
[ "#" fragment ]
absoluteURI = scheme ":" ( hier-part / opaque-part )
relativeURI = ( net-path / abs-path / rel-path )
[ "?" query ]
hier-part = ( net-path / abs-path ) [ "?" query ]
net-path = "//" authority [ abs-path ]
abs-path = "/" path-segments
rel-path = rel-segment [ abs-path ]
rel-segment = 1*( unreserved / escaped / ";" / "@"
/ "&" / "=" / "+" / "$" / "," )
opaque-part = uric-no-slash *uric
uric = reserved / unreserved / escaped
uric-no-slash = unreserved / escaped / ";" / "?" / ":"
/ "@" / "&" / "=" / "+" / "$" / ","
path-segments = segment *( "/" segment )
segment = *pchar *( ";" param )
scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
authority = srvr / reg-name
srvr = [ [ userinfo "@" ] hostport ]
reg-name = 1*( unreserved / escaped / "$" / ","
/ ";" / ":" / "@" / "&" / "=" / "+" )
query = *uric
userinfo = ( user ) [ ":" password ] "@"
user = 1*( unreserved / escaped
/ user-unreserved )
user-unreserved = "&" / "=" / "+" / "$" / "," / ";"
/ "?" / "/"
password = *( unreserved / escaped
/ "&" / "=" / "+" / "$" / "," )
hostport = host [ ":" port ]
host = hostname / IPv4address / IPv6reference
hostname = *( domainlabel "." ) toplabel [ "." ]
domainlabel = alphanum / alphanum *( alphanum / "-" )
alphanum
toplabel = ALPHA / ALPHA *( alphanum / "-" )
alphanum
IPv4address = 1*3DIGIT "." 1*3DIGIT "." 1*3DIGIT "."
1*3DIGIT
IPv6reference = "[" IPv6address "]"
IPv6address = hexpart [ ":" IPv4address ]
hexpart = hexseq / hexseq "::" [ hexseq ] / "::"
[ hexseq ]
hexseq = hex4 *( ":" hex4)
hex4 = 1*4HEXDIG
port = 1*19DIGITnext up: the actual MRCP message structure, which is the main character of this whole grammar 🦋 every message starts with a start-line, then headers, then an optional body 🧚
; generic-message is the top-level rule
generic-message = start-line message-header CRLF
[ message-body ]
message-body = *OCTET
start-line = request-line / response-line / event-line
request-line = mrcp-version SP message-length SP method-name
SP request-id CRLF
response-line = mrcp-version SP message-length SP request-id
SP status-code SP request-state CRLF
event-line = mrcp-version SP message-length SP event-name
SP request-id SP request-state CRLF
method-name = generic-method
/ synthesizer-method
/ recognizer-method
/ recorder-method
/ verifier-method
generic-method = "SET-PARAMS"
/ "GET-PARAMS"
request-state = "COMPLETE"
/ "IN-PROGRESS"
/ "PENDING"
event-name = synthesizer-event
/ recognizer-event
/ recorder-event
/ verifier-event
message-header = 1*(generic-header / resource-header / generic-field)
generic-field = field-name ":" [ field-value ]
field-name = token
field-value = *LWS field-content *( CRLF 1*LWS field-content)
field-content = <the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string>
resource-header = synthesizer-header
/ recognizer-header
/ recorder-header
/ verifier-header
generic-header = channel-identifier
/ accept
/ active-request-id-list
/ proxy-sync-id
/ accept-charset
/ content-type
/ content-id
/ content-base
/ content-encoding
/ content-location
/ content-length
/ fetch-timeout
/ cache-control
/ logging-tag
/ set-cookie
/ vendor-specific
; -- content-id is as defined in RFC 2392, RFC 2046 and RFC 5322
; -- accept and accept-charset are as defined in RFC 2616
mrcp-version = "MRCP" "/" 1*2DIGIT "." 1*2DIGIT
message-length = 1*19DIGIT
request-id = 1*10DIGIT
status-code = 3DIGIT
channel-identifier = "Channel-Identifier" ":"
channel-id CRLF
channel-id = 1*alphanum "@" 1*alphanum
active-request-id-list = "Active-Request-Id-List" ":"
request-id *("," request-id) CRLF
proxy-sync-id = "Proxy-Sync-Id" ":" 1*VCHAR CRLF
content-base = "Content-Base" ":" absoluteURI CRLF
content-length = "Content-Length" ":" 1*19DIGIT CRLF
content-type = "Content-Type" ":" media-type-value CRLF
media-type-value = type "/" subtype *( ";" parameter )
type = token
subtype = token
parameter = attribute "=" value
attribute = token
value = token / quoted-string
content-encoding = "Content-Encoding" ":"
*WSP content-coding
*(*WSP "," *WSP content-coding *WSP )
CRLF
content-coding = token
content-location = "Content-Location" ":"
( absoluteURI / relativeURI ) CRLF
cache-control = "Cache-Control" ":"
[*WSP cache-directive
*( *WSP "," *WSP cache-directive *WSP )]
CRLF
fetch-timeout = "Fetch-Timeout" ":" 1*19DIGIT CRLF
cache-directive = "max-age" "=" delta-seconds
/ "max-stale" ["=" delta-seconds ]
/ "min-fresh" "=" delta-seconds
delta-seconds = 1*19DIGIT
logging-tag = "Logging-Tag" ":" 1*UTFCHAR CRLF
vendor-specific = "Vendor-Specific-Parameters" ":"
[vendor-specific-av-pair
*(";" vendor-specific-av-pair)] CRLF
vendor-specific-av-pair = vendor-av-pair-name "="
value
vendor-av-pair-name = 1*UTFCHAR
set-cookie = "Set-Cookie:" SP set-cookie-string
set-cookie-string = cookie-pair *( ";" SP cookie-av )
cookie-pair = cookie-name "=" cookie-value
cookie-name = token
cookie-value = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
cookie-octet = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
token = <token, defined in [RFC2616], Section 2.2>
cookie-av = expires-av / max-age-av / domain-av /
path-av / secure-av / httponly-av /
extension-av / age-av
expires-av = "Expires=" sane-cookie-date
sane-cookie-date = <rfc1123-date, defined in [RFC2616], Section 3.3.1>
max-age-av = "Max-Age=" non-zero-digit *DIGIT
non-zero-digit = %x31-39
domain-av = "Domain=" domain-value
domain-value = <subdomain>
path-av = "Path=" path-value
path-value = <any CHAR except CTLs or ";">
secure-av = "Secure"
httponly-av = "HttpOnly"
extension-av = <any CHAR except CTLs or ";">
age-av = "Age=" delta-secondsand that's the whole grammar, fr fr 🫶 every header, start-line and cookie bit gets its shape from up here, so when you're building a parser, this is the page you keep open 🔥
Okay, the grammar keeps going, and it's giving a lot of headers 💅
; Synthesizer ABNF
synthesizer-method = "SPEAK"
/ "STOP"
/ "PAUSE"
/ "RESUME"
/ "BARGE-IN-OCCURRED"
/ "CONTROL"
/ "DEFINE-LEXICON"
synthesizer-event = "SPEECH-MARKER"
/ "SPEAK-COMPLETE"
synthesizer-header = jump-size
/ kill-on-barge-in
/ speaker-profile
/ completion-cause
/ completion-reason
/ voice-parameter
/ prosody-parameter
/ speech-marker
/ speech-language
/ fetch-hint
/ audio-fetch-hint
/ failed-uri
/ failed-uri-cause
/ speak-restart
/ speak-length
/ load-lexicon
/ lexicon-search-order
jump-size = "Jump-Size" ":" speech-length-value CRLF
speech-length-value = numeric-speech-length
/ text-speech-length
text-speech-length = 1*UTFCHAR SP "Tag"
numeric-speech-length = ("+" / "-") positive-speech-length
positive-speech-length = 1*19DIGIT SP numeric-speech-unit
numeric-speech-unit = "Second"
/ "Word"
/ "Sentence"
/ "Paragraph"
kill-on-barge-in = "Kill-On-Barge-In" ":" BOOLEAN
CRLF
speaker-profile = "Speaker-Profile" ":" uri CRLF
completion-cause = "Completion-Cause" ":" cause-code SP
cause-name CRLF
cause-code = 3DIGIT
cause-name = *VCHAR
completion-reason = "Completion-Reason" ":"
quoted-string CRLF
voice-parameter = voice-gender
/ voice-age
/ voice-variant
/ voice-name
voice-gender = "Voice-Gender:" voice-gender-value CRLF
voice-gender-value = "male"
/ "female"
/ "neutral"
voice-age = "Voice-Age:" 1*3DIGIT CRLF
voice-variant = "Voice-Variant:" 1*19DIGIT CRLF
voice-name = "Voice-Name:"
1*UTFCHAR *(1*WSP 1*UTFCHAR) CRLF
prosody-parameter = "Prosody-" prosody-param-name ":"
prosody-param-value CRLF
prosody-param-name = 1*VCHAR
prosody-param-value = 1*VCHAR
timestamp = "timestamp" "=" time-stamp-value
time-stamp-value = 1*20DIGIT
speech-marker = "Speech-Marker" ":"
timestamp
[";" 1*(UTFCHAR / %x20)] CRLF
speech-language = "Speech-Language" ":" 1*VCHAR CRLF
fetch-hint = "Fetch-Hint" ":" ("prefetch" / "safe") CRLF
audio-fetch-hint = "Audio-Fetch-Hint" ":"
("prefetch" / "safe" / "stream") CRLF
failed-uri = "Failed-URI" ":" absoluteURI CRLF
failed-uri-cause = "Failed-URI-Cause" ":" 1*UTFCHAR CRLF
speak-restart = "Speak-Restart" ":" BOOLEAN CRLF
speak-length = "Speak-Length" ":" positive-length-value
CRLF
positive-length-value = positive-speech-length
/ text-speech-length
load-lexicon = "Load-Lexicon" ":" BOOLEAN CRLF
lexicon-search-order = "Lexicon-Search-Order" ":"
"<" absoluteURI ">" *(" " "<" absoluteURI ">") CRLF
; Recognizer ABNF
recognizer-method = recog-only-method
/ enrollment-method
recog-only-method = "DEFINE-GRAMMAR"
/ "RECOGNIZE"
/ "INTERPRET"
/ "GET-RESULT"
/ "START-INPUT-TIMERS"
/ "STOP"
enrollment-method = "START-PHRASE-ENROLLMENT"
/ "ENROLLMENT-ROLLBACK"
/ "END-PHRASE-ENROLLMENT"
/ "MODIFY-PHRASE"
/ "DELETE-PHRASE"
recognizer-event = "START-OF-INPUT"
/ "RECOGNITION-COMPLETE"
/ "INTERPRETATION-COMPLETE"
recognizer-header = recog-only-header
/ enrollment-header
recog-only-header = confidence-threshold
/ sensitivity-level
/ speed-vs-accuracy
/ n-best-list-length
/ input-type
/ no-input-timeout
/ recognition-timeout
/ waveform-uri
/ input-waveform-uri
/ completion-cause
/ completion-reason
/ recognizer-context-block
/ start-input-timers
/ speech-complete-timeout
/ speech-incomplete-timeout
/ dtmf-interdigit-timeout
/ dtmf-term-timeout
/ dtmf-term-char
/ failed-uri
/ failed-uri-cause
/ save-waveform
/ media-type
/ new-audio-channel
/ speech-language
/ ver-buffer-utterance
/ recognition-mode
/ cancel-if-queue
/ hotword-max-duration
/ hotword-min-duration
/ interpret-text
/ dtmf-buffer-time
/ clear-dtmf-buffer
/ early-no-match
enrollment-header = num-min-consistent-pronunciations
/ consistency-threshold
/ clash-threshold
/ personal-grammar-uri
/ enroll-utterance
/ phrase-id
/ phrase-nl
/ weight
/ save-best-waveform
/ new-phrase-id
/ confusable-phrases-uri
/ abort-phrase-enrollment
confidence-threshold = "Confidence-Threshold" ":"
FLOAT CRLF
sensitivity-level = "Sensitivity-Level" ":" FLOAT
CRLF
speed-vs-accuracy = "Speed-Vs-Accuracy" ":" FLOAT
CRLF
n-best-list-length = "N-Best-List-Length" ":" 1*19DIGIT
CRLF
input-type = "Input-Type" ":" inputs CRLF
inputs = "speech" / "dtmf"
no-input-timeout = "No-Input-Timeout" ":" 1*19DIGIT
CRLF
recognition-timeout = "Recognition-Timeout" ":" 1*19DIGIT
CRLF
waveform-uri = "Waveform-URI" ":" ["<" uri ">"
";" "size" "=" 1*19DIGIT
";" "duration" "=" 1*19DIGIT] CRLF
recognizer-context-block = "Recognizer-Context-Block" ":"
[1*VCHAR] CRLF
start-input-timers = "Start-Input-Timers" ":"
BOOLEAN CRLF
speech-complete-timeout = "Speech-Complete-Timeout" ":"
1*19DIGIT CRLF
speech-incomplete-timeout = "Speech-Incomplete-Timeout" ":"
1*19DIGIT CRLF
dtmf-interdigit-timeout = "DTMF-Interdigit-Timeout" ":"
1*19DIGIT CRLF
dtmf-term-timeout = "DTMF-Term-Timeout" ":" 1*19DIGIT
CRLF
dtmf-term-char = "DTMF-Term-Char" ":" VCHAR CRLF
save-waveform = "Save-Waveform" ":" BOOLEAN CRLF
new-audio-channel = "New-Audio-Channel" ":"
BOOLEAN CRLF
recognition-mode = "Recognition-Mode" ":"
"normal" / "hotword" CRLF
cancel-if-queue = "Cancel-If-Queue" ":" BOOLEAN CRLF
hotword-max-duration = "Hotword-Max-Duration" ":"
1*19DIGIT CRLF
hotword-min-duration = "Hotword-Min-Duration" ":"
1*19DIGIT CRLF
interpret-text = "Interpret-Text" ":" 1*VCHAR CRLF
dtmf-buffer-time = "DTMF-Buffer-Time" ":" 1*19DIGIT CRLF
clear-dtmf-buffer = "Clear-DTMF-Buffer" ":" BOOLEAN CRLF
early-no-match = "Early-No-Match" ":" BOOLEAN CRLF
num-min-consistent-pronunciations =
"Num-Min-Consistent-Pronunciations" ":" 1*19DIGIT CRLF
consistency-threshold = "Consistency-Threshold" ":" FLOAT
CRLF
clash-threshold = "Clash-Threshold" ":" FLOAT CRLF
personal-grammar-uri = "Personal-Grammar-URI" ":" uri CRLF
enroll-utterance = "Enroll-Utterance" ":" BOOLEAN CRLF
phrase-id = "Phrase-ID" ":" 1*VCHAR CRLF
phrase-nl = "Phrase-NL" ":" 1*UTFCHAR CRLF
weight = "Weight" ":" FLOAT CRLF
save-best-waveform = "Save-Best-Waveform" ":"
BOOLEAN CRLF
new-phrase-id = "New-Phrase-ID" ":" 1*VCHAR CRLF
confusable-phrases-uri = "Confusable-Phrases-URI" ":"
uri CRLF
abort-phrase-enrollment = "Abort-Phrase-Enrollment" ":"
BOOLEAN CRLF
; Recorder ABNF
recorder-method = "RECORD"
/ "STOP"
/ "START-INPUT-TIMERS"
recorder-event = "START-OF-INPUT"
/ "RECORD-COMPLETE"
recorder-header = sensitivity-level
/ no-input-timeout
/ completion-cause
/ completion-reason
/ failed-uri
/ failed-uri-cause
/ record-uri
/ media-type
/ max-time
/ trim-length
/ final-silence
/ capture-on-speech
/ ver-buffer-utterance
/ start-input-timers
/ new-audio-channel
record-uri = "Record-URI" ":" [ "<" uri ">"
";" "size" "=" 1*19DIGIT
";" "duration" "=" 1*19DIGIT] CRLF
media-type = "Media-Type" ":" media-type-value CRLF
max-time = "Max-Time" ":" 1*19DIGIT CRLF
trim-length = "Trim-Length" ":" 1*19DIGIT CRLF
final-silence = "Final-Silence" ":" 1*19DIGIT CRLF
capture-on-speech = "Capture-On-Speech " ":"
BOOLEAN CRLFOkay, the Verifier ABNF is next and it's giving a whole lineup 💅
; Verifier ABNF
verifier-method = "START-SESSION"
/ "END-SESSION"
/ "QUERY-VOICEPRINT"
/ "DELETE-VOICEPRINT"
/ "VERIFY"
/ "VERIFY-FROM-BUFFER"
/ "VERIFY-ROLLBACK"
/ "STOP"
/ "CLEAR-BUFFER"
/ "START-INPUT-TIMERS"
/ "GET-INTERMEDIATE-RESULT"
verifier-event = "VERIFICATION-COMPLETE"
/ "START-OF-INPUT"
verifier-header = repository-uri
/ voiceprint-identifier
/ verification-mode
/ adapt-model
/ abort-model
/ min-verification-score
/ num-min-verification-phrases
/ num-max-verification-phrases
/ no-input-timeout
/ save-waveform
/ media-type
/ waveform-uri
/ voiceprint-exists
/ ver-buffer-utterance
/ input-waveform-uri
/ completion-cause
/ completion-reason
/ speech-complete-timeout
/ new-audio-channel
/ abort-verification
/ start-input-timers
/ input-type
repository-uri = "Repository-URI" ":" uri CRLF
voiceprint-identifier = "Voiceprint-Identifier" ":"
vid *[";" vid] CRLF
vid = 1*VCHAR ["." 1*VCHAR]
verification-mode = "Verification-Mode" ":"
verification-mode-string
verification-mode-string = "train" / "verify"
adapt-model = "Adapt-Model" ":" BOOLEAN CRLF
abort-model = "Abort-Model" ":" BOOLEAN CRLF
min-verification-score = "Min-Verification-Score" ":"
[ %x2D ] FLOAT CRLF
num-min-verification-phrases = "Num-Min-Verification-Phrases"
":" 1*19DIGIT CRLF
num-max-verification-phrases = "Num-Max-Verification-Phrases"
":" 1*19DIGIT CRLF
voiceprint-exists = "Voiceprint-Exists" ":"
BOOLEAN CRLF
ver-buffer-utterance = "Ver-Buffer-Utterance" ":"
BOOLEAN CRLF
input-waveform-uri = "Input-Waveform-URI" ":" uri CRLF
abort-verification = "Abort-Verification " ":"
BOOLEAN CRLFNext up is a bestie-level tip about the new SDP session-level attribute 🦋 The original says to check out Paragraph 5 for the full tea, so go read that when you get a sec 🌸
cmid-attribute = "a=cmid:" identification-tag
identification-tag = token16. XML Schemas ✨
16.1. NLSML Schema Definition 💖
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
targetNamespace="urn:ietf:params:xml:ns:mrcpv2"
xmlns="urn:ietf:params:xml:ns:mrcpv2"
elementFormDefault="qualified"
attributeFormDefault="unqualified" >
<xs:annotation>
<xs:documentation> Natural Language Semantic Markup Schema
</xs:documentation>
</xs:annotation>
<xs:include schemaLocation="enrollment-schema.rng"/>
<xs:include schemaLocation="verification-schema.rng"/>
<xs:element name="result">
<xs:complexType>
<xs:sequence>
<xs:element name="interpretation" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element name="instance">
<xs:complexType mixed="true">
<xs:sequence minOccurs="0">
<xs:any namespace="##other" processContents="lax"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="input" minOccurs="0">
<xs:complexType mixed="true">
<xs:choice>
<xs:element name="noinput" minOccurs="0"/>
<xs:element name="nomatch" minOccurs="0"/>
<xs:element name="input" minOccurs="0"/>
</xs:choice>
<xs:attribute name="mode"
type="xs:string"
default="speech"/>
<xs:attribute name="confidence"
type="confidenceinfo"
default="1.0"/>
<xs:attribute name="timestamp-start"
type="xs:string"/>
<xs:attribute name="timestamp-end"
type="xs:string"/>
</xs:complexType>
</xs:element>
</xs:sequence>
<xs:attribute name="confidence" type="confidenceinfo"
default="1.0"/>
<xs:attribute name="grammar" type="xs:anyURI"
use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="enrollment-result"
type="enrollment-contents"/>
<xs:element name="verification-result"
type="verification-contents"/>
</xs:sequence>
<xs:attribute name="grammar" type="xs:anyURI"
use="optional"/>
</xs:complexType>
</xs:element>
<xs:simpleType name="confidenceinfo">
<xs:restriction base="xs:float">
<xs:minInclusive value="0.0"/>
<xs:maxInclusive value="1.0"/>
</xs:restriction>
</xs:simpleType>
</xs:schema>16.2. Enrollment Results Schema Definition 🌈
<?xml version="1.0" encoding="UTF-8"?>
<!-- MRCP Enrollment Schema
(See http://www.oasis-open.org/committees/relax-ng/spec.html)
-->
<grammar datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes"
ns="urn:ietf:params:xml:ns:mrcpv2"
xmlns="http://relaxng.org/ns/structure/1.0">
<start>
<element name="enrollment-result">
<ref name="enrollment-content"/>
</element>
</start>
<define name="enrollment-content">
<interleave>
<element name="num-clashes">
<data type="nonNegativeInteger"/>
</element>
<element name="num-good-repetitions">
<data type="nonNegativeInteger"/>
</element>
<element name="num-repetitions-still-needed">
<data type="nonNegativeInteger"/>
</element>
<element name="consistency-status">
<choice>
<value>consistent</value>
<value>inconsistent</value>
<value>undecided</value>
</choice>
</element>
<optional>
<element name="clash-phrase-ids">
<oneOrMore>
<element name="item">
<data type="token"/>
</element>
</oneOrMore>
</element>
</optional>
<optional>
<element name="transcriptions">
<oneOrMore>
<element name="item">
<text/>
</element>
</oneOrMore>
</element>
</optional>
<optional>
<element name="confusable-phrases">
<oneOrMore>
<element name="item">
<text/>
</element>
</oneOrMore>
</element>
</optional>
</interleave>
</define>
</grammar>16.3. Verification Results Schema Definition 🎀
okay bestie, this is the RELAX NG schema for verification results 💅 it's the grammar that says what a verification-result looks like, so every element and value has to match exactly, no cap ✨
<?xml version="1.0" encoding="UTF-8"?>
<!-- MRCP Verification Results Schema
(See http://www.oasis-open.org/committees/relax-ng/spec.html)
-->
<grammar datatypeLibrary="http://www.w3.org/2001/XMLSchema-datatypes"
ns="urn:ietf:params:xml:ns:mrcpv2"
xmlns="http://relaxng.org/ns/structure/1.0">
<start>
<element name="verification-result">
<ref name="verification-contents"/>
</element>
</start>
<define name="verification-contents">
<element name="voiceprint">
<ref name="firstVoiceprintContent"/>
</element>
<zeroOrMore>
<element name="voiceprint">
<ref name="restVoiceprintContent"/>
</element>
</zeroOrMore>
</define>
<define name="firstVoiceprintContent">
<attribute name="id">
<data type="string"/>
</attribute>
<interleave>
<optional>
<element name="adapted">
<data type="boolean"/>
</element>
</optional>
<optional>
<element name="needmoredata">
<ref name="needmoredataContent"/>
</element>
</optional>
<optional>
<element name="incremental">
<ref name="firstCommonContent"/>
</element>
</optional>
<element name="cumulative">
<ref name="firstCommonContent"/>
</element>
</interleave>
</define>
<define name="restVoiceprintContent">
<attribute name="id">
<data type="string"/>
</attribute>
<element name="cumulative">
<ref name="restCommonContent"/>
</element>
</define>
<define name="firstCommonContent">
<interleave>
<element name="decision">
<ref name="decisionContent"/>
</element>
<optional>
<element name="utterance-length">
<ref name="utterance-lengthContent"/>
</element>
</optional>
<optional>
<element name="device">
<ref name="deviceContent"/>
</element>
</optional>
<optional>
<element name="gender">
<ref name="genderContent"/>
</element>
</optional>
<zeroOrMore>
<element name="verification-score">
<ref name="verification-scoreContent"/>
</element>
</zeroOrMore>
</interleave>
</define>
<define name="restCommonContent">
<interleave>
<optional>
<element name="decision">
<ref name="decisionContent"/>
</element>
</optional>
<optional>
<element name="device">
<ref name="deviceContent"/>
</element>
</optional>
<optional>
<element name="gender">
<ref name="genderContent"/>
</element>
</optional>
<zeroOrMore>
<element name="verification-score">
<ref name="verification-scoreContent"/>
</element>
</zeroOrMore>
</interleave>
</define>
<define name="decisionContent">
<choice>
<value>accepted</value>
<value>rejected</value>
<value>undecided</value>
</choice>
</define>
<define name="needmoredataContent">
<data type="boolean"/>
</define>
<define name="utterance-lengthContent">
<data type="nonNegativeInteger"/>
</define>
<define name="deviceContent">
<choice>
<value>cellular-phone</value>
<value>electret-phone</value>
<value>carbon-button-phone</value>
<value>unknown</value>
</choice>
</define>
<define name="genderContent">
<choice>
<value>male</value>
<value>female</value>
<value>unknown</value>
</choice>
</define>
<define name="verification-scoreContent">
<data type="float">
<param name="minInclusive">-1</param>
<param name="maxInclusive">1</param>
</data>
</define>
</grammar>17. References 🦋
17.1. Normative References 🌸
ok so here's the normative reference list, every single entry, lined up so you can find them easily 🫶
[ISO.8859-1.1987]International Organization for Standardization, "Information technology - 8-bit single byte coded graphic - character sets - Part 1: Latin alphabet No. 1, JTC1/ SC2", ISO Standard 8859-1, 1987.[RFC0793]Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.[RFC1035]Mockapetris, P., "Domain names - implementation and specification", STD 13, RFC 1035, November 1987.[RFC2119]Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.[RFC2326]Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time Streaming Protocol (RTSP)", RFC 2326, April 1998.[RFC2392]Levinson, E., "Content-ID and Message-ID Uniform Resource Locators", RFC 2392, August 1998.[RFC2483]Mealling, M. and R. Daniel, "URI Resolution Services Necessary for URN Resolution", RFC 2483, January 1999.[RFC2616]Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.[RFC3023]Murata, M., St. Laurent, S., and D. Kohn, "XML Media Types", RFC 3023, January 2001.[RFC3261]Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002.[RFC3264]Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, June 2002.[RFC3550]Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.[RFC3629]Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003.[RFC3688]Mealling, M., "The IETF XML Registry", BCP 81, RFC 3688, January 2004.[RFC3711]Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004.[RFC3986]Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", STD 66, RFC 3986, January 2005.[RFC4145]Yon, D. and G. Camarillo, "TCP-Based Media Transport in the Session Description Protocol (SDP)", RFC 4145, September 2005.[RFC4288]Freed, N. and J. Klensin, "Media Type Specifications and Registration Procedures", BCP 13, RFC 4288, December 2005.[RFC4566]Handley, M., Jacobson, V., and C. Perkins, "SDP: Session Description Protocol", RFC 4566, July 2006.[RFC4568]Andreasen, F., Baugher, M., and D. Wing, "Session Description Protocol (SDP) Security Descriptions for Media Streams", RFC 4568, July 2006.[RFC4572]Lennox, J., "Connection-Oriented Media Transport over the Transport Layer Security (TLS) Protocol in the Session Description Protocol (SDP)", RFC 4572, July 2006.[RFC5226]Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008.[RFC5234]Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, January 2008.[RFC5246]Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.2", RFC 5246, August 2008.[RFC5322]Resnick, P., Ed., "Internet Message Format", RFC 5322, October 2008.[RFC5646]Phillips, A. and M. Davis, "Tags for Identifying Languages", BCP 47, RFC 5646, September 2009.[RFC5888]Camarillo, G. and H. Schulzrinne, "The Session Description Protocol (SDP) Grouping Framework", RFC 5888, June 2010.[RFC5905]Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithms Specification", RFC 5905, June 2010.[RFC5922]Gurbani, V., Lawrence, S., and A. Jeffrey, "Domain Certificates in the Session Initiation Protocol (SIP)", RFC 5922, June 2010.[RFC6265]Barth, A., "HTTP State Management Mechanism", RFC 6265, April 2011.[W3C.REC-semantic-interpretation-20070405]Tichelen, L. and D. Burke, "Semantic Interpretation for Speech Recognition (SISR) Version 1.0", World Wide Web Consortium Recommendation REC-semantic-interpretation-20070405, April 2007,<http://www.w3.org/TR/2007/REC-semantic-interpretation-20070405>.[W3C.REC-speech-grammar-20040316]McGlashan, S. and A. Hunt, "Speech Recognition Grammar Specification Version 1.0", World Wide Web Consortium Recommendation REC-speech-grammar-20040316, March 2004,<http://www.w3.org/TR/2004/REC-speech-grammar-20040316>.[W3C.REC-speech-synthesis-20040907]Walker, M., Burnett, D., and A. Hunt, "Speech Synthesis Markup Language (SSML) Version 1.0", World Wide Web Consortium Recommendation REC-speech-synthesis-20040907, September 2004,<http://www.w3.org/TR/2004/REC-speech-synthesis-20040907>.[W3C.REC-xml-names11-20040204]Layman, A., Bray, T., Hollander, D., and R. Tobin, "Namespaces in XML 1.1", World Wide Web Consortium First Edition REC-xml-names11-20040204, February 2004,<http://www.w3.org/TR/2004/REC-xml-names11-20040204>.
17.2. Informative References ✨
okay so these are the informative refs 💅 they're here to help you understand the bigger picture, not strictly required reading, but they're still good to know about 🧚 here's the full list:
[ISO.8601.1988]International Organization for Standardization, "Data elements and interchange formats - Information interchange - Representation of dates and times", ISO Standard 8601, June 1988.[Q.23]International Telecommunications Union, "Technical Features of Push-Button Telephone Sets", ITU-T Q.23, 1993.[RFC2046]Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046, November 1996.[RFC2818]Rescorla, E., "HTTP Over TLS", RFC 2818, May 2000.[RFC4217]Ford-Hutchinson, P., "Securing FTP with TLS", RFC 4217, October 2005.[RFC4267]Froumentin, M., "The W3C Speech Interface Framework Media Types: application/voicexml+xml, application/ssml+xml, application/srgs, application/srgs+xml, application/ccxml+xml, and application/pls+xml", RFC 4267, November 2005.[RFC4301]Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005.[RFC4313]Oran, D., "Requirements for Distributed Control of Automatic Speech Recognition (ASR), Speaker Identification/Speaker Verification (SI/SV), and Text-to-Speech (TTS) Resources", RFC 4313, December 2005.[RFC4395]Hansen, T., Hardie, T., and L. Masinter, "Guidelines and Registration Procedures for New URI Schemes", BCP 35, RFC 4395, February 2006.[RFC4463]Shanmugham, S., Monaco, P., and B. Eberman, "A Media Resource Control Protocol (MRCP) Developed by Cisco, Nuance, and Speechworks", RFC 4463, April 2006.[RFC4467]Crispin, M., "Internet Message Access Protocol (IMAP) - URLAUTH Extension", RFC 4467, May 2006.[RFC4733]Schulzrinne, H. and T. Taylor, "RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals", RFC 4733, December 2006.[RFC4960]Stewart, R., "Stream Control Transmission Protocol", RFC 4960, September 2007.[RFC6454]Barth, A., "The Web Origin Concept", RFC 6454, December 2011.[W3C.REC-emma-20090210]Johnston, M., Baggia, P., Burnett, D., Carter, J., Dahl, D., McCobb, G., and D. Raggett, "EMMA: Extensible MultiModal Annotation markup language", World Wide Web Consortium Recommendation REC-emma-20090210, February 2009,<http://www.w3.org/TR/2009/REC-emma-20090210>.[W3C.REC-pronunciation-lexicon-20081014]Baggia, P., Bagshaw, P., Burnett, D., Carter, J., and F. Scahill, "Pronunciation Lexicon Specification (PLS)", World Wide Web Consortium Recommendation REC-pronunciation-lexicon-20081014, October 2008,<http://www.w3.org/TR/2008/REC-pronunciation-lexicon-20081014>.[W3C.REC-voicexml20-20040316]Danielsen, P., Porter, B., Hunt, A., Rehor, K., Lucas, B., Burnett, D., Ferrans, J., Tryphonas, S., McGlashan, S., and J. Carter, "Voice Extensible Markup Language (VoiceXML) Version 2.0", World Wide Web Consortium Recommendation REC-voicexml20-20040316, March 2004,<http://www.w3.org/TR/2004/REC-voicexml20-20040316>.[refs.javaSpeechGrammarFormat]Sun Microsystems, "Java Speech Grammar Format Version 1.0", October 1998.
Appendix A. Contributors 🌸
these are the amazing people who helped build this spec, so let's give them the flowers they deserve 💐
Pierre Forgues 🫶
Pierre Forgues
Nuance Communications Ltd.
1500 University Street
Suite 935
Montreal, Quebec
Canada H3A 3S7
EMail: forgues@nuance.comCharles Galles 🦋
Charles Galles
Intervoice, Inc.
17811 Waterview Parkway
Dallas, Texas 75252
USA
EMail: charles.galles@intervoice.comKlaus Reifenrath 💜
Klaus Reifenrath
Scansoft, Inc
Guldensporenpark 32
Building D
9820 Merelbeke
Belgium
EMail: klaus.reifenrath@scansoft.comAppendix B. Acknowledgements 🎀
the whole squad who made this possible, obsessed with all of you 😭
- Andre Gillet (Nuance Communications)
- Andrew Hunt (ScanSoft)
- Andrew Wahbe (Genesys)
- Aaron Kneiss (ScanSoft)
- Brian Eberman (ScanSoft)
- Corey Stohs (Cisco Systems, Inc.)
- Dave Burke (VoxPilot)
- Jeff Kusnitz (IBM Corp)
- Ganesh N. Ramaswamy (IBM Corp)
- Klaus Reifenrath (ScanSoft)
- Kristian Finlator (ScanSoft)
- Magnus Westerlund (Ericsson)
- Martin Dragomirecky (Cisco Systems, Inc.)
- Paolo Baggia (Loquendo)
- Peter Monaco (Nuance Communications)
- Pierre Forgues (Nuance Communications)
- Ran Zilca (IBM Corp)
- Suresh Kaliannan (Cisco Systems, Inc.)
- Skip Cave (Intervoice, Inc.)
- Thomas Gal (LumenVox)
The chairs of the SPEECHSC work group are Eric Burger (Georgetown University) and Dave Oran (Cisco Systems, Inc.).
Many thanks go in particular to Robert Sparks, Alex Agranovsky, and Henry Phan, who were there at the end to dot all the i's and cross all the t's. 💖
Authors' Addresses 🦋
and finally, the two people who wrote it, with their contact info 💅
Daniel C. Burnett
Daniel C. Burnett
Voxeo
189 South Orange Avenue #1000
Orlando, FL 32801
USA
EMail: dburnett@voxeo.comSaravanan Shanmugham
Saravanan Shanmugham
Cisco Systems, Inc.
170 W. Tasman Dr.
San Jose, CA 95134
USA
EMail: sarvi@cisco.comThat's the whole RFC, bestie! 🎉 Thanks for reading all 45 parts with me 🌈👑
📦 where this page comes from
- ietf: rfc6787 2012-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms