Transcription API

Realtime transcription

WSS wss://global.rt.speechmatics.com/v2/

Transcribe audio as you stream it, over one WebSocket connection. Open the connection and send StartRecognition with your audio format and transcription settings; then stream audio as binary messages, and transcripts arrive as they are ready. Every other message is a JSON object that names itself in its message field.

New to the Realtime API? Start with the quickstart.

Handshake

GET wss://global.rt.speechmatics.com/v2/

Routes each connection to the nearest region, for the lowest latency. To keep data in one region instead, connect to that region's endpoint, such as wss://eu.rt.speechmatics.com/v2. See Supported endpoints.

Authentication

Any one of:

apiKey Authorization: Bearer <token>
Your API key, for server-side connections.
temporaryKey ?jwt=<key>
A temporary key, for connections from a browser, which cannot set the Authorization header — and should not hold your long-lived API key. For example, wss://global.rt.speechmatics.com/v2?jwt=<temporary-key>.

Responses

Switching Protocols. Connected: send StartRecognition to begin.

A successful handshake request:

GET /v2/ HTTP/1.1
Host: global.rt.speechmatics.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: ujRTbIaQsXO/0uCbjjkSZQ==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Authorization: Bearer wmz9fkLJM6U5NdyaG3HLHybGZj65PXp
User-Agent: Python/3.8 websockets/8.1

And its response:

HTTP/1.1 101 Switching Protocols
Server: nginx/1.17.8
Date: Wed, 06 Jan 2021 11:01:05 GMT
Connection: upgrade
Upgrade: WebSocket
Sec-WebSocket-Accept: 87kiC/LI5WgXG52nSylnfXdz260=

Method Not Allowed. The request method is not GET.

Send client → server

StartRecognition

Initiates a new recognition session.

message "StartRecognition"
audio_format AudioFormatone of
AudioFormatRaw Raw optional: Raw audio samples, described by the following additional mandatory fields:
type "raw"
encoding RawAudioEncodingEnum"pcm_f32le""pcm_s16le""mulaw"
sample_rate integer: The sample rate of the audio in Hz.
AudioFormatFile File optional: Choose this option to send audio encoded in a recognized format. The AddAudio messages have to provide all the file contents, including any headers. The file is usually not accepted all at once, but segmented into reasonably sized messages.

Note: Only the following formats are supported: wav, mp3, aac, ogg, mpeg, amr, m4a, mp4, flac
type "file"
transcription_config TranscriptionConfig: Contains configuration for this recognition session.
language string: Language model to process the audio input, normally specified as an ISO language code. The value must be consistent with the language code used in the API endpoint URL.
language_hints string[] optional: The languages expected in the audio, specified as ISO Language Codes. Biases language detection toward these languages without restricting it to them. Only used by the melia-1 model, which requires language to be set to multi. See Language hints.
domain string optional: Request a specialized model based on 'language' but optimized for a particular field, e.g. finance or medical.
output_locale OutputLocale≥ 1 characters optional: Configure locale for outputted transcription. See output formatting.
additional_vocab VocabListone of≤ 20000 items optional: Configure custom dictionary. Default is an empty list. You should be aware that there is a performance penalty (latency degradation and memory increase) from using additional_vocab, especially if you use a large word list. When initializing a session that uses additional_vocab in the config, you should expect a delay of up to 15 seconds (depending on the size of the list).
String string≥ 1 characters optional: Pass a non-empty string to add a single word to the dictionary.
AdditionalVocabObject Object optional: Pass an object to add a single word to the dictionary, with an array of words which it sounds like.
content string≥ 1 characters
sounds_like string[]≥ 1 items optional
diarization DiarizationConfigdefault: "none""none""speaker""channel""channel_and_speaker" optional: Set to speaker to apply Speaker Diarization to the audio.
max_delay numberdefault: 4≥ 0.7, ≤ 4 optional: This is the delay in seconds between the end of a spoken word and returning the Final transcript results. See Latency for more details
max_delay_mode MaxDelayModeConfigdefault: "flexible""flexible""fixed" optional: This allows some additional time for Smart Formatting.
speaker_diarization_config SpeakerDiarizationConfig optional
max_speakers integer≥ 2 optional: Configure the maximum number of speakers to detect. See Max Speakers.
prefer_current_speaker booleandefault: false optional: When set to true, reduces the likelihood of incorrectly switching between similar sounding speakers. See Prefer Current Speaker.
speaker_sensitivity number<float>≥ 0, ≤ 1 optional
get_speakers boolean optional: If true, speaker identifiers will be returned at the end of transcript.
speakers SpeakersInputItem[] optional: Use this option to provide speaker labels linked to their speaker identifiers. When passed, the transcription system will tag spoken words in the transcript with the provided speaker labels whenever any of the specified speakers is detected in the audio. A maximum of 50 speakers identifiers across all speakers can be provided.
label unknown`not` has no display form: Speaker label, which must not match the format used internally (e.g. S1, S2, etc)
speaker_identifiers Array<string<bytes>>≥ 1 itemsunique items
audio_filtering_config AudioFilteringConfig optional: Puts a lower limit on the volume of processed audio by using the volume_threshold setting. See Audio Filtering.
volume_threshold number<float>≥ 0, ≤ 100 optional
transcript_filtering_config TranscriptFilteringConfig optional
remove_disfluencies boolean optional: When set to true, removes disfluencies from the transcript. See Removing disfluencies
replacements WordReplacementItem[] optional: A list of replacement rules to apply to the transcript. Each rule consists of a pattern to match and a replacement string. See Word replacement
from string
to string
enable_partials booleandefault: false optional: Whether or not to send Partials (i.e. AddPartialTranslation messages) as well as Finals (i.e. AddTranslation messages) See Partial transcripts.
enable_entities booleandefault: true optional
operating_point OperatingPointdeprecated"standard""enhanced""melia-1" optionaldeprecated: Deprecated: Kept for backward compatibility only. Use model instead going forward.
model Modeldefault: "standard""standard""enhanced""melia-1" optional: Which model you wish to use. See Models for more details.
punctuation_overrides PunctuationOverrides optional: Options for controlling punctuation in the output transcripts. See Punctuation Settings
permitted_marks string[] optional: The punctuation marks which the client is prepared to accept in transcription output, or the special value 'all' (the default). Unsupported marks are ignored. This value is used to guide the transcription process.
sensitivity number<float>≥ 0, ≤ 1 optional: Ranges between zero and one. Higher values will produce more punctuation. The default is 0.5.
conversation_config ConversationConfig optional: This mode will detect when a speaker has stopped talking. The end_of_utterance_silence_trigger is the time in seconds after which the server will assume that the speaker has finished speaking, and will emit an EndOfUtterance message. A value of 0 disables the feature. Supported by Enhanced and Standard. Melia 1 rejects end_of_utterance_silence_trigger in conversation_config; use client-side turn detection and send ForceEndOfUtterance instead.
end_of_utterance_silence_trigger number<float>default: 0≥ 0, ≤ 2 optional
channel_diarization_labels string[] optional
translation_config TranslationConfig optional: Specifies various configuration values for translation. All fields except target_languages are optional, using default values when omitted.
target_languages string[]: List of languages to translate to from the source transcription language. Specified as an ISO Language Code.
enable_partials booleandefault: false optional: Whether or not to send Partials (i.e. AddPartialTranslation messages) as well as Finals (i.e. AddTranslation messages).
audio_events_config AudioEventsConfig optional: Contains configuration for Audio Events
types AudioEventType[] optional: List of Audio Event types to enable.

AddAudio

A binary chunk of audio. The server confirms receipt by sending an AudioAdded message.

application/octet-stream

string<binary>: Raw audio bytes, in the encoding and sample rate given by audio_format in StartRecognition.

AddChannelAudio

Audio belonging to a specific channel.

message "AddChannelAudio"
channel string: The channel identifier to which the audio belongs.
data string: The audio data in base64 format.

EndOfStream

Declares that the client has no more audio to send.

message "EndOfStream"
last_seq_no integer

EndOfChannel

Declares that the channel has no more audio to send.

message "EndOfChannel"
channel string: The channel identifier to which the audio belongs.
last_seq_no integer

ForceEndOfUtterance

Requests a finalized transcript followed by an EndOfUtterance message. Supported by Enhanced, Standard, and Melia 1. The client decides when to send this message using its own turn detection.

message "ForceEndOfUtterance"
channel string optional: The channel to request finalized transcript from. This field is only seen in multichannel.
timestamp number<float>≥ 0 optional: Timestamp of the audio data that corresponds to the force end of utterance request. It's the number of seconds since the beginning of the audio.

SetRecognitionConfig

Allows the client to re-configure the recognition session.

The language field can be included in SetRecognitionConfig, but its value must match the language specified in the initial StartRecognition message. Attempting to change the language or any other transcription configuration parameter not listed below will result in an error.

message "SetRecognitionConfig"
transcription_config MidSessionTranscriptionConfig: Contains configuration for this recognition session.
language string optional: Language model to process the audio input, normally specified as an ISO language code. The value must be consistent with the language code used in the API endpoint URL.
max_delay numberdefault: 4≥ 0.7, ≤ 4 optional: This is the delay in seconds between the end of a spoken word and returning the Final transcript results. See Latency for more details
max_delay_mode MaxDelayModeConfigdefault: "flexible""flexible""fixed" optional: This allows some additional time for Smart Formatting.
audio_filtering_config AudioFilteringConfig optional: Puts a lower limit on the volume of processed audio by using the volume_threshold setting. See Audio Filtering.
volume_threshold number<float>≥ 0, ≤ 100 optional
enable_partials booleandefault: false optional: Whether or not to send Partials (i.e. AddPartialTranslation messages) as well as Finals (i.e. AddTranslation messages) See Partial transcripts.
conversation_config ConversationConfig optional: This mode will detect when a speaker has stopped talking. The end_of_utterance_silence_trigger is the time in seconds after which the server will assume that the speaker has finished speaking, and will emit an EndOfUtterance message. A value of 0 disables the feature. Supported by Enhanced and Standard. Melia 1 rejects end_of_utterance_silence_trigger in conversation_config; use client-side turn detection and send ForceEndOfUtterance instead.
end_of_utterance_silence_trigger number<float>default: 0≥ 0, ≤ 2 optional

GetSpeakers

Requests any detected speaker identifiers to be returned.

message "GetSpeakers"
final boolean optional: Optional. This flag controls when speaker identifiers are returned. Defaults to false if omitted. When false, multiple GetSpeakers requests can be sent during transcription, each returning the speaker identifiers generated so far. To reduce the chance of empty results, send requests after at least one TranscriptAdded message is received to make sure that the server has processed some audio. When true, speaker identifiers are returned only once at the end of the transcription, regardless of how many final: true requests are sent. Even with final: true requests, you can still send final: false requests to receive intermediate speaker identifier updates.

Receive server → client

RecognitionStarted

Server response to StartRecognition, acknowledging that a recognition session has started.

message "RecognitionStarted"
orchestrator_version string optional
id string optional
language_pack_info LanguagePackInfo optional: Properties of the language pack.
language_description string optional: Full descriptive name of the language, e.g. 'Japanese'.
word_delimiter string: The character to use to separate words.
writing_direction WritingDirectionEnum"left-to-right""right-to-left" optional: The direction that words in the language should be written and read in.
itn boolean optional: Whether or not ITN (inverse text normalization) is available for the language pack.
adapted boolean optional: Whether or not language model adaptation has been applied to the language pack.
channel_ids string[] optional

AudioAdded

Server response to AddAudio, indicating that audio has been added successfully.

When clients send audio faster than real-time, the server may read data slower than it's sent. If binary AddAudio messages exceed the server's internal buffer, the server will process other WebSocket messages until buffer space is available. Clients receive AudioAdded responses only after binary data is read. This can fill TCP buffers, potentially causing WebSocket write failures and connection closure with prejudice. Clients can monitor the WebSocket's bufferedAmount attribute to prevent this.

message "AudioAdded"
seq_no integer

ChannelAudioAdded

Server response to AddChannelAudio, indicating that audio has been added successfully.

message "ChannelAudioAdded"
seq_no integer
channel string

AddPartialTranscript

A partial transcript is a transcript that can be changed in a future AddPartialTranscript as more words are spoken until the AddTranscript Final message is sent for that audio.

Partials will only be sent if transcription_config.enable_partials is set to true in the StartRecognition message.

The message structure is the same as AddTranscript, with a few limitations.

For AddPartialTranscript messages the confidence field for alternatives has no meaning and should not be relied on.

message "AddPartialTranscript"
format string optional: Speechmatics JSON output format version number.
metadata RecognitionMetadata
start_time number<float>
end_time number<float>
transcript string: The entire transcript contained in the segment in text format. Providing the entire transcript here is designed for ease of consumption; we have taken care of all the necessary formatting required to concatenate the transcription results into a block of text. This transcript lacks the detailed information however which is contained in the results field of the message - such as the timings and confidences for each word.
results RecognitionResult[]
type RecognitionResultTypeEnum"word""punctuation""entity"
start_time number<float>
end_time number<float>
attaches_to AttachesToEnum"next""previous""none""both" optional
is_eos boolean optional
alternatives RecognitionAlternative[] optional
content string: A word or punctuation mark.
confidence number<float>: A confidence score assigned to the alternative. Ranges from 0.0 (least confident) to 1.0 (most confident).
language string optional: The language that the alternative word is assumed to be spoken in. For the standard and enhanced models this is always the language requested in the initial StartRecognition message. The melia-1 model detects the language per word, so this value can vary across the transcript.
display RecognitionDisplay optional: Information about how the word/symbol should be displayed.
direction DirectionEnum"ltr""rtl": Either ltr for words that should be displayed left-to-right, or rtl vice versa.
speaker string optional: Label indicating who said that word. Only set if diarization is enabled.
tags RecognitionAlternativeTagsEnum[]"disfluency""profanity" optional: This is a set list of profanities and disfluencies respectively that cannot be altered by the end user. [disfluency] is present in the supported languages, and [profanity] is present in English, Spanish, and Italian
volume number<float>≥ 0, ≤ 100 optional
entity_class string optional: For 'entity' results only, the class the entity has been formatted as. Examples: 'date', 'money', 'number'
spoken_form SpokenFormRecognitionResult[] optional: For 'entity' results only, the spoken_form is the transcript of the individual words directly spoken.
alternatives RecognitionAlternative[]
content string: A word or punctuation mark.
confidence number<float>: A confidence score assigned to the alternative. Ranges from 0.0 (least confident) to 1.0 (most confident).
language string optional: The language that the alternative word is assumed to be spoken in. For the standard and enhanced models this is always the language requested in the initial StartRecognition message. The melia-1 model detects the language per word, so this value can vary across the transcript.
display RecognitionDisplay optional: Information about how the word/symbol should be displayed.
direction DirectionEnum"ltr""rtl": Either ltr for words that should be displayed left-to-right, or rtl vice versa.
speaker string optional: Label indicating who said that word. Only set if diarization is enabled.
tags RecognitionAlternativeTagsEnum[]"disfluency""profanity" optional: This is a set list of profanities and disfluencies respectively that cannot be altered by the end user. [disfluency] is present in the supported languages, and [profanity] is present in English, Spanish, and Italian
end_time number<float>
start_time number<float>
type SpokenFormRecognitionResultTypeEnum"word""punctuation"
written_form WrittenFormRecognitionResult[] optional: For 'entity' results only, the written_form is a standardized form of the spoken words. It contains the formatted entity split into individual words.
alternatives RecognitionAlternative[]
content string: A word or punctuation mark.
confidence number<float>: A confidence score assigned to the alternative. Ranges from 0.0 (least confident) to 1.0 (most confident).
language string optional: The language that the alternative word is assumed to be spoken in. For the standard and enhanced models this is always the language requested in the initial StartRecognition message. The melia-1 model detects the language per word, so this value can vary across the transcript.
display RecognitionDisplay optional: Information about how the word/symbol should be displayed.
direction DirectionEnum"ltr""rtl": Either ltr for words that should be displayed left-to-right, or rtl vice versa.
speaker string optional: Label indicating who said that word. Only set if diarization is enabled.
tags RecognitionAlternativeTagsEnum[]"disfluency""profanity" optional: This is a set list of profanities and disfluencies respectively that cannot be altered by the end user. [disfluency] is present in the supported languages, and [profanity] is present in English, Spanish, and Italian
end_time number<float>
start_time number<float>
type WrittenFormRecognitionResultTypeEnum"word"
channel string optional: The channel identifier to which the audio belongs. This field is only seen in multichannel.
forced boolean optional: Whether this message was triggered by a ForceEndOfUtterance message. This field is only seen on forced messages, where its value is true.

AddTranscript

Contains the final transcript of a part of the audio that the client has sent.

message "AddTranscript"
format string optional: Speechmatics JSON output format version number.
metadata RecognitionMetadata
start_time number<float>
end_time number<float>
transcript string: The entire transcript contained in the segment in text format. Providing the entire transcript here is designed for ease of consumption; we have taken care of all the necessary formatting required to concatenate the transcription results into a block of text. This transcript lacks the detailed information however which is contained in the results field of the message - such as the timings and confidences for each word.
results RecognitionResult[]
type RecognitionResultTypeEnum"word""punctuation""entity"
start_time number<float>
end_time number<float>
attaches_to AttachesToEnum"next""previous""none""both" optional
is_eos boolean optional
alternatives RecognitionAlternative[] optional
content string: A word or punctuation mark.
confidence number<float>: A confidence score assigned to the alternative. Ranges from 0.0 (least confident) to 1.0 (most confident).
language string optional: The language that the alternative word is assumed to be spoken in. For the standard and enhanced models this is always the language requested in the initial StartRecognition message. The melia-1 model detects the language per word, so this value can vary across the transcript.
display RecognitionDisplay optional: Information about how the word/symbol should be displayed.
direction DirectionEnum"ltr""rtl": Either ltr for words that should be displayed left-to-right, or rtl vice versa.
speaker string optional: Label indicating who said that word. Only set if diarization is enabled.
tags RecognitionAlternativeTagsEnum[]"disfluency""profanity" optional: This is a set list of profanities and disfluencies respectively that cannot be altered by the end user. [disfluency] is present in the supported languages, and [profanity] is present in English, Spanish, and Italian
volume number<float>≥ 0, ≤ 100 optional
entity_class string optional: For 'entity' results only, the class the entity has been formatted as. Examples: 'date', 'money', 'number'
spoken_form SpokenFormRecognitionResult[] optional: For 'entity' results only, the spoken_form is the transcript of the individual words directly spoken.
alternatives RecognitionAlternative[]
content string: A word or punctuation mark.
confidence number<float>: A confidence score assigned to the alternative. Ranges from 0.0 (least confident) to 1.0 (most confident).
language string optional: The language that the alternative word is assumed to be spoken in. For the standard and enhanced models this is always the language requested in the initial StartRecognition message. The melia-1 model detects the language per word, so this value can vary across the transcript.
display RecognitionDisplay optional: Information about how the word/symbol should be displayed.
direction DirectionEnum"ltr""rtl": Either ltr for words that should be displayed left-to-right, or rtl vice versa.
speaker string optional: Label indicating who said that word. Only set if diarization is enabled.
tags RecognitionAlternativeTagsEnum[]"disfluency""profanity" optional: This is a set list of profanities and disfluencies respectively that cannot be altered by the end user. [disfluency] is present in the supported languages, and [profanity] is present in English, Spanish, and Italian
end_time number<float>
start_time number<float>
type SpokenFormRecognitionResultTypeEnum"word""punctuation"
written_form WrittenFormRecognitionResult[] optional: For 'entity' results only, the written_form is a standardized form of the spoken words. It contains the formatted entity split into individual words.
alternatives RecognitionAlternative[]
content string: A word or punctuation mark.
confidence number<float>: A confidence score assigned to the alternative. Ranges from 0.0 (least confident) to 1.0 (most confident).
language string optional: The language that the alternative word is assumed to be spoken in. For the standard and enhanced models this is always the language requested in the initial StartRecognition message. The melia-1 model detects the language per word, so this value can vary across the transcript.
display RecognitionDisplay optional: Information about how the word/symbol should be displayed.
direction DirectionEnum"ltr""rtl": Either ltr for words that should be displayed left-to-right, or rtl vice versa.
speaker string optional: Label indicating who said that word. Only set if diarization is enabled.
tags RecognitionAlternativeTagsEnum[]"disfluency""profanity" optional: This is a set list of profanities and disfluencies respectively that cannot be altered by the end user. [disfluency] is present in the supported languages, and [profanity] is present in English, Spanish, and Italian
end_time number<float>
start_time number<float>
type WrittenFormRecognitionResultTypeEnum"word"
channel string optional: The channel identifier to which the audio belongs. This field is only seen in multichannel.
forced boolean optional: Whether this message was triggered by a ForceEndOfUtterance message. This field is only seen on forced messages, where its value is true.

AddPartialTranslation

Contains a work-in-progress translation of a part of the audio that the client has sent.

message "AddPartialTranslation"
format string optional: Speechmatics JSON output format version number.
language string: Language translation relates to given as an ISO language code.
results TranslatedSentence[]
content string
start_time number<float>: The start time (in seconds) of the original transcribed audio segment
end_time number<float>: The end time (in seconds) of the original transcribed audio segment
speaker string optional: The speaker that uttered the speech if speaker diarization is enabled

AddTranslation

Contains the final translation of a part of the audio that the client has sent.

message "AddTranslation"
format string optional: Speechmatics JSON output format version number.
language string: Language translation relates to given as an ISO language code.
results TranslatedSentence[]
content string
start_time number<float>: The start time (in seconds) of the original transcribed audio segment
end_time number<float>: The end time (in seconds) of the original transcribed audio segment
speaker string optional: The speaker that uttered the speech if speaker diarization is enabled

EndOfTranscript

Server response to EndOfStream, after the server has finished sending all AddTranscript messages.

message "EndOfTranscript"

AudioEventStarted

Start of an audio event detected.

message "AudioEventStarted"
event AudioEventStartData
type AudioEventType: The type of audio event that has started or ended. See our list of supported Audio Event types.
start_time number<float>: The time (in seconds) of the audio corresponding to the beginning of the audio event.
confidence number<float>≥ 0, ≤ 1: A confidence score assigned to the audio event. Ranges from 0.0 (least confident) to 1.0 (most confident).
channel string optional: The channel identifier to which the audio belongs. This field is only seen in multichannel.

AudioEventEnded

End of an audio event detected.

message "AudioEventEnded"
event AudioEventEndData
type AudioEventType: The type of audio event that has started or ended. See our list of supported Audio Event types.
end_time number<float>
channel string optional: The channel identifier to which the audio belongs. This field is only seen in multichannel.

EndOfUtterance

Indicates the end of an utterance, triggered by a configurable period of non-speech or a client-sent ForceEndOfUtterance message. Enhanced and Standard support automatic silence-based detection, configured by the end_of_utterance_silence_trigger parameter in conversation_config (see End of utterance). Melia 1 does not support automatic silence-based detection and rejects this setting. With Melia 1, send ForceEndOfUtterance to receive an EndOfUtterance message.

Like punctuation, an EndOfUtterance has zero duration.

message "EndOfUtterance"
metadata EndOfUtteranceMetadata
start_time number<float> optional: The time (in seconds) that the end of utterance was detected.
end_time number<float> optional: The time (in seconds) that the end of utterance was detected.
channel string optional: The channel identifier to which the EndOfUtterance message belongs. This field is only seen in multichannel.
forced boolean optional: Whether this message was triggered by a ForceEndOfUtterance message. This field is only seen on forced messages, where its value is true.

LanguageInfo

Reports the word delimiter and writing direction for a language detected in the audio.

Sent only by the melia-1 model, which labels each word with the language it was spoken in. The message is emitted once per language code, immediately before that language first appears in an AddPartialTranscript or AddTranscript message. Use it to set word spacing and text direction when rendering the transcript (see LanguageInfo message).

message "LanguageInfo"
language string: The detected language code.
word_delimiter string: The delimiter to place between words of this language when reconstructing the transcript.
writing_direction WritingDirectionEnum"left-to-right""right-to-left": The direction that words in the language should be written and read in.
partial boolean optional: Whether the message was triggered by a Partial transcript. This field is only seen when its value is true.

Info

Additional information sent from the server to the client.

message "Info"
type InfoTypeEnumone of: The kind of information an Info message carries.
"recognition_quality": Informs the client what particular quality-based model is used to handle the recognition. Sent to the client immediately after the WebSocket handshake is completed.
"concurrent_session_usage": Informs the client of their quota for concurrent sessions and how much of it they are using. Sent to the client immediately after the WebSocket handshake is completed.
reason string
code integer optional
seq_no integer optional
quality string optional: Only set when type is recognition_quality. Quality-based model name. It is one of "telephony", "broadcast". The model is selected automatically, for high-quality audio (12kHz+) the broadcast model is used, for lower quality audio the telephony model is used.
usage number optional: Only set when type is concurrent_session_usage. Indicates the current usage (number of active concurrent sessions).
quota number optional: Only set when type is concurrent_session_usage. Indicates the current quota (maximum number of concurrent sessions allowed).
last_updated string optional: Only set when type is concurrent_session_usage. Indicates the timestamp of the most recent usage update, in the format YYYY-MM-DDTHH:MM:SSZ (UTC). This value is updated even when usage exceeds the quota, as it represents the most recent known data. In some cases, it may be empty or outdated due to internal errors preventing successful update.
region string optional: Only set when type is concurrent_session_usage. The region that reported this usage, for example eu.

Warning

Warning messages sent from the server to the client.

message "Warning"
type WarningTypeEnumone of: The kind of problem a Warning message reports.
"duration_limit_exceeded": The maximum allowed duration of a single utterance to process has been exceeded. Any AddAudio messages received that exceed this limit are confirmed with AudioAdded, but are ignored by the transcription engine. Exceeding the limit triggers the same mechanism as receiving an EndOfStream message, so the Server will eventually send an EndOfTranscript message and suspend.
"unsupported_translation_pair": One of the requested translation target languages is unsupported (given the source audio language). The error message specifies the unsupported language pair.
"idle_timeout": Informs that the session is approaching the idle duration limit (no audio data sent within the last hour), with a reason of the form:

Session will timeout in {time_remaining}m due to inactivity, no audio sent within the last {time_elapsed}m

Currently the server will send messages at 15, 10 and 5m prior to timeout, and will send a final error message on timeout, before closing the connection with the code 1008. (see Realtime limits for more information).
"session_timeout": Informs that the session is approaching the max session duration limit (maximum session duration of 48 hours), with a reason of the form:

Session will timeout in {time_remaining}m due to max duration, session has been active for {time_elapsed}m

Currently the server will send messages at 45, 30 and 15m prior to timeout, and will send a final error message on timeout, before closing the connection with the code 1008. (see Realtime limits for more information).
"empty_translation_target_list": No supported translation target languages specified. Translation will not run.
"add_audio_after_eos": Protocol specification doesn't allow adding audio after EndOfStream has been received. Any AddAudio messages after this, will be ignored.
"speaker_id": Informs the client about any speaker ID related issues.
reason string
code integer optional
seq_no integer optional
duration_limit number optional: Only set when type is duration_limit_exceeded. Indicates the limit that was exceeded (in seconds).

Error

Error messages sent from the server to the client. After any error, the transcription is terminated and the connection is closed.

message "Error"
type ErrorTypeEnumone of: The kind of error an Error message reports.

invalid_message, protocol_error and unknown_error can be triggered as a response to any type of messages.
"invalid_message": The message received was not understood.
"invalid_model": Unable to use the model for the recognition. This can happen if the language is not supported at all, or is not available for the user.
"invalid_language": The requested language is not valid or is not supported.
"invalid_config": The config received contains some wrong or unsupported fields, or too many translation target languages were requested.
"invalid_audio_type": Audio type is not supported, is deprecated, or the audio_type is malformed.
"invalid_output_format": Output format is not supported, is deprecated, or the output_format is malformed.
"not_authorised": User was not recognised, or the API key provided is not valid.
"not_allowed": User is not allowed to use this message (is not allowed to perform the action the message would invoke).
"job_error": Unable to do any work on this job, the server might have timed out etc.
"protocol_error": Message received was syntactically correct, but could not be accepted due to protocol limitations. This is usually caused by messages sent in the wrong order.
"quota_exceeded": Maximum number of concurrent connections allowed for the contract has been reached
"timelimit_exceeded": Usage quota for the contract has been reached
"idle_timeout": Idle duration limit was reached (no audio data sent within the last hour), a closing handshake with code 1008 follows this in-band error.
"session_timeout": Max session duration was reached (maximum session duration of 48 hours), a closing handshake with code 1008 follows this in-band error.
"unknown_error": An error that did not fit any of the types above.
reason string
code integer optional
seq_no integer optional

SpeakersResult

Server response to GetSpeakers request returning any detected speaker identifiers.

message "SpeakersResult"
speakers SpeakersResultItem[]
label string≥ 1 characters: Speaker label.
speaker_identifiers Array<string<bytes>>≥ 1 itemsunique items

Connection close

An Error message can be followed by a WebSocket close frame. Its code, and the reason it carries:

Close code	Reason
1003	`protocol_error`
1008	`policy_violation`
1011	`internal_error`
4001	`not_authorised`
4003	`not_allowed`
4004	`invalid_model`
4005	`quota_exceeded`
4006	`timelimit_exceeded`
4013	`job_error`