HTTP API reference
Complete field reference for the nemo-speech serve HTTP API. Every endpoint follows the same conventions:
Auth: if the server was started with an API key, send Authorization: Bearer <key> to /v1 routes (WebSocket clients may use ?api_key=<key>). The playground, health, readiness, and version routes are intentionally unauthenticated.
Errors: non-2xx responses carry {"error": {"message": "...", "type": "invalid_request_error" | "server_error"}}.
Audio uploads: multipart form with the uncompressed RIFF/WAVE file in file. Mono or stereo PCM16 and float32 WAVs at 8-96 kHz are accepted; stereo is downmixed to mono.
Endpoints return 501 when their capability is not included in the build and a server_error when its model is not loaded. GET /v1/models lists the active capabilities.
OpenAI SDK compatibility covers model listing and the documented subsets of /v1/audio/transcriptions and /v1/audio/speech; it does not extend to other OpenAI APIs. Realtime transcription uses the WebSocket contract below, not the OpenAI Realtime API.
Service
Method + pathPurposeGET /bundled playground UIGET /health, GET /readycompact health status ({"status": "ok", ...}) / detailed readiness ({"ready": true, ...})GET /versionbuild versionGET /v1/modelsloaded models and capabilitiesGET /v1/realtime/healthVoiceChat WebSocket readinessPOST /v1/audio/transcriptions
Speech-to-text (OpenAI-compatible multipart subset). Realtime transcription uses the /v1/audio/transcriptions/realtime WebSocket described below.
FieldTypeDefaultDescriptionmodelstringloaded ASR modelaccepted for client compatibility; this server uses its one loaded ASR modelfileWAV uploadrequiredaudio to transcribelanguagestringmodel defaultlanguage code; prompt-conditioned models select their language promptresponse_formatstringjsonjson, verbose_json (adds words/timestamps), text, srt, or vttautomatic_punctuationbooltruepunctuation + capitalizationverbatimboolfalseskip inverse text normalizationprofanity_filterboolfalsemask words from the configured listdiarizationboolfalsetag words with speakers (requires verbose_json and a diarizer model)max_speaker_countintignoreddeprecated compatibility field; Sortformer v2 supports up to four speakersspeech_contextsJSON arraynoneword boosting, [{"phrases": ["..."], "boost": N}] - same shape as gRPC; scoring:
promptstringnoneOpenAI-compat: one boosted phrase at boost 10curl -X POST http://127.0.0.1:8080/v1/audio/transcriptions \ -F "[email protected]" -F "response_format=verbose_json" \ -F 'speech_contexts=[{"phrases": ["Kowalczyk"], "boost": 3.0}]'Response (json): {"text": "..."}. verbose_json adds task, language, duration, and words[] (word, start, end, confidence, and speaker when diarization is on). SRT and WebVTT responses use the same readable, punctuation-aware cue grouping as the file CLI.
WebSocket /v1/audio/transcriptions/realtime
Live PCM16 transcription using a project-specific event protocol. The server sends session.created on connect. Optionally send one session.update JSON event (rejected once audio has started); then binary little-endian PCM16 frames (or base64 chunks in input_audio_buffer.append); finish with input_audio_buffer.commit. input_audio_buffer.clear or response.cancel discards buffered audio (input_audio_buffer.cleared).
session.update -> {"type": "session.update", "session": {...}} fields:
FieldTypeDefaultDescriptionsample_rateintmodel input rate (16000 for the shipped models)PCM sample rate, 8000-96000languagestringmodel defaultlanguage codeautomatic_punctuationbooltruepunctuation + capitalizationverbatimboolfalseskip inverse text normalizationprofanity_filterboolfalsemask words from the configured listword_timestampsboolfalseword timings on final eventsspeaker_diarizationboolfalsetag words with speakers; requires a loaded diarizermax_speaker_countintignoreddeprecated compatibility field; Sortformer v2 supports up to four speakersendpointing_msnumberserver defaultend-of-utterance silence thresholdspeech_contextsarraynoneword boosting, as in /v1/audio/transcriptionspromptstringnoneOpenAI-compat: one boosted phrase at boost 10Server events: session.created, session.updated, conversation.item.input_audio_transcription.delta (partials), .completed (finals, with words when requested), input_audio_buffer.committed, input_audio_buffer.cleared, and error.
For backward compatibility, /v1/realtime serves this transcription protocol when VoiceChat is not loaded. Use the explicit audio-namespaced path for new transcription clients.
VoiceChat WebSocket /v1/realtime and /realtime
When a VoiceChat model is loaded, both paths expose the full-duplex Riva VoiceChat session protocol used by the reference client and compatible realtime integrations. The server sends session.created on connection. Send session.update before audio begins; session configuration is immutable after the first audio frame.
session.audio.input.format accepts "pcm16" or an object such as {"type":"audio/pcm","rate":24000}. Input must be mono little-endian PCM16 at 16-48 kHz. session.audio.output.format must select PCM16 at 24 kHz. Audio may be sent as binary frames or as base64 in input_audio_buffer.append. Output is base64 PCM16 in 80 ms response.output_audio.delta packets.
session fieldTypeDefaultDescriptionaudio.input.formatstring or objectPCM16 at 24 kHzclient input encoding and rateaudio.output.formatstring or objectPCM16 at 24 kHzoutput encoding; only 24 kHz PCM16 is supportedinstructionsstringserver VoiceChat promptsystem instructions, including conversational policytoolsarray or JSON string[]OpenAI-format tool definitions; ack_messages is supportedResponse events include response.created, response.output_audio.delta, response.output_audio.done, response.output_audio_transcript.delta, response.output_audio_transcript.done, and response.done. User turns emit input_audio_buffer.speech_started, transcription delta/completed events, and input_audio_buffer.speech_stopped. Every event includes an event_id.
Tool requests arrive as response.function_call_arguments.done, with call_id, name, and JSON-encoded arguments. Return the result on the same socket:
{ "type": "conversation.item.create", "item": { "type": "function_call_output", "call_id": "call_123", "output": "{\"status\":\"ok\"}" } }Send session.close to finish cleanly. The server drains residual input, emits response completion events, then sends session.end with received, sent, dropped, inference, and audio-duration counters. See the
for the complete event flow.
POST /v1/audio/speech
Text-to-speech (OpenAI-compatible JSON subset).
FieldTypeDefaultDescriptionmodelstringloaded TTS modelaccepted for client compatibility; this server uses its one loaded TTS modelinputstringrequiredtext to synthesizevoicestringmodel defaultlocal voice name, model-qualified voice name, or zero-based speaker indexlanguagestringmodel defaultlanguage codespeednumber1.0only 1.0 is accepted; other values return 400sample_rateintmodel defaultoutput sample rate, from 8000 Hz through the model rate (22050 Hz for the supported NanoCodec model)response_formatstringwavwav or pcmResponse: mono signed PCM16, either in a WAV container or raw little-endian bytes with the matching content type. The complete audio is buffered before the HTTP response; streaming synthesis is not part of this compatibility subset.
Local voice names are case-insensitive and are listed in the voices field of the speech entry returned by GET /v1/models. <model-id>.<voice> is also accepted. default and supported OpenAI voice aliases such as alloy select the server's configured default local speaker; they do not provide the corresponding hosted OpenAI voices. An unrecognized local name returns 400.
POST /v1/translations
Text translation (JSON body).
FieldTypeDefaultDescriptioninputstring or arrayrequiredtext(s) to translatesource_languagestringrequiredsource language codetarget_languagestringrequiredtarget language codeResponse: {"translations": [{"text": "..."}]}.
POST /v1/audio/translations
Speech translation (ASR -> NMT, multipart). Accepts the ASR common fields: automatic_punctuation, verbatim, profanity_filter, speech_contexts, prompt - identical semantics to /v1/audio/transcriptions - plus:
FieldTypeDefaultDescriptionfileWAV uploadrequiredsource speechlanguagestringautosource language codetarget_languagestringen-UStarget language coderesponse_formatstringjsonjson, verbose_json, or textResponse (json): {"text": "..."} (the translation); verbose_json adds task, language, and duration.
POST /v1/audio/speech/translations
Speech-to-speech translation (ASR -> NMT -> TTS, multipart; extension). Same fields as /v1/audio/translations, except target_language is required and response_format is audio:
FieldTypeDefaultDescriptiontarget_languagestringrequiredtarget language coderesponse_formatstringwavwav or pcmvoicestringmodel defaultTTS voice for the translated audio; follows /v1/audio/speech voice rulessample_rateintmodel defaultoutput rate, from 8000 Hz through the loaded TTS model rateResponse: translated mono signed PCM16 audio in the requested container.
POST /v1/audio/diarizations
Speaker segmentation without transcription. Alias: /v1/diarizations.
FieldTypeDefaultDescriptionfileWAV uploadrequiredaudio to segmentmodestringstreamingstreaming for long-form audio, or full-attention offline for recordings up to about 6.6 minutesResponse: {"segments": [{"start": s, "end": s, "speaker": n}]} (1-based speaker ids).
Request mode=offline uses full attention. It is distinct from diar.preset: offline, which still uses the streaming path.