The runtime loads one GGUF per ASR model. Ready-to-run Q8 GGUFs are published alongside the original checkpoints on Hugging Face and indexed by the CLI:
nemo-speech model list nemo-speech pull nemotron-3.5nemotron-3.5 is the default when --model is omitted. A short name, full repository ID, or existing local GGUF path can be passed to --model.
Nemotron 3.5 (0.6B, multilingual, prompt-conditioned RNNT)
A cache-aware FastConformer-RNNT with language-ID prompt conditioning across 40+ language-locales. Hugging Face:
nvidia/nemotron-3.5-asr-streaming-0.6b
nemo-speech pull nemotron-3.5Select a language such as en-US or es-ES, or use auto for model-based detection. Structured results include the selected or detected language:
nemo-speech transcribe audio.wav \ --model nemotron-3.5 \ --language auto \ --jsonWhen ITN is configured with a parent grammar directory (en/, es/, ...), the same explicit or auto-detected language code selects the grammar used for the final transcript. Unsupported languages remain unchanged.
The CLI uses this model by default. It supports whole-file recognition, recorded streaming with --stream, and live microphone transcription.
Nemotron-Speech Streaming (0.6B, cache-aware RNNT)
English FastConformer-RNNT for whole-file or cache-aware streaming inference. Hugging Face:
nvidia/nemotron-speech-streaming-en-0.6b
nemo-speech pull nemotron-enParakeet TDT (0.6B v3, multilingual, offline transducer)
Token-and-Duration Transducer: the joint predicts each token together with its frame span. 25 European languages, self-punctuating. Hugging Face:
nemo-speech pull parakeet-tdtThe model does not support cache-aware streaming; inference is full-utterance only. Streaming requests are rejected with an error; use offline recognition (nemo-speech transcribe, POST /v1/audio/transcriptions, or gRPC Recognize).
Parakeet CTC (1.1B, offline / buffered streaming)
English FastConformer-CTC for whole-file recognition or overlapping buffered streaming. Hugging Face:
nemo-speech pull parakeet-ctcConverting custom ASR checkpoints
Use the root
converter for a custom checkpoint or alternate quantization. Follow the
to set up its isolated Python environment and choose a supported source. ASR head type (CTC, RNNT, or TDT) is auto-detected; use --head-type only when an override is needed.
Quantization (--outtype)
python3 convert_model.py model.nemo --outfile model.gguf --outtype q8_0Quantization applies to linear and pointwise-convolution weights. Other tensors retain their supported floating-point formats.
--outtypeformatbytes/elemuse caseq8_0 (default)Q8_01.062compact, high-quality defaultbf16BF162.000modern NVIDIA / ARM v9fp16F162.000Apple Silicon, older GPUsq6_kQ6_K0.820smaller artifact, more quantizationq5_kQ5_K0.688smaller artifact, more quantizationq4_kQ4_K0.562compact K-quantnvfp4NVFP40.562FP4; native acceleration on supported Blackwell GPUsmxfp4MXFP40.531compact FP4; acceleration depends on the backendq8_0 is the portable default; pass --outtype to choose a different size/precision tradeoff. K-quants (q4_k/q5_k/q6_k) require inner dim divisible by 256; any tensor that fails alignment falls back to F16 and is reported by the converter. NVFP4 and MXFP4 require inner dim divisible by 64 and use the same fallback. Validate FP4 accuracy and performance on the target model and backend before deployment.
CUDA batching: planar Q8 layout
The default Q8 layout is portable. For high-concurrency CUDA inference, the converter can instead produce a planar Q8 layout:
python3 convert_model.py model.nemo --outfile model.planar.q8_0.gguf \ --outtype q8_0 --q8-layout planarPlanar Q8 is CUDA-only. Keep a default-layout artifact for other backends.
Companion models (optional)
These optional GGUFs are loaded alongside the ASR model and can be changed without reconverting it. Enable them with their runtime options; see
.
Silero VAD
Used for
and VAD-driven
. Convert from the public Silero VAD package:
pip install "silero-vad==6.2.0" python3 convert_model.py silero --outfile models/silero-v6.2.0.gguf # offline alternative, using an existing whisper.cpp Silero checkpoint:# python3 convert_model.py silero --outfile models/silero-v6.2.0.gguf \# --from-whisper-ggml /path/to/silero-v6.2.0-ggml.binSource:
(the pip package), or whisper.cpp's bundled checkpoint for the offline path.
Sortformer speaker diarization
Used for ASR speaker tags and standalone nemo-speech diarize. Sortformer v2 supports up to four speakers, with stateful streaming for long recordings and full-attention inference for short recordings. Convert it with:
python3 convert_model.py nvidia/diar_streaming_sortformer_4spk-v2 \ --outfile models/sortformer-v2-f32.gguf # --outtype f32 is the default; f16 and q8_0 produce smaller artifacts.Enable with --diar-model models/sortformer-v2-f32.gguf; streaming geometry comes from --diar-preset (see
). Segment postprocessing defaults follow the checkpoint and may need tuning for your audio.
Source:
nvidia/diar_streaming_sortformer_4spk-v2
.
PnC (punctuation + capitalization)
Used for
restores casing and . , ? for models that emit lowercase unpunctuated text (e.g. Parakeet CTC). Use a compatible PnC GGUF, or convert a local NeMo BERT punctuation-and-capitalization .nemo checkpoint directly with convert_model.py:
python3 convert_model.py pnc.nemo --outfile pnc-bert.q8_0.gguf --outtype q8_0