# NVIDIA Nemotron 3.5 ASR Streaming Explained: How It Works and How to Run It
- URL: https://www.dhruvvakharwala.dev/blogs/nvidia-nemotron-3-5-asr-streaming-explained
- Author: Dhruv Vakharwala (https://www.dhruvvakharwala.dev/about)
- Published: November 9, 2026
- Category: Engineering
- Keywords: nemotron 3.5 asr, nemotron asr streaming, cache-aware streaming asr, fastconformer rnnt, nemotron 3.5 transcribe, streaming speech recognition, nemo asr tutorial, fluidaudio nemotron, openmdw license, multilingual asr model
> NVIDIA Nemotron 3.5 ASR streaming explained: mel frames, cache-aware FastConformer, RNN-T, language prompts, chunk tiers, the 30B Transcribe, and code.
**TL;DR:** **NVIDIA Nemotron 3.5 ASR Streaming** (released 2026-06-04) is a 600M-parameter speech recognizer that turns live audio into punctuated, capitalized text in 40 language-locales from one checkpoint. Audio becomes a log-mel spectrogram (100 frames per second), a 24-layer **cache-aware FastConformer** encoder reads it in chunks of 80 ms to 1120 ms while keeping a fixed-size cache of past attention and convolution state, and an **RNN-T** decoder emits tokens from a 13,087-piece vocabulary. A language prompt (`en-US`, `hi-IN`, or `auto`) tells it which script to write. Caching is the point: every audio frame is encoded once, so latency stays flat however long you talk. A second model, **Nemotron 3.5 Transcribe 30B-A4B** (2026-10-28), is a different animal: an offline audio encoder plus a Mixture-of-Experts language model with about 4.3B active parameters. Both ship under the OpenMDW-1.1 license. I run the 0.6B model on a Mac through FluidAudio's Core ML port for TapTalk's Hindi, so this post explains the model from the ground up and shows the NeMo and Swift code. My own benchmark tables get a separate write-up.
## What is Nemotron 3.5 ASR and why does it exist?
Speech recognition has two jobs people tend to blur together. One is **transcribing a recording**: you have the whole file, accuracy is everything, and a few seconds of delay do not matter. The other is **transcribing live speech**: a person is talking right now, words should appear while they speak, and the model never gets to see the future. Voice agents, live captions and dictation apps need the second job.
NVIDIA's answer is the Nemotron speech family. The line starts with an English-only streaming model and grows into a multilingual one, plus a much bigger offline model that shares the name. Here is the timeline, with dates taken from the model cards, the release notes and the posts themselves:
```mermaid
timeline
title NVIDIA speech models behind this post
2025-08-14 : Parakeet TDT 0.6B v3 on Hugging Face
: 25 European languages, offline, CC BY 4.0
2026-01-06 : Nemotron Speech ASR announced at CES
: cache-aware streaming, 24 ms median time to final
2026-03-13 : nemotron-speech-streaming-en-0.6b card
: English only, 80 to 1120 ms
2026-06-04 : Nemotron 3.5 ASR Streaming 0.6B
: 40 language-locales, language prompt
2026-10-28 : Nemotron 3.5 Transcribe 30B-A4B
: offline speech LLM, FluidAudio 0.18.0 adds Lite
```
Three things to keep straight, because the names collide:
1. **Nemotron 3.5 ASR Streaming 0.6B** (`nvidia/nemotron-3.5-asr-streaming-0.6b`). The streaming recognizer. This is what most of the post is about.
2. **Nemotron 3.5 Transcribe 30B-A4B** (`nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B`). The offline speech LLM. Different architecture, different job.
3. **Nemotron 3.5 Transcribe Lite**. Not an NVIDIA release. It is FluidInference's Core ML port of only the encoder and the CTC timestamp head of the 30B model, with no language model attached. It arrived in FluidAudio 0.18.0.
One more name to avoid confusing: FluidAudio 0.18.0 also added "Nemotron Unified ASR 0.6B", a different NVIDIA model covering 25 European languages. It is not part of this post.
NVIDIA's own account of why the first model exists is about cost, not just accuracy. Old-style "buffered" streaming re-encodes overlapping windows of audio, so the same frames are computed again and again. NVIDIA announced the cache-aware design at CES with a claim of stable latency and a 24 ms median time to final transcript:
[Post on X](https://x.com/i/status/2008654492204441862)
The English model arrived first. The 3.5 model is its multilingual successor: per its model card it is "developed based on" `nvidia/nemotron-speech-streaming-en-0.6b`, with language-ID prompt conditioning added. Together AI posted a summary on release day:
[Post on X](https://x.com/i/status/2062520605102993436)
---
## How does audio become a log-mel spectrogram?
A neural network does not hear audio. It sees numbers. The microphone gives you a stream of **samples**: 16,000 per second for this model (16 kHz, mono). One second of speech is 16,000 floating point numbers, and a waveform of that shape is a poor input. The interesting information, which sounds are being made, lives in how the energy is spread across frequencies over time.
So the first step is a **spectrogram**. Slide a short window along the audio, take a Fourier transform of each window, and you get a column of numbers: how much energy sits in each frequency band right now. Stack the columns left to right and you have a picture with time on the x axis and frequency on the y axis. The **mel** part squashes the frequency axis to match human hearing, which is finer at low pitch and coarser at high pitch. The **log** part compresses loudness the same way, because ears are roughly logarithmic too.
For this model the features are 128 mel bins, with one new column every 10 ms (a hop of 160 samples). Those two numbers come from the model's `metadata.json` in the Core ML port and from FluidAudio's config, and they fix all the arithmetic that follows.
**Worked example, one second of audio:**
| Quantity | Calculation | Result |
|---|---|---|
| Samples | 16 kHz x 1 s | 16,000 |
| Mel frames | 16,000 samples / 160 samples per hop | 100 frames |
| Values in the feature matrix | 100 frames x 128 bins | 12,800 numbers |
| Encoder frames | 100 mel frames / 8 (the encoder downsamples 8x) | 12.5 frames, so one every 80 ms |
The 8x downsampling matters. The FastConformer encoder shrinks the time axis by 8 right at its input, so it works with 12.5 vectors per second instead of 100. NVIDIA's own write-up credits this (versus 4x in older Conformers) for making the model cheap to run.
**What a chunk of 560 ms means:**
- 560 ms is 8,960 samples.
- That is 56 mel frames (560 / 10).
- That is 7 encoder frames (56 / 8, or 560 / 80).
That last number lines up with NVIDIA's configuration. The card sets latency with `att_context_size=[56, r]`, where `r` is the number of **right-context** frames (lookahead) in 80 ms units, and each step processes `r + 1` frames. For `r = 6` that is 7 frames, which is 560 ms. For `r = 13` it is 14 frames, 1120 ms.
The picture above is synthetic, drawn to show the frame arithmetic rather than a real recording. The arithmetic itself is exact.
---
## What does the FastConformer encoder do?
The encoder is the heavy part: 24 layers, a hidden size of 1024, about 600M parameters in total with the decoder. It is a **Conformer**, a stack that mixes two ideas in every layer:
- **Self-attention** lets each frame look at other frames and decide which matter. It is how the model understands that a sound belongs to a word that began half a second earlier.
- **Convolution** looks at a small neighborhood of nearby frames. It is good at local patterns like the shape of a consonant.
"Fast" refers to the aggressive downsampling above, from NVIDIA's FastConformer paper (arXiv 2305.05084). The encoder turns each 80 ms step of audio into a 1024-dimension vector that describes what was said there, in context.
For a recording you can let every frame attend to every other frame. For live speech you cannot, because the future has not happened yet. That is where the streaming design comes in.
---
## Cache-aware streaming vs buffered vs full-context: why does the cache keep latency flat?
There are three ways to run an encoder on audio, and the difference is easiest to see by counting how often each frame gets computed.
**Full context (offline).** Wait for the whole file, run the encoder once, every frame sees everything. Best accuracy, latency equals the length of the recording, and attention cost grows with the square of the length. Parakeet TDT 0.6B v3 works this way for normal use.
**Buffered (chunked) streaming.** Keep a sliding window, say the last 4 seconds, and every second re-run the encoder on the whole window so each new frame has context. It works with any model, but it recomputes. With a 4 s window and a 1 s step, the encoder processes 4 seconds of audio for every 1 second that arrives, 4x the work. Shrink the step for lower latency and the redundancy gets worse. NVIDIA's post describes it as re-reading the last few pages every time you turn a page.
**Cache-aware streaming.** The model is trained so that each layer only needs a fixed amount of past state. At inference you save that state after every chunk and feed it back with the next one. Each frame is encoded exactly once. The state to carry is small and constant, which the Core ML port's metadata makes concrete. For the 2240 ms build it lists:
| Cache | Shape | Meaning |
|---|---|---|
| Channel (attention) | `[1, 24, 42, 1024]` | 24 layers x 42 past frames x 1024 dims |
| Time (convolution) | `[1, 24, 1024, 8]` | 24 layers x 1024 channels x 8 past frames |
**Worked example, the cache size:** 24 x 42 x 1024 = 1,032,192 values for attention, plus 24 x 1024 x 8 = 196,608 for convolution, so 1,228,800 values in total. In 16-bit floats that is about 2.5 MB. After a minute of speech it is still 2.5 MB, because old frames fall out of the window as new ones enter. In NVIDIA's reference configuration the left context is 56 frames, which at 80 ms each is 4.48 seconds of history.
Here is one chunk's trip through the pipeline, with the cache loop made explicit:
```mermaid
sequenceDiagram
participant Mic as Microphone
participant Mel as Log-mel
participant Enc as Encoder (24 layers)
participant Cache as Cache (about 2.5 MB)
participant Dec as RNN-T decoder
participant App as App
Mic->>Mel: chunk N (for example 560 ms, 8960 samples)
Mel->>Enc: 56 mel frames
Enc->>Cache: read past attention and conv state
Cache-->>Enc: 42 to 56 past frames per layer
Enc->>Cache: write updated state (old frames fall out)
Enc->>Dec: 7 encoder frames, one per 80 ms
loop each encoder frame
Dec->>Dec: emit tokens until blank
end
Dec->>App: new text (partial transcript)
Mic->>Mel: chunk N+1 (only new audio)
```
```mermaid
flowchart LR
A["New audio chunk
(80 to 1120 ms)"] --> B["Log-mel
128 bins, 10 ms hop"]
B --> C["24 x FastConformer layer
reads cache, writes cache"]
C --> D["Encoder frames
one per 80 ms"]
D --> E["RNN-T joint network"]
F["Prediction network
(LSTM, last tokens)"] --> E
E --> G{"blank?"}
G -- "no: emit token" --> F
G -- "yes: next frame" --> C
H["Language prompt
128-way one-hot"] --> E
I[("Cache
attention + conv state")] <--> C
G -- "tokens" --> J["Text
punctuated, capitalized"]
style C fill:#3b82f6,stroke:#3b82f6,color:#fff
style I fill:#141414,stroke:#3b82f6,color:#fff
style H fill:#141414,stroke:#3f9c74,color:#fff
```
NVIDIA's concurrency numbers show why this matters for servers: on an H100 with 320 ms chunks, the post reports 560 concurrent streams against 180 for a buffered baseline (3x), and a Modal test of 127 WebSocket clients at the 560 ms setting held a median delay of 182 ms. One caveat the comments raised and the author answered: the baseline model in that comparison was larger, chosen for similar accuracy, so not all of the 3x is the cache alone. For one person on one laptop the saving shows up as a constant, small amount of work per chunk.
The architecture is described in NVIDIA's "Stateful Conformer with Cache-based Inference for Streaming ASR" paper, arXiv 2312.17279.
---
## How does the RNN-T decoder turn encoder frames into text?
The encoder gives you a vector every 80 ms. The decoder has to decide which tokens, if any, belong to each vector. Nemotron uses an **RNN-T** (recurrent neural network transducer), which has three parts:
1. The **encoder** (above), producing one vector per time step.
2. A **prediction network**, a small recurrent network that reads the tokens emitted so far. In the port it is a 2-layer LSTM with hidden size 640. It acts as a built-in language model of the output.
3. A **joint network** that combines the current encoder frame with the prediction network state and scores every token in the vocabulary, plus a special **blank**.
The loop, per encoder frame: ask the joint network for scores. If the best choice is a real token, emit it, update the prediction network and ask again on the same frame (a frame can produce several tokens, like the pieces of a long word). If the best choice is blank, move on to the next frame. Pure greedy decoding, which is what FluidAudio uses, just takes the highest score at every step.
The vocabulary is a SentencePiece set of **13,087 tokens** plus one blank (index 13,087 in the port's metadata). Tokens are word pieces, not whole words, which is why a rare name comes out as several fragments. Because RNN-T is naturally left to right and only needs the frames seen so far, it fits streaming without tricks. That is the main reason streaming recognizers like this use a transducer.
Two small details are easy to miss. First, the model is trained to produce punctuation and capitalization directly, using synthetic labels (NVIDIA says punctuation and capitalization came from Qwen3-32B), so there is no separate punctuation model. Second, in `auto` language mode the model appends a language tag such as `` after the end of a sentence. Both NeMo (`strip_lang_tags`) and Transformers (`skip_special_tokens`) let you drop it.
---
## How does language prompting work (en, hi, auto)?
A multilingual model has a problem: the same sounds can be written in different scripts. Hindi speech can be Devanagari or romanized. If the model has to guess, it will sometimes guess wrong.
Nemotron 3.5 fixes this by telling it. Per the model card, a **128-dimension one-hot language vector** is broadcast across time, concatenated with the encoder output, and projected into the decoder. You pick the language with `target_lang`: a locale like `en-US`, `es-ES` or `hi-IN`, or `auto` to let the model detect it.
In the Core ML port the same idea appears as an integer `prompt_id` input. The metadata's `prompt_dictionary` maps names to ids, for example:
| Prompt | `prompt_id` |
|---|---|
| `en-US` / `en` | 0 |
| `en-GB` | 1 |
| `hi-IN` / `hi` | 6 |
| `auto` | 101 (the default) |
The table has 128 slots in total. What does the prompt change in practice? It picks the output script and language, not the acoustics. I tested this on Hindi (numbers in the teaser below): prompting `en` on Hindi speech produced a score of 84.97% error, because the model faithfully wrote English-looking text. Prompting `hi` and `auto` landed within 0.2 points of each other.
The 40 locales are tiered on the model card:
- **Transcription-ready (19 locales)**, including English, Spanish, French, Italian, Portuguese, Dutch, German, Turkish, Russian, Arabic, Hindi, Japanese, Korean, Vietnamese and Ukrainian.
- **Broad coverage (13)**, including Polish, Swedish, Czech, Danish, Finnish, Mandarin and Romanian.
- **Adaptation-ready (8)**, such as Greek, Hebrew, Thai and Latvian. NVIDIA says these need fine-tuning on in-domain data before they transcribe well.
A caution I wrote about in the [TapTalk rebuild post](https://www.dhruvvakharwala.dev/blogs/rebuilding-taptalk-parakeet-nemotron-live-typing): the advertised language list is not the same as the scripts the vocabulary can write. Counting the 13,087 tokens by Unicode block, Devanagari has 196 pieces, Arabic 252, CJK 6,907, and several Indic scripts have none.
---
## What is the latency vs accuracy trade-off across chunk tiers?
The chunk size is the knob. A bigger chunk means the encoder waits for more audio before it speaks, but each frame also gets more lookahead, so it is more accurate. NVIDIA exposes five settings through `att_context_size=[56, r]`:
| `r` (right context) | Chunk | Latency tier |
|---|---|---|
| 0 | 1 frame | 80 ms |
| 1 | 2 frames | 160 ms |
| 3 | 4 frames | 320 ms |
| 6 | 7 frames | 560 ms |
| 13 | 14 frames | 1120 ms |
You pick the tier at inference time. There is no retraining, one checkpoint serves all of them.
NVIDIA's multilingual card reports FLEURS error rates averaged by tier (with the language given explicitly). The transcription-ready group goes from 10.38 at 80 ms to 8.84 at 1.12 s. English alone goes from 9.43 to 7.91, Spanish from 4.87 to 4.11. Japanese, Korean and Mandarin are scored as character error rate, and NVIDIA notes its text normalization is imperfect, so those figures can overstate real error.
For the English-only predecessor, the card gives Open ASR Leaderboard averages that make the shape clear:
```mermaid
xychart-beta
title "English streaming model, average WER (%) by chunk latency"
x-axis ["80 ms", "160 ms", "560 ms", "1120 ms"]
y-axis "Open ASR average WER (%)" 6 --> 9
bar [8.43, 7.67, 7.07, 6.93]
```
The first doubling of latency buys the biggest drop, and returns shrink: going from 560 ms to 1120 ms doubles the wait for 0.14 points. For a voice agent, where every 100 ms is felt, 160 to 560 ms is the interesting range. For dictation, where the user is already waiting on a hotkey release, a bigger chunk is free accuracy.
**Where do 2240 ms and 4480 ms come from?** They are not on NVIDIA's cards. NVIDIA lists 80 to 1120 ms. The FluidInference Core ML port, `FluidInference/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b-CoreML`, builds four tiers instead, in folders `560ms`, `1120ms`, `2240ms` and `4480ms`, for each of two models (`latin/` with a 2,828-token vocabulary for en, es, fr, it, pt and de; `multilingual/` with the full 13,087). In the 2240 ms build the metadata shows 224 mel frames per chunk, 9 frames of pre-encode cache, `att_context_size` of `[42, 13]` and the cache shapes quoted above. So the port feeds the encoder more audio per call to amortize the cost on Apple hardware; it is a packaging choice, not an NVIDIA setting. The port's own README states that 560 ms is off the trained attention tiling, with a small quality cost, and recommends 2240 ms as the default. One more rule from that README that bit me in practice: the full-vocabulary joint network only runs efficiently at the 2 s tier on the Neural Engine, so use 2240 ms for Hindi, Japanese and Chinese.
---
## How is the 30B-A4B Transcribe model different?
Transcribe is built for a different job, and it is worth reading what NVIDIA says about it: "built for recorded audio, not live streaming." Instead of a transducer it is a **speech-augmented language model**. Audio goes through an encoder, the encoder output is projected into the embedding space of a language model, and the language model writes the transcript token by token, the way a chatbot writes a reply.
From the model card:
- **Language model:** a Mamba-2 and Transformer hybrid Mixture-of-Experts, built on Nemotron-3.5 Lightning 30B-A3B.
- **Audio side:** about 600M-parameter ASR Transformer encoder plus a roughly 99M-parameter diarization (speaker) encoder.
- **Size:** about 34B parameters total, about 4.3B **active** per token.
- **Context:** 65,536 tokens, recordings up to an hour.
- **Extras:** CTC heads for word timestamps, speaker attribution, translation, language identification, audio question answering and prompts in plain language.
- **Release:** 2026-10-28.
**What does A4B mean?** In a Mixture-of-Experts model the feed-forward layers are split into many "experts", and a router sends each token to only a few of them. The model stores all the experts (34B parameters on disk) but only uses a few per token. "30B-A4B" is shorthand for roughly 30B total and 4B **A**ctive. The 4.3B on the card is the language model's roughly 3.5B active parameters plus the audio encoders, which fit together as 3.5 + 0.8. You pay compute for 4.3B, but you still need memory for all 34B, which is why NVIDIA targets one 80 GB H100.
On accuracy the card is strong. The nine-dataset English Open ASR mean is 4.71, with LibriSpeech test-clean at 1.09 and test-other at 2.60, and the seven-language multilingual mean is 4.62 (Hindi 9.15). NVIDIA's launch post gives a 5.33% English leaderboard average and says it ranks first among open-weight models. Those two averages are different measurements, so quote the one you mean. Its limitations are plain too: offline only, and audio understanding and reasoning evaluated only in English.
**Transcribe Lite**, the third name, cuts the language model away. FluidInference's Core ML build keeps the encoder and the CTC timestamp head, and FluidAudio 0.18.0 (2026-10-28) shipped it with parallel speaker labels. The release notes claim a 49.5 minute meeting in about 5.5 seconds on an M5 Pro. Without the LLM it is a plain CTC recognizer, and as the numbers below show, you lose the accuracy that the language model provided.
---
## When would I pick which model?
| If you need | Pick | Why |
|---|---|---|
| Live captions or voice agent on a GPU | Nemotron 3.5 ASR Streaming 0.6B at 160 to 560 ms | cache-aware, flat latency, many concurrent streams |
| English only, streaming | `nemotron-speech-streaming-en-0.6b` | NVIDIA recommends the English model for English-only use |
| Dictation on a Mac | Core ML port, `2240ms` tier | runs on the Neural Engine, best throughput |
| Hindi or non-European languages on device | Core ML `multilingual/` build | best of the models I tested on Hindi and Hinglish |
| Best accuracy on recordings, with diarization | Transcribe 30B-A4B | LLM decoder, timestamps, speakers, but needs an H100-class GPU |
| Meeting transcripts on Apple Silicon, fast | Transcribe Lite | quick, but trails the 0.6B streaming model in my tests |
| 25 European languages, offline, small | Parakeet TDT 0.6B v3 | CC BY 4.0, 8,192-token vocabulary, no Devanagari |
---
## How do I run Nemotron 3.5 ASR with NeMo?
The Hugging Face card gives the loader. Install NeMo's ASR collection, then:
```python
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(
model_name="nvidia/nemotron-3.5-asr-streaming-0.6b"
)
# offline-style call; target_lang picks the language prompt ("auto" detects)
transcriptions = asr_model.transcribe(["file.wav"], target_lang="en-US")
print(transcriptions[0])
```
I did not run this snippet myself (my benchmarks used the Core ML port, not PyTorch), so check the return type against your NeMo version. For real streaming, use the cache-aware script that ships in the NeMo repo. This is the invocation from the model card, where `att_context_size` takes the right-context values `0, 1, 3, 6, 13`:
```bash
python examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
model_path=${MODEL_PATH} \
dataset_manifest=${MANIFEST_PATH} \
output_path=${OUTPUT_FOLDER} \
target_lang=en-US \
att_context_size="[56,13]" \
strip_lang_tags=true
```
Input is mono WAV plus a JSON-lines manifest with `audio_filepath` and `text` (the reference transcript). Set `target_lang=auto` for detection. With the Hugging Face Transformers route (version 5.13 or newer), the processor takes `language="en-US"` or `language="auto"`.
## How do I run it on a Mac with FluidAudio?
There is no PyTorch on the Neural Engine, so Apple Silicon needs a Core ML conversion. FluidInference publishes one and [FluidAudio](https://github.com/FluidInference/FluidAudio) is the Swift library that runs it. The multilingual streaming manager is present in both 0.17.5 and 0.18.0 (0.18.0 is the release that also added Transcribe Lite). This snippet uses only methods I checked in the FluidAudio 0.18.0 source:
```swift
import FluidAudio
// Downloads /ms from the Hugging Face repo and caches it.
// hi-IN routes to the full-vocabulary `multilingual/` folder.
let dir = try await StreamingNemotronMultilingualAsrManager.downloadVariant(
languageCode: "hi-IN",
chunkMs: 2240
)
let manager = StreamingNemotronMultilingualAsrManager()
try await manager.loadModels(from: dir)
await manager.setLanguage("hi-IN") // picks the prompt_id
// samples: [Float], 16 kHz mono. Feed audio as it arrives.
_ = try await manager.process(samples: samples)
let partial = await manager.getPartialTranscript()
let text = try await manager.finish() // flushes the last partial chunk
let detected = await manager.detectedLanguage()
await manager.reset()
```
`process` returns an empty string and delivers text through `getPartialTranscript()` or a callback set with `setPartialCallback`. `finishWithTokenTimings()` returns per-token timings if you need word times. The CLI equivalent from the port's README is:
```bash
fluidaudiocli nemotron-multilingual-transcribe \
--input audio.wav \
--model-dir latin/2240ms \
--language de-DE
```
The package's own doc sample passes `[Float]` to `process(audioBuffer:)`, which does not match the source. The method that takes `[Float]` is `process(samples:)`. Apple Silicon only. The port's README also says access is granted through a Discord server, so check the repo page before building a release that depends on the download.
---
## What did I see on my Mac?
I run Nemotron 3.5 ASR through this exact port inside TapTalk, on an M3 Pro with FluidAudio 0.17.5 (Transcribe Lite through 0.18.0), as the engine for Hindi (before the rebuild TapTalk ran Whisper, which is gone from the local path). A full benchmark write-up follows, so here are only the headline numbers.
| Measurement | Result |
|---|---|
| Indian English (Svarah) WER at 0.56 / 1.12 / 2.24 / 4.48 s | 13.24 / 12.29 / 12.06 / 12.20% |
| Hindi-English speech (Monsoon hi-IN) error, Nemotron vs Orukeet (a Parakeet TDT variant) and Parakeet | 14.18% vs about 92 to 93% |
| Same Monsoon set, prompt `en` instead of `hi` | 84.97% |
| Transcribe Lite on Svarah | 21.23% vs 12.06% for the 0.6B |
Two limitations belong here. The model returned **empty text on 704 of 1,214 clips shorter than one second**, and it **deleted the first word on 44.9% of Monsoon clips**. Putting one second of silence in front of the audio brought the first-word loss down to the other models' rate. Tier changes cost little: 1.12 s was 0.2 to 0.3 points worse than 2.24 s, 0.56 s about 1.2 to 1.3 points worse. This is a single machine and one run per set, so treat gaps inside a confidence interval as ties. The next post covers the method and the full tables.
---
## What does the OpenMDW-1.1 license allow?
Both the 3.5 streaming model and the 30B Transcribe model are governed by **OpenMDW-1.1**, and the FluidInference Core ML port carries the same license. That is a change from the English streaming model, which uses the NVIDIA Open Model License. In plain terms, from the license text:
- You may use, modify and distribute the model materials free of charge, with broad rights ("deal in the Model Materials without restriction") covering copyright, patent and database rights. The model card also marks the 3.5 model as ready for commercial use.
- If you redistribute any part, you must keep the license and the copyright and origin notices.
- If you sue claiming the model infringes a patent or copyright, your rights under the license end, unless you are answering a suit someone else started.
- The license puts no restrictions or obligations on the outputs the model generates.
- It is provided "as is": no warranty, and you do your own diligence on third-party rights and consent.
I am not a lawyer. For a product, read the text yourself at openmdw.ai, and also read the license of the datasets you use to evaluate. Parakeet TDT 0.6B v3 is CC BY 4.0, a different regime, and the Transcribe card notes that a few bundled components are Apache 2.0.
---
## What was it trained on?
For the 0.6B multilingual model, NVIDIA lists the proprietary Riva multilingual ASR set, NVIDIA Granary, Multilingual LibriSpeech, Mozilla Common Voice, FLEURS and VoxPopuli with Europarl-ASR, in the 10,000 to 1,000,000 hour range. Some labels are synthetic, produced by an ensemble of ASR models. The English predecessor lists about 530k hours, mostly the Riva set and Granary-derived YouTube-Commons, YODAS2, Mosel and LibriLight. The 30B model lists about 1.72M unique hours, including Granary v2 data (NVIDIA's post puts the public Granary v2 release at 463,062 hours across 48 languages, a figure for the dataset, not for Transcribe's training mix).
The card for the 0.6B does not list code-switched speech. That detail is why Hinglish, where Hindi and English alternate in one sentence, is a stress test rather than a covered case. The 30B card does list code-switching.
---
## Where it falls short, and who else noticed
Third-party reactions are useful here. The creator of parakeet.cpp posted that a CPU-only C++ runtime reproduces the model:
[Post on X](https://x.com/i/status/2063207757692441056)
My own known issues, from running the Core ML port:
- Empty output on many clips under one second.
- First-word drops when speech starts right at the first sample. A short silence prefix is a cheap fix.
- Hindi costs about 3.9 tokens per word because only 196 of 13,087 pieces are Devanagari, so it decodes close to character by character.
- 560 ms is off the trained attention tiling, and FluidAudio's docs warn that long continuous sessions at that tier emit sparse punctuation.
---
## FAQ
### What is NVIDIA Nemotron 3.5 ASR Streaming?
A 600M-parameter speech recognizer released on 2026-06-04 that turns live audio into punctuated, capitalized text in 40 language-locales from one checkpoint, under the OpenMDW-1.1 license.
### How does cache-aware streaming work in Nemotron 3.5 ASR?
The 24-layer FastConformer encoder reads audio in chunks of 80 ms to 1120 ms and keeps a fixed-size cache of past attention and convolution state. Every audio frame is encoded once, so latency stays flat however long you talk. An RNN-T decoder then emits tokens from a 13,087-piece vocabulary.
### How do I choose the language in Nemotron 3.5 ASR?
Pass a language prompt such as `en-US`, `hi-IN` or `auto`. It tells the model which script to write.
### What is the difference between Nemotron 3.5 ASR and Nemotron 3.5 Transcribe 30B-A4B?
The ASR model is a 0.6B streaming recognizer. Transcribe 30B-A4B (released 2026-10-28) is an offline audio encoder plus a Mixture-of-Experts language model with about 4.3B active parameters. Both ship under the OpenMDW-1.1 license.
### Can I run Nemotron 3.5 ASR on a Mac?
Yes. FluidAudio's Core ML port runs the 0.6B model on Apple Silicon, and this post shows the Swift code next to the NeMo Python version.
---
**Takeaway:** Nemotron 3.5 ASR Streaming is a small, openly licensed recognizer built so that each audio frame is computed once: log-mel features at 100 frames per second, an 8x-downsampled cache-aware FastConformer, an RNN-T decoder and a language prompt that selects the script. Pick the chunk tier by what you can afford to wait: 160 to 560 ms for agents, a second or two for dictation. Do not confuse it with Transcribe 30B-A4B, an offline speech LLM that needs a big GPU, or with Transcribe Lite, a third-party port of its encoder with no language model. On a Mac the Core ML port makes it practical, and the known gaps (short clips, first word) are fixable at the application layer.
## References
- [nvidia/nemotron-3.5-asr-streaming-0.6b model card](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b) (architecture, tiers, languages, WER, license, usage)
- [NVIDIA-Nemotron-3.5-ASR-Streaming-Multilingual-0.6b model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-ASR-Streaming-Multilingual-0.6b)
- [nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B)
- [nvidia/nemotron-speech-streaming-en-0.6b model card](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b) (English predecessor, leaderboard WER)
- [NVIDIA: Nemotron Speech ASR, scaling voice agents](https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents) (buffered vs cache-aware, concurrency)
- [NVIDIA: Fine-tuning Nemotron 3.5 ASR](https://huggingface.co/blog/nvidia/fine-tuning-nemotron-35-asr) (language prompts, `att_context_size`, streaming script)
- [NVIDIA: Nemotron 3.5 Transcribe and Granary v2](https://huggingface.co/blog/nvidia/nemotron-transcribe-granary)
- [nvidia/parakeet-tdt-0.6b-v3 model card](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)
- [Stateful Conformer with Cache-based Inference for Streaming ASR (arXiv 2312.17279)](https://arxiv.org/abs/2312.17279)
- [Fast Conformer with Linearly Scalable Attention (arXiv 2305.05084)](https://arxiv.org/abs/2305.05084)
- [OpenMDW-1.1 license text](https://openmdw.ai/license/1-1/)
- [FluidInference Core ML port of Nemotron 3.5 ASR Streaming](https://huggingface.co/FluidInference/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b-CoreML)
- [FluidAudio v0.18.0 release notes](https://github.com/FluidInference/FluidAudio/releases/tag/v0.18.0) and the [FluidAudio repository](https://github.com/FluidInference/FluidAudio)
- [Rebuilding TapTalk with Parakeet and Nemotron](https://www.dhruvvakharwala.dev/blogs/rebuilding-taptalk-parakeet-nemotron-live-typing) (why Hindi routes to Nemotron)
- [NVIDIA AI Developer on X, 2026-01-06](https://x.com/NVIDIAAIDev/status/2008654492204441862), [Together AI on X, 2026-06-04](https://x.com/togethercompute/status/2062520605102993436), [Ettore Di Giacinto on X, 2026-06-06](https://x.com/mudler_it/status/2063207757692441056)