# Nemotron 3.5 ASR Benchmark on a Mac Neural Engine: Accuracy, Speed, and Every Way It Fails
- URL: https://www.dhruvvakharwala.dev/blogs/nemotron-3-5-asr-benchmark-mac-neural-engine
- Author: Dhruv Vakharwala (https://www.dhruvvakharwala.dev/about)
- Published: November 11, 2026
- Category: Engineering
- Keywords: nemotron 3.5 asr benchmark, nemotron 3.5 asr mac, nemotron core ml, fluidaudio benchmark, parakeet tdt 0.6b v3 vs nemotron, hinglish speech recognition, indian english asr wer, apple neural engine speech recognition, streaming asr chunk size latency, nemotron 3.5 transcribe lite
> Nemotron 3.5 ASR and Transcribe Lite on an M3 Pro via Core ML vs Orukeet and Parakeet: WER, speed, chunk tiers, empty output, first-word loss.
**TL;DR:** I benchmarked **Nemotron 3.5 ASR** (NVIDIA's 0.6B streaming model) and **Nemotron 3.5 Transcribe Lite** (FluidInference's encoder and CTC-head port of NVIDIA's new 30B Transcribe) through Core ML on an Apple M3 Pro, with [FluidAudio](https://github.com/FluidInference/FluidAudio) 0.17.5 and 0.18.0. The opponents were **Orukeet** and **Parakeet TDT 0.6B v3**. Same runner, same scorer, 13,783 clips, six test sets, paired speaker bootstrap intervals. Results: Nemotron 3.5 ASR scores **12.06%** on Indian English (Svarah) against 7.96% for Orukeet and 8.46% for Parakeet, and **2.70%** on LibriSpeech test-clean against 1.51% and 1.63%. But on conversational Hindi it is the only usable model: **14.18%** against about 92% for the two Latin-only models. Transcribe Lite trails everywhere (21.23% on Svarah, 42.24% on Hindi) and costs a flat 257 ms per clip. The 2240 ms chunk tier is the sweet spot: 1120 ms costs 0.2 to 0.3 points, 560 ms over 1 point. And it has two failure modes that average WER hides: empty output on **704 of 1,214 clips under one second**, and the **first word deleted** on 44.9% of Hindi clips.
## Why I ran this benchmark
I build [TapTalk](https://www.dhruvvakharwala.dev/blogs/building-taptalk-local-speech-to-text-macos), a free on-device dictation app for macOS. It routes each dictation to one of several recognizers: Orukeet for English and European languages, Parakeet as a fallback, and **Nemotron 3.5 ASR** for Hindi and everything else. I wrote about the first routing in the [Parakeet and Nemotron rebuild](https://www.dhruvvakharwala.dev/blogs/rebuilding-taptalk-parakeet-nemotron-live-typing) and about Orukeet in the [Silero VAD and Core ML post](https://www.dhruvvakharwala.dev/blogs/taptalk-orukeet-silero-vad-core-ml-microphone).
When NVIDIA released Nemotron 3.5 Transcribe on October 28, 2026, and FluidAudio 0.18.0 shipped a Core ML port of part of it the same day, I wanted numbers instead of impressions. The questions were: does multi-context training cost accuracy, is the gap on code-switched Hindi caused by the language prompt or by the model's size, and what do the failures actually look like.
This post is the measurement side. If you want to know **what the model is** (cache-aware streaming, RNN-T, the language prompt, attention context), read the [Nemotron 3.5 ASR explainer](https://www.dhruvvakharwala.dev/blogs/nvidia-nemotron-3-5-asr-streaming-explained) first. I will not re-derive the architecture here. I will use five facts from it:
- Nemotron 3.5 ASR is a 0.6B cache-aware streaming RNN-T. It takes a language prompt that picks the output script.
- It reads audio in chunks. FluidInference exports four chunk sizes: 560, 1120, 2240 and 4480 ms.
- Its vocabulary has 13,087 pieces, of which only 196 are Devanagari.
- Nemotron 3.5 **Transcribe** is a different model: a 30B-A4B speech LLM. "Transcribe Lite" is FluidInference's name for its ~600M encoder plus a CTC timestamp head, with the LLM removed. It is not an NVIDIA release.
- Both Nemotron models use the OpenMDW-1.1 license.
Two posts from launch day, for context on what Transcribe is claimed to do:
[Post on X](https://x.com/i/status/2115462300241141882)
Artificial Analysis, an independent benchmarking site, posted its own result about ten minutes before NVIDIA's post: a tie for first among open-weight speech-to-text models at 2.8% WER.
[Post on X](https://x.com/i/status/2115459886045942161)
Neither of those is about the 0.6B streaming model or about the Lite port, and that distinction matters for everything below. The full Transcribe model uses its LLM; Lite does not.
```mermaid
timeline
title How the numbers were collected
2026-10-14 : Baseline run on M3 Pro
: Orukeet, Parakeet, Nemotron 2240 ms on 6 sets
: Short-clip padding probe
2026-10-28 : NVIDIA releases Transcribe
: FluidAudio 0.18.0 adds Transcribe Lite
2026-10-29/30 : Lite on 6 sets (1 h 35 min)
: Nemotron tiers 560, 1120, 4480 ms
: Prompt overrides auto and en
2026-10-30 : Leading-silence runs
: Hinglish, region and onset probes
```
---
## How I set up the benchmark
### The machine and the runner
Everything ran on one **Apple M3 Pro** with **macOS 27.0.1** on AC power. The Neural Engine (ANE) is a shared resource on Apple silicon: the camera, audio effects and other apps can use it. A video call roughly doubled encoder time in an earlier experiment, so I checked for one before the 29 and 30 October runs, and TapTalk was quit for those. I did not log that check for 14 October.
FluidAudio is the Swift library that runs these models on Core ML. I used two versions side by side:
- **0.17.5** for Orukeet, Parakeet and Nemotron 3.5 ASR, which is what TapTalk ships.
- **0.18.0** only for Transcribe Lite, because that engine did not exist before it.
The runner is a small Swift command-line tool, `ttasr`. It does one thing: read a manifest of clips, transcribe each one with one engine, and write a JSON row per clip with the text, the wall-clock milliseconds and the audio length. Each engine gets what TapTalk gives it: 16 kHz mono float samples and a fresh decoder state for every clip. Orukeet and Parakeet load through `AsrModels.loadLocal` with the encoder on `.cpuAndNeuralEngine`, exactly how TapTalk loads Orukeet. Nemotron goes through FluidAudio's streaming multilingual manager with the language set first and a reset per clip, the same order the app uses.
A clip the engine rejects stays in the denominator as an empty hypothesis. FluidAudio's TDT path refuses audio under 0.3 s (89 Svarah clips), and so does TapTalk's own speech gate. My first Svarah run aborted on those clips, which taught me to score a rejection as "nothing said" instead of crashing.
```mermaid
flowchart LR
M["Sealed manifest
clips, sha256, references,
license, speaker key"] --> R["ttasr runner
FluidAudio 0.17.5 / 0.18.0"]
A["Audio
16 kHz mono"] --> R
L{{"Shared ANE lock
one Core ML run at a time"}} --> R
R --> H["Per-clip hypotheses
text, ms, seconds, error"]
H --> N["Pinned normalizer
(or OIWER lattice for Hindi)"]
N --> C["Errors / words
per clip"]
C --> B["Paired speaker bootstrap
10,000 resamples"]
B --> T["WER + 95% CI"]
style R fill:#3b82f6,stroke:#3b82f6,color:#fff
style L fill:#c2772e,stroke:#c2772e,color:#fff
style T fill:#3f9c74,stroke:#3f9c74,color:#fff
```
### Benchmark hygiene
Speed numbers on a laptop are easy to ruin, so these were the rules:
- **One Core ML job at a time.** Every run takes a shared lock directory before starting and waits if another process holds it. The lock exists because I had a second experiment running in parallel on the same machine, and two ANE users skew each other's timings.
- **AC power throughout; no video call and TapTalk quit (checked before the 29 and 30 October runs).**
- **Warm runs.** The model loads once per run and each speed figure is a median over 168 to 6,656 clips, so the first-call compile cost is out of the median. Lite also calls an explicit `warmup()` before the first clip.
- **A pause between runs.** The loop sleeps 15 s after each model to let the machine settle.
- **Model order.** On 14 October, for each corpus Orukeet, Parakeet and Nemotron ran back to back in that fixed order; Lite, the chunk tiers, the prompts and the lead-silence runs were queued separately on 29 and 30 October. I did not randomize the order, I did not repeat runs (decoding is greedy, so accuracy is deterministic), and the speed medians come from 168 to 6,656 clips per run. I did not interleave clip by clip, and I did not measure thermal state, so treat the speed numbers as approximate.
- **Never compare across machines.** FluidInference's published numbers come from an M5 Pro (macOS 26.5 for the Nemotron 3.5 ASR card). I quote them as reported, not as comparable.
### Why the scorer is pinned
Word error rate (WER) hides a lot of choices, and the biggest is text normalization: is "Mr." the same as "mister", does case count, are "e-mail" and "email" one word? I used the same pinned normalizer as the Orukeet team's evaluation code, so my tables line up with theirs. It also merges compound words, which matters more than you would think: Transcribe Lite scores 5.00% on LibriSpeech test-clean with that scorer and **5.48%** with a plain Whisper-style normalizer and no compound merge. FluidInference reports 5.50% for the same model, so my runner reproduces it to 0.02 points with a plain Whisper-style normalizer.
### WER with a worked example
WER is edit distance on words, divided by the number of words in the reference:
```text
WER = (substitutions + deletions + insertions) / words in the reference
```
A toy example (not from any dataset):
```text
reference: the cat sat on the mat
hypothesis: the cat sit on mat today
S D I
```
One substitution (sat/sit), one deletion (the second "the"), one insertion ("today"). That is 3 errors over 6 reference words, so WER is 50%. Two things follow. WER can exceed 100% if a model inserts lots of words. And it is **not** a count of wrong sentences: the sets here have 6,000 to 70,000 reference words, so I always print errors and words next to the percentage, for instance 8,419 errors over 69,808 words for Nemotron on Svarah.
The error types turn out to be informative. On Svarah, Nemotron's 8,419 errors are 4,524 substitutions, **2,655 deletions** and 1,240 insertions. Orukeet's 5,560 are 3,279, 720 and 1,561. Nemotron deletes words nearly four times as often as Orukeet, and that is the thread that runs through the failure modes below.
### Confidence intervals, without the jargon
A difference of 0.5 points between two models on one test set could be real or luck. A **bootstrap** answers "how much would that difference move if I had a slightly different test set?" by simulating it: draw a new test set by sampling with replacement from the one I have, recompute the difference, and repeat 10,000 times. The middle 95% of those differences is the confidence interval.
Two details make it trustworthy here:
- **Paired.** Both models are scored on the same resampled clips, so a hard clip hurts both and cancels out of the difference.
- **By speaker.** I resample whole speakers, not single clips. Clips from the same person are not independent (same voice, same room, same accent), and resampling clips would give intervals that are too tight.
If the interval contains zero, the two models tie. About 2,500 words resolves a gap of roughly 5 errors; below that, I call it a tie.
### OIWER for Hindi
Plain WER cannot score Hindi that people actually write. "kitna" and the Devanagari spelling of it are the same word, and Hinglish speakers write English loanwords either way ("hotel" or its Devanagari form). A single reference string would count a correct word in the other script as an error, and a model that writes perfectly correct romanized Hindi would score near 100%.
The Monsoon dataset solves this with a **lattice reference**: for each span of the sentence it lists every accepted written form, Devanagari and romanized. I score it with **OIWER**, using the `voi-oiwer` 0.1.4 package, the scorer the dataset card points to. A word is right if it matches any accepted form of its span. The word count differs slightly per model (13,437 to 13,631) because the aligner counts compound spans per hypothesis.
This matters for reading the table later. Orukeet and Parakeet scoring about 92% is not a script-bias artifact. The lattice accepts romanized Hindi, and they still fail, because they cannot write Hindi at all. Their tokenizer has zero Devanagari pieces and no Hindi training.
---
## What did I test on, and why those datasets?
I wanted sets that match what real users do: Indian-accented English, conversational Hindi with English mixed in, and standard English anchors so I can check my runner against published numbers.
| Set | Size | What it is | License |
|---|---|---|---|
| Svarah (AI4Bharat) | 6,656 clips, 9.61 h, 115 speakers, 19 first languages | Indian English: read, extempore and command-style prompts, recorded in an app. Main set | CC-BY-4.0, gated (login and accepted terms) |
| EdAcc Indian English | 168 clips, 0.58 h, 5 speakers | Spontaneous Zoom conversation, Indian English subset | CC-BY-SA-4.0 |
| Monsoon hi-IN (Voice Arena) | 753 clips, 1.33 h, 468 speakers | Conversational Hindi with natural English code-mixing, lattice references | CC-BY-4.0 |
| LibriSpeech test-clean | 2,620 clips, 5.40 h | Read English audiobooks | CC-BY-4.0 |
| LibriSpeech test-other | 2,939 clips, 5.34 h | Harder read English | CC-BY-4.0 |
| FLEURS English | 647 clips, 1.77 h | Read English sentences | CC-BY-4.0 |
Why these:
- **Svarah** has the speakers I care about: 115 of them across 19 Indian first languages, with age and gender labels, so I can see where a model breaks down. It has no speaker ID column, so I derive a key from the speaker profile fields. That gives 115 keys against 117 speakers in the paper, so at most two profile collisions.
- **EdAcc** adds spontaneous speech, with a caveat I report openly: only 5 speakers, and three monologues of one speaker hold about half the words. I also report the view without them, "under 30 s".
- **Monsoon** is the only public set I found with lattice references for Hinglish. It is new 2026 data with the private split withheld by the publisher.
- **LibriSpeech and FLEURS** are the parity checks. If my runner cannot reproduce a model card, nothing else it says is believable.
Two honesty flags. **LibriSpeech test-other is Orukeet's adaptation data**: the final training stage used all 2,939 recordings and the checkpoint was selected on them, so I report it but it is not evidence. And contamination for the Nemotron models on any public Indian set is unverifiable: the 0.6B was trained on a proprietary multilingual set, and Transcribe on 1.72 million hours including web-derived speech.
I treat the Svarah data as gated: it needs a login and accepted terms, so this post only uses aggregate numbers, never transcript text.
---
## What are the results? The main WER table
Lower is better. The last column is the paired speaker bootstrap interval of Orukeet minus Parakeet, in percentage points (pp).
| Set / view | Orukeet | Parakeet v3 | Nemotron 3.5 ASR | Transcribe Lite | Orukeet - Parakeet, 95% CI |
|---|---|---|---|---|---|
| Svarah, all | **7.96%** (5,560 / 69,808) | 8.46% (5,904) | 12.06% (8,419) | 21.23% (14,818) | [-0.68, -0.30] |
| Svarah, clips 1 s or longer | **7.52%** | 7.95% | 10.93% | 20.31% | [-0.63, -0.24] |
| EdAcc, all | **6.94%** (415 / 5,976) | 7.03% (420) | 11.40% (681) | 12.30% (735) | [-0.81, +0.17] |
| EdAcc, under 30 s | **10.55%** | 11.09% | 17.95% | 18.85% | [-1.20, -0.06] |
| Monsoon hi-IN, OIWER | 92.28% | 92.95% | **14.18%** (1,929 / 13,601) | 42.24% (5,758) | [-1.03, +0.09] |
| LibriSpeech test-clean | **1.51%** (802 / 53,000) | 1.63% (866) | 2.70% (1,430) | 5.00% (2,650) | [-0.22, -0.04] |
| LibriSpeech test-other (Orukeet trained on it) | 3.02% | 3.40% | 6.24% | 9.40% | [-0.54, -0.25] |
| FLEURS English | **4.23%** | 4.42% | 7.34% | 11.58% | [-0.45, +0.06] |
```mermaid
xychart-beta
title "Svarah Indian English, WER % (69,808 words)"
x-axis ["Orukeet", "Parakeet v3", "Nemotron 3.5 ASR", "Transcribe Lite"]
y-axis "WER %" 0 --> 25
bar [7.96, 8.46, 12.06, 21.23]
```
### What each model did
**Orukeet** has the fewest errors on every English set. Against Parakeet the gap is small but consistent: 0.5 points on Svarah (about 6% relative), with an interval that excludes zero. It has fewer errors than Parakeet on 14 of 19 first languages, ties on Telugu, and trails by 1 to 5 errors on four (Assamese, Bodo, Gujarati, Maithili), every one under 0.3 points. It wins on both genders and every age group. On EdAcc, FLEURS and Monsoon the interval crosses zero, so those are ties. I would not oversell it: this is a half-point gap, and the "small" 172-clip Svarah subset actually flips sign (Orukeet 8.06% vs Parakeet 7.64%, interval -0.33 to +1.20). That is why I only use the full set for decisions.
**Parakeet v3** is within 0.1 to 0.5 points of Orukeet on every English set. Same vocabulary, same decoder, nearly the same speed.
**Nemotron 3.5 ASR** trails both on English by 3.6 to 4.1 points on Svarah and 1.1 to 1.2 on LibriSpeech clean. It is a 0.6B streaming model built to emit text while you are still talking, and it pays for that. What it has that nothing else here has is the ability to write Hindi: **14.18%** OIWER on Monsoon against about 92%. For TapTalk that is the whole point of keeping it.
**Transcribe Lite** trails Nemotron 3.5 ASR on every held-out set. The paired intervals against Nemotron are +8.23 to +10.13 pp on Svarah, +26.96 to +29.24 on Monsoon, and +2.04 to +2.57 on test-clean. On EdAcc it ties (interval -0.56 to +2.82, but with 5 speakers that is a weak statement). Its errors are overwhelmingly **substitutions**: 11,423 of its 14,818 Svarah errors. It writes broken subword pieces. In the output I saw doubled fragments ("pouredred", "Conconcord") and, inside Hindi output, stray Bengali script. It does almost no damage by omission: 3 empty outputs on all of Svarah, against 737 for Nemotron.
The test-clean number needs a caveat. NVIDIA reports 1.09% for the **full** Transcribe model, and that figure uses the LLM that Lite does not have. My 5.00% is for the encoder and CTC head only (5.48% with plain normalization, against FluidInference's own 5.50% for that same split). Do not read Lite as a stand-in for the 30B model.
**LibriSpeech test-other** is the one place I would not trust Orukeet's lead: it trained on that set.
### Does it hold by accent and region?
Svarah labels speakers by first language, gender, age and state. Aggregate: Orukeet 5.1% to 11.7% by first language, Nemotron 8.6% to 16.9%, Lite 14.6% to 28.8%. The hardest first languages for every model are Bodo, Nepali, Sindhi and Assamese. By zone (zonal council of the speaker's state of residence):
| Zone | Words | Orukeet | Parakeet v3 | Nemotron 3.5 ASR | Transcribe Lite |
|---|---|---|---|---|---|
| North | 9,644 | 7.71% | 8.42% | 11.97% | 20.22% |
| East | 19,573 | 8.38% | 8.85% | 12.55% | 21.92% |
| Northeast | 7,308 | 10.47% | 10.36% | 15.71% | 25.05% |
| West | 14,800 | 6.55% | 7.20% | 9.84% | 19.47% |
| South | 18,241 | 7.75% | 8.27% | 11.86% | 20.84% |
Northeast is hardest for everyone, West is easiest. The only zone where Orukeet is not first is Northeast, where Parakeet has 757 errors to Orukeet's 765, a tie. (There is a sixth zone, Central, with 2 speakers, and I draw no conclusion from it.) Zone follows where the speaker lives, not their mother tongue: the data has Urdu first-language speakers in Gujarat and Maharashtra, for example.
Gender and age barely move any model. Nemotron on Svarah: 12.2% female, 11.9% male; 12.6% for ages 18 to 30 and 11.6% for 30 to 45. The one group that stands out is speakers over 60, where Lite jumps to 25.7% against its 21.2% overall.
---
## Does it match the published numbers?
This is the check that makes the rest believable. Orukeet's model card gives NeMo FP32 greedy-decoding numbers with the same scorer:
| Set | Orukeet here / published | Parakeet here / published |
|---|---|---|
| LibriSpeech test-clean | 1.51% / 1.46% | 1.63% / 1.53% |
| LibriSpeech test-other | 3.02% / 2.86% | 3.40% / 3.14% |
| FLEURS English | 4.23% / 3.82% | 4.42% / 4.28% |
The runner reproduces them within 0.05 to 0.41 points. Orukeet's FLEURS gap (+0.41) is wider than Parakeet's (+0.14), consistent with the 6-bit quantization palette costing about 0.35 points in an earlier lab run. Core ML quantization is the likely explanation for the rest.
For the new paths:
| Model, set | Here | Reported by FluidInference |
|---|---|---|
| Transcribe Lite, test-clean, plain WER | 5.48% (2,904 / 53,000) | 5.50% |
| Nemotron 3.5 ASR, FLEURS English raw field, 560 / 1120 / 2240 / 4480 ms | 9.22 / 9.05 / 9.00 / 9.04% | 9.15 / 8.64 / 8.76 / 8.78% |
The Nemotron per-tier numbers are within 0.07 to 0.41 points. FluidInference used a different normalizer on an M5 Pro, so a small gap is expected. One more FLEURS detail: its raw transcription field keeps parenthetical text that Whisper's normalizer deletes from the reference only, which adds 1.7 to 1.8 points to every model. I score FLEURS on its normalized field in the main table.
---
## How much does chunk size cost? The latency tiers
Nemotron 3.5 ASR is exported in four chunk sizes. The weights are the same; only the amount of audio per encoder call changes. All four use the same attention context. The model card states that 560 ms is off the trained attention tiling. NVIDIA's own lower-latency settings (80 to 560 ms) are not exported for Core ML, so this is the range FluidAudio users can actually pick.
| Set | 560 ms | 1120 ms | 2240 ms (TapTalk) | 4480 ms |
|---|---|---|---|---|
| Svarah | 13.24% | 12.29% | **12.06%** | 12.20% |
| EdAcc | 12.13% | 11.61% | 11.40% | 11.24% |
| Monsoon, OIWER | 15.51% | 14.45% | 14.18% | 14.09% |
| FLEURS English | 7.51% | 7.39% | 7.34% | 7.38% |
| Svarah empty outputs | 938 | 879 | 737 | 682 |
| Svarah median ms per clip | 124.3 | 85.2 | 86.9 | 110.0 |
| Svarah RTFx | 33 | 50 | 53 | 45 |
Paired intervals against the 2240 ms tier (positive means worse than 2240):
| Set | 560 ms | 1120 ms | 4480 ms |
|---|---|---|---|
| Svarah | [+0.96, +1.41] | [+0.12, +0.34] | [+0.06, +0.23] |
| Monsoon | [+0.89, +1.77] | [+0.07, +0.48] | [-0.21, +0.01] |
| EdAcc | [+0.37, +1.65] | [+0.03, +0.76] | [-0.58, +0.03] |
| FLEURS | [-0.23, +0.54] | [-0.04, +0.14] | [-0.04, +0.13] |
```mermaid
xychart-beta
title "Monsoon Hindi OIWER % by chunk tier (ms)"
x-axis ["560", "1120", "2240", "4480"]
y-axis "OIWER %" 13 --> 16
line [15.51, 14.45, 14.18, 14.09]
```
What this says:
- **1120 ms costs 0.2 to 0.3 points** on Svarah and Monsoon. **560 ms costs 1.1 to 1.3.** FLEURS (clean read speech) barely moves at all.
- **4480 ms buys nothing** over 2240 ms and runs slower.
- Odd detail: the per-clip time is not monotonic. 1120 ms is the fastest tier at 85 ms median, 560 ms is the slowest at 124 ms. Smaller chunks mean more encoder calls per clip, and the per-call overhead eats the gain. 560 ms also has a p90 of 288 ms on Svarah, against 159 ms at 2240 ms.
- The tier with fewest empty outputs is the largest one (682 at 4480 ms, 938 at 560 ms).
For a dictation app that transcribes after key-up, 2240 ms is the right default and 1120 ms is a real option if every millisecond of end-of-speech latency matters. For live typing, where you want text as you speak, smaller chunks have to be weighed against a 1 to 1.3 point accuracy cost. A caveat about what I could not test: whether multi-context training costs accuracy at a given latency, compared with a model trained at one fixed context. That needs a fixed-context checkpoint at each latency, and none is public. The Core ML exports also fix one attention context. So I measured the multi-context side only and cannot say whether the 0.2 to 1.3 point cost is the price of multi-context training.
---
## How fast is it? Per-audio-second compute and where it runs
Median milliseconds per clip, with RTFx (audio seconds processed per second of compute; bigger is faster), model loaded and warm:
| Set | Orukeet | Parakeet v3 | Nemotron 3.5 ASR | Transcribe Lite (GPU) |
|---|---|---|---|---|
| Svarah | 41.1 / 117x | 43.7 / 109x | 86.9 / 53x | 256.6 / 20x |
| EdAcc | 42.2 / 196x | 47.2 / 183x | 93.1 / 66x | 256.2 / 41x |
| Monsoon | 45.7 / 87x | 53.0 / 68x | 110.7 / 51x | 256.7 / 25x |
| LibriSpeech clean | 44.4 / 145x | 48.7 / 131x | 95.0 / 65x | 257.0 / 29x |
| FLEURS English | 50.8 / 177x | 56.0 / 159x | 135.7 / 68x | 258.4 / 38x |
Turn RTFx into compute per second of audio on Svarah and you get roughly **8.5 ms** for Orukeet, **9.2 ms** for Parakeet, **18.9 ms** for Nemotron and **50 ms** for Lite. The real-time factor (compute over audio length) is the inverse: Orukeet runs at about 0.009, Nemotron at about 0.019.
```mermaid
xychart-beta
title "Svarah: median ms per clip (warm, M3 Pro)"
x-axis ["Orukeet", "Parakeet v3", "Nemotron 3.5 ASR", "Transcribe Lite"]
y-axis "ms per clip" 0 --> 300
bar [41.1, 43.7, 86.9, 256.6]
```
Where each one runs matters:
- **Orukeet and Parakeet**: the encoder is on `.cpuAndNeuralEngine` in my runner, the TapTalk configuration. In an earlier profile, Orukeet's encoder took 26.7 ms on the ANE against 71 to 73 ms on the GPU for an 11-second clip.
- **Nemotron 3.5 ASR**: I did not override its placement; FluidAudio's Nemotron manager defaults to `.cpuAndNeuralEngine`. FluidInference's model card describes it as a Neural Engine ship, and the encoder is int8 for the outer layers and 6-bit palettized in the middle. The card reports 80x RTFx on FLEURS English for the multilingual model at 2 s on an M5 Pro. I get 68x on an M3 Pro. Different chips, so that is context, not a comparison.
- **Transcribe Lite**: runs on the **GPU**, FluidAudio's default for it. I did not force it onto the ANE. A fp16 encoder of this size takes a fixed 48-second window for every clip, so a 2-second clip costs the same ~257 ms as a 30-second one. That is why its RTFx rises with clip length (20x on short Svarah clips, 41x on EdAcc). FluidInference reports 537x on a 49.5-minute recording on an M5 Pro, which is the case Lite is built for: long audio, with speaker labels from a separate diarizer. For short dictation it is the wrong shape.
These are **throughput** figures, not key-up-to-paste latency. In TapTalk, Orukeet's warm key-up-to-text path measured about 207 ms in the app (see [the Orukeet post](https://www.dhruvvakharwala.dev/blogs/taptalk-orukeet-silero-vad-core-ml-microphone)), which includes the audio path and paste, not just the model. A run with leading silence counts the added silence as audio, so its speed is not comparable. I did not measure energy or other chips.
```mermaid
quadrantChart
title Accuracy vs latency on Svarah (position from measured values)
x-axis Faster --> Slower
y-axis Less accurate --> More accurate
quadrant-1 Accurate but slow
quadrant-2 Fast and accurate
quadrant-3 Fast and weak
quadrant-4 Slow and weak
Orukeet: [0.14, 0.68]
Parakeet v3: [0.15, 0.66]
Nemotron 3.5 ASR: [0.29, 0.52]
Transcribe Lite: [0.86, 0.15]
```
The chart places models using Svarah median ms per clip divided by 300 on the x-axis, and one minus WER divided by 25 on the y-axis. It is a picture of the table above, not new data: the two Parakeet-family models are fastest and most accurate on English, and Lite is neither.
---
## Where does Nemotron fail? The failure modes WER hides
### Empty output on short clips
Nemotron returned empty text on **737 of 6,656 Svarah clips** (11%). Most of those are very short. On the 1,214 clips under 1 second it is empty on **704, or 58%**. Padding the **end** of the clip with silence does not help. I tried it three ways:
| Run (Svarah clips under 1 s: 1,313 words) | Errors / words | WER | Empty |
|---|---|---|---|
| Nemotron, no pad | 931 / 1,313 | 70.9% | 704 |
| Nemotron, pad to 1.0 s | 931 / 1,313 | 70.9% | 704 |
| Nemotron, pad to 2.24 s | 931 / 1,313 | 70.9% | 704 |
| Nemotron, pad to 3.0 s | 1,065 / 1,313 | 81.1% | 704 |
| Nemotron, 1.0 s silence **before** | 659 / 1,313 | 50.2% | 508 |
| Orukeet, no pad | 412 / 1,313 | 31.4% | 90 |
| Orukeet, pad to 1.0 s | 367 / 1,313 | 28.0% | 6 |
| Parakeet, no pad | 459 / 1,313 | 35.0% | 97 |
| Transcribe Lite | 904 / 1,313 | 68.8% | 3 |
Padding after the audio does exactly nothing at 1.0 and 2.24 s (identical rows) and hurts at 3.0 s. Silence **before** the audio recovers 196 of the 704. Orukeet, by contrast, is helped by padding to 1 s (31.4% to 28.0%, empties 90 to 6). Lite almost never returns empty, but its WER on these clips is 68.8%, so non-empty is not the same as correct.
These clips are English (Svarah, English prompt), so I did not test Hindi here, but a very short Hindi dictation like "haan" or "theek hai" may well come back as nothing.
### Is the first word dropped?
This is the failure that moves the headline numbers. I measured how often the **first reference word** is deleted, among clips of three or more words that produced output:
| Set | Nemotron | Nemotron, 1.0 s silence first | Orukeet | Parakeet | Lite |
|---|---|---|---|---|---|
| LibriSpeech clean (leading silence in audio) | 0.1% | - | 0.0% | 0.1% | 0.1% |
| FLEURS English | 1.4% | - | 0.0% | 0.0% | 0.2% |
| Svarah | 7.4% | 2.4% | 1.8% | 1.8% | 1.5% |
| EdAcc | 30.4% | 15.3% | 15.6% | 14.5% | 6.0% |
| Monsoon (OIWER alignment) | 44.9% | 19.3% | 63.3% | 65.2% | 19.8% |
Read this carefully. LibriSpeech and FLEURS clips have natural leading silence, so every model is fine. Svarah, EdAcc and Monsoon clips often start right on speech, and Nemotron loses the first word on **7.4%, 30.4% and 44.9%** of them. Every chunk tier does it (44.7 to 44.9% on Monsoon). Orukeet and Parakeet's high Monsoon numbers are because they cannot write Hindi at all, which is not the same failure. EdAcc and Monsoon are cut from conversations, so 15 to 20% is a segmentation floor for everyone.
Prepending **1.0 s of silence** brings Nemotron to the other models' rate and lowers WER:
| Nemotron 3.5 ASR, 2240 ms | Svarah | EdAcc | Monsoon |
|---|---|---|---|
| No silence (what TapTalk does today) | 12.06% (8,419) | 11.40% (681) | 14.18% (1,929) |
| 0.5 s silence first | - | - | 13.57% (1,846) |
| 1.0 s silence first | 11.49% (8,018) | 10.84% (648) | 13.54% (1,840) |
| 2.24 s silence first | - | - | 13.32% (1,811) |
| 1.0 s minus none, 95% CI (pp) | [-0.86, -0.30] | [-1.84, -0.12] | [-1.13, -0.16] |
On Monsoon the middle words are deleted at 3.7%, the first word at 44.9%. The model's problem here is not Hindi; it is the first sample. What I could not tell: whether the cause is the model's first chunk or FluidAudio's streaming state. That would need emitted-token tracing, which I did not do. It also means my "Nemotron trails on English by 3.6 points" figure contains some avoidable loss. With 1 s of lead silence Svarah drops to 11.49%, still 3.5 points behind Orukeet.
### Hinglish: tokens, errors, and the prompt
The right question before blaming the language prompt or the 600M size for the Hinglish gap is whether training coverage and tokens per word explain it. So I measured both.
**Training coverage (as reported on NVIDIA's model cards):** the 0.6B lists no code-switched data (Riva multilingual, Granary, MLS, Common Voice, FLEURS, VoxPopuli). Transcribe lists code-switched speech explicitly and reports 9.40% OIWER on Monsoon hi (full model, with the LLM).
**Tokens per word.** I tokenized the Monsoon reference lattice slots with each model's own tokenizer (ids verified equal to the Core ML bundles': 13,087 of 13,087 and 23,862 of 23,862):
| Words | Nemotron 3.5 ASR (13,087 pieces, 196 Devanagari) | Transcribe CTC (23,862 pieces, 1,112 Devanagari) | Parakeet / Orukeet (8,192 pieces, 0 Devanagari) |
|---|---|---|---|
| Hindi, Devanagari | 3.91 | 1.90 | unknown token |
| Hindi, romanized | 2.50 | 2.00 | 2.07 |
| English-origin, written in Devanagari | 5.17 | 3.17 | unknown token |
| English-origin, Latin | 3.01 | 2.34 | 2.11 |
| Svarah English | 2.41 | 2.00 | 1.82 |
Nemotron 3.5 ASR spells Devanagari almost character by character. A Hindi word costs 1.6 times the tokens of an English word, and an English loanword written in Devanagari 2.1 times. One public Monsoon clip shows it: the Devanagari form of "expense" is 7 pieces for the 0.6B and 5 for Transcribe's CTC tokenizer. The 0.6B heard "express" for "expense" three times in that clip, which looks like an acoustic confusion rather than a tokenization effect.
**Errors by word origin** (OIWER per lattice slot, `hi` prompt):
| Model | Hindi words | English-origin words | First word: error / deleted | Middle: error / deleted | Last: error / deleted |
|---|---|---|---|---|---|
| Nemotron 3.5 ASR | 13.4% (1,613 / 12,078) | 11.5% (107 / 928) | 52.2% / 44.9% | 10.2% / 3.7% | 15.8% / 6.8% |
| same, 1.0 s silence first | - | - | 27.1% / 19.3% | 11.1% / 4.0% | 18.7% / 9.1% |
| Transcribe Lite | 36.9% | **72.7%** | 39.6% / 19.8% | 39.1% / 6.4% | 42.5% / 3.8% |
This answers the question. Nemotron 3.5 ASR does **not** fail on the code-switched words: English-origin 11.5% against Hindi 13.4%. Its biggest single loss is the first word of the clip (52.2% error). Lite is the model that fails on English-origin words: 72.7%, because its CTC head writes them as broken Devanagari or mixed-script pieces. ("English-origin" is a heuristic: a slot whose Latin form is a LibriSpeech or FLEURS word, minus romanized Hindi homographs, 928 of 13,006 slots. I did not hand-label them.)
**The prompt.** The language prompt is a one-hot vector concatenated to the encoder output before the decoder (per NVIDIA's card), so the acoustics are prompt-free. Monsoon, 2240 ms:
| Prompt | OIWER | Deletions | Empty clips |
|---|---|---|---|
| `hi` (TapTalk) | 14.18% | 880 | 31 |
| `auto` | 14.39% (CI vs `hi`: +0.04 to +0.40) | 918 | 33 |
| `en` | 84.97% | 9,003 | 219 |
`auto` detects Hindi and writes Devanagari, at a 0.2-point cost. `en` forces Latin output and deletes most Hindi words. So the prompt picks the script, and it is not the cause of the Hinglish gap. One caveat: the pieces I show come from re-tokenizing output text, not from the ids the model emitted.
---
## What does this change for TapTalk's engine routing?
TapTalk's routing is by language and engine. This benchmark confirms it and shows what to fix:
| Decision | Evidence | Verdict |
|---|---|---|
| English and European dictation | Orukeet 7.96% vs Nemotron 12.06% on Svarah, 1.51% vs 2.70% on clean | Keep Orukeet as default, Parakeet as fallback |
| Hindi and other scripts | Nemotron 14.18% vs about 92% for the Latin-only pair | Keep Nemotron, no alternative among these four |
| Transcribe Lite as an engine | 21.23% Svarah, 42.24% Monsoon, 257 ms flat per clip | Not a candidate. The 30B model needs its LLM |
| Chunk tier | 1120 ms: +0.2 to 0.3 pp, 560 ms: over +1 pp | Stay on 2240 ms |
| Language prompt | `hi` 14.18%, `auto` 14.39% | `auto` is a safe fallback when language is unknown |
| Hindi first-word loss | 44.9% first word deleted on Monsoon, 13.54% with 1 s of lead | Add pre-roll or a silence prefix |
| Very short clips | 58% empty under 1 s on Svarah (English prompt), padding after does nothing | Prepend silence, expect some empties |
The two behaviours I would fix first both sit in the audio, not the model. A pre-roll (keep the last fraction of a second before key-down) or a silence prefix before the first sample recovers most of the first-word loss, and it recovers 196 of the 704 empty clips too. I have not applied it to the app. The benchmark did not change TapTalk; it told me where to look. I also have not tested real microphone input inside TapTalk, where VAD trimming changes what the model sees at the start.
---
## FAQ
### How accurate is Nemotron 3.5 ASR on a Mac?
On an M3 Pro through Core ML it scored 12.06% word error rate on Indian English (Svarah) and 2.70% on LibriSpeech test-clean, behind Orukeet (7.96% and 1.51%) and Parakeet TDT 0.6B v3 (8.46% and 1.63%).
### Is Nemotron 3.5 ASR good for Hindi?
It was the only usable model of the four on conversational Hindi (Monsoon): 14.18% against about 92% for Orukeet and Parakeet, which cannot write Devanagari.
### Which Nemotron 3.5 ASR chunk size should I use?
2240 ms was the sweet spot. Dropping to 1120 ms cost 0.2 to 0.3 points of WER, and 560 ms cost over 1 point.
### How does Nemotron 3.5 Transcribe Lite compare?
It trailed on every set, 21.23% on Svarah and 42.24% on Hindi, and cost a flat 257 ms per clip.
### What are Nemotron 3.5 ASR's failure modes?
Empty output on 704 of 1,214 clips under one second, and the first word deleted on 44.9% of Hindi clips. Average WER hides both.
---
**Takeaway:** Nemotron 3.5 ASR is a specialist, not a general upgrade. It is 3.6 to 4.1 points behind Orukeet and Parakeet on Indian English and twice as slow per clip, but it is the only one of these models that writes Hindi (14.18% against about 92%). Its two real defects are in how audio starts: it deletes the first word on 44.9% of Hindi clips and returns nothing on 58% of clips under a second, and a second of silence before the audio recovers most of the first-word loss and 196 of the 704 empty clips. Transcribe Lite is a long-audio tool, not a dictation engine. Pick the 2240 ms tier. And do not trust a WER table that does not print errors over words and a confidence interval next to each number.
## What I did not measure
- **Real microphone input** inside TapTalk, other chips (M1, M2, M5), or energy use.
- **Fixed-context Nemotron baselines**, so I cannot say what multi-context training costs.
- **NVIDIA's 80 to 560 ms contexts**: not exported for Core ML.
- **The full Transcribe model with its LLM.** Lite is only the encoder and CTC head.
- **Emitted token ids**: the probes re-tokenize text.
- **The root cause of the onset loss**: model first chunk or FluidAudio streaming state.
- **Code-switched Hindi-English on MUCS 2021**: it needs a validated transliterator for scoring Latin output.
- **Speed with the GPU/ANE placement forced** for Lite and Nemotron.
## References and sources
- [Nemotron 3.5 ASR explainer on this blog](https://www.dhruvvakharwala.dev/blogs/nvidia-nemotron-3-5-asr-streaming-explained)
- [FluidAudio v0.18.0 release](https://github.com/FluidInference/FluidAudio/releases/tag/v0.18.0), including Nemotron 3.5 Transcribe Lite
- [FluidInference Nemotron 3.5 ASR Streaming Multilingual 0.6B Core ML](https://huggingface.co/FluidInference/Nemotron-3.5-ASR-Streaming-Multilingual-0.6b-CoreML) (gated model card)
- [FluidInference Nemotron 3.5 Transcribe Lite, Core ML](https://huggingface.co/FluidInference/nemotron-3.5-transcribe-lite-coreml)
- [NVIDIA Nemotron 3.5 Transcribe 30B-A4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B) and [NVIDIA Nemotron 3.5 ASR streaming 0.6B](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
- [Svarah (AI4Bharat)](https://huggingface.co/datasets/ai4bharat/Svarah), [EdAcc](https://huggingface.co/datasets/edinburghcstr/edacc), [Monsoon hi-IN (Voice Arena)](https://huggingface.co/datasets/VoiceArena/MonsoonASR-Open-ASR-leaderboard-hi-IN), [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr), [FLEURS](https://huggingface.co/datasets/google/fleurs)
- [Oruk](https://oruk.ai), the makers of Orukeet
- Earlier posts: [Building TapTalk](https://www.dhruvvakharwala.dev/blogs/building-taptalk-local-speech-to-text-macos), [Rebuilding TapTalk around Parakeet and Nemotron](https://www.dhruvvakharwala.dev/blogs/rebuilding-taptalk-parakeet-nemotron-live-typing), [Orukeet, Silero VAD and Core ML](https://www.dhruvvakharwala.dev/blogs/taptalk-orukeet-silero-vad-core-ml-microphone), [FluidAudio PR 941](https://www.dhruvvakharwala.dev/blogs/fluidaudio-mlmultiarray-zero-fill-pr-941)
- Measurements: Apple M3 Pro, macOS 27.0.1, AC power, FluidAudio 0.17.5 and 0.18.0, runs on 2026-10-14 and 2026-10-29/30. Every figure not marked "reported" is my own measurement.