# Speculative Listening Explained: What a Full-Duplex Voice Paper Means for On-Device Speech Recognition

- URL: https://www.dhruvvakharwala.dev/blogs/speculative-listening-paper-on-device-asr
- Author: Dhruv Vakharwala (https://www.dhruvvakharwala.dev/about)
- Published: November 8, 2026
- Category: Engineering
- Keywords: speculative listening, speculative decoding, full-duplex speech model, on-device asr, streaming speech recognition, lossless speculative decoding, voice assistant latency, dictation latency, ticktock-bench

> Speculative listening decodes the next audio chunk before it arrives: 2.6-2.8x faster, lossless. Why it fails for ASR, and where it works in dictation.

**TL;DR:** **Speculative listening** is a new training-free trick from the paper "Decode Before You Hear" (arXiv 2611.06982, 4 November 2026). Full-duplex voice models read audio in 1 second chunks and sit idle for 70-89% of each listen chunk. The paper uses that idle time to guess the model's next output as if the next chunk were silence, then checks the guess in the same forward pass that reads the real audio. The output is identical to normal decoding. On an A100 that gives 2.6-2.8x faster speak chunks and a 73% shorter wait before the answer starts, because 95.3% of chunks come out the same under silence. I read it for my own dictation research, and the honest result is this: **it does not transfer to transcription while someone is talking**, since a transcript always depends on new audio (the model that reacts most to user speech accepted only 37.9% of guesses while the user spoke). It transfers exactly at push-to-talk key-up, where the future really is silence. The biggest win is not the speech model at all but the Smart Mode text rewrite: 841 of 1102 ms in one logged dictation. I am working on this paper right now. None of the experiments has finished yet, so every gain below is a ceiling or an estimate, not a measurement, and I will publish the measured results once they are in.


![Cover card for the speculative listening paper breakdown showing 70-89% idle GPU, 2.6-2.8x speedup, 73% faster response start and 37.9% acceptance during user speech](https://www.dhruvvakharwala.dev/images/blog/speculative-listening-paper-on-device-asr/cover.png)

## What is speculative listening, and why did I read a voice-agent paper?

I build a dictation app called TapTalk. You hold a key, talk, release, and text appears in whatever app has focus. The whole product is one number: the time from key-up to text. I have written about [rebuilding it around NVIDIA Parakeet and Nemotron](https://www.dhruvvakharwala.dev/blogs/rebuilding-taptalk-parakeet-nemotron-live-typing), about [moving the microphone and voice activity detection into Swift](https://www.dhruvvakharwala.dev/blogs/taptalk-orukeet-silero-vad-core-ml-microphone), and about [a 20 ms bug I fixed in FluidAudio](https://www.dhruvvakharwala.dev/blogs/fluidaudio-mlmultiarray-zero-fill-pr-941). Every one of those posts is about shaving milliseconds off that wait.

So when a paper claims a 73% cut in response delay for speech models without changing the output, I read it. It is about a different problem. The paper targets **full-duplex voice assistants** that listen and talk at the same time, not speech recognition. My question was narrow: which parts of this trick survive when the model is a transcriber and the user is a person holding a key?

This post goes through the paper in order, then through my own analysis. I did the analysis by reading the paper, the authors' public repository, FluidAudio's source and my own earlier measurements. The paper is by Yutong Bian and eight co-authors (Yanglin Zhang, Yuxiang Wang, Luoyuan Zhang, Rongshen He, Zihao Wang, Junwen Qiu, Jiaqi Li and Zhizheng Wu) from The Chinese University of Hong Kong, Shenzhen and Amphion Technology. Its title is "Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models."

---

## How do full-duplex voice models work, and why is the GPU idle?

A normal voice assistant is turn-based. You speak, a voice activity detector decides you are done, the model thinks, then it answers. That feels like a walkie-talkie. A **full-duplex** model keeps listening while it talks, so it can be interrupted, can say "mm-hm", and can decide on its own when you have finished a thought.

The paper studies one family of these: **chunk-interleaved** models. Audio arrives in fixed 1 second chunks. For each chunk the model reads the audio, then decides one of three things:

- keep listening and output nothing,
- start speaking (or writing a text channel),
- call a tool, such as a search or a calendar lookup.

Reading a second of audio is cheap. A "listen" chunk needs one forward pass over the new audio, which takes the GPU a fraction of the second. The paper measures that the GPU sits idle for **70-89% of the chunk duration** on listen chunks. Then the user stops talking, the model decides to answer, and the work flips. Generating an answer is **serial**: one token per forward pass, each waiting for the last. A tool call takes tens of passes.

| Chunk type | What the model does | Work after the audio arrives |
|---|---|---|
| Listen | Reads audio, outputs nothing | One prefill pass, then the GPU waits |
| Speak | Reads audio, generates reply tokens | Prefill plus one pass per token |
| Tool call | Reads audio, emits a structured call | Prefill plus tens of serial passes |

That imbalance is the problem. Under plain autoregressive (AR) decoding the paper measures 417 to 1089 ms of extra delay before a response starts, depending on the model. Human turn gaps are about 200 ms, so the assistant feels slow. A tool call finishes 1.8-2.6 s after its audio arrives, which misses the 1 s chunk deadline every time. One of the four models, LWS, misses the deadline on 95% of its speak chunks.

![Timeline of 1 second chunks comparing normal decoding with speculative listening: normal decoding leaves the GPU idle then overruns the deadline, speculative listening fills the idle gap with a draft](https://www.dhruvvakharwala.dev/images/blog/speculative-listening-paper-on-device-asr/timeline.png)

Here is the same idea as a schedule. The blue bars are real work, the grey stretches are the idle gap, and the dark blue region in the second row is where speculation goes.

```mermaid
gantt
    title Two listen chunks and an answer, GPU view (illustrative widths)
    dateFormat x
    axisFormat %L ms
    section Normal decoding
    Read chunk 1 audio        :a1, 0, 150
    Idle                      :crit, i1, 150, 1000
    Read chunk 2 audio        :a2, 1000, 1150
    Idle                      :crit, i2, 1150, 2000
    Serial answer decode      :a3, 2000, 3400
    section Speculative listening
    Read chunk 1 audio        :b1, 0, 150
    Draft chunk 2 under silence :active, d1, 150, 1000
    Verify and read chunk 2   :b2, 1000, 1150
    Draft chunk 3 under silence :active, d2, 1150, 2000
    Verify and read chunk 3   :b3, 2000, 2150

```

The widths here are schematic, not the paper's measurements. The point is the shape: the idle stretches in the top lane are exactly what the bottom lane spends on guesses.

---

## What is speculative decoding, and why is it lossless?

Speculative listening is a variation on an older idea, so the lineage matters.

A language model generates one token at a time because each token depends on the one before it. But checking a sequence of tokens is not serial. If someone hands you a guess for the next five tokens, one forward pass over those five positions tells you what the model would have chosen at each of them. That is how a transformer trains, and it is how it scores.

**Speculative decoding** (Leviathan et al. and Chen et al., 2022-2023) uses that asymmetry:

1. A cheap **draft** produces k guessed tokens. Usually that is a small model.
2. The big model runs **one forward pass** over all k guesses and computes its own choice at every position.
3. Keep the guesses up to the first position where the big model disagrees, then take the big model's own token there.
4. Repeat from the new position.

The result is **lossless** for greedy decoding. Every kept token is, by construction, the token the big model would have picked anyway. A wrong guess costs a little wasted compute, never a wrong word. For sampling at a temperature, the standard accept/reject rule keeps the output distribution identical to the big model's.

### A worked example of the acceptance math

Say the draft is right on any single token with probability p = 0.8, independently, and you draft k = 4 tokens. A pass keeps tokens until the first miss and always gains one token from the verifier. The expected tokens per pass is the geometric sum:

```text
E[tokens per pass] = (1 - p^(k+1)) / (1 - p)
                   = (1 - 0.8^5) / (1 - 0.8)
                   = (1 - 0.32768) / 0.2
                   = 3.36
```

So instead of 1.0 token per big-model pass you get about 3.4. The speedup is smaller than 3.4x because the draft costs time, but the big model's passes drop by that factor. If p falls to 0.5 with the same k, the same formula gives 1.94 tokens per pass. Acceptance rate is everything.

The usual drafts are a small separate model, or n-gram lookups that copy phrases already seen in the prompt (Prompt Lookup Decoding), or a recycled pool of earlier guesses (Token Recycling, Lookahead decoding). Those are the baselines in the paper. They need no training, but they only start after the audio has arrived.

---

## How does speculative listening work, step by step?

The paper's twist is **when** the drafting happens. Normal speculative decoding drafts after the input arrives. Speculative listening drafts **before** the input arrives, using the idle window, and it uses **the model itself** as the draft model. The authors call this input-time self-speculative decoding.

The assumption is a hypothesis about the future: the next chunk will be **digital silence**. Under that assumption the model can run ahead.

```mermaid
sequenceDiagram
    participant U as User audio
    participant M as Model (GPU)
    participant K as KV cache
    Note over M: Chunk t-1 finished, GPU idle
    M->>K: Snapshot position
    M->>M: Encode one chunk of digital silence
    M->>M: Greedy decode a draft until end-of-chunk or idle budget
    M->>K: Roll cache back to snapshot, keep draft tokens
    U->>M: Real chunk t arrives
    M->>M: ONE pass over real audio plus draft tokens
    M->>M: Compare draft with own argmax at each position
    Note over M: Keep matching prefix, plus one corrected token
    M->>U: Output identical to normal greedy decoding

```

Step by step:

1. **Silence stand-in.** The model's audio encoder embeds one chunk of digital silence. The authors' port of the method reportedly encodes this once at startup and reuses it for every session.
2. **Draft under silence.** After chunk t-1, the model decodes its own greedy continuation as if chunk t were that silence. It stops at the end-of-chunk token or when the idle time budget is spent.
3. **Roll back.** The draft tokens are stored, and the KV cache (the model's working memory of the conversation) is restored to where it was before the guess.
4. **Verify during prefill.** When the real chunk arrives, the model has to run a forward pass over the new audio anyway. The draft tokens ride along in the same pass. That one pass reads the audio **and** scores every guess. A correct draft costs **zero extra forward passes**.
5. **Accept the prefix.** Tokens are kept up to the first mismatch with the model's true argmax, plus one corrected token. The output is identical to plain greedy decoding, and the post-audio passes can never exceed what autoregressive decoding needed.

![Three cases of token-level verification: silence guess right on a listen chunk, miss at the first token when the user starts speaking, and a partial hit of an already drafted answer](https://www.dhruvvakharwala.dev/images/blog/speculative-listening-paper-on-device-asr/accept.png)

### Two extras for the misses

Pure silence drafts fail when the world changes, so the paper adds two things:

- **Runner-up branch.** If the primary draft finishes before the idle window ends, a second draft starts from the model's second-best first token. This hedges the most common miss, which is the listen-or-speak decision.
- **Cross-chunk hypothesis pool.** Every draft from earlier chunks is stored and indexed by n-gram, as in lookahead decoding. When the model starts a tool call, an earlier chunk may already have drafted most of it.

### Why does the silence guess work so well?

The paper first ran an oracle probe: decode each chunk twice, once with the real audio and once with silence, and compare. Two observations:

| Observation | Number |
|---|---|
| Chunks whose output is identical under silence (4 models pooled) | **95.3%** |
| Mismatched chunks that diverge at the very first output token | **87.7%** (rounded to 88% in the abstract) |

Most chunks are listen chunks, where the model outputs "keep listening" whatever the next second sounds like. When the output changes, it almost always changes at token one, the decision to start talking. That is why a runner-up branch on the first token catches so much.

---

## What are the paper's results?

The authors built **TickTock-Bench**: 8 tasks, 760 sessions, 15,883 one-second chunks. The tasks cover factual QA (Web Questions, Llama Questions, TriviaQA), open answers (AlpacaEval), multi-step reasoning (Big Bench Audio), turn-taking and overlapping speech (Full-Duplex-Bench v1 and v1.5), and tool calls (v3). The main results ran on one A100, one session at a time, bf16, greedy decoding, batch size 1, and only the text backbone (the "Thinker") was timed, not speech synthesis.

Four models were tested, all with 1 s chunks: the authors' own Streaming SLM (trained from the turn-based Qwen2.5-Omni-7B Thinker), plus Gander, Realtime-Venus and LWS, all three built on MiniCPM-o.

### Speedup on speak chunks

| Model | Speak-chunk speedup (mean of tasks) | Mean accepted tokens per pass |
|---|---|---|
| Streaming SLM | 2.76x | 2.63 |
| Gander | 2.60x | 2.20 |
| Realtime-Venus | 2.69x | 2.15 |
| LWS | 2.68x | 2.52 |
| Baselines (PLD, Token Recycling, Lookahead) | 0.99-1.15x | 1.01-1.23 |

The model-free baselines barely beat 1x, because they start after the audio is in and have nothing to copy. Tool calls are the dramatic case: **23.41x** for Streaming SLM and 3.45x for Gander.

### Where the tool-call gain comes from

The ablation separates the components (Llama Questions for speak, tool-call benchmark for tool):

| Method | SLM speak | SLM tool | Gander speak | Gander tool |
|---|---|---|---|---|
| Autoregressive | 1.00 | 1.00 | 1.00 | 1.00 |
| + silence draft | 2.88 | 1.42 | 2.64 | 1.21 |
| + hypothesis pool | 2.87 | 22.04 | 2.97 | 2.04 |
| + runner-up branch | 2.87 | 23.41 | 3.00 | 3.45 |

For speak chunks the silence draft does nearly all of the work. For tool calls the **pool** does: a tool call is mostly already drafted in earlier chunks, so a pool that remembers earlier drafts turns 1.42x into 22.04x.

### How much sooner does the answer start?

```mermaid
xychart-beta
    title "Response onset latency, ms (lower is better)"
    x-axis ["SLM AR", "SLM SL", "Gander AR", "Gander SL", "Venus AR", "Venus SL", "LWS AR", "LWS SL"]
    y-axis "milliseconds" 0 --> 1200
    bar [417, 74, 763, 228, 565, 241, 1089, 204]

```

On average response onset drops 73%, to 74-241 ms. Tool-call latency drops by 72-96% (2623 to 112 ms for Streaming SLM). The maximum lag behind real time per session falls by 94.4% on average. On the tool-call benchmark, Streaming SLM went from 35% of sessions lagging under AR to 0%.

### Quality did not drop

An AlpacaEval judge score (1 to 5) came out the same or marginally higher with speculative listening: 3.10 vs 3.13, 3.56 vs 3.63, 2.98 vs 3.00, 3.02 vs 3.07. In float32 the outputs match AR token for token on nearly all sessions. In bf16, rounding can change some outputs, which the paper treats as a numerics effect, not a flaw in the method.

### Where the gains erode

The authors are open about three limits, and one more shows up in their own tables:

| Condition | Speak / tool speedup |
|---|---|
| Greedy, 1 session | 2.9x / 22.7x |
| Temperature 0.3 | 2.1x / 13.3x |
| Temperature 1.0 | 1.4x / 5.4x |
| 4 concurrent sessions on one A100 | 2.0x / 10.5x |

More sessions on a GPU shrink the idle window, and more sampling randomness lowers acceptance. The method is aimed at low-concurrency, latency-sensitive serving. It also does not cover parallel-stream models such as Moshi, which decode every 80 ms and would need a different formulation, and the paper times only the text backbone.

---

## Where does speculative listening break? When the output depends on the audio

Everything above rests on one fact: in these models, what a chunk produces barely depends on that chunk's audio. Look at the acceptance table by chunk type (the paper's Table 7: Llama Questions, 1,423 chunks per model, with tool calls from the tool-call benchmark).

| Model | All chunks | Listen | Speak | Tool call | While the user speaks |
|---|---|---|---|---|---|
| Streaming SLM | 98.3% | 97.6% | 99.8% | 80.0% | 94.2% |
| Gander | 99.5% | 99.6% | 98.6% | 34.0% | 99.5% |
| Realtime-Venus | 83.3% | 81.4% | 90.0% | n/a | **37.9%** |
| LWS | 99.9% | 99.8% | 100.0% | n/a | 100.0% |

Three of the four models produce nearly the same output with or without the next second of user audio. Realtime-Venus is the one that reacts most to speech, and its acceptance while the user talks collapses to 37.9%. It is also the model with the weakest onset gain in the table (565 to 241 ms, 57%, against 70-82% for the others).

Two more caveats from my reading, flagged as mine:

- **The question-answering sessions are mostly silence.** The repository README describes a QA session as 0.5 s of silence, the question, then about 10 s of silence. For most of those chunks the silence hypothesis is literally true.
- **The "token one" finding leans on one model.** Weighting the per-model rates by chunk count (my own derivation, not a number the authors state) puts roughly 238 of about 270 mismatched chunks on Realtime-Venus.

The paper reports no energy figures and no accounting of the extra compute the drafts use, which matters a lot for a laptop. As of 7 November 2026 the authors' GitHub repository holds only a README, a license and images, and the README says "Code and TickTock-Bench release in preparation." A work-in-progress port to sglang-omni exists on a branch of a separate repository, and that is where the encode-silence-once detail comes from.

None of this undermines the paper. It tells me where the method is strong: a model whose output is mostly a function of its own plan.

---

## Does speculative listening work for speech recognition?

ASR is the opposite case. A transcriber's output at chunk t **is** the transcription of chunk t. If the next second contains the words "send it Friday", a silence draft predicts "no new words" and fails at token one, which is exactly where the paper's misses already pile up.

There is outside evidence for how hard guessing the future of speech is. Speculative Speech Recognition (SSR, arXiv 2407.04641) used an audio-prefixed language model to guess upcoming words, and even when the best of 8 guesses counts, it retrieved only **35.7% of suffixes up to one second ahead** correctly.

There is a second wall: the encoder. A streaming ASR model runs an audio encoder over each chunk, and the encoder output depends on the real audio. No check is cheaper than running the encoder. So a draft can only save **decoder** work, and in the transducer models TapTalk runs (RNN-T and TDT style), decoder work is small. In my own profile of an 11 s Orukeet clip the joint network loop was about 20 ms (roughly 120 calls at 0.17 ms) against a 27 ms encoder. Argmax's WhisperKit paper likewise left speculative decoding out of its Whisper Large v3 Turbo build, because drafter overhead cut the practical speedup to 1.25x.

So mid-utterance, the literal method is out. Here is my table of the cases in TapTalk:

| Case in TapTalk | Does the silence premise hold? | Why |
|---|---|---|
| Any ASR chunk containing speech | No | Output depends on new audio; accepted drafts only land on silent chunks, which have nothing to save |
| Orukeet re-transcription (bidirectional encoder) | Partly | Earlier encoder outputs shift every step, so the old hypothesis must be recomputed |
| Nemotron key-up flush | **Exactly**, if no samples arrived after the guess | FluidAudio's `finish()` zero-pads the leftover tail to one chunk, so it already is the silence hypothesis |
| Listen-or-speak decision | Removed | Key-up is an explicit end-of-turn signal, so the paper's main failure point does not exist |
| Smart Mode text rewrite | **Exactly** when the final transcript equals the guessed one | A rewrite of an identical prompt is a valid output at any temperature |

---

## Where does the idea hold exactly? Push-to-talk key-up

The paper's weakest point is guessing whether the user will start talking. Push-to-talk deletes that uncertainty. When I release the key, I have declared that nothing more is coming. After key-up the future really is silence.

```mermaid
stateDiagram-v2
    [*] --> Recording: key down
    Recording --> Recording: mic buffers arrive, ASR streams
    Recording --> Flush: key up
    Flush --> Rewrite: final transcript
    Rewrite --> Paste: Smart Mode text ready
    Paste --> [*]
    note right of Flush
      Future is exactly silence.
      finish() already pads zeros.
    end note
    note right of Rewrite
      Depends only on the transcript.
      Mostly known before key-up.
    end note

```

FluidAudio's Nemotron path already does the paper's silence move by accident. When I release the key, `finish()` pads the leftover tail with zeros up to one chunk and flushes. That zero-padded chunk **is** the silence hypothesis, so for the ASR side of key-up there is nothing to invent.

Where the time actually goes is more interesting. In one logged Smart Mode dictation, key-up to paste took 1102 ms, and the LLM rewrite step took **841 ms** of it. Audio handling was 88 ms, speech recognition 170 ms and the paste 3 ms. In my 41-dictation Smart Mode benchmark (3 runs each, measured on 10 October 2026) the LLM step had a median of 203 ms and a p90 of 441 ms, so 841 ms is a long one, not a typical one. Smart Mode runs a small Qwen 2.5 1.5B model (4-bit) through llama.cpp, and my estimate, not a measurement, is that most of that time is spent generating output tokens rather than reading the prompt.

That is the real opportunity. The rewrite depends only on the transcript. By key-up I usually already have almost all of the transcript. So I can start the rewrite **during recording**, on the transcript so far, and keep it. At key-up:

- If the final transcript is byte-identical to the one I drafted from, paste the stored rewrite. The LLM left the key-up path completely, with zero loss.
- If the end changed, I pay for a check of the old rewrite plus the changed tail instead of a full regeneration.

For the exact-match path llama.cpp's server already has what I need. `llama-server` build b10107 reuses its KV cache for a growing prompt, accepts prefill-only requests with `n_predict: 0`, and can pin a slot. It does not accept a client-supplied draft, but byte equality of transcripts needs none. Voice-agent systems that start LLM work before input ends report real savings: PredGen reports about 2x lower latency in simulated experiments, Google reported about 200 ms from end-to-end prefetching, and Deepgram's eager end-of-turn trims 100-200 ms at the cost of 50-70% more LLM calls.

### The trap: real silence is not digital zeros

Exactness has one trap. The microphone samples between my last guess and the key-up are not digital zeros. Both of my transducer families have been reported to react to what silence looks like: Parakeet TDT v3 returned nothing after 0.4-0.6 s of leading silence (FluidAudio issue 746) and Nemotron multilingual dropped isolated short words (issue 838). Both issues are now closed upstream, but I would still not trust a "the tail looks quiet" check.

My proposed fix, which I have not tested, is to **canonicalize the tail**. At key-up the ASR input becomes the buffer cut at the last detected speech end plus a fixed pad P of silence. A guess made at a pause of at least P uses the same cut. The guessed input and the final input are then byte-identical whenever no speech followed the pause. The hit rate becomes the share of dictations where I paused at least P before releasing the key. The cost is a new ASR baseline, which has to tie the old one on word error rate before I trust it.

---

## What is the experiment plan for on-device dictation?

My research notes rank eight training-free experiments, R1 to R8, by milliseconds gained per unit of effort. Four gates run first, because they decide whether the ranked items are worth building:

| Gate | What it checks | Stop rule |
|---|---|---|
| G1 | Reproduce the Smart Mode 203 / 441 ms baseline on the 41 benchmark dictations | If it does not reproduce, fix the harness first. I am not sure the audio and key-up timestamps of those dictations were kept |
| G2 | Per-chunk Nemotron latency on the current macOS | FluidAudio issue 1116 reports 2.7x real time for the 2240 ms tier on a macOS 27.3 beta (first run, 9.2x on the second), against my 12-13 ms per chunk on 27.0.1. If the slow regime shows up, defer experiments that add encoder calls |
| G3 | `llama-bench` for Qwen 2.5 1.5B at 4-bit, plus the time to score a batch of 16-128 tokens | Calibrates the gain model for R1 and R4 |
| G4 | Interference between two models running together (ASR on the Neural Engine, Qwen on the GPU) | A slowdown above about 1.2x forces speculation to fire only at pauses |

```mermaid
quadrantChart
    title Experiments by effort and expected gain (my judgment)
    x-axis Low effort --> High effort
    y-axis Small gain --> Large gain
    quadrant-1 Plan carefully
    quadrant-2 Do first
    quadrant-3 Skip for now
    quadrant-4 Only if cheap
    R1 Smart Mode exact match: [0.2, 0.9]
    R2 Nemotron streaming: [0.15, 0.6]
    R3 Silence state: [0.3, 0.7]
    R4 Verify on misses: [0.55, 0.65]
    R5 Speculative final: [0.3, 0.2]
    R6 Silence LocalAgreement: [0.55, 0.3]
    R7 Speculative partials: [0.6, 0.4]
    R8 Transcribe 30B-A4B: [0.9, 0.5]

```

The placement is my own judgment from the cost and risk columns, not a measurement. The ranked list:

| Rank | Experiment | Expected gain | Cost and risk |
|---|---|---|---|
| R1 | **Smart Mode exact-match speculation.** Draft the rewrite at voice-activity pauses, paste it when the final transcript equals the guessed one | On hits the LLM leaves the key-up path: up to 841 ms in the logged example, 203 ms median | Low effort (TapTalk orchestration plus existing llama-server features). Unknown hit rate, tail sensitivity, GPU busy while recording, energy |
| R2 | **Stream Nemotron during recording** with the public API (`process` per mic buffer, `finish()` at key-up). This is not speculation | Key-up ASR from about 15.6 ms per audio second (about 156 ms at 10 s, derived) to roughly one padded chunk, an estimated 15-25 ms | Low. Streaming hypotheses may differ from batch ones, so I check for byte identity |
| R3 | **A silence state computed once**, restored at key-down as a cold-start prior | Accuracy at no run-time cost (numbers below) | Low to medium. Needs a small FluidAudio patch, or one extra encoder call per dictation in the public-API variant |
| R4 | **Verify the old rewrite on misses**, at temperature 0, plus incremental re-speculation | A miss costs one check pass plus the changed tail. With illustrative numbers (hit rate 0.5, 60 tokens at 10 ms), about 90 ms instead of about 600 ms | Medium. Needs a llama.cpp patch or a custom verifier. Moving from temperature 0.3 to 0 changes output quality, so it needs a side-by-side review |
| R5 | **Speculative ASR final** outside Smart Mode | At most 47/65 ms (Orukeet flush p50/p90), at most one chunk for Nemotron | Small gain, plus about 26.5 ms encoder cost per guess |
| R6 | **Silence-perturbation LocalAgreement** for Orukeet live text | Commit delay (median 0.70 s) down by up to one 0.5 s step | Medium, and about double the compute (94 to about 190 ms per audio second) |
| R7 | **Nemotron speculative partials**: literal speculative listening for live display | Display delay below the measured 1.31/2.20 s commit delay | Medium. Last word may flash then change; must stay out of committed text |
| R8 | **NVIDIA Transcribe 30B-A4B as a key-up verifier** | An estimate only: about 1.0-1.2 s per 11 s clip down to about 0.4-0.6 s | High. New runtime, about 21 GB, so it needs a 36 GB machine |

R8 is the closest literal match to the paper, because that model is a speech LLM and appending a draft to the audio prefill is exactly what speculative listening does. On a mixture-of-experts model, a prefill over about 140 audio tokens probably already touches most experts, so extra draft tokens should cost little while about 40 serial decode steps each re-read the active weights. That is an inference, and no Mac data for that model exists yet.

### The one that costs nothing: a silence state, and a suspicious zero

R3 started from a measurement I already had, and then a code-reading finding made it more interesting.

I measured that Nemotron 3.5 ASR drops speech that starts at the first sample. On Monsoon, a Hindi corpus, the first word was deleted on **44.9% of clips**. Putting silence in front of the audio changed that:

| Nemotron, 2240 ms tier | Svarah | EdAcc | Monsoon |
|---|---|---|---|
| No silence (TapTalk today) | 12.06% | 11.40% | 14.18% |
| 1.0 s of silence first | 11.49% | 10.84% | 13.54% |
| 2.24 s of silence first | not run | not run | 13.32% |

These are word error rates from my own runs (Monsoon 1929 errors in 13601 words, falling to 1811 in 13598 with 2.24 s). With 1 s of silence first, the first-word deletion rate on Monsoon fell from 44.9% to 19.3%, and on Svarah the number of clips under 1 s that came back empty fell from 704 to 508 of 1214. The 95% confidence interval for the 1 s Monsoon change is -1.13 to -0.16 points, so it is a real if modest gain.

The catch is that this pre-roll costs a model call at run time. The authors' port of speculative listening points at a cheaper route: **compute the silence state once and reuse it**. That is the same trick as their startup-time silence embedding.

Then reading FluidAudio's Nemotron buffer code turned up this: the first chunk's 9 pre-encode mel frames are seeded with **0.0 in log-mel space, while real silence sits around -16.6**. Padding with 0.0 tells the model the signal is far louder than silence, so that could be a cause of the first-word drop. I have **not tested this**. It is a reading of the code. The test costs no compute: seed the first mel cache with silence values and check the first-word deletion rate. If it works, the fix is free.

---

## What would speculative listening NOT help with?

My notes include a list of things not to build. A few are worth saying out loud because they are tempting:

| Do not | Why |
|---|---|
| Run silence drafts on ASR speech chunks | Acceptance roughly equals the share of silent chunks, which hold nothing to save (37.9% for Venus during speech, 35.7% in SSR) |
| Verify the previous TDT path in one batched joint call | Next to a measured dead end (my batched K-frame joint was slower), and the loop is only about 20 ms on an 11 s clip |
| Use n-gram or prompt-lookup drafts for the Smart Mode rewrite | Measured, no gain at temperature 0. Only the model's own earlier rewrite is a well-aligned draft |
| Use the Transcribe CTC head as a drafter | Measured Svarah 21.23% (Transcribe Lite, encoder plus CTC head), and a misspelled piece ends the accepted prefix |
| Accept a draft because the tail "looks quiet" | Both transducers are sensitive to silence, so use the canonical tail and check for exact identity |

---

## What is the honest status?

**I am working on this paper right now, and none of R1 to R8 has finished yet.** The analysis in this post is from reading, not from measuring, and the measured results will get their own write-up. Here is what is measured and what is not:

| Claim | Status |
|---|---|
| Smart Mode took 841 of 1102 ms in one logged dictation; LLM median 203 ms, p90 441 ms in the 41-dictation benchmark | Measured (earlier runs) |
| Silence pre-roll lowers WER: Monsoon 14.18% to 13.54% (1 s), 13.32% (2.24 s); Svarah 12.06% to 11.49%; EdAcc 11.40% to 10.84% | Measured |
| Nemotron key-up cost of about 15.6 ms per audio second | Measured input, derived totals |
| Streaming during recording cuts key-up to about one chunk | Estimate |
| Exact-match Smart Mode hit rate | Unknown. It decides everything for R1 |
| The 0.0 versus -16.6 mel seed causes first-word drops | Untested hypothesis from code reading |
| Transcribe 30B-A4B key-up time of 0.4-0.6 s | Estimate scaled from another model |

Two unknowns decide the ranking, and no literature can supply them: how often I pause before releasing the key, and how stable a Qwen rewrite stays as its input grows. G1 plus a replay of R1 measures both cheaply. If the hit rate is high, key-up to paste for Smart Mode should move toward the roughly 260 ms that is left in the logged example (audio 88, speech recognition 170, paste 3), and live typing and Smart Mode would stop being mutually exclusive, since the transcript I need during recording becomes the draft source. If the hit rate is low, the miss-path check in R4 still turns most of a miss into one batched pass, and R2 and R3 pay off either way because they rest on measured numbers.

I also should say what speculative listening itself never does: it never changes accuracy. It is lossless by construction. Any accuracy gain only appears if the latency it frees is spent on something better, such as a larger model or a longer context.

---

## FAQ

### What is speculative listening?

A training-free technique from the paper "Decode Before You Hear" (arXiv 2611.06982). A full-duplex voice model uses the time it would otherwise sit idle to guess its next output as if the next audio chunk were silence, then checks the guess in the same forward pass that reads the real audio. The output is identical to normal decoding.

### How much faster does speculative listening make voice models?

On an A100 the paper reports 2.6-2.8x faster speak chunks and a 73% shorter wait before the answer starts, because 95.3% of chunks come out the same under silence.

### Does speculative listening work for speech recognition?

Not while someone is talking. A transcript always depends on the new audio, and the model in the paper that reacts most to user speech accepted only 37.9% of guesses while the user spoke. It transfers exactly at push-to-talk key-up, where the future really is silence.

### Is the speculative listening code available?

Not yet. As of 7 November 2026 the authors' GitHub repository holds only a README, a license and images, and the README says the code and TickTock-Bench release are in preparation.

---

**Takeaway:** The lasting idea in "Decode Before You Hear" is not "decode under silence". It is: **find the part of the output that does not depend on input still to come, and compute it while you wait.** In a full-duplex model that part is almost everything, which is why the paper gets 2.6-2.8x and 73% faster starts. In speech recognition it is almost nothing, except the flush. In push-to-talk dictation it is the whole Smart Mode rewrite once the key is up. Key-up even deletes the paper's main source of misses, the listen-or-speak decision. I have not measured any of this yet. The first experiments are cheap: seed the mel cache with real silence, stream Nemotron during recording, and replay my 41 benchmark dictations against an exact-match rewrite.


---

## References

- [Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models (arXiv 2611.06982)](https://arxiv.org/abs/2611.06982), and the [HTML version](https://arxiv.org/html/2611.06982)
- [Speculative-Listening repository (README only as of 7 November 2026)](https://github.com/petrichor20211/Speculative-Listening)
- [Hugging Face paper page for 2611.06982](https://huggingface.co/papers/2611.06982)
- [Fast Inference from Transformers via Speculative Decoding (Leviathan et al.)](https://arxiv.org/abs/2211.17192)
- [Speculative Speech Recognition (arXiv 2407.04641)](https://arxiv.org/html/2407.04641v1)
- [PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction (arXiv 2506.15556)](https://arxiv.org/html/2506.15556v2)
- [Low latency speech recognition using end-to-end prefetching (Google Research)](https://research.google/pubs/low-latency-speech-recognition-using-end-to-end-prefetching/)
- [Deepgram Flux eager end-of-turn](https://developers.deepgram.com/docs/flux/voice-agent-eager-eot)
- [WhisperKit paper, including why speculative decoding was left out of the Turbo build (arXiv 2507.10860)](https://arxiv.org/pdf/2507.10860)
- [FluidAudio on GitHub](https://github.com/FluidInference/FluidAudio), issues [746](https://github.com/FluidInference/FluidAudio/issues/746), [838](https://github.com/FluidInference/FluidAudio/issues/838) and [1116](https://github.com/FluidInference/FluidAudio/issues/1116)
- [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
- My earlier posts: [Rebuilding TapTalk](https://www.dhruvvakharwala.dev/blogs/rebuilding-taptalk-parakeet-nemotron-live-typing), [Orukeet, Silero VAD and Core ML microphone](https://www.dhruvvakharwala.dev/blogs/taptalk-orukeet-silero-vad-core-ml-microphone), [FluidAudio zero-fill fix](https://www.dhruvvakharwala.dev/blogs/fluidaudio-mlmultiarray-zero-fill-pr-941)