Speculative Listening Explained: What a Full-Duplex Voice Paper Means for On-Device Speech Recognition

◆ IMG800×600

What is speculative listening, and why did I read a voice-agent paper?

I build a dictation app called TapTalk. You hold a key, talk, release, and text appears in whatever app has focus. The whole product is one number: the time from key-up to text. I have written about rebuilding it around NVIDIA Parakeet and Nemotron, about moving the microphone and voice activity detection into Swift, and about a 20 ms bug I fixed in FluidAudio. Every one of those posts is about shaving milliseconds off that wait.

So when a paper claims a 73% cut in response delay for speech models without changing the output, I read it. It is about a different problem. The paper targets full-duplex voice assistants that listen and talk at the same time, not speech recognition. My question was narrow: which parts of this trick survive when the model is a transcriber and the user is a person holding a key?

This post goes through the paper in order, then through my own analysis. I did the analysis by reading the paper, the authors' public repository, FluidAudio's source and my own earlier measurements. The paper is by Yutong Bian and eight co-authors (Yanglin Zhang, Yuxiang Wang, Luoyuan Zhang, Rongshen He, Zihao Wang, Junwen Qiu, Jiaqi Li and Zhizheng Wu) from The Chinese University of Hong Kong, Shenzhen and Amphion Technology. Its title is "Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models."

How do full-duplex voice models work, and why is the GPU idle?

A normal voice assistant is turn-based. You speak, a voice activity detector decides you are done, the model thinks, then it answers. That feels like a walkie-talkie. A full-duplex model keeps listening while it talks, so it can be interrupted, can say "mm-hm", and can decide on its own when you have finished a thought.

The paper studies one family of these: chunk-interleaved models. Audio arrives in fixed 1 second chunks. For each chunk the model reads the audio, then decides one of three things:

• keep listening and output nothing,
• start speaking (or writing a text channel),
• call a tool, such as a search or a calendar lookup.

Reading a second of audio is cheap. A "listen" chunk needs one forward pass over the new audio, which takes the GPU a fraction of the second. The paper measures that the GPU sits idle for 70-89% of the chunk duration on listen chunks. Then the user stops talking, the model decides to answer, and the work flips. Generating an answer is serial: one token per forward pass, each waiting for the last. A tool call takes tens of passes.

That imbalance is the problem. Under plain autoregressive (AR) decoding the paper measures 417 to 1089 ms of extra delay before a response starts, depending on the model. Human turn gaps are about 200 ms, so the assistant feels slow. A tool call finishes 1.8-2.6 s after its audio arrives, which misses the 1 s chunk deadline every time. One of the four models, LWS, misses the deadline on 95% of its speak chunks.

◆ IMG800×600

Here is the same idea as a schedule. The blue bars are real work, the grey stretches are the idle gap, and the dark blue region in the second row is where speculation goes.

The widths here are schematic, not the paper's measurements. The point is the shape: the idle stretches in the top lane are exactly what the bottom lane spends on guesses.

What is speculative decoding, and why is it lossless?

Speculative listening is a variation on an older idea, so the lineage matters.

A language model generates one token at a time because each token depends on the one before it. But checking a sequence of tokens is not serial. If someone hands you a guess for the next five tokens, one forward pass over those five positions tells you what the model would have chosen at each of them. That is how a transformer trains, and it is how it scores.

Speculative decoding (Leviathan et al. and Chen et al., 2022-2023) uses that asymmetry:

1. A cheap draft produces k guessed tokens. Usually that is a small model.
2. The big model runs one forward pass over all k guesses and computes its own choice at every position.
3. Keep the guesses up to the first position where the big model disagrees, then take the big model's own token there.
4. Repeat from the new position.

The result is lossless for greedy decoding. Every kept token is, by construction, the token the big model would have picked anyway. A wrong guess costs a little wasted compute, never a wrong word. For sampling at a temperature, the standard accept/reject rule keeps the output distribution identical to the big model's.

A worked example of the acceptance math

Say the draft is right on any single token with probability p = 0.8, independently, and you draft k = 4 tokens. A pass keeps tokens until the first miss and always gains one token from the verifier. The expected tokens per pass is the geometric sum:

E[tokens per pass] = (1 - p^(k+1)) / (1 - p)
                   = (1 - 0.8^5) / (1 - 0.8)
                   = (1 - 0.32768) / 0.2
                   = 3.36

So instead of 1.0 token per big-model pass you get about 3.4. The speedup is smaller than 3.4x because the draft costs time, but the big model's passes drop by that factor. If p falls to 0.5 with the same k, the same formula gives 1.94 tokens per pass. Acceptance rate is everything.

The usual drafts are a small separate model, or n-gram lookups that copy phrases already seen in the prompt (Prompt Lookup Decoding), or a recycled pool of earlier guesses (Token Recycling, Lookahead decoding). Those are the baselines in the paper. They need no training, but they only start after the audio has arrived.

How does speculative listening work, step by step?

The paper's twist is when the drafting happens. Normal speculative decoding drafts after the input arrives. Speculative listening drafts before the input arrives, using the idle window, and it uses the model itself as the draft model. The authors call this input-time self-speculative decoding.

The assumption is a hypothesis about the future: the next chunk will be digital silence. Under that assumption the model can run ahead.

Step by step:

1. Silence stand-in. The model's audio encoder embeds one chunk of digital silence. The authors' port of the method reportedly encodes this once at startup and reuses it for every session.
2. Draft under silence. After chunk t-1, the model decodes its own greedy continuation as if chunk t were that silence. It stops at the end-of-chunk token or when the idle time budget is spent.
3. Roll back. The draft tokens are stored, and the KV cache (the model's working memory of the conversation) is restored to where it was before the guess.
4. Verify during prefill. When the real chunk arrives, the model has to run a forward pass over the new audio anyway. The draft tokens ride along in the same pass. That one pass reads the audio and scores every guess. A correct draft costs zero extra forward passes.
5. Accept the prefix. Tokens are kept up to the first mismatch with the model's true argmax, plus one corrected token. The output is identical to plain greedy decoding, and the post-audio passes can never exceed what autoregressive decoding needed.

◆ IMG800×600

Two extras for the misses

Pure silence drafts fail when the world changes, so the paper adds two things:

• Runner-up branch. If the primary draft finishes before the idle window ends, a second draft starts from the model's second-best first token. This hedges the most common miss, which is the listen-or-speak decision.
• Cross-chunk hypothesis pool. Every draft from earlier chunks is stored and indexed by n-gram, as in lookahead decoding. When the model starts a tool call, an earlier chunk may already have drafted most of it.

Why does the silence guess work so well?

The paper first ran an oracle probe: decode each chunk twice, once with the real audio and once with silence, and compare. Two observations:

Most chunks are listen chunks, where the model outputs "keep listening" whatever the next second sounds like. When the output changes, it almost always changes at token one, the decision to start talking. That is why a runner-up branch on the first token catches so much.

What are the paper's results?

The authors built TickTock-Bench: 8 tasks, 760 sessions, 15,883 one-second chunks. The tasks cover factual QA (Web Questions, Llama Questions, TriviaQA), open answers (AlpacaEval), multi-step reasoning (Big Bench Audio), turn-taking and overlapping speech (Full-Duplex-Bench v1 and v1.5), and tool calls (v3). The main results ran on one A100, one session at a time, bf16, greedy decoding, batch size 1, and only the text backbone (the "Thinker") was timed, not speech synthesis.

Four models were tested, all with 1 s chunks: the authors' own Streaming SLM (trained from the turn-based Qwen2.5-Omni-7B Thinker), plus Gander, Realtime-Venus and LWS, all three built on MiniCPM-o.

Speedup on speak chunks

The model-free baselines barely beat 1x, because they start after the audio is in and have nothing to copy. Tool calls are the dramatic case: 23.41x for Streaming SLM and 3.45x for Gander.

Where the tool-call gain comes from

The ablation separates the components (Llama Questions for speak, tool-call benchmark for tool):

For speak chunks the silence draft does nearly all of the work. For tool calls the pool does: a tool call is mostly already drafted in earlier chunks, so a pool that remembers earlier drafts turns 1.42x into 22.04x.

How much sooner does the answer start?

On average response onset drops 73%, to 74-241 ms. Tool-call latency drops by 72-96% (2623 to 112 ms for Streaming SLM). The maximum lag behind real time per session falls by 94.4% on average. On the tool-call benchmark, Streaming SLM went from 35% of sessions lagging under AR to 0%.

Quality did not drop

An AlpacaEval judge score (1 to 5) came out the same or marginally higher with speculative listening: 3.10 vs 3.13, 3.56 vs 3.63, 2.98 vs 3.00, 3.02 vs 3.07. In float32 the outputs match AR token for token on nearly all sessions. In bf16, rounding can change some outputs, which the paper treats as a numerics effect, not a flaw in the method.

Where the gains erode

The authors are open about three limits, and one more shows up in their own tables:

More sessions on a GPU shrink the idle window, and more sampling randomness lowers acceptance. The method is aimed at low-concurrency, latency-sensitive serving. It also does not cover parallel-stream models such as Moshi, which decode every 80 ms and would need a different formulation, and the paper times only the text backbone.

Where does speculative listening break? When the output depends on the audio

Everything above rests on one fact: in these models, what a chunk produces barely depends on that chunk's audio. Look at the acceptance table by chunk type (the paper's Table 7: Llama Questions, 1,423 chunks per model, with tool calls from the tool-call benchmark).

Three of the four models produce nearly the same output with or without the next second of user audio. Realtime-Venus is the one that reacts most to speech, and its acceptance while the user talks collapses to 37.9%. It is also the model with the weakest onset gain in the table (565 to 241 ms, 57%, against 70-82% for the others).

Two more caveats from my reading, flagged as mine:

• The question-answering sessions are mostly silence. The repository README describes a QA session as 0.5 s of silence, the question, then about 10 s of silence. For most of those chunks the silence hypothesis is literally true.
• The "token one" finding leans on one model. Weighting the per-model rates by chunk count (my own derivation, not a number the authors state) puts roughly 238 of about 270 mismatched chunks on Realtime-Venus.

The paper reports no energy figures and no accounting of the extra compute the drafts use, which matters a lot for a laptop. As of 7 November 2026 the authors' GitHub repository holds only a README, a license and images, and the README says "Code and TickTock-Bench release in preparation." A work-in-progress port to sglang-omni exists on a branch of a separate repository, and that is where the encode-silence-once detail comes from.

None of this undermines the paper. It tells me where the method is strong: a model whose output is mostly a function of its own plan.

Does speculative listening work for speech recognition?

ASR is the opposite case. A transcriber's output at chunk t is the transcription of chunk t. If the next second contains the words "send it Friday", a silence draft predicts "no new words" and fails at token one, which is exactly where the paper's misses already pile up.

There is outside evidence for how hard guessing the future of speech is. Speculative Speech Recognition (SSR, arXiv 2407.04641) used an audio-prefixed language model to guess upcoming words, and even when the best of 8 guesses counts, it retrieved only 35.7% of suffixes up to one second ahead correctly.

There is a second wall: the encoder. A streaming ASR model runs an audio encoder over each chunk, and the encoder output depends on the real audio. No check is cheaper than running the encoder. So a draft can only save decoder work, and in the transducer models TapTalk runs (RNN-T and TDT style), decoder work is small. In my own profile of an 11 s Orukeet clip the joint network loop was about 20 ms (roughly 120 calls at 0.17 ms) against a 27 ms encoder. Argmax's WhisperKit paper likewise left speculative decoding out of its Whisper Large v3 Turbo build, because drafter overhead cut the practical speedup to 1.25x.

So mid-utterance, the literal method is out. Here is my table of the cases in TapTalk:

Where does the idea hold exactly? Push-to-talk key-up

The paper's weakest point is guessing whether the user will start talking. Push-to-talk deletes that uncertainty. When I release the key, I have declared that nothing more is coming. After key-up the future really is silence.

FluidAudio's Nemotron path already does the paper's silence move by accident. When I release the key, finish() pads the leftover tail with zeros up to one chunk and flushes. That zero-padded chunk is the silence hypothesis, so for the ASR side of key-up there is nothing to invent.

Where the time actually goes is more interesting. In one logged Smart Mode dictation, key-up to paste took 1102 ms, and the LLM rewrite step took 841 ms of it. Audio handling was 88 ms, speech recognition 170 ms and the paste 3 ms. In my 41-dictation Smart Mode benchmark (3 runs each, measured on 10 October 2026) the LLM step had a median of 203 ms and a p90 of 441 ms, so 841 ms is a long one, not a typical one. Smart Mode runs a small Qwen 2.5 1.5B model (4-bit) through llama.cpp, and my estimate, not a measurement, is that most of that time is spent generating output tokens rather than reading the prompt.

That is the real opportunity. The rewrite depends only on the transcript. By key-up I usually already have almost all of the transcript. So I can start the rewrite during recording, on the transcript so far, and keep it. At key-up:

• If the final transcript is byte-identical to the one I drafted from, paste the stored rewrite. The LLM left the key-up path completely, with zero loss.
• If the end changed, I pay for a check of the old rewrite plus the changed tail instead of a full regeneration.

For the exact-match path llama.cpp's server already has what I need. llama-server build b10107 reuses its KV cache for a growing prompt, accepts prefill-only requests with n_predict: 0, and can pin a slot. It does not accept a client-supplied draft, but byte equality of transcripts needs none. Voice-agent systems that start LLM work before input ends report real savings: PredGen reports about 2x lower latency in simulated experiments, Google reported about 200 ms from end-to-end prefetching, and Deepgram's eager end-of-turn trims 100-200 ms at the cost of 50-70% more LLM calls.

The trap: real silence is not digital zeros

Exactness has one trap. The microphone samples between my last guess and the key-up are not digital zeros. Both of my transducer families have been reported to react to what silence looks like: Parakeet TDT v3 returned nothing after 0.4-0.6 s of leading silence (FluidAudio issue 746) and Nemotron multilingual dropped isolated short words (issue 838). Both issues are now closed upstream, but I would still not trust a "the tail looks quiet" check.

My proposed fix, which I have not tested, is to canonicalize the tail. At key-up the ASR input becomes the buffer cut at the last detected speech end plus a fixed pad P of silence. A guess made at a pause of at least P uses the same cut. The guessed input and the final input are then byte-identical whenever no speech followed the pause. The hit rate becomes the share of dictations where I paused at least P before releasing the key. The cost is a new ASR baseline, which has to tie the old one on word error rate before I trust it.

What is the experiment plan for on-device dictation?

My research notes rank eight training-free experiments, R1 to R8, by milliseconds gained per unit of effort. Four gates run first, because they decide whether the ranked items are worth building:

The placement is my own judgment from the cost and risk columns, not a measurement. The ranked list:

R8 is the closest literal match to the paper, because that model is a speech LLM and appending a draft to the audio prefill is exactly what speculative listening does. On a mixture-of-experts model, a prefill over about 140 audio tokens probably already touches most experts, so extra draft tokens should cost little while about 40 serial decode steps each re-read the active weights. That is an inference, and no Mac data for that model exists yet.

The one that costs nothing: a silence state, and a suspicious zero

R3 started from a measurement I already had, and then a code-reading finding made it more interesting.

I measured that Nemotron 3.5 ASR drops speech that starts at the first sample. On Monsoon, a Hindi corpus, the first word was deleted on 44.9% of clips. Putting silence in front of the audio changed that:

These are word error rates from my own runs (Monsoon 1929 errors in 13601 words, falling to 1811 in 13598 with 2.24 s). With 1 s of silence first, the first-word deletion rate on Monsoon fell from 44.9% to 19.3%, and on Svarah the number of clips under 1 s that came back empty fell from 704 to 508 of 1214. The 95% confidence interval for the 1 s Monsoon change is -1.13 to -0.16 points, so it is a real if modest gain.

The catch is that this pre-roll costs a model call at run time. The authors' port of speculative listening points at a cheaper route: compute the silence state once and reuse it. That is the same trick as their startup-time silence embedding.

Then reading FluidAudio's Nemotron buffer code turned up this: the first chunk's 9 pre-encode mel frames are seeded with 0.0 in log-mel space, while real silence sits around -16.6. Padding with 0.0 tells the model the signal is far louder than silence, so that could be a cause of the first-word drop. I have not tested this. It is a reading of the code. The test costs no compute: seed the first mel cache with silence values and check the first-word deletion rate. If it works, the fix is free.

What would speculative listening NOT help with?

My notes include a list of things not to build. A few are worth saying out loud because they are tempting:

What is the honest status?

I am working on this paper right now, and none of R1 to R8 has finished yet. The analysis in this post is from reading, not from measuring, and the measured results will get their own write-up. Here is what is measured and what is not:

Two unknowns decide the ranking, and no literature can supply them: how often I pause before releasing the key, and how stable a Qwen rewrite stays as its input grows. G1 plus a replay of R1 measures both cheaply. If the hit rate is high, key-up to paste for Smart Mode should move toward the roughly 260 ms that is left in the logged example (audio 88, speech recognition 170, paste 3), and live typing and Smart Mode would stop being mutually exclusive, since the transcript I need during recording becomes the draft source. If the hit rate is low, the miss-path check in R4 still turns most of a miss into one batched pass, and R2 and R3 pay off either way because they rest on measured numbers.

I also should say what speculative listening itself never does: it never changes accuracy. It is lossless by construction. Any accuracy gain only appears if the latency it frees is spent on something better, such as a larger model or a longer context.

FAQ

What is speculative listening?

A training-free technique from the paper "Decode Before You Hear" (arXiv 2611.06982). A full-duplex voice model uses the time it would otherwise sit idle to guess its next output as if the next audio chunk were silence, then checks the guess in the same forward pass that reads the real audio. The output is identical to normal decoding.

How much faster does speculative listening make voice models?

On an A100 the paper reports 2.6-2.8x faster speak chunks and a 73% shorter wait before the answer starts, because 95.3% of chunks come out the same under silence.

Does speculative listening work for speech recognition?

Not while someone is talking. A transcript always depends on the new audio, and the model in the paper that reacts most to user speech accepted only 37.9% of guesses while the user spoke. It transfers exactly at push-to-talk key-up, where the future really is silence.

Is the speculative listening code available?

Not yet. As of 7 November 2026 the authors' GitHub repository holds only a README, a license and images, and the README says the code and TickTock-Bench release are in preparation.

References

• Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models (arXiv 2611.06982), and the HTML version
• Speculative-Listening repository (README only as of 7 November 2026)
• Hugging Face paper page for 2611.06982
• Fast Inference from Transformers via Speculative Decoding (Leviathan et al.)
• Speculative Speech Recognition (arXiv 2407.04641)
• PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction (arXiv 2506.15556)
• Low latency speech recognition using end-to-end prefetching (Google Research)
• Deepgram Flux eager end-of-turn
• WhisperKit paper, including why speculative decoding was left out of the Turbo build (arXiv 2507.10860)
• FluidAudio on GitHub, issues 746, 838 and 1116
• llama.cpp server documentation
• My earlier posts: Rebuilding TapTalk, Orukeet, Silero VAD and Core ML microphone, FluidAudio zero-fill fix