Rebuilding TapTalk: Ripping Out Whisper for NVIDIA Parakeet, Live Typing, and On-Device Multilingual Dictation

The first version shipped on Whisper. This one doesn't.

When I wrote up the first TapTalk, the part I was proudest of was the inference engine: whisper.cpp running the encoder on the Neural Engine, with parameters hand-tuned to the exact chip. I detected M1 through M5 with sysctl, gated flash attention on chip family, matched the thread count to real performance cores, and sized the audio context to the length of what you actually said. It was a genuinely satisfying pile of engineering.

I deleted all of it.

Not because it was wrong - it worked, and the reasoning still holds. I deleted it because a better answer showed up. Whisper is a fine model, but for push-to-talk dictation there are now purpose-built recognizers that are faster, punctuate cleanly, stream token by token, and - crucially - come pre-converted to Core ML by people who do that conversion for a living. Once I leaned on those, my chip-tuning layer was solving a problem I no longer had.

So this is the honest version of a rebuild post: what I tore out, what replaced it, and the decisions I can defend with a measurement instead of a vibe. If you just want the app, the download is at the bottom, and it is a one-line brew install.

Here is the shape of what changed:

Deleting the part I was most proud of

The old architecture put recognition in Rust. Rust drove whisper.cpp, whisper.cpp drove Core ML, and I spent real effort tuning the layer in between. The new architecture moves recognition into Swift entirely, and the reasoning is in the repo's architecture.md in one blunt sentence:

Recognition is not Rust work, which is less obvious. Parakeet and Nemotron are Core ML models executing on the Neural Engine; the host language is a thin shim over Apple's accelerator either way. Running them from Swift removes a layer of indirection and gets first-class Core ML APIs. An earlier version ran whisper.cpp inside the Rust core and was removed for exactly this reason.

That is the whole trade. When the model is a .mlmodelc bundle running on the ANE, the host language is not doing the compute - Apple's runtime is. Calling it from Rust meant an FFI hop and a C++ engine in the middle purely so I could say the core owned inference. Calling it from Swift gets me Core ML's own APIs with nothing between me and the accelerator, and it lets me hand model loading, chunking, and format handling to a library built for exactly this.

So the split moved. Rust still owns everything that has to be fast and deterministic on a real-time thread. Swift owns everything that talks to Apple's frameworks, and recognition turned out to be one of those things.

The Rust FFI surface got smaller as a result. It exports a Recorder (warm up, start, stop, plus level and chunk callbacks), a ModelManager for the LLM download only, and the optional OpenAI cloud path. That is it. Every #[uniffi::export] still lives in a single lib.rs, so the entire boundary is reviewable in one file - and because the Swift bindings are generated at build time and not committed, deleting a Rust export shows up as a Swift compile error on the next build. The compiler is the completeness check.

The Swift side got the same discipline, which matters more now that there are several engines instead of one. Every recognizer - Parakeet, Nemotron, the streaming one - is an actor conforming to a single contract (isInstalled, download, ensureLoaded, unload, transcribe, plus a couple of download-hygiene helpers), so nothing that calls an engine cares which engine it is holding. Adding a new one means touching nine places: the LocalEngine enum and its exhaustive switches, the plan that decides which engine to load, both halves of applying that plan, the release path, the download-residue sweep, the transcribe dispatch, and a catalog card. The exhaustive switches will not compile until every one is updated - the same trick as the Rust boundary, on the other side of it. On both sides of the bridge, the compiler refuses to build a half-added engine.

pub fn stop(&self) -> Result<RecordingResult, CoreError> {
    let samples = self.inner.stop().map_err(|msg| CoreError::Audio { msg })?;
    let trimmed = audio::trim_silence(&samples).map_err(|msg| CoreError::Audio { msg })?;

    // Too short to be a real utterance - return empty so the UI shows
    // "Too short - hold longer" instead of a hallucinated transcript.
    if trimmed.len() < audio::MIN_SPEECH_SAMPLES {
        return Ok(RecordingResult { samples: Vec::new(), sample_count: 0, duration_secs: 0.0 });
    }

    // Lift quiet/murmured speech toward conversational loudness - helps every engine.
    let mut processed = trimmed;
    let _gain = audio::apply_agc(&mut processed);
    // ... return processed samples to Swift
}

That short-clip guard is new and it earns its keep: a push-to-talk key gets tapped by accident constantly, and handing 80 ms of silence to a recognizer is an invitation to hallucinate a word. Below ~300 ms of real speech, TapTalk returns nothing and the pill says "Too short - hold longer" rather than pasting an invented transcript.

Two engines, and a vocabulary that decides which languages are even possible

The default engine is NVIDIA Parakeet TDT 0.6B v3 (~490 MB). It is fast, it punctuates, it auto-detects the language across English and about two dozen European ones, and it streams - which is the property that makes live typing possible at all. For most people dictating in English, it is the whole story.

The second engine is NVIDIA Nemotron 3.5 ASR Multilingual 0.6B (~640 MB), and it exists because Parakeet has a failure mode that is worse than not working. Parakeet cannot produce Devanagari. Ask it to transcribe Hindi and it does not fail loudly - it emits confident romanized nonsense. A recognizer that is wrong quietly is more dangerous than one that refuses, so Hindi and everything non-European routes to Nemotron instead.

Nemotron was not a guess. I measured candidates on 418 Hindi clips from FLEURS:

Nemotron won on accuracy, and it cost no new dependency - it ships inside the same FluidAudio I was already using for Parakeet, and it needed no change to the deployment target.

But the detail I did not expect, the one worth the whole section, is this: a model's advertised language list is not the same as the languages it can physically emit. I counted the Unicode blocks in Nemotron's 13,087-token vocabulary. Devanagari has 196 tokens, Arabic 252, CJK 6,907, Kana 217, Latin 2,567. And Bengali, Gujarati, Tamil, Telugu, Kannada, Malayalam, and Punjabi have zero. Not slow, not inaccurate - impossible. There is no token the decoder could emit to write those scripts, no matter how clearly you speak.

So the language picker in TapTalk is not the model's brochure list. It is restricted to the scripts the vocabulary actually contains, measured from the model itself:

// Languages this model can actually produce, with the exact prompt keys it recognizes.
// Script coverage comes from counting Unicode blocks in its 13,087-token vocabulary...
// Bengali, Gurmukhi, Gujarati, Tamil, Telugu, Kannada and Malayalam have *zero*, so
// those are omitted rather than offered and silently emitted as garbage.
nonisolated static let supportedLanguages: [(id: String?, label: String)] = [
    (nil, "Auto-detect"),
    ("en", "English"), ("hi", "Hindi"), ("mr-IN", "Marathi"), ("ur-PK", "Urdu"),
    ("es", "Spanish"), ("fr", "French"), ("de", "German"),
    ("ja-JP", "Japanese"), ("zh-CN", "Chinese"),
]

Offering a language you cannot render is a worse experience than not offering it, because the user speaks, gets garbage, and blames themselves. Counting tokens is a boring afternoon that removes a whole class of that.

Hinglish: romanizing Devanagari without a rulebook

Here is a very specific thing a lot of people who type in Hindi actually do: they speak Hindi, but they want it written in the Roman alphabet, the way Hindi chat looks in practice. "main kal aaunga", not the Devanagari. TapTalk does this, and getting it right taught me something about small models.

The obvious approach is rule-based transliteration. There is a mature library for it - ICU - and it is deterministic and fast. It also produces kaiphe for cafe and mitinga for meeting. It destroys exactly the English loanwords that make code-switched Hinglish readable, because it transliterates the sounds without knowing that some of those sounds were English words wearing a Devanagari costume. Deterministic and wrong is still wrong.

So romanization runs through the local LLM instead, and the prompt is few-shot rather than descriptive. I tried describing the task in prose; the model translated to English, dropped words, and changed the verb person. Shown five worked pairs - each mapping a Devanagari sentence to its chat-Roman form - it became stable and correct. The examples also teach the two things a rulebook cannot: drop the inherent schwa (so the Devanagari for "kal" romanizes to kal, not kala) and leave English loanwords in English (so "meeting" stays meeting, never mitinga).

The part that actually gets tricky in production is deciding when to romanize, and it is a five-condition gate where each condition rules out a real bug:

static func shouldRomanize(...) -> Bool {
    guard script == .roman,               // the user asked for Roman output
          transcriptionEngine == .local,  // cloud is a different model with its own output
          localEngine == .nemotron,       // Parakeet can't produce Devanagari anyway
          language == hindiLanguageCode    // <- the one that bites
    else { return false }
    return containsDevanagari(text)        // English dictation costs nothing even with Roman on
}

The check that actually matters is language == "hi". Marathi is also written in Devanagari, so a script check alone would happily romanize Marathi using a prompt whose five examples are entirely Hindi - and produce confident wrong output. The transliteration is language-specific, not script-specific, and only the language check catches that.

Live typing: words on the screen while you are still talking

The biggest new feature you can feel is live typing. Turn it on and text appears in the focused field as you speak, instead of all at once on release. It runs on a third model - NVIDIA Parakeet Realtime EOU 120M (~440 MB), a separate optional download - streaming 320 ms chunks at roughly 5.7% WER and 14x real time. The Rust audio callback, which already exists, just forwards chunks to a streaming sink; a Swift actor drains them in arrival order and emits partial transcripts.

The interesting engineering is not the model, it is putting the text on screen without it flickering or fighting the app you are typing into. Streaming means the recognizer revises its own tail - "wreck a nice beach" becomes "recognize speech" - so the inserter has to reconcile what is on screen with the latest hypothesis, not just append. It diffs the common prefix, deletes the changed tail, and types the new one:

private func reconcile(to target: String) {
    let prefixCount = commonPrefixCount(inserted, target)
    let deleteCount = inserted.count - prefixCount
    let newTail = String(target.dropFirst(prefixCount))

    // Try the Accessibility range-replace fast path first (flicker-free, single undo).
    if useAX, let element = focusedElementIfStill() {
        if axReplaceTail(element: element, deleteCount: deleteCount, with: newTail) {
            inserted = target
            return
        }
        useAX = false   // first failure -> drop to pasteboard for the rest of the session
    }

    if deleteCount > 0 { backspace(deleteCount) }
    if !newTail.isEmpty { pasteInsert(newTail) }
    inserted = target
}

Two macOS realities shaped this. First, the clean path - selecting the tail via kAXSelectedTextRange and overwriting it via kAXSelectedText - is flicker-free and a single undo, but it does not work everywhere. So when it fails once, I stop trying it for the session and fall back to pasteboard-plus-Cmd-V, which works everywhere paste works. Why paste and not synthesized keystrokes? Because Chromium-based apps - VS Code, Slack, Discord, most Electron - silently drop CGEvent Unicode keystrokes, since those events carry virtualKey = 0 and Chromium ignores them. Paste is the only universally accepted insertion method on macOS.

Second, focus tracking has an Electron trap. The naive check is "did the focused element's process change?" But Electron apps run each window in a helper process, so the focused element's PID jitters between updates and would falsely trip a focus-lost bail-out mid-sentence. TapTalk tracks the frontmost application's PID instead, which is stable per app, so it keeps typing into VS Code without deciding VS Code left.

And because every insert goes through the pasteboard, each one is stamped transient and concealed so clipboard managers - including macOS 26's built-in Clipboard History and tools like Maccy or Paste - skip it. Your dictation does not pollute your clipboard history, and whatever you had copied is restored when the session ends. Live typing is mutually exclusive with Smart Mode, for the obvious reason: a rewrite needs the whole transcript, and streaming does not have it yet.

Smart Mode grew up, then learned to do one thing at a time

Smart Mode is the second hotkey - hold it instead of the plain one and the transcript runs through the local LLM (Qwen 2.5 1.5B via a pinned llama.cpp) before it lands. In v1 it had one job: read the frontmost app and rewrite to fit. Now it has three composable modes - Polish (strip fillers, fix grammar, keep your wording), Restructure (honor mid-sentence self-corrections, drop what you retracted), and Match the app (rewrite for wherever you are typing) - plus the Hinglish romanization above.

The context it reads got sharper too. Matching the app is not just the app name; it reads the focused window title, because that is often the only way to tell Gmail from YouTube inside the same browser. The Accessibility read carries a 250 ms timeout, because an AX request to a hung app otherwise blocks forever - and this is on the paste path, where a hang is a bug you feel. Browsers themselves are not matched against a hardcoded list; TapTalk asks Launch Services which apps are registered to open https, so Arc, Dia, Zen, Orion, and whatever ships next month all resolve as browsers without me maintaining a list.

But the real lesson was about the model, and it surprised me. The three cleanup modes refuse to stack. I wanted "polish AND match-the-app" to compose. Measured against the shipped 1.5B model, any cleanup clause placed beside the destination clause caused the model to ignore the destination entirely - it anchored on "tidy this text" and echoed the dictation instead of rewriting it. Reproduced 3 out of 3 with Polish, 3 out of 3 with Restructure, 3 out of 3 with the destination clause moved first. It is a capability limit of a small model following a multi-part instruction, not a wording problem.

So the modes have a precedence order instead of being concatenated - romanization outranks match-the-app, which outranks polish and restructure - and only the winning clause is sent:

if romanize { return romanizeClause }         // romanization outranks everything

var parts: [String] = [base]
if options.contains(.smart) {
    parts.append(smartClause(context))        // match-the-app supersedes the cleanup modes
} else {
    if options.contains(.polish) { parts.append(polishClause) }
    if options.contains(.restructure) { parts.append(restructureClause) }
}
parts.append(closing)

Almost nothing is lost by the precedence, because the modes overlap: rewriting a rambling dictation into an email inherently drops the fillers and honors the self-corrections. Two findings held across everything I tried with this size of model. Few-shot beats prose, decisively. And it does one job, or none - give it one instruction and it lands, give it two and they cancel out.

One more small thing that is invisible but matters: the prompt is assembled in a fixed order with all the variable content (the app name, the window title) last, so the stable prefix is as long as possible and the server-side cache can reuse it across dictations. Moving the app context earlier would bust the cache on every window-title change.

The pipeline, end to end

Here is one full pass, from the key going down to the text appearing. It is worth seeing whole, because a few decisions only make sense in context.

There is no network call in that diagram, and that is the point. The one exception is the optional OpenAI cloud engine, which is off by default and only runs if you turn it on and provide your own key. A LatencyTrace records each stage and emits one line per dictation, so I can see where the time actually goes instead of guessing:

log stream --predicate 'subsystem == "talk.tap.app" && category == "latency"'
# keyup->paste total=1102ms audio=88 asr=170 llm=841 paste=3

That breakdown is honest about where the cost is: with Smart Mode on, the LLM rewrite dominates. Plain dictation - no LLM - is the asr=170 path plus a few milliseconds of paste, which is why it feels instant.

One reliability detail sits underneath all of this. The CGEvent tap that watches the hotkey can be silently disabled by macOS - taps die on callback timeouts and across sleep and wake - and a dictation app whose hotkey quietly stops responding is one you uninstall. So a health check re-enables the tap if it finds it inert, and rebuilds it from scratch if re-enabling does not take. It is defensive code for a failure that is invisible until it happens to you mid-sentence.

The unglamorous parts of the audio path

Recognition changed, but the audio path in front of it got quietly better too.

Gain for quiet speakers. New in the Rust core is a small AGC pass that lifts murmured or quiet dictation toward broadcast loudness (-23 dBFS), bypasses anything already at a conversational level, caps the boost at +20 dB so near-silence is not blown up into noise, and limits peaks. Two linear passes, effectively free, and it helps every engine because they were all trained on audio at a sane level.

let mut gain = (TARGET_RMS / rms).min(MAX_GAIN);
let peak = samples.iter().fold(0.0f32, |m, &s| m.max(s.abs()));
if peak > 0.0 && peak * gain > PEAK_LIMIT {
    gain = PEAK_LIMIT / peak;   // don't let the boost clip
}

A VAD threshold tuned down, not up. Silence trimming still uses Silero VAD, but the confidence threshold dropped from Silero's default 0.5 to 0.35, deliberately, to catch murmured dictation that the default treats as silence. Padding went to six chunks (~190 ms) on each side so soft word onsets survive the trim. And the window size, 512 samples, is not a knob - Silero v5 requires exactly that at 16 kHz. I know because I tried larger windows: 3x faster, and they clipped up to 2976 ms of opening speech on 34 of 59 real clips while returning perfectly plausible probabilities. The crate accepts them without complaint. That is the kind of "optimization" that passes every test and ruins the product.

Models that let go of memory. An idle dictation tool should not squat on the Neural Engine. TapTalk releases loaded models after 30 minutes idle - but the timer is only a backstop. The real signal is a memory-pressure handler that drops everything the instant the system asks for the RAM:

let source = DispatchSource.makeMemoryPressureSource(eventMask: [.warning, .critical], queue: .main)
source.setEventHandler { [weak self] in
    self?.releaseIdleEngines()
    LlamaServerManager.shared.stop()
}

A timer alone gets this backwards: it releases when RAM is free and holds when it is scarce. Tying the release to actual pressure means a long idle hold never competes with whatever you are really doing. It helps that Core ML weights are memory-mapped - the app sits around 164 MB resident with Parakeet loaded, against ~490 MB on disk. And releasing is not quite as simple as dropping a reference: a Core ML model held inside an engine actor outlives the Swift handle, so unload() has to call the model's cleanup() explicitly. Drop the reference alone and the Neural Engine keeps the weights loaded anyway.

Switching engines without a race. Loading a model is async, and a user flipping between engines in Settings can supersede a load that is still in flight - and the first, now-stale load must not win over the second. So every load bumps a generation counter, and each async continuation re-checks that it is still the current generation before it commits its result. Loads and unloads also chain through a single task, so an unload from a discarded plan can never land after the next load has already finished. It is one generation-guard pattern reused everywhere it is needed, rather than a new ad-hoc guard invented at each call site - which is how these races usually slip in.

What I measured and rejected

The most useful thing I can leave in the repo is the list of things that looked good and were not:

• Apple Foundation Models for the rewrite. Works, handles Hindi, costs zero disk. But a reused session accumulates its transcript, so dictation N would see dictation N-1's text; a fresh session per dictation is required, and that costs 1121 ms against llama.cpp's 513 ms. It also threw guardrailViolation on ordinary text.
• llama.cpp speculative decoding. The theory was a 2-3x win, since rewriting mostly copies its input. Measured at temperature 0 with a fixed seed and identical token counts: no gain, and 3% slower on the copy-heavy case it was supposed to help. An earlier "20% win" at temperature 0.3 was just variable output length, not speed.
• A smaller rewrite model. A 0.6B would be ~2.5x faster than the 1.5B, but the 1.5B already fabricates occasionally - it once invented a closing sentence that was never dictated. The right thing to measure before shrinking is the fabrication rate, not the latency.
• Rule-based transliteration. Covered above: kaiphe and mitinga. Deterministic and wrong.

None of these are failures. They are the measurements that make the shipped choices defensible, and writing them down is cheaper than earning them twice.

What is still on the list

I want to be straight about the edges, same as last time.

• Not notarized yet. Gatekeeper still grumbles on first launch. The Homebrew cask handles it for you by clearing the quarantine flag on install, but a manual DMG needs a xattr command or a right-click Open. Notarization is paperwork, and it is coming.
• Benchmarks are per-model, not exhaustive. I have WER numbers for the ASR choices and latency traces for the pipeline, but not a full latency-per-chip-per-model matrix. That harness would let me make precise claims and catch regressions instead of trusting perceived latency.
• Downloads do not resume. A failed model transfer restarts at byte zero. It streams to a .partial file and renames on completion so a crash never leaves a file that looks whole, but resume is a real feature that is not there yet.
• Apple Silicon only. The whole performance story is the Neural Engine. The Rust core is portable enough that a CPU-only path is possible; it just was not the point.

Conclusion

The lesson from this rebuild is the one that is easy to say and hard to do: be willing to delete your favorite code. My chip-tuned Whisper engine was good work, and it was the right thing to remove the moment a better-fitting set of models existed with the hard Core ML conversion already done. What replaced it is less code on my side and a better tool: faster recognition, a language picker restricted to the scripts the model can actually emit, and text that appears in the field as you speak. The privacy story never changed, because it was never a feature to add - it was the default the whole thing was built to keep.

Download

TapTalk is free, open source, and on-device by default. macOS 14 or later, any Apple Silicon Mac.

• Homebrew (recommended):

brew install --cask vakharwalad23/tap/taptalk

The cask downloads the DMG and clears the Gatekeeper quarantine flag for you, so it launches on the first try.

• Website: taptalk.dhruvvakharwala.dev - what it does, in one page.
• Download the app: TapTalk Releases - pre-built DMG for Apple Silicon.
• Product Hunt: TapTalk on Product Hunt - if it has been useful, an upvote helps a lot.
• Source: github.com/vakharwalad23/tap-talk - Rust core, SwiftUI app, MIT. Stars and issues welcome.

Building from source is still one command:

git clone https://github.com/vakharwalad23/tap-talk.git
cd tap-talk
make run

If a manual DMG install is flagged as "damaged" on first launch, that is Gatekeeper being cautious about an unnotarized build - clear the quarantine flag:

xattr -dr com.apple.quarantine "/Applications/TapTalk.app"

References

• TapTalk website - the product page
• TapTalk on GitHub - the full source
• TapTalk on Product Hunt - the launch
• The first TapTalk post - the Whisper-based v1 this rebuilds
• FluidAudio - the Core ML speech runtime for Parakeet and Nemotron
• NVIDIA Parakeet TDT 0.6B v3 - the default ASR model
• FLEURS - the multilingual speech benchmark used for the Hindi WER numbers
• Silero VAD - the voice activity detector
• llama.cpp - the local LLM runtime for Smart Mode
• Qwen 2.5 - the 1.5B rewrite model
• UniFFI - the Rust-to-Swift bindings generator