TapTalk Gets Faster: Orukeet, Silero VAD on Core ML, and the Microphone That Would Not Let Go

Where the last post left off

When I rebuilt TapTalk around NVIDIA Parakeet and Nemotron, recognition moved into Swift and the Rust core shrank to audio, voice activity, gain and downloads. I said at the time that Rust owned "everything that has to be fast and deterministic on a real-time thread." That sentence aged badly, and this post is mostly about why.

Three things happened in the five releases since then (0.4.0 to 0.5.2). A better model showed up. The audio path turned out to be the slowest part of the key-up path, and it was slow for a reason Rust could not fix. And the microphone, the one piece of hardware a dictation app cannot work without, produced three separate bugs that real users hit.

If you only want the app, the download is at the bottom and it is still a one-line brew install. Here is the shape of what changed:

Orukeet: a better Parakeet, from oruk.ai

Oruk builds speech understanding models, and Orukeet is their 25-language recognizer built on top of NVIDIA Parakeet TDT 0.6B v3. The idea is surprisingly small. Half of the encoder's temporal depthwise filters are replaced with 12,288 fitted, frozen Gabor kernels - classic signal-processing filters with a fixed shape instead of learned weights - and the remaining parameters are trained on multilingual and multi-accent data. Everything else stays: same architecture, same TDT decoder, same tokenizer. I checked the vocabulary and it is byte-identical to Parakeet v3, all 8,192 tokens.

That last detail decided where it fits in TapTalk. Same 25 European languages, no Devanagari, so Orukeet replaces Parakeet as the default for English and European dictation, and Nemotron still handles Hindi and everything else.

Oruk's own model card reports Orukeet beating Parakeet on 61 of 74 test splits, with FLEURS pooled WER of 9.85% against 11.01%. Those are numbers from their training stack, though, and I do not ship on a model card. What I care about is the model running through Core ML on a Mac, decoded the way TapTalk decodes. So I ran the full FLEURS test set myself: all 25 languages, 20,146 clips, both models on the same on-device path.

That is a 15.6% relative reduction in word errors, and it reproduces the paper's 23-of-25 result. It only loses French and German. The biggest gains are on the languages where Parakeet struggles most - Latvian improves by 8 points of WER, Maltese by 7. The absolute numbers run higher than Oruk's because my number normalization is lighter and Core ML decodes in int8 instead of NeMo; the relative result holds.

One thing I want to flag, because I nearly fooled myself with it. My first comparison was a 60-clip sample across six languages, and it said the two models were tied. It was wrong. The sample missed exactly the higher-error languages where Orukeet gains the most. A small benchmark that agrees with your prior is the most dangerous kind, and the full run is the only number I cite now.

The Core ML conversion, twice

Orukeet ships as NeMo, ONNX and GGUF, but not Core ML. So I converted it myself into the bundle layout FluidAudio expects (Preprocessor, Encoder, Decoder, JointDecisionv3), and it worked - but it ran about 14 ms slower than Parakeet. Not because of the architecture; my int8 encoder simply compressed worse than FluidInference's own Parakeet conversion.

Then I met Nathan Roll, Orukeet's creator at Oruk, and he did the conversion properly. His version quantizes 699 inference tensors to a 6-bit lookup table with FP16, which closed the size gap (about 467 MB), and adds a greedy profile that drops a top-64 joint calculation TapTalk never uses, since auto-detect never re-ranks hypotheses. Same transcripts on 128 FLEURS clips, 9.5% faster on his M5 Max. That is the bundle TapTalk downloads.

Measured on my M3 Pro with the in-app latency trace, here is what the switch costs:

So Orukeet is not just more accurate, it is a touch faster, and it has no cold-start penalty because it preloads at launch. I would have shipped it at equal speed for the accuracy alone. I would not have shipped it slower - a change that regresses key-up to paste does not land in TapTalk, whatever else it fixes.

Compiled on your Mac, and recompiled when macOS changes

Nathan's bundle ships as .mlpackage files rather than precompiled .mlmodelc, because a compiled Core ML model is specific to the OS and Core ML version that built it. So TapTalk compiles Orukeet on your Mac once, after the download, and stamps the result with the OS build. After a macOS upgrade the stamp no longer matches and it quietly recompiles, without re-downloading 467 MB:

private static func currentOSStamp() -> String { ProcessInfo.processInfo.operatingSystemVersionString }

// compilePackages is not idempotent, so start clean.
nonisolated static func compile() throws {
    try? fm.removeItem(at: compiledDir())
    try OrukeetLocalModels.compilePackages(from: packagesDir(), to: compiledDir())
    try currentOSStamp().write(to: stampFile(), atomically: true, encoding: .utf8)
}

That is why the Orukeet installer has a distinct "compiling" phase that the other engines do not. The download itself is checked before anything is unpacked: TapTalk reads a manifest from the same immutable Hugging Face revision as the archive and verifies the filename, the pinned SHA-256 and the byte count, so a truncated or altered archive is rejected instead of installed. Nathan contributed that check, along with AsrModels.loadLocal in FluidAudio, which lets TapTalk open the bundle with FluidAudio's own loader instead of assembling the models by hand.

Switching the default engine on existing users is the part where it is easy to be rude. New installs get Orukeet. Existing Parakeet users see a one-time, dismissible banner that downloads and compiles Orukeet and then switches to it. Parakeet is never removed - it stays the fallback, and it is the only engine the live-typing model can drive, because there is no streaming Orukeet. So anyone already using live typing is never offered the switch at all. Taking away a feature someone uses to give them a better number is not an upgrade.

One licensing note, because it is a first for this stack: Orukeet is CC-BY-SA-4.0, a ShareAlike license. TapTalk stays MIT because it never bundles the weights. They download at runtime, from the publisher, to Application Support - the same rule every other model in TapTalk already follows.

A 24 ms fix that went upstream

While profiling Orukeet I found something that had nothing to do with Orukeet. After every transcription, FluidAudio handed its 240,000-sample input buffer back to a cache and cleared it, one element at a time, boxing each zero as an NSNumber. That cost about 20 ms per dictation, spent writing zeros after the models had already finished. The decoder state was copied the same slow way.

I sent the fix upstream as FluidAudio#941: bulk memset and memcpy over contiguous storage. The review made it better twice - the maintainer pointed out that the clear on return was dead work, since the next user overwrites the whole buffer anyway, so it is now gone entirely; and that a bulk write over a padded Core ML array could run past its allocation, so the fast path only applies to contiguous storage. Measured through Orukeet on an 11-second clip, transcribe went from 72.6 ms to 48.2 ms with identical transcripts. It shipped in FluidAudio 0.17.0, and TapTalk picked it up in 0.5.0.

Moving audio out of Rust, and Silero onto Core ML

Here is the problem the first two posts did not have a number for. In the Rust core, recording worked like this: capture with cpal while the key is down, then on key-up resample the whole clip to 16 kHz and run Silero VAD over all of it through ONNX Runtime to find where the speech starts and ends. Only then could recognition begin.

That design puts the entire voice-activity pass on the key-up path, which is exactly where the user is waiting. Silero cost about 2.5 ms per second of audio, plus the full-clip resample, plus about 100 ms extra on the first dictation of a process while ONNX Runtime warmed up. For a 17 to 21 second dictation the stop step alone took 53 to 66 ms, and the audio stage as a whole ranged from 59 to 189 ms. None of that was Rust being slow. It was the work happening at the wrong time.

The fix was to do the work while you are still talking, and once I was rewriting it, the same argument that moved recognition to Swift applied to audio too: the code that talks to Apple's audio hardware belongs next to Apple's audio APIs. The Rust core now only downloads the LLM and runs the optional cloud path. Everything else lives in a small Swift package, TapTalkAudio, with its own tests.

The real-time thread does almost nothing

Core Audio calls your input block on a real-time thread with a hard deadline. Allocate, take a lock, or even retain an object there and you risk a glitch. So the sink node's block does one thing: downmix the frames into a ring buffer that was allocated once, up front.

let ring = SampleRing(capacity: Int(format.sampleRate * Self.ringSeconds))
let writer = ring.writer
let node = AVAudioSinkNode { _, frames, list in
    writer.writeDownmix(list, frames: Int(frames))
    return noErr
}

The ring is a single-producer, single-consumer buffer of raw pointers, so the real-time side never touches reference counting. The producer publishes how many samples it has written with a release store, and the consumer reads it with an acquire load. Swift's Atomic type needs macOS 15 and TapTalk supports 14, so those two operations are a few lines of C in their own module. If the consumer ever falls behind, the writer drops samples and counts them rather than block, and the pipeline logs the count. With four seconds of capacity and a drain every 32 ms, it only fills if the consumer stalls for seconds.

Silero runs while you speak

The consumer is an actor, RecordingPipeline, that wakes every 32 ms while you are recording. It drains whatever arrived, sends the level to the floating pill, converts to 16 kHz with AVAudioConverter, and scores every complete 512-sample window with Silero as soon as it exists:

while (scoredWindows + 1) * size <= all.count {
    let start = scoredWindows * size
    let probability = try detector.speechProbability(
        window: UnsafeBufferPointer(rebasing: all[start..<(start + size)]))
    span.record(window: scoredWindows, probability: probability)
    scoredWindows += 1
}

By the time you release the key, every window but the last one is already scored. Key-up pauses the engine, converts and scores the final ~32 ms, trims to the speech span with the same six windows of padding as before, and applies gain to quiet speech. On those same 17 to 21 second clips that stop step now takes 0.7 to 1.2 ms, against 53 to 66 ms for the Rust stage it replaced.

The Silero model is FluidInference's 32 ms Core ML conversion of Silero v6. It is 0.9 MB, downloaded at first launch from a pinned Hugging Face revision and checked against SHA-256 before use. It is a stateful model - an LSTM - so the wrapper carries the hidden state, the cell state and the last 64 samples of context from one window into the next, exactly the way the reference implementation does:

let output = try model.prediction(from: input)
// ...
Self.copy(newHidden, into: hidden)
Self.copy(newCell, into: cell)
// The last 64 samples of this window become the next window's context.
audio.withUnsafeMutableBufferPointer(ofType: Float.self) { buffer, _ in
    guard let base = buffer.baseAddress else { return }
    base.update(from: base + SpeechTrim.windowSize, count: Self.contextSize)
}

Get that context handoff wrong and the probabilities still look plausible, which is the worst way for a voice detector to be broken. The thresholds did not move either - 512-sample windows, a 0.35 threshold, six windows of padding, 300 ms minimum - so transcripts are unchanged. Only when the work happens changed.

The Neural Engine was not the reason

Here is the part I did not expect. I assumed "Silero on Core ML" meant "Silero on the Neural Engine." It does not. Asking Core ML for its compute plan shows every Silero op placed on the CPU, under every compute-unit setting, and the timings are identical either way. So TapTalk loads it with .cpuOnly, which skips compiling for the Neural Engine at load (18 ms instead of 43 ms) for zero runtime cost.

The speedup over the Rust version came from two other things: Core ML's CPU path is faster than ONNX Runtime's for this model, and the work moved off the key-up path entirely. Neither has anything to do with the accelerator. Writing that down matters, because "we moved it to Core ML for the ANE" is the story you would tell if you did not check.

The numbers

Measured on an M3 Pro, Release build, live dictation:

The remaining 8 ms median is mostly AVAudioEngine.pause() itself, about 7 ms. ONNX Runtime is gone from the app completely, which is where most of the size went.

Live typing got two fixes in the same release. The last word is no longer cut off when you release the key - finishing a session now closes the input stream and waits for the streaming model to drain before it reads the final text, instead of racing it. And switching live typing on in Settings warms the model right away, so the first live dictation starts immediately instead of a few seconds late.

The microphone that would not let go

A dictation app has one job before any model runs: open the microphone when you press the key, and let go of it when you release. It turned out I had three separate bugs there.

Bug one: the orange dot that stayed on

The symptom was easy to see once you looked: the moment TapTalk launched, macOS's orange microphone indicator came on and stayed on. Worse, AirPods started sounding muffled - "dumpy" is the word that stuck - and stayed that way until the first dictation.

In the first post I wrote this about my persistent-stream trick: "the stream is paused, not destroyed, between recordings, so the OS mic indicator behaves - but it's the kind of thing I want to keep an eye on across macOS releases." It was the right thing to keep an eye on, for the wrong reason.

The root cause was in cpal 0.15.3, the Rust audio library: building an input stream also starts it, inside build_input_stream. And TapTalk's warm_up() built the stream at launch, so that the first dictation would not pay for the microphone permission check. The intent was to prepare the microphone. The effect was to open it, from launch until the first dictation paused it.

The muffled AirPods are the same bug seen from the other side. Bluetooth headphones have two modes: a high-quality one for listening, and a headset mode that also carries the microphone at much lower quality. Whenever anything on the Mac records from them, macOS switches them to headset mode. A microphone held open by an app in the menu bar means your music sounds like a phone call all day.

The Swift rewrite fixed it by construction. AVAudioEngine.prepare() builds the whole graph at launch without starting it. start() runs on key-down, and pause() on key-up releases the device. To prove it rather than assume it, I wrote a small monitor on Core Audio's process objects (macOS 14.2+ exposes which process is using input, through kAudioHardwarePropertyProcessObjectList and kAudioProcessPropertyIsRunningInput). With 0.5.1: no microphone at launch, none while idle, held only while the hotkey is down, released at key-up. One amusing find from the monitor: whenever any app records, Siri's com.apple.CoreSpeech joins and lingers about a second after. That is not TapTalk.

I also measured the obvious "safer" alternative - fully stopping the engine after every dictation. It costs 48 ms more to first audio on the next press (98 ms against 50 ms), and building the graph lazily on key-down costs 106 ms more. Since pausing already releases the microphone, both would make every dictation slower for no benefit.

Bug two: a crash after waking from sleep

Shortly after 0.5.0 shipped, a user reported that TapTalk quit the first time they pressed the hotkey after their Mac woke from sleep. They sent logs, and the question I always ask first is whether it is my bug or something odd on their machine. It was mine.

The crash was an Objective-C exception from AVAudioEngine: Input HW format and tap format not matching. Swift cannot catch Objective-C exceptions, so it does not throw an error - it aborts the app.

Here is what happens. TapTalk prepares the audio graph once and keeps it, rebuilding only when the audio configuration changes. Sleep and wake is exactly when input devices change: AirPods reconnect, an iPhone used as a microphone disappears, a USB or display microphone re-enumerates. The rebuild connected the input node using its output format, and after a device change the output side of the input node still describes the old device. Connecting the new device with the old device's format is what raised the exception.

I reproduced it before touching any code. A sample-rate flip on the same device does not trigger it, because AVAudioEngine converts internally. What does is the device itself going away: I created a public aggregate device of the built-in microphone, made it the default input, let TapTalk prepare against it, then destroyed it while idle. Next key press, crash, every time.

The fix has two parts. Wire the graph with the input node's hardware format, which describes whatever device is actually connected:

// The hardware side: after a device change the output side keeps the old device's format, and connecting with it throws.
let format = input.inputFormat(forBus: 0)

And never let AVAudioEngine take the app down again. A five-line Objective-C target catches any exception from graph wiring and hands it back to Swift as a value:

NSException *_Nullable tt_catch_exception(NS_NOESCAPE void (^block)(void)) {
    @try {
        block();
        return nil;
    } @catch (NSException *exception) {
        return exception;
    }
}

Attaching, connecting and preparing the graph all run inside it. If macOS ever refuses the audio setup again, TapTalk shows an error, marks the graph for rebuild, and the next key press tries again on a fresh graph instead of quitting. Version 0.5.1 shipped the next day with the full root cause written up in the issue.

Bug three is a feature: use the built-in microphone

Fixing the stuck microphone left one honest problem. Even with the microphone held only while you hold the key, if your AirPods are the Mac's input, dictating still switches them to headset mode for the length of the dictation, and macOS takes a moment to switch them back. If you dictate while listening to something, every dictation makes it sound worse for a few seconds.

So 0.5.2 adds Settings > Microphone > Use the built-in microphone. Turn it on and TapTalk records from the Mac's own microphone, so nothing ever records from the headphones and they stay in their high-quality mode. Laptop microphones are good, and you are sitting in front of it anyway.

Choosing a microphone with AVAudioEngine is not a property on the engine. The input node wraps a Core Audio output unit, and you point that unit at a device:

static func device(
    for preference: InputPreference, among devices: [InputDeviceInfo], systemDefault: AudioDeviceID
) -> AudioDeviceID? {
    if preference == .builtIn, let builtIn = devices.first(where: \.isBuiltIn) {
        return builtIn.id
    }
    return systemDefault == kAudioObjectUnknown ? nil : systemDefault
}

// The input node records from one Core Audio device; pointing it at the chosen one is what selects the microphone.
private func select(_ device: AudioDeviceID, on input: AVAudioInputNode) {
    guard let unit = input.audioUnit else { return }
    var id = device
    let status = AudioUnitSetProperty(
        unit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Global, 0, &id,
        UInt32(MemoryLayout<AudioDeviceID>.size))
    // ...
}

"Built-in" is identified by the device's transport type, kAudioDeviceTransportTypeBuiltIn, not by its name, which is localized and differs between Macs. The device is selected on every graph build, before the hardware format is read - so the crash fix from bug two still sees the right device. Changing the setting only marks the graph for rebuild; it applies on the next key press and never opens the microphone early. Selecting the device does not force a rebuild on every press either: a repeat start takes 74 ms to first audio, against about 135 ms right after a switch.

Mac mini, Mac Studio and Mac Pro have no built-in microphone, so on those the option is simply not shown. An option that cannot work is worse than no option.

Verifying this one needed a trick. Core Audio's per-process device lists name the physical device under an aggregate, so they could not tell me which path was recording. Instead I made an aggregate device the system default and watched whether the aggregate itself was running (kAudioDevicePropertyDeviceIsRunningSomewhere). Setting off: the aggregate runs while recording. Setting on: it never does, and the built-in microphone records. Flipping it back and forth on a live engine followed every change, and nothing was held after key-up.

The pipeline, end to end

Here is one full pass again, because almost every box changed since the last diagram.

The Rust core is no longer in that diagram at all. It still exists - it downloads the LLM for Smart Mode and runs the optional OpenAI cloud path - but plain dictation never crosses the FFI boundary anymore.

What I measured and rejected

Same as last time, the most useful thing I can leave behind is the list of things that looked good and were not:

• FluidAudio's 256 ms Silero model. It is 2.7x cheaper per second of audio than the 32 ms model, and it is FluidAudio's default. But it merges eight frames with a noisy-OR, which scored plain background noise at 0.56 to 0.73. At thresholds up to 0.7 it left the leading silence untrimmed on two of three real clips, and trimming properly needed about 0.85, which drops murmured speech. The 32 ms model matches the old trim within one window.
• Silero on the Neural Engine. Covered above: Core ML places every op on the CPU anyway. The win was never the ANE.
• Fully stopping the engine after each dictation. +48 ms to first audio, no benefit, because pausing already releases the microphone.
• My own Core ML conversion of Orukeet. It worked and was 14 ms slower than it needed to be. Nathan's 6-bit conversion with the greedy profile replaced it.
• Moving the Smart Mode LLM to Core ML. With all this Core ML work, I tested whether the rewrite model should follow. Same model, two runtimes: Gemma 3 1B took 1073 ms on the Neural Engine against 231 ms with llama.cpp on the GPU, and Qwen 3.5 0.8B took 4.3 seconds against 0.2, because the Core ML build processes the prompt one token at a time. I also re-measured Apple Foundation Models on macOS 27: much better than before (573 ms, no guardrail errors, and by far the best at Hindi romanization), but still almost three times slower than the shipped Qwen 2.5 1.5B at 203 ms. The Neural Engine is wonderful for speech models. It is the wrong place for a small LLM that has to answer in 200 ms.

What is still on the list

• Not notarized yet. Same as before. The Homebrew cask clears the quarantine flag for you; a manual DMG needs xattr or a right-click Open.
• Older chips. Every number in this post is from an M3 Pro. The real test of the latency work is an M1, and I want the same key-up-to-paste traces from the oldest and newest Apple Silicon before I call it done.
• No streaming Orukeet. Live typing still runs on Parakeet's streaming model. A streaming Orukeet would let the more accurate engine type as you speak too, and it is the next thing I am looking at with Oruk.
• Downloads still do not resume. A failed model download starts again from zero.

Conclusion

The thread through all of this is timing. The Silero rewrite did not make voice detection smarter - same model, same windows, same thresholds. It made it happen while you are talking instead of while you are waiting, and that alone took a third off key-up to paste. The microphone bug was the same mistake in reverse: work meant to happen early (preparing the microphone) that quietly became work that never stopped (holding it open). And Orukeet is a reminder that a better model is sometimes a small change - a set of frozen Gabor filters on an architecture I already ran - as long as you measure it on the path your users actually take, with enough data to not fool yourself.

Download

TapTalk is free, open source, and on-device by default. macOS 14 or later, any Apple Silicon Mac. The current version is 0.5.2.

• Homebrew (recommended):

brew install --cask vakharwalad23/tap/taptalk

The cask downloads the DMG and clears the Gatekeeper quarantine flag for you, so it launches on the first try. Older versions stay installable as pinned casks, for example [email protected].

• Website: thetaptalk.com - what it does, in one page.
• Download the app: TapTalk Releases - pre-built DMG for Apple Silicon.
• Product Hunt: TapTalk on Product Hunt - if it has been useful, an upvote helps a lot.
• Source: github.com/vakharwalad23/tap-talk - SwiftUI app, Swift audio package, small Rust core, MIT. Stars and issues welcome.

Building from source is still one command:

git clone https://github.com/vakharwalad23/tap-talk.git
cd tap-talk
make run

If a manual DMG install is flagged as "damaged" on first launch, that is Gatekeeper being cautious about an unnotarized build - clear the quarantine flag:

xattr -dr com.apple.quarantine "/Applications/TapTalk.app"

References

• TapTalk website - the product page
• TapTalk on GitHub - the full source
• The first TapTalk post - the Whisper-based v1
• The Parakeet rebuild - Parakeet, Nemotron, live typing
• oruk.ai - Oruk, the speech models behind Orukeet
• Orukeet on Hugging Face - model card, weights and the Core ML bundle
• Orukeet Core ML conversion - Nathan Roll's conversion and parity benchmark
• FluidAudio - the Core ML speech runtime
• FluidAudio#941 - the decoder buffer fix
• FluidAudio#928 - AsrModels.loadLocal, by Nathan Roll
• Silero VAD - the voice activity detector
• Issue #22 - the crash after wake, root cause and fix
• FLEURS - the multilingual benchmark behind the WER numbers
• NVIDIA Parakeet TDT 0.6B v3 - the model Orukeet builds on
• llama.cpp - the local LLM runtime for Smart Mode