20 Milliseconds of Writing Zeros: Profiling FluidAudio and Fixing It Upstream

The number that did not belong

In the last TapTalk post I wrote about switching the default engine to Orukeet and moving audio into Swift. Both were about one number: the time from releasing the hotkey to text appearing in your app. Every millisecond on that path is one the user waits for.

Recognition is the biggest piece of it, and TapTalk does not run recognition itself. It hands 16 kHz audio to FluidAudio, which runs Parakeet-style models (Parakeet, and Orukeet, which shares its architecture) through Core ML. So when I wanted to know where recognition time went, guessing was not good enough. I wrote a small profiler, ttprof, that times each stage inside one AsrManager.transcribe call, and ran it on an 11-second clip (the JFK "ask not" fixture from Orukeet's repository) on my M3 Pro.

Look at that second row. The encoder is the bulk of a 600-million-parameter model running on dedicated silicon, and it takes 26.7 ms. Right behind it, at 20.5 ms, is writing zeros into memory. A modern Mac can clear a buffer of that size in a few microseconds. Something was taking roughly three thousand times longer than it should.

A loop that boxes every zero

FluidAudio keeps a small cache of MLMultiArrays so it does not allocate a fresh input buffer for every transcription. When a transcription finishes, the buffer goes back to the cache, and the cache cleared it first:

func returnArray(_ array: MLMultiArray) {
    // ...
    if arrays.count < maxCacheSize / max(cache.count, 1) {
        array.resetData(to: 0)
        arrays.append(array)
        cache[key] = arrays
    }
}

And resetData was this:

func resetData(to value: NSNumber) {
    for i in 0..<count {
        self[i] = value
    }
}

Two details make that expensive. The buffer is the preprocessor input, shape [1, 240000] in float32 - fifteen seconds of 16 kHz audio, so 240,000 elements. And MLMultiArray's subscript is an Objective-C API that takes and returns NSNumber. Every one of those 240,000 writes is a dynamic message send, a check of the array's data type, an unboxing of the number and a store, at around 90 nanoseconds each. In a release build that adds up to 21.5 ms. In a debug build it is 42.9 ms.

The timing made it worse. returnArray runs on the way out of transcribe, after the encoder and decoder are done, so the user is waiting for the transcript while the library writes zeros into a buffer nobody is looking at.

There was a second loop just like it. copyData(from:) copied element by element through the same subscript, and TdtDecoderState(from:) uses it to copy the decoder's LSTM state, which since an earlier change runs before every recoverable inference. Smaller buffers, same pattern.

The first fix: memset

The obvious fix is to stop going through the subscript and write the backing memory directly. MLMultiArray exposes it through withUnsafeMutableBytes, so the first version of the PR did three things:

• Zero for any data type is one memset over the array's bytes.
• Other values fill through a typed pointer for float32, float64 and int32, and other types keep the element loop.
• copyData(from:) is one memcpy when shape, data type and strides match, and keeps the loop otherwise.

The numbers were what you would expect:

Transcripts were identical. I added tests, including two timing budgets that fail on main (42.9 ms and 62.9 ms against a 5 ms budget), ran the full FluidAudio suite with model downloads - 2,534 tests - and opened the PR.

I also wrote this in the description, as a feature: "withUnsafeMutableBytes reports the full backing extent, so the ANE-aligned arrays with padded strides are covered too." That sentence was the bug.

What review found

A fast PR is easy. The review is where it became a correct one, in three rounds.

Round one: overlapping memory

The automated Copilot review flagged the copy path first. memcpy has undefined behavior when the source and destination overlap, and that can happen here: destination.copyData(from: destination) is a valid call, and FluidAudio creates zero-copy views that share one allocation. The old element loop was slow but safe in that case. My "optimization" turned a safe copy into undefined behavior. The fix is memmove semantics, which in Swift is what UnsafeMutableRawBufferPointer.copyMemory(from:) gives you.

Round two: the maintainer's review

Then FluidAudio's maintainer (@Alex-Wengg) reviewed it, and the blocking issue was the sentence I was proud of.

Core ML arrays meant for the Neural Engine are often padded: a [10, 10] array can have strides of [16, 1], so each row of ten values sits in sixteen slots. withUnsafeMutableBytes reports the whole strided span - stride times rows times element size - not ten times ten elements. Every array FluidAudio allocated itself had that full span, so the bulk write was in bounds. But ANEMemoryOptimizer.createZeroCopyView builds a view over someone else's storage, pads the view's innermost stride, and bounds-checked it by element count. A padded view over a tighter source passes that check, and then a bulk memset over its "full extent" writes past the end of the source allocation. My fast path would have turned a quiet layout quirk into a memory overrun.

The fix came in two parts. The bulk paths only run when the storage is exactly contiguous - the byte span equals the element count times the element size - and padded layouts keep the element loop, so nothing past the last element is ever written. And the root cause got fixed too: the zero-copy view now bounds its padded span, not its element count.

// The view pads its innermost stride, so its storage span can exceed its element count;
// bound the span, not the count, or a padded view would run past a tighter source.
let strides = calculateOptimalStrides(for: shape, dataType: sourceArray.dataType)
let viewSpan = shape.isEmpty ? 0 : strides[0].intValue * shape[0].intValue
let sourceSpan =
    sourceArray.shape.isEmpty ? 0 : sourceArray.strides[0].intValue * sourceArray.shape[0].intValue

guard offset + viewSpan <= sourceSpan else {
    throw DiarizerError.invalidArrayBounds
}

The review also caught that one of my tests could not fail. The padded-stride test dirtied only the logical elements and then checked for zeros, but the allocator already zero-clears the padding, so reverting to the old loop still passed. It now poisons the entire backing store with NaN first, and new sentinel tests check that the slots after the last element - and between rows - are never touched. And my 5 ms timing budgets would have been flaky in FluidAudio's parallel CI, where the timed region includes an actor hop on a loaded machine. They still run locally and skip under CI.

Then the non-blocking notes, and one of them was better than my whole PR.

The best optimization was deleting the call

The maintainer pointed out that the clear on return was dead work. The only code that takes an array from that cache is in AsrManager, and it immediately copies the full extent of new audio into it. Short recordings are zero-padded before that, elsewhere. So the zeros written on return were always overwritten before anyone read them.

I had made a useless operation 3,500 times faster. The right move was to stop doing it:

/// Return an array to the cache for reuse. Its contents are kept: every consumer of `getArray`
/// overwrites the full extent before use, so clearing here would be wasted work.
func returnArray(_ array: MLMultiArray) {
    // ...
    if arrays.count < maxCacheSize / max(cache.count, 1) {
        arrays.append(array)
        cache[key] = arrays
    }
}

The cost on that path went from 20 ms to 6 microseconds with memset, and then to nothing. The faster helpers still earn their place, because they serve everything else: the decoder state copy, warm-up, and the other callers of the fill helper.

The review noticed one more thing: FluidAudio had three different fill helpers. One in the decoder, one in ModelWarmup built on vDSP_vfill, and an older reset(to:) extension with fourteen callers that only covered the logical count and silently did nothing for data types other than float32 and int32. They are now one pair of helpers in Shared, reset(to:) delegates to it (which fixed its padded-array gap for free), and the vDSP version is gone. I also added a note that -0.0 takes the zero path and lands as +0.0, and a float64 test.

Round three: overlapping views with different layouts

Nathan Roll from Oruk, who builds Orukeet and reviewed the fix with me, sent one more case. The bulk copy was now overlap-safe, but the element-by-element fallback - used when layouts differ - read and wrote in the same pass. Two overlapping views with different layouts could read a value the loop had already overwritten. His test went red first (0.0 where it expected 1.0); the fix is one line that reads the whole source before writing anything. That fallback is not reached on any hot path, so the extra allocation costs nothing that matters.

Here is the merged helper, all of it:

/// Fills every element with `value`.
///
/// Contiguous storage fills in bulk: zero is one `memset` for every data type, other values
/// fill through a typed pointer for float32, float64 and int32. Padded strides and other data
/// types fill element by element, so nothing past the last element is ever written. `value` is
/// compared as an `NSNumber`, so `-0.0` takes the zero path and lands as `+0.0`.
func resetData(to value: NSNumber) {
    let elementSize = ANEMemoryUtils.getElementSize(for: dataType)
    let filled = withUnsafeMutableBytes { bytes, _ -> Bool in
        guard bytes.count == count * elementSize, let base = bytes.baseAddress else {
            return false
        }
        if value == 0 {
            memset(base, 0, bytes.count)
            return true
        }
        switch dataType {
        case .float32:
            bytes.bindMemory(to: Float.self).update(repeating: value.floatValue)
        case .float64:
            bytes.bindMemory(to: Double.self).update(repeating: value.doubleValue)
        case .int32:
            bytes.bindMemory(to: Int32.self).update(repeating: value.int32Value)
        default:
            return false
        }
        return true
    }
    if filled {
        return
    }
    for i in 0..<count {
        self[i] = value
    }
}

/// Copies every element from `source`.
///
/// Identical contiguous layouts copy the storage in bulk; that copy is overlap-safe, so two
/// views of one allocation may overlap. Any other pair reads the whole source before writing,
/// so overlapping views with different layouts stay safe too.
func copyData(from source: MLMultiArray) {
    let elementSize = ANEMemoryUtils.getElementSize(for: dataType)
    if dataType == source.dataType, shape == source.shape, strides == source.strides {
        let copied = withUnsafeMutableBytes { destination, _ -> Bool in
            guard destination.count == count * elementSize else {
                return false
            }
            source.withUnsafeBytes { destination.copyMemory(from: $0) }
            return true
        }
        if copied {
            return
        }
    }
    let values = (0..<count).map { source[$0] }
    for i in 0..<count {
        self[i] = values[i]
    }
}

Compare it with the first version. The fast paths are narrower, there is a guard that can fall back, the copy is a memmove, and the hottest call site is deleted rather than optimized. Every one of those changes came from someone else reading the code.

Proving nothing else changed

A performance PR to a library other people ship has to show two things: that it is faster, and that it changes nothing else.

Same output. Transcripts were identical on every measurement. Nathan's review harness, which I re-ran on my M3 Pro, compared 400 transcriptions with and without the fix: zero changes, and 28 to 29 ms saved in each of its three configurations, 34 to 42% of the whole call.

Same behavior. On the final head, the full FluidAudio suite ran serially with model downloads enabled: 2,543 tests, 0 failures, plus 64 of 64 Swift Testing cases.

Faster, measured carefully. I built the commit before the PR and the final head side by side, wired the same profiler to each, and ran two interleaved pairs of 20 warm runs on the same 11-second clip with the same Orukeet bundle:

Interleaving matters. Run all the "before" numbers and then all the "after" numbers, and anything that drifts in between - thermals, a background index, Spotlight - lands on one side only.

One measurement detail cost me an evening and is worth passing on. My first full run looked wrong: the encoder took about 64 ms instead of 26 ms. I was on a video call. Camera and audio effects run on the same Neural Engine, and they were sharing it with my benchmark. I deleted that run and started over on AC power with nothing else using the ANE, and I now check that the encoder measures near 26 ms before trusting any number from that machine.

There was also a nice cross-check. Separately, I had written my own decode loop for Orukeet, without FluidAudio, to see how fast the decoding could be. It ran the 11-second clip in 48 ms against FluidAudio's 73 to 77 ms at the time. After this fix, FluidAudio runs it in 48.2 ms. The gap between "the library" and "hand-written" was the zero-fill, almost exactly.

Shipping it

The PR merged on September 22: nine commits, nine files, 339 lines added and 66 removed, most of the additions being tests. FluidAudio 0.17.0 shipped it the next day. TapTalk moved from FluidAudio 0.15.8 to 0.17.5 in its 0.5.0 release, so every TapTalk dictation now skips those 20 ms. So does every other app that runs Parakeet-style models through FluidAudio, without changing a line.

What else the profile said

Fixing the zero-fill was the obvious win in that table. The profile also answered questions I would otherwise have argued about:

• The GPU is not faster here. The same encoder took 73 ms on the M3 Pro's GPU against 26.7 ms on the Neural Engine, 2.8x slower, even though FluidAudio and Nathan measured the GPU faster on newer M5-class chips. The right compute unit is a per-chip measurement, never a global default.
• Batching the joint network made it slower. The joint calls are the next-biggest cost, about 20 ms in about 120 small calls, so I built a joint that scores eight frames per call. On my test clips it cut 46 calls to 40, but each batched call cost 0.61 ms against 0.17 ms, and decoding went from 20 ms to 36 ms. Parakeet's TDT decoder already skips blank frames, so the joint runs about once per token and there is little to batch. That lever is closed.

Conclusion

The bug was not exotic. It was a for loop through a convenient API, written for small arrays and later handed a large one, sitting on the one path where every millisecond is felt. Nobody would have found it by reading the code, because nothing about it looks slow. The profile found it in one run.

The more useful lesson came after. My first version was fast and wrong in two ways I had not considered, and it kept an operation that should not have existed. Three reviewers - a bot, the library's maintainer and the person whose model I was running - each found something different, and the merged code is smaller, safer and faster than what I submitted.

Where to find it

• The PR: FluidInference/FluidAudio#941 - the diff, the review thread and the measurements.
• FluidAudio: github.com/FluidInference/FluidAudio - the fix is in 0.17.0 and later.
• TapTalk: thetaptalk.com, or brew install --cask vakharwalad23/tap/taptalk - free, open-source, on-device dictation for macOS, running on this fix since 0.5.0.

References

• FluidAudio#941 - bulk-fill MLMultiArray resets and copies
• FluidAudio - Core ML speech models for Apple platforms
• FluidAudio#928 - AsrModels.loadLocal, by Nathan Roll
• oruk.ai and Orukeet on Hugging Face - the model I was profiling
• MLMultiArray - Apple's Core ML array type, including withUnsafeMutableBytes
• The last TapTalk post - Orukeet, Silero VAD on Core ML, and the microphone fixes
• TapTalk on GitHub - the app this was all for