Anemone: LLM Inference on the Apple Neural Engine, With a GPU Drafter, Exact Output and Half the Energy

◆ IMG800×600

Why I wanted LLM inference on the Apple Neural Engine

Every Apple Silicon Mac since 2020 has a Neural Engine, a block of silicon built for matrix math at low power. Almost no LLM software uses it. MLX, llama.cpp and Ollama all run on the GPU. The ANE sits idle while your fan spins.

I have a reason to care. I build TapTalk, a dictation app that runs speech models on the ANE through Core ML, and I sent a fix upstream to FluidAudio, the Core ML speech library it uses (the zero-fill fix). So I know how good the ANE is when a model fits it. I also know the other side. On 2026-10-10 I tried the obvious thing for TapTalk's Smart Mode, an LLM cleaning up transcripts, and compared Core ML LLM runtimes against llama.cpp on the GPU. The Core ML ones were 4.6x to 21x slower on the same weights. I kept llama.cpp and moved on.

Ten days later DeepSeek open-sourced DeepInfer, the inference engine behind their own serving, and I forked it. The idea that caught me was small: launch the next GPU step while the CPU is still finishing the previous result, so the accelerator never waits on the host. I wrote a long explainer on it, DeepInfer: how DeepSeek's inference engine works, so I will not repeat it. The other thing I had come across was DeepSeek's DSpark drafters, on Hugging Face since June: tiny speculative-decoding models published for Qwen3, with a head that predicts how likely each guess is to be accepted.

On 2026-10-20 I sat down and asked a plain question: what from this engine could carry over to a Mac, with the Neural Engine as the main chip? Nothing ports directly. DeepInfer serves one 552B mixture-of-experts model on NVIDIA and Ascend hardware, with CUDA graphs and expert parallelism. But two ideas travel: drafting with DSpark, and overlapping work so no device waits. The research that day gave me the physical limits, and they were not encouraging. Here they are.

Why the ANE is hard for LLM decoding

Generating one token means reading every weight of the model once. The compute per weight is tiny (one multiply-add per weight per token), so the speed limit is how fast bytes arrive from memory:

time per token = bytes of weights read / bandwidth

Worked example for Qwen3-8B on the ANE. I split the model into four chunks of 9 layers, each weight stored in 4 bits, which comes to 3.84 GB in total. A timing study on random-weight layers measured about 57 GB/s of weight streaming on the ANE:

3.84 GB / 57 GB/s = 0.067 s = 67 ms per token  ->  about 15 tokens/s

Measured on the real model: 65.8 ms per token, 15.1 tokens/s. The formula is almost exact. Plain MLX on the GPU decodes the same model at 30.4 tokens/s, about 33 ms per token. The GPU streams weights roughly twice as fast as the ANE, so on plain decoding the ANE loses by about 2x.

I also saw it on a tiny model. Apple's own llm-benchmark on Qwen3-0.6B: the ANE decodes 86.7 tok/s, the GPU 234 tok/s. Prefill, though, went the other way: 4,942 tok/s on the ANE against 3,939 on the GPU. That is the clue.

Prefill processes many tokens per weight read. When a call carries many tokens, the weights are streamed once and used many times, and the bottleneck moves from bandwidth to compute. The ANE has much more compute than bandwidth. Here is the cost of one call through the four Qwen3-8B chunks as I widen it:

Going from 8 to 64 tokens costs 29% more time for 8x the tokens. Going from 8 to 16 adds 0.18 ms per extra token, against 65.8 ms for a token decoded alone.

◆ IMG800×600

That is the whole bet. One token at a time, the ANE is the slower chip. Sixteen tokens at a time, it costs the same as one. If something could hand it sixteen plausible tokens to check per call, the ANE would turn its slow weight reads into a throughput advantage, at far lower power than the GPU. That something is speculative decoding, and I covered the basics in Bear the Tokens and swapping vLLM's suffix drafter. The short version: a cheap drafter guesses several tokens, the big model verifies all of them in one forward pass, and you keep the longest correct prefix plus one token the big model produces for free.

How does Anemone work?

Anemone splits the job by what each chip is good at:

• The 8B model lives on the ANE. It has spare compute to verify 16 tokens per call.
• The drafter lives on the GPU. Its steps are small and sequential, and the GPU streams weights twice as fast. DSpark's drafter is 5 transformer layers that read the big model's hidden states from layers 1, 9, 17, 25 and 33.
• The CPU runs a thin loop: accept, align, stream text.

The decode loop

Each cycle sends the last accepted token to the drafter, gets 7 guesses with a confidence each, and builds one 16-row verify call: the last token, the 7 guesses, runner-up tokens as side branches, and a few replayed tokens (more on those below). The ANE answers with its own top choice at every row. The loop walks the guesses in order and keeps them until the first mismatch, then takes the ANE's next token. A guess survives only if it equals what the 8B would have chosen anyway. That is why the output is exact.

A cycle costs 21.3 ms of drafting and 64.4 ms of verifying, plus about 0.1 ms on the CPU. It keeps 3.01 guesses on average, plus the free token, so 4.0 tokens per 86 ms.

◆ IMG800×600

Plug that into the speed formula, where a is the mean guesses accepted per cycle, t_d the draft time, t_v the verify time and t_h the host time:

tokens/s = (a + 1) / (t_d + t_v + t_h)

Worked example from the lab prototype on the math prompt: a = 5.07, t_d = 21.1 ms, t_v = 67.5 ms gives 6.07 / 0.0886 = 68.5 tok/s, against 65.6 measured, so the host took about 4 ms per cycle there. Speculation beats plain decoding when a is larger than (t_d + t_v) / t_plain - 1. On the ANE that is (21.1 + 67.5) / 65.8 - 1 = 0.35: every prompt clears it, because the ANE's single token is so slow the bar is low. On the GPU with MLX, plain decode is 32.5 ms and a cycle 110 to 116 ms, so the drafter must accept more than about 2.4 to 2.6 guesses, and only math and code prompts do. This is why a drafter helps the ANE more than it helps the GPU.

The parts that took real work

Chunks and in-graph argmax. Qwen3-8B has 36 layers. I export it as four Core AI programs of 9 layers. The last one also holds the final norm, the 151,936-wide output layer and the argmax, so only token ids and hidden states leave the ANE. Reading 151,936 logits back to the CPU for every row would waste time and bandwidth.

Palettized weights. Each chunk stores weights as 4-bit indices into a lookup table of 16 fp16 values, one table per group of 32 output rows. That is Apple's own LUT4 format, so the weights export through Core AI unchanged. The cost in bits is 4 + 16 * 16 / (32 * n_in) per weight; for an input width of 4096 that is 4.002 bits. Chunks come out at 880 MB each, 1.2 GB for the last.

Fixed shapes. Each chunk has one function per pair of context size (256, doubling up) and call width. The ANE compiler specializes for static shapes, so there is no "width 1". The engine always asks for width 16 to verify and width 64 to prefill, and picks the smallest context that fits so short chats stay fast.

The KV write offset must be a multiple of 4. KV-cache writes on the ANE failed with "Request cancelled" unless the write offset was a multiple of 4. So when the accepted length is not aligned, the call replays up to 3 earlier tokens to start on a multiple of 4. The last token, 7 guesses and up to 3 replays fit exactly inside width 16, which is why 16 is the width I verify with.

The drafter as a Core AI GPU program. DSpark's drafter reads the 8B's hidden states into its own context, drafts 7 tokens with a runner-up for each in one call, and has every shape fixed: positions, counts and offsets are inputs, attention reads the whole cache through a mask. When I let shapes change, Core AI recompiled on every call, about 350 ms each. I picked this runtime by measuring it against an MLX drafter on the same ANE target: Core AI program 41.5 tok/s with 2.83 accepted, MLX 38.0 and 40.6 with 2.74 (both measured on battery power). A second runtime to ship was not worth the tie.

Trees. A width-16 call has room for more than one chain. The engine fills the free slots with the drafter's runner-up tokens as one-token side branches, at the depths where the chain is most likely to miss. Same verify width, same cost, 6% more accepted tokens per cycle, 4% faster (45.9 against 44.0 tok/s).

I should say what I did not build. DeepInfer's real trick overlaps CPU post-processing of step N with the launch of step N+1. In Anemone the host time is 0.1 ms per cycle, so there is nothing to hide, and the one overlap experiment I did run (below) lost. The idea of "keep the devices busy" carried over; the mechanism did not.

How accurate are 4-bit weights on the ANE?

Exact output relative to what? Anemone's speculative output equals plain greedy decoding of the same ANE weights, and anemone bench checks every answer token for token. But those weights are 4-bit, so the more honest question is how close they sit to the original bf16 model.

My first attempt was Apple's default palettizer in Core AI. I measured it with KL divergence: feed the same text to the bf16 model and the 4-bit model and average the difference between their next-token distributions (lower is better), plus top-1 agreement, how often both pick the same next token.

Apple's default was far off, 4x worse than MLX. So I built a quantizer in two steps, still inside the same LUT4 format:

1. Activation-aware k-means. Pick each group's 16 table values by weighted k-means, where every input column counts by how much signal it carries (the sum of squared calibration inputs).
2. GPTQ onto the fixed table. Keep the 16 values, and change only which entry each weight snaps to. Snap one input column at a time, and push each column's rounding error onto the columns not yet snapped, using the inverse Hessian of the layer. Because every group still holds 16 values, the result exports through Core AI unchanged: same size, same speed as Apple's default, 7.3x lower KL on the lab run (0.341 to 0.047).

On the first run GPTQ took docs KL from 0.136 to 0.043 in simulation, and the ANE measured 0.044, so the simulator I used for quick checks matched the hardware within 0.004 KL.

Here is the catch I caught late. My first calibration used code and docs from the same repository as my evaluation files, so the 0.047 was partly a domain match. Calibrating on Python standard library files alone scored 0.107, worse than MLX. Adding five machine-learning library modules (transformers, torch, huggingface_hub, none from the docs repo) fixed it: 0.059 docs, 0.076 chat. I put that caveat in the README and made it the converter's default. Chat quality was the weak spot, so I added the model's own answers to chat prompts as calibration text: 16 answers took chat KL from 0.139 to 0.086, and 48 answers took the 8B to 0.064 and the 4B from 0.133 to 0.087 (MLX: 0.128). The 4B and 14B still trail MLX slightly on chat KL in the folders on Hugging Face (14B: 0.068 against 0.058), because those were converted before the 48-answer change. I would rather say that than hide it.

I also tried lookup tables with other bit widths. 6-bit tables score KL 0.025 (1.5x the bytes, so the model gets slower), and 3-bit tables score 0.202 docs and 0.391 chat, which breaks chat. Four bits it is.

Energy per token: how I measured it and what I got

Anemone's headline is energy, so the method matters. macOS 27 keeps the CPU, ANE and DRAM energy counters frozen unless you run as root, so there is no "ANE-only" number. I measured whole-system power from the SMC PSTR sensor and GPU power from IOReport, both readable without root, sampled every 100 ms.

Each engine ran all 6 prompts in one window plus a 2-second tail, after a 10 s settle and 10 s of idle, in two interleaved rounds. The SMC reading is smoothed (about 4 s to rise, 2 s to fall), hence the tail:

E = mean_power * (generation_time + 2 s) / tokens_generated

Worked example, Anemone tree round 1: 20.77 W x (26.72 s + 2 s) / 1224 tokens = 0.487 J per token. The whole-system number includes the display and background processes, the same for every engine. Idle on this Mac is 1.5 to 2.8 W.

M3 Pro 18 GB, macOS 27.0.1, AC power, 6 prompts x 256 tokens, greedy, thinking off, all rows the same night (2026-10-24):

Read it carefully. mlx-dspark is 1.35x faster than Anemone. Anemone uses 1.45x less energy per token at 55% of the power, and 2.5x less than plain MLX. Its GPU draws about 1 W (mlx-dspark's draws 14 W). Against plain MLX it is 1.5x faster. Anemone running with no drafter at all is slower than MLX and no better on energy (1.10 against 1.20 J), which is the first half of this post's argument in one row.

The "equal to plain greedy" column needs a note. mlx-dspark is a good project and I am glad it exists. But its 8-row batched check rounds differently from single-token decoding, so its text drifted away from MLX's own plain greedy output on 4 of the 6 prompts, first at tokens 24, 109, 126 and 173. A tiny numerical difference flips a close call and the rest of the reply follows a different path. Both texts are fine; they are just not the same text.

I did not always have an honest baseline. On 2026-10-21 my lab number looked great: ANE plus an MLX drafter at 42.6 tok/s against 35.1 for "MLX plus DSpark", a 1.21x win. That comparison used my own MLX loop, which verified 16 rows with stock kernels. On 2026-10-24 I ran mlx-dspark properly, with its own tuned kernels, and the "1.41x faster than MLX plus DSpark" line in my README became "0.74x". I rewrote the README that night. The honest framing, energy first, came out of that correction.

Three more models, same method and setup (the 4B the same night, the 14B on 2026-10-25, R1 on 2026-10-27, except mlx-dspark's R1 lossless count, which is from 2026-10-25):

The 14B energy numbers are one round each; for R1 only Anemone has two rounds; the 4B and 8B have two for every engine. R1 always thinks, and DeepSeek publishes no drafter for it, so I use the Qwen3-8B drafter, which suits Qwen3's regular non-thinking answers far better than R1's thinking text. It accepts 1.53 guesses per cycle there against 3.05 on Qwen3-8B. Anemone still matches plain MLX's speed on it at 1.5x less energy.

There is one more thing the energy story buys, and it is why the GPU matters. With Anemone and mlx-dspark each running an 8B at the same time on one 18 GB Mac, I got about 40 + 49 = 88 tokens/s of combined output, against 63 for the fastest single engine. The ANE and GPU barely slow each other down: nine layers take 23.6 ms on the GPU alone, 15.4 ms on the ANE alone and 25.0 ms with both running. That was a rough one-off measurement, not a benchmark I would defend to the decimal, but it is the clearest picture of what "leaves the GPU free" means.

The 14B night: 2.9 tokens/s to 28.5

Qwen3-14B was supposed to be a routine conversion. The 4B went through the converter with no code changes. The first 14B build ran at 2.9 tokens/s, verify taking 1,338 ms per call where the 8B takes 65.

I did not know why. Same chip, same code path, only about 1.7x more weights. So I exported one layer alone and timed it at different widths:

The 14B's MLP cost grew linearly with the number of tokens. At 32 tokens a layer took 64 ms; the ANE was running the MLP once per token. Attention, in the same layer, stayed flat.

The cause is how Apple's iOS reference implementation lays out the MLP. It reshapes the tokens onto the batch axis of the 1x1 convolutions. For the 8B's smaller weights (25 MB per projection) the ANE copes. For the 14B's larger ones (45 MB per projection at 4 bits), it falls back to one pass per token, about 2.1 ms each. Apple's attention keeps the tokens on the width axis and costs the same at every width.

The fix is a few lines in the converter: put the MLP's tokens on the width axis too, like attention. Same weights, same result to within 1.3e-7 in fp32. The 14B layer compiles in 31 s instead of 229 s, a full chunk in about 2.6 minutes instead of 19, and the model went from 2.9 to 28.5 tokens/s. The 8B did not change (47.3 against 47.4 tok/s), since its MLP was already flat, but I kept the fix as the default.

What did not work

This section is longer than I expected, and it is the part I learned the most from. All of these were measured on the same Mac, losslessness checked.

DFlash instead of DSpark. DFlash (z-lab's drafter) drafts a block of 16 in one pass. On the ANE, 16 verify rows cost the same as 8, so a longer block looks free. But on my prompts it accepted 3.84 tokens per round against 4.02 for the DSpark tree, won only on math, and as a Core AI program cost 23.0 ms per call against 16.3 for DSpark. Not worth porting.

Trees in Python. In the Python lab, width-32 trees accepted 20% more tokens (3.29 against 2.74 per cycle) but the verify cost 74.8 ms against 67.5 and the extra drafting ate the rest: 41.1 tok/s against 41.3 for a chain, 37.4 at width 64. They only paid once the engine moved to Swift with a drafter that exports its runner-ups in the same call.

Keeping the GPU awake. The drafter takes 16.3 ms alone, but 21 ms in the engine, because the GPU goes cold during each 64 ms ANE check (25 ms after 20 ms idle, 32 to 37 ms after 64 ms). A tiny dispatch every 2 to 5 ms during the verify cut the draft to 17 to 19.5 ms and gained 2% to 5% speed, but raised power by 2.3 W so energy per token got 9% worse (0.48 to 0.52 J). I optimize energy first, so no.

Drafting during the check. This was my attempt at DeepInfer's zero-overhead scheduling idea: while the ANE checks cycle k, the GPU drafts cycle k+1 for the likeliest outcomes of that check. Results, all lossless:

Guesses came true on only 0% to 28% of checks, and drafts made without the newest hidden states accept fewer tokens. The ANE check itself did not slow down with the GPU working beside it (62.8 against 63.2 ms), so the concurrency was real; the prediction was the problem.

Skipping drafts that will miss. 26% of chain cycles accept nothing. A perfect skip rule would add 6% (43.7 to 46.3 tok/s). But the last cycles do not predict the next miss: skipping after one empty cycle gave 40.4, after two 42.5.

Chaining the GPU draft into the ANE verify with no CPU wait. On the first export, chaining saved 3.7 ms per cycle (83.7 against 87.4). On the published folder it saves 0.3 ms (82.05 against 82.38). I re-measured after I noticed the README leaned on the old number. The engine just waits for each call.

The drafter on the ANE. The GPU drafter program does not compile for the ANE: Unsupported mps.complex_to_planar op, from Apple's RoPE for macOS. By arithmetic an ANE export plus a CPU roll would take about 19 ms against 21.5 ms on the GPU. Not enough to chase.

32K context. The 16384 export compiled and works: a 10,923-token prompt decodes at 10.6 tok/s (6.2 plain), lossless. The 32768 export fails in the ANE compiler (MLIR MPS to ANEC conversion failed), and a softmax split into 16384-key blocks fails identically, so it is not the softmax. The nastier part: Core AI does not raise an error. The program loads in 2 seconds and silently runs somewhere else, one layer taking 21 to 28 ms where the ANE does a layer in about 1.7 ms. An engine that asks for the ANE gets a slow model and no warning. So I added one: Core AI's cache leaves a region named ..._ANE_region_... in a manifest for each function placed on the ANE, and Anemone now reads it after every load and names any function without one.

Splitting one verify across the ANE and GPU. I parked this on arithmetic, and I did not measure it. A column split needs about two joins per layer, 18 per chunk at 0.2 to 0.5 ms, which eats the roughly 6 ms per chunk it would save.

A few things also turned out to be long-context limits, not dead ends. Apple's attention for the ANE transposes the scores so the context sits on the channel axis, and at 8192 that made a chunk 46.25 ms against 21.72 at 4096. The weights read does not explain it (the extra 151 MB of KV cache accounts for 2.6 ms of the 24.5 ms jump). One last-axis softmax per KV group, which I call row-wise attention, gives the same result within 6e-7 and brings a chunk to 30.96 ms. At long context the GPU still wins on decode: on a 5,423-token prompt mlx-dspark decodes at 31.3 tok/s, plain MLX 25.4, Anemone 17.9, though Anemone prefills 13% faster and uses 2.3x less energy for the request (456 J against 1,069 J and 1,077 J).

The night the Mac kernel-panicked

At 05:43 on 2026-10-25 my M3 Pro kernel-panicked (a watchdog timeout, 56 swapfiles, low swap space) while an unattended queue of exports was running. It took the session, the queue, my scratch space and all the logs in /private/tmp with it.

The cause was one scratch script. I was re-exporting the 14B's last chunk, and my script called Apple's palettize_pytorch_model at module level, with no if __name__ == "__main__": guard. That function starts 32 worker processes with Python's spawn start method. Under spawn, every worker re-imports the main script. Each worker re-ran my script, rebuilt a 10 GB fp32 14B chunk, called the palettizer, and spawned 32 more. The converter's own entry point was guarded and would have been safe, which is why the problem hid for days.

# simplified from the script that panicked the Mac: no guard, heavy work at import time
from coreai_models.export.compression import palettize_pytorch_model

chunk = build_chunk()                 # 10 GB fp32, rebuilt by every worker
palettize_pytorch_model(chunk)        # spawns 32 workers, each re-runs this file

# the fix: one guarded entry point
def main() -> None:
    chunk = build_chunk()
    palettize_pytorch_model(chunk)

if __name__ == "__main__":
    main()

What changed afterwards, as rules:

• Every heavy job (exports, quantization, big benchmarks) runs under a memguard.sh wrapper that kills the whole process group if swap grows by 6 GB or free memory drops under 6%.
• Any Python file that palettizes or uses multiprocessing gets a guarded main(). No exceptions for one-off scripts.
• The converter builds chunk layers on the meta device (build peak from 10.2 GB to 0.5 GB, since the checkpoint tensors are memory-mapped) and palettizes with 4 workers. The 14B's last chunk now exports in 92 s at a 9.8 GB peak, down from 12.8 GB with 32 workers.
• One heavy job at a time. Three at once is what got my earlier upload killed by the low-memory reaper at 00:35 on the 24th.

The reboot had one more surprise. Afterwards, Core AI returned a nil error (Foundation._GenericObjCError.nilError) for every cached program written before the crash, instead of recompiling. That leads to the next section.

Compile times and the cache that disappears

The first time a model loads, Core AI compiles every function for the ANE. On the M3 Pro that is 10 minutes for the 4B, 12 to 14 for the 8B, 19 to 23 for the 8192 and 16384 exports, and 22 for the 14B. The result is cached per executable, so later loads take a few seconds.

Two findings made that cost smaller or at least predictable.

Compile time scales with the number of functions. Core AI compiles every function of a program the moment it opens it. Loading one function afterwards takes 0.1 s. One chunk with 20 functions (5 context sizes x widths 8, 16, 32, 64) compiled in 195.8 s; the same chunk with only widths 16 and 64 (10 functions) took 98.7 s. The engine never called widths 8 and 32, so the converter now exports only 16 and 64: the whole 8B folder went from 816.2 s to 414.1 s with the same speed (47.6 and 47.1 against 47.4 and 47.4 tok/s).

Low disk evicts the compiled copies. For a day, models that had compiled fine would later fail to load or compile again, with 6 to 24 GB free on the disk. I traced it to a second cache: the ANE daemon keeps its own compiled copies, which macOS counts as purgeable space. In a controlled test I filled the disk until a write failed, then freed it. Free space jumped by 23.3 GB (macOS had purged that much), Core AI's own cache stayed whole at 25.7 GB, and the 8B's next load took 753.4 s, a full recompile, where it had taken 10.0 s before. A compiled model costs about 2.4x its weights on disk in total. With 60 GB free, the 4B, 8B and 14B compiled one after another and all reloaded in 8 to 15 s with nothing stale. I cannot read the daemon's cache directory, so the exact mechanism is inferred from disk accounting, not seen directly. But the advice is practical: leave about 20 GB free for the 8B. Anemone also deletes a stale entry (named by its program hash) and recompiles when a load returns nil.

Timeline: eight days from idea to four published models

The plan I made on day one was "start with DeepSeek models, then Qwen". It ended up the other way around: DeepSeek's drafters are trained for Qwen3, and the DeepSeek model in the lineup is R1-0528-Qwen3-8B, a distilled Qwen3. Git log has 93 commits across both branches.

Rough shape of the work: one day of research, a lab of numbered experiments (e01 to e22, each one a script and a section in a findings file) built on Apple's open-source coreai-models exporters, then a rewrite into a Swift engine once the lab reproduced. The engine matched the Python lab's acceptance per prompt token for token, which was the check that told me the port was faithful. A seven-millisecond bug showed up on the way: building input arrays element by element with NDArray(scalars:) cost 7 ms per draft. One memcpy-style copy took the mean from 40.7 to 45.5 tok/s.

How do I run it?

Requirements first, because they are real. An Apple Silicon Mac with macOS 27 (Core AI) and Xcode 27 to build. Tested only on an M3 Pro with 18 GB. The first run of each model compiles it for the ANE, 10 to 23 minutes depending on size, and needs free disk, about 20 GB for the 8B. If your Mac is on macOS 26 or older, this will not run.

git clone https://github.com/vakharwalad23/Anemone.git
cd Anemone
swift build -c release
.build/release/anemone run "Explain how a CPU cache works."

The first run downloads the Qwen3-8B folder (6 GB) from the Anemone collection on Hugging Face, compiles it, and answers. Pick another model with --model or ANEMONE_MODEL; the four folders are vakharwalad23/Qwen3-4B-Anemone, Qwen3-8B-Anemone, Qwen3-14B-Anemone and DeepSeek-R1-0528-Qwen3-8B-Anemone. The other commands:

.build/release/anemone chat
.build/release/anemone bench
.build/release/anemone serve --port 8080

bench runs six fixed prompts plain and speculative and prints speed, acceptance and the lossless check. This is real output from the Qwen3-8B folder with the tree drafter (the same run as the table above):

$ anemone bench --model models/qwen3-8b-4k-tree
math      plain  16.1 tok/s   speculative  74.3 tok/s   accepted 5.53   lossless yes
code      plain  16.0 tok/s   speculative  69.3 tok/s   accepted 4.93   lossless yes
explain   plain  16.1 tok/s   speculative  44.7 tok/s   accepted 2.91   lossless yes
summary   plain  16.0 tok/s   speculative  33.9 tok/s   accepted 1.95   lossless yes
advice    plain  16.0 tok/s   speculative  36.5 tok/s   accepted 2.21   lossless yes
translate plain  15.9 tok/s   speculative  16.8 tok/s   accepted 0.56   lossless yes
mean      plain  16.0 tok/s   speculative  45.9 tok/s   accepted 3.01   lossless 6/6

Look at the spread. Math and code, where the drafter guesses well, run at 69 to 74 tok/s. Translation barely beats plain decoding at 16.8. Speculative decoding is only as good as the drafter's guesses. serve is an OpenAI-compatible API on 127.0.0.1:8080 with streaming, greedy only (sampling fields are ignored), one request at a time. Point any OpenAI client at http://127.0.0.1:8080/v1.

From Swift, add the package and call it:

// Package.swift
.package(url: "https://github.com/vakharwalad23/Anemone.git", branch: "main")

import Anemone

let model = try await LanguageModel.load(from: ModelFolder.download("vakharwalad23/Qwen3-8B-Anemone"))
for try await event in model.generate([ChatMessage(.user, "Write a haiku about the sea.")]) {
    switch event {
    case .text(let piece): print(piece, terminator: "")
    case .finished(let response): print("\n\(response.stats.tokensPerSecond) tok/s")
    }
}

The converter is a uv project (anemone-convert quantize, target, drafter, package) that takes a bf16 checkpoint through calibration, k-means plus GPTQ and Core AI export, and produces a model folder. Palettizing the 8B takes about 30 minutes on the CPU. Export takes 99 to 167 s per chunk with a 9.2 GB peak. Three folders shipped with the v0.1.0 release (2026-10-26) and R1 followed a day later; the Qwen3 folders there still carry four call widths, so their first load is twice as long as it needs to be, and I will convert them again when I have a night for it.

A side-track: Orukeet's encoder on the Neural Engine

Once the toolkit worked, I wondered what carried over to my other love, speech. I collaborate with Nathan Roll from Oruk, whose Orukeet model is the default in TapTalk, a fine-tune of NVIDIA Parakeet TDT 0.6B v3. Speculative decoding does not carry over: the encoder holds 97% of the weights and already runs all frames in one call. What does carry is the Core AI export, the placement checks and the compile-cache lessons.

The work is on the public experiments/speech-research branch (it is not part of the SDK). The short version, M3 Pro, clips up to 15 seconds, against TapTalk's 6-bit Core ML build timed the same night in A-B-B-A order:

Both draw about 16 W while they run and leave the GPU idle; the energy saving is a shorter run. The int8 encoder is two 12-layer programs with windows of 2.5, 5, 10 and 15 seconds (12.1, 12.5, 15.8 and 30.2 ms).

Getting there taught me the ANE's rules the hard way. Apple's stock export ran on the GPU (96.6 ms per 15 s window); it took fp16 masks and a 4D conv module to put every op on the ANE. Compressed weights only reached the ANE as 1x1 Conv2d linears, at most about 18 layers per program. A program that began or ended on a LayerNorm failed to compile, until I multiplied by the frame mask at the split boundary and cut at 12 layers. Lookup-table 6-bit weights compiled at 2 layers per program and not at 6, so they do not work for this model yet.

Treat this as early. It is one session on one machine, batch transcription only, with clips over 15 s cut, no live typing, and the power runs were not repeated on another day. I wrote each step's report in the branch so the numbers are checkable.

FAQ

What is Anemone?

An open-source (MIT) Swift engine that runs a large language model on the Apple Neural Engine and a small drafter on the GPU. Its output matches plain greedy decoding token for token.

How fast is Anemone on an M3 Pro?

Qwen3-8B generates 46 tokens/s at 0.49 J per token. mlx-dspark is faster at 62 tok/s but uses 0.71 J per token, and plain MLX uses 1.20 J at 30 tok/s.

Why run an LLM on the Apple Neural Engine?

Energy and a free GPU. Anemone uses 2.5x less energy per token than plain MLX, and the GPU stays almost idle (about 1 W) while the output still matches greedy decoding exactly.

Which models does Anemone support?

Qwen3-4B, Qwen3-8B, Qwen3-14B and DeepSeek-R1-0528-Qwen3-8B, all published on Hugging Face.

What do I need to run Anemone?

macOS 27, Xcode 27 and an Apple Silicon Mac (tested only on an M3 Pro with 18 GB). The first load compiles for many minutes and needs plenty of free disk.

What comes next

Open work, in the order I plan to do it: converting the published Qwen3 folders again (widths 16 and 64 for half the first load, 48 chat answers for better chat KL); long prompts (32K needs the KV cache and mask split into halves, faster prefill, and faster decode where the GPU still wins); wider trees (offline, 36-row trees keep 26% more tokens per check on the 8B and 32% more on R1, at a 9% dearer width-32 check); and numbers from other Macs. If you have an M1, M2, M4 or M5, run anemone bench --json bench.json on AC power and open an issue with the file. It records the chip, memory, macOS build and power source. That is the most useful contribution right now.

References

• Anemone on GitHub: code, README, docs/architecture.md, docs/math.md, docs/benchmarks.md (every run with its method), MIT license
• Anemone models on Hugging Face: Qwen3-4B, 8B, 14B and DeepSeek-R1-0528-Qwen3-8B folders
• Anemone release v0.1.0
• Speech research branch: Orukeet on the ANE, reports S0 to S3 and S7
• DeepInfer by DeepSeek, Apache 2.0, and my explainer on how it works
• DSpark drafter for Qwen3-8B from DeepSeek
• DeepSeek open-sources DeepInfer (TechNode, 2026-10-21)
• mlx-dspark, the GPU baseline I compare against
• MacPaw research: Neural Engine, ANE with a DSpark drafter on a 2.6B model
• apple/coreai-models, Apple's Core AI exporters that the converter builds on
• ANEMLL, earlier work on LLMs on the Neural Engine
• FluidAudio and Orukeet
• My earlier posts: TapTalk, TapTalk with Orukeet, the FluidAudio fix, Bear the Tokens, swapping vLLM's suffix drafter