Two Gates and a Process Boundary: Swapping vLLM's Suffix Drafter Without a Fork

This is the deep dive on the injection mechanism I skimmed over in the Bear the Tokens writeup. That post was about the throughput number. This one is about the part I actually spent the afternoon on: getting a patch to land in a library that spawns its own processes.

Everything below is against vLLM 0.15.1. Line numbers move between releases. The shape of the problem does not.

What I was trying to do

vLLM supports --speculative-config '{"method":"suffix", ...}'. Suffix decoding is model-free speculation: keep a suffix tree over tokens you have already generated, match the tail of the current sequence against it, and emit the most frequent continuation as a draft. The model verifies the whole draft in one forward pass and rejects anything wrong, so the output never changes, only the speed.

I had written my own drafter and wanted vLLM to use it in place of the built-in one.

The two approaches most people reach for first are both bad:

• Fork vLLM. You now own a merge conflict forever.
• Edit the installed package under site-packages. Your fix exists on exactly one machine and evaporates the next time anyone runs pip install.

I wanted something that survives a clean install on a fresh box.

First, find what is actually stopping you

vLLM ships the interface for suffix decoding but defers the real work to Snowflake's arctic_inference. That dependency is enforced in two separate places, and you have to clear both.

Gate 1 is at config time. When you pass method="suffix", SpeculativeConfig validation calls _validate_suffix_decoding() in vllm/config/speculative.py:

def _validate_suffix_decoding(self):
    if not has_arctic_inference():
        raise ImportError("Arctic Inference is required for suffix decoding. ...")

has_arctic_inference() lives in vllm/utils/import_utils.py and is just _has_module("arctic_inference"), an "is this importable" probe. If it returns False you get an ImportError before the server finishes booting.

Gate 2 is at runtime. vLLM's own proposer, in vllm/v1/spec_decode/suffix_decoding.py, does from arctic_inference.suffix_decoding import SuffixDecodingCache. So even past the config check, it still needs the package to draft anything.

Two gates, two overrides. Make the availability probe return True, and replace the proposer class with mine so the arctic import never runs at all.

Wrong turn 1: patching the function where it is defined

The obvious move is to patch the probe at its source:

import vllm.utils.import_utils as import_utils
import_utils.has_arctic_inference = lambda: True

The validator still raises. This is plain Python, and it is worth internalizing because it bites in every codebase, not just this one.

speculative.py imports the symbol by name:

from vllm.utils.import_utils import LazyLoader, has_arctic_inference

That statement runs once, at import time, and copies the function object's reference into speculative's own module namespace. It is not a live link back to import_utils. When _validate_suffix_decoding later calls the bare name has_arctic_inference, Python resolves it against speculative's globals, which still hold the original function. Rebinding the attribute on import_utils changes a name that nobody is reading anymore.

Here is the whole lesson with vLLM removed:

# mod_a.py
def flag():
    return False

# mod_b.py
from mod_a import flag        # binds a reference into mod_b's globals
def check():
    return flag()             # resolves against mod_b's globals

# patching mod_a.flag does NOT change mod_b.check()
# patching mod_b.flag DOES

So you patch the name in the module that imported it:

import vllm.config.speculative as speculative
speculative.has_arctic_inference = lambda: True

I patch both, guarded with hasattr, since the original module is what anything doing a dynamic lookup later would consult.

Wrong turn 2: patching from my own script

With the right name patched, it worked when I built the config myself in a notebook and did nothing when I ran vllm serve. That is a process boundary.

vllm serve does not run in one process. Engine args are parsed and the config is built in the front-end. The engine itself runs in a separate EngineCore process. My monkeypatch ran in whichever process executed my script, and nothing else inherited it.

This is where it pays to read the source instead of guessing. In 0.15.1 there are exactly two calls to load_general_plugins():

• EngineArgs.__post_init__ in vllm/engine/arg_utils.py, which runs in the front-end before the config is constructed
• EngineCore.__init__ in vllm/v1/engine/core.py, carrying the comment "plugins need to be loaded at the engine/scheduler level too"

Those two load points line up exactly with the two gates. Gate 1 is checked while the config is built in the front-end. Gate 2 bites when the proposer is instantiated inside EngineCore. A plugin registered through the entry point runs in both. A patch in your own main runs in one.

The vector that works

vLLM discovers plugins through a setuptools entry point group called vllm.general_plugins. You declare it in pyproject.toml:

[project.entry-points."vllm.general_plugins"]
t4_spec_patch = "t4_spec_patch:register"

And register() does the two overrides:

import os
import sys

_done = False


def register():
    global _done
    if _done:
        return
    _done = True

    # gate 1: un-gate the native suffix path, in the module that imported the symbol
    import vllm.utils.import_utils as import_utils
    import_utils.has_arctic_inference = lambda: True
    import vllm.config.speculative as speculative
    if hasattr(speculative, "has_arctic_inference"):
        speculative.has_arctic_inference = lambda: True

    # gate 2: swap the proposer so the arctic import never runs
    import vllm.v1.worker.gpu_model_runner as gmr
    from t4_spec_patch.proposer import T4SuffixProposer
    gmr.SuffixDecodingProposer = T4SuffixProposer

    print(f"[t4_spec_patch] registered, pid={os.getpid()}", file=sys.stderr, flush=True)

Then pip install . and run vLLM normally. Nothing on disk is modified and there is no fork.

Three details worth knowing:

• Make register() idempotent. It gets called more than once per run. The _done guard keeps it cheap and predictable.
• Import vLLM modules inside the function, not at module top level. The plugin is loaded early and you do not want to fight import ordering.
• VLLM_PLUGINS is an allowlist. In vllm/plugins/__init__.py the loader does if allowed_plugins is None or plugin.name in allowed_plugins. Unset, everything discovered loads. Set, only what you listed loads, and your plugin silently does not run. Check this first when a correctly installed plugin never fires.

Verify it actually loaded

Print a stamp with the pid, and check you see it more than once:

[t4_spec_patch] registered, pid=1234
[t4_spec_patch] registered, pid=1267

At tensor parallel 1 you should get two: the front-end that parses the args, and EngineCore. Zero means the entry point is not installed at all, which is almost always the wrong virtualenv or a pip install run from the wrong directory.

Caveats, and what I actually verified

These are internal APIs. has_arctic_inference, SuffixDecodingProposer, and the module paths they live in are not public contract, and they will move. Pin the vLLM version, keep the surface you touch small, and make the failure loud rather than silent.

I also want to be precise about scope. Everything here was run and verified at tensor parallel 1, where the model runner lives inside EngineCore. With TP > 1 vLLM spawns separate worker processes, and I did not find an explicit load_general_plugins() call on that path in 0.15.1. If you need this on multi-GPU, check that first rather than trusting my two-process picture.

That correction is worth stating plainly, because I got it wrong the first time I wrote it up: I had assumed the entry point ran in "every process vLLM spawns, including the workers." It does not. It runs in two specific places, and the patch works because those two places happen to be exactly where the two gates live.

What I used it for

The drafter itself is a from-scratch implementation of SuffixDecoding (Oliaro, Jia, Campos, Qiao, NeurIPS 2025 Spotlight, arXiv:2411.04975), whose reference implementation lives in ArcticInference. The algorithm is theirs. What is mine is a dependency-free version: one global suffix tree, Numba-JIT compiled, open-addressing hash table, greedy most-frequent-child drafting, in about 200 lines with no arctic_inference import anywhere.

On the contest workload (200 requests, 50 concurrent, 512 in and 512 out, greedy with ignore-eos, Qwen2.5-0.5B on a single Tesla T4) it lands around 12 verified tokens per forward pass and about 8.7k output tok/s. That workload is degenerate on purpose, so treat the number as characterizing the benchmark rather than the method.

Code, benchmark harness, and the longer writeup on the drafter itself: github.com/vakharwalad23/vllm-suffix-decoding-t4

References

• vllm-suffix-decoding-t4 (my repo)
• Bear the Tokens: FP16 and suffix decoding on a T4 (my earlier post)
• vLLM (GitHub)
• vLLM plugin system docs
• Arctic Inference (GitHub)
• SuffixDecoding: A Model-Free Approach to Speeding Up LLM Inference (arXiv 2411.04975)
• Enabling Suffix Decoding in vLLM (RFC, issue 18037)
• setuptools entry points