Parallel draft models can draft an entire block of tokens in a single forward pass, but the authors argue that the officially released systems do not fully exploit this speedup: in single-request scenarios, only a few tokens are accepted per step despite reading all target weights; in batched scenarios, per-request verification overhead roughly halves the speedup; and the optimal approach varies depending on the serving scenario, draft model, target model, and GPU. SpecPilot is an automated optimization loop: in each round, a cost model identifies the component of the current system furthest from its upper bound, multiple agents explore directions in parallel and merge successful changes, and a kernel sub-agent specifically tunes kernels that are furthest from the memory bandwidth limit; token forcing compares each output token one-by-one against the original to prevent "reward hacking"; structured memory carries verified mechanisms to the next round and new deployments. The repository releases the inference engines (based on a trimmed SGLang) that it found for 6 different deployments, along with launch, benchmarking, and validation scripts.
Likes 16 · Code Stars 82
Reasons
- The authors claim no training and no quality loss, with speedups relative to autoregressive decoding increasing from 4.46x in the original to 10.87x for single requests, and from 1.94x to 3.93x for batched requests; the README table shows Qwen3.8-27B + DFlash2 on H100 improving from 3.95x to 8.86x, and includes speedups and confidence intervals for four H100 settings rerun from fresh clones.
- The discovered mechanisms can be read individually: the lossless 12-bit weight format W12 (splitting BF16 into three 4-bit planes and reconstructing bit-by-bit in GEMM registers, which the authors claim alone speeds up BF16 decoding for 8–30B dense models by 1.15–1.26x), two-level verification, tree verification for target models with mixed loop structures, and three batching mechanisms.
- Reproducible: code is Apache-2.0, each setting uses only one GPU, and scripts handle environment setup, downloading checkpoints at fixed versions, launching an OpenAI-compatible service, and comparative benchmarking; on the day of the Hugging Face post, it received 16 likes and 82 code stars (indicating attention only).
Usage
- Set up the environment: bash scripts/setup_env.sh (Python 3.11, torch 2.13 + CUDA 13.0, dependencies locked); download checkpoints for a specific setting: bash scripts/download_models.sh h100_c1_qwen38-27b_dflash2 (the setting name is the directory name under settings/).
- Launch the service: bash scripts/serve.sh h100_c1_qwen38-27b_dflash2 final 0 launches the SpecPilot engine on GPU 0 at port 30000; replace final with stock / ar to launch the original and autoregressive baselines; the interface is compatible with SGLang HTTP and OpenAI.
- Benchmarking: bash scripts/bench.sh <setting> final-1 0, then use scripts/compare.py speed to compare speedups and scripts/score.py to score gsm8k / humaneval / math500 / mbpp; each engine requires kernel compilation on first launch, so run a warm-up pass before timing.
Risks
- Narrow hardware support: tested only on H100 80GB and RTX PRO 6000 Blackwell 96GB; kernels for RTX PRO 6000 are compiled only for sm_120a; torch 2.13 + CUDA 13.0 requires R580 drivers, and older drivers rely on forward compatibility libraries.
- Covers only 6 fixed combinations (Qwen3.8-27B / Muse-Glimmer-30B with DFlash2 / DSpark); the repository contains the discovered engines, but there is no visible method to launch the agent optimization loop itself, so it cannot be directly reused for different models or draft models.
- "Lossless" means that under greedy decoding, every token is determined by the target model; the README states that outputs are not bitwise identical to autoregressive decoding (due to different computation orders in batched verification, near-ties may flip), and the included scores are for integrity checks only; all speedup numbers are as reported by the authors.
Editor's note: Suitable for teams already serving Qwen3.8-27B with DFlash2 / DSpark on a single GPU; readers interested in inference optimization should focus on the W12 weight format and the two-level verification parts.