Qwen3.8-27B External Draft Model: NVIDIA's Speculative Decoding Acceleration

Released by NVIDIA on 11/17, this external draft model was trained using Model Optimizer (4.21B parameters, based on Qwen3.5-4B, sharing the tokenizer and vocabulary with the target model). During serving, it proposes a set of candidate tokens in one pass according to the target model's sampling settings. Qwen3.8-27B verifies these using rejection sampling, ensuring the final output distribution is identical to that of the target model generating alone, while only reducing decoding latency. Training used 199,998 single-turn prompts from Nemotron-Post-Training-Dataset-v2, with responses generated by Qwen3.8-27B; the probability distribution of the first 20 tokens at each position was recorded as the training signal.

Original title: nvidia/Qwen3.5-4B-Qwen3.8-27B-Draft

Released by NVIDIA on 11/17, this external draft model was trained using Model Optimizer (4.21B parameters, based on Qwen3.5-4B, sharing the tokenizer and vocabulary with the target model). During serving, it proposes a set of candidate tokens in one pass according to the target model's sampling settings. Qwen3.8-27B verifies these using rejection sampling, ensuring the final output distribution is identical to that of the target model generating alone, while only reducing decoding latency. Training used 199,998 single-turn prompts from Nemotron-Post-Training-Dataset-v2, with responses generated by Qwen3.8-27B; the probability distribution of the first 20 tokens at each position was recorded as the training signal.

Why it matters The output distribution is completely consistent with the original model, reducing only decoding latency; NVIDIA claims that the average number of accepted tokens per verification is significantly higher than that of the multi-token prediction heads built into the models themselves.

Upvotes 63 · Likes 67 · Downloads 290

Reasons

  • On the qualitative subset of SPEED-Bench with a draft block size of 16, the average acceptance length is 5.58, compared to 3.81 for Qwen3.8-27B's built-in MTP head; on the 32K long-context subset, it is 5.11 versus 3.86.
  • The same series also includes two smaller draft models: Qwen3.5-0.8B (3.92 / 3.60) and Llama-3.2-1B (3.42 / 3.07), evaluated on the same data and benchmarks.
  • By category, RAG, multilingual tasks, and summarization achieve the highest acceptance lengths (6.58, 6.55, 6.19), while role-playing and writing have the lowest (4.41, 4.59).

Usage

  • Deploy as an external draft model for Qwen3.8-27B: the card lists vLLM as a supported engine and notes it can also be loaded in SGLang and TRT-LLM; inputs must use the Qwen3.8-27B tokenizer.
  • The draft samples with temperature 1.0, top-k 20, and top-p 0.95, consistent with the target model's serving settings; the training sequence length is 12,288 tokens.

Risks

  • The card does not provide launch commands or actual speedup factors, only acceptance lengths; NVIDIA primarily targets deployment on NVIDIA Groq 3 LPUs, listing only Blackwell as a supported GPU architecture.
  • The draft model itself has 4.21B parameters and must share VRAM with the 27B target model; additional VRAM requirements are not specified in the card.
  • The license is the NVIDIA Open Model License (commercial use allowed), not Apache. This entry has not been run.

Open

Hugging Face Trending

Explore the latest AI Radar updates