Qwen3.8-Flash-Next Slim Edition: Cutting Out a Quarter of the Routing Experts

An unofficial slimmed-down version based on nvidia/Qwen3.8-Flash-Next-FP4-Local: Neural architecture search was used to select which routing experts to retain in each decoder MoE layer and MTP layer, reducing the count from 512 to 384; top-10 routing, shared experts, attention, the vision tower, and the tokenizer remain unchanged. Weight size decreased from 102.14 GB to 85.66 GB (approximately −16.1%). Subsequently, three rounds of GRPO-style LoRA training with verifiable rewards were performed (the final round targeting CVSS score completion and structured vulnerability discovery output), and these have been merged into the full model. The author reports that in a single evaluation run, AIME 2026 scored 95.0 (reference value 93.3), MMLU 87.37, while GPQA-Diamond 75.76 and MMMU 78.22 were slightly below reference values.

Original title: autotrust/Qwen3.8-Flash-Next-E384-FP4-V2

An unofficial slimmed-down version based on nvidia/Qwen3.8-Flash-Next-FP4-Local: Neural architecture search was used to select which routing experts to retain in each decoder MoE layer and MTP layer, reducing the count from 512 to 384; top-10 routing, shared experts, attention, the vision tower, and the tokenizer remain unchanged. Weight size decreased from 102.14 GB to 85.66 GB (approximately −16.1%). Subsequently, three rounds of GRPO-style LoRA training with verifiable rewards were performed (the final round targeting CVSS score completion and structured vulnerability discovery output), and these have been merged into the full model. The author reports that in a single evaluation run, AIME 2026 scored 95.0 (reference value 93.3), MMLU 87.37, while GPQA-Diamond 75.76 and MMMU 78.22 were slightly below reference values.

Why it matters The same GPU can accommodate more KV cache or concurrency; the author lists both scores on standard problems and unresolved issues, facilitating assessment

Upvotes 69 · Likes 2439 · Downloads 55866

Rationale

  • The compression method differs from standard quantization: it reduces the number of experts on top of an already FP4 base, while the number of experts activated per token remains unchanged
  • The author verified this on a single B200 using vLLM 0.31, providing complete launch parameters and a portable script, serve_vllm.sh

Usage

  • After running export PYTHONPATH="$PWD/vllm_patch", use vllm serve . with parameters such as --quantization modelopt_mixed --kv-cache-dtype fp8 --reasoning-parser qwen3 --tool-call-parser qwen3_xml (see the model card for the full command)
  • The repository's bundled vllm_patch/sitecustomize.py is required; otherwise, the FP4 weight layout of this checkpoint cannot be read

Risks

  • This is an unofficial derivative; neither Alibaba nor NVIDIA endorses it; the license is the NVIDIA Open Model Agreement
  • The author explicitly states that CVSS calculation remains unreliable: only approximately 4.5% (CVSS3) and 4.2% (CVSS4) of completed answers match the reference implementation exactly
  • All evaluations are single-run results; reference values are taken from previously published figures and were not re-run under identical conditions; this entry has not been run

Open

Hugging Face Trending

Explore the latest AI Radar updates