ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-NVFP4-8xV100-SXM2-PP2xTP4

The author first loads all routing experts into the GPU using IQ2-level compressed formats, then progressively replaces them with weights from NVIDIA's official NVFP4 package based on usage frequency, until no more can fit: out of 15360 routing experts, 6087 retain NVFP4 precision, handling 79% to 85% of the decoding routing in the measured samples. The author states that the version from 11 13 achieves a single-request generation speed of approximately 117.7 tokens per second, with code-type lengths of 1K to 256K yielding 143 to 175 tokens per second, and a next-token match rate with the NVFP4 reference of 95.1%; it is slower than the same author's faster Volta sPLIt but closer to official precision.

The author first loads all routing experts into the GPU using IQ2-level compressed formats, then progressively replaces them with weights from NVIDIA's official NVFP4 package based on usage frequency, until no more can fit: out of 15360 routing experts, 6087 retain NVFP4 precision, handling 79% to 85% of the decoding routing in the measured samples. The author states that the version from 11 13 achieves a single-request generation speed of approximately 117.7 tokens per second, with code-type lengths of 1K to 256K yielding 143 to 175 tokens per second, and a next-token match rate with the NVFP4 reference of 95.1%; it is slower than the same author's faster Volta sPLIt but closer to official precision.

Why It Matters Determines which experts retain original precision based on per-byte usage frequency; all experts reside in VRAM without traversing PCIe; speed, precision, and failed attempts are disclosed by date to facilitate reproduction.

Star count 18

Rationale

  • The repository was created on 11 10, with code and documentation under the Apache-2.0 license; having read the README, every figure is labeled with measurement date, conditions, and raw logs
  • The README separately lists parts that "did not succeed or are not yet good enough," such as the quality of fast prompt reading which remains unconfirmed

Usage

  • Requires two 4 cards on an NVLink board totaling 8 V100-SXM2-32GB units: compile the engine per the README, download the 556 GB Hugging Face package and verify it with SHA256SUMS, then launch using serve_hot_iq.sh

Risks

  • High hardware barrier prevents ordinary users from reproducing results; speed and precision figures come from the author's own machine
  • The fast prompt reading feature, enabled by default, alters output, and the author notes its quality is unconfirmed

Open

GitHub

Explore the latest AI Radar updates