DeepSeek has open-sourced DeepInfer, the inference framework that supports its large-scale online services. This version offers only offline (batch) inference and currently supports DeepSeek-V4.1-Flash.
Stars 23403
Confidence: High (80) · Event time 2026-10-15
Reasoning
- The repository was created on 10-16 (recent activity, not within the past 24 hours); the GitHub deepseek-ai source failed to fetch in the previous period and entered the candidate list for the first time in this period.
- Officially described as the same framework used for online services: featuring custom kernels such as FlashMLA, DeepGEMM, and DeepEP; Attention DP / CP and MoE EP + EPLB parallelism; DSpark speculative decoding with verification scheduled by confidence; and asynchronous scheduling.
- Supports batch-invariant inference and deterministic sampling, ensuring consistent output for the same input and sampling parameters (NVIDIA only); supports constrained output via regex, JSON Schema, or EBNF.
- The repository includes offline evaluation scripts for AIME 2026, HLE, and GPQA Diamond; star count only reflects attention level.
Usage
- Environment: Linux, GCC 13+, Python 3.12; NVIDIA requires CUDA 13, Ascend requires CANN 9.2. In an empty Python 3.12 environment, export DEEPINFER_ACCELERATOR=nvidia (use ascend for Ascend) and then run ./build.sh.
- python examples/offline_inference.py downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 to the Hugging Face cache by default; if local weights are already available, set DEEPINFER_HF_MODEL_DIR to point to them. For multi-GPU usage, use parameters such as --ep 8, --ep 2 --dp 8, etc.
- Deterministic output: Set DEEPINFER_BATCH_INVARIANT=1 and DEEPINFER_ENABLE_MTP_VARLEN=0, and pass a fixed seq_seed to SamplingParams; for structured output, pass so_args to SamplingParams.
Risks
- Supports only NVIDIA SM100 architecture GPUs and Huawei Ascend 950 NPUs; consumer-grade graphics cards are not supported. Official performance charts are based on measurements using 4 cards for prefill and 64 cards for decode.
- This version does not include a KV cache system, Prefill/Decode disaggregated deployment, or multi-replica load balancing, so it cannot be directly used to build a complete online service; currently supports only the V4.1-Flash model.