The repository does not depend on Flax, Equinox, or PyTorch; parameters are plain dictionaries, and the entire model is written in deepseek_v3.py: MLA caches only a small latent vector per token, DeepSeekMoE consists of 1 shared experts plus 16 routed experts with load balancing based on expert bias, along with a multi-token prediction module. The author trained a model with 3620 million parameters (1360 million active per token) using public domain detective novels, running it for about 2.6 hours on an M5 MacBook Air; validation loss was nearly identical whether load balancing bias was disabled or enabled, but expert load distribution was significantly more uniform, consistent with the paper's claims.
Why It Matters Latent attention, mixture of experts, auxiliary-loss-free load balancing, and multi-token prediction from the paper each correspond to a single function, allowing one to understand the structure of V3 by reading from top to bottom.
Star count 102
Reasons
- The repository was created on 11/13, licensed under MIT; the README has been read, accompanied by a long-form article with paragraph-by-paragraph explanations and reproducible data hashes
- The author explicitly states this is an architecture demonstration, not a usable language model, and lists unimplemented parts (FP8 training, DualPipe, YaRN, etc.)
Usage
- Requires an Apple Silicon Mac, Python 3.12, and uv: after cloning, install pinned dependencies according to the README, run prepare_data.py to prepare data, and train.py for training; if you don't have a Mac, you can run it on CPU following the "Running on CPU" section
Risks
- The released checkpoints are Python pickle files, which may execute code upon loading; the author advises loading only files from trusted sources
- Training result numbers come from a single run by the author