Edge0 is an open-source streaming MoE inference framework: expert weights are stored on disk and loaded on demand, so peak memory depends on the currently active experts rather than the total parameter count; a trained pre-routing head predicts the experts needed for the next token one step in advance, with the author claiming up to 59% improvement in decoding throughput; a 4-bit base model is paired with distilled LoRA to compensate for quantization loss. On 11 4, the GGML-based unified engine edge0-infer was released, running on macOS, iOS, Android, and Windows via a single C++ interface, including command-line tools, an OpenAI-compatible local service, and chat application examples; the accompanying Edge0-35B-A3B is based on Qwen3.6-35B-A3B, with the author's self-tests showing an average score on five OpenCompass tasks 3.9 points lower than the original.
Why it matters Predicting which experts to use one step in advance allows disk reads and computation to overlap, enabling large MoE models to run locally on smartphones and standard computers.
Stars 9670 (+483 / 24h)
Reasons
- 483 new stars in 24 hours (indicating attention only)
- The README provides the author's benchmark tables across platforms: decoding on an M5 MacBook using Metal achieves approximately 24.8 tok/s with peak memory of 3.43 GiB, while an iPhone 17 Pro Max achieves approximately 13.2 tok/s
Usage
- Compile according to the Getting Started guide for edge0-infer, use the included resource tool to download and convert Edge0-35B-A3B-preview, then launch the command-line interface or local service
Risks
- Speed and scores are based on the author's self-tests; initial setup requires approximately 18 GB of raw weights and approximately 20 GB of space for converted resources
- The unified engine currently supports only the 35B tier, with the 8B tier still in the planning stage; SSD speed directly impacts generation speed