llama-modchip is a set of ggml/llama.cpp kernel patches, quantization transcoders, and runtime tools aimed at fitting sparse MoE models entirely into system memory for CPU-only inference. The author claims to have further compressed Unsloth's DeepSeek-V4-Flash UD-IQ1_S version to 56.56 GiB (approximately 1.71 bpw), achieving approximately 44 tok/s prefill and 9.1–14.7 tok/s generation on a machine with 64GB of memory and 16 cores of Zen 5, supporting a 256K context; for 30B-class MoE models (such as Nemotron-3.5-30B-A3B), the author reports 25–60 tok/s on laptops with 16–32GB of memory. It also includes self-speculative drafting and KV snapshot recovery to speed up agent sessions with long prompts.
Why it matters Large MoE models do not necessarily require a GPU: by streaming cold experts from memory, local deployment becomes feasible even on standard desktops.
Stars 785 (+287 / 24h)
Reasons
- 287 new stars in 24 hours (indicating attention only)
- The README lists the quantization formats used for each tensor type, memory usage, and reproducible quality test scripts
Usage
- After cloning the repository, run ./modchip chat for automatic configuration; the model uses custom tensor types, so the original llama.cpp, Ollama, and LM Studio cannot read it—you must use the programs included in the repository
Risks
- Speed and quality metrics are self-reported by the author; the quality score for the DeepSeek version is compared against the already heavily compressed UD-IQ1_S (with more than 1 bits), not against the original weights
- The impact of extremely low-bit quantization on complex tasks is unknown; the repository was created on 10 29