A local inference engine written in pure C, specifically designed to run massive Mixture-of-Experts (MoE) models: the components required for every token (attention, shared experts, and word embeddings) remain resident in memory, while experts selected by the router are loaded from disk on demand, with a cache remembering your frequently used experts; no GPU is required, but if one is available, the hottest experts and dense layers are offloaded to it.
Stars 46561 (+675 / 24h)
Rationale
- The author's premise: MoE models are large on disk but use only a small portion per token. Taking GLM-5.2 as an example, it has 744B total parameters but uses approximately 40B per token, meaning the model does not need to fit entirely into VRAM, as long as it is "placed correctly." The author emphasizes that the default strategy does not silently alter model precision or routing; limited memory only affects speed.
- The README lists 13 engines: language models include GLM-5.2/5.3, Kimi K3, DeepSeek V4 / V4.1 Flash, MiMo-V2.6, Qwen3.6 / Qwen3.8, etc., along with text-to-image generation via Qwen-Image-2.1 and decision models for binary classification tasks. It can also directly read published GGUF files, supporting CPU, Vulkan (any GPU, including integrated graphics), and CUDA.
coli serveprovides both OpenAI-compatible and Anthropic-compatible interfaces (allowing direct integration with Claude Code), tool calling, image input, and text-to-image interfaces. It includes a built-in web dashboard to visualize when each expert is activated. Speeds for each model are marked with the test machine used (self-tested by the author).
Usage
- Minimum 8 GB of RAM (16 GB or more recommended) and 22 GB of disk space for the smallest model; GPU is optional. On Windows, download the ZIP and double-click
START-HERE.bat; on Linux / macOS, clone the repository and run./start-here.sh. - The installer detects RAM, disk, CPU, and GPU to recommend suitable models, automatically compiles or downloads the engine, supports resumable model downloads, and then opens http://127.0.0.1:8000/. Use
--listto check if each model can run on your machine,--modelto specify a model, and--backend vulkan / cuda / cputo specify the backend. - Integration with other programs: Use http://127.0.0.1:8000/v1，Anthropic for the OpenAI base URL and http://127.0.0.1:8000；也可以让 for the Anthropic base URL. AI assistants can install it on your behalf by following
docs/AI\_SETUP.md, or use the MCP tools provided bycoli mcp.
Risks
- The speed of large models is determined by disk performance. The numbers provided by the author mostly range from 0.05 to a few tokens per second, with the largest Kimi K3 taking about 9.4 seconds per token, making it more suitable for non-urgent tasks. Downloading large models often requires hundreds of GBs, so a fast NVMe drive is necessary.
- The engine is licensed under Apache-2.0, but models have their own licenses: The README notes that Qwen-Image-2.1 is under a non-commercial license, and Muse Spark Open is subject to the Meta license agreement. All speed figures were measured by the author on the listed machines.