A local inference tool based on llama.cpp: it offloads experts from MoE models that exceed memory or VRAM capacity to an SSD for on-demand loading, runs using standard GGUF files, and provides a browser-based chat interface and an OpenAI-compatible API; its approach is similar to the top-recommended colibri, but it currently supports only two models.
Stars 259 (+32 / 24h)
Reasoning
- The models currently tested are DeepSeek-V4.1-Flash-1017 and Muse-Spark-Open, both community-converted GGUF versions; the author compared prefill and generation speeds against the original llama.cpp and colibri on four machines: a MacBook Pro (M5 Max), an RTX 5060 Ti, an RTX 5090, and a DGX Spark, and published the measurement methodology
- No memory parameters need to be configured; the program automatically adjusts based on available memory; the first run measures machine performance
Usage
- Apple Silicon macOS or Linux with NVIDIA GPUs: curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh (compiles from source, takes about 2 to 12 minutes, no sudo required)
- inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF for terminal chat; inferssd serve -hf … to launch the web interface and http://127.0.0.1:8080/v1 API; use --model-dir to specify the drive for storing models
Risks
- The first run requires downloading approximately 470 GiB of model files; Linux requires the CUDA toolkit, liburing headers, and systemd with cgroup v2 enabled; Windows support is still under development, with no precompiled packages available
- Comparison figures are self-tested by the author; model files are community-converted versions, and weight licenses are subject to the respective publishers; the license is Apache-2.0, and the repository was created on 10 22