inference-optimisation covers single-user and concurrent decoding, speculative decoding, prefill, KV cache capacity, startup time, and time to first token. It requires the agent to first calculate theoretical limits based on bandwidth and compute, then change only one variable at a time, alternating measurements between control groups on both sides, log every result (including failures) in a ledger, and finally publish only after passing long-context retrieval and full-load stability checks, verified with a clean, pinned version. It includes benchmarking, limit calculation, and queuing scripts written using only the Python standard library and bash, capable of benchmarking any OpenAI-compatible service.
Why it matters It codifies the author's tuning practices across various GPUs into rules: change only one thing at a time, sandwich changes with control groups, and only count results that pass long-context retrieval and full-load stability checks.
Stars 1114 (+184 / 24h)
Reasoning
- 24 hours added 184 stars, total stars 1114 (indicating attention only); MIT license, created on 11 16
- The README lists before-and-after tuning data from the author's Intel Arc B390 integrated graphics, Arc Pro B70, RTX 3090, and 4 H200 cards, with links to raw logs and recipes
Usage
- Copy skills/inference-optimisation from the repository to ~/.claude/skills/, then tell the agent to "improve decoding and prefill speed for the model on :8000 within a 40 GiB memory limit"; without an agent, you can directly run ceiling.py to calculate limits and bench.py to benchmark
Risks
- Speedup figures are results from the author's own hardware (the author claims single-user decoding on B390 improved from 43.6 to 53 tok/s); results will vary with different models, engines, or GPUs
- Tuning requires repeatedly restarting inference services and long-duration stress testing; it is best performed on machines not serving external traffic