A hands-on course for systems engineers: Implement Qwen3 inference from scratch on Apple silicon using MLX arrays, starting with matrix multiplication, then progressively adding KV caching, quantization, continuous batching, and paged KV to build a mini vLLM, and finally constructing a coding agent.
Stars 4851 (+22 / 24h)
Rationale
- Four-week roadmap: In week 1, implement attention, RoPE, GQA, RMSNorm, MLP, sampling, and the autoregressive loop using only mlx.core; in week 2, add KV caching, 4-bit quantized matrix-vector multiplication, and SIMD prefilling; in week 3, implement continuous batching, chunked admission, and paged KV, allowing decode attention and FlashAttention to read pages directly
- For each operator discussed, you are required to implement it yourself in Python, C++, or Metal, without using MLX's pre-optimized versions; MLX serves only as a correctness reference and performance baseline
- The coding agent for week 4 is released daily, currently open through day 9: file modification after approval, single validation commands, checkpointing and recovery, context compression, deterministic evaluation, etc.
Usage
- The textbook is at skyzh.github.io/tiny-llm; start with the environment setup chapter; check your environment with: pdm install -v, pdm run check-installation, pdm run test-refsol -- -- -k week_1
- Exercises are located in the tiny_llm package, with tiny_llm_ref serving as the reference solution for testing; follow the order of the book's table of contents
Risks
- Requires a Mac with Apple silicon; the author warns that starting from day 3 of week 4, the agent will send file contents to the model, modify files, and run commands, so use a disposable workspace without secrets
- This is an educational project, not a production inference framework; the repository has existed since 2025, licensed under Apache-2.0