hyperionaidev/smolmaxxing

smolmaxxing is a set of llama.cpp patches: it implements kernels for Cortex-A55 and Apple A7 supporting 1-bit, 2-bit, and ternary formats, removes rarely used output rows from the vocabulary to save memory, and for Mixture-of-Experts models keeps only frequently used experts in memory while loading the rest from flash storage and prefetching the next layer. The author claims that a Xiaomi TV box can run the 16B Ling-mini-2.0 at 5.1 tokens per second, and a 2022 OnePlus phone can run Qwen3-Next-80B-A3B; the repository includes complete records of 165 experiments, including failed attempts.

smolmaxxing is a set of llama.cpp patches: it implements kernels for Cortex-A55 and Apple A7 supporting 1-bit, 2-bit, and ternary formats, removes rarely used output rows from the vocabulary to save memory, and for Mixture-of-Experts models keeps only frequently used experts in memory while loading the rest from flash storage and prefetching the next layer. The author claims that a Xiaomi TV box can run the 16B Ling-mini-2.0 at 5.1 tokens per second, and a 2022 OnePlus phone can run Qwen3-Next-80B-A3B; the repository includes complete records of 165 experiments, including failed attempts.

Why it matters By placing rarely used experts of Mixture-of-Experts models in flash storage for on-demand loading, devices with only about 1GB of memory can run models with tens of billions of parameters; all figures were measured on-device.

Stars 699 (+61 / 24h)

Reasons

  • 61 new stars in 24 hours (indicating attention only); MIT license
  • For each device, the chip, available memory, everyday model, and maximum model are specified along with on-device measured speeds, and all benchmark logs are made public

Usage

  • For TV boxes and other ARMv8.2 Android devices, follow the instructions in docs/tv-box-setup.md; Android phones with 8GB of memory can directly download the mobile GGUF files uploaded by the author to Hugging Face and run them according to docs/phone.md

Risks

  • Model files for TV boxes and iPads require a patched version of llama.cpp to load; the iPad solution requires a jailbroken iOS 12 device
  • Speed and quality data were measured by the author on their own devices

Open

GitHub

Explore the latest AI Radar updates