MiniMind has open-sourced a complete training pipeline for an ultra-small language model with approximately 64M parameters, covering data cleaning, pre-training, supervised fine-tuning, LoRA, DPO, PPO / GRPO / CISPO, tool calling, multi-turn agent reinforcement learning, and distillation. It also includes a MoE version and extensions for vision and multimodal tasks. The author states that one epoch of the SFT stage takes about 2 hours on a single 3090. The weights are hosted on Hugging Face and ModelScope, with online demos available for experience.
Why it matters The core algorithms are written from scratch in native PyTorch, without relying on high-level interfaces from transformers, trl, or peft, making it ideal for understanding each step of large model training in detail.
Stars 65991 (+192 / 24h)
Reasons
- Gained 192 stars in 24 hours, reaching a total of 65991 stars; ranked #14 on both Trendshift and GitHub Trending that day (indicating attention only); licensed under Apache-2.0
- The README changelog shows the release of the minimind-3 series in 4, with new multi-turn Tool-Use reinforcement learning scripts added in 3
Usage
- Prepare datasets according to the README, then run training scripts for pre-training, SFT, etc., in sequence; for a quick experience, you can directly use the online Demo on ModelScope
Risks
- This is an educational-grade small model, with capabilities far inferior to general-purpose large models; the repository was created in 2024, and there have been no new major versions in recent months
- The README notes that weights for every experimental branch are not necessarily maintained continuously