Original title: MinimaLabs/KLM-1.6
MinimaLabs' 99M parameter English base model, continued training from the checkpoint of KLM 1.5 before decay. The only major change is extending the learning rate decay phase from 20B tokens to 100B tokens. Training consists of three stages: first, warm-up using Token-Superposition Training (where each input position is the average of 7 consecutive tokens), which the authors state covers 128B tokens with approximately 18B compute; then training for 80B tokens at a constant learning rate; finally, decaying over 100B tokens. The architecture is a hybrid of 24 layers of Gated DeltaNet-2 and attention 3:1, along with n-gram memory for 12.5 ten-thousand bigrams and 3.8 ten-thousand trigrams. The authors report an intelligence index of 33.95 on the Open SLM leaderboard, ranking first in all three tiers below 100M, 150M, and 250M.
Why it matters A training recipe sample for models under one hundred million parameters: token superposition warm-up, linear attention hybrid architecture, and n-gram memory are all documented in the model card.
Upvotes 51 · Likes 51 · Downloads 666
Reasons
- Training data sources, three-stage training process, optimizer settings, and architecture parameters are fully documented in the model card, facilitating reproduction of small model training.
- Pure PyTorch implementation, requiring only transformers; can run on CPU.
Usage
- Load and continue text generation using AutoModelForCausalLM.from_pretrained("MinimaLabs/KLM-1.6", trust_remote_code=True).
- It is a base model; fine-tuning is required for dialogue or reasoning tasks. The tokenizer has reserved special tokens such as <|im_start|> and <think>.
Risks
- Supports English only, with a context length of 2,048 tokens; the authors state that knowledge is thin and stated facts are often incorrect.
- No safety alignment has been performed; you must add your own filtering before direct user use.
- Leaderboard comparisons are cited by the authors and have not been independently run for this entry.