Original title: Interactive-Active-AI/NextTransformer
Open-source training framework and experiment logs: Trained and compared recurrent depth, sparse attention, linear attention, residual stream variants, and hybrid architectures (GQA, NSA, MoBA, GDN, etc.) under identical conditions—same dataset, same 49.152B tokens, and same theoretical training compute per token—evaluating at the same checkpoints. The README states that 31 full training runs, 36 model cards, and complete protocols have been released, and provides an Arena for AI agents to research new architectures under the same rules; a request for comments on scaling to 200B and 400B tokens was initiated on 11 5.
Why it matters Experimental conditions vary across architecture papers; this project standardizes conditions and maintains auditable records, facilitating comparison and reproduction by researchers.
Stars 319 (+35 / 24h)
Reason
- 35 new stars in 24 hours, total stars 319 (reflects attention only).
Risks
- The README states that one round of pretraining for a model with approximately 1.4B parameters requires 2–4 days on a single node with 8 cards, making it difficult for individuals to reproduce
- The author of Arena v2 states it will be released before 11 14, but functionality is currently incomplete