Original title: AuroraAI-Research/Aurora2-400M-A20M
AuroraAI-Research 11 18 uploaded research base model: approximately 4 billion total parameters, activating approximately 1900 thousand per token, 12 layers, hidden dimension 192, 32 routed experts per layer (top-1) plus 1 shared experts, context 2,048 tokens, self-trained byte-level BPE tokenizer with a 16,384 vocabulary (digits split character by character). Trained on 1000 billion tokens in approximately 36 hours on a single RTX Pro 6000, with data primarily consisting of ClimbMix web text, along with FineMath, Wikipedia, mathematical reasoning, and smoltalk dialogue data. Only pre-training was performed; no instruction fine-tuning or RLHF.
Why it matters Fully discloses architecture details, data ratios, and training time, enabling individual researchers to reproduce small-scale MoE experiments, Apache license
Rising score 31 · Likes 42 · Downloads 643
Reasons
- The model card details all architectural techniques: hashed n-gram embeddings, embedding skip connections, value residuals, per-head attention gating, learnable routing biases, as well as load balancing and router z-loss coefficients
- Token-preserving top-1 routing, using the Muon optimizer; suitable for researching small models, sparse expert routing, and training efficiency
- The repository provides weights in three precisions: FP32, BF16, and FP16
Usage
- Load AuroraAI-Research/Aurora2-400M-A20M with transformers (the tokenizer is in the repository root and automatically adds a BOS token at the beginning), use generate for continuation, with a temperature of 0.5 in the model card example
- BF16 / FP16 weights are located in the bf16/ and fp16/ subdirectories respectively; the inference implementation uses BF16 activations, and a CUDA GPU supporting BF16 is recommended
- For batch inference, use a batch size of 1 or batches without padding; beam search is not supported
Risks
- This is a base model without instruction fine-tuning and cannot be used as a chat assistant; only approximately twenty million parameters are activated per token, resulting in very limited knowledge, reasoning, and long-range coherence
- Primarily English data, with a context of only 2,048 tokens; the model card does not provide benchmark scores, and licenses for each training dataset must be verified before downstream use. This entry has not been run