Aurora2: An ultra-small MoE model pre-trained from scratch on a single GPU

AuroraAI-Research 11 18 uploaded research base model: approximately 4 billion total parameters, activating approximately 1900 thousand per token, 12 layers, hidden dimension 192, 32 routed experts per layer (top-1) plus 1 shared experts, context 2,048 tokens, self-trained byte-level BPE tokenizer with a 16,384 vocabulary (digits split character by character). Trained on 1000 billion tokens in approximately 36 hours on a single RTX Pro 6000, with data primarily consisting of ClimbMix web text, along with FineMath, Wikipedia, mathematical reasoning, and smoltalk dialogue data. Only pre-training was performed; no instruction fine-tuning or RLHF.

Original title: AuroraAI-Research/Aurora2-400M-A20M

AuroraAI-Research 11 18 uploaded research base model: approximately 4 billion total parameters, activating approximately 1900 thousand per token, 12 layers, hidden dimension 192, 32 routed experts per layer (top-1) plus 1 shared experts, context 2,048 tokens, self-trained byte-level BPE tokenizer with a 16,384 vocabulary (digits split character by character). Trained on 1000 billion tokens in approximately 36 hours on a single RTX Pro 6000, with data primarily consisting of ClimbMix web text, along with FineMath, Wikipedia, mathematical reasoning, and smoltalk dialogue data. Only pre-training was performed; no instruction fine-tuning or RLHF.

Why it matters Fully discloses architecture details, data ratios, and training time, enabling individual researchers to reproduce small-scale MoE experiments, Apache license

Rising score 31 · Likes 42 · Downloads 643

Reasons

  • The model card details all architectural techniques: hashed n-gram embeddings, embedding skip connections, value residuals, per-head attention gating, learnable routing biases, as well as load balancing and router z-loss coefficients
  • Token-preserving top-1 routing, using the Muon optimizer; suitable for researching small models, sparse expert routing, and training efficiency
  • The repository provides weights in three precisions: FP32, BF16, and FP16

Usage

  • Load AuroraAI-Research/Aurora2-400M-A20M with transformers (the tokenizer is in the repository root and automatically adds a BOS token at the beginning), use generate for continuation, with a temperature of 0.5 in the model card example
  • BF16 / FP16 weights are located in the bf16/ and fp16/ subdirectories respectively; the inference implementation uses BF16 activations, and a CUDA GPU supporting BF16 is recommended
  • For batch inference, use a batch size of 1 or batches without padding; beam search is not supported

Risks

  • This is a base model without instruction fine-tuning and cannot be used as a chat assistant; only approximately twenty million parameters are activated per token, resulting in very limited knowledge, reasoning, and long-range coherence
  • Primarily English data, with a context of only 2,048 tokens; the model card does not provide benchmark scores, and licenses for each training dataset must be verified before downstream use. This entry has not been run

Open

Hugging Face Trending

Explore the latest AI Radar updates