A single researcher built two language models from scratch, covering everything from the data pipeline to the training kernels: Praxis (2.94B total parameters, 0.46B activated per pass) and Logos (10.23B total parameters, 1.18B activated per pass). Both use a Mixture-of-Experts (MoE) architecture mixing Mamba-2 and attention, processing each token through the entire network twice using the same weights ("take a second look"). Each was trained on 1.02T tokens with a context length of 131,072. The second half of the report details what went wrong: the configured optimizer didn't actually take effect, an n-gram memory module learned to disable itself, five fine-tuning choices degraded capabilities, and under group-mean normalization, length rewards penalized "long correct answers." The authors provide a proof and a one-line truncation fix.
Likes 4 · Code Stars 0
Reasons
- Clear efficiency focus: The authors claim the training data is about one-twelfth of the median for comparable public models, with fewer activated parameters per token, placing it on the Pareto frontier for both data volume and compute; it ranks second among 30 models in mathematical benchmarks (AIME 2025 67.50%).
- Ready to use: Apache-2.0 weights are on Hugging Face, along with FP8, NVFP4, GGUF, and MLX 4/8bit versions; the author's custom inference engine Dromos (Apache-2.0) supports NVIDIA, Apple silicon, and CPUs, providing an OpenAI-compatible interface.
- Valuable failure records: The issue where length rewards reverse sign when the group pass rate exceeds 2/3 is a directly checkable pitfall for those doing reinforcement learning for reasoning training.
Usage
- Fastest:
curl -fsSL https://github.com/lernex/dromos/releases/latest/download/install.sh | sh, thendromos run logos:fp8for terminal chat, ordromos serve logos:fp8to start an OpenAI-compatible interface (port 8080);dromos doctorshows which formats your machine can run. - Choose format by GPU (model card): Use BF16 for 32 GB or more, FP8 for 16/24 GB, NVFP4 for 8/12 GB (readable only by Dromos), MLX for Mac, and GGUF Q4_K_M (llama.cpp) for CPU / laptops.
- Transformers:
AutoModelForCausalLM.from_pretrained("Lernex/Metis-1.6-Logos", trust_remote_code=True), requires transformers 4.57 or higher; for thinking mode, recommend temperature 0.8 and repetition_penalty 1.05, not lower than 0.6.
Risks
- The authors explicitly state that code, instruction following, and long-context capabilities are weak; the average score across 40 benchmarks ranks seventh among the 17 models in its comparison group, within one point of Qwen3-4B.
- Dromos's decoding speedup figures over vLLM come from the author's tests on a single RTX 5070 Ti;
trust_remote_codeis required, and under llama.cpp / Ollama, the repetition penalty window must be adjusted according to the model card, otherwise thinking mode tends to loop. - Personal project; benchmarks were self-tested by the author within the same evaluation framework, with no third-party reproduction.