Long-context reinforcement learning requires data that is long, difficult, and automatically verifiable. Manual annotation is expensive, external corpora are insufficient, and relying on stronger teacher models is becoming increasingly difficult. LongZero allows the model being trained to generate its own questions: it first writes questions, supporting evidence, and answers in a compact structured format, then expands the evidence into multiple long documents to create multi-hop reasoning questions; it also adds multi-turn retrieval questions from long conversations constructed using the model's own responses. Only questions where the model sometimes answers correctly and sometimes incorrectly are retained, and training is performed using GRPO.
Likes 4 · Code Stars 4
Reasons
- Does not rely on external corpora or stronger teacher models: As the model being trained becomes stronger, the difficulty of the data also increases, generated by the model itself, which the authors consider a sustainable path.
- The authors claim that on two models and five long-context reasoning benchmarks (LongBench multi-document QA, LongBench v2, FRAMES, AA-LCR, LongReason), it outperforms baselines using reinforcement learning with synthetic data from external corpora (the abstract does not provide specific numbers).
- High reproducibility: Data synthesis, training (a modified verl snapshot), and evaluation scripts are all provided. Synthetic training data for Qwen3-4B-Thinking-2507 and Qwen3-8B is hosted on Hugging Face, and the results/ directory contains per-question scores and recalculation scripts.
- Code is MIT licensed.
Usage
- Code: github.com/zhennan1/LongZero; Data: hf download zhennan1/LongZero --repo-type dataset --include "data/*" (each model has 2,304 reasoning + 1,152 retrieval training samples).
- Environment: Linux, Python 3.12, CUDA; The versions recorded in the README are PyTorch 2.9.0+cu128, vLLM 0.11.1, FlashAttention 2.8.3, followed by pip install --no-deps -e training/verl_4b (or verl_8b).
- Training: MODEL_PATH=… DATA_DIR=data/qwen3_4b bash training/train_4b.sh; Single node with 8 GPUs, 64 prompts × 8 responses per step, learning rate 1e-6, the paper reports taking step 72 for the 4B model and step 108 for the 8B model.
- Generating your own data: First start the model being trained with vllm serve, then run synthesis/gen_pipeline.py and repeat_eval.py to filter questions.
Risks
- Trained model weights are not released, only data and code are provided; you must train it yourself to use it.
- The training threshold is a single node with 8 GPUs; validation was performed only on two models: Qwen3 4B / 8B.
- AA-LCR evaluation requires an OpenAI-compatible judge model API; performance conclusions are reported by the authors, and the abstract does not provide specific improvement margins.