A preference learning method for compressing chain-of-thought length. Instead of broadly rewarding short answers, it identifies candidate states in the model's own reasoning trajectories where "the information is sufficient but the model keeps thinking." It freezes the reasoning prefix up to that step, regenerates shorter continuations from the same state with verified answers, forms preference pairs of "continue thinking vs. wrap up" under the same state, and then uses DPO training to teach the model when to stop. All training data is derived from the model's self-generated mathematical reasoning trajectories.
Likes 65 · Code Stars 6
Rationale
- The authors report that across five mathematical and three code benchmarks, mathematical output length decreased by 41.7%–45.4% and code output length decreased by 23.4%–36.3%, with accuracy remaining flat or improving; AUC was the best across all code benchmarks. Training solely on mathematical data also transfers to code.
- Weights are available: Hugging Face MuyaoYuan/LoCo-7B (a compressed version of DeepSeek-R1-Distill-Qwen-7B, MIT license), directly loadable in vLLM; the repository includes the full pipeline for data generation, training, and evaluation, with code under MIT license.
- Ranked third in daily likes on Hugging Face (indicating attention only); the data generation approach of "creating preference pairs only on overthinking states" is not tied to a specific model and can be adapted to other reasoning models.
Usage
- Direct use: Load MuyaoYuan/LoCo-7B in vLLM (model card example: max_model_len=36864, temperature 0.6, top_p 0.95, max_tokens 32768); the prompt requires step-by-step reasoning and placing the answer in \boxed{}.
- Generate your own data: Download the base model DeepSeek-R1-Distill-Qwen-7B and the judge model Qwen3-30B-A3B-Instruct-2507, then run the six scripts from 1_rollout to 6_build_dpo under construction/ sequentially on the math14k problem set included in the repository to obtain out/6_build_dpo/dpo.jsonl.
- Training and evaluation: Run bash script/train/run.sh (default 4 GPUs, 2 epochs, DeepSpeed ZeRO-2), use python train/merge.py to merge LoRA into the base model, and run bash script/eval/run.sh math500 or aime24 aime25 for evaluation.
Risks
- Only one 7B checkpoint is released; the base model is an earlier R1 distilled model. Switching to other reasoning models requires re-running data generation and training, and data generation also requires a 30B-scale judge model.
- The development environment uses NVIDIA H20, CUDA 12.8, and Python 3.12, with strictly locked dependency versions (vllm 0.23.0, trl 0.29.1, etc.).
- Length and accuracy figures are as reported by the authors; the paper only evaluated mathematical and code tasks, and there is no data on whether premature wrapping up occurs in open-ended dialogue.