Empowering agents to improve using their own past attempts without updating parameters: The same model first makes several attempts at a task, then a "coach" invocation synthesizes these execution logs and artifacts into a step-by-step guide (including specific paths, values, commands, and pitfalls to avoid). Finally, a "student" re-executes the task from the initial state, deciding whether to follow, adjust, or skip steps based on actual observations. The attempts, synthesis, and execution all use the same base model; there is no stronger teacher.
Likes 11 · Code Stars 0
Reasons
- The authors claim an average improvement of 9.9 percentage points (GPT series) and 8.8 percentage points (Claude series) over execution without guidance across four benchmarks: Terminal-Bench 2.0, Spider2-Lite, AndroidWorld, and BrowseComp-Plus.
- More token-efficient than directly injecting history: On BrowseComp-Plus, the synthesized guidance uses 98.2% fewer tokens (GPT) and 95.5% fewer tokens (Claude) than the modified AggAgent (as claimed by the authors).
- The README provides a cost comparison: A low-cost model with ReGE achieves an average success rate of 67.4% on 468 tasks, comparable to more expensive models, with a per-task cost approximately 28% of theirs (including the cost of generating attempts, calculated at list prices, as claimed by the authors).
Usage
- Code: github.com/TIGER-AI-Lab/ReGE (MIT, Python 3.11+), pip install -r requirements.txt; all model calls use an OpenAI-compatible interface, with keys stored in .env.
- One pipeline script per benchmark: scripts/run_tb2.sh gpt (Terminal-Bench 2.0), run_spider.sh, run_androidworld.sh, run_bcp.sh, followed by python scripts/summarize.py to aggregate success rates; replace gpt with claude to switch model series.
- The core lies in the coach prompt and guidance synthesis, found in src/guidance_moa_experiment.py and src/aggregate_deliverable.py, which can be adapted to your own multi-retry or parallel agent workflows.
Risks
- The premise is that the same task can be retried repeatedly and reset to its initial state; each question requires 4 attempts, resulting in a total cost approximately 5 times that of a single execution (README table). On the AndroidWorld subset in README table 5, the low-cost model with guidance actually shows a slight decline.
- Terminal-Bench requires Docker, AndroidWorld requires multiple Android emulators, and BrowseComp-Plus requires a self-hosted vector retrieval service, making environment setup heavy; all figures are reported by the authors.
Open · Code · Paper Page