Most existing MCP environment expansions follow a "first fabricate databases and scenarios, then derive tools and tasks" approach, resulting in tool interfaces that do not match reality and tool combinations that have not been executed. ToolSpace takes the opposite approach: using deterministic rules (without relying on LLM judgment) to mine three types of dependencies—data, state, and retry—from adjacent calls in real MCP services and open-source agent trajectories. It builds a weighted tool combination graph and clusters it into environments; it synthesizes databases only for environments that modify state, while replaying recorded return values for query-type and stateless tools. It then performs state-aware random walks with rollback, decompiling each call chain into natural language tasks with assertion checks, retaining only samples that can be successfully replayed from the initial state.
Likes 34 · Code Stars 28
Reasons
- The authors claim that on seven long-horizon tool benchmarks, the average score for Qwen3.6-35B-A3B rose from 23.0 to 41.0 (reaching 43.2 with RL), and GLM-5.2 rose from 44.6 to 56.6, comparable to Claude Opus 5; ablation studies show that "dependency edges observed during execution" significantly outperform edges judged by LLMs, with scores increasing as more data and environments are added.
- The same pipeline creates harder Pro versions of AutomationBench and tau3-Bench Banking; the authors claim that 7 frontier models saw score drops of up to 38.2 points; the repository releases task packages and evaluation scripts for AutomationBench Pro and Tau3 Banking Pro.
- Weights are available: Hugging Face dongguanting/ToolSpace-8B (MIT), ToolSpace-35B-A3B (SFT + Agent RL, Apache-2.0), and ToolSpace-744B-A40B (GLM-5.2 base, SFT only, MIT); Hugging Face daily likes 34 and code stars 28 (indicating attention only).
Usage
- Offline experience of the synthesis pipeline (no API Key required): cd Tool-Space && uv sync --locked, uv run python run_pipeline.py demo --output outputs/demo, then uv run python run_pipeline.py verify --tasks outputs/demo/tasks; environment execution and replay are real, while model responses use fixed examples.
- Deploying the model (model card example, 4 cards, 64K context window): vllm serve dongguanting/ToolSpace-35B-A3B --tensor-parallel-size 4 --max-model-len 65536 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3.
- Evaluation: cp benchmark/.env.example benchmark/.env and fill in the address of the model under test, cd benchmark && bash setup_all.sh, then SCOPE=smoke RUN_NAME=my-model-smoke bash AutomationBench/run.sh for a smoke test; for the Pro version, use evaluate.py validate / run under Automation-Pro/.
Risks
- Default online evaluation requires an API Key for the third-party relay service Boyue (user simulator, retrieval, and review models), as well as Serper / SerpAPI and a Hugging Face token; these can be replaced with other OpenAI-compatible services, but require manual configuration.
- The public version only uses AutomationBench as an example (47 services, 758 interfaces); the complete tool graph from the paper is not included; the README states that the paper describes 600 AutomationBench Pro tasks, but the public package contains only 300 frozen tasks.
- There is no LICENSE file in the repository root; the README only states that third-party components retain their respective licenses; the three checkpoints have different licenses, and all scores are reported by the authors.