When multimodal large models perform robot or household planning, they often lack knowledge about when a specific action can be executed in the current environment and how the world state changes after execution. CausGround does not modify model parameters; instead, it distills interaction records (observations before and after actions, along with success or failure outcomes) into action-centric causal knowledge: the preconditions and state effects of each action are organized within a world state representation based on "object attributes + object relations." During planning, relevant dependencies are extracted from the graph according to the task, preconditions are back-chained, and effects are forward-chained, then provided as context to the large model.
Likes 3 · Code Stars 1
Reasons
- Training-free and model-agnostic: The authors report that on a modified VirtualHome, the average task success rate improved from 24.9% to 53.3%, RoboCasa closed-loop planning improved from 43.0% to 57.0%, and SWITCH interactive reasoning improved from 25.8% to 31.9%; the README table covers three backbones: Qwen3.5-9B, 27B, and 35B-A3B.
- You can build graphs for your own environment: Each record requires "pre-action observation, action, post-action observation, and result"; if you already have structured world states, only Python 3.10+ is needed, while text or image observations are automatically extracted using Qwen3.5-9B via vLLM.
- The code, VirtualHome interaction data (hosted on Hugging Face at YijunShen/CausGround), and RoboCasa graphs have all been released, with query results directly output as context.txt for use by the large model.
Usage
- Code: github.com/Intelligent-Interaction/CausGround; to build a VirtualHome graph: download trajectories.jsonl.xz, then run python -m causground build --input … --effect-policy virtualhome --output outputs/virtualhome.
- Retrieve task context: python -m causground query --graph … --task "Turn on the computer." --mode causal (causal back-chains from the goal, semantic uses text retrieval, hybrid combines both; semantic defaults to offline TF-IDF, but can also switch to Qwen3-Embedding-0.6B).
- Your own environment: python -m causground build --format text|image --backend vllm --model-path models/Qwen3.5-9B --input your_trajectories.
Risks
- The repository outputs context for planning models (context.txt / subgraph.json); the README does not include a complete planning-execution closed loop or evaluation scripts for the benchmarks in the paper, so reproducing the reported success rates requires custom integration.
- Sufficient interaction records are required to build the graph (VirtualHome used 60,957 records); the cold-start cost for entirely new environments is not specified; the README does not state the license.
- The performance improvements are as reported by the authors, and real-robot experiments are described only qualitatively.