A "dual-layer flywheel" training framework for computer-use agents. The outer layer connects open-world corpora with the model's current weaknesses to guide the next round of data generation and training; the inner layer first mines real-world workflows from instructional videos as seed tasks (reconstructing operation trajectories via two paths: semantic triggers in transcription text and visual triggers from screen changes), then abstracts these seeds into reusable "capability primitives" to synthesize new tasks with controllable quantity, difficulty, and verifiers. During trajectory collection, errors encountered in the real environment trigger snapshot rollbacks and retries based on feedback. The resulting CUA-Master-27B is based on Qwen3.5-27B, trained through multiple rounds of SFT and RL, and supports both pure GUI and hybrid GUI+CLI operations.
Likes 92 · Code Stars 86
Reasons
- The authors report scores on OSWorld Verified 80.2, OSWorld 2.0 36.9, WeaveBench 43.9, and Agents' Last Exam 25.0; the model card additionally lists WindowsAgentArena 56.2, MacOSWorld 43.0, and MacAgentBench 59.9, providing the average time per task (approximately 8 minutes for OSWorld Verified).
- Both the weights and the entire data pipeline are released: Hugging Face yxl66666/CUAMaster-27B, with the repository containing command-line tools for the full workflow from video retrieval, transcription, applicability scoring, trajectory reconstruction, and task synthesis to sandbox collection and ms-swift SFT data conversion.
- Ranked third in daily likes on Hugging Face (indicating attention only); the approach of "mining seed tasks from instructional videos" does not rely on self-generated model experience and can be used independently to generate training data for your own agents.
Usage
- Installation:
pip install -e ".\[seeding,annotator,collection,dev\]", thencp .env.example .envto fill in API keys and sandbox service addresses. - Data Generation:
python -m cuamaster.seeding.harvesting.search\_clito search for instructional videos →…harvesting.downloadto download →…transcription.transcribeto transcribe using faster-whisper →…scoring.scoreto assign scores from 0 to 5 →…reconstruction.constructto reconstruct seed trajectories;cuamaster-synthesizeto synthesize new tasks. - Trajectory Collection: First perform a smoke test without a virtual machine using
cuamaster-collect --config configs/collection/mock\_command.example.json; for formal collection, connect your own virtual machine orchestration via the SandboxProvider interface;cuamaster-datasetto export SFT data. - Model: Hugging Face yxl66666/CUAMaster-27B, with action coordinates normalized to [0,1000]; the model card advises running in a sandbox and only relaxing permissions after verifying results.
Risks
- The model card does not specify a license or provide inference deployment commands; the code is marked as Apache-2.0 in pyproject, but there is no LICENSE file in the repository root, so confirmation with the authors is required before commercial use.
- The complete workflow depends on external APIs (annotation models), video downloads, and self-built virtual machine sandboxes, resulting in high reproduction costs; the README for the RL section only mentions using task verifiers for outcome rewards, with no ready-made scripts provided.
- Scores are reported by the authors, with long tasks taking anywhere from a few minutes to over an hour per task; the agent modifies files and executes commands, so do not run it directly on your primary machine.