Mage-VL-Cowork: Microsoft's 4B computer operation agent, capable of both clicking and writing code

A computer operation agent trained by the Microsoft Mage team on the Mage-VL 4B intermediate checkpoint (Mage-ViT visual encoder + Qwen3-4B). The model features two action channels: computer_use for viewing screenshots, clicking the mouse, and typing on the keyboard, and bash for running commands and using Python to process files. The model selects the channel for each step autonomously, without requiring an external scheduler. Training consists of five stages: first, separately practicing interface localization, interface operation, and terminal programming, followed by combined mixed SFT and online reinforcement learning. Officially, it achieves 66.0% on OSWorld-Verified (with a 50 step budget), marking the best performance at the 4B scale; online reinforcement learning improves performance by 10.16 percentage points over mixed SFT, while also reducing the number of calls and output length.

Original title: microsoft/Mage-VL-Cowork

A computer operation agent trained by the Microsoft Mage team on the Mage-VL 4B intermediate checkpoint (Mage-ViT visual encoder + Qwen3-4B). The model features two action channels: computer_use for viewing screenshots, clicking the mouse, and typing on the keyboard, and bash for running commands and using Python to process files. The model selects the channel for each step autonomously, without requiring an external scheduler. Training consists of five stages: first, separately practicing interface localization, interface operation, and terminal programming, followed by combined mixed SFT and online reinforcement learning. Officially, it achieves 66.0% on OSWorld-Verified (with a 50 step budget), marking the best performance at the 4B scale; online reinforcement learning improves performance by 10.16 percentage points over mixed SFT, while also reducing the number of calls and output length.

Why it matters Small models can now serve as computer operation agents, using scripts for file modifications and clicks for interface interactions, resulting in fewer steps

Rising score 65 · Likes 217 · Downloads 3714

Reasons

  • Structured files like tables, documents, and slides are modified via code, while screenshot-based clicking is reserved for canvas editing and real-time interface states, reducing step-by-step mouse clicks
  • The bf16 weights are approximately 9.48 GB; the official team provides an SGLang Docker image and transformers inference examples
  • The model card fully details the training pipeline and task synthesis method (Mage-CUA-Forge), licensed under Apache-2.0

Usage

  • Launch the service using SGLang (requires trust_remote_code; recommended context length 32,768, temperature 0.7, top_p 0.8), or use the official Docker image directly
  • Use the system prompt and the definitions of the two tools exactly as provided in the model card

Risks

  • Custom architecture llava_onevision2, which cannot be loaded as a standard Qwen3-VL
  • Performance metrics are self-reported by the official team; the model card only specifies English
  • The model directly operates the computer and executes shell commands; it is recommended to run it in a virtual machine or sandbox
  • This entry has not been tested; refer to the model card for actual performance

Open

Hugging Face Trending

Explore the latest AI Radar updates