TraceTrial-Bench: Agent Harness Reliability Evaluation (Trial by Trace)

A benchmark that scores programming/operations agents by splitting them into two layers: "model" and "harness." The harness is responsible for executing tools, managing context, requesting authorization, spawning sub-agents, and determining task completion. 95 tasks cover 7 reliability dimensions, including context, execution, governance, lifecycle, observability, tools, and verification. By collecting execution traces, it determines whether a failure should be attributed to the model or the harness.

A benchmark that scores programming/operations agents by splitting them into two layers: "model" and "harness." The harness is responsible for executing tools, managing context, requesting authorization, spawning sub-agents, and determining task completion. 95 tasks cover 7 reliability dimensions, including context, execution, governance, lifecycle, observability, tools, and verification. By collecting execution traces, it determines whether a failure should be attributed to the model or the harness.

Likes 253 · Code Stars 15

Rationale

  • The authors ran 3,420 trials across 4 types of harnesses (Claude Code, Codex, OpenClaw, DeepSeek Harness) × 9 models: harness scores ranged from 59.8% to 72.0%; 31.1% of the runs completed the task successfully but violated at least one harness specification, and only 8.4% passed both layers (according to the authors).
  • It answers "what is missed when looking only at task success rates": for example, tool errors being silently swallowed or the model bypassing harness defects are invisible in success rates.
  • When building agent harnesses or selecting among various programming agents, you can use the same set of tasks and the same model to compare harnesses horizontally.

Usage

  • Code and tasks: github.com/TraceLite-AI/TraceTrial-Bench (MIT), requires Python 3.11+ and Docker; use WSL on Windows. There is also an online leaderboard and a Hugging Face Space.
  • Getting started: After pip install -e ., copy configs/models.example.toml and fill in the model address and API key; build the harness image with tracetrial_bench image build --harness openclaw, then run a single task with tracetrial_bench run --task Task/C1-escalation-export-001 --harness openclaw --model <alias> --mode docker.
  • For the full benchmark, use --task-manifest Task/final.manifest and specify --judge-model; view the two-layer scores and attribution with tracetrial_bench report <run_directory>.

Risks

  • Scoring the harness layer relies on LLM-based evaluation, so the choice of judge model affects the results; all tasks call real model APIs within Docker, so running the full benchmark requires corresponding API quotas.
  • The verification dimension has only 1 tasks, resulting in thin coverage; all harness rankings and figures are reported by the authors.

Open · Code · Hugging Face Paper Page

Hugging Face Daily Papers

Explore the latest AI Radar updates