An agent benchmark simulating Indian retail banking customer service: The model under test acts as a bank assistant, while another model plays the customer making requests such as reporting lost cards, opening fixed deposits, or getting early withdrawal quotes. The assistant must check records, follow bank policies, and call tools in sequence. Unlike benchmarks that only look at the database state at the end of the conversation, this one also checks whether every required step was performed and whether any changes were made that the customer did not request.
Likes 8 · Code Stars 6
Reasons
- Captures a common evaluation loophole: The authors state that when only the final record is checked, every model has 6.5–12.0% of sessions judged as successful, but in reality, they skipped required steps or made unnecessary changes.
- Comprehensive scale and engineering: 1000 tasks, 50 scenario categories, 33 banking tools plus one human-transfer tool, 944 simulated customers, and 24 knowledge base documents; scoring does not require a judge model, the clock is frozen, citation numbers are determined by the seed, and it can be replayed byte-for-byte.
- The README provides reference scores for six open-source models (Qwen3.8-27B and the Gemma 4 series, all 1000 questions pass@1), broken down by sequential execution, edge cases, minimal tool usage, and control groups.
- Installs directly as the tau2 command, connecting to any OpenAI-compatible endpoint or hosted API supported by litellm; the code and data are MIT licensed, and all data is synthetic.
Usage
- Installation: Python 3.12/3.13, pip install -r requirements.txt → python scripts/fetch_data.py (fetches data from Hugging Face NPCI/tau-indian-banking) → pip install --no-deps -e . (uv is also supported).
- Running: tau2 run --domain indian_banking --agent-llm openai/<model-under-test> --user-llm openai/<customer-simulation-model> --task-split-name base …; reference settings include temperature 0, maximum 8192 output tokens per turn, and context of at least 32768.
- Scoring: python scripts/score.py data/simulations/<run>/results.json; add --compare to compare two runs; use tau2 view to browse conversations.
- To save costs for a trial run, use lite (200 questions) or --num-tasks 5; the README specifically warns that running the full set with paid APIs costs significantly more than a single request.
Risks
- The score is an attribute of the combination of "model under test + customer simulation model + data split"; it is not comparable if the customer simulation model is changed. The README states that with the same configuration, approximately 15–25% of sessions flip results on repeated runs, with mean fluctuations of about ±1–2 points.
- The domain is a fictional Indian bank with synthetic data; it is unverified whether the conclusions transfer to other industries or real-world customer service.
- Reference scores are only for open-source models, and all use the same customer simulation model (MiniMax-M2.7).