All benchmarksCustomer service agents In plain words Example taskSource
τ²-bench (tau-squared-bench)
Act as a support agent who follows company rules.
- Released
- Jun 2025
- Made by
- Sierra, with University of Toronto
- Score
- pass^k: how reliably a task succeeds across k repeated tries
Like a phone support rep who must follow policy while the customer also presses buttons on their own phone.
A customer's mobile data is slow. The agent must find the cause and guide them through checks such as airplane mode.
A conversation with a simulated customer. Both sides can use tools to change a shared system, like a phone account.How it is marked
The final state of the system, such as account and phone settings, is checked to see if the problem was really fixed.
Why it matters
Businesses deploy AI agents for customer service. This tests both following rules and guiding a real person.
Good to know
- Covers airline, retail and telecom; telecom proved hardest in the paper.
- The customer is played by another AI, which can make its own mistakes.
- A newer version, τ³-bench, adds banking knowledge and voice tasks.