Skip to main content
τ-bench (Yao et al., 2024, from Sierra) introduced the setup sandtable is built on. Each domain has a database, Python API tools over it and a Markdown policy. Each task gives an LLM-simulated user an instruction that is hidden from the agent, and the trial is graded on the final database state. The same paper introduced pass^k: “the chance that all k i.i.d. task trials are successful, averaged across tasks”. sandtable does not replace it. It takes the same method and makes it cheap to apply to one customer’s workflow, as a file in their repo. The code is in sierra-research/tau-bench and its successor sierra-research/tau2-bench, both MIT-licensed (LICENSE).

​
Different jobs

τ-bench is a benchmark with a public leaderboard. Its domains are hand-built so that models and agents can be compared on the same tasks, and each version added new ones:
  • τ-bench: airline and retail.
  • τ²-bench: adds a telecom domain with dual control, where the simulated user also has tools and changes the shared environment.
  • τ³-bench, released as version 1.0.0 of tau2-bench: adds a banking domain in which the agent works from about 700 knowledge documents (τ-Knowledge), and a full-duplex voice mode (τ-Voice).
sandtable is a harness for your own deployment. The domain is the customer’s API, SOP and records. The question is not “which model is best” but “is this agent ready for this customer, and did the last prompt change break a rule?”

​
How grading differs

Both grade deterministically on the world’s end state, apart from the natural-language assertions some τ²/τ³ tasks use, so that is not the main difference. The differences are where the expected outcome comes from, and when rules are checked.
  • τ-bench replays a reference sequence of tool calls on a fresh database and compares the agent’s final database with the result. It also checks that required values appear in the agent’s replies (paper, section 3). τ²-bench and τ³-bench let each task combine database checks, assertions on the environment, required replies, required actions and natural-language assertions (evaluation docs). All of these are deterministic except the natural-language assertions, which an LLM judges. Some τ³ retail tasks use them.
  • sandtable has no reference trajectory. Each task declares the end state it expects: the records that should exist, change or stay untouched. The domain declares invariants, which are checked before every tool call runs, against the world at that moment. An agent that attempts a forbidden action fails the trial, even if the API would reject it and even if the final state looks right.
What τ-bench’s approach does better: comparing the whole database catches any write that nobody thought to check. A sandtable task checks only the records it declares, plus the invariants, so the task has to say what should stay unchanged. In τ-bench, policy rules count through the outcome. Some restrictions are enforced by the tool code, which returns an error, and others are not. The paper says so directly: a passing trial “might be a necessary but not sufficient condition for a successful episode e.g., the agent might issue the return without explicit user confirmation, which violates the policy”. tau2-bench also has an optional LLM review that flags agent and user errors in each conversation (tau2 review, CLI reference). It is separate from the reward (reviewer.py).

​
Side by side

​
Choose τ-bench when

  • You are comparing models or agent architectures on shared, public tasks, and want numbers others can reproduce and compare on the leaderboard.
  • You need what its newer domains test: a user who also acts on the environment (telecom), work from a large set of knowledge documents (banking), or full-duplex voice.
  • You want every write checked against a reference database, not only the records a task names.
  • You want a Gymnasium environment with train and test splits, to train agents.
  • You are doing research on tool-agent-user interaction.

​
Choose sandtable when

  • The domain is a specific customer’s systems and rules, and it changes with them.
  • You need hard rules (verify before booking, never exceed a ceiling, never reveal a field) checked at the moment of each call, so an attempt fails even when the API would have refused it.
  • You want the suite in the customer’s repo, gated in CI with --fail-under, with a report you can hand over.

​
Use both

Run τ-bench to choose a model family, then run sandtable on the customer’s domain to decide whether a specific agent and prompt are ready to go live. Exporting sandtable domains to τ-bench’s format is on the roadmap (#18). If you publish results produced with sandtable, please cite τ-bench too (CITATION.cff). Last reviewed October 2026. Something out of date or unfair? Open an issue.