Skip to main content
sandtable borrows its method from τ-bench and aims at a different job: rehearsing one customer’s deployment, in that customer’s repo, on every change. This page lists the closest projects and what each one is best at. Each claim links to the project’s own docs, repo or paper, so you can check it. If something here is out of date or unfair, please open an issue.

τ-bench

The method sandtable builds on, and a benchmark for comparing models.

LangWatch Scenario

Agent tests in code, with a simulated user and a judge agent.

IntellAgent

Many scenarios generated from a prompt or policy document, critiqued by a model.

Hosted platforms

Simulation inside agent platforms, voice testing services and eval platforms.

​
Method

  • τ-bench, τ²-bench and τ³-bench (Sierra Research, MIT: tau-bench, tau2-bench). τ-bench introduced the setup sandtable uses: a policy, tools over a database, an LLM-simulated user with an instruction hidden from the agent, grading on the final database state, and pass^k. Its domains (airline, retail, then telecom in τ²-bench and banking and voice in τ³-bench) are hand-built benchmarks for comparing models, with a public leaderboard. Adding a domain means writing Python plus a policy, a database and tasks as data files. Use it to compare models on public domains; use sandtable to build and gate a domain for your own customer. Exporting sandtable domains to τ-bench’s format is on the roadmap (#18).
  • τ^τ-bench (Hyper-τ) (Sierra Research, MIT: repo, paper). Evaluates how well a coding agent builds a customer-service agent from evidence such as policy documents, support transcripts and a client REST API, then scores the built agent with τ³-bench. It benchmarks agent builders. It is not a tool for testing your own deployment.

​
Open-source simulation frameworks

  • IntellAgent (Plurai, Apache-2.0). Takes the agent’s system prompt or a policy document, plus its tools and a database schema, builds a policy graph, and generates many scenarios with database rows at varying complexity. It simulates the user and has an LLM critique each conversation. sandtable drafts a smaller suite from the customer’s artifacts, has a person review it, and grades deterministically. Comparison.
  • LangWatch Scenario (Apache-2.0). A library for agent tests in code, with a user simulator agent and a judge agent, inside pytest, Vitest or Jest. It also tests voice agents and has a red-team agent, and pairs with the LangWatch platform. Comparison.
  • DeepEval’s ConversationSimulator (Apache-2.0). Role-plays a user from a scenario, a persona and an expected outcome against your chatbot, then scores the conversation with DeepEval’s multi-turn metrics. Tool state lives in your own application (model callback).
  • promptfoo’s simulated user provider (MIT). Multi-turn conversations between your agent and a simulated user, inspired by τ-bench, inside promptfoo’s general eval framework. You mock tools with your own callbacks, and --repeat runs each test several times.
  • ArkSim (Arklex, Apache-2.0). Synthetic users with their own profile, goal and knowledge level hold multi-turn conversations with your agent. Every turn is scored on metrics such as helpfulness, faithfulness and goal completion, and it runs in CI as a quality gate.
  • Rehearsal (MIT, with a hosted service; the Python package is rehearsal-kit). Points at an application’s source repository and published container images, starts the real application in an isolated sandbox with fictional data, and decides every outcome by reading the application’s own database, with no model grading. It makes some requests fail on purpose, and an improvement agent proposes the next agent version from failed jobs. It is the closest in spirit. It fits best when you own the application’s code and images. sandtable targets deployments where you only have what a customer hands over (API docs, SOPs, transcripts), and keeps the world declarative, local and in the repo.

​
Mock servers and simulated worlds

Classic API mocks either keep no state between requests (Prism) or keep state you configure endpoint by endpoint: WireMock scenarios, Microcks scripts, or Mockoon CRUD routes over in-memory data buckets. They are built for API consumers, not for grading an agent against business rules. Newer projects build worlds for agents:
  • faultbench (Apache-2.0, pre-alpha) is the nearest of these to sandtable. You declare fake, stateful services in a YAML file, it serves them as MCP tools with injected faults, records every call, and lets you assert on the world’s end state from pytest, over repeated runs.
  • mockworld (Apache-2.0) runs fake services such as a payment provider, an inbox or a CRM as local MCP servers, with seeded, deterministic state and faults.
  • mockd (Apache-2.0) is a multi-protocol mock server with stateful CRUD, chaos injection and an MCP server that lets AI assistants create mocks.
sandtable keeps state minimal and declarative because its job is grading: the world is the thing being asserted on, with rules checked before each call and simulated callers driving the conversation. A black-box mode that serves the world over REST and MCP is planned (#14).

​
Runtime guardrails

Guardrail layers such as Invariant Guardrails (Apache-2.0) check rules on tool calls in production, between the application and its MCP servers or LLM provider. They enforce rules on live traffic; sandtable tests whether the agent follows the rules before it ships. They combine well.

​
Commercial platforms

  • Sierra (Simulations) generates tests from a customer’s SOPs, knowledge bases and transcripts and runs them on every change, inside its own platform. Decagon (Simulations, Agent Versioning) generates tests from its Agent Operating Procedures and production conversations and tests every agent version.
  • Coval, Hamming, Cekura, Bluejay, Roark and Vapi do simulation and regression testing for voice agents, and several for chat agents too.
  • LangSmith and Braintrust offer datasets, judges, repeated trials and CI gating for LLM apps generally.
Details, and when to choose each, are on the hosted platforms page.

​
Where sandtable fits

Every rule init --sop drafts quotes the SOP sentence it comes from and stays review: pending until a person accepts it (guide), so a failing trial points at a sentence in the customer’s own procedure.

​
What sandtable does not do (yet)

A fair comparison includes the gaps. Today sandtable is text-only (no audio, timing or interruption testing), has no LLM-judged soft scores for tone or wording, no production monitoring, no test generation from production traffic, and no hosted UI: reports are files. It is early alpha. See the roadmap for what is planned. Last reviewed October 2026.