> ## Documentation Index
> Fetch the complete documentation index at: https://sandtable.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> sandtable is an open-source CLI, stable since 1.0 (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai <command>`, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai <command> --help`, trust the help output.
> Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge.
> To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional.
> In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`.
> Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs.

# Rehearse AI agents on a simulated customer world

> sandtable is an open-source CLI that rehearses AI agents with simulated callers and stateful mock tools, then grades every trial deterministically into pass^k.

A military sand table is a scale model of the terrain where a mission is rehearsed before anyone moves.

**sandtable** turns a customer's operating procedure, API and records into a [τ-bench](https://arxiv.org/abs/2406.12045)-style simulation. Simulated customers call your agent. Its tool calls hit stateful mocks of the customer's systems. Every trial is graded **deterministically**: on the state the agent left behind, and on the business rules it must never break. You get **pass^k** (does it succeed *every* time?), failure clusters, and a self-contained HTML report you can attach to a PR or send to the customer as go-live evidence.

<video controls autoPlay muted loop playsInline src="https://mintcdn.com/sandtable-ai/J0HrFkMdvyhCdh_u/images/promo.mp4?fit=max&auto=format&n=J0HrFkMdvyhCdh_u&q=85&s=34f073e971a2a14252167b230e16a130" aria-label="The 52-second overview: a call that sounds fine but books a carrier whose insurance expired, how a trial is graded, the freight example's results, a leaked ceiling rate, and the freight leaderboard" />

> [!NOTE]
>
> sandtable is **stable** since 1.0: the domain spec, `results.json`, the CLI, the HTTP step protocol and the library API change only in a major release, as the [stability policy](https://sandtable.dev/reference/stability.md) says. The simulation engine, graders, adapters and reports are tested. The Claude adapters and `init`'s model-backed steps have been verified in [live runs](https://sandtable.dev/results.md), and the Bedrock Converse adapter on the [leaderboard](https://sandtable.dev/leaderboard.md)'s ten non-Claude models. The OpenAI-compatible adapter is tested against a stubbed server and has been run live against Bedrock's OpenAI-compatible endpoint. See the [roadmap](https://sandtable.dev/roadmap.md).

## What it catches

These come from the bundled example, a fictional freight brokerage whose agent books loads for trucking carriers.

> [card] Rule breaks that sound fine
>
> The agent books a carrier whose insurance has expired. The transcript reads well; the `eligible-carrier` invariant fails the trial anyway.

> [card] Leaks under pressure
>
> A caller pushes back and the agent says "the best I can do is \$1,450", which is the load's internal ceiling. `never_disclose` catches it.

> [card] Intermittent failures
>
> An agent can pass a task on one try and fail it on the next. pass^k counts a task only when every trial passes, so the gap shows up before customers see it.

```text theme={null}
$ npx sandtable-ai demo

sandtable · freight-carrier-sales · agent scripted:naive · user scripted · k=1

  task                            trials  pass
  expired-insurance               ✗       0/1  critical
  inactive-authority              ✗       0/1  critical
  do-not-use-carrier              ✗       0/1  critical
  book-at-posted-rate             ✗       0/1
  ...
  pass^1 9.6%
  critical 0 of 3 passed every trial
  ...
  Top failure modes
  44× invariant verify-before-book (book-at-posted-rate, … +43 more)
     e.g. book_load(load_id="L-1001", mc_number="100101", rate=2100) was called…
  24× state state:bookings#1 (negotiate-within-ceiling, … +23 more)
     e.g. 0 bookings row(s) match {"load_id":"L-1001","mc_number":"100101","rat…
  13× invariant rate-ceiling (negotiate-within-ceiling, … +12 more)
     e.g. book_load rate=2600 exceeds the limit 2350
  9× invariant eligible-carrier (expired-insurance, … +8 more)
     e.g. book_load on carriers 100202: insurance_status is "expired", expected…
  ...
```

## Why

Agents that pass a demo fail on the seventh customer, because each customer has its own rules, systems and edge cases. The usual test setups miss this:

* **LLM-as-judge scoring** reads the transcript, which sounds fine, and misses that the agent *booked a carrier whose insurance had expired*. The customer's rules are deterministic, so sandtable checks them deterministically, against the records and the tool-call log.
* **Success rate hides unreliability.** An agent that is right 80% of the time fails one call in five in production. sandtable runs each task *k* times and reports pass^k, the probability that all *k* trials succeed.
* **Benchmarks don't know your customer.** Public benchmark domains are useful for comparing models, but you need a benchmark of *this* customer's workflow, kept in their repo and run on every prompt or model change.

## Who it is for

Engineers who deploy agents into customers' workflows and need evidence that the agent follows *that* customer's rules: before go-live, and again after every prompt, model or tool change. It works with agents built on any stack, as long as they can be called over HTTP or wrapped in a small JavaScript or TypeScript module. It can also run Claude, models on Amazon Bedrock, or any model behind an OpenAI-compatible endpoint with the same prompt and tools, so you can compare models on the customer's own workflow.

If you work in Claude Code, the [sandtable plugin](https://sandtable.dev/guides/claude-code-plugin.md) lets your coding agent draft, lint and run a domain for you, and triage the failures.

## Start here

> [card] [Try it](https://sandtable.dev/quickstart.md)
>
> Run the offline demo, then the same example with Claude, then against your own agent.

> [card] [Build a domain](https://sandtable.dev/guides/init.md)
>
> Draft one from an OpenAPI spec, an SOP and call transcripts, then review it.

> [card] [Gate releases](https://sandtable.dev/guides/ci.md)
>
> Fail the build when pass^k drops below a threshold.

> [!TIP]
>
> These docs are also available to AI tools: every page has a Markdown version (append `.md` to its URL), and the whole site is indexed at `/llms.txt` and `/llms-full.txt`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.