# sandtable

> Open-source CLI that rehearses AI agents on a simulated copy of a customer's world: simulated callers, stateful mock tools, business rules checked on every tool call, deterministic grading and pass^k.

> ## Agent Instructions
> sandtable is an early-alpha, open-source CLI (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai <command>`, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai <command> --help`, trust the help output.
> Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge.
> To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional.
> In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`.
> Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs.

- [Rehearse AI agents on a simulated customer world](https://sandtable.dev/index.md): sandtable is an open-source CLI that rehearses AI agents with simulated callers and stateful mock tools, then grades every trial deterministically into pass^k.
- [Quickstart: rehearse your first AI agent in five minutes](https://sandtable.dev/quickstart.md): Install sandtable, run the offline freight demo with no API key, rerun it with Claude on both sides, then point it at your own agent over HTTP.
- [How sandtable rehearses and grades an AI agent](https://sandtable.dev/how-it-works.md): What happens in one trial and in one run: a fresh mock world, a simulated user with a hidden goal, your agent's tool calls and deterministic graders.
- [Domains: a simulated copy of one customer's world](https://sandtable.dev/concepts/domain.md): A domain is a folder of plain files for one customer workflow: the records, API, rules and conversations that sandtable uses to rehearse an agent.
- [Invariants vs expectations: how trials are graded](https://sandtable.dev/concepts/invariants.md): How sandtable grades without a judge model: invariants check business rules before every tool call, and expectations check each task's end state.
- [pass^k: measure how reliably an AI agent succeeds](https://sandtable.dev/concepts/pass-k.md): pass^k is the chance that all k trials of a task succeed. Why it matters for customer-facing agents, how sandtable computes it, and how to choose k.
- [Noop baseline and lint: check a suite before model runs](https://sandtable.dev/concepts/baselines.md): Know the floor before you read a score: run the do-nothing agent, and prove offline that every task is ready, solvable, non-vacuous and graded.
- [Review workflow: approve drafted rules and tasks](https://sandtable.dev/concepts/review-workflow.md): Drafted rules and tasks stay pending until a person accepts them. Use sandtable review, validate --strict and provisional reports to keep it that way.
- [Draft a domain from an OpenAPI spec, SOP and transcripts](https://sandtable.dev/guides/init.md): Use sandtable init to draft world tables, tools, invariants and tasks from a customer's OpenAPI spec, SOP and call transcripts, then review each item.
- [Connect your agent: Claude, Bedrock, OpenAI-compatible or HTTP](https://sandtable.dev/guides/bring-your-agent.md): Rehearse Claude, models on Amazon Bedrock, any OpenAI-compatible endpoint, an agent behind HTTP, or a JS/TS module, and choose the simulated user.
- [Test agents built with Vercel AI SDK, OpenAI Agents or Pipecat](https://sandtable.dev/guides/frameworks.md): Worked examples that rehearse agents built with the Vercel AI SDK, the OpenAI Agents SDK, the Claude Agent SDK, Pipecat, LiveKit Agents or a plain HTTP service.
- [Claude Code plugin: draft, lint and run sandtable from Claude](https://sandtable.dev/guides/claude-code-plugin.md): Install the sandtable plugin for Claude Code: three skills and a subagent that draft a domain, review and lint it, run it against your agent, triage failures.
- [Test AI agents in CI: gate pull requests on pass^k](https://sandtable.dev/guides/ci.md): Gate pull requests on agent reliability: free offline checks on every change, and a model-backed rehearsal that fails the build below a pass^k threshold.
- [Write tasks that separate good agents from careless ones](https://sandtable.dev/guides/write-tasks.md): Write tasks that a correct agent passes and a careless one fails: firm hidden goals, outcomes graded on state, balanced refusals and a reference solution.
- [Domain spec: every field of sandtable.yaml and tasks.yaml](https://sandtable.dev/reference/spec.md): Every field of sandtable.yaml and tasks.yaml, the files that define a domain: world tables, tools, where clauses, invariants, personas and tasks.
- [CLI reference: every sandtable command and flag](https://sandtable.dev/reference/cli.md): Every sandtable command and flag (init, validate, lint, review, run, leaderboard, demo, report and regrade) with provider options, environment variables and exit codes.
- [HTTP agent protocol: test an agent that runs as a service](https://sandtable.dev/reference/http-protocol.md): How sandtable talks to an agent that runs as a service, in any language: one POST per step, answered with a reply to the user or tool calls to run.
- [results.json schema: scores, trials and failure modes](https://sandtable.dev/reference/results-schema.md): The file every run writes: suite scores, failure modes, per-tag scores, usage, and every trial's transcript, tool calls, checks and world diff.
- [Freight leaderboard: how models do on a carrier-sales desk](https://sandtable.dev/leaderboard.md): How the freight leaderboard was run: one fictional carrier-sales domain, 50 rehearsed calls, four trials each, the same simulated caller for every model, deterministic grading, and how to reproduce it.
- [Live results: real Claude runs of the freight example](https://sandtable.dev/results.md): Measured pass^k, cost, latency and failure modes from real Claude runs of the bundled freight example, plus what sandtable init drafted from its sources.
- [Alternatives to sandtable: τ-bench, Scenario, IntellAgent](https://sandtable.dev/compare/overview.md): The closest projects to sandtable, from τ-bench to simulation frameworks, mock worlds and hosted platforms: what each is best at, and where sandtable fits.
- [sandtable vs τ-bench: your customer's workflow or a benchmark](https://sandtable.dev/compare/tau-bench.md): sandtable applies τ-bench's method to one customer's workflow, in their repo. How grading and domains differ, and when to use τ-bench, sandtable or both.
- [sandtable vs LangWatch Scenario: declared world or test code](https://sandtable.dev/compare/langwatch-scenario.md): LangWatch Scenario writes agent tests in code with a judge agent; sandtable grades a declared world deterministically. What each does best, and when to use it.
- [sandtable vs IntellAgent: a reviewed suite or generated tests](https://sandtable.dev/compare/intellagent.md): IntellAgent generates many scenarios from a prompt or policy and a model critiques them; sandtable grades a reviewed suite deterministically. When to use each.
- [sandtable vs hosted agent simulation and voice testing tools](https://sandtable.dev/compare/hosted-platforms.md): Simulation in Sierra and Decagon, voice testing in Vapi, Coval, Hamming and Cekura, and LangSmith or Braintrust evals, compared with a harness in your repo.
- [Roadmap: what sandtable does today and what comes next](https://sandtable.dev/roadmap.md): What works in sandtable today, what ships next and what comes later, release by release, with links to the GitHub issues that track each item.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.