Install
- npm
- Python launcher
- From source
Needs Node 20 or later.These docs run the CLI as
npx sandtable-ai <command>, which uses the copy installed in the project when there is one. Keep the -ai: npm has an unrelated package called sandtable, and without the suffix npx downloads and runs that package whenever sandtable-ai isn’t installed in the project.Run it
1
Run the offline demo
results.json and report.html under .sandtable/runs/ and opens the first report in your browser. Pass --no-open to skip that.2
Read the report
The HTML report is one self-contained file. For every task it shows each trial’s transcript, the tool calls with their arguments and results, every check that passed or failed, and the changes the agent made to the world. Read a failing trial before trusting any score: it tells you whether the agent, the task or the simulated caller was wrong.
3
Run it with Claude
Claude plays both the carrier and the agent, two trials per task by default.A cheaper model for the simulated caller keeps the cost down. The run holds one conversation per task and trial (
- Anthropic API
- Amazon Bedrock
-k, default 2), so its cost grows with both. For scale: on the example’s first 10 tasks, 20 trials, Claude Opus 5.5 as the agent and Claude Haiku 5.5 as the caller cost about $0.58 at list prices (results). The full example has more tasks and costs proportionally more.4
Point it at your own agent
Any agent that answers one POST per step can be rehearsed. Here is a toy one that only verifies the caller’s MC number:Start it in one terminal, then run the freight example against it in another:It scores pass^1 36.0% (18 of 50). It never books a load, so it passes only tasks where the right outcome is to book nothing. The noop baseline, an agent that never calls a tool, scores 32.0% (16 of 50) for the same reason. The toy agent passes two more tasks than noop,
double-brokering-on-yes and broker-claims-carrier: in both, the agent must look the carrier up with verify_carrier and then book nothing, and the toy agent does look it up. The noop score is the floor of a suite, and the first number to know about it.examples/freight works from any directory: when there is no such folder where you run the command, sandtable uses the copy bundled with the package and prints its path.Next
How it works
Worlds, tools, invariants, tasks and grading in one page.
Run against your agent
Claude, Bedrock, OpenAI-compatible, HTTP and JS/TS module adapters in detail.
Draft a domain with init
Build a domain for your own customer from their documents.
Use it in CI
Gate pull requests on pass^k.
Use it from Claude Code
Install the plugin and let your coding agent draft, lint and run a domain.