npx sandtable-ai --version prints the version and npx sandtable-ai <command> --help lists a command’s options. From a clone of the repo, run npm run sandtable -- <command>.
validate, lint, review, run and leaderboard accept examples/freight and examples/clinic outside a clone too. When the current directory has no folder of that name, they use the copy bundled with the package and print its path on stderr. A folder of that name in the current directory always wins, and any other path is used as given.
init
Draft a domain from a customer’s artifacts. Review the output before trusting it. See Draft a domain with init.
At least one of
--openapi, --sop and --transcripts is required. They run in that order. Without --force, --openapi refuses to overwrite existing files.
validate
Check a domain for schema and cross-reference errors.⚠ line is a warning, not an error, and never fails validate, not even with --strict. It flags a never_disclose field that no tool returns and the policy doesn’t state, which can never fire, since a value counts only once the agent has been shown it.
lint
Prove every task is ready, solvable, non-vacuous and graded, without any model call. See The noop baseline and lint.
Each task prints as
✓ (clean), ! (warnings) or ✗ (errors), followed by whether it expects a write, is ready, is solvable (solvable ? when it has no reference), and whether the noop baseline passes it.
review
List drafted items that still need a person to accept them. See Review workflow.run
Run every task k times against an agent and grade the results.--agent accepts:
Run against your agent explains each adapter and the packages and credentials it needs.
The run prints a summary (per-task results, pass^k, latency, cost, per-tag scores, the rate of each soft invariant and the top failure modes) and the paths of the files it wrote. See results.json.
leaderboard
Rank several agents on one domain under the same simulated users, next to a do-nothing baseline.models.yaml above) lists the agents, with an optional shared caller and k:
leaderboard/models.yaml, and the Bedrock prices it runs with, prices/bedrock.eu-west-1.json.
Each model’s full
results.json and report.html go in their own folder, named after its label (Claude Opus 5.5 → claude-opus-5-5/). Two labels that make the same folder name are refused before anything runs. leaderboard.json, LEADERBOARD.md and leaderboard.html rank them by pass^k, with pass^1, rule violations, the top failure, cost per trial and p95 turn latency. When the domain has soft invariants, a Soft rules column shows how often each model broke each one, such as counter-at-ceiling 37.5% (3/8); it never affects the ranking. Cost per trial counts the agent’s model only, since every model faces the same caller; leaderboard.json also has the total with the caller (costUsd, costPerTrialUsd). The do-nothing baseline is always added, so a noop entry in the models file is skipped.
A model’s existing results are reused when they match the run: the same agent spec, domain name, k, simulated caller and tasks, with no crashed trial, and the agent and the caller given the same inputs (the policy, the tools, the seed records and each task’s caller instructions, which results.json keeps as an inputs hash). Each model’s folder keeps the agent spec in run.json for this check. An interrupted or paid run then resumes instead of repeating work: models with matching results are reused, and models with none run. Stored results that don’t match, or have no run.json, are never replaced, since they may be the only copy of paid conversations. The board stops before it runs anything and lists each model whose results it would replace, with why (for example they are for k=4, not k=1). Run a check of a few tasks, another k or another caller with another -o. To rerun a model, delete or move its folder first; --no-resume reruns every model and replaces all their results. Reused results are never ranked on the verdicts they were stored with: their conversations are graded again under the domain’s current rules, as regrade does, without calling a model. An edit to an invariant or an expectation then updates every model’s results.json and report.html, and the output says how many verdicts changed. A model with crashed trials (expired credentials, throttling that outlasted the retries) shows as an error instead of a score, and the next run retries it unless it has stored results to reuse. So does the do-nothing baseline when the caller it runs under crashes. When every model shows as an error, because its agent couldn’t be built or any of its trials crashed, leaderboard still writes the board, then points at the ✗ failed lines that give each model’s error, and exits 1, so a CI job that runs it fails. One model with a score is enough for exit 0, and the baseline doesn’t count; a models file of noop entries only, which leaves just the baseline, also exits 0. Even under --no-resume, a run with crashed trials never replaces the model’s stored results, which may be the only copy of paid conversations: it is written to a crashed/ folder inside the model’s folder. The model’s next run without a crash removes that folder; while stored results are reused next to it, the output says it is there, so you can delete it before publishing the folder. Keep the caller fixed and disclose which model plays it, since it may favour its own family.
A claude: entry can set effort, the Claude agent’s reasoning effort: { label: Claude Sonnet 5.5, agent: claude:claude-sonnet-5-5, effort: high }. Without it, a Claude agent runs at medium, as in run; a Claude caller always runs at low. An effort on any other agent is an error, since only the Claude agent has the setting. Each Claude model’s run.json records the effort it ran at, and results stored at another effort aren’t reused. A run.json from before effort was recorded counts as medium. LEADERBOARD.md and leaderboard.html show each Claude model’s effort next to its name, as in Claude Sonnet 5.5 (effort high), and its row in leaderboard.json has it as effort.
Each model’s run.json also records what ran its trials: sandtable (the version, and the git commit with dirty: true for uncommitted changes, when sandtable runs from its own clone), provider when the model or the caller is a Claude model, and region when either calls Bedrock. The region is --region, else AWS_REGION or AWS_DEFAULT_REGION, else the region in the AWS profile’s config, and the leaderboard sends every Bedrock call there, Claude’s included. The commit pins the prompts and the adapter settings, such as the caller’s effort, the Claude agent’s default effort and max_tokens. leaderboard.json records the sandtable that graded and ranked the rows, and LEADERBOARD.md and leaderboard.html name its version and commit. With --prices, leaderboard.json also records prices: { file, sha256 }: the file’s path relative to the working directory, or only its name when it is outside it, so no home directory is published, and a sha256 of its bytes. Every cost column is priced with that file, reused rows included, so the board can be checked against it. The built-in Claude prices are code, pinned by the commit. A run from a checkout with uncommitted changes prints a warning. Stored results are reused only on the provider and region the model uses now, since price, routing and the served model can differ. Results from another provider or region, or with a run.json from before either was recorded, aren’t reused, and the output says why (they ran in eu-west-1, not us-east-1). A model that uses neither, such as an openai: agent with a scripted caller, still reuses its results when they change. When Bedrock is in use and no region comes from --region, the environment or the AWS profile, leaderboard stops before it runs or rewrites anything.
demo
Rehearse the bundled freight carrier-sales example.
The offline demo runs the
scripted:naive and scripted:careful agents against scripted callers and writes a report for each. The naive agent’s report opens in the browser, and only when the output is a terminal.
report
Re-render the HTML report for a past run, for example after upgrading sandtable.
It prints the path it wrote.
regrade
Re-score a past run’s stored trials after editing how the domain grades them: an invariant, an expectation or anever_disclose rule. No model is called, so a fixed grading rule doesn’t cost a rerun.
It replays each trial’s tool calls in order against a fresh copy of the world, grades the conversation as a live run would, and recomputes every score. The transcripts, token usage and timing stay as they were recorded. It prints the summary, how many verdicts changed and which, and the new pass^k.
It refuses, and writes nothing, when
--domain is another domain: the name in its sandtable.yaml isn’t the domain that results.json records. The error names both. Pass the domain the run used. If you renamed the domain since the run, restore its name or rerun it with run.
It also refuses, and writes nothing, when the stored conversations no longer fit the domain:
- the agent or the caller would be given something else now: the policy, a tool’s schema, or a task’s caller prompt, opening line or turn limit (and its script, for the scripted caller).
results.jsonkeeps a hash of them asinputs. - a tool call would now return something other than what the agent saw, for example after an edit to the seed records or to what a tool does.
- a task it ran is gone.
run. Tasks added since the run, with their seed records, don’t block it unless they change what a stored tool call returns.
The hash covers what sandtable’s own adapters give a model. A custom agent adapter receives the whole domain, so a field it reads beyond the system prompt and the tool schemas is not covered: after editing one, rerun instead of regrading.
Provider options
init, run, leaderboard and demo accept: