> ## Documentation Index > Fetch the complete documentation index at: https://sandtable.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > sandtable is an early-alpha, open-source CLI (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai `, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai --help`, trust the help output. > Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge. > To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional. > In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`. > Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs. # CLI reference: every sandtable command and flag > Every sandtable command and flag (init, validate, lint, review, run, leaderboard, demo, report and regrade) with provider options, environment variables and exit codes. ```text theme={null} npx sandtable-ai [options] ``` | Command | What it does | | - | - | | [`init`](#init) | draft a domain from a customer's artifacts | | [`validate`](#validate) | check a domain for schema and cross-reference errors | | [`lint`](#lint) | prove every task is ready, solvable, non-vacuous and graded, without model calls | | [`review`](#review) | list drafted items that still need a person to accept them | | [`run`](#run) | run every task k times against an agent and grade the results | | [`leaderboard`](#leaderboard) | rank several agents on one domain under the same simulated users, next to a do-nothing baseline | | [`demo`](#demo) | rehearse the bundled freight example | | [`report`](#report) | re-render the HTML report for a past run | | [`regrade`](#regrade) | re-score a past run's stored trials under the domain's current rules, without calling a model | `npx sandtable-ai --version` prints the version and `npx sandtable-ai --help` lists a command's options. From a clone of the repo, run `npm run sandtable -- `. `validate`, `lint`, `review`, `run` and `leaderboard` accept `examples/freight` and `examples/clinic` outside a clone too. When the current directory has no folder of that name, they use the copy bundled with the package and print its path on stderr. A folder of that name in the current directory always wins, and any other path is used as given. ## init Draft a domain from a customer's artifacts. Review the output before trusting it. See [Draft a domain with init](/guides/init). ```bash theme={null} npx sandtable-ai init --openapi api.yaml --sop policy.md --transcripts calls/ -o sandtable ``` | Option | Description | | - | - | | `--openapi ` | OpenAPI 3.x spec (YAML or JSON): drafts tables and tools, no model needed | | `--sop ` | SOP or policy document: drafts invariants with Claude, each quoting its source | | `--transcripts ` | call or chat transcripts (a file, or a directory of `.txt`/`.md` files): drafts personas and tasks with Claude | | `-o, --out ` | domain directory to create, or to add to when it already exists. Default: `sandtable` | | `-m, --model ` | Claude model for the drafting steps. Default: `claude-opus-5-5` | | `--force` | overwrite existing files when drafting from `--openapi` | | `--provider`, `--region`, `--aws-profile` | where Claude runs; see [provider options](#provider-options) | At least one of `--openapi`, `--sop` and `--transcripts` is required. They run in that order. Without `--force`, `--openapi` refuses to overwrite existing files. ## validate Check a domain for schema and cross-reference errors. ```bash theme={null} npx sandtable-ai validate ./my-domain [--strict] ``` | Argument / option | Description | | - | - | | `` | domain directory (with `sandtable.yaml`) or the file itself | | `--strict` | also fail while drafted items are pending review | ```text theme={null} ✓ freight-carrier-sales: 8 tables, 9 tools, 18 invariants, 50 tasks ``` A `⚠` line is a warning, not an error, and never fails `validate`, not even with `--strict`. It flags a `never_disclose` field that no tool returns and the policy doesn't state, which can never fire, since a value counts only once the agent has been shown it. ## lint Prove every task is ready, solvable, non-vacuous and graded, without any model call. See [The noop baseline and lint](/concepts/baselines). ```bash theme={null} npx sandtable-ai lint ./my-domain ``` | Argument | Description | | - | - | | `` | domain directory (with `sandtable.yaml`) or the file itself | Each task prints as `✓` (clean), `!` (warnings) or `✗` (errors), followed by whether it expects a write, is ready, is solvable (`solvable ?` when it has no `reference`), and whether the noop baseline passes it. ## review List drafted items that still need a person to accept them. See [Review workflow](/concepts/review-workflow). ```bash theme={null} npx sandtable-ai review ./my-domain ``` | Argument | Description | | - | - | | `` | domain directory (with `sandtable.yaml`) or the file itself | ## run Run every task k times against an agent and grade the results. ```bash theme={null} npx sandtable-ai run ./my-domain --agent http://localhost:8787/step -k 4 --fail-under 85 ``` | Argument / option | Description | | - | - | | `` | domain directory (with `sandtable.yaml`) or the file itself | | `-a, --agent ` | the agent under test; see below. Default: `claude` | | `-u, --user ` | the simulated user: `claude`, `claude:`, `openai:` or `scripted`. Default: `claude` | | `--openai-base-url ` | OpenAI-compatible endpoint for `openai:`. Default: `OPENAI_BASE_URL`, else `https://api.openai.com/v1`; the key comes from `OPENAI_API_KEY` | | `-k, --trials ` | trials per task; pass^k is reported for every k up to n. Default: `1` | | `-t, --tasks ` | comma-separated task ids to run (all by default) | | `-c, --concurrency ` | episodes run in parallel. Default: `4` | | `-o, --out ` | output directory for `results.json` and `report.html`. Default: `.sandtable/runs/-` | | `--no-html` | skip the HTML report | | `--fail-under ` | exit 1 if pass^k (k = `--trials`) is below this percentage, `0` to `100`. A trailing `%` is accepted | | `--open` | open the HTML report in the browser when the run finishes | | `--prices ` | extra model prices for the cost estimate: `{ "": { "input", "output", "cacheRead", "cacheWrite" } }` in USD per million tokens. For Bedrock, [`scripts/bedrock-prices.mjs`](https://github.com/sandtable-ai/sandtable/blob/main/scripts/bedrock-prices.mjs) in the repo writes this file from the AWS Pricing API, and with `--verified` takes the models it doesn't list from [`prices/bedrock.verified.json`](https://github.com/sandtable-ai/sandtable/blob/main/prices/bedrock.verified.json), prices copied from AWS's model pages with their source | | `--provider`, `--region`, `--aws-profile` | where Claude runs, and the AWS settings for `bedrock:`; see [provider options](#provider-options) | `--agent` accepts: | Spec | Agent | | - | - | | `claude`, `claude:` | the Claude reference agent | | `bedrock:` | any tool-capable model on Amazon Bedrock, through the Converse API | | `openai:` | any model behind an OpenAI-compatible Chat Completions endpoint | | `http://…`, `https://…` | your agent, over the [HTTP step protocol](/reference/http-protocol) | | a file path | a JS/TS module that default-exports an `AgentAdapter`, or a function returning one | | `noop` | the do-nothing baseline | | `scripted:naive`, `scripted:careful` | rule-based demo agents for the bundled freight example | | `scripted:clinic-naive`, `scripted:clinic-careful` | rule-based demo agents for the bundled clinic example | [Run against your agent](/guides/bring-your-agent) explains each adapter and the packages and credentials it needs. The run prints a summary (per-task results, pass^k, latency, cost, per-tag scores, the rate of each soft invariant and the top failure modes) and the paths of the files it wrote. See [results.json](/reference/results-schema). ## leaderboard Rank several agents on one domain under the same simulated users, next to a do-nothing baseline. ```bash theme={null} npx sandtable-ai leaderboard examples/freight -m models.yaml -k 4 \ --provider bedrock --region eu-west-1 --prices bedrock-prices.json ``` The models file (`models.yaml` above) lists the agents, with an optional shared caller and k: ```yaml theme={null} user: claude:claude-sonnet-5-5 # every model faces the same simulated callers k: 4 models: - { label: Claude Opus 5.5, agent: claude:claude-opus-5-5 } - { label: GPT-6.1 Sol, agent: bedrock:global.openai.gpt-6.1-sol } ``` The repo has the freight leaderboard's own, [`leaderboard/models.yaml`](https://github.com/sandtable-ai/sandtable/blob/main/leaderboard/models.yaml), and the Bedrock prices it runs with, [`prices/bedrock.eu-west-1.json`](https://github.com/sandtable-ai/sandtable/blob/main/prices/bedrock.eu-west-1.json). | Argument / option | Description | | - | - | | `` | domain directory (with `sandtable.yaml`) or the file itself | | `-m, --models ` | **required**: the models file, YAML: `{ user?, k?, models: [{ label, agent, effort? }] }`; `agent` takes the same specs as `run --agent`; `effort` (`low`, `medium`, `high`, `xhigh` or `max`) is for `claude:` agents only | | `-u, --user ` | simulated user for every model; overrides the models file. Default: `claude` | | `-k, --trials ` | trials per task; overrides the models file. Default: `4` | | `-t, --tasks ` | comma-separated task ids to run, for every model and the baseline. Default: all tasks | | `-c, --concurrency ` | episodes in parallel per model. Default: `4` | | `--openai-base-url ` | OpenAI-compatible endpoint for `openai:` entries | | `-o, --out ` | output directory. Default: `leaderboard/` | | `--no-resume` | rerun every model and replace its stored results | | `--prices ` | extra model prices for the cost column | | `--provider`, `--region`, `--aws-profile` | see [provider options](#provider-options) | Each model's full `results.json` and `report.html` go in their own folder, named after its label (`Claude Opus 5.5` → `claude-opus-5-5/`). Two labels that make the same folder name are refused before anything runs. `leaderboard.json`, `LEADERBOARD.md` and `leaderboard.html` rank them by pass^k, with pass^1, rule violations, the top failure, cost per trial and p95 turn latency. When the domain has soft invariants, a **Soft rules** column shows how often each model broke each one, such as `counter-at-ceiling 37.5% (3/8)`; it never affects the ranking. Cost per trial counts the agent's model only, since every model faces the same caller; `leaderboard.json` also has the total with the caller (`costUsd`, `costPerTrialUsd`). The do-nothing baseline is always added, so a `noop` entry in the models file is skipped. A model's existing results are reused when they match the run: the same agent spec, domain name, k, simulated caller and tasks, with no crashed trial, and the agent and the caller given the same inputs (the policy, the tools, the seed records and each task's caller instructions, which `results.json` keeps as an `inputs` hash). Each model's folder keeps the agent spec in `run.json` for this check. An interrupted or paid run then resumes instead of repeating work: models with matching results are reused, and models with none run. Stored results that don't match, or have no `run.json`, are never replaced, since they may be the only copy of paid conversations. The board stops before it runs anything and lists each model whose results it would replace, with why (for example `they are for k=4, not k=1`). Run a check of a few tasks, another k or another caller with another `-o`. To rerun a model, delete or move its folder first; `--no-resume` reruns every model and replaces all their results. Reused results are never ranked on the verdicts they were stored with: their conversations are graded again under the domain's current rules, as [`regrade`](#regrade) does, without calling a model. An edit to an invariant or an expectation then updates every model's `results.json` and `report.html`, and the output says how many verdicts changed. A model with crashed trials (expired credentials, throttling that outlasted the retries) shows as an error instead of a score, and the next run retries it unless it has stored results to reuse. So does the do-nothing baseline when the caller it runs under crashes. When every model shows as an error, because its agent couldn't be built or any of its trials crashed, `leaderboard` still writes the board, then points at the `✗ failed` lines that give each model's error, and exits 1, so a CI job that runs it fails. One model with a score is enough for exit 0, and the baseline doesn't count; a models file of `noop` entries only, which leaves just the baseline, also exits 0. Even under `--no-resume`, a run with crashed trials never replaces the model's stored results, which may be the only copy of paid conversations: it is written to a `crashed/` folder inside the model's folder. The model's next run without a crash removes that folder; while stored results are reused next to it, the output says it is there, so you can delete it before publishing the folder. Keep the caller fixed and disclose which model plays it, since it may favour its own family. A `claude:` entry can set `effort`, the Claude agent's reasoning effort: `{ label: Claude Sonnet 5.5, agent: claude:claude-sonnet-5-5, effort: high }`. Without it, a Claude agent runs at `medium`, as in `run`; a Claude caller always runs at `low`. An `effort` on any other agent is an error, since only the Claude agent has the setting. Each Claude model's `run.json` records the effort it ran at, and results stored at another effort aren't reused. A `run.json` from before effort was recorded counts as `medium`. `LEADERBOARD.md` and `leaderboard.html` show each Claude model's effort next to its name, as in `Claude Sonnet 5.5 (effort high)`, and its row in `leaderboard.json` has it as `effort`. Each model's `run.json` also records what ran its trials: `sandtable` (the `version`, and the git `commit` with `dirty: true` for uncommitted changes, when sandtable runs from its own clone), `provider` when the model or the caller is a Claude model, and `region` when either calls Bedrock. The region is `--region`, else `AWS_REGION` or `AWS_DEFAULT_REGION`, else the region in the AWS profile's config, and the leaderboard sends every Bedrock call there, Claude's included. The commit pins the prompts and the adapter settings, such as the caller's effort, the Claude agent's default effort and `max_tokens`. `leaderboard.json` records the `sandtable` that graded and ranked the rows, and `LEADERBOARD.md` and `leaderboard.html` name its version and commit. With `--prices`, `leaderboard.json` also records `prices: { file, sha256 }`: the file's path relative to the working directory, or only its name when it is outside it, so no home directory is published, and a sha256 of its bytes. Every cost column is priced with that file, reused rows included, so the board can be checked against it. The built-in Claude prices are code, pinned by the commit. A run from a checkout with uncommitted changes prints a warning. Stored results are reused only on the provider and region the model uses now, since price, routing and the served model can differ. Results from another provider or region, or with a `run.json` from before either was recorded, aren't reused, and the output says why (`they ran in eu-west-1, not us-east-1`). A model that uses neither, such as an `openai:` agent with a scripted caller, still reuses its results when they change. When Bedrock is in use and no region comes from `--region`, the environment or the AWS profile, `leaderboard` stops before it runs or rewrites anything. ## demo Rehearse the bundled freight carrier-sales example. ```bash theme={null} npx sandtable-ai demo # offline: two rule-based agents, no API key npx sandtable-ai demo --live --user-model claude-haiku-5-5 # Claude plays the caller and the agent ``` | Option | Description | | - | - | | `--live` | use Claude as both the agent and the simulated user (needs `ANTHROPIC_API_KEY` or `ant auth login`, or `--provider bedrock`) | | `-k, --trials ` | trials per task in `--live` mode. Default: `2` | | `--agent-model ` | model for the agent in `--live` mode | | `--user-model ` | model for the simulated caller in `--live` mode; a cheaper model keeps runs cheap | | `--no-open` | do not open the HTML report in the browser | | `--provider`, `--region`, `--aws-profile` | where Claude runs; see [provider options](#provider-options) | The offline demo runs the `scripted:naive` and `scripted:careful` agents against scripted callers and writes a report for each. The naive agent's report opens in the browser, and only when the output is a terminal. ## report Re-render the HTML report for a past run, for example after upgrading sandtable. ```bash theme={null} npx sandtable-ai report .sandtable/runs//results.json [-o report.html] ``` | Argument / option | Description | | - | - | | `` | path to a `results.json` | | `-o, --out ` | output HTML path. Default: `report.html` next to `results.json` | It prints the path it wrote. ## regrade Re-score a past run's stored trials after editing how the domain grades them: an invariant, an expectation or a `never_disclose` rule. No model is called, so a fixed grading rule doesn't cost a rerun. ```bash theme={null} npx sandtable-ai regrade .sandtable/runs//results.json --domain examples/freight ``` | Argument / option | Description | | - | - | | `` | path to a `results.json` | | `-d, --domain ` | **required**: the domain the run used, as edited since: a directory (with `sandtable.yaml`) or the file itself. `results.json` records only the domain's `name`, not where it is, so there is no default | | `-o, --out ` | output directory for `results.json` and `report.html`. Default: the results' own directory, replacing them | | `--no-html` | skip the HTML report | It replays each trial's tool calls in order against a fresh copy of the world, grades the conversation as a live run would, and recomputes every score. The transcripts, token usage and timing stay as they were recorded. It prints the summary, how many verdicts changed and which, and the new pass^k. It refuses, and writes nothing, when `--domain` is another domain: the `name` in its `sandtable.yaml` isn't the `domain` that `results.json` records. The error names both. Pass the domain the run used. If you renamed the domain since the run, restore its `name` or rerun it with `run`. It also refuses, and writes nothing, when the stored conversations no longer fit the domain: * the agent or the caller would be given something else now: the policy, a tool's schema, or a task's caller prompt, opening line or turn limit (and its script, for the scripted caller). `results.json` keeps a hash of them as `inputs`. * a tool call would now return something other than what the agent saw, for example after an edit to the seed records or to what a tool does. * a task it ran is gone. Rerun those with `run`. Tasks added since the run, with their seed records, don't block it unless they change what a stored tool call returns. The hash covers what sandtable's own adapters give a model. A custom agent adapter receives the whole domain, so a field it reads beyond the system prompt and the tool schemas is not covered: after editing one, rerun instead of regrading. ## Provider options `init`, `run`, `leaderboard` and `demo` accept: | Option | Description | | - | - | | `--provider ` | where Claude runs: `anthropic` or `bedrock`. Default: `SANDTABLE_PROVIDER`, else `anthropic` | | `--region ` | AWS region for Bedrock. Default: `AWS_REGION` | | `--aws-profile ` | AWS profile for Bedrock. Default: `AWS_PROFILE` | ## Environment variables | Variable | Used for | | - | - | | `ANTHROPIC_API_KEY` | Claude on the Anthropic API (agent, simulated user, `init`) | | `SANDTABLE_PROVIDER` | default for `--provider` | | `SANDTABLE_BEDROCK_CLIENT` | `classic` (default: cross-region inference profiles) or `mantle` (Bedrock's Messages-API endpoint) | | `AWS_REGION`, `AWS_DEFAULT_REGION` | Bedrock region when `--region` is not given | | `AWS_PROFILE` | AWS profile when `--aws-profile` is not given; other standard AWS credential variables also work | | `OPENAI_API_KEY` | bearer token for `openai:`; required only for the default OpenAI endpoint | | `OPENAI_BASE_URL` | default for `--openai-base-url` | ## Exit codes | Code | When | | - | - | | `0` | success | | `1` | an error (invalid domain, unknown agent or user spec, missing credentials or package), `validate --strict` with pending items, `lint` errors, `run` below `--fail-under`, a `run` or `demo --live` in which every trial ended in an error (missing credentials or package, an unreachable agent), or a `leaderboard` on which every model failed or crashed | This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.