> ## Documentation Index > Fetch the complete documentation index at: https://sandtable.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > sandtable is an open-source CLI, stable since 1.0 (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai `, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai --help`, trust the help output. > Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge. > To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional. > In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`. > Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs. # Get a customer's go-live sign-off on an AI agent > Turn a rehearsal into a page a customer's operations lead can read and sign: each rule of their procedure with its quoted source, the scenarios that checked it, which mistakes the rehearsal catches, and what it doesn't cover. The engineer's `report.html` answers "what failed and why". Before go-live, the customer asks something else: does the agent follow our procedure, how do you know, and what haven't you checked? `report --audience customer` answers that on one page, in plain words, from a run you already have. ## 1. Rehearse your agent Run the suite with several conversations per scenario, so a mistake that happens one time in four has a chance to show up: ```bash theme={null} npx sandtable-ai run ./my-domain --agent http://localhost:8787/step -k 4 -o runs/agent ``` The bundled freight example runs offline, so you can try every step here for free: ```bash theme={null} npx sandtable-ai run examples/freight --agent scripted:careful --user scripted -k 2 -o runs/agent ``` ## 2. Rehearse a careless agent A rule is **checked** in a conversation when it applied to what the agent did: the agent booked a load, so the rate ceiling was checked. That doesn't show the scenario would catch an agent breaking it. A booking at the posted rate checks the ceiling without testing it. The only evidence that your scenarios catch a mistake is a run in which an agent made it and the conversation failed. So rehearse a careless agent too. `--without-policy` makes one from a built-in model agent: the model keeps its tools, the generic instructions and each scenario's context, but not the customer's procedure. It acts sensibly without knowing their rules, so each rule it breaks that fails a conversation is one your scenarios catch. A soft rule it breaks is reported, never caught. It calls a model, so it costs money like any rehearsal: ```bash theme={null} npx sandtable-ai run ./my-domain --agent claude:claude-haiku-5-5 --without-policy -o runs/careless ``` Expect it to catch fewer rules than you might think. A capable model is cautious by default, and the tools' descriptions often carry the rules themselves ("Book a load for a vetted carrier"). On five freight tasks, Haiku without the policy still turned down the ineligible carriers and broke only one rule that fails a conversation: keeping the price ceiling to itself. An earlier or deliberately weakened version of your own agent usually makes more of the mistakes your scenarios are there to catch. Haiku is already the smallest Claude model; with `openai:` or `bedrock:` you can also try a smaller model of another family. Rules it never breaks stay *not shown caught* on the page, which is the honest answer. The do-nothing `noop` agent won't do, because it never acts. The freight example ships a careless agent of its own, so this step is free there: `scripted:naive` books without vetting and pays whatever is asked. ```bash theme={null} npx sandtable-ai run examples/freight --agent scripted:naive --user scripted -o runs/careless ``` A careless run fails, which is the point: `run` exits 1 on its failed critical scenarios. ## 3. Write the page ```bash theme={null} npx sandtable-ai report runs/agent/results.json --audience customer --probe runs/careless/results.json ``` It writes `customer-report.html` next to `results.json` (or where `-o` says): one static file with no script, which you can attach, print or save as PDF. The conversations that show a rule broken or caught are open, so they print from any browser. The lists of scenarios print open from Chrome or Edge, and other examples print only when you open them first. It holds: * **A summary** in plain words: how many scenarios passed every conversation, critical scenarios, rules checked and broken, what the careless agent was caught doing, and the **open items before go-live**. Open items are drafted items still pending review, conversations that weren't graded, failed critical scenarios, broken rules, and scenarios that didn't pass every conversation. * **Each rule** of the procedure, with the sentence of the customer's document it comes from. For each rule: * how many conversations checked it, and whether any broke it; * whether the rehearsal caught the careless agent breaking it, or the rule is *not shown caught*. A soft rule never fails a conversation, so it is reported, never caught; * the scenarios that checked it, and an example conversation. * **Every scenario** and its result, critical ones first. * **What the rehearsal doesn't cover**: rules no conversation checked, actions the agent never took, soft rules, and the limits of a simulation (simulated callers and records, and nothing graded beyond the written rules and outcomes). * **A sign-off block** for a name, a role and a date. Each rule's quote comes from its `source` in the spec, which `init --sop` writes when it drafts the rule. Add one to a rule you wrote by hand, with the document and the exact sentence: ```yaml theme={null} - id: rate-ceiling description: Never book above the load's max_rate. source: { file: policy.md, quote: "**Never book above `max_rate`.**" } limit: { tool: book_load, arg: rate, lte: { table: loads, key_arg: load_id, field: max_rate } } ``` A source changes nothing about grading, so stored runs still [regrade](https://sandtable.dev/reference/cli.md#regrade). ## 4. Close the gaps before you send it Read the page as the customer will, then fix what it shows: * **Open items**: fix the agent or the spec, then rerun. A drafted item pending review is the customer's rule, not yet confirmed: review it with them. * **Rules not checked**: add a scenario that gives the agent a reason to take the action the rule governs. `lint` lists the rules no scenario puts in play before you spend anything on a run. * **Rules not shown caught**: the careless agent never broke them, so no scenario has shown it catches that mistake. Add a scenario that tempts the agent, such as a caller who pushes past the limit, or a careless agent that makes the mistake. The decision to go live is the customer's. The page gives them the evidence and its limits, so they can make that decision. > [!NOTE] > > The page reads the run's [coverage](https://sandtable.dev/reference/cli.md#coverage), which runs record since sandtable-ai 1.1. For an older run, [`regrade`](https://sandtable.dev/reference/cli#regrade) it first. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.