> ## Documentation Index > Fetch the complete documentation index at: https://sandtable.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > sandtable is an early-alpha, open-source CLI (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai `, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai --help`, trust the help output. > Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge. > To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional. > In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`. > Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs. # Invariants vs expectations: how trials are graded > How sandtable grades without a judge model: invariants check business rules before every tool call, and expectations check each task's end state. Every trial is graded by two kinds of checks. Both are deterministic: they read the world and the tool-call log, never a judge model's opinion. | | Invariants | Expectations | | - | - | - | | Declared | once, at the domain level | per task, under `expect` | | Apply to | every task | the task that declares them | | Express | rules the agent must never break | what success looks like for this conversation | | Checked | before every tool call, against the world at that moment; `never_disclose` on every agent message | at the end of the trial, against the final world and the call log | | Fail when | the agent *attempts* a forbidden action, even if the API would reject it | the end state or the call log does not match | ## Invariants: rules checked at decision time An invariant is evaluated before each tool call runs, against the world as it was when the agent decided. That timing matters: * **Attempting is the bug.** If the agent tries to book a carrier whose insurance has expired, the trial fails, even if a `check` then rejects the booking. An agent that relies on the API to stop it will eventually meet an API that doesn't. * **Order is visible.** `before` invariants see the call history, so "verify the carrier before booking it" fails when the verification comes after the booking, or was for a different carrier. * **The domain states the rule once.** It then holds in every task, including tasks written later, and tasks a drafting step adds. The six kinds: | Kind | Rule it expresses | Example | | - | - | - | | `before` | call A successfully before B, for the same arguments | verify a carrier before booking it | | `require` | the record an argument points at must match conditions | only book carriers with valid insurance | | `limit` | an argument stays within a bound, constant or from a record | never book above the load's `max_rate` | | `match` | two records referenced by the call agree on a field | carrier equipment matches load equipment | | `never` | never call a tool, or never with certain arguments | no refunds over \$500 | | `never_disclose` | never say a protected value in any message | never reveal the internal rate ceiling | The [spec](/reference/spec#invariants) has the exact syntax. ### Soft invariants: measure what the policy allows Some behaviour is allowed but worth measuring. The freight policy lets the agent offer a load's `max_rate` as a counter-offer, but an agent that always concedes to the ceiling leaves margin on the table. Mark such a rule `soft: true`: it is checked like any other, but never fails a trial. Runs report how often it was broken, out of the trials where it applied: ```text theme={null} Soft rules (reported, never fail a trial) counter-at-ceiling 37.5% 3 of 8 trials that called log_offer ``` Two agents with the same pass^k can differ here. Keep a rule hard whenever the policy forbids the behaviour. ## Expectations: what success looks like Expectations belong to one task and describe the outcome its conversation should reach: ```yaml theme={null} expect: state: - { table: bookings, where: { load_id: L-1001, mc_number: "100101" }, count: 1 } - { table: loads, key: L-1001, fields: { status: booked } } called: [verify_carrier, book_load] # each succeeded at least once not_called: [transfer_to_human] # never attempted reply_includes: ["confirmation"] # use sparingly ``` * `state` is the main signal: the records the conversation should have created or changed, or left alone (`count: 0`, or `key` with the original `fields`). * `called` and `not_called` cover actions that leave no record, such as a lookup that must happen. * `reply_includes` matches text in what the agent said. Wording varies between runs and models, so prefer state. ## Which one to use * If it should hold in **every** conversation, it is an invariant: "never book above the ceiling", "verify before booking". * If it is the **outcome** of one scenario, it is an expectation: "this caller ends up with L-1004 booked", "nothing is booked when the caller insists on \$1,600". * If you find yourself adding the same `not_called` to many tasks, it is probably an invariant (`never`, or `never` with `when`). * Keep invariants as strict as the SOP and no stricter. A rule like "never transfer to a human" written as an invariant fails correct handoffs everywhere. ## How failures show up Each failed check is recorded with a kind, an id and a readable detail, and the run clusters them into failure modes: | Kind | Id | Example detail | | - | - | - | | `invariant` | the invariant's id | `book_load rate=2600 exceeds the limit 2350` | | `state` | `state:[]#` | `loads L-1001: status is "available", expected "booked"` | | `called` | `called:` | `verify_carrier was never called successfully` | | `not_called` | `not_called:` | `transfer_to_human was called 1x` | | `reply` | `reply:` | `agent never said "confirmation"` | | `runtime` | `runtime` | the agent endpoint returned an error | ## Why no judge The customer's rules are deterministic, so sandtable checks them deterministically. That makes a failure reproducible ("it booked carrier 100202, whose insurance was expired"), explainable to someone who did not write the test, and free to evaluate. It also means sandtable does not grade tone, empathy or wording today. Judged soft scores are planned as a separate score that never changes pass/fail ([#16](https://github.com/sandtable-ai/sandtable/issues/16)). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.