> ## Documentation Index > Fetch the complete documentation index at: https://sandtable.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > sandtable is an early-alpha, open-source CLI (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai `, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai --help`, trust the help output. > Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge. > To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional. > In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`. > Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs. # Domain spec: every field of sandtable.yaml and tasks.yaml > Every field of sandtable.yaml and tasks.yaml, the files that define a domain: world tables, tools, where clauses, invariants, personas and tasks. A domain lives in a directory with a `sandtable.yaml`. Any top-level value that is a relative path to a `.md`, `.yaml` or `.yml` file is inlined, so you can keep `policy: ./policy.md` and `tasks: ./tasks.yaml` in separate files. A path to a missing file is a validation error, so a misspelled path can't become the policy text. Run `npx sandtable-ai validate ` after every edit. It checks the schema and every cross-reference (tables, tools, params, personas). ```yaml sandtable.yaml theme={null} name: string # required description: string # who the agent works for; shown to both agent and simulated user policy: string | ./file.md # required: the SOP the agent must follow (the reference agent's system prompt) max_turns: 20 # user turns per trial before it is cut off world: { tables: ... } # required tools: [...] # required, at least one invariants: [...] personas: [...] tasks: [...] | ./tasks.yaml ``` > [!NOTE] > > The schema is strict: an unknown key anywhere is a validation error, not a silent no-op. A typo such as `invariant:` instead of `invariants:` fails `sandtable validate`. ## Templates Inside tool `ops`, `checks`, invariant `where` clauses and task assertions: * `"$arg"` is replaced by the tool call's argument value, keeping its type. * `"${arg}"` interpolates inside a longer string: `"Load ${load_id} is not available"`. * An argument's name is a letter or `_`, then letters, digits or `_`. Any other string is literal text, even when it starts with `$`: `"$5"`, `"$$"`, `"$/hr"`, `"$USD/kg"`, `"$@type"`. A whole string such as `"$USD"` is a reference. * In a tool's `ops` and `checks`, every reference must name one of the tool's params, or `sandtable validate` fails with `never resolves: tool "get_load" has no param "loadId"`. It would be empty at run time. * A condition whose argument was not passed (an optional param) is dropped, so it filters nothing. ## Where clauses `where`, `fields` and `when` are maps of field to condition. A plain value means equality. An object of operators means all of them must hold: | Operator | Meaning | | - | - | | `$eq`, `$ne` | equal / not equal | | `$in`, `$nin` | in / not in a list | | `$gt`, `$gte`, `$lt`, `$lte` | numeric comparison (strings compare lexically, so ISO dates work) | | `$exists` | `true`: the field is set and not null; `false`: it is missing or null | | `$contains` | case-insensitive substring | ## world ```yaml theme={null} world: tables: loads: key: load_id # primary key field rows: # seed data; each trial starts from a fresh copy - { load_id: L-1001, origin: "Dallas, TX", status: available, max_rate: 2350 } bookings: key: booking_id # created rows without a key get "-" ``` Every seed row needs its key field, and keys must be unique within a table. ## tools A tool is what the agent sees (name, description, JSON-schema params) plus what it does to the world. ```yaml theme={null} - name: book_load # ^[a-zA-Z0-9_-]{1,64}$ description: What it does, written for a model. params: load_id: { type: string } # string | number | integer | boolean rate: { type: number, description: "All-in USD" } note: { type: string, required: false } equipment: { type: string, required: false, enum: [dry_van, reefer] } status: { type: string, required: false, default: available } # used when the agent omits it checks: # preconditions the real API enforces; failing one = tool error, no writes # (skipped when the key comes from an optional arg that was not passed) - { table: loads, key: $load_id, where: { status: available }, error: "Load ${load_id} is not available" } ops: # run in order, atomically (all or nothing) - create: bookings values: { load_id: $load_id, rate: $rate } - update: loads key: $load_id set: { status: booked } returns: 0 # which op's result the agent gets back (default 0) ``` | Op | Effect | Returns | | - | - | - | | `get:
, key` | read one row; error if missing | the row | | `list:
, where?, limit?` | filter rows | array of rows | | `create:
, values` | insert a row | the new row | | `update:
, key, set` | merge fields into a row; error if missing | the updated row | | `respond: ` | no state change | the value (templated) | Arguments are validated before anything runs. Unknown argument names, missing required ones and wrong types come back to the agent as tool errors. Representations are coerced leniently: `"2,100"` becomes `2100` for a number param, and `100101` becomes `"100101"` for a string param. There is no `delete` op, and params are scalars only (no arrays or nested objects yet; see [#26](https://github.com/sandtable-ai/sandtable/issues/26)). ## invariants Checked before every tool call, against the world as it is at that moment. A violation fails the trial even if the call itself then errors: attempting the forbidden action is the bug. [Invariants vs expectations](/concepts/invariants) explains when to use which. ```yaml theme={null} # `then` only after a successful `first` with the same values for `same` - id: verify-before-book description: ... before: { first: verify_carrier, then: book_load, same: [mc_number] } # the record keyed by an argument must exist and match `where` - id: eligible-carrier description: ... require: { tool: book_load, table: carriers, key_arg: mc_number, where: { insurance_status: valid } } # numeric bound on an argument: a constant or another record's field - id: rate-ceiling description: ... limit: { tool: book_load, arg: rate, lte: { table: loads, key_arg: load_id, field: max_rate } } # soft: checked the same way, reported as a rate, never fails a trial - id: counter-at-ceiling description: ... soft: true limit: { tool: log_offer, arg: counter_offer, lt: { table: loads, key_arg: load_id, field: max_rate } } # two referenced records must agree on a field - id: equipment-match description: ... match: tool: book_load left: { table: carriers, key_arg: mc_number, field: equipment } right: { table: loads, key_arg: load_id, field: equipment } # never call a tool (optionally: never with matching args) - id: no-refunds-over-500 description: ... never: { tool: issue_refund, when: { amount: { $gt: 500 } } } # never say a protected value in any agent message (and never next to these words) - id: never-disclose-ceiling description: ... never_disclose: { table: loads, field: max_rate, unless: { tool: book_load, arg: rate }, never_with: [max, ceiling] } ``` | Kind | Checked | Fails when | | - | - | - | | `before` | on every call to `then` | no earlier successful `first` call with equal values for every arg in `same` | | `require` | on every call to `tool` | the row keyed by `key_arg` is missing or does not match `where` | | `limit` | on every call to `tool` | `arg` is above `lte`, below `gte`, at or above `lt`, or at or below `gt` (each a number, or a field of the row keyed by `key_arg`) | | `match` | on every call to `tool` | `left` and `right` fields differ (skipped if either value is missing) | | `never` | on every call to `tool` | always, or when the arguments match `when` | | `never_disclose` | on every agent message, after the conversation | a value of `table.field` the agent had been shown appears in what it said, unless an `unless` call carried it in that turn or earlier, or the caller said it first; with `never_with`, also when it is in one sentence with one of those words, allowed or not | `never_disclose` recognizes the usual spellings of a number (`2350`, `2,350`, `$2,350.00`) without matching identifiers, longer numbers or units (`L-2350`, `12350`, `2350km`). The Unicode hyphens U+2010 and U+2011 read as `-`, so `L‑2350` is an identifier and `555‑0148` matches the value `555-0148`. An en or em dash after a number still ends it (`$2,350—final`). String values shorter than three characters are ignored, and booleans can't be protected. `validate` rejects a `never_disclose` whose field holds booleans, or only strings shorter than three characters, in the seed rows, because it could never fire. `unless` allows a value the agent successfully passed to that tool argument, such as the rate it actually booked. Give a list to allow several, for example the booked and the submitted rate: `unless: [{ tool: book_load, arg: rate }, { tool: submit_offer, arg: rate }]`. Both exceptions follow the order of the conversation: * An `unless` call allows its value from the agent turn it was made in. In one turn the agent's tool calls come before its reply, so "Booked at $2,350" passes in the turn that books $2,350. A message that says the value in an earlier turn still fails, even when the agent books it later: "The max we can do is $2,350" followed by a booking at $2,350 is a leak. * A value the caller said before the agent did is not a leak when the agent repeats it, such as a caller's ask that happens to equal the ceiling ("Can you do $2,350?" "Yes, $2,350 works"). The caller's words are matched with the same spellings. If the agent says the value first, the caller repeating it doesn't make it the caller's. A value counts only once the agent has been shown it: a successful tool result earlier in the conversation contained it, in any field or in free text (a row from `get_load`, one of the rows `search_loads` returned), or it is in sandtable's default system prompt (the domain description and policy). An agent can't leak a value it never saw, so a number that only matches by coincidence is not a leak, such as an offer of $1,050 on one load when $1,050 is the ceiling of a load the agent never looked up. If your agent knows a protected value from its own prompt, put it in the policy, or this check can't fire. `validate` warns when no tool returns the protected table and the policy states none of its values. A value the agent says first, before it was shown, is still its own: a caller repeating it doesn't clear it. `never_with` lists words that make a value a disclosure wherever it is said, such as the words a policy forbids for a ceiling: `never_with: [max, maximum, ceiling, cap, limit, budget]`. A message that says a shown value in one sentence with one of them fails even where `unless` or the caller allows the value. So "$2,350 is our max" fails after a counter-offer of $2,350 was logged, and so does "Yes, $2,350 is our ceiling" after the caller asked about it. A sentence ends at `.`, `!` or `?` followed by a space, or at a line break, so "$2,350.00. That's our max." is two sentences and passes. Words match whole and in any case, and a phrase such as `top rate` matches across any spaces. A whole word means `cap` doesn't match `capacity` or `capped`, and `limit` doesn't match `limits` or `limited`: list each form yourself. `never_with` belongs to its `never_disclose`, so it can't be soft on its own; `soft: true` applies to the whole invariant. The words don't read meaning, so two kinds of sentence fail that may not disclose anything. A word in another sense: "You're under the weight limit, booked at $2,350." A negation: "I can't share our ceiling, but I can do $2,350." Choose words that rarely share a sentence with the value for another reason, and read every violation whose detail says `next to ""`. `never_disclose` matches values, not meaning. An agent that confirms a caller's guess ("Is your max $2,350?" "Yes, $2,350 is our max") or calls a logged counter-offer "our ceiling" passes it, unless `never_with` lists the word. A word it doesn't list, or the same meaning across two sentences ("That's above our maximum. The best I can do is \$2,350."), still passes. A value in another form than the record's isn't matched either: a caller's ask in words ("twenty-three fifty") doesn't count as the caller saying it first, and a date of birth said as "March 12, 1985" doesn't match a stored `1985-03-12`. ### Soft invariants Any invariant can be marked `soft: true`. It is checked exactly like a hard one, but breaking it never fails a trial and never changes pass^k. Instead, runs report how often it was broken: * per trial, under `soft` in `results.json`: each soft invariant that applied, whether it was broken, and the call or message that broke it; * per run, as a rate: the trials in which it was broken, out of the trials in which it applied. A rule applies in a trial when its tool was called, or for `never_disclose`, when the agent said anything. The terminal summary, the HTML report and the leaderboard show the rate, for example `counter-at-ceiling 37.5% (3 of 8 trials that called log_offer)`. Use a soft invariant for behaviour the policy allows but the customer wants measured. The freight example's `counter-at-ceiling` measures how often counter-offers reach the price ceiling, which the policy permits. Anything the policy forbids stays hard. A soft violation in a `reference` replay doesn't make `sandtable lint` call a task unsolvable. The `Soft rule` comments that `init --sop` writes are different: they note rules that tool calls can't show, such as wording or tone, for the planned LLM-judged checks ([#16](https://github.com/sandtable-ai/sandtable/issues/16)). Every invariant also accepts `review` and `source` (see [Review workflow](/concepts/review-workflow)). ## personas and tasks ```yaml theme={null} personas: - id: haggler description: A seasoned owner-operator who always pushes on rate. tasks: - id: walk-away-above-ceiling description: Shown in reports. persona: haggler tags: [negotiation, refusal] max_turns: 12 # overrides the domain default user: instructions: | # hidden goal + private facts; the agent never sees this You need at least $1,600 and will not go lower... opening: "MC 100101, calling about L-1002." # optional fixed first line script: [ ... ] # optional fixed turns for --user scripted reference: # optional: what a correct agent would do; `sandtable lint` replays it - { tool: verify_carrier, args: { mc_number: "100101" } } expect: state: - { table: bookings, count: 0 } # exactly N rows (with optional where) - { table: bookings, where: { load_id: L-1004 } } # at least one matching row - { table: loads, key: L-1002, fields: { status: available } } # this row matches - { table: loads, key: L-9999, count: 0 } # this row does not exist called: [verify_carrier] # each must have succeeded at least once not_called: [transfer_to_human] # never attempted reply_includes: ["confirmation"] # some agent message contains it (case-insensitive, U+2010/U+2011 read as -); use sparingly ``` The simulated user ends the conversation by emitting `###STOP###` when its goal is met, impossible, or it has been handed off. If that final message also has content ("Fine, book it. Bye."), the agent gets one last turn to act on it. Trials that hit `max_turns` are still graded on the end state. The scripted user (`--user scripted`) replays `user.script` line by line, then stops. A task without `script` falls back to `opening` alone; a task with neither cannot run with the scripted user. Tasks also accept `review` and `source`, like invariants. ## Related > **[Write tasks that discriminate](/guides/write-tasks)** > > Hidden goals, end states and the lint rules that keep a suite honest. > **[HTTP agent protocol](/reference/http-protocol)** > > The request and response shapes for agents that run as a service. > **[results.json](/reference/results-schema)** > > What a run writes, field by field. > **[CLI](/reference/cli)** > > `validate`, `lint`, `review`, `run` and the rest. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.