sandtable.yaml. Any top-level value that is a relative path to a .md, .yaml or .yml file is inlined, so you can keep policy: ./policy.md and tasks: ./tasks.yaml in separate files. A path to a missing file is a validation error, so a misspelled path can’t become the policy text.
Run npx sandtable-ai validate <dir> after every edit. It checks the schema and every cross-reference (tables, tools, params, personas).
sandtable.yaml
The schema is strict: an unknown key anywhere is a validation error, not a silent no-op. A typo such as
invariant: instead of invariants: fails sandtable validate.Templates
Inside toolops, checks, invariant where clauses and task assertions:
"$arg"is replaced by the tool call’s argument value, keeping its type."${arg}"interpolates inside a longer string:"Load ${load_id} is not available".- An argument’s name is a letter or
_, then letters, digits or_. Any other string is literal text, even when it starts with$:"$5","$$","$/hr","$USD/kg","$@type". A whole string such as"$USD"is a reference. - In a tool’s
opsandchecks, every reference must name one of the tool’s params, orsandtable validatefails withnever resolves: tool "get_load" has no param "loadId". It would be empty at run time. - A condition whose argument was not passed (an optional param) is dropped, so it filters nothing.
Where clauses
where, fields and when are maps of field to condition. A plain value means equality. An object of operators means all of them must hold:
world
tools
A tool is what the agent sees (name, description, JSON-schema params) plus what it does to the world.
Arguments are validated before anything runs. Unknown argument names, missing required ones and wrong types come back to the agent as tool errors. Representations are coerced leniently:
"2,100" becomes 2100 for a number param, and 100101 becomes "100101" for a string param.
There is no delete op, and params are scalars only (no arrays or nested objects yet; see #26).
invariants
Checked before every tool call, against the world as it is at that moment. A violation fails the trial even if the call itself then errors: attempting the forbidden action is the bug. Invariants vs expectations explains when to use which.never_disclose recognizes the usual spellings of a number (2350, 2,350, $2,350.00) without matching identifiers, longer numbers or units (L-2350, 12350, 2350km). The Unicode hyphens U+2010 and U+2011 read as -, so L‑2350 is an identifier and 555‑0148 matches the value 555-0148. An en or em dash after a number still ends it ($2,350—final). String values shorter than three characters are ignored, and booleans can’t be protected. validate rejects a never_disclose whose field holds booleans, or only strings shorter than three characters, in the seed rows, because it could never fire. unless allows a value the agent successfully passed to that tool argument, such as the rate it actually booked. Give a list to allow several, for example the booked and the submitted rate: unless: [{ tool: book_load, arg: rate }, { tool: submit_offer, arg: rate }].
Both exceptions follow the order of the conversation:
- An
unlesscall allows its value from the agent turn it was made in. In one turn the agent’s tool calls come before its reply, so “Booked at 2,350. A message that says the value in an earlier turn still fails, even when the agent books it later: “The max we can do is 2,350 is a leak. - A value the caller said before the agent did is not a leak when the agent repeats it, such as a caller’s ask that happens to equal the ceiling (“Can you do 2,350 works”). The caller’s words are matched with the same spellings. If the agent says the value first, the caller repeating it doesn’t make it the caller’s.
get_load, one of the rows search_loads returned), or it is in sandtable’s default system prompt (the domain description and policy). An agent can’t leak a value it never saw, so a number that only matches by coincidence is not a leak, such as an offer of 1,050 is the ceiling of a load the agent never looked up. If your agent knows a protected value from its own prompt, put it in the policy, or this check can’t fire. validate warns when no tool returns the protected table and the policy states none of its values. A value the agent says first, before it was shown, is still its own: a caller repeating it doesn’t clear it.
never_with lists words that make a value a disclosure wherever it is said, such as the words a policy forbids for a ceiling: never_with: [max, maximum, ceiling, cap, limit, budget]. A message that says a shown value in one sentence with one of them fails even where unless or the caller allows the value. So “2,350 was logged, and so does “Yes, 2,350.00. That’s our max.” is two sentences and passes. Words match whole and in any case, and a phrase such as top rate matches across any spaces. A whole word means cap doesn’t match capacity or capped, and limit doesn’t match limits or limited: list each form yourself. never_with belongs to its never_disclose, so it can’t be soft on its own; soft: true applies to the whole invariant.
The words don’t read meaning, so two kinds of sentence fail that may not disclose anything. A word in another sense: “You’re under the weight limit, booked at 2,350.” Choose words that rarely share a sentence with the value for another reason, and read every violation whose detail says next to "<word>".
never_disclose matches values, not meaning. An agent that confirms a caller’s guess (“Is your max 2,350 is our max”) or calls a logged counter-offer “our ceiling” passes it, unless never_with lists the word. A word it doesn’t list, or the same meaning across two sentences (“That’s above our maximum. The best I can do is $2,350.”), still passes. A value in another form than the record’s isn’t matched either: a caller’s ask in words (“twenty-three fifty”) doesn’t count as the caller saying it first, and a date of birth said as “March 12, 1985” doesn’t match a stored 1985-03-12.
Soft invariants
Any invariant can be markedsoft: true. It is checked exactly like a hard one, but breaking it never fails a trial and never changes pass^k. Instead, runs report how often it was broken:
- per trial, under
softinresults.json: each soft invariant that applied, whether it was broken, and the call or message that broke it; - per run, as a rate: the trials in which it was broken, out of the trials in which it applied. A rule applies in a trial when its tool was called, or for
never_disclose, when the agent said anything.
counter-at-ceiling 37.5% (3 of 8 trials that called log_offer). Use a soft invariant for behaviour the policy allows but the customer wants measured. The freight example’s counter-at-ceiling measures how often counter-offers reach the price ceiling, which the policy permits. Anything the policy forbids stays hard. A soft violation in a reference replay doesn’t make sandtable lint call a task unsolvable.
The Soft rule comments that init --sop writes are different: they note rules that tool calls can’t show, such as wording or tone, for the planned LLM-judged checks (#16).
Every invariant also accepts review and source (see Review workflow).
personas and tasks
###STOP### when its goal is met, impossible, or it has been handed off. If that final message also has content (“Fine, book it. Bye.”), the agent gets one last turn to act on it. Trials that hit max_turns are still graded on the end state.
The scripted user (--user scripted) replays user.script line by line, then stops. A task without script falls back to opening alone; a task with neither cannot run with the scripted user.
Tasks also accept review and source, like invariants.
Related
Write tasks that discriminate
Hidden goals, end states and the lint rules that keep a suite honest.
HTTP agent protocol
The request and response shapes for agents that run as a service.
results.json
What a run writes, field by field.
CLI
validate, lint, review, run and the rest.