> ## Documentation Index > Fetch the complete documentation index at: https://sandtable.dev/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > sandtable is an early-alpha, open-source CLI (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai `, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai --help`, trust the help output. > Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge. > To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional. > In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`. > Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs. # Write tasks that separate good agents from careless ones > Write tasks that a correct agent passes and a careless one fails: firm hidden goals, outcomes graded on state, balanced refusals and a reference solution. A task discriminates when a correct agent passes it and a careless or do-nothing agent fails it. Most weak suites fail that test in one of two ways: tasks that a do-nothing agent passes, and tasks that a correct agent fails because the task, not the agent, is wrong. These practices avoid both. ## Anatomy of a good task ```yaml theme={null} - id: negotiate-within-ceiling description: Carrier haggles; the agent closes between the caller's floor and the max rate. persona: haggler tags: [negotiation] user: instructions: | You are Riley, owner-operator, Blue Mesa Trucking, MC 100101, dry van. You want load L-1001 (Dallas to Atlanta). Open by asking for $2,600. You will accept any offer of $2,250 or more, but never less than $2,250. If the agent won't reach $2,250, say no thanks and end the call. script: - MC 100101, Blue Mesa. I'm looking at L-1001. - I need $2,600 for that one. - Okay, how about $2,300? - Deal, book it at 2300. - Thanks! reference: - { tool: verify_carrier, args: { mc_number: "100101" } } - { tool: get_load, args: { load_id: L-1001 } } - { tool: book_load, args: { load_id: L-1001, mc_number: "100101", rate: 2300 } } expect: state: - { table: bookings, where: { load_id: L-1001, mc_number: "100101", rate: { $gte: 2250, $lte: 2350 } }, count: 1 } ``` It has a firm hidden goal with numbers, a clear exit, a script for offline runs, a reference that proves it can be passed, and an expectation on state that accepts every correct outcome (any rate between the caller's floor and the ceiling) and nothing else. ## Practices **1. Grade the outcome on state** Expect the records the conversation should create or change: a booking with these fields, a row whose status changed, an escalation created. State is what the customer's systems will actually hold. Use `reply_includes` sparingly: wording varies between runs and models. **2. Accept every correct outcome** Use ranges and operators instead of one exact value when the policy allows several: `rate: { $gte: 2250, $lte: 2350 }`. Do not forbid acceptable alternatives. In the first live runs, a task that forbade any handoff failed agents that handed off correctly *after* a valid booking; it was relaxed to grade the booking itself. **3. Assert what must not change** For refusal tasks, check that nothing happened where it matters: `{ table: bookings, count: 0 }`, or `{ table: loads, key: L-1002, fields: { status: available } }`. **4. Make refusals cost something** A refusal graded only on "nothing changed" is passed by an agent that never does anything. Add what a correct refusal requires, such as `called: [verify_carrier]` (it looked the carrier up before declining) or an escalation row. Then the [noop baseline](/concepts/baselines) fails it. **5. Balance write and no-write tasks** Include tasks that must end with a write and tasks that must end without one. A suite of only refusals lets a do-nothing agent score well; a suite of only happy paths never tests restraint. Tag them (`happy-path`, `negotiation`, `refusal`, `handoff`) so per-tag scores show the balance. **6. Give the caller a firm, specific goal** Write `user.instructions` as private facts plus limits: who they are, what they want, the number they will not cross, and when to end the call ("Once you have a confirmation number, say thanks and end the call"). Vague goals make the simulated user improvise, which adds noise instead of testing the agent. Do not give the caller needs the agent cannot meet unless that is the test. **7. Follow the policy, not the transcript** When a task comes from a real call, the expected outcome is what the SOP requires, even if the human rep did something else. That is how `init` drafts tasks from transcripts, and how you should review them. **8. Prove it is solvable** Add a `reference`: the tool calls a correct agent would make. `sandtable lint` replays it against a fresh world and checks that it passes every expectation without breaking an invariant. A task no agent could pass is a broken task. **9. Make it run offline** Add a `user.script` so the task runs with `--user scripted`, for smoke tests in CI and for debugging graders without model calls. ## Make hard tasks hard for the right reasons When capable models pass every task, the suite stops telling them apart. Raise the difficulty the way real calls do, not with traps: * **More to get right in one call.** Two loads, a booking and a callback, or a rep-confirm load next to an instant one. Grade each outcome, such as each booking, the callback's load ID, and the round numbering on each load. * **Details that change mid-call.** A corrected MC number, a new pickup date after an offer, a corrected empty date. Grade the version the caller gave last. * **Callers who test the rules.** Another company's MC, a load another company will haul, a question about another load's ceiling. Write every rule such a task needs into the policy in plain words first, and ask whether two people who know the domain would give the same pass or fail. The freight example's [hard tier](https://github.com/sandtable-ai/sandtable/tree/main/examples/freight#the-hard-tier) shows each task next to the policy rule it tests. ## Use personas for behaviour, not goals A persona changes *how* the caller behaves: impatient, evasive, friendly, pushy on price. The task's instructions hold *what* they want. Reusing a few personas across many tasks shows whether an agent holds the policy under pressure, not just in polite conversations. ## Check your suite ```bash theme={null} npx sandtable-ai validate ./my-domain # schema and cross-references npx sandtable-ai lint ./my-domain # ready, solvable, non-vacuous, graded npx sandtable-ai run ./my-domain --agent noop --user scripted # the floor ``` Then run the real agent, open the report, and read the failing trajectories. For each failure, decide whether the agent, the task or the simulated caller was wrong, and fix the right one. A suite earns trust one corrected task at a time. ## Checklist * [ ] Every task has `state` or `called` expectations, and lint reports it ready. * [ ] Every task that expects a write has a `reference`, and lint reports it solvable. * [ ] Refusal tasks require an action, so the noop baseline fails them. * [ ] Expectations accept every outcome the policy allows. * [ ] Hidden goals state limits and an exit. * [ ] Every task has a `user.script`. * [ ] Names, numbers and records are fictional. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.