> ## Documentation Index
> Fetch the complete documentation index at: https://sandtable.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> sandtable is an early-alpha, open-source CLI (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai <command>`, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai <command> --help`, trust the help output.
> Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge.
> To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional.
> In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`.
> Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs.

# results.json schema: scores, trials and failure modes

> The file every run writes: suite scores, failure modes, per-tag scores, usage, and every trial's transcript, tool calls, checks and world diff.

`sandtable run` writes two files to its output directory (`--out`, or `.sandtable/runs/<timestamp>-<domain>/`):

* `results.json`: a `SuiteResult`, described below. It is the source of truth for a run.
* `report.html`: a self-contained rendering of it, with no external assets, that you can attach to a pull request or send to a customer. `npx sandtable-ai report results.json` re-renders it.

The TypeScript types are exported from the package (`SuiteResult`, `TrialResult`, `CheckResult` and the rest). While sandtable is alpha, fields may still be added or change; `version` will change when the format breaks.

## SuiteResult

| Field | Type | Description |
| - | - | - |
| `version` | `1` | format version |
| `runId` | string | unique per run, e.g. `20261018T210512-k3f9`; prefixes the agent session ids |
| `domain` | string | the domain's `name` |
| `agent` | string | the agent's name, such as `claude:claude-opus-5-5`, `openai:gpt-4.1-mini`, `http://localhost:8787/step` (an HTTP agent is named by its URL) |
| `user` | string | the simulated user's name, such as `claude:claude-haiku-5-5` or `scripted` |
| `k` | number | trials per task |
| `inputs` | string | a hash of what the agent and the simulated user were given in these tasks: the agent's system prompt and tool schemas, and each task's caller prompt, opening line and turn limit (and script, for the scripted user). [`sandtable regrade`](/reference/cli#regrade) re-scores the run only while it is unchanged. Absent in results from before regrade |
| `startedAt` | string | ISO 8601 start time |
| `durationMs` | number | wall-clock duration |
| `tasks` | `TaskSummary[]` | one entry per task, with every trial |
| `passHat` | object | pass^j for every j from 1 to k, keyed `"1"`, `"2"`, …, as fractions from 0 to 1 |
| `failureModes` | `FailureMode[]` | failed checks clustered across trials, most frequent first |
| `byTag` | `TagScore[]` | scores per task tag (`untagged` for tasks without tags), weakest first |
| `soft` | `SoftScore[]` | one entry per soft invariant in the domain, in spec order; empty when there are none |
| `usage` | `Usage` | tokens and estimated cost, summed over trials |
| `pendingReview` | `PendingItem[]` | drafted items not yet accepted; while non-empty, the run is provisional |
| `latency` | object | `agentTurnP50Ms` and `agentTurnP95Ms`: time the agent took per user turn, tool calls included (`null` with no turns) |

### TaskSummary

| Field | Type | Description |
| - | - | - |
| `id` | string | task id |
| `description` | string | task description, if any |
| `tags` | string\[] | task tags |
| `trials` | `TrialResult[]` | ordered by trial number |
| `passes` | number | trials that passed |

### FailureMode

| Field | Type | Description |
| - | - | - |
| `id` | string | the check id, such as `verify-before-book` or `state:bookings#1` |
| `kind` | string | the check kind (see below) |
| `count` | number | failed trials containing this check |
| `tasks` | string\[] | tasks where it failed |
| `example` | string | the detail from the first occurrence |

### TagScore

| Field | Type | Description |
| - | - | - |
| `tag` | string | the tag |
| `tasks` | number | tasks with this tag |
| `passHat1` | number | pass^1 over those tasks |
| `passHatK` | number | pass^k over those tasks |

### SoftScore

How often a soft invariant (`soft: true`) was broken. Soft invariants never fail a trial or change pass^k.

| Field | Type | Description |
| - | - | - |
| `id` | string | invariant id |
| `description` | string | invariant description |
| `tool` | string | the tool whose calls it checks; absent for `never_disclose`, which checks what the agent says |
| `applied` | number | trials in which it applied: its tool was called, or for `never_disclose`, the agent spoke |
| `broken` | number | trials in which it was broken at least once |
| `rate` | number or `null` | `broken / applied`, from 0 to 1; `null` when it never applied |
| `tasks` | string\[] | tasks with at least one trial that broke it |
| `example` | string | the detail of the first violation, when there is one |

### PendingItem

| Field | Type | Description |
| - | - | - |
| `kind` | `"tool"`, `"invariant"`, `"task"` | what is pending |
| `id` | string | its name or id |
| `source` | object | `file` and optional `quote`, for invariants and tasks drafted from a source |

## TrialResult

| Field | Type | Description |
| - | - | - |
| `taskId` | string | task id |
| `trial` | number | trial number, from 0 |
| `passed` | boolean | true only if every check passed |
| `checks` | `CheckResult[]` | every check, passed or failed |
| `violations` | `Violation[]` | hard invariant violations; any one fails the trial |
| `soft` | `SoftCheck[]` | each soft invariant that applied in this trial, and whether it was broken |
| `events` | `TrajectoryEvent[]` | the conversation and tool calls, in order |
| `changes` | `Change[]` | the difference between the world's seed and its final state |
| `endReason` | string | `user_stop`, `max_turns` or `error` |
| `turns` | number | agent replies |
| `durationMs` | number | trial duration |
| `usage` | `Usage` | tokens and estimated cost for this trial |
| `error` | string | present when the trial ended with an error |

### CheckResult

| Field | Type | Description |
| - | - | - |
| `id` | string | invariant id, `state:<table>[<key>]#<n>`, `called:<tool>`, `not_called:<tool>`, `reply:<phrase>` or `runtime` |
| `kind` | string | `invariant`, `state`, `called`, `not_called`, `reply` or `runtime` |
| `passed` | boolean | whether it passed |
| `detail` | string | a readable explanation, such as `book_load rate=2600 exceeds the limit 2350` |

### Violation

| Field | Type | Description |
| - | - | - |
| `invariant` | string | invariant id |
| `description` | string | invariant description |
| `detail` | string | what happened |
| `callIndex` | number | for tool-call violations: the index of the call |
| `messageIndex` | number | for `never_disclose`: the index of the agent message |
| `soft` | `true` | present on a soft invariant's violation |

### SoftCheck

| Field | Type | Description |
| - | - | - |
| `id` | string | the soft invariant's id |
| `broken` | boolean | broken at least once in this trial |
| `violations` | `Violation[]` | each time it was broken, with the call or message |

### TrajectoryEvent

One of:

```jsonc theme={null}
{ "type": "user", "text": "MC 100101, Blue Mesa. I'm looking at L-1001." }
{ "type": "agent", "text": "L-1001 pays $2100. Want me to book it?", "latencyMs": 812 }
{ "type": "tool", "index": 0, "name": "get_load", "args": { "load_id": "L-1001" },
  "ok": true, "result": { }, "violations": [] }   // or "ok": false with "error"
```

A tool event's `violations` lists the ids of hard invariants that call violated. When it also broke soft invariants, `soft` lists their ids.

### Change

| Field | Type | Description |
| - | - | - |
| `table` | string | table name |
| `key` | string | row key |
| `kind` | string | `insert`, `update` or `delete` |
| `before` | object | the row before (updates and deletes) |
| `after` | object | the row after (inserts and updates) |

## Usage

```jsonc theme={null}
// illustrative numbers
{
  "agent": { "claude-opus-5-5": { "calls": 12, "input": 1200, "output": 3000, "cacheRead": 40000, "cacheWrite": 8000 } },
  "user":  { "claude-haiku-5-5": { "calls": 8, "input": 6000, "output": 900, "cacheRead": 0, "cacheWrite": 0 } },
  "costUsd": 0.11385
}
```

Tokens are grouped by role and model. `input` excludes cache reads and writes, following the Anthropic convention. The Claude, Bedrock Converse and OpenAI-compatible adapters record usage automatically; custom modules can call `recordUsage`; the HTTP adapter cannot see your agent's model calls, so it records none.

`maxTokensStops` counts the calls whose reply stopped at the model's output-token limit: stop reason `max_tokens` on Claude and Converse, `finish_reason: "length"` on an OpenAI-compatible server. What the agent or caller said after such a call may be cut off or empty, and the grader scores it as said. The count also includes replies sandtable discarded and asked for again, such as a Converse reply whose `<reasoning>` never closed, since every call is recorded. The field appears only once a model has such a call, so a missing field means none did, or that the run is older than the field. The terminal summary and the HTML report show the count when it isn't zero.

`costUsd` is an estimate at list prices from `PRICES` in `src/run/usage.ts`, which holds verified prices for Claude models, plus any prices passed with `--prices`. If any model in the run has no known price, `costUsd` is `null` and reports show `n/a`.

Claude's usage is keyed by the model that served it. On the Anthropic API, that is the model Claude reports, which a refusal fallback can change. On Bedrock, it is the requested ID, such as `global.anthropic.claude-opus-5-5` or `eu.anthropic.claude-opus-5-5`. A geo cross-region profile (`eu.`, `us.`, `apac.`, `us-gov.` …) is priced at 1.1× the built-in price, since "Regional and multi-region endpoints include a 10% premium over global endpoints" for Claude 4.5 models and later ([Anthropic pricing](https://platform.claude.com/docs/en/about-claude/pricing)). A `global.` profile and the Anthropic API pay the built-in price. A price passed with `--prices` for the exact ID is used as is. An inference profile ARN names no model, so its usage is recorded under the bare name Claude reports and always priced at the built-in (global) price; a `--prices` entry for the ARN is never read. A bare `anthropic.` ID, such as an in-region call, also gets the built-in (global) price, which is unverified for in-region calls. Runs from before keyed Bedrock Claude usage by the bare model name, so they're priced as global.

## Recipes

```bash theme={null}
# pass^k
jq '.passHat' results.json

# every failed trial, with its failing checks
jq -r '.tasks[] | .id as $t | .trials[] | select(.passed | not)
  | "\($t)#\(.trial): \([.checks[] | select(.passed | not) | .id] | join(", "))"' results.json

# failure modes, most frequent first
jq -r '.failureModes[] | "\(.count)x \(.kind) \(.id)"' results.json

# soft invariants: how often each was broken
jq -r '.soft[] | "\(.id): \(.broken) of \(.applied) trials"' results.json
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.