results.json schema: scores, trials and failure modes

sandtable run writes two files to its output directory (--out, or .sandtable/runs/<timestamp>-<domain>/):

results.json: a SuiteResult, described below. It is the source of truth for a run.
report.html: a self-contained rendering of it, with no external assets, that you can attach to a pull request or send to a customer. npx sandtable-ai report results.json re-renders it.

The TypeScript types are exported from the package (SuiteResult, TrialResult, CheckResult and the rest). While sandtable is alpha, fields may still be added or change; version will change when the format breaks.

SuiteResult

Field	Type	Description
`version`	`1`	format version
`runId`	string	unique per run, e.g. `20261018T210512-k3f9`; prefixes the agent session ids
`domain`	string	the domain’s `name`
`agent`	string	the agent’s name, such as `claude:claude-opus-5-5`, `openai:gpt-4.1-mini`, `http://localhost:8787/step` (an HTTP agent is named by its URL)
`user`	string	the simulated user’s name, such as `claude:claude-haiku-5-5` or `scripted`
`k`	number	trials per task
`inputs`	string	a hash of what the agent and the simulated user were given in these tasks: the agent’s system prompt and tool schemas, and each task’s caller prompt, opening line and turn limit (and script, for the scripted user). `sandtable regrade` re-scores the run only while it is unchanged. Absent in results from before regrade
`startedAt`	string	ISO 8601 start time
`durationMs`	number	wall-clock duration
`tasks`	`TaskSummary[]`	one entry per task, with every trial
`passHat`	object	pass^j for every j from 1 to k, keyed `"1"`, `"2"`, …, as fractions from 0 to 1
`failureModes`	`FailureMode[]`	failed checks clustered across trials, most frequent first
`byTag`	`TagScore[]`	scores per task tag (`untagged` for tasks without tags), weakest first
`soft`	`SoftScore[]`	one entry per soft invariant in the domain, in spec order; empty when there are none
`usage`	`Usage`	tokens and estimated cost, summed over trials
`pendingReview`	`PendingItem[]`	drafted items not yet accepted; while non-empty, the run is provisional
`latency`	object	`agentTurnP50Ms` and `agentTurnP95Ms`: time the agent took per user turn, tool calls included (`null` with no turns)

TaskSummary

Field	Type	Description
`id`	string	task id
`description`	string	task description, if any
`tags`	string[]	task tags
`trials`	`TrialResult[]`	ordered by trial number
`passes`	number	trials that passed

FailureMode

Field	Type	Description
`id`	string	the check id, such as `verify-before-book` or `state:bookings#1`
`kind`	string	the check kind (see below)
`count`	number	failed trials containing this check
`tasks`	string[]	tasks where it failed
`example`	string	the detail from the first occurrence

TagScore

Field	Type	Description
`tag`	string	the tag
`tasks`	number	tasks with this tag
`passHat1`	number	pass^1 over those tasks
`passHatK`	number	pass^k over those tasks

SoftScore

How often a soft invariant (soft: true) was broken. Soft invariants never fail a trial or change pass^k.

Field	Type	Description
`id`	string	invariant id
`description`	string	invariant description
`tool`	string	the tool whose calls it checks; absent for `never_disclose`, which checks what the agent says
`applied`	number	trials in which it applied: its tool was called, or for `never_disclose`, the agent spoke
`broken`	number	trials in which it was broken at least once
`rate`	number or `null`	`broken / applied`, from 0 to 1; `null` when it never applied
`tasks`	string[]	tasks with at least one trial that broke it
`example`	string	the detail of the first violation, when there is one

PendingItem

Field	Type	Description
`kind`	`"tool"`, `"invariant"`, `"task"`	what is pending
`id`	string	its name or id
`source`	object	`file` and optional `quote`, for invariants and tasks drafted from a source

TrialResult

Field	Type	Description
`taskId`	string	task id
`trial`	number	trial number, from 0
`passed`	boolean	true only if every check passed
`checks`	`CheckResult[]`	every check, passed or failed
`violations`	`Violation[]`	hard invariant violations; any one fails the trial
`soft`	`SoftCheck[]`	each soft invariant that applied in this trial, and whether it was broken
`events`	`TrajectoryEvent[]`	the conversation and tool calls, in order
`changes`	`Change[]`	the difference between the world’s seed and its final state
`endReason`	string	`user_stop`, `max_turns` or `error`
`turns`	number	agent replies
`durationMs`	number	trial duration
`usage`	`Usage`	tokens and estimated cost for this trial
`error`	string	present when the trial ended with an error

CheckResult

Field	Type	Description
`id`	string	invariant id, `state:<table>[<key>]#<n>`, `called:<tool>`, `not_called:<tool>`, `reply:<phrase>` or `runtime`
`kind`	string	`invariant`, `state`, `called`, `not_called`, `reply` or `runtime`
`passed`	boolean	whether it passed
`detail`	string	a readable explanation, such as `book_load rate=2600 exceeds the limit 2350`

Violation

Field	Type	Description
`invariant`	string	invariant id
`description`	string	invariant description
`detail`	string	what happened
`callIndex`	number	for tool-call violations: the index of the call
`messageIndex`	number	for `never_disclose`: the index of the agent message
`soft`	`true`	present on a soft invariant’s violation

SoftCheck

Field	Type	Description
`id`	string	the soft invariant’s id
`broken`	boolean	broken at least once in this trial
`violations`	`Violation[]`	each time it was broken, with the call or message

TrajectoryEvent

One of:

{ "type": "user", "text": "MC 100101, Blue Mesa. I'm looking at L-1001." }
{ "type": "agent", "text": "L-1001 pays $2100. Want me to book it?", "latencyMs": 812 }
{ "type": "tool", "index": 0, "name": "get_load", "args": { "load_id": "L-1001" },
  "ok": true, "result": { }, "violations": [] }   // or "ok": false with "error"

A tool event’s violations lists the ids of hard invariants that call violated. When it also broke soft invariants, soft lists their ids.

Change

Field	Type	Description
`table`	string	table name
`key`	string	row key
`kind`	string	`insert`, `update` or `delete`
`before`	object	the row before (updates and deletes)
`after`	object	the row after (inserts and updates)

Usage

// illustrative numbers
{
  "agent": { "claude-opus-5-5": { "calls": 12, "input": 1200, "output": 3000, "cacheRead": 40000, "cacheWrite": 8000 } },
  "user":  { "claude-haiku-5-5": { "calls": 8, "input": 6000, "output": 900, "cacheRead": 0, "cacheWrite": 0 } },
  "costUsd": 0.11385
}

Tokens are grouped by role and model. input excludes cache reads and writes, following the Anthropic convention. The Claude, Bedrock Converse and OpenAI-compatible adapters record usage automatically; custom modules can call recordUsage; the HTTP adapter cannot see your agent’s model calls, so it records none. maxTokensStops counts the calls whose reply stopped at the model’s output-token limit: stop reason max_tokens on Claude and Converse, finish_reason: "length" on an OpenAI-compatible server. What the agent or caller said after such a call may be cut off or empty, and the grader scores it as said. The count also includes replies sandtable discarded and asked for again, such as a Converse reply whose <reasoning> never closed, since every call is recorded. The field appears only once a model has such a call, so a missing field means none did, or that the run is older than the field. The terminal summary and the HTML report show the count when it isn’t zero. costUsd is an estimate at list prices from PRICES in src/run/usage.ts, which holds verified prices for Claude models, plus any prices passed with --prices. If any model in the run has no known price, costUsd is null and reports show n/a. Claude’s usage is keyed by the model that served it. On the Anthropic API, that is the model Claude reports, which a refusal fallback can change. On Bedrock, it is the requested ID, such as global.anthropic.claude-opus-5-5 or eu.anthropic.claude-opus-5-5. A geo cross-region profile (eu., us., apac., us-gov. …) is priced at 1.1× the built-in price, since “Regional and multi-region endpoints include a 10% premium over global endpoints” for Claude 4.5 models and later (Anthropic pricing). A global. profile and the Anthropic API pay the built-in price. A price passed with --prices for the exact ID is used as is. An inference profile ARN names no model, so its usage is recorded under the bare name Claude reports and always priced at the built-in (global) price; a --prices entry for the ARN is never read. A bare anthropic. ID, such as an in-region call, also gets the built-in (global) price, which is unverified for in-region calls. Runs from before keyed Bedrock Claude usage by the bare model name, so they’re priced as global.

Recipes

# pass^k
jq '.passHat' results.json

# every failed trial, with its failing checks
jq -r '.tasks[] | .id as $t | .trials[] | select(.passed | not)
  | "\($t)#\(.trial): \([.checks[] | select(.passed | not) | .id] | join(", "))"' results.json

# failure modes, most frequent first
jq -r '.failureModes[] | "\(.count)x \(.kind) \(.id)"' results.json

# soft invariants: how often each was broken
jq -r '.soft[] | "\(.id): \(.broken) of \(.applied) trials"' results.json

Get started

Concepts

Guides

Reference

Resources

results.json schema: scores, trials and failure modes

SuiteResult

TaskSummary

FailureMode

TagScore

SoftScore

PendingItem

TrialResult

CheckResult

Violation

SoftCheck

TrajectoryEvent

Change

Usage

Recipes

​SuiteResult

​TaskSummary

​FailureMode

​TagScore

​SoftScore

​PendingItem

​TrialResult

​CheckResult

​Violation

​SoftCheck

​TrajectoryEvent

​Change

​Usage

​Recipes

SuiteResult

TaskSummary

FailureMode

TagScore

SoftScore

PendingItem

TrialResult

CheckResult

Violation

SoftCheck

TrajectoryEvent

Change

Usage

Recipes