> ## Documentation Index
> Fetch the complete documentation index at: https://sandtable.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> sandtable is an early-alpha, open-source CLI (TypeScript, MIT): https://github.com/sandtable-ai/sandtable. Run it as `npx sandtable-ai <command>`, never with the bare name: npm's `sandtable` is an unrelated package. If a flag here disagrees with `npx sandtable-ai <command> --help`, trust the help output.
> Grading is deterministic: pass/fail comes from the world's end state, invariants and the tool-call log, never from an LLM judge.
> To draft a domain for a user, read /guides/init and /reference/spec, then run `npx sandtable-ai validate`, `lint` and `review` before any model-backed run. Keep all example data fictional.
> In Claude Code, the sandtable plugin teaches these steps as skills: `/plugin marketplace add sandtable-ai/sandtable`, then `/plugin install sandtable@sandtable`.
> Every page is available as Markdown by appending .md to its URL. /llms.txt lists every page, /llms-full.txt contains the whole site, and https://sandtable.dev/mcp is a search MCP server for these docs.

# Test AI agents in CI: gate pull requests on pass^k

> Gate pull requests on agent reliability: free offline checks on every change, and a model-backed rehearsal that fails the build below a pass^k threshold.

The goal is simple: a prompt tweak, model swap or tool change that breaks a customer rule should fail the build, the same way a broken unit test does. `sandtable` gives you two tiers for that.

| Tier | Commands | Cost | Run it |
| - | - | - | - |
| Checks | `validate --strict`, `lint` | free, no model calls, no secrets | on every pull request |
| Rehearsal | `run -k <n> --fail-under <pct>` | model calls for the agent and the simulated user | on pull requests that touch the agent, or nightly |

## Exit codes

Every command exits 1 when it should block a merge, so they work as plain CI steps:

| Command | Exits 1 when |
| - | - |
| `validate --strict` | the domain is invalid, or a drafted item is still `review: pending` |
| `lint` | any task is not graded, not ready, vacuous or not solvable |
| `run --fail-under <pct>` | pass^k, with k = `--trials`, is below `pct` |
| `run`, `demo --live` | every trial ended in an error, so nothing was graded: missing credentials or package, or an agent that isn't running. This holds without `--fail-under` too |
| `leaderboard` | every model is an error row, so the board ranks none of them: its agent couldn't be built, or any of its trials crashed (missing credentials or package, the wrong provider or region, throttling, an agent that isn't running). The do-nothing baseline doesn't count |
| any command | the domain cannot be loaded, an adapter cannot be built, or another error stops it |

In `run`, when only some trials end in an error, each counts as a failed trial, not a skipped one. In `leaderboard`, any crashed trial turns that model into an error row instead of a score, and the board exits 1 only when every model is one. Reporting infrastructure errors separately is planned ([#27](https://github.com/sandtable-ai/sandtable/issues/27)).

## A GitHub Actions workflow

This assumes the domain lives in `./sandtable`, `sandtable-ai` is pinned in your `devDependencies`, and your agent can be started as a local service on port 8787.

```yaml .github/workflows/sandtable.yml theme={null}
name: sandtable
on:
  pull_request:

jobs:
  checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx sandtable-ai validate ./sandtable --strict
      - run: npx sandtable-ai lint ./sandtable

  rehearse:
    needs: checks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Start the agent under test
        run: |
          node agent-server.mjs &   # your agent, listening on :8787
          sleep 3                   # or poll your agent's health endpoint
      - name: Rehearse
        run: >
          npx sandtable-ai run ./sandtable
          --agent http://localhost:8787/step
          --user claude:claude-haiku-5-5
          -k 3 --fail-under 85
          --out .sandtable/ci
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: sandtable-report
          path: .sandtable/ci
```

The uploaded artifact holds `report.html` and `results.json`, so a reviewer can open the failing transcripts from the pull request.

> [!NOTE]
>
> While sandtable is alpha, pin an exact version of `sandtable-ai` in `package.json` (for example `npm install --save-dev --save-exact sandtable-ai`): the spec and outputs may change between releases.

### Other agent setups

**JS/TS module**

```yaml theme={null}
- run: npx sandtable-ai run ./sandtable --agent ./agent.ts -k 3 --fail-under 85
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

A `.ts` agent module needs Node 22.18 or later on the runner.

**Claude on Bedrock**

```yaml theme={null}
- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
    aws-region: eu-west-1
- run: npm install --no-save @anthropic-ai/bedrock-sdk
- run: npx sandtable-ai run ./sandtable --provider bedrock --agent claude -k 3 --fail-under 85
```

`--provider bedrock` applies to the simulated user too, so no Anthropic API key is needed.

**OpenAI-compatible**

```yaml theme={null}
- run: >
    npx sandtable-ai run ./sandtable --agent openai:gpt-4.1-mini
    --user claude:claude-haiku-5-5 -k 3 --fail-under 85
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

## Choosing the threshold and k

* **Measure first.** Run the suite a few times on `main` and set `--fail-under` a little below what the current agent reliably scores. Raise it as the agent improves.
* **Know the floor.** Run the [noop baseline](/concepts/baselines) once. A threshold close to the noop score gates nothing.
* **Use k of 3 or 4 on pull requests.** It catches tasks that fail one time in three. Raising k with the same threshold makes the gate stricter. See [choosing k](/concepts/pass-k#choosing-k).
* **Keep the simulated user fixed and cheap.** `--user claude:claude-haiku-5-5` keeps cost down, and a fixed caller means a score change comes from the agent.
* **Expect noise on small suites.** With 10 tasks, one flaky task moves the score by up to 10 points.

## Cost

Each run costs tasks × k conversations. For scale: the first 10 tasks of the bundled freight example, 2 trials each, with Claude Opus 5.5 as the agent and Claude Haiku 5.5 as the caller, cost about \$0.58 at list prices ([results](/results)). The terminal summary and report show an estimated cost for Claude models, and for any other model whose prices you pass with `--prices`; without a price, cost shows `n/a`.

## Coming next

Comparing a run against a stored baseline and posting pass^k deltas as a pull request comment are planned ([#12](https://github.com/sandtable-ai/sandtable/issues/12), [#13](https://github.com/sandtable-ai/sandtable/issues/13)), as are critical tasks that fail the build on their own ([#28](https://github.com/sandtable-ai/sandtable/issues/28)).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.