Test AI agents in CI: gate pull requests on pass^k

The goal is simple: a prompt tweak, model swap or tool change that breaks a customer rule should fail the build, the same way a broken unit test does. sandtable gives you two tiers for that.

Tier	Commands	Cost	Run it
Checks	`validate --strict`, `lint`	free, no model calls, no secrets	on every pull request
Rehearsal	`run -k <n> --fail-under <pct>`	model calls for the agent and the simulated user	on pull requests that touch the agent, or nightly

Exit codes

Every command exits 1 when it should block a merge, so they work as plain CI steps:

Command	Exits 1 when
`validate --strict`	the domain is invalid, or a drafted item is still `review: pending`
`lint`	any task is not graded, not ready, vacuous or not solvable
`run --fail-under <pct>`	pass^k, with k = `--trials`, is below `pct`
`run`, `demo --live`	every trial ended in an error, so nothing was graded: missing credentials or package, or an agent that isn’t running. This holds without `--fail-under` too
`leaderboard`	every model is an error row, so the board ranks none of them: its agent couldn’t be built, or any of its trials crashed (missing credentials or package, the wrong provider or region, throttling, an agent that isn’t running). The do-nothing baseline doesn’t count
any command	the domain cannot be loaded, an adapter cannot be built, or another error stops it

In run, when only some trials end in an error, each counts as a failed trial, not a skipped one. In leaderboard, any crashed trial turns that model into an error row instead of a score, and the board exits 1 only when every model is one. Reporting infrastructure errors separately is planned (#27).

A GitHub Actions workflow

This assumes the domain lives in ./sandtable, sandtable-ai is pinned in your devDependencies, and your agent can be started as a local service on port 8787.

.github/workflows/sandtable.yml

name: sandtable
on:
  pull_request:

jobs:
  checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx sandtable-ai validate ./sandtable --strict
      - run: npx sandtable-ai lint ./sandtable

  rehearse:
    needs: checks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - name: Start the agent under test
        run: |
          node agent-server.mjs &   # your agent, listening on :8787
          sleep 3                   # or poll your agent's health endpoint
      - name: Rehearse
        run: >
          npx sandtable-ai run ./sandtable
          --agent http://localhost:8787/step
          --user claude:claude-haiku-5-5
          -k 3 --fail-under 85
          --out .sandtable/ci
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: sandtable-report
          path: .sandtable/ci

The uploaded artifact holds report.html and results.json, so a reviewer can open the failing transcripts from the pull request.

While sandtable is alpha, pin an exact version of sandtable-ai in package.json (for example npm install --save-dev --save-exact sandtable-ai): the spec and outputs may change between releases.

Other agent setups

JS/TS module
Claude on Bedrock
OpenAI-compatible

- run: npx sandtable-ai run ./sandtable --agent ./agent.ts -k 3 --fail-under 85
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

A .ts agent module needs Node 22.18 or later on the runner.

- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
    aws-region: eu-west-1
- run: npm install --no-save @anthropic-ai/bedrock-sdk
- run: npx sandtable-ai run ./sandtable --provider bedrock --agent claude -k 3 --fail-under 85

--provider bedrock applies to the simulated user too, so no Anthropic API key is needed.

- run: >
    npx sandtable-ai run ./sandtable --agent openai:gpt-4.1-mini
    --user claude:claude-haiku-5-5 -k 3 --fail-under 85
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Choosing the threshold and k

Measure first. Run the suite a few times on main and set --fail-under a little below what the current agent reliably scores. Raise it as the agent improves.
Know the floor. Run the noop baseline once. A threshold close to the noop score gates nothing.
Use k of 3 or 4 on pull requests. It catches tasks that fail one time in three. Raising k with the same threshold makes the gate stricter. See choosing k.
Keep the simulated user fixed and cheap. --user claude:claude-haiku-5-5 keeps cost down, and a fixed caller means a score change comes from the agent.
Expect noise on small suites. With 10 tasks, one flaky task moves the score by up to 10 points.

Cost

Each run costs tasks × k conversations. For scale: the first 10 tasks of the bundled freight example, 2 trials each, with Claude Opus 5.5 as the agent and Claude Haiku 5.5 as the caller, cost about $0.58 at list prices (results). The terminal summary and report show an estimated cost for Claude models, and for any other model whose prices you pass with --prices; without a price, cost shows n/a.

Coming next

Comparing a run against a stored baseline and posting pass^k deltas as a pull request comment are planned (#12, #13), as are critical tasks that fail the build on their own (#28).

Get started

Concepts

Guides

Reference

Resources

Test AI agents in CI: gate pull requests on pass^k

Exit codes

A GitHub Actions workflow

Other agent setups

Choosing the threshold and k

Cost

Coming next

​Exit codes

​A GitHub Actions workflow

​Other agent setups

​Choosing the threshold and k

​Cost

​Coming next

Exit codes

A GitHub Actions workflow

Other agent setups

Choosing the threshold and k

Cost

Coming next