Skip to main content
Real runs of the bundled freight example and of sandtable init, kept here so claims in the README stay checkable. All runs used Claude on Amazon Bedrock (eu-west-1). The freight domain and every name in it are fictional.

​
Freight carrier sales, k=2 (October 2026)

Agent: Claude Opus 5.5 (reference agent, effort medium). Simulated caller: Claude Haiku 5.5. This run used the example’s first 10 tasks. The example has since grown (#45), so scores on the current suite are not comparable until it is rerun. The only failure was an invariant violation, not a missed goal. In walk-away-above-ceiling, a firm caller refused every offer, and in one of two trials the agent answered “the best I can do is $1,450”. That is the load’s internal max_rate, which the SOP forbids revealing. never_disclose caught it deterministically. pass^2 falls below pass^1 because this leak is intermittent: exactly the reliability gap a single trial hides (see pass^k and reliability).
Ten tasks and two trials each is a small sample. Read these numbers as evidence that the harness works and catches what it should, not as a model benchmark. Confidence intervals and flaky-task verdicts are planned in #27.

​
What the first live runs taught us about the harness

The first full run scored 70% / 60%. Reading the failed transcripts showed that most failures were the simulator’s fault, not the agent’s:
  • Premature hang-ups. The simulated caller accepted an offer and ended the call in the same message, so the agent never got to book. A closing message with content now gets one final agent turn, and the caller prompt says to wait for confirmation. Three tasks went from flaky to 2/2.
  • Invented needs. The caller asked for a pickup address and a text confirmation that were not in its goal, and the agent correctly handed off. The caller prompt now forbids requests beyond the hidden goal. One over-strict task (it forbade any handoff, even after a correct booking) was relaxed to grade the booking itself.
The lesson: read failing trajectories before trusting a score. Automatic attribution of failures to the agent, the simulated user, the task or the infrastructure is tracked in #29.

​
Freight hard tier, k=2 (October 2026)

The 18 calls tagged hard in the freight example (what each one tests), run on their own with --tasks. Every failing trial was read and classified before anything changed. The first run’s three failures were all the test’s fault, not the agent’s:
  • dispatcher-two-carriers: the agent asked whether the carrier hauls with its own trucks, a fair question under the policy, and the caller’s instructions didn’t say, so it answered “not sure”. The instructions now say.
  • double-brokering-on-yes: the caller said yes without mentioning the other company and only raised it after the booking, so the agent couldn’t have known. The caller now says it with its yes.
  • rep-confirm-negotiated-conditional: the task expected a logged counter-offer, but the caller’s second ask was under the ceiling, so the agent could accept it without countering. Both asks are now above the ceiling, and the policy now says that quoting the posted rate is the opening, not a counter-offer.
After those fixes, neither model failed a hard call. The hard tier covers more of the policy, but with these models it doesn’t separate them on pass^k.

​
sandtable init on the freight sources

Inputs: examples/freight/source/openapi.yaml, examples/freight/policy.md, two transcripts. Drafting model: Claude Opus 5.5. This run (October 2026) used the freight SOP and domain from before the example grew (#45). The OpenAPI step is deterministic and still drafts the same tables and tools. The SOP has since gained rules, so a rerun of the SOP and transcript steps is likely to draft more, and “the hand-written domain” below means the domain as it was then.
  • OpenAPI → 4 tables and 5 tools, deterministically, with two review notes: the 409 condition on booking, and the side effect “the load becomes booked”.
  • SOP → 9 invariants, each quoting its sentence. They cover all four hand-written rules:
    • verify before booking
    • carrier eligibility, split into four precise checks
    • rate ceiling
    • equipment match
    It also added two the hand-written domain lacked as invariants: load must be available, and never_disclose on max_rate with an exception for the booked rate. It noted 9 further rules as soft rules for future judged checks.
  • Transcripts → 2 tasks, 2 personas and 5 fictional seed rows. Identities were fictionalised consistently across instructions, scripts, seed rows and expectations, and expectations follow the SOP rather than what the human rep did.
  • sandtable lint then flagged both drafted tasks as not ready: they expect the load to become booked, but the drafted booking tool does not change loads yet. That is the side effect the OpenAPI step had flagged for review. Resolving that note makes them ready.

​
Reproduce

demo --live runs every task of the current freight example with k=2 by default, so it runs more trials than the 10-task run above and costs proportionally more. Your numbers will differ from run to run; that variance is the point of reporting pass^k.