How it drops
If an agent succeeds on every task with the same probability p, and trials are independent, pass^k is simply pk:
Real agents are not that uniform: most tasks pass every time and a few are flaky. pass^k finds those few. In a live run on the first 10 tasks of the freight example, Claude Opus 5.5 scored pass^1 95% and pass^2 90%: one task leaked an internal price in one of its two trials, a failure a single trial would have missed half the time.
How sandtable computes it
For each task with n trials and c successes, sandtable uses the unbiased estimator from τ-bench:C is the binomial coefficient. pass^1 is the plain success rate. A run with -k 4 reports pass^1 through pass^4, overall and per tag.
Worked example, one task, 4 trials, 3 passed:
pass^k, not pass@k
pass@k is the probability that at least one of k trials succeeds. It suits settings where you can pick the best attempt, such as generating code and running tests. A customer-facing agent gets one attempt per conversation, so sandtable reports pass^k. A pass@k column, flaky-task verdicts and confidence intervals are planned (#27).Choosing k
Cost and time grow linearly with tasks × k, so pick k by purpose:
Two things limit what a score can tell you:
- Few tasks means coarse scores. With 10 tasks, one flaky task moves pass^k by up to 10 points. Treat small differences between runs as noise until intervals land.
- Variation needs a source. Trials differ because the model samples and the simulated user improvises. A deterministic agent replaying a scripted user gives identical trials, so pass^k equals pass^1.
Gating on it
--fail-under compares the highest pass^k a run computes (k = --trials) against a percentage, and exits 1 when it is below: