report.html answers “what failed and why”. Before go-live, the customer asks something else: does the agent follow our procedure, how do you know, and what haven’t you checked? report --audience customer answers that on one page, in plain words, from a run you already have.
1. Rehearse your agent
Run the suite with several conversations per scenario, so a mistake that happens one time in four has a chance to show up:2. Rehearse a careless agent
A rule is checked in a conversation when it applied to what the agent did: the agent booked a load, so the rate ceiling was checked. That doesn’t show the scenario would catch an agent breaking it. A booking at the posted rate checks the ceiling without testing it. The only evidence that your scenarios catch a mistake is a run in which an agent made it and the conversation failed. So rehearse a careless agent too.--without-policy makes one from a built-in model agent: the model keeps its tools, the generic instructions and each scenario’s context, but not the customer’s procedure. It acts sensibly without knowing their rules, so each rule it breaks that fails a conversation is one your scenarios catch. A soft rule it breaks is reported, never caught. It calls a model, so it costs money like any rehearsal:
openai: or bedrock: you can also try a smaller model of another family. Rules it never breaks stay not shown caught on the page, which is the honest answer. The do-nothing noop agent won’t do, because it never acts.
The freight example ships a careless agent of its own, so this step is free there: scripted:naive books without vetting and pays whatever is asked.
run exits 1 on its failed critical scenarios.
3. Write the page
customer-report.html next to results.json (or where -o says): one static file with no script, which you can attach, print or save as PDF. The conversations that show a rule broken or caught are open, so they print from any browser. The lists of scenarios print open from Chrome or Edge, and other examples print only when you open them first. It holds:
- A summary in plain words: how many scenarios passed every conversation, critical scenarios, rules checked and broken, what the careless agent was caught doing, and the open items before go-live. Open items are drafted items still pending review, conversations that weren’t graded, failed critical scenarios, broken rules, and scenarios that didn’t pass every conversation.
- Each rule of the procedure, with the sentence of the customer’s document it comes from. For each rule:
- how many conversations checked it, and whether any broke it;
- whether the rehearsal caught the careless agent breaking it, or the rule is not shown caught. A soft rule never fails a conversation, so it is reported, never caught;
- the scenarios that checked it, and an example conversation.
- Every scenario and its result, critical ones first.
- What the rehearsal doesn’t cover: rules no conversation checked, actions the agent never took, soft rules, and the limits of a simulation (simulated callers and records, and nothing graded beyond the written rules and outcomes).
- A sign-off block for a name, a role and a date.
source in the spec, which init --sop writes when it drafts the rule. Add one to a rule you wrote by hand, with the document and the exact sentence:
4. Close the gaps before you send it
Read the page as the customer will, then fix what it shows:- Open items: fix the agent or the spec, then rerun. A drafted item pending review is the customer’s rule, not yet confirmed: review it with them.
- Rules not checked: add a scenario that gives the agent a reason to take the action the rule governs.
lintlists the rules no scenario puts in play before you spend anything on a run. - Rules not shown caught: the careless agent never broke them, so no scenario has shown it catches that mistake. Add a scenario that tempts the agent, such as a caller who pushes past the limit, or a careless agent that makes the mistake.