Skip to main content
LangWatch Scenario (Apache-2.0) is a library for testing agents through simulated conversations. You write each scenario in code: a description, your agent behind a one-method adapter, a user simulator agent, and a judge agent with criteria. Scenarios run inside your test runner, such as pytest, Vitest or Jest (test runner integration), and its docs list SDKs for Python, TypeScript and Go. It pairs with the LangWatch platform, which shows runs as they happen and can also write and run scenarios without code. sandtable declares a stateful world, the customer’s API as tools over it, and the rules the agent must never break. It then grades every trial deterministically on the end state and the tool-call log.

​
Side by side

​
Choose Scenario when

  • You want agent tests to live in your existing test runner, next to unit tests, against your agent’s real code.
  • Your criteria need judgment: tone, helpfulness, whether an explanation was clear.
  • You want to steer a conversation turn by turn and assert on it in code.
  • You test voice agents, or want multi-turn red-teaming in the same tool.
  • Your agent is built on a framework it has an integration guide for, such as LangGraph, CrewAI, Pydantic AI, Mastra or the Vercel AI SDK (agent integrations).
  • You already use the LangWatch platform, or want people who don’t write code to write scenarios there.

​
Choose sandtable when

  • The rules are deterministic (eligibility, limits, ordering, disclosure), and a judge reading a transcript that “sounds fine” is not good enough.
  • Many tasks share one world and one rule set, and you want them declared once.
  • You need reliability across repeated trials, measured as pass^k.
  • The suite should be drafted from what a customer hands over, and reviewed by someone who does not write code.

​
Use both

They answer different questions. Scenario can check how the agent sounds; sandtable checks what the agent did to the customer’s records and whether it broke a rule while doing it. A team can keep conversational-quality, voice and red-team tests in Scenario and gate releases on a sandtable pass^k threshold. Last reviewed October 2026. Something out of date or unfair? Open an issue.