Connect your agent: Claude, Bedrock, OpenAI-compatible or HTTP

​

Pick an adapter

export ANTHROPIC_API_KEY=...
npx sandtable-ai run ./my-domain --agent claude -k 4

npm install --save-dev sandtable-ai @anthropic-ai/bedrock-sdk
npx sandtable-ai run ./my-domain --provider bedrock --region eu-west-1 --agent claude -k 4

OPENAI_API_KEY=sk-... npx sandtable-ai run ./my-domain --agent openai:gpt-4.1-mini -k 4

npx sandtable-ai run ./my-domain --agent http://localhost:8787/step -k 4

npx sandtable-ai run ./my-domain --agent ./my-agent.ts -k 4

​

Claude on the Anthropic API

• Authentication uses ANTHROPIC_API_KEY.
• The system prompt, tools and history form an append-only prefix that is cached automatically, which keeps repeated trials cheap.
• A user turn ends after 12 model calls; the reply is then [agent exceeded its tool-call budget for this turn].
• Where the model supports it, sandtable opts into server-side refusal fallbacks on the Anthropic API. A fallback means another model served that turn. To measure one model strictly, build the agent in code with claudeAgent({ fallbacks: false }).
• Tokens are recorded per model, and cost is estimated at Anthropic list prices.

​

Claude on Amazon Bedrock

npm install --save-dev sandtable-ai @anthropic-ai/bedrock-sdk
npx sandtable-ai demo --live --provider bedrock --region eu-west-1 --aws-profile my-profile

​

Other models on Amazon Bedrock

npm install --save-dev sandtable-ai @aws-sdk/client-bedrock-runtime
npx sandtable-ai run ./my-domain --agent bedrock:qwen.qwen3-32b-v1:0 -k 4

• <model-id> is the Bedrock model or inference profile ID, passed through unchanged. The model must be enabled for your account in that region.
• The region comes from --region, then AWS_REGION, then AWS_DEFAULT_REGION. Credentials come from the default AWS chain, which honours AWS_PROFILE. Passing --aws-profile also needs @aws-sdk/credential-providers, installed the same way.
• A turn allows up to 12 model calls of up to 16,000 output tokens each, reasoning included, the same as the Claude adapter. Output stopped by a guardrail or content filter becomes [agent output was filtered].
• A tool call runs when its reply ends with tool_use or end_turn; Grok 4.7 sometimes sends a call with end_turn. A call in a reply that ended any other way, such as cut off at the output limit, filtered or malformed, is not run, and the agent gets an error result for it.
• Reasoning never reaches the caller or the graders. Reasoning blocks are left out of the reply, and so is reasoning a model writes into its text as <reasoning>…</reasoning>, as gpt-oss sometimes does. A reply that opens <reasoning> and never closes it is requested again, up to twice. The discarded reply appears in neither the conversation nor the transcript, but its tokens, cost and time count toward the turn.
• Prompt caching is automatic on Claude and Amazon Nova models. The adapter marks the tools, the system prompt and the newest message as cacheable, so each call pays the cache rate for context it already sent. Nova’s cache point stays on the caller’s latest message, because Nova rejects one next to tool calls and results. Other models get none, because Bedrock rejects the whole request when one is present; neither do Claude models older than Claude 3.5 Haiku, which AWS documents as not supporting prompt caching. Some of the other models, such as OpenAI’s GPT models and Grok, cache on their own, and those tokens are recorded as cache reads too. A model is recognised by its ID, so an application inference profile ARN gets no cache points. Caching changes token counts and cost, never the prompt the model reads.
• This is an agent adapter only. The simulated user stays Claude; add --provider bedrock to run that on Bedrock too.
• Tokens are recorded. Cost shows n/a unless you pass the model’s prices with --prices: sandtable holds verified prices only for Claude models.

​

OpenAI-compatible endpoints

# OpenAI
OPENAI_API_KEY=sk-... npx sandtable-ai run ./my-domain --agent openai:gpt-4.1-mini -k 4

# OpenRouter: one key for GPT, Gemini, Llama, Qwen, DeepSeek and more
OPENAI_API_KEY=$OPENROUTER_API_KEY npx sandtable-ai run ./my-domain -k 4 \
  --openai-base-url https://openrouter.ai/api/v1 --agent openai:google/gemini-2.5-flash

# A local model on Ollama, as both agent and caller: no key needed
npx sandtable-ai run ./my-domain --openai-base-url http://localhost:11434/v1 \
  --agent openai:llama3.1:8b --user openai:llama3.1:8b

• The base URL is --openai-base-url, else OPENAI_BASE_URL, else https://api.openai.com/v1. Agent and user share it.
• OPENAI_API_KEY, when set, is sent as a bearer token. It is required only for the default OpenAI URL; local servers such as Ollama and vLLM need none.
• The model name passes through unchanged: openai:llama3.1:8b asks for llama3.1:8b.
• The caller stays Claude (ANTHROPIC_API_KEY) unless you pass --user. Keep the caller the same across agents, so that only the agent changes.
• The model must support tool calling on that server. vLLM, for example, needs --enable-auto-tool-choice and a --tool-call-parser.

{
  "model": "gpt-4.1-mini",
  "messages": [
    { "role": "system", "content": "..." },
    { "role": "user", "content": "MC 100101, anything Dallas to Atlanta?" }
  ],
  "tools": [{ "type": "function", "function": { "name": "verify_carrier", "description": "...", "parameters": { "type": "object", "properties": { } } } }]
}

• Each entry in tool_calls runs against the simulated world. Its result goes back as a role: "tool" message with the same tool_call_id. A turn ends after maxToolRounds model calls (default 12).
• Arguments that are not valid JSON go back to the model as a tool error it can correct. The attempt still shows in the trajectory as a failed call.
• History is append-only. Assistant messages go back as the endpoint returned them, including provider fields such as reasoning and Gemini’s thought signatures.
• Inline <think>…</think> blocks are dropped from the reply, so neither the caller nor the graders see them.

​

Compare models in code

import { claudeUser, loadDomain, openaiCompatAgent, runSuite } from "sandtable-ai";

const domain = await loadDomain("./my-domain");
const user = claudeUser({ model: "claude-haiku-5-5" }); // the same caller for every agent
for (const model of ["openai/gpt-4.1", "google/gemini-2.5-flash", "meta-llama/llama-3.3-70b-instruct"]) {
  const agent = openaiCompatAgent({ baseUrl: "https://openrouter.ai/api/v1", apiKey: process.env.OPENROUTER_API_KEY, model });
  const result = await runSuite({ domain, agent, user, k: 4 });
  console.log(model, result.passHat["4"]);
}

​

Your agent over HTTP

npx sandtable-ai run ./my-domain --agent http://localhost:8787/step -k 4

​

Your agent as a JS or TS module

import type { AgentAdapter } from "sandtable-ai";

const agent: AgentAdapter = {
  name: "my-agent",
  async createSession({ system, tools, sessionId, recordUsage }) {
    // system: sandtable's default prompt (domain description + policy); use your own if you have one
    // tools: JSON-schema tool definitions, in the shape most model APIs accept
    return {
      async reply(userMessage, callTool) {
        // run your own loop; every callTool() hits the simulated world and is graded
        const carrier = await callTool("verify_carrier", { mc_number: "100101" });
        return carrier.ok ? "You're all set to book." : "I couldn't find that MC number.";
      },
      async close() {
        // optional: called when the trial ends
      },
    };
  },
};
export default agent;

npx sandtable-ai run ./my-domain --agent ./my-agent.ts --user scripted

• callTool(name, args) resolves to { ok, result?, error? }. Invalid arguments and failed checks come back as ok: false with an error your agent can show the model.
• Call recordUsage(model, { input, output, cacheRead, cacheWrite }) after each model call if you want tokens and cost in the report. It is optional. Add maxTokensStops: 1 for a call whose reply stopped at the model’s output-token limit, so the report counts it.
• A .ts module needs Node 22.18 or later, which strips types natively. On older Node, compile it to .js first.

​

The simulated user