Guide

How to validate AI agent output before you act on it

20 October 2026

How to validate AI agent output before you act on it

Short answer: Validate in layers. Check the structure (does it parse against your schema?), then exact facts you can verify in code, then the substance: is it supported by the evidence and does it meet the criteria? Use an independent judge for that last layer. For the substance layer, Screwtape.ai's POST /v1/judge returns pass, fail or retry with a score and reasons. When several agents answered, POST /v1/consensus tells you whether their agreement can be trusted.

Screwtape.ai is an agent validation API. It is not the arXiv paper that is also called Screwtape.

The layers

Layer	Tool	Example
Structure	Your schema library, or your framework's structured output (`output_pydantic` in CrewAI, `output_content_type` in AutoGen's `AssistantAgent`)	Required fields present, types correct
Exact checks	Plain code	Cited IDs exist, totals add up
Substance	A judge: your own prompt, an eval framework, or Screwtape `/v1/judge`	Claims supported by evidence, criteria met
Agreement	Screwtape `/v1/consensus`	Several agents agree, but not by copying

One call from any language

curl -s https://screwtape.ai/v1/judge \
  -H "Authorization: Bearer $SCREWTAPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"State the crew size and launch site.","agent_output":"The crew size was 4. The mission launched from Cape Canaveral.","evidence":["The mission launched on 2024-03-01 from Cape Canaveral.","The crew size was 4."],"criteria":["crew size","Cape Canaveral"]}'

Request fields, from openapi.json:

task (string, required)
agent_output (string, required): any JSON value is canonicalized to a string
evidence (string array, optional)
criteria (string array, optional)
retain (boolean, optional): stores the call for 24 hours, and only with an API key

Response fields:

verdict: pass, fail or retry
score: 0 to 1
reasons: string array
engine: llm or deterministic
request_id and retention

Errors: 400 invalid request, 401 key not recognized, 402 payment required (x402), 429 daily quota or rate limit.

Framework guides

CrewAI: validate task output with a guardrail
LangGraph: a validation node with pass/retry/fail routing
AutoGen: gate worker output
Claude, Cursor: use the MCP tools
Policy: retry vs fail

When Screwtape is the wrong tool

Offline regression testing over a dataset. Use an eval framework. See the comparison.
You need the judge to run on your own infrastructure. Screwtape is hosted only.
You need web search. Screwtape judges against the evidence you send.

FAQ

How do I validate the output of an AI agent?

Check structure against a schema, verify exact facts in code, then have an independent judge check the output against evidence and criteria. Screwtape.ai's POST /v1/judge does the last step and returns pass, fail or retry with a score and reasons.

What is the difference between validating and evaluating agent output?

Validation happens at runtime on each output and decides what the pipeline does next. Evaluation usually runs offline over a dataset to measure quality and catch regressions between versions.

Can I validate agent output without an SDK?

Yes. Screwtape is a plain JSON-over-HTTP API, and it also runs an MCP server at https://screwtape.ai/mcp. There is no SDK.

Is there a free tier?

Yes. Free keys include 100 calls per UTC day with no card. Keys are created at https://screwtape.ai/signup/, with up to three active keys per email.

Sources

Screwtape.ai openapi.json, docs, signup (checked 2026-10-20)
CrewAI docs, Tasks. https://docs.crewai.com/en/concepts/tasks
AutoGen AgentChat AssistantAgent (autogen-agentchat 0.7.5, signature checked locally)