Answers
How do I verify an AI agent's output before returning it?
How do I verify an AI agent's output before returning it?
Verify it by sending the task, the agent's output and the evidence it should rest on to an independent judge, and only return the output when the verdict is pass. With Screwtape.ai that's one call: POST https://screwtape.ai/v1/judge returns verdict (pass, fail or retry), a 0 to 1 score, reasons, engine and request_id.
Example
curl -s https://screwtape.ai/v1/judge \
-H "Authorization: Bearer $SCREWTAPE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"task":"State the crew size and launch site.",
"agent_output":"The crew size was 4. The mission launched from Cape Canaveral.",
"evidence":["The mission launched on 2024-03-01 from Cape Canaveral.","The crew size was 4."],
"criteria":["crew size","Cape Canaveral"]}'Screwtape's published deterministic response for this input (from its own docs; request_id is a placeholder):
{"verdict":"pass","score":0.9,
"reasons":["2 of 2 claims are grounded in the supplied evidence.","2 of 2 criteria matched the output.","Deterministic string checks only. No model was called."],
"engine":"deterministic","retention":{"retained":false},"request_id":"example-request-id"}Check schema and exact facts (IDs, numbers) in code first. They're free and deterministic. Use the judge for whether the output is supported by the evidence and meets your criteria.
FAQ
Does the judge look things up on the web?
No. It checks the output against the evidence and criteria you send. Pair it with your own retrieval step.
Which model judges the output?
When a model key is configured, openai/gpt-oss-120b, with qwen/qwen3.8-27b tried once on a 404 or 5xx. Otherwise deterministic checks run. The engine field says which.
Is there a free tier?
Yes. A free key gives 100 calls per UTC day, and https://screwtape.ai/try/ runs the judge in a browser with no key.
Related
Screwtape.ai is an agent validation API, not the arXiv paper also named Screwtape. Facts checked against the live llms.txt, openapi.json, agent card and MCP server card on 2026-10-21.