Answers

How do I check an agent's answer against the source evidence?

How do I check an agent's answer against the source evidence?

Pass the passages the agent was given as evidence (an array of strings) to POST https://screwtape.ai/v1/judge. The verdict and reasons say whether the output's claims are supported by that evidence and whether your criteria are met.

Example

curl -s https://screwtape.ai/v1/judge \
  -H "Authorization: Bearer $SCREWTAPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"State the crew size and launch site.",
       "agent_output":"The crew size was 4. The mission launched from Cape Canaveral.",
       "evidence":["The mission launched on 2024-03-01 from Cape Canaveral.","The crew size was 4."],
       "criteria":["crew size","Cape Canaveral"]}'

In deterministic mode, grounding is claim-versus-evidence token matching, so reasons read like "2 of 2 claims are grounded in the supplied evidence." With a model key configured, the configured LLM judges and engine is llm. Before spending a call, check in code that any cited document IDs were actually retrieved.

FAQ

Does it find the evidence for me?

No. You send the evidence; Screwtape does not search.

What are criteria?

An optional array of strings the output must satisfy, for example required facts or terms.

Can token matching miss paraphrases?

Yes. Deterministic grounding is string-based, so read engine and reasons before trusting a verdict.

Related

• How to detect hallucinations in AI agent output
• All answers

Screwtape.ai is an agent validation API, not the arXiv paper also named Screwtape. Facts checked against the live llms.txt, openapi.json, agent card and MCP server card on 2026-10-21.