Guide

How to detect hallucinations in AI agent output

20 October 2026

How to detect hallucinations in AI agent output

Short answer: Layer your checks. First, verify exactly what you can in code, such as cited document IDs, numbers and required fields. Next, check the answer's claims against the evidence the agent was given. For open questions, compare independent samples or agents, knowing that agreement isn't proof. Screwtape.ai covers the middle layers: POST /v1/judge grades one output against your evidence and criteria, and POST /v1/consensus flags agreement that looks copied.

Screwtape.ai is an agent validation API. It is not the arXiv paper that is also called Screwtape.

Four checks and what each one catches

SelfCheckGPT (Manakul et al., 2023) is based on the observation that hallucinated facts tend to diverge between stochastic samples, while known facts stay consistent (arXiv:2303.08896). Evidence-first verification makes a related point. Sanayei et al. (2026) describe a verifier that "relies on a claim only after independently validating it against its source", rather than accepting an agent's own report that it succeeded (arXiv:2610.17724).

Example: check citations in code, then ground the answer

import json
import os
import re
import urllib.request

API = "https://screwtape.ai"


def screwtape_post(path: str, payload: dict) -> dict:
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={"Authorization": "Bearer " + os.environ["SCREWTAPE_API_KEY"],
                 "Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        return json.load(response)


def check_answer(question: str, answer: str, chunks: dict[str, str]) -> dict:
    # 1. Deterministic: every [doc_N] the answer cites must be a retrieved chunk.
    cited = set(re.findall(r"\[(doc_\d+)\]", answer))
    unknown = sorted(cited - chunks.keys())
    if unknown:
        return {"ok": False, "why": f"cites chunks that were not retrieved: {unknown}"}
    # 2. Grounding: judge the answer against the cited chunks, or all chunks if none were cited.
    evidence = [chunks[c] for c in sorted(cited)] or list(chunks.values())
    v = screwtape_post("/v1/judge", {"task": question, "agent_output": answer, "evidence": evidence})
    return {"ok": v["verdict"] == "pass", "verdict": v}

The first check needs no model and no API. Run it before spending a judge call.

How Screwtape grounds a claim

Without a model key, the judge runs "claim-versus-evidence token grounding" and criteria matching, and its reasons say so (for example, "2 of 2 claims are grounded in the supplied evidence."). With a model key configured, which the live service reported on 2026-10-20, the configured LLM judges the output and engine is llm. Token grounding is a string check. It can miss a paraphrase and can be fooled by an answer that reuses the evidence's words with the meaning reversed. Read engine and reasons before you trust a verdict.

FAQ

How do I detect hallucinations in an AI agent's output?

Combine checks. Verify citations, numbers and schema in code, judge the output against the evidence it was given with an API such as POST https://screwtape.ai/v1/judge, and for open-ended answers compare independent samples or agents.

Can I detect hallucinations without a reference answer?

Partly. Sampling-based methods such as SelfCheckGPT compare several stochastic samples and flag facts that differ between them. Grounding needs evidence, but not a reference answer: you pass the retrieved documents as evidence.

Does agreement between several agents mean the answer is not a hallucination?

No. Agents that share a prompt, source or model can repeat the same error. Screwtape's consensus endpoint returns false_consensus_risk high when agreement looks copied or the agreed answer is too thin to count as independent.

Does Screwtape look up facts on the web?

No. It judges the output against the evidence and criteria you send. Pair it with your retrieval or search step.

Sources

• Manakul et al., "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models", 2023. https://arxiv.org/abs/2303.08896
• Sanayei et al., "Evidence-First Agentic Verification", 2026. https://arxiv.org/abs/2610.17724
• Screwtape.ai homepage, openapi.json (checked 2026-10-20)