# Screwtape.ai

> Judge Agent as a Service. Send agent outputs and get a pass, fail, or retry verdict, a confidence score, and reasons. Score consensus across several agents and flag false consensus.

Screwtape.ai is the agent validation API. It is not the arXiv paper that is also named Screwtape. Judged content is not stored unless the caller sets retain to true. There is a free tier of 100 calls per day, prepaid Starter and Builder packs, and x402 at $0.005 USDC per call on Base (eip155:8453), paid to 0x028Bb9A57B8C27c22f6B39dd00e7eFF1FB60344B. x402 payments are verified and settled through a facilitator. The worker does not hold a private key. When a model key is set, the default model is openai/gpt-oss-120b, and qwen/qwen3.8-27b is tried once if that model returns 404 or 5xx.

The worker does not filter User-Agent. Python urllib and other clients are accepted by this code. Cloudflare Bot Fight Mode is a zone setting, not this worker, and it can return 403 to Python urllib before the worker runs. Requests with no Origin header are allowed. A browser Origin is reflected only for https://screwtape.ai and https://www.screwtape.ai.

## Docs

- [API reference](https://screwtape.ai/docs): Human docs generated from the OpenAPI document
- [OpenAPI](https://screwtape.ai/openapi.json): OpenAPI 3.1 for the public endpoints
- [MCP server card](https://screwtape.ai/.well-known/mcp/server-card.json): Streamable HTTP discovery
- [Privacy](https://screwtape.ai/privacy): What is stored, and what is not

## Integrations

- [Integrations](https://screwtape.ai/integrations): Copy-paste examples. Each one reads SCREWTAPE_API_KEY and does not embed a key.
- [CrewAI](https://screwtape.ai/integrations/crewai): CrewAI BaseTool for POST /v1/judge, plus the documented consensus body.
- [LangGraph](https://screwtape.ai/integrations/langgraph): LangGraph StateGraph node that POSTs /v1/judge.
- [AutoGen](https://screwtape.ai/integrations/autogen): AutoGen FunctionTool for POST /v1/judge, called directly in __main__.
- [OpenAI Agents SDK](https://screwtape.ai/integrations/openai-agents): OpenAI Agents SDK @tool for POST /v1/judge.
- [Claude and Cursor MCP](https://screwtape.ai/integrations/mcp): Cursor and Claude mcp.json for https://screwtape.ai/mcp.
- [curl](https://screwtape.ai/integrations/curl): curl for POST /v1/judge and POST /v1/consensus.

## Blog

- [Blog](https://screwtape.ai/blog): Notes on the API's real behavior
- [False consensus in multi-agent swarms and how to catch it](https://screwtape.ai/blog/false-consensus): How POST /v1/consensus scores agreement and false-consensus risk, with the deterministic results for five answer sets.

## Machine-readable

- [llms-full.txt](https://screwtape.ai/llms-full.txt): The API reference, the integration examples, and the false-consensus article
- [A2A agent card](https://screwtape.ai/.well-known/agent-card.json): Judge and consensus skills. This server does not implement A2A SendMessage.

## Optional

- [Health](https://screwtape.ai/v1/health): Liveness, no secrets
- [Get an API key](https://screwtape.ai/signup): Free tier, 100 calls per day

# API reference

# Screwtape.ai

Screwtape.ai returns a pass, fail, or retry verdict on another agent's output, and scores consensus across several agents, including false-consensus risk. It is the agent validation API, not the arXiv paper also named Screwtape. Authenticate with Authorization: Bearer or, for a single call, an x402 PAYMENT-SIGNATURE header. That payment is $0.005 USDC on Base (eip155:8453) to 0x028Bb9A57B8C27c22f6B39dd00e7eFF1FB60344B, verified and settled through a facilitator. The worker does not hold a private key. When a model key is configured, the default model is openai/gpt-oss-120b, with qwen/qwen3.8-27b tried once if the primary model returns 404 or 5xx. Judged content is not stored unless retain is true. The public spec omits operator-only routes.

Base URL: https://screwtape.ai

## GET /v1/health

Health

Liveness. Does not require an API key and does not report secrets.

- `200`: Service is up.

## POST /v1/judge

Judge one agent output

Returns a verdict, a confidence score, reasons, and a request id. Free keys include 100 calls per UTC day. Prepaid credits are used after that. With no API key, the response is 402 and describes an x402 payment of $0.005 USDC on Base, which is verified and settled through a facilitator.

```json
{
  "task": "State the crew size and launch site.",
  "agent_output": "The crew size was 4. The mission launched from Cape Canaveral.",
  "evidence": [
    "The mission launched on 2024-03-01 from Cape Canaveral.",
    "The crew size was 4."
  ],
  "criteria": [
    "crew size",
    "Cape Canaveral"
  ]
}
```

- `200`: Verdict.
- `400`: Invalid request.
- `401`: API key was not recognized.
- `402`: Payment required. PAYMENT-REQUIRED carries the x402 requirements. A valid payment is verified and settled through a facilitator.
- `429`: Daily quota or rate limit exceeded.

## POST /v1/consensus

Score consensus across agents

Returns the consensus answer, an agreement score, false-consensus risk, and a request id.

```json
{
  "task": "State the crew size.",
  "answers": [
    {
      "agent_id": "alpha",
      "output": "The crew size was 4."
    },
    {
      "agent_id": "beta",
      "output": "The crew size was 4."
    },
    {
      "agent_id": "gamma",
      "output": "The crew size was 4."
    }
  ]
}
```

- `200`: Consensus.
- `400`: Invalid request.
- `401`: API key was not recognized.
- `402`: Payment required. A valid payment is verified and settled through a facilitator.
- `429`: Daily quota or rate limit exceeded.

## POST /v1/keys

Create a free API key

Returns the secret key once. Only a hash is stored. Limited to three active keys per email.

```json
{
  "email": "dev@example.com",
  "name": "Local dev"
}
```

- `201`: Key created.
- `400`: Invalid signup.
- `429`: Signup rate limit or email cap.

## GET /v1/keys/me

Current key

Usage and credit balance for the Bearer key. The secret is not returned.

- `200`: Key metadata.
- `401`: API key was not recognized.

## DELETE /v1/keys/me

Delete the current key and its data

Deletes the key hash, daily counters, payment records, and any opt-in retained judgments.

- `200`: Deleted.
- `401`: API key was not recognized.

## GET /v1/retention

List opt-in retained calls

Returns calls this key asked to retain. Rows expire after 24 hours.

- `200`: Retained rows.
- `401`: API key was not recognized.

## DELETE /v1/retention/{id}

Delete one retained call

- `id`: Retention id returned when retain was true.

- `200`: Deleted.
- `401`: API key was not recognized.
- `404`: No such row for this key.

## POST /v1/billing/checkout

Start a Stripe test-mode checkout

Starter is $30 for 10,000 calls. Builder is $100 for 50,000 calls. Requires a Bearer key. Live-mode Stripe keys are refused. Credits are applied by the webhook, not by the success page.

```json
{
  "pack": "starter"
}
```

- `200`: Checkout URL.
- `401`: API key was not recognized.
- `503`: Stripe test checkout is not configured, or a live key was refused.

## POST /mcp

MCP Streamable HTTP

JSON-RPC for initialize, tools/list, and tools/call. Tools are judge and consensus. tools/call uses the same auth and quota as the REST API. GET is reserved and returns 405 because this server does not open an SSE stream.

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/list"
}
```

- `200`: JSON-RPC result.
- `202`: Notification accepted.
- `402`: tools/call requires a key or an x402 payment, which is verified and settled through a facilitator.

# Integrations

## CrewAI

A CrewAI BaseTool that POSTs /v1/judge. The script also POSTs the documented consensus body.

```python
"""CrewAI tool that POSTs /v1/judge to Screwtape.ai."""
import json
import os
import urllib.request
from crewai import Agent
from crewai.tools import BaseTool
from pydantic import BaseModel, Field

API = "https://screwtape.ai"


def screwtape_post(path: str, payload: dict) -> dict:
    """POST JSON to Screwtape.ai. Requires SCREWTAPE_API_KEY."""
    key = os.environ["SCREWTAPE_API_KEY"]
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + key,
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(request) as response:
        body = json.load(response)
        if not isinstance(body, dict):
            raise RuntimeError("expected a JSON object")
        return body


JUDGE = {
    "task": "State the crew size and launch site.",
    "agent_output": "The crew size was 4. The mission launched from Cape Canaveral.",
    "evidence": [
        "The mission launched on 2024-03-01 from Cape Canaveral.",
        "The crew size was 4."
    ],
    "criteria": [
        "crew size",
        "Cape Canaveral"
    ]
}

CONSENSUS = {
    "task": "State the crew size.",
    "answers": [
        {
            "agent_id": "alpha",
            "output": "The crew size was 4."
        },
        {
            "agent_id": "beta",
            "output": "The crew size was 4."
        },
        {
            "agent_id": "gamma",
            "output": "The crew size was 4."
        }
    ]
}

class JudgeInput(BaseModel):
    task: str = Field(..., description="The task the other agent was given.")
    agent_output: str = Field(..., description="The output to judge.")


class ScrewtapeJudgeTool(BaseTool):
    name: str = "screwtape_judge"
    description: str = "Judge an agent output. Returns JSON with verdict, score, and reasons."
    args_schema: type[BaseModel] = JudgeInput

    def _run(self, task: str, agent_output: str) -> str:
        return json.dumps(screwtape_post("/v1/judge", {"task": task, "agent_output": agent_output}))


def reviewer() -> Agent:
    """Build a CrewAI agent that can call the judge tool. Requires your CrewAI model setup."""
    return Agent(
        role="Reviewer",
        goal="Judge another agent's answer before accepting it.",
        backstory="You call screwtape_judge and report the verdict, score, and reasons.",
        tools=[ScrewtapeJudgeTool()],
        verbose=False,
    )


if __name__ == "__main__":
    print(json.dumps(screwtape_post("/v1/judge", JUDGE)))
    print(json.dumps(screwtape_post("/v1/consensus", CONSENSUS)))
```

## LangGraph

A one-node LangGraph graph. The judge node POSTs /v1/judge and stores the JSON on verdict.

```python
"""LangGraph node that POSTs /v1/judge to Screwtape.ai."""
import json
import os
import urllib.request
from typing import TypedDict

from langgraph.graph import END, StateGraph

API = "https://screwtape.ai"


def screwtape_post(path: str, payload: dict) -> dict:
    """POST JSON to Screwtape.ai. Requires SCREWTAPE_API_KEY."""
    key = os.environ["SCREWTAPE_API_KEY"]
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + key,
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(request) as response:
        body = json.load(response)
        if not isinstance(body, dict):
            raise RuntimeError("expected a JSON object")
        return body


JUDGE = {
    "task": "State the crew size and launch site.",
    "agent_output": "The crew size was 4. The mission launched from Cape Canaveral.",
    "evidence": [
        "The mission launched on 2024-03-01 from Cape Canaveral.",
        "The crew size was 4."
    ],
    "criteria": [
        "crew size",
        "Cape Canaveral"
    ]
}

CONSENSUS = {
    "task": "State the crew size.",
    "answers": [
        {
            "agent_id": "alpha",
            "output": "The crew size was 4."
        },
        {
            "agent_id": "beta",
            "output": "The crew size was 4."
        },
        {
            "agent_id": "gamma",
            "output": "The crew size was 4."
        }
    ]
}

class State(TypedDict, total=False):
    task: str
    agent_output: str
    evidence: list[str]
    criteria: list[str]
    verdict: dict


def judge(state: State) -> dict:
    payload: dict = {"task": state["task"], "agent_output": state["agent_output"]}
    if "evidence" in state:
        payload["evidence"] = state["evidence"]
    if "criteria" in state:
        payload["criteria"] = state["criteria"]
    return {"verdict": screwtape_post("/v1/judge", payload)}


graph = StateGraph(State)
graph.add_node("judge", judge)
graph.set_entry_point("judge")
graph.add_edge("judge", END)
app = graph.compile()


if __name__ == "__main__":
    print(app.invoke({
        "task": JUDGE["task"],
        "agent_output": JUDGE["agent_output"],
        "evidence": JUDGE["evidence"],
        "criteria": JUDGE["criteria"],
    }))
    print(screwtape_post("/v1/consensus", CONSENSUS))
```

## AutoGen

An AutoGen FunctionTool for POST /v1/judge. The file does not construct a model client.

```python
"""AutoGen FunctionTool that POSTs /v1/judge to Screwtape.ai."""
import json
import os
import urllib.request
from autogen_core.tools import FunctionTool

API = "https://screwtape.ai"


def screwtape_post(path: str, payload: dict) -> dict:
    """POST JSON to Screwtape.ai. Requires SCREWTAPE_API_KEY."""
    key = os.environ["SCREWTAPE_API_KEY"]
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + key,
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(request) as response:
        body = json.load(response)
        if not isinstance(body, dict):
            raise RuntimeError("expected a JSON object")
        return body


JUDGE = {
    "task": "State the crew size and launch site.",
    "agent_output": "The crew size was 4. The mission launched from Cape Canaveral.",
    "evidence": [
        "The mission launched on 2024-03-01 from Cape Canaveral.",
        "The crew size was 4."
    ],
    "criteria": [
        "crew size",
        "Cape Canaveral"
    ]
}

CONSENSUS = {
    "task": "State the crew size.",
    "answers": [
        {
            "agent_id": "alpha",
            "output": "The crew size was 4."
        },
        {
            "agent_id": "beta",
            "output": "The crew size was 4."
        },
        {
            "agent_id": "gamma",
            "output": "The crew size was 4."
        }
    ]
}

def screwtape_judge(task: str, agent_output: str) -> str:
    """Judge an agent output. Returns JSON with verdict, score, and reasons."""
    return json.dumps(screwtape_post("/v1/judge", {"task": task, "agent_output": agent_output}))


judge_tool = FunctionTool(screwtape_judge, description=screwtape_judge.__doc__ or "Judge an agent output.")

# Pass judge_tool to AssistantAgent(tools=[judge_tool]). This file does not construct a model client.


if __name__ == "__main__":
    print(screwtape_judge(JUDGE["task"], JUDGE["agent_output"]))
    print(json.dumps(screwtape_post("/v1/judge", JUDGE)))
    print(json.dumps(screwtape_post("/v1/consensus", CONSENSUS)))
```

## OpenAI Agents SDK

An OpenAI Agents SDK tool for POST /v1/judge. __main__ calls the wrapped function and the consensus example.

```python
"""OpenAI Agents SDK tool that POSTs /v1/judge to Screwtape.ai."""
import json
import os
import urllib.request
from agents import Agent
from agents.decorators import tool

API = "https://screwtape.ai"


def screwtape_post(path: str, payload: dict) -> dict:
    """POST JSON to Screwtape.ai. Requires SCREWTAPE_API_KEY."""
    key = os.environ["SCREWTAPE_API_KEY"]
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + key,
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(request) as response:
        body = json.load(response)
        if not isinstance(body, dict):
            raise RuntimeError("expected a JSON object")
        return body


JUDGE = {
    "task": "State the crew size and launch site.",
    "agent_output": "The crew size was 4. The mission launched from Cape Canaveral.",
    "evidence": [
        "The mission launched on 2024-03-01 from Cape Canaveral.",
        "The crew size was 4."
    ],
    "criteria": [
        "crew size",
        "Cape Canaveral"
    ]
}

CONSENSUS = {
    "task": "State the crew size.",
    "answers": [
        {
            "agent_id": "alpha",
            "output": "The crew size was 4."
        },
        {
            "agent_id": "beta",
            "output": "The crew size was 4."
        },
        {
            "agent_id": "gamma",
            "output": "The crew size was 4."
        }
    ]
}

@tool
def screwtape_judge(task: str, agent_output: str) -> str:
    """Judge an agent output.

    Args:
        task: The task the other agent was given.
        agent_output: The output to judge.
    """
    return json.dumps(screwtape_post("/v1/judge", {"task": task, "agent_output": agent_output}))


def reviewer() -> Agent:
    """Build an Agents SDK agent that can call the judge tool. Runner.run needs your OpenAI key."""
    return Agent(
        name="Reviewer",
        instructions="Call screwtape_judge before you accept an answer. Report the verdict, score, and reasons.",
        tools=[screwtape_judge],
    )


if __name__ == "__main__":
    print(screwtape_judge.__wrapped__(JUDGE["task"], JUDGE["agent_output"]))
    print(json.dumps(screwtape_post("/v1/consensus", CONSENSUS)))
```

## Claude and Cursor MCP

mcp.json for the Streamable HTTP server at https://screwtape.ai/mcp. Tools are judge and consensus.

Cursor:

```json
{
  "mcpServers": {
    "screwtape": {
      "url": "https://screwtape.ai/mcp",
      "headers": {
        "Authorization": "Bearer ${env:SCREWTAPE_API_KEY}"
      }
    }
  }
}
```

Claude:

```json
{
  "mcpServers": {
    "screwtape": {
      "type": "http",
      "url": "https://screwtape.ai/mcp",
      "headers": {
        "Authorization": "Bearer ${SCREWTAPE_API_KEY}"
      }
    }
  }
}
```

## curl

A shell script that POSTs the documented judge body and the documented consensus body.

```bash
#!/bin/sh
set -eu
if [ -z "${SCREWTAPE_API_KEY:-}" ]; then
  echo "Set SCREWTAPE_API_KEY" >&2
  exit 1
fi

curl -sS https://screwtape.ai/v1/judge \
  -H "Authorization: Bearer ${SCREWTAPE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @- <<'JSON'
{
  "task": "State the crew size and launch site.",
  "agent_output": "The crew size was 4. The mission launched from Cape Canaveral.",
  "evidence": [
    "The mission launched on 2024-03-01 from Cape Canaveral.",
    "The crew size was 4."
  ],
  "criteria": [
    "crew size",
    "Cape Canaveral"
  ]
}
JSON

echo

curl -sS https://screwtape.ai/v1/consensus \
  -H "Authorization: Bearer ${SCREWTAPE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @- <<'JSON'
{
  "task": "State the crew size.",
  "answers": [
    {
      "agent_id": "alpha",
      "output": "The crew size was 4."
    },
    {
      "agent_id": "beta",
      "output": "The crew size was 4."
    },
    {
      "agent_id": "gamma",
      "output": "The crew size was 4."
    }
  ]
}
JSON
```

# False consensus in multi-agent swarms and how to catch it

20 October 2026

Screwtape.ai is the agent validation API. An unrelated arXiv paper is also called Screwtape.

POST /v1/consensus takes a task and two to sixteen answers. Each answer has an output and an optional agent_id. The JSON body includes consensus_answer, agreement_score, false_consensus_risk (low, medium, or high), reasons, clusters, and engine. It also includes request_id and retention. The figures below are the output of the deterministic function the API returns when no model key is configured, for these five requests.

## How a score is chosen

Answers are read in request order. Tokens are lowercase runs of letters and digits of length at least 2. Stemming removes a trailing ing when the token is longer than 5 characters, a trailing ed when it is longer than 4, and a trailing s when it is longer than 4 and does not end in ss. A fixed stopword list is dropped. Similarity is the Jaccard score of the two token sets: the size of the intersection divided by the size of the union.

An answer joins the first cluster whose first member has Jaccard similarity of at least 0.55. Otherwise it starts a new cluster. The largest cluster wins. Equal sizes keep the earlier cluster. agreement_score is that cluster's size divided by the number of answers, rounded to two decimal places. consensus_answer is the output of the cluster's first member.

false_consensus_risk is chosen in this order. It is high when agreement is at least 0.8 and the mean pairwise Jaccard inside the winning cluster is at least 0.9. Mean pairwise Jaccard is 1 when the winning cluster has one answer. It is high when agreement is at least 0.8 and the representative has fewer than 4 tokens. It is medium when agreement is at least 0.6 and some answer outside the winning cluster has Jaccard similarity below 0.35 with the representative. It is low when agreement is below 0.6. Otherwise it stays low. Every deterministic result ends with this reason: Agreement is token overlap, not a model judgment.

## Five answer sets

### Verbatim copies

Three agents returned the same sentence.

```json
{
  "task": "State the crew size.",
  "answers": [
    {
      "agent_id": "alpha",
      "output": "The crew size was 4."
    },
    {
      "agent_id": "beta",
      "output": "The crew size was 4."
    },
    {
      "agent_id": "gamma",
      "output": "The crew size was 4."
    }
  ]
}
```

Deterministic result for that request:

```json
{
  "consensus_answer": "The crew size was 4.",
  "agreement_score": 1,
  "false_consensus_risk": "high",
  "reasons": [
    "Answers are nearly verbatim. High agreement may be copied rather than independent.",
    "Agreement is token overlap, not a model judgment."
  ],
  "clusters": [
    {
      "size": 3,
      "agent_ids": [
        "alpha",
        "beta",
        "gamma"
      ],
      "representative": "The crew size was 4."
    }
  ],
  "engine": "deterministic"
}
```

### A majority and one dissent

Two agents returned the same sentence. The third answered a different question.

```json
{
  "task": "State the crew size.",
  "answers": [
    {
      "agent_id": "a",
      "output": "The crew size was 4."
    },
    {
      "agent_id": "b",
      "output": "The crew size was 4."
    },
    {
      "agent_id": "c",
      "output": "Mars is a dry planet with two small moons and a thin atmosphere"
    }
  ]
}
```

Deterministic result for that request:

```json
{
  "consensus_answer": "The crew size was 4.",
  "agreement_score": 0.67,
  "false_consensus_risk": "medium",
  "reasons": [
    "A minority answer materially disagrees with the majority.",
    "Agreement is token overlap, not a model judgment."
  ],
  "clusters": [
    {
      "size": 2,
      "agent_ids": [
        "a",
        "b"
      ],
      "representative": "The crew size was 4."
    },
    {
      "size": 1,
      "agent_ids": [
        "c"
      ],
      "representative": "Mars is a dry planet with two small moons and a thin atmosphere"
    }
  ],
  "engine": "deterministic"
}
```

### No shared majority

Three agents returned unrelated sentences.

```json
{
  "task": "Describe the finding.",
  "answers": [
    {
      "agent_id": "a",
      "output": "Alpha bravo charlie delta echo foxtrot"
    },
    {
      "agent_id": "b",
      "output": "Hotel india juliet kilo lima mike"
    },
    {
      "agent_id": "c",
      "output": "November oscar papa quebec romeo sierra"
    }
  ]
}
```

Deterministic result for that request:

```json
{
  "consensus_answer": "Alpha bravo charlie delta echo foxtrot",
  "agreement_score": 0.33,
  "false_consensus_risk": "low",
  "reasons": [
    "Agents disagree, so there is no false consensus and no stable majority.",
    "Agreement is token overlap, not a model judgment."
  ],
  "clusters": [
    {
      "size": 1,
      "agent_ids": [
        "a"
      ],
      "representative": "Alpha bravo charlie delta echo foxtrot"
    },
    {
      "size": 1,
      "agent_ids": [
        "b"
      ],
      "representative": "Hotel india juliet kilo lima mike"
    },
    {
      "size": 1,
      "agent_ids": [
        "c"
      ],
      "representative": "November oscar papa quebec romeo sierra"
    }
  ],
  "engine": "deterministic"
}
```

### Shared wording that is not verbatim

Three agents share most tokens, and each changes the last token.

```json
{
  "task": "List the tokens.",
  "answers": [
    {
      "agent_id": "a",
      "output": "alpha bravo charlie delta echo foxtrot golf"
    },
    {
      "agent_id": "b",
      "output": "alpha bravo charlie delta echo foxtrot hotel"
    },
    {
      "agent_id": "c",
      "output": "alpha bravo charlie delta echo foxtrot india"
    }
  ]
}
```

Deterministic result for that request:

```json
{
  "consensus_answer": "alpha bravo charlie delta echo foxtrot golf",
  "agreement_score": 1,
  "false_consensus_risk": "low",
  "reasons": [
    "The majority agrees without a verbatim pile-on or a strong dissent.",
    "Agreement is token overlap, not a model judgment."
  ],
  "clusters": [
    {
      "size": 3,
      "agent_ids": [
        "a",
        "b",
        "c"
      ],
      "representative": "alpha bravo charlie delta echo foxtrot golf"
    }
  ],
  "engine": "deterministic"
}
```

### A short agreed answer

Both answers are short. They overlap, and they are not the same token set.

```json
{
  "task": "Name the pair.",
  "answers": [
    {
      "agent_id": "a",
      "output": "alpha bravo"
    },
    {
      "agent_id": "b",
      "output": "alpha bravo charlie"
    }
  ]
}
```

Deterministic result for that request:

```json
{
  "consensus_answer": "alpha bravo",
  "agreement_score": 1,
  "false_consensus_risk": "high",
  "reasons": [
    "The agreed answer is too thin to treat as an independent consensus.",
    "Agreement is token overlap, not a model judgment."
  ],
  "clusters": [
    {
      "size": 2,
      "agent_ids": [
        "a",
        "b"
      ],
      "representative": "alpha bravo"
    }
  ],
  "engine": "deterministic"
}
```

## How to catch it

Read false_consensus_risk before you treat consensus_answer as confirmed. high means the winning cluster cleared the verbatim rule or the short-answer rule above. medium means agreement cleared 0.6 and an answer outside that cluster fell below the dissent threshold, so read every cluster. low with agreement_score below 0.6 means there is no stable majority. low with a higher agreement_score means the winning cluster cleared 0.55 without the verbatim rule or the dissent rule. The shared-wording example is that last case: agreement_score is 1 and the risk is low.

clusters lists each group: size, agent_ids, and the representative output. reasons states which rule fired. When the risk is high, those reasons say the answers may be copied rather than independent, or that the agreed text is too thin to treat as an independent consensus.

## When a model key is set

With a model key, the API asks the configured model for consensus_answer, agreement_score, false_consensus_risk, and reasons. A usable reply replaces those fields, keeps the deterministic clusters, sets engine to llm, and appends: Scored by the configured LLM. Clusters below are from deterministic overlap.

A 404 or a status of 500 or higher from the primary model tries the fallback model once, when that fallback differs from the primary. Any other HTTP status, including 400, keeps the deterministic result. A thrown request, including a timeout, does the same and does not try the fallback. Those failures append: LLM call failed. Used deterministic checks. If the model returns an object that is missing a required field, the deterministic result is kept and this reason is appended: LLM response was unusable. Used deterministic checks.

## What the HTTP response adds

The handler adds request_id and retention to the result above. retention.retained is false unless the request sets retain to true and the caller uses an API key. A keyless call that sets retain to true gets retained false and this reason: Retention requires an API key. An opt-in row is kept for 24 hours. The samples above omit request_id because that value is assigned per call.

- [API reference](https://screwtape.ai/docs)
- [curl example for the same judge and consensus bodies](https://screwtape.ai/integrations/curl)