Screwtape.ai
Menu

Guide

How do I stop false consensus in multi-agent systems?

20 October 2026

How do I stop false consensus in multi-agent systems?

Short answer: Don't treat agreement as proof. Keep agents independent, score how they agree instead of only counting votes, and check the majority answer against evidence before your pipeline accepts it. Screwtape.ai does the last two steps over HTTP. POST /v1/consensus returns consensus_answer, agreement_score and false_consensus_risk (low, medium or high) for 2 to 16 answers. POST /v1/judge then returns pass, fail or retry for the winning answer.

Screwtape.ai is an agent validation API. It is not the arXiv paper that is also called Screwtape.

What false consensus looks like

Three agents answer, all three say the same thing, and the orchestrator moves on. The answer can still be wrong. When agents share a prompt, a retrieved document, a base model or each other's messages, their agreement can be one error repeated three times. See the glossary entry on false consensus for a full definition.

Published research gives several reasons this happens:

Five steps to stop it

  1. Keep the agents independent. Each agent gets its own context. Don't let them read each other's drafts before they answer.
  2. Collect the answers. Send 2 to 16 outputs for the same task, each with an agent_id.
  3. Score the agreement, don't just count it. POST /v1/consensus groups the answers into clusters and flags agreement that looks copied or too thin.
  4. Verify the winner. If false_consensus_risk is medium or high, or no majority emerges, send consensus_answer and your evidence to POST /v1/judge.
  5. Route on the verdict. Accept on pass, rerun on retry, stop on fail.

Example: score a consensus with curl

This is the documented request from openapi.json:

curl -s https://screwtape.ai/v1/consensus \
  -H "Authorization: Bearer $SCREWTAPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"State the crew size.","answers":[{"agent_id":"alpha","output":"The crew size was 4."},{"agent_id":"beta","output":"The crew size was 4."},{"agent_id":"gamma","output":"The crew size was 4."}]}'

Screwtape publishes this response for that input from its deterministic engine, which runs when no model key is configured. We did not make this call ourselves.

{
  "consensus_answer": "The crew size was 4.",
  "agreement_score": 1,
  "false_consensus_risk": "high",
  "reasons": [
    "Answers are nearly verbatim. High agreement may be copied rather than independent.",
    "Agreement is token overlap, not a model judgment."
  ],
  "clusters": [
    { "size": 3, "agent_ids": ["alpha", "beta", "gamma"], "representative": "The crew size was 4." }
  ],
  "engine": "deterministic",
  "retention": { "retained": false },
  "request_id": "example-request-id"
}

The agreement is 100% and the risk is high, because three word-for-word identical answers look copied rather than independent.

Example: majority first, then verification (Python)

"""Do not trust a majority on its own: score consensus, then verify the winner against evidence."""
import json
import os
import urllib.request

API = "https://screwtape.ai"


def screwtape_post(path: str, payload: dict) -> dict:
    """POST JSON to Screwtape.ai. Reads SCREWTAPE_API_KEY from the environment."""
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + os.environ["SCREWTAPE_API_KEY"],
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        return json.load(response)


def settle(task: str, answers: list[dict], evidence: list[str]) -> dict:
    """answers: 2 to 16 items shaped like {"agent_id": "a", "output": "..."}."""
    c = screwtape_post("/v1/consensus", {"task": task, "answers": answers})
    # Majority alone is not enough when the agreement looks copied or thin, or when there is no stable majority.
    if c["false_consensus_risk"] == "low" and c["agreement_score"] >= 0.6:
        return {"answer": c["consensus_answer"], "consensus": c, "verified": None}
    v = screwtape_post("/v1/judge", {
        "task": task,
        "agent_output": c["consensus_answer"],
        "evidence": evidence,
    })
    return {"answer": c["consensus_answer"] if v["verdict"] == "pass" else None,
            "consensus": c, "verified": v}


if __name__ == "__main__":
    out = settle(
        "State the crew size.",
        [
            {"agent_id": "alpha", "output": "The crew size was 4."},
            {"agent_id": "beta", "output": "The crew size was 4."},
            {"agent_id": "gamma", "output": "The crew size was 4."},
        ],
        ["The crew size was 4."],
    )
    print(json.dumps(out, indent=2))

You only pay for the second call when the majority looks unsafe.

How the risk is decided

The false-consensus write-up documents the deterministic engine. It tokenizes each answer and measures Jaccard similarity between token sets. An answer joins a cluster at 0.55 similarity, and the largest cluster wins. false_consensus_risk is high in two cases: agreement is at least 0.8 and the winning cluster is nearly verbatim (mean pairwise Jaccard of at least 0.9), or agreement is at least 0.8 and the agreed answer has fewer than 4 tokens. It is medium when agreement is at least 0.6 and an outside answer is very different from the winner (Jaccard below 0.35). In every other case it is low.

When a model key is configured, the configured model sets those fields and engine is llm. The default model is openai/gpt-oss-120b, and qwen/qwen3.8-27b is tried once if that model returns a 404 or 5xx. The clusters still come from deterministic overlap. On 2026-10-20 the live /v1/health endpoint reported llm_configured: true, so real responses may say engine: "llm" and word their reasons differently from the sample above.

Limits

FAQ

Is high agreement between agents a bad sign?

Not by itself. It's a bad sign when the agreeing answers are near copies of each other, or when the agreed answer is too short to carry independent evidence. Screwtape's consensus endpoint returns false_consensus_risk high in exactly those two cases. Its documented shared-wording example shows an agreement_score of 1 with a risk of low.

How many agent answers can I send to the consensus endpoint?

Between 2 and 16 answers per call. Each answer needs an output string and can carry an optional agent_id.

What should I do when false_consensus_risk is medium or high?

Don't accept consensus_answer yet. Read the clusters and reasons, then verify the answer against evidence with POST /v1/judge, or rerun the agents with separate contexts.

Does this work with CrewAI, LangGraph and AutoGen?

Yes. There is no SDK, so any HTTP client works. Screwtape also runs an MCP server at https://screwtape.ai/mcp with tools named judge and consensus.

Is the content I send stored?

Not by default. Judged content is stored only if you set retain to true and authenticate with an API key, and then only for 24 hours.

What does it cost?

A free key from https://screwtape.ai/signup/ gives 100 calls per UTC day with no card. Without a key, a single call can be paid with x402 for $0.005 USDC on Base. Prepaid packs are listed, but card checkout is in Stripe test mode in this version.

Sources