Guide

Majority voting vs independent verification for LLM agents

20 October 2026

Majority voting vs independent verification for LLM agents

Short answer: Majority voting works when agents make independent mistakes. Verification checks an answer against something outside the agents, such as evidence, criteria or a separate judge. Use voting to pick a candidate. Then verify the candidate whenever the agreement looks copied, thin or split, because a vote can't catch an error that every agent shares.

Screwtape.ai is an agent validation API. It is not the arXiv paper that is also called Screwtape.

What the research says

• Voting over samples helps. Self-consistency samples several reasoning paths and takes the most common answer. Wang et al. (2022) showed it improves chain-of-thought reasoning (arXiv:2203.11171).
• Debate isn't automatically better. Smit et al. (2023) found that multi-agent debate "in their current form, do not reliably outperform" self-consistency and ensembling, and that debate results depend heavily on hyperparameters (arXiv:2311.17371).
• Debate can help factuality. Du et al. (2023) reported fewer hallucinations when several model instances debate over multiple rounds (arXiv:2305.14325).
• Voting assumes independent errors, and that assumption is weakening. Goel et al. (2025) found model mistakes becoming more similar as capability grows (arXiv:2502.04313).

Side by side

The two approaches complement each other. Vote to choose a candidate, and verify when the vote can't be trusted.

Example: vote, then verify only when needed

"""Do not trust a majority on its own: score consensus, then verify the winner against evidence."""
import json
import os
import urllib.request

API = "https://screwtape.ai"


def screwtape_post(path: str, payload: dict) -> dict:
    """POST JSON to Screwtape.ai. Reads SCREWTAPE_API_KEY from the environment."""
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + os.environ["SCREWTAPE_API_KEY"],
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        return json.load(response)


def settle(task: str, answers: list[dict], evidence: list[str]) -> dict:
    """answers: 2 to 16 items shaped like {"agent_id": "a", "output": "..."}."""
    c = screwtape_post("/v1/consensus", {"task": task, "answers": answers})
    # Majority alone is not enough when the agreement looks copied or thin, or when there is no stable majority.
    if c["false_consensus_risk"] == "low" and c["agreement_score"] >= 0.6:
        return {"answer": c["consensus_answer"], "consensus": c, "verified": None}
    v = screwtape_post("/v1/judge", {
        "task": task,
        "agent_output": c["consensus_answer"],
        "evidence": evidence,
    })
    return {"answer": c["consensus_answer"] if v["verdict"] == "pass" else None,
            "consensus": c, "verified": v}


if __name__ == "__main__":
    out = settle(
        "State the crew size.",
        [
            {"agent_id": "alpha", "output": "The crew size was 4."},
            {"agent_id": "beta", "output": "The crew size was 4."},
            {"agent_id": "gamma", "output": "The crew size was 4."},
        ],
        ["The crew size was 4."],
    )
    print(json.dumps(out, indent=2))

The 0.6 threshold mirrors the documented deterministic rules: below 0.6 agreement, Screwtape reports no stable majority. It's a reasonable default, not a value the API requires. Tune it for your workload.

When to skip verification

• When false_consensus_risk is low and the agents really did work from separate context.
• When no evidence exists to check against. In that case, the judge can only assess the answer against criteria and the task.

FAQ

Is majority voting enough for a multi-agent LLM system?

It's enough only when the agents' errors are independent. If they share a prompt, a retrieved document or each other's messages, they can agree on the same mistake. Verify the winner against evidence in those cases.

What is the difference between self-consistency and multi-agent consensus?

Self-consistency samples several reasoning paths from one model and takes the most common answer. Multi-agent consensus compares answers from separate agents, which may use different prompts, tools or models. Both rely on errors being independent.

Does multi-agent debate fix false consensus?

Not reliably. Smit et al. (2023) found that debate did not reliably beat self-consistency or ensembling without tuning. Debate also lets agents read each other's answers, which can produce agreement that isn't independent.

Which Screwtape endpoint is the vote and which is the verification?

POST /v1/consensus scores agreement across 2 to 16 answers and returns false_consensus_risk. POST /v1/judge checks one answer against optional evidence and criteria and returns pass, fail or retry with a 0 to 1 score and reasons.

Sources

• Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models", 2022. https://arxiv.org/abs/2203.11171
• Smit et al., "Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs", 2023. https://arxiv.org/abs/2311.17371
• Du et al., "Improving Factuality and Reasoning in Language Models through Multiagent Debate", 2023. https://arxiv.org/abs/2305.14325
• Goel et al., "Great Models Think Alike and this Undermines AI Oversight", 2025. https://arxiv.org/abs/2502.04313
• Screwtape.ai openapi.json and false-consensus write-up (checked 2026-10-20)