Answers
What's the difference between agreement_score and false_consensus_risk?
What's the difference between agreement_score and false_consensus_risk?
agreement_score measures how much the answers agree: the share of answers in the largest cluster. false_consensus_risk (low, medium or high) says whether that agreement looks trustworthy, so you can have agreement_score: 1 and still get high risk when the answers are near-verbatim copies.
Example
curl -s https://screwtape.ai/v1/consensus \
-H "Authorization: Bearer $SCREWTAPE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"task":"State the crew size.","answers":[
{"agent_id":"alpha","output":"The crew size was 4."},
{"agent_id":"beta","output":"The crew size was 4."},
{"agent_id":"gamma","output":"The crew size was 4."}]}'Published deterministic response: "agreement_score": 1, "false_consensus_risk": "high", reasons "Answers are nearly verbatim. High agreement may be copied rather than independent." and "Agreement is token overlap, not a model judgment.", and one cluster of size 3.
In the deterministic mode, as described on https://screwtape.ai/blog/false-consensus/:
- Risk is high when agreement is at least 0.8 and the answers are near-verbatim, or the representative answer is very short.
- Risk is medium when agreement is at least 0.6 and an outside answer barely overlaps the majority.
- Otherwise risk is low.
clusters lists each group's size, agent_ids and representative.
FAQ
Is high agreement good?
Not by itself. High agreement with high risk suggests copying; verify the consensus answer against evidence.
Can risk be low with full agreement?
Yes. Answers that share wording but aren't near-verbatim can get agreement 1 with low risk.
What is consensus_answer?
The representative answer of the largest cluster.
Related
Screwtape.ai is an agent validation API, not the arXiv paper also named Screwtape. Facts checked against the live llms.txt, openapi.json, agent card and MCP server card on 2026-10-21.