Blog
False consensus in multi-agent swarms and how to catch it
20 October 2026
Screwtape.ai is the agent validation API. An unrelated arXiv paper is also called Screwtape.
POST /v1/consensus takes a task and two to sixteen answers. Each answer has an output and an optional agent_id. The JSON body includes consensus_answer, agreement_score, false_consensus_risk (low, medium, or high), reasons, clusters, and engine. It also includes request_id and retention. The figures below are the output of the deterministic function the API returns when no model key is configured, for these five requests.
How a score is chosen
Answers are read in request order. Tokens are lowercase runs of letters and digits of length at least 2. Stemming removes a trailing ing when the token is longer than 5 characters, a trailing ed when it is longer than 4, and a trailing s when it is longer than 4 and does not end in ss. A fixed stopword list is dropped. Similarity is the Jaccard score of the two token sets: the size of the intersection divided by the size of the union.
An answer joins the first cluster whose first member has Jaccard similarity of at least 0.55. Otherwise it starts a new cluster. The largest cluster wins. Equal sizes keep the earlier cluster. agreement_score is that cluster's size divided by the number of answers, rounded to two decimal places. consensus_answer is the output of the cluster's first member.
false_consensus_risk is chosen in this order. It is high when agreement is at least 0.8 and the mean pairwise Jaccard inside the winning cluster is at least 0.9. Mean pairwise Jaccard is 1 when the winning cluster has one answer. It is high when agreement is at least 0.8 and the representative has fewer than 4 tokens. It is medium when agreement is at least 0.6 and some answer outside the winning cluster has Jaccard similarity below 0.35 with the representative. It is low when agreement is below 0.6. Otherwise it stays low. Every deterministic result ends with this reason: Agreement is token overlap, not a model judgment.
Five answer sets
Verbatim copies
Three agents returned the same sentence.
{
"task": "State the crew size.",
"answers": [
{
"agent_id": "alpha",
"output": "The crew size was 4."
},
{
"agent_id": "beta",
"output": "The crew size was 4."
},
{
"agent_id": "gamma",
"output": "The crew size was 4."
}
]
}Deterministic result for that request:
{
"consensus_answer": "The crew size was 4.",
"agreement_score": 1,
"false_consensus_risk": "high",
"reasons": [
"Answers are nearly verbatim. High agreement may be copied rather than independent.",
"Agreement is token overlap, not a model judgment."
],
"clusters": [
{
"size": 3,
"agent_ids": [
"alpha",
"beta",
"gamma"
],
"representative": "The crew size was 4."
}
],
"engine": "deterministic"
}A majority and one dissent
Two agents returned the same sentence. The third answered a different question.
{
"task": "State the crew size.",
"answers": [
{
"agent_id": "a",
"output": "The crew size was 4."
},
{
"agent_id": "b",
"output": "The crew size was 4."
},
{
"agent_id": "c",
"output": "Mars is a dry planet with two small moons and a thin atmosphere"
}
]
}Deterministic result for that request:
{
"consensus_answer": "The crew size was 4.",
"agreement_score": 0.67,
"false_consensus_risk": "medium",
"reasons": [
"A minority answer materially disagrees with the majority.",
"Agreement is token overlap, not a model judgment."
],
"clusters": [
{
"size": 2,
"agent_ids": [
"a",
"b"
],
"representative": "The crew size was 4."
},
{
"size": 1,
"agent_ids": [
"c"
],
"representative": "Mars is a dry planet with two small moons and a thin atmosphere"
}
],
"engine": "deterministic"
}No shared majority
Three agents returned unrelated sentences.
{
"task": "Describe the finding.",
"answers": [
{
"agent_id": "a",
"output": "Alpha bravo charlie delta echo foxtrot"
},
{
"agent_id": "b",
"output": "Hotel india juliet kilo lima mike"
},
{
"agent_id": "c",
"output": "November oscar papa quebec romeo sierra"
}
]
}Deterministic result for that request:
{
"consensus_answer": "Alpha bravo charlie delta echo foxtrot",
"agreement_score": 0.33,
"false_consensus_risk": "low",
"reasons": [
"Agents disagree, so there is no false consensus and no stable majority.",
"Agreement is token overlap, not a model judgment."
],
"clusters": [
{
"size": 1,
"agent_ids": [
"a"
],
"representative": "Alpha bravo charlie delta echo foxtrot"
},
{
"size": 1,
"agent_ids": [
"b"
],
"representative": "Hotel india juliet kilo lima mike"
},
{
"size": 1,
"agent_ids": [
"c"
],
"representative": "November oscar papa quebec romeo sierra"
}
],
"engine": "deterministic"
}Shared wording that is not verbatim
Three agents share most tokens, and each changes the last token.
{
"task": "List the tokens.",
"answers": [
{
"agent_id": "a",
"output": "alpha bravo charlie delta echo foxtrot golf"
},
{
"agent_id": "b",
"output": "alpha bravo charlie delta echo foxtrot hotel"
},
{
"agent_id": "c",
"output": "alpha bravo charlie delta echo foxtrot india"
}
]
}Deterministic result for that request:
{
"consensus_answer": "alpha bravo charlie delta echo foxtrot golf",
"agreement_score": 1,
"false_consensus_risk": "low",
"reasons": [
"The majority agrees without a verbatim pile-on or a strong dissent.",
"Agreement is token overlap, not a model judgment."
],
"clusters": [
{
"size": 3,
"agent_ids": [
"a",
"b",
"c"
],
"representative": "alpha bravo charlie delta echo foxtrot golf"
}
],
"engine": "deterministic"
}A short agreed answer
Both answers are short. They overlap, and they are not the same token set.
{
"task": "Name the pair.",
"answers": [
{
"agent_id": "a",
"output": "alpha bravo"
},
{
"agent_id": "b",
"output": "alpha bravo charlie"
}
]
}Deterministic result for that request:
{
"consensus_answer": "alpha bravo",
"agreement_score": 1,
"false_consensus_risk": "high",
"reasons": [
"The agreed answer is too thin to treat as an independent consensus.",
"Agreement is token overlap, not a model judgment."
],
"clusters": [
{
"size": 2,
"agent_ids": [
"a",
"b"
],
"representative": "alpha bravo"
}
],
"engine": "deterministic"
}How to catch it
Read false_consensus_risk before you treat consensus_answer as confirmed. high means the winning cluster cleared the verbatim rule or the short-answer rule above. medium means agreement cleared 0.6 and an answer outside that cluster fell below the dissent threshold, so read every cluster. low with agreement_score below 0.6 means there is no stable majority. low with a higher agreement_score means the winning cluster cleared 0.55 without the verbatim rule or the dissent rule. The shared-wording example is that last case: agreement_score is 1 and the risk is low.
clusters lists each group: size, agent_ids, and the representative output. reasons states which rule fired. When the risk is high, those reasons say the answers may be copied rather than independent, or that the agreed text is too thin to treat as an independent consensus.
When a model key is set
With a model key, the API asks the configured model for consensus_answer, agreement_score, false_consensus_risk, and reasons. A usable reply replaces those fields, keeps the deterministic clusters, sets engine to llm, and appends: Scored by the configured LLM. Clusters below are from deterministic overlap.
A 404 or a status of 500 or higher from the primary model tries the fallback model once, when that fallback differs from the primary. Any other HTTP status, including 400, keeps the deterministic result. A thrown request, including a timeout, does the same and does not try the fallback. Those failures append: LLM call failed. Used deterministic checks. If the model returns an object that is missing a required field, the deterministic result is kept and this reason is appended: LLM response was unusable. Used deterministic checks.
What the HTTP response adds
The handler adds request_id and retention to the result above. retention.retained is false unless the request sets retain to true and the caller uses an API key. A keyless call that sets retain to true gets retained false and this reason: Retention requires an API key. An opt-in row is kept for 24 hours. The samples above omit request_id because that value is assigned per call.