Benchmark

False-Consensus Benchmark

An independent judge for your agents. Screwtape isn't tied to any agent framework or AI model provider, so its verdicts are never grading their own work. Every verdict comes with a signed receipt you can verify yourself, which gives you a record you can trust when it matters.

The False-Consensus Benchmark (FCB) is an open dataset for one question: when several agents agree, is that agreement evidence the answer is right?

What false consensus is

False consensus is agreement that is not good evidence. It happens in two ways:

The agents agree and are wrong. A shared misconception, a wrong premise in the prompt, or the same arithmetic slip gives every agent the same wrong answer.
The agents agree because they are not independent. They share a model, a context, or each other's output, so five answers carry the evidence of one.

Majority voting assumes errors are independent. When they are correlated, voting makes a wrong answer look confirmed.

What is in the benchmark

144 hand-written cases in eight categories. Each case has a question, 2 to 16 agent answers, the correct answer, and the risk an ideal detector should report.

Category	Cases	Example
Majority wrong on a verifiable fact	16	Three agents name Sydney as the capital of Australia
Echo or paraphrase cascade	16	Later agents copy the first answer, sometimes word for word
Anchoring on an injected wrong premise	16	A planted note says a kilobyte is 1024 bytes in SI units
Lone correct dissenter	16	Three agents say Ganymede is Saturn's largest moon; one says Titan
Genuine independent agreement (control)	28	Different model families reach the same right answer in their own words
Genuine disagreement (control)	20	No answer reaches 60 percent
Correct but unsupported by evidence	16	The answer is right, but the supplied evidence never says it
Numeric or calculation consensus error	16	Most agents say the ball costs 10 cents

A third of the cases are controls. A detector that flags everything gets no credit: false alarms on genuine agreement are measured and reported next to the catch rate.

Every fact is stable and checkable, and every calculation was recomputed. 95 cases are original. 27 questions come from TruthfulQA (Apache-2.0) and 22 from GSM8K (MIT), each cited to its row. All agent answers were written for the benchmark.

How it is scored

Each case is sent once to POST /v1/consensus as a task and its answers. The scorer compares the returned false_consensus_risk (low, medium or high) and consensus_answer with the labels.

Metric	What it measures
Risk-classification accuracy and macro-F1	How often the risk level matches the label, across all three levels
Catch rate on wrong-majority cases	Share of the 74 wrong-majority cases flagged medium or high
False-alarm rate on genuine agreement	Share of the 28 genuine-agreement controls flagged medium or high
Consensus-answer correctness	Share of the 124 cases with a majority where `consensus_answer` is right

Results will be reported per category and per engine (deterministic or model-based), with every raw response and its receipt published so anyone can re-score or verify them.

Results

Pending: not yet run.

No system has been scored on this benchmark yet, including Screwtape. When results are published, this section will give the dataset version, the run date, the engine, every metric with its sample size, and a link to the raw predictions.

Limitations

144 cases is small. Differences of a few cases per category are noise.
The agent answers are written by hand. Real transcripts are longer and messier.
Labels describe what an ideal detector should say. Some wrong majorities look exactly like independent agreement in the text, and only a detector that checks content can catch them.
English, short factual and arithmetic questions only.
The TruthfulQA and GSM8K questions are public and may be in model training data.

Get the data

Dataset: cases.jsonl, CC BY 4.0
Schema: schema.json, JSON Schema draft 2020-12
Code: validate.py and score.py, MIT
Methodology: README

Run it yourself with a free key (100 calls per UTC day):

pip install jsonschema
python3 validate.py
export SCREWTAPE_API_KEY=...
python3 score.py --out runs/my-run

Cite

Screwtape.ai (2026). Screwtape False-Consensus Benchmark (FCB), version 0.1. https://screwtape.ai/benchmark/

Please also cite TruthfulQA (Lin, Hilton and Evans, 2022) and GSM8K (Cobbe et al., 2021) if you use the adapted items.

Files

README.md · schema.json · cases.jsonl · validate.py · score.py · tools/build_cases.py · LICENSE-CODE · LICENSE-DATA · NOTICE