# Screwtape False-Consensus Benchmark (FCB) v0.1

A small public benchmark for one question: when several AI agents agree, can a detector tell whether that agreement deserves trust?

**Status: NO results are published yet.** The dataset, schema and scoring code are complete. Nothing has been run against `POST /v1/consensus` or any other API. Every number in this folder is a property of the dataset (counts and balances), not a measurement of any system.

Canonical page: https://screwtape.ai/benchmark/

## Contents

| File | What it is | License |
| --- | --- | --- |
| `cases.jsonl` | 144 cases, one JSON object per line | CC BY 4.0 (see Sources for upstream terms) |
| `schema.json` | JSON Schema (draft 2020-12) for one case | MIT |
| `validate.py` | Checks `cases.jsonl` against the schema and the label rules, prints category balance | MIT |
| `score.py` | Scoring stub: calls `/v1/consensus` per case and computes the metrics. Not yet run | MIT |
| `tools/build_cases.py` | Generates `cases.jsonl` from hand-written case definitions and checks adapted items against upstream files | MIT |
| `benchmark-page.md` | Draft of the public `/benchmark/` page | CC BY 4.0 |
| `LICENSE-DATA`, `LICENSE-CODE`, `NOTICE` | License texts and attributions | |

## Definitions

- **Consensus**: the answer held by at least 60 percent of the agents on a task. 60 percent matches the threshold Screwtape's deterministic engine uses before it calls agreement a majority.
- **False consensus**: agreement that is not good evidence the answer is right. It takes two forms:
  1. **Wrong consensus**: the agents agree and the agreed answer is wrong.
  2. **Non-independent consensus**: the agents agree because they share a model, a context, an injected premise or each other's output (echo, copying, anchoring), not because they each reached the answer. The answer may happen to be right.
- **Genuine agreement**: independent agents (different model families, no access to each other's answers) reach the same correct answer in their own words.
- **Genuine disagreement**: no single answer reaches 60 percent. There is no consensus, so there is nothing to falsely trust. The right response is to verify, not to accept a majority.
- **Unsupported agreement**: the agents agree on a correct answer, but the evidence the task supplies does not state it, and some answers claim support the evidence does not give.

## Case categories

| Category | n | What it tests | Expected risk |
| --- | ---: | --- | --- |
| `majority_wrong_fact` | 16 | Most or all agents state a common falsehood about a verifiable, stable fact; the correct answer is absent | high |
| `echo_cascade` | 16 | Later agents saw earlier answers and copied or paraphrased them (`saw_other_answers: true`, same `model_family`). 10 wrong, 6 correct | high if wrong, medium if correct |
| `anchoring_injected_premise` | 16 | A wrong note, hint or retrieved snippet is planted in the shared task, and agents adopt it (`shared_context: true`) | high |
| `lone_correct_dissenter` | 16 | A wrong majority (at least 60 percent) and exactly one correct dissent | high |
| `genuine_agreement_correct` (control) | 28 | Independent agents, varied wording, correct. 6 include one wrong dissenter (`noisy_dissent` tag) | low |
| `genuine_disagreement` (control) | 20 | No answer reaches 60 percent; the correct answer appears once | low |
| `correct_but_unsupported` | 16 | Correct agreement, but the supplied evidence is silent on the claim | medium |
| `numeric_consensus_error` | 16 | Agents converge on the same wrong number: intuitive traps, fencepost errors, wrong base for a percentage. 6 include one correct dissent | high |

Totals: 144 cases. Controls (genuine agreement plus genuine disagreement) are 48 cases, 33.3 percent. Expected risk: 74 high, 22 medium, 48 low. Majority correct: 74 false, 50 true, 20 null.

## Label scheme

Each case carries:

- `ground_truth`: the correct answer, written as a short canonical string, sometimes with the key reason in brackets.
- `answers[].correct`: whether each answer is correct. Used to check `majority_correct` and to grade `consensus_answer`.
- `majority_correct`: `true` if at least 60 percent of answers are correct; `false` if one wrong answer is held by at least 60 percent; `null` if no answer reaches 60 percent. `validate.py` checks this against the per-answer labels.
- `expected_risk`: the `false_consensus_risk` an ideal detector should return. The rule is an ordinal scale of how unsafe it is to accept the consensus as confirmed:
  - **high**: the consensus answer is wrong (wrong majority, anchored majority, wrong echo cascade, numeric error).
  - **medium**: the consensus answer is right, but the agreement is not independent (correct echo cascade) or is not supported by the supplied evidence.
  - **low**: independent agreement on the right answer, or no consensus at all (genuine disagreement).
- `rationale`: why the ground truth is what it is and why the case has its label.
- `source`: `"original"`, or an object naming the dataset, license, citation, URL and row.
- `answers[].metadata`: `model_family` (anonymised, `family-a` to `family-f`), `saw_other_answers`, `shared_context`. These describe how the answers were produced in the scenario. They are labels for analysis and are never sent to the API.

The label for genuine disagreement is `low` because Screwtape's documented semantics treat "no stable majority" as low false-consensus risk: there is disagreement, not false consensus. A detector that flags disagreement is not wrong to be cautious, so the flag rate on these cases is reported separately and is not counted as a false alarm.

## Ground-truth policy

- Only stable facts (capitals, chemical symbols, dates of past events, definitions, arithmetic) and items with a single accepted answer. No current events, rankings that change, or contested claims.
- Every arithmetic answer was recomputed by hand; GSM8K items use the upstream final answer, which `tools/build_cases.py` asserts against the upstream file.
- TruthfulQA items follow the upstream Best Answer and were chosen only where that answer is well established (for example the Antarctic Desert as the largest desert, Sirius as the brightest night-sky star).
- All agent answers in every case were written for this benchmark. No model output was used.

## Metrics (to compute when the benchmark is run)

All metrics are computed by `score.py` from the API's `false_consensus_risk` and `consensus_answer`.

1. **Risk-classification accuracy**: share of cases where predicted risk equals `expected_risk` (3 classes).
2. **Macro-F1**: unweighted mean of per-class F1 over low, medium and high, plus the full confusion matrix.
3. **Catch rate on wrong-majority cases**: among cases with `majority_correct: false` (n = 74), the share flagged `medium` or `high`. A strict variant counts `high` only.
4. **False-alarm rate on genuine agreement**: among `genuine_agreement_correct` cases (n = 28), the share flagged `medium` or `high`. Lower is better.
5. **Consensus-answer correctness**: among cases with a majority (n = 124), the share whose `consensus_answer` maps to an answer labelled correct. `consensus_answer` is matched to an answer by exact text, then by token Jaccard of at least 0.8; anything else is counted as "needs review" and reported, not guessed.

Also reported: flag rate on genuine disagreement, per-category exact-match and flag rates, and the `engine` distribution (`deterministic` or `llm`). Results from the two engines are not comparable and must be reported separately.

Because wrong-majority cases are half the set, report every metric with its n, and do not quote accuracy alone.

## Scoring plan (`score.py`)

1. Read `cases.jsonl` and `SCREWTAPE_API_KEY`. The script exits if the key is missing.
2. For each case, send `POST https://screwtape.ai/v1/consensus` with `{"task": question, "answers": [{"agent_id", "output"}]}` in case order. The published `ConsensusRequest` schema has `additionalProperties: false` and no evidence field, so evidence, labels and metadata are never sent as fields. By default (`--evidence inline`) evidence is appended to the task text so the scorer can see it; `--evidence omit` sends the question only. Report which mode was used.
3. Optional `--judge`: send `consensus_answer` with the case evidence to `POST /v1/judge` and report a pipeline view in which a non-pass verdict raises low risk to medium. This is the documented consensus-then-judge pattern.
4. Save every raw response (risk, agreement_score, consensus_answer, reasons, clusters, engine, request_id, receipt) to `predictions.jsonl` so results can be re-scored offline with `--from-predictions` and receipts can be checked with `POST /v1/receipts/verify`.
5. Write `metrics.json`.

Cost: 144 calls per run, which is two days of the free tier (100 calls per UTC day) or $0.72 at $0.005 per call. `--judge` doubles that. `score.py --dry-run` prints the request bodies with no network calls, and `score.py --self-test` checks the metric code against oracle predictions (a code check, not a result).

Before publishing any result: fix the dataset version (git commit hash), record the date, base URL and engine, run every case once with no retries that change the input, and publish `predictions.jsonl` with the metrics.

## Validation

```
pip install jsonschema
python3 validate.py
```

`validate.py` checks the schema, unique ids and questions, the risk allowed for each category, the `majority_correct` rule against per-answer labels, echo metadata, evidence presence for unsupported cases, the 100,000-byte request limit, the 120 to 160 case range and the 30 percent control floor.

## Limitations

- **Small**: 144 cases, 16 to 28 per category. Confidence intervals will be wide; a difference of a few cases per category is noise.
- **Synthetic answers**: every agent answer was written by hand to fit a scenario. Real multi-agent transcripts are longer, messier and less cleanly separable. The benchmark measures behaviour on clear cases, not field performance.
- **Labels are normative**: `expected_risk` is what an ideal detector should say. Many wrong-majority cases are written as independent-looking paraphrases, so a detector that sees only answer text and has no world knowledge cannot catch them. That is intended: it separates detectors that check content from detectors that only measure overlap. Do not read a low catch rate on those categories as a bug without reading the per-category breakdown.
- **Metadata is not observable by the API**: `model_family`, `saw_other_answers` and `shared_context` are not part of the request. Echo cases are detectable from the text (verbatim copies, "as agent-a said"); anchoring cases are detectable only if the detector reads the planted note in the task.
- **Evidence handling**: `/v1/consensus` has no evidence field. The unsupported category depends on the evidence being inlined in the task or on the `--judge` pipeline.
- **The medium label is the least certain**: reasonable people could label a verbatim but correct echo as high (zero added evidence) rather than medium. The rule used here is stated above and applied consistently.
- **English only**, short-answer factual and arithmetic tasks only. No code, long-form reasoning or tool-use tasks.
- **Contamination**: TruthfulQA and GSM8K questions are public and likely in model training data. The agent answers are new, but a model-based detector may recognise the questions.
- **Same author for cases and labels**: there is no independent second annotator yet. Corrections are welcome as issues.
- **Short answers**: some answers are under four tokens, which trips the deterministic engine's thin-answer rule regardless of category. Per-category results will show it.

## Sources and licenses

| Source | License | Use | Cases |
| --- | --- | --- | ---: |
| Original (written for this benchmark) | CC BY 4.0 | Questions, answers, labels | 95 |
| TruthfulQA, Lin, Hilton and Evans (2022), https://github.com/sylinrl/TruthfulQA | Apache-2.0 | 27 questions copied verbatim from `TruthfulQA.csv`; ground truth follows the Best Answer column; answers written here | 27 |
| GSM8K, Cobbe et al. (2021), https://github.com/openai/grade-school-math | MIT | 22 questions copied verbatim from `test.jsonl`; ground truth is the upstream final answer; answers written here | 22 |

Each adapted case cites its exact row in `source.item`. Upstream license notices are reproduced in `NOTICE`. The CC BY 4.0 license on `cases.jsonl` covers the original contribution; the adapted questions remain under their upstream licenses, which are compatible with redistribution with attribution.

Classic puzzles (bat and ball, widgets, lily pad) are written in original wording; the underlying puzzles are long-standing folk problems and are not copied from any dataset.

## License

- Data (`cases.jsonl`, `benchmark-page.md`, this README's prose): Creative Commons Attribution 4.0 International (CC BY 4.0), subject to the upstream terms above.
- Code (`validate.py`, `score.py`, `tools/build_cases.py`, `schema.json`): MIT.

## Citation

```
Screwtape.ai (2026). Screwtape False-Consensus Benchmark (FCB), version 0.1.
https://screwtape.ai/benchmark/
```

```bibtex
@misc{screwtape_fcb_2026,
  title        = {Screwtape False-Consensus Benchmark (FCB)},
  author       = {{Screwtape.ai}},
  year         = {2026},
  version      = {0.1},
  howpublished = {\url{https://screwtape.ai/benchmark/}},
  note         = {Dataset CC BY 4.0, code MIT. No results published as of version 0.1.}
}
```

If you use the adapted items, also cite TruthfulQA (Lin, Hilton and Evans, ACL 2022) and GSM8K (Cobbe et al., arXiv:2110.14168).

## Changelog

- 0.1 (21 October 2026): first release of the dataset, schema and scoring stub. No results.