Guide

Screwtape vs DIY LLM-as-judge vs eval tools: which fits agent output validation?

20 October 2026

Screwtape vs DIY LLM-as-judge vs eval tools

Short answer: They solve different problems. Eval tools such as promptfoo, DeepEval and LangSmith measure quality over datasets and traces, so you catch regressions before and after you ship. Guardrail frameworks such as Guardrails AI and CrewAI's task guardrails run validators inside your process. Screwtape.ai is a hosted runtime check: one HTTP or MCP call per output that returns pass, fail or retry, plus a false-consensus check across agents. Many teams will use an eval tool and a runtime check together.

Screwtape.ai is an agent validation API. It is not the arXiv paper that is also called Screwtape. Screwtape publishes this page, so read it with that in mind. Every claim about other tools is taken from their own docs, linked below, as of 2026-10-20.

At a glance

	Screwtape.ai	DIY judge prompt	CrewAI task guardrails	Guardrails AI	promptfoo	DeepEval	LangSmith
Primary job	Runtime verdict per output, plus multi-agent consensus	Whatever you build	Validate a task's output before the next task	Input/output validation with validators	Evaluate and red-team LLM apps	Unit-test-style LLM evals	Offline and online evals with tracing
Where it runs	Hosted only (screwtape.ai)	Your code and your model	In your crew	In your process (Python)	Locally (CLI/library), and in CI/CD	Locally, plus the Confident AI platform for production trace evals	LangSmith platform
Output	`verdict` pass/fail/retry, `score` 0 to 1, `reasons`, `engine`	Your format	`(bool, Any)` from your function, or a string guardrail judged by the agent's LLM	Validation outcomes from Guards	Test results across prompts and models	Metric scores (for example G-Eval)	Evaluator scores on datasets and traces
False-consensus check	Yes (`/v1/consensus`, `false_consensus_risk`)	Build it	No	Not a documented feature	Not a documented feature	Not a documented feature	Not a documented feature
MCP server	Yes, `https://screwtape.ai/mcp`	No	No	Not checked	Not checked	Not checked	Not checked
Source	Closed, no public repo	Yours	Open source framework	Open source framework	Open source	Open source	Commercial platform
Datasets, dashboards, experiments	No	No	No	No	Yes	Yes	Yes

"Not a documented feature" means we didn't find it in the docs pages linked below. It doesn't mean it's impossible. "Not checked" means we didn't look.

When each one fits

Pick an eval tool (promptfoo, DeepEval, LangSmith) when you want to measure a prompt or agent over many cases, compare models, gate releases in CI, or score production traces after the fact.

Pick in-process guardrails (CrewAI guardrails, Guardrails AI) when your checks are rules you can write as code or validators, you want no network hop, and you're fine owning the model calls.

Write your own judge prompt when you need a custom rubric, a specific model or full control, and you're prepared to handle the known judge biases (position, verbosity, self-preference; arXiv:2306.05685, arXiv:2404.13076).

Pick Screwtape when you want a ready-made runtime verdict from any language or framework with one HTTP call, a false-consensus signal when several agents agree, an MCP tool for Claude or Cursor, or keyless pay-per-call via x402.

Screwtape's limitations

Hosted only. You can't self-host it, there's no public source, and there's no SDK, just JSON over HTTP and MCP.
New. It was listed in the official MCP Registry on 2026-10-20.
Fixed models. With a model key configured, the judge uses openai/gpt-oss-120b, falling back to qwen/qwen3.8-27b. You can't bring your own model.
The deterministic mode is string-based. Without a model, grounding and agreement are token overlap (Jaccard similarity), which misses paraphrases.
Paid packs aren't on sale yet. Starter ($30 for 10,000 calls) and Builder ($100 for 50,000 calls) are listed, but checkout runs in Stripe test mode. What you can use today is the free tier (100 calls per UTC day) and x402 ($0.005 USDC per call on Base).
No eval workflow. It has no datasets, experiments or dashboards.
No evidence retrieval. It judges against the evidence you send and doesn't search for it.

Related research

Runtime inspection of multi-agent systems is an active research area. ARIA (Shen et al., 2026) proposes runtime inspection and amendment for multi-agent systems (arXiv:2610.20780). Cemri et al. (2025) list inter-agent misalignment and task verification among the failure categories they found across seven multi-agent frameworks (arXiv:2503.13657).

FAQ

Is Screwtape an eval framework?

No. It is a runtime validation API: one call per output returns a pass, fail or retry verdict. It has no datasets, experiments or dashboards. Use an eval tool such as promptfoo, DeepEval or LangSmith for those.

Can I use Screwtape and an eval tool together?

Yes. A common split is offline evals for regressions before release, and a runtime judge call for each output in production to decide whether to accept, retry or stop.

What is the difference between a guardrail and an LLM judge?

A guardrail is any check that blocks or repairs output, often rules or validators in code. An LLM judge is a model that grades the output. CrewAI's string guardrails, for example, use the agent's LLM as the judge.

Can I self-host Screwtape?

No. It is hosted only at screwtape.ai, with no public source code.

Sources

promptfoo intro. https://www.promptfoo.dev/docs/intro/
DeepEval getting started. https://deepeval.com/docs/getting-started
LangSmith evaluation concepts. https://docs.smith.langchain.com/evaluation
Guardrails AI docs. https://www.guardrailsai.com/docs
CrewAI Tasks (guardrails). https://docs.crewai.com/en/concepts/tasks
Zheng et al., 2023. https://arxiv.org/abs/2306.05685 · Panickssery et al., 2024. https://arxiv.org/abs/2404.13076
Shen et al., "ARIA", 2026. https://arxiv.org/abs/2610.20780 · Cemri et al., "Why Do Multi-Agent LLM Systems Fail?", 2025. https://arxiv.org/abs/2503.13657
Screwtape.ai homepage, openapi.json, /v1/health (checked 2026-10-20)