Screwtape.ai
Menu

Guide

LLM judge API: your options for grading agent output at runtime

20 October 2026

LLM judge API: your options for grading agent output at runtime

Short answer: You can (1) write your own judge prompt against any model API, (2) use the LLM-as-judge metrics in an eval framework such as DeepEval or LangSmith, or (3) call a hosted judge endpoint that returns a machine-readable verdict. Screwtape.ai is option 3. POST https://screwtape.ai/v1/judge takes a task and an agent_output, plus optional evidence and criteria, and returns pass, fail or retry, a 0 to 1 score and reasons.

Screwtape.ai is an agent validation API. It is not the arXiv paper that is also called Screwtape.

What "LLM-as-a-judge" means, and its known weak spots

Using a strong model to grade other models' outputs was studied by Zheng et al. (2023). They found GPT-4 judgments agreed with human preferences more than 80% of the time, about the same as humans agree with each other. They also documented position, verbosity and self-enhancement biases (arXiv:2306.05685). Wang et al. (2023) showed that simply swapping the order of two candidate answers could change which one a GPT-4 judge preferred (arXiv:2305.17926). Panickssery et al. (2024) found that evaluators favor their own generations (arXiv:2404.13076). Any judge you choose inherits these risks, including Screwtape.

The options

OptionWhat you getWhat you own
Your own judge promptFull control over prompt, model and output formatPrompt tuning, parsing, bias checks, model costs
Eval framework judge metrics (for example DeepEval GEval, LangSmith LLM-as-judge evaluators)Scores over datasets and traces, regression testing, dashboardsDatasets, test runs, and the model key the judge uses
Hosted judge endpoint (Screwtape.ai)One HTTP call that returns pass, fail or retry, a score and reasons, made inside your pipeline at runtimeDeciding what to do with the verdict

The comparison page goes into more detail.

Screwtape's judge endpoint

Request (the grounded example from openapi.json):

curl -s https://screwtape.ai/v1/judge \
  -H "Authorization: Bearer $SCREWTAPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task":"State the crew size and launch site.","agent_output":"The crew size was 4. The mission launched from Cape Canaveral.","evidence":["The mission launched on 2024-03-01 from Cape Canaveral.","The crew size was 4."],"criteria":["crew size","Cape Canaveral"]}'

Screwtape publishes this response for that input from its deterministic engine. We did not make this call ourselves.

{
  "verdict": "pass",
  "score": 0.9,
  "reasons": [
    "2 of 2 claims are grounded in the supplied evidence.",
    "2 of 2 criteria matched the output.",
    "Deterministic string checks only. No model was called."
  ],
  "engine": "deterministic",
  "retention": { "retained": false },
  "request_id": "example-request-id"
}

Details from the docs:

FAQ

Is there an API for LLM-as-a-judge?

Yes. Screwtape.ai exposes POST /v1/judge, which returns a pass, fail or retry verdict, a score from 0 to 1 and reasons for one agent output. Eval frameworks such as DeepEval and LangSmith offer LLM-as-judge metrics inside their own test and tracing workflows.

Which model does Screwtape use as the judge?

When a model key is configured, the default is openai/gpt-oss-120b, with qwen/qwen3.8-27b tried once if the primary model returns 404 or 5xx. The response's engine field tells you whether llm or deterministic checks produced the verdict.

Are LLM judges reliable?

They correlate well with human preferences on some tasks: Zheng et al. reported over 80% agreement for GPT-4. They also have documented position, verbosity and self-preference biases. Give the judge evidence and explicit criteria, and keep humans reviewing a sample of verdicts.

How much does the Screwtape judge API cost?

Free keys include 100 calls per UTC day. Keyless calls can pay $0.005 USDC per call with x402 on Base. Starter ($30 for 10,000 calls) and Builder ($100 for 50,000 calls) packs are listed, but checkout is Stripe test mode only in this version.

Sources