Guide

What is an LLM-as-judge API? How to call one for agent output

25 October 2026

llm-as-judge · ai-agents · evaluation · api · python

This article was created with the help of AI by the Screwtape.ai team. Product details were checked against screwtape.ai/pricing, /docs and /openapi.json on 2026-10-25.

An LLM-as-judge API is a hosted endpoint that takes a task and another model's output, then returns a structured judgment: a verdict, a score, and the reasons behind it. You use one when an agent's output has to be checked before anything downstream trusts it, and you don't want to write and maintain the judging prompt yourself. Screwtape.ai's version is POST /v1/judge. It returns pass, fail or retry, a score from 0 to 1, and a list of reasons.

Screwtape.ai decides whether an AI agent is allowed to spend money, and keeps the proof. The judge in this post is the same kind of check, pointed at any agent output instead of a payment.

What does "LLM-as-judge" mean?

LLM-as-judge is the pattern of using one language model to evaluate the output of another. The judge gets the original task, the candidate answer and, ideally, the evidence the answer should be based on. It returns a decision your code can branch on.

An LLM-as-judge API moves that judge behind an HTTP endpoint. Your agent framework sends JSON and gets JSON back. You don't have to manage the judge prompt, the parsing or the model choice.

Two definitions used in the rest of this post:

Verdict: a categorical decision about one output. In Screwtape it's one of pass, fail or retry.
Grounding: whether the claims in an output are supported by the evidence you supplied, as opposed to the model's own memory.

When should you use a judge API instead of a regex or a schema check?

Use a schema validator when the question is structural, for example "Is this valid JSON with a price field?" Use a judge when the question is about content:

Did the worker answer the task that was actually asked?
Is each claim backed by the evidence you retrieved?
Does the output cover the criteria you care about, such as a named entity, a date or a required section?

Those questions come up at hand-off points in multi-agent systems. A researcher passes notes to a writer, a planner passes steps to an executor, or a sub-agent returns a result to an orchestrator. Each hand-off is a place where one unchecked mistake gets copied forward. If several agents then agree on it, you get the false consensus described in Stop false consensus in multi-agent swarms.

A judge is not a replacement for tests or schema checks. It's an extra gate for the things those checks can't see.

What does Screwtape's `/v1/judge` endpoint accept and return?

Here's the contract as published in openapi.json:

Field	Required	Type	Meaning
`task`	yes	string	What the agent was asked to do
`agent_output`	yes	string	The output being judged
`evidence`	no	string[]	Source passages the output should be grounded in
`criteria`	no	string[]	Things the output must cover
`retain`	no	boolean	Store this call for 24 hours (API-key callers only)

The request schema sets additionalProperties: false, so any extra fields are rejected.

The 200 response always contains:

verdict: pass, fail or retry
score: a number from 0 to 1
reasons: an array of strings explaining the verdict
request_id: an id for support and logging
engine: llm or deterministic
retention: { "retained": false } unless you opted in

The other documented status codes are 400 (invalid request), 401 (API key not recognized), 402 (payment required, when you send no key) and 429 (daily quota or rate limit exceeded).

Which model does the judging?

When a model key is configured, the default model is openai/gpt-oss-120b. If Groq fails, Workers AI is tried next. According to the privacy page, the default OpenAI-compatible base URL is Groq.

With no model key, the service uses a deterministic engine, which does string checks of claims against evidence and of criteria against the output. No model is called in that mode. The engine field in every response tells you which one ran. On 2026-10-20, GET /v1/health reported "llm_configured": true for the live deployment.

How do I call the judge API with curl?

This is the grounded example from the OpenAPI spec:

curl -s https://screwtape.ai/v1/judge \
  -H "Authorization: Bearer $SCREWTAPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task": "State the crew size and launch site.",
    "agent_output": "The crew size was 4. The mission launched from Cape Canaveral.",
    "evidence": ["The mission launched on 2024-03-01 from Cape Canaveral.", "The crew size was 4."],
    "criteria": ["crew size", "Cape Canaveral"]
  }'

If you leave out the Authorization header, you don't get a 401. You get a 402 Payment Required, and the body includes a signup_url where you can get a free key. The key is checked before the body, so a request with no key gets the 402 even if the body is invalid. With a valid key, a body that's missing agent_output returns a 400. A key the API doesn't recognize returns a 401.

How do I call it from Python?

This client uses requests. It treats 402 separately so a missing key is obvious rather than looking like a generic HTTP error:

# judge_requests.py  (pip install requests)
import os
import requests

API = "https://screwtape.ai/v1/judge"


def judge(task, agent_output, evidence=None, criteria=None, timeout=30):
    """POST /v1/judge. Returns the parsed JSON body; raises on non-200."""
    body = {"task": task, "agent_output": agent_output}
    if evidence:
        body["evidence"] = evidence
    if criteria:
        body["criteria"] = criteria
    headers = {"Content-Type": "application/json"}
    key = os.environ.get("SCREWTAPE_API_KEY")
    if key:
        headers["Authorization"] = f"Bearer {key}"
    r = requests.post(API, json=body, headers=headers, timeout=timeout)
    if r.status_code == 402:
        # No key: the body points to the free signup page.
        raise RuntimeError(f"402: no API key. Get a free one at {r.json().get('signup_url')}")
    r.raise_for_status()  # 400 invalid body, 401 unknown key, 429 quota
    return r.json()


if __name__ == "__main__":
    result = judge(
        task="State the crew size and launch site.",
        agent_output="The crew size was 4. The mission launched from Cape Canaveral.",
        evidence=[
            "The mission launched on 2024-03-01 from Cape Canaveral.",
            "The crew size was 4.",
        ],
        criteria=["crew size", "Cape Canaveral"],
    )
    print(result["verdict"], result["score"], result["engine"])
    for reason in result["reasons"]:
        print("-", reason)

We ran this script with no key set. It raised the 402 error with the signup link, which is the documented behavior. With a key, it prints the verdict, score, engine and reasons.

We use requests rather than urllib for a reason. llms.txt notes that the API itself doesn't filter by User-Agent, but Cloudflare's Bot Fight Mode can return 403 to Python's default Python-urllib agent before the request reaches the API. If you must use urllib, set an explicit User-Agent.

How should my code act on the verdict?

The API returns a verdict. What you do with it is your own policy. A common mapping:

pass: hand the output to the next step.
retry: run the worker again and include the reasons in its prompt.
fail: stop, and escalate to a human or a different agent.

How you treat score is up to you. The spec defines its range (0 to 1) but not a recommended threshold, so pick one from your own data. Agent guardrails with retry builds a bounded loop around this mapping, and Validating CrewAI and LangGraph worker output wires it into those two frameworks.

What gets stored when I send output to a judge API?

By default, nothing. According to the privacy page, judged content stays in memory for the life of the request and isn't written to the database or to logs. Each request logs four fields: request id, endpoint, status and latency.

If you send "retain": true with an API key, that request and its result are stored for 24 hours. You can list them with GET /v1/retention and delete one with DELETE /v1/retention/{id}. Calls without an API key can't turn retention on. If a model key is configured, the content you send is forwarded to the model provider for that request.

FAQ

Is LLM-as-judge the same as evaluation? It's one way to do evaluation. Offline evals score a dataset after the fact. A judge API is usually called inline, on each output, so the result can change what your agent does next.

Do I need evidence to use /v1/judge? No. Only task and agent_output are required. Without evidence, though, there's nothing to check grounding against, so include it whenever your agent used retrieved sources.

What does it cost? Free keys include 100 calls per UTC day, with up to three active keys per email and no card needed. Prepaid packs: Trial, $5 for 1,000 calls (one per customer); Starter, $40 for 10,000 calls; and Builder, $175 for 50,000 calls. Team is $199/mo and Finance Control is $399/mo. Card payment through PayPal is live. See screwtape.ai/pricing.

Can I use it from an MCP client? Yes. The same judge is exposed as an MCP tool at https://screwtape.ai/mcp. See MCP tools for fact-checking agents.

Is the deterministic engine an LLM? No. It does string-matching checks and says so in its reasons. Check the engine field if you need to know which one ran.

Try it

Get a free API key (100 calls per UTC day, no card needed): screwtape.ai/signup. Then run the script above with SCREWTAPE_API_KEY set. The full contract is in the API reference.