Screwtape.ai
Menu

Guide

Retry vs fail: what an agent pipeline should do with a judge's verdict

20 October 2026

Retry vs fail: what an agent pipeline should do with a judge's verdict

Short answer: Accept on pass. On retry, send the judge's reasons back to the agent and try again, but only up to a fixed cap of 2 or 3 attempts. On fail, stop: log the request_id and reasons and route to a fallback or a human. Screwtape.ai's POST /v1/judge returns one of those three verdicts with a 0 to 1 score, so the policy is a three-way branch in your own code.

Screwtape.ai is an agent validation API. It is not the arXiv paper that is also called Screwtape.

Why three verdicts instead of a boolean

A boolean forces you to choose between always retrying, which wastes calls, and never retrying, which throws away fixable work. A third verdict lets the judge say "close, try again" separately from "wrong". Screwtape doesn't publish the score thresholds that separate retry from fail, so treat the verdict as the decision and the reasons as the explanation. Don't build your own thresholds on score unless you've calibrated them on your own data.

A reusable policy function

"""A small verdict-to-action policy you can reuse in any orchestrator."""
import json
import os
import urllib.request

API = "https://screwtape.ai"


def screwtape_post(path: str, payload: dict) -> dict:
    """POST JSON to Screwtape.ai. Reads SCREWTAPE_API_KEY from the environment."""
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + os.environ["SCREWTAPE_API_KEY"],
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        return json.load(response)


def run_with_policy(task, produce, evidence=None, criteria=None, max_attempts=3):
    """produce(task, feedback) -> str. Returns (output or None, last verdict, attempts)."""
    feedback: list[str] = []
    verdict: dict = {}
    for attempt in range(1, max_attempts + 1):
        output = produce(task, feedback)
        body = {"task": task, "agent_output": output}
        if evidence:
            body["evidence"] = evidence
        if criteria:
            body["criteria"] = criteria
        verdict = screwtape_post("/v1/judge", body)
        if verdict["verdict"] == "pass":
            return output, verdict, attempt
        if verdict["verdict"] == "fail":
            break                     # do not spend more calls on a hard failure
        feedback = verdict["reasons"]  # retry: feed the reasons back to the agent
    return None, verdict, attempt


if __name__ == "__main__":
    out, v, n = run_with_policy(
        "State the crew size.",
        lambda task, fb: "The crew size was 4.",
        evidence=["The crew size was 4."],
    )
    print(out, v.get("verdict"), n)

Tested on 2026-10-20: the function sent a schema-valid body to the live API, which returned 401 for the placeholder key. Plug in any produce(task, feedback) callable, such as a CrewAI kickoff, a LangGraph invoke or an AutoGen run.

Choosing the cap

Framework mapping

FrameworkHow to apply the policy
CrewAIA guardrail returning (False, reasons) makes CrewAI retry up to guardrail_max_retries. For a hard stop on fail, judge outside the crew. See validate CrewAI output.
LangGraphA router returns accept, worker or reject from add_conditional_edges. See validation node.
AutoGenGate in code after agent.run. See AutoGen checks.

FAQ

When should an agent retry instead of failing?

Retry when the judge says the output is fixable, which Screwtape signals with the verdict retry, and you still have attempts left. Give the agent the judge's reasons so the next attempt changes something. Stop on fail.

How many retries should an LLM agent get?

Two or three is a common cap. Base it on cost and on how often retries actually turn into a pass in your logs.

What do pass, fail and retry mean in Screwtape?

They are the three values of the verdict field returned by POST /v1/judge, alongside a score from 0 to 1, a list of reasons, the engine that produced the verdict (llm or deterministic) and a request_id.

Should I log judge results?

Log the request_id, verdict, score and reasons in your own system. Screwtape stores judged content only when you set retain to true with an API key, and then for 24 hours.

Sources