Answers
How do I use a pass/retry/fail verdict in an agent loop?
How do I use a pass/retry/fail verdict in an agent loop?
Branch three ways: on pass return the output, on retry send the judge's reasons back to the agent and try again up to a fixed cap, and on fail stop. Screwtape.ai's POST /v1/judge returns exactly these three values in verdict, with a 0 to 1 score and the reasons.
Example (Python, stdlib)
import json, os, urllib.request
def judge(task, output, evidence=None):
body = {"task": task, "agent_output": output, **({"evidence": evidence} if evidence else {})}
req = urllib.request.Request("https://screwtape.ai/v1/judge", data=json.dumps(body).encode(), method="POST",
headers={"Authorization": "Bearer " + os.environ["SCREWTAPE_API_KEY"], "Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=60) as r:
return json.load(r)
def run(task, produce, evidence, max_attempts=3):
feedback = None
for _ in range(max_attempts):
output = produce(task, feedback)
v = judge(task, output, evidence)
if v["verdict"] == "pass":
return output
if v["verdict"] == "fail":
return None # stop: wrong, not fixable
feedback = v["reasons"] # retry: feed the reasons back
return NoneEach judge call counts as one call (100 per UTC day on a free key). The cap bounds both cost and latency. Log request_id with each verdict.
FAQ
Why three verdicts instead of a boolean?
retry separates fixable output from wrong output, so you only spend another generation when the judge thinks it can be fixed.
What thresholds separate retry from fail?
Screwtape doesn't publish them. Treat verdict as the decision and reasons as the explanation.
How many retries should I allow?
A small fixed cap such as 2 or 3. Tune it from how often retries turn into pass in your logs.
Related
Screwtape.ai is an agent validation API, not the arXiv paper also named Screwtape. Facts checked against the live llms.txt, openapi.json, agent card and MCP server card on 2026-10-21.