Screwtape.ai
Menu

Guide

How to check AutoGen worker output before another agent uses it

20 October 2026

How to check AutoGen worker output before another agent uses it

Short answer: If the check must always happen, run it in code between agents. Call worker.run(...), take the last message, send it to POST /v1/judge and continue only on pass. If you'd rather let a reviewer agent decide when to check, load Screwtape's judge and consensus tools from its MCP server with mcp_server_tools(StreamableHttpServerParams(url="https://screwtape.ai/mcp", ...)).

Screwtape.ai is an agent validation API. It is not the arXiv paper that is also called Screwtape.

Option 1: a hard gate in code

A tool is optional for the model: it can skip the call. A gate in code can't be skipped.

"""AutoGen: check a worker agent's answer in code before you accept it."""
import asyncio
import json
import os
import urllib.request

from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient

API = "https://screwtape.ai"
TASK = "State the crew size and launch site."
EVIDENCE = [
    "The mission launched on 2024-03-01 from Cape Canaveral.",
    "The crew size was 4.",
]


def screwtape_post(path: str, payload: dict) -> dict:
    """POST JSON to Screwtape.ai. Reads SCREWTAPE_API_KEY from the environment."""
    request = urllib.request.Request(
        API + path,
        data=json.dumps(payload).encode(),
        headers={
            "Authorization": "Bearer " + os.environ["SCREWTAPE_API_KEY"],
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=60) as response:
        return json.load(response)


async def main() -> None:
    worker = AssistantAgent(
        name="worker",
        model_client=OpenAIChatCompletionClient(model="gpt-4o-mini"),  # needs OPENAI_API_KEY
        system_message="Answer only from the facts in the task.",
    )
    result = await worker.run(task="Facts: " + " ".join(EVIDENCE) + " " + TASK)
    answer = result.messages[-1].content

    verdict = screwtape_post("/v1/judge", {
        "task": TASK,
        "agent_output": answer,
        "evidence": EVIDENCE,
    })
    print(verdict["verdict"], verdict["score"], verdict["reasons"])
    if verdict["verdict"] != "pass":
        raise SystemExit("worker output not accepted")


if __name__ == "__main__":
    asyncio.run(main())

Option 2: Screwtape's MCP tools for a reviewer agent

Tested on 2026-10-20 with autogen-agentchat 0.7.5, autogen-ext 0.7.5 and mcp 1.28.1. mcp_server_tools discovered ['judge', 'consensus'] from the live server. Listing tools needs no key, but calling them needs a Bearer key or an x402 payment.

"""AutoGen: load Screwtape's judge and consensus tools from its MCP server."""
import asyncio
import os

from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.tools.mcp import StreamableHttpServerParams, mcp_server_tools


async def main() -> None:
    server = StreamableHttpServerParams(
        url="https://screwtape.ai/mcp",
        headers={"Authorization": "Bearer " + os.environ["SCREWTAPE_API_KEY"]},
    )
    tools = await mcp_server_tools(server)  # judge, consensus
    print([t.name for t in tools])
    reviewer = AssistantAgent(
        name="reviewer",
        model_client=OpenAIChatCompletionClient(model="gpt-4o-mini"),  # needs OPENAI_API_KEY
        tools=tools,
        system_message="Call the judge tool on the worker's answer before you accept it. Report the verdict and reasons.",
    )
    result = await reviewer.run(task=(
        "Task: State the crew size and launch site. "
        "Worker answer: 'The crew size was 5. The mission launched from Vandenberg.' "
        "Evidence: 'The mission launched on 2024-03-01 from Cape Canaveral.', 'The crew size was 4.'"
    ))
    print(result.messages[-1].content)


if __name__ == "__main__":
    asyncio.run(main())

Option 3: a FunctionTool

Screwtape's AutoGen integration page wraps POST /v1/judge in autogen_core.tools.FunctionTool and passes it to AssistantAgent(tools=[judge_tool]).

Which option to choose

NeedUse
Every worker answer must be checkedOption 1 (gate in code)
A reviewer agent should decide what to check, and should also score consensusOption 2 (MCP tools)
One tool, no MCP dependencyOption 3 (FunctionTool)

FAQ

How do I validate the output of an AutoGen agent?

Read the final message from the TaskResult that agent.run returns, send it to a validator such as POST https://screwtape.ai/v1/judge with the task and any evidence, and only pass it on if the verdict is pass.

Can AutoGen use a remote MCP server over streamable HTTP?

Yes. autogen_ext.tools.mcp provides StreamableHttpServerParams (with url and headers fields) and mcp_server_tools, which return the server's tools for an AssistantAgent.

Why not just add a critic agent?

A critic built on the same model family may share the worker's blind spots and favor similar outputs. Goel et al. (2025) found LLM judges favor models similar to themselves. An external check against evidence adds a signal the agents don't produce themselves.

Do I need an OpenAI key for these examples?

Only for the AutoGen model client that runs the agents. The Screwtape calls need SCREWTAPE_API_KEY, which is a free key with 100 calls per UTC day.

Sources