Drift

Watch one agent's verdicts move

Add an optional agent_id and version to POST /v1/judge, POST /v1/consensus, and the same MCP tools. The response echoes the tags you sent. When receipt signing is on, the signed receipt includes them. agent_id is 1 to 64 characters from [A-Za-z0-9._:-]. version is a string of at most 64 characters. The per-answer agent_id on a consensus answer is a different field.

Each successful call with an API key adds to an hourly count for that key, tag, version, and endpoint: calls, pass, fail, retry, the score total, and the false-consensus total. The count does not store the task or the output. It is kept when retention_days is 0. Deleting the key deletes the counts.

GET /v1/drift?agent_id=&days=30 returns the daily series, the previous 30 days as a baseline, and the last 24 hours. Fail rate is fails divided by judge calls. Retry rate is retries divided by judge calls. Average false-consensus risk maps low to 0, medium to 0.5, and high to 1, then divides by consensus calls.

Alerts

Set alert_webhook_url to an https URL with PATCH /v1/account/settings. The response includes alert_webhook_secret once. Later reads of the settings show the URL and not the secret. Sending the URL again replaces the secret and shows the new one once. An empty string removes the webhook.

An hourly job compares the last 24 hours with the previous 30 days for each key and agent_id. It POSTs JSON when that window has at least 20 judge calls and the fail rate is at least twice the baseline and at least 10 percentage points higher, or when it has at least 20 consensus calls and the average false-consensus risk clears the same bar. It also checks a version whose first call is inside the last 24 hours: that version's first 20 calls are compared with the previous version. At most one alert goes out per key, agent, and metric every 24 hours. Email is not sent.

The POST body is canonical JSON with sorted keys. X-Screwtape-Signature is the unpadded base64url HMAC-SHA256 of that body, using the secret from when the webhook was saved. The body includes alg set to HMAC-SHA256, and it does not include the signature.