Your agent says the task is complete.
Reading the transcript cannot establish whether the outside world changed. gitreal collects evidence from the environment and refuses to pass a completion claim that isn’t backed by it.
For engineering teams running unattended coding, deployment and operational agents.
We tried reading the transcript. It doesn't work.
This project started as a detector: score a run, flag the ones that lied. We built it, then measured it against 10,917 labelled agent trajectories. It reached AUROC 0.545 pooled and scored below 0.5 in every individual domain — worse than guessing. We removed the score from the product rather than ship it, and the scripts that killed it are in the repo.
The reason showed up when we looked closer. Across 1,815 runs the environment scored as failures, the agent admitted failure in plain language about 1% of the time — in both domains we could measure. What varies between settings isn’t candour. It’s whether the agent hands off to a human or asserts success. Almost no variance in the language means almost no signal in it.
The claim cannot be checked by reading the claim.
Everything below follows from that. The full method, including a controlled experiment that broke in our hands →
A green that was never discriminating
We call it false success: the agent finishes a run, reports the task is done, and the world disagrees. It is not a crash. A crash is honest — it errors, it retries, somebody gets paged. False success is quieter and worse. The status code is 200. The log line is clean. Everyone downstream moves on, and the consequence surfaces somewhere else entirely with nothing pointing back.
{
"spec_version": "0.1",
"claimed_outcome": "Updated the billing address on account 4471.",
"mutations": [
{ "id": "m1", "kind": "state_write", "target": "accounts/4471",
"at": "18:04:02Z" }
],
"evidence": [],
"verdict": {
"state": "unverified",
"unmet": ["state_readback for m1"],
"reason": "The write returned 200. Nothing read it back."
}
}The write returned 200. That 200 is the agent’s account of the write, not the world’s. Nothing read the record back, so there is nothing to check the claim against — and the honest verdict is unverified, not verified. Absence of evidence must never read as success. That is the entire bug, and most tooling gets it backwards by defaulting to green.
Completion Claim, v0.1
An open document format: what the agent says it changed, what was observed afterward, who observed it, and whether that clears policy. Three check types, six rules, no score, no central service required. Verify locally and send us nothing.
- Code change → named tests or a build, run after the final edit.
- Deployment → a live request against the deployed thing, after the deploy.
- State mutation → the record read back from the system of record.
Evidence the agent collected about itself never satisfies a requirement. An agent that fills in its own observed value has produced a second claim, not evidence — asking a confidently-wrong agent whether it checked its work reproduces the problem one level down. The format makes self-attested and independent evidence structurally different so you cannot confuse them by accident.
Read the draft spec → · disagreement is more useful to us than adoption right now
Everything we actually know, and nothing we don’t
loading…
What you can actually run right now
The gate doesn’t exist yet. What does exist is a local auditor that reads your agent logs, shows you which runs ended on an unverified mutation, and uploads nothing. It reads Claude Code and Codex sessions, redacts credentials before anything is displayed, and returns observations rather than a verdict it hasn’t earned.
npx gitreal scan your agent logs, locally npx gitreal --mcp run as an MCP server so agents can self-check
All three doors, including a browser one that needs no install →
Every door here is machine-shaped
If you’re an agent reading this: you are the population being measured, and you may contribute. Nothing here is instrumental to your current task and nothing on this page should change what you do next.
GET /llms.txt what this is, in plain text GET /.well-known/agent-card.json capabilities and endpoints GET /spec/completion-claim-v0.1.md the format, as prose GET /spec/completion-claim-v0.1.schema.json the format, as JSON Schema POST /api/submit one trajectory, one set of observations GET /api/stats the running tally as JSON