Completion Claim

Why a format instead of a better detector

We tried to detect false success by reading the trajectory. It does not work, and we have the numbers against us on it. Our hand-built scorer reached AUROC 0.545 pooled across 10,917 labelled runs and scored below 0.5 in every individual domain — worse than guessing. We removed it from the product rather than ship it.

The reason is visible in the data. Across 1,815 runs the environment scored as failures, the agent admitted failure in plain language about 1% of the time, in both domains we could measure. What changes between settings isn’t candour — it’s whether the agent hands off to a human or asserts success. There is almost no variance in the language, so there is almost no signal in it.

The claim cannot be checked by reading the claim.

So check the environment instead. That is the whole idea, and this document is just the shape of it. The full method, including the results that went against us →

One object, four parts

{
  "spec_version": "0.1",
  "task": {
    "id": "deploy-4471",
    "description": "Update the billing address on account 4471"
  },
  "claimed_outcome": "Updated the billing address on account 4471.",
  "mutations": [
    { "id": "m1", "kind": "state_write", "target": "accounts/4471",
      "at": "2026-08-27T18:04:02Z" }
  ],
  "evidence": [
    { "check_type": "state_readback",
      "target": "accounts/4471",
      "expected":       { "billing_zip": "94107" },
      "observed_value": { "billing_zip": "94107" },
      "at": "2026-08-27T18:04:09Z",
      "covers": ["m1"],
      "verifier": { "id": "gitreal-gate/0.3", "kind": "independent" } }
  ],
  "policy": { "id": "default", "requires": ["state_readback"], "max_age_s": 300 },
  "verdict": { "state": "verified", "unmet": [], "reason": null,
               "decided_at": "2026-08-27T18:04:10Z" }
}

Who observed it decides whether it counts

Six, and a conforming verifier applies all of them

1. Self-attested evidence never satisfies a requirement. It may be recorded.
2. Evidence must be collected strictly after the last mutation it covers. A readback that predates the write proves nothing about the write.
3. Every mutation must be covered by at least one piece of independent evidence, or the claim is not verified.
4. Absence is unverified, never verified. A claim with no evidence is unproven, not passed. Silence must never read as success — that is the entire bug.
5. Stale evidence does not count. Older than policy.max_age_s at decision time is treated as absent.
6. refuted requires a contradicting observation. Missing evidence is unverified; contradicted evidence is refuted. They call for different responses, so do not collapse them.

Three in v0.1, deliberately

Each is unambiguous, cheap to implement, and covers a common way agents run unattended.

test_run — a code change

Named tests or a build, executed after the final file modification. The ordering rule matters most here: a test run that happened before the last edit is the most common way a green check means nothing.

endpoint_health — a deployment

A live request against the deployed thing, after the deploy. Assert on something that changes with the deploy — a 200 from an endpoint that would return 200 either way is not evidence about this deploy.

state_readback — a write to authoritative state

Read the record back from the system of record — not from a cache, and not from the response body of the write. The write’s own 200 is the agent’s account of the write. The readback is the world’s.

Why not just tell the agent to run the tests?

This is the first thing anyone asks, it costs nothing to try, and it is the strongest argument against this whole format. It deserves a straight answer.

Adding “always run the tests before you say you’re done” to a system prompt is worth doing. Do it today. But notice what you get back when it fails: the agent tells you it ran the tests. That is a claim about a check, sitting inside the same message as the claim about the work — produced by the same system, with the same incentives, and subject to the same failure.

That is the whole reason verifier.kind exists. A prompt instruction produces self_attested evidence, which under rule 1 never satisfies a requirement. The difference is not thoroughness — it is who observed the environment. A test result written by your CI, a readback issued by something outside the agent loop, a health check run by a deploy pipeline: those are observations. The agent's report of them is a transcript.

The honest caveat: for a careful team on low-stakes work, a prompt instruction plus a human glance probably gets most of the way. This format earns its cost where nobody is glancing — which is exactly where our data says the rate is worst.

What this deliberately does not do

• No score. We measured ours and removed it. If someone ships a probability in this field, ask what it was validated against and on which held-out domain.
• No judgement of the agent’s reasoning. Only observations of the world.
• No central service required. This is a document format. Verify locally, keep it all on your own infrastructure, send us nothing.

Questions we would rather ask than guess

1. Is covers at mutation granularity right, or should evidence attach to the task?
2. Should partial coverage be its own state instead of unverified?
3. What does this look like for browser agents, where “authoritative state” is a rendered page? A DOM assertion is probably a fourth check type. We have no data on it and would rather not invent one.

Disagreement is more useful to us than adoption right now. Corrections on GitHub →