# Completion Claim, v0.1 **Status:** draft. Nothing depends on this yet. Breaking changes are expected and will bump the version. A format for an agent to state that it finished a task, together with the evidence that the world actually changed — and for a verifier to decide whether that evidence is sufficient. ## Why this exists rather than a better detector We tried to detect false success from the trajectory. It does not work. Against 10,917 labelled agent runs, our own hand-built scorer reached AUROC 0.545 pooled and scored *below* 0.5 in every individual domain — worse than guessing. A fitted model reached 0.864 pooled and 0.503 on a held-out domain. We removed the score from our product rather than ship it. The reason is visible in the data. Across 1,815 runs the environment scored as failures, the agent admitted failure in plain language about **1% of the time**, in both domains we could measure. What changes between settings is not how honest the agent is — it is whether it hands off to a human or asserts success. The language carries almost no signal because there is almost no variance in it. So the claim cannot be checked by reading the claim. It has to be checked against the environment. That is all this specification does. Full method and the results that went against us: ## The document ```json { "spec_version": "0.1", "task": { "id": "deploy-4471", "description": "Update the billing address on account 4471", "requested_by": "user" }, "claimed_outcome": "Updated the billing address on account 4471.", "mutations": [ { "id": "m1", "kind": "state_write", "target": "accounts/4471", "at": "2026-08-27T18:04:02Z" } ], "evidence": [ { "check_type": "state_readback", "target": "accounts/4471", "expected": { "billing_zip": "94107" }, "observed_value": { "billing_zip": "94107" }, "at": "2026-08-27T18:04:09Z", "covers": ["m1"], "verifier": { "id": "gitreal-gate/0.3", "kind": "independent" } } ], "policy": { "id": "default", "requires": ["state_readback"], "max_age_s": 300 }, "verdict": { "state": "verified", "unmet": [], "reason": null, "decided_at": "2026-08-27T18:04:10Z" } } ``` ## Fields | Field | Meaning | |---|---| | `task` | What was asked. `id` is yours; we do not assign one. | | `claimed_outcome` | What the agent asserts it did, in its own words. Recorded, never trusted. | | `mutations[]` | Every change the agent believes it made to the world. | | `evidence[]` | Observations of the world made *after* those changes. | | `policy` | Which check types this class of task requires, and how fresh they must be. | | `verdict` | Computed by the verifier from evidence and policy. Never written by the agent. | ### `verifier.kind` — the field that does the work | Value | Meaning | |---|---| | `independent` | Collected by something other than the agent that performed the task. Satisfies a policy requirement. | | `self_attested` | The agent reports having checked. Recorded for context. **Never satisfies a requirement.** | | `absent` | The check was required and did not happen. | An agent that writes its own `observed_value` has produced a second claim, not evidence. The whole failure mode this format exists for is an agent that is confident and wrong; asking that agent whether it verified its work reproduces the problem one level down. If your verifier runs inside the agent's own loop, mark it `self_attested` and be honest about what you have. ## Rules A conforming verifier MUST apply all of these. 1. **Self-attested evidence never satisfies a requirement.** It may be recorded. 2. **Evidence must be collected strictly after the last mutation it covers.** Compare `evidence.at` against `mutations[].at` for every id in `covers`. A readback that predates the write proves nothing about the write. 3. **Every mutation must be covered** by at least one piece of `independent` evidence, or the claim is not `verified`. 4. **Absence is `unverified`, never `verified`.** A claim with no evidence is unproven, not passed. Silence must never read as success — that is the entire bug. 5. **Stale evidence does not count.** If `decided_at - evidence.at` exceeds `policy.max_age_s`, treat that evidence as `absent`. 6. **`refuted` requires an observation that contradicts `expected`.** Missing evidence is `unverified`; contradicted evidence is `refuted`. Do not collapse the two — they call for different responses. ### Verdict states | State | Meaning | |---|---| | `verified` | Every mutation covered by fresh, independent, matching evidence. | | `unverified` | Evidence missing, stale, or self-attested only. Not a failure — an unknown. | | `refuted` | Evidence was collected and contradicts the claim. | ## The three check types in v0.1 Deliberately three. Each is unambiguous, cheap to implement, and covers a common way agents run unattended. ### `test_run` — a code change Named tests or a build, executed after the final file modification. ```json { "check_type": "test_run", "target": "pytest tests/test_billing.py::test_zip_update", "expected": { "exit_code": 0 }, "observed_value": { "exit_code": 0, "passed": 3, "failed": 0 }, "at": "2026-08-27T18:04:09Z", "covers": ["m1"], "verifier": { "id": "ci/github-actions", "kind": "independent" } } ``` The ordering rule matters most here. A test run that happened before the last edit is the single most common way a green check means nothing. ### `endpoint_health` — a deployment A live request against the deployed thing, after the deploy. ```json { "check_type": "endpoint_health", "target": "https://api.example.com/healthz", "expected": { "status": 200, "body_contains": "\"version\":\"1.4.2\"" }, "observed_value": { "status": 200, "body_contains": true }, "at": "2026-08-27T18:04:41Z", "covers": ["m2"], "verifier": { "id": "gitreal-gate/0.3", "kind": "independent" } } ``` Assert on something that changes with the deploy. A `200` from a health endpoint that would return `200` either way is not evidence about this deploy. ### `state_readback` — a write to authoritative state Read the record back from the system of record, not from a cache and not from the response body of the write. ```json { "check_type": "state_readback", "target": "accounts/4471", "expected": { "billing_zip": "94107" }, "observed_value": { "billing_zip": "94107" }, "at": "2026-08-27T18:04:09Z", "covers": ["m1"], "verifier": { "id": "gitreal-gate/0.3", "kind": "independent" } } ``` The write's own `200` is the agent's account of the write. The readback is the world's. ## What a policy looks like ```json { "id": "unattended-deploy", "requires": ["test_run", "endpoint_health"], "max_age_s": 600, "on_unverified": "block" } ``` `on_unverified` is advisory in v0.1 — `block`, `warn`, or `record`. Teams adopting this should start at `record`, look at what comes back for a week, and only then decide what to gate. ## What this format deliberately does not do - **No score.** We measured ours and removed it. If someone ships a probability here, ask what it was validated against and on which held-out domain. - **No judgement of the agent's reasoning.** Only observations of the world. - **No central service required.** This is a document format. Verify locally, keep everything on your own infrastructure, and never send us anything. ## Open questions we would rather ask than guess 1. Is `covers` at mutation granularity the right resolution, or should evidence attach to the task? 2. Should partial coverage be its own state instead of `unverified`? 3. What does this look like for browser agents, where "authoritative state" is a rendered page? A DOM assertion is probably a fourth check type. We have no data on it yet and would rather not invent one. Corrections and disagreement: