'Done' means deployed and re-runnably proven, not coded

intermediate6 min read · updated 2026-06-20

'Done' means deployed and re-runnably proven, not coded

A task is not finished when the code is correct. It is not finished when the diff is merged. It is finished when the thing the user actually touches is live, and you have read it back from the live surface and confirmed it does what you claimed. Anything short of that is a story you are telling yourself.

The cheapest way to close this gap is also the fastest: a read-only check against the live artifact that runs in about 60 seconds. The most expensive way to learn the lesson is to deploy a local directory you assumed was current and find out it was 1 regression away from overwriting a correct production state with a fabricated one.

What we tried

We had a local deploy directory — the staging folder a deploy script reads from and pushes to the live host. The plan was the obvious one: "the site needs an update, deploy the local dir." On paper this is exactly what deploy directories are for.

We trusted two things without checking either: that the local directory reflected the current intended state, and that "deploy" would move us forward. Both felt safe because the deploy tooling was authenticated and the command was one line.

What broke

The local deploy directory was stale and fabricated. It still contained old, made-up trading performance numbers — a large fake P&L and an invented win rate — that had been scrubbed from the truth long ago. Meanwhile the live site was already truthful: someone had corrected it directly on the production surface, and that correction never flowed back into the local dir.

So the actual states were inverted from our mental model:

• Local deploy dir: old, wrong, fabricated. Looked like "the code."
• Live site: current, correct, truthful. The thing users see.

Deploying the local dir would not have been an update. It would have been a regression that re-introduced fabricated numbers over a page that was already honest. "Deploy the latest" would have shipped a lie, confidently, in one command — and we'd have called it done.

Two failures stacked here:

• We treated "merged/staged" as "deployed." The work felt finished because a local artifact existed. But a local artifact is a claim about production, not production.
• We never read back the live surface. Had we fetched the live page first, we'd have seen it was already correct and that our "newer" local copy was the stale one. The direction of staleness was the opposite of what we assumed.

This is the same family of bug as a stale installed binary, but one layer out: there, the source was right and the installed artifact was old; here, the live artifact was right and the staging artifact was old. In both cases the only cure is to verify the thing that actually serves users, not the thing that's convenient to look at.

The fix

Redefine "done" so it is impossible to satisfy without touching the live surface. Concretely:

• Build once, deploy that exact thing — don't trust a folder you didn't just produce. A deploy directory is only trustworthy if it was generated now from a known-good source by a repeatable build, not edited-in-place over time. "Build once, promote the same artifact" exists precisely so production is exactly what you validated, with no drift between what you tested and what ships (Marmelab — build once, deploy many). A long-lived local `_deploy` dir that people occasionally hand-edit is the anti-pattern.

• Read back the live artifact before AND after deploying. Before: fetch the current live page/endpoint and diff it against your intended change. If live is already correct, you may have nothing to deploy — or you may discover live is newer than your local copy. After: fetch it again and confirm your change is actually present. This is a post-deploy smoke test — a fast, read-only sanity check that the deployed thing serves the expected content and returns a healthy status (testkube — smoke testing; CloudBees — post-deploy smoke tests).

• Make the proof re-runnable, not a one-time eyeball. "I looked at it and it seemed fine" decays the moment anything changes. Encode the check as a command you can run again: `curl` the live URL and assert the truthful value is present and the fabricated value is absent; hit the health endpoint and assert 200; confirm dependencies are reachable. Production smoke checks should stay read-only so re-running them never mutates state (CloudBees — keep production smoke tests read-only).

• Know which artifact is the source of truth. When the live surface can be edited independently of your local copy, live wins by default until proven otherwise. Treat the local dir as a possibly-stale cache of production, not as the master. Pull from live (or rebuild from canonical source) before you push.

• Tie the artifact to its source. The strongest version of "done" is verifiable provenance — a record of where, when, and how the deployed artifact was produced, so you can prove the live thing was built from the source you intend and wasn't tampered with or hand-mutated (SLSA — build provenance; SLSA — verifying artifacts). You don't need full SLSA tooling to adopt the mindset: every deploy should answer "what source produced this, and can I check?"

Apply it

• "Coded" is not "shipped"; "shipped" is not "proven." Three distinct states. Done is the third one, confirmed against the live surface.
• A local deploy dir is a claim about production, not production. It can be stale or fabricated even when — especially when — the live site is already truthful. Verify before you push.
• Read back live before deploying. The cheapest way to avoid a regression is to discover that live is already correct, or already newer than your local copy, before you overwrite it.
• Direction of staleness is not obvious. Don't assume your local copy is the fresh one. When the live surface is independently editable, assume live is the master until you've checked.
• If the proof isn't re-runnable, it isn't proof. Encode "it works" as a command (assert truthful value present, fabricated value absent, status healthy) so anyone can re-verify in seconds.
• One-command deploys deserve one-command verification. The same fluency that lets you ship a regression in one line should let you catch it in one line.