A bug you fixed last week is live again. Nobody touched that file. No revert, no
bad merge, no rollback. The fix simply evaporated. This is one of the most
confusing classes of regression in real engineering, and the cause is almost
always the same: the fix was committed to a feature branch, but the running
system loads code from whatever branch happens to be checked out — and that
branch never got the fix.
A live voice subsystem kept breaking the same way: false wakes, empty commands,
restart storms. We diagnosed it, fixed it carefully, and committed the fix —
restored a 861-line superset of the loop, added the empty-wake cooldown, the
120Hz high-pass filter, the stream-reopen logic — and verified it working live.
Done. We moved on.
Days later the exact same symptoms came back. The fix was real and still sitting
in git history. But the live process imports its code from a working directory
that is a symlink into the repo, and that working directory had a *different
branch* checked out — a branch that predated the fix. The moment someone ran a
`git checkout` to switch branches for unrelated work, every file in the working
tree was silently rewritten to that branch's version, and the running system
picked up the old, broken loop on its next restart.
The root cause is not a bug in your code. It is a property of git itself.
When you pass a branch name to `git checkout`, the command moves `HEAD` and
updates the files in your working directory to match the target branch
(100% of tracked files that
differ are replaced). This is documented, expected, and exactly what you asked
for — but it means a checkout is a full content swap of the working tree, not
just a pointer change. If your fix lives on branch A and you check out branch B,
the files on disk now contain branch B's version of that code. Your fix is gone
from disk even though it is safe in history.
Now combine that with how live systems actually load code:
commit.** A daemon, a launchd service, a `python -m` process, or a symlinked
deploy reads whatever bytes are on disk right now.
green PR sitting unmerged ships nothing. A fix on `claude/product-areas` does
nothing for a service running off `main`.
fixed code in memory and looked healthy. The next restart — a crash, a
deploy, a reboot — loaded the now-reverted file and the bug returned.
So three independent facts have to line up, and when they don't, you get a
phantom regression: (1) the fix is on branch A, (2) the working tree gets
switched to branch B, (3) the live process restarts and re-imports the file. No
single step looks wrong in isolation, which is why this wastes hours.
In DORA terms, this is textbook change failure: a regression that forces
unplanned forward-fix work
(regression = rework).
Elite teams keep change failure rate around
5% while low performers run
as high as
30% — and "the fix was on
a branch" is exactly the kind of self-inflicted failure that pushes a team into
the low cluster while feeling like everyone is working hard.
The durable rule is one sentence: **a fix is not done until it is on the branch
the live system actually runs.** Everything else is verification of that.
imports from — check the symlink target, the deploy script, or the service
working directory. That is your "main" for this system, whatever it is
literally named.
feature branch is a proposal. Merge (or cherry-pick) it onto the branch of
record so a checkout of another branch can never silently un-apply it. In
our case the voice fixes were re-landed directly on `main` (commit `42a3513`)
specifically so a branch switch could not re-break them.
the branch of record should show the fix; the file in the runtime's working
directory should contain it. Don't verify on your laptop's copy — verify
where the process reads from.
code in memory, restart the service and check a live signal (a log line, a
threshold, a behavior) that only the fixed version produces. "It works
without restarting" is not proof; the next restart is the test.
half code (in git) and half environment (in a launchd plist, an env var, a
secret). The git half can land perfectly and the system still misbehaves
because the env half wasn't deployed. Land both, and write down which is
which.
branch right now and the service restarts, is my fix still live?"* If the
answer is no, you are not done.
ships nothing — verify the bytes on the branch and working tree the runtime
reads.
symlinked or floating working directory, that is a regression trap; pin it to
a known branch or deploy from an immutable artifact instead of a live checkout.
checkout that drops the fix also turns a test red instead of silently shipping
the old bug.
The deeper lesson: in any system where the live runtime loads from a working
directory, "the branch that is checked out" is a piece of production state. Treat
a stray `git checkout` with the same respect you'd give a deploy, because to the
running process, it is one.