A False Silence Floor Causes Restart Storms: Calibrate Thresholds to Measured Reality

intermediate6 min read · updated 2026-06-20

A False Silence Floor Causes Restart Storms: Calibrate Thresholds to Measured Reality

A threshold that should describe the world is only as good as the measurement behind it. Pick a number out of the air — "this is what silence sounds like," "this is the energy a wake word needs" — and set it above the level reality actually produces, and the system stops perceiving the world correctly. In an always-on listener, that single bad constant doesn't just degrade quality. It triggers a self-reinforcing failure: the process declares itself broken, the supervisor kills and respawns it, and the cycle repeats forever.

What We Tried

We ran an always-on voice loop ("hey ace") under a process supervisor. The loop had two threshold constants baked in as literals:

• A silence / true-silence floor — the RMS level below which the loop concluded the microphone was dead and bailed out so the supervisor could restart it. This was a real, useful self-heal: a wedged mic should trigger a restart.
• A wake threshold — the confidence score an openWakeWord detection had to clear before we treated it as a real "hey ace."

Both numbers were chosen by intuition, not by measuring the actual device in the actual room.

What Broke

Two independent failures, same root cause: a threshold set above the level reality produces.

1. The silence floor sat above the room's real quiet floor. We set the true-silence cutoff at an RMS of 0.0008. The room's genuinely-quiet ambient level was lower than that. So during normal silence the loop's own audio read below the "dead mic" threshold, concluded the mic was dead, and exited — even though the mic was perfectly healthy. The supervisor faithfully respawned it, the new process immediately read silence below the floor again, and exited again. We got a restart storm: a fresh restart roughly every 90 seconds, all night, with `mic_silent` and "wake word not configured" spam filling the logs. The self-heal designed to recover a dead mic was instead causing the outage.

2. The wake threshold was set above the user's real voice scores. Separately, the wake confidence bar was set so high that the user's own "hey ace" — which scored across a wide band — kept landing below the bar and was silently rejected. The system wasn't deaf because of hardware; it was deaf because the gate was calibrated against an imaginary, louder, more confident speaker than the one actually talking to it.

Both are the same bug wearing two costumes: the threshold encoded a belief about reality that reality did not match. When the gate is too high, true positives (real silence is normal, real voice is a wake) get classified as failures, and downstream machinery — a supervisor, a restart policy — amplifies the misclassification into a loop.

This is a known shape. Robust voice-activity detection in the literature does not hardcode a single energy cutoff; it tracks an adaptive noise floor and bounds the threshold so it can't go oversensitive in quiet rooms or under-sensitive in loud ones (adaptive noise floor tracking). Production VAD guidance is the same: thresholds must adjust to ambient noise to avoid false triggers, not assume one number fits every device and room.

The Fix

Three layers, in order of importance:

• Measure the real floor, then calibrate below it. We lowered the true-silence constant from 0.0008 to 0.00001 — a value verified to sit under the room's actual quiet level, so genuine silence no longer reads as a dead mic. We dropped the wake bar to match where the user's voice actually scored, recovering the rejected true positives.
• Make every threshold env-overridable, not a recompile. The new floor reads from an environment variable with the literal only as a default. A reality that varies by room, mic, and person must be tunable in the field — calibration you can't change without a rebuild will drift out of true and stay there. (In this codebase the live values, e.g. `UTAH_WAKE_THRESHOLD` and the silence floor, deliberately live in the supervisor's launch config, not in committed source, so a branch switch can't silently re-break them.)
• Don't let a misclassification spin freely. A self-heal that respawns on "dead mic" must not fire in a tight loop when its own trigger is wrong. The infrastructure world solved this with backoff: after a failed restart the delay grows exponentially (e.g. 10s, 20s, 40s) up to a cap of 300 seconds, so a faulty component can't overwhelm the system with constant restarts. Even with a correctly calibrated floor, a bounded backoff turns a storm into a slow, visible, survivable degradation.

We regression-locked the corrected floor in tests and verified live: `deaf:false`, zero new restarts over the watch window. The mic was never broken. The number was.

Apply It

• Never hardcode a threshold that's supposed to describe the physical world. Silence levels, wake confidences, light/temperature cutoffs, latency SLOs — measure the actual environment first, then set the constant relative to that measurement, with margin on the correct side.
• Calibrate to the quietest/weakest real signal, not the loudest imaginary one. Your floor must sit below genuine ambient, your wake bar below the real speaker's worst real utterance. Set it where reality lives, not where you wish it lived.
• Make thresholds env-overridable. Anything that varies by device, room, or user belongs in config you can tune without a rebuild — and pin the live values somewhere a branch switch or redeploy can't quietly revert.
• Prefer adaptive over fixed where you can. Track a running noise floor and bound it, rather than committing to one number for every environment.
• Backoff every self-heal. A restart-on-failure with no exponential backoff and no cap turns one wrong threshold into an all-night outage. Add backoff so a misclassification degrades slowly and stays observable instead of hammering.
• When a self-heal causes the outage, suspect its trigger. If "recovery" runs in a tight loop, the thing it's reacting to is almost always misclassified — fix the measurement, not the restart count.