A wrapper reporting a component 'down' can be lying: confirm with the component's own check before trusting a proxy's status

advanced6 min read · updated 2026-06-20

A wrapper reporting a component "down" can be lying

A status field is only as honest as the code that produced it. When a wrapper, proxy, or health endpoint says "component X is down," that string is a claim made by the wrapper - not by component X. If the wrapper's own machinery can fail in a way that looks like the component failed, you will burn hours fixing a thing that was never broken.

This entry is the generalized lesson from a real incident: a capability layer reported `daemon_alive: false` and surfaced "daemon down / ledger capabilities blocked." The daemon was fine. The database was fine. The wrapper was lying because of an asyncio reentrancy bug in its own health probe.

What we tried

The user-facing symptom was "daemon down, ledger capabilities blocked." The obvious suspects were the two heavy moving parts:

• The database. PostgreSQL on port :5433 - is it up, is it accepting connections, is the supervisor flapping? We checked it. It was healthy.
• The daemon itself. Is the process alive, is its socket bound, did it crash on boot? We checked it. It was alive and answering its own internal `call_sync` check.

So both components the wrapper accused were demonstrably healthy when interrogated directly. That mismatch - "the proxy says dead, the thing itself says alive" - is the whole lesson.

What broke

The capability layer's health probe ran an async call from inside a thread that already had a running event loop. Concretely, the wrapper's code path (`_utah_ipc`) tried to drive a synchronous bridge - `call_sync` / `anyio.run()` - on the same thread where the host framework's event loop was already running.

Python forbids this. You get RuntimeError: asyncio.run() cannot be called from a running event loop (the "this event loop is already running" family of errors). The probe raised, the wrapper caught the exception, and - critically - it interpreted "my probe threw" as "the daemon is dead," emitting `daemon_alive: false`.

Why was a loop already running on that thread? Because of how the host framework schedules work. Sync tools are dispatched to a thread pool while async tools run inline on the event-loop thread - so an async-shaped handler executes on the loop thread, and any attempt to start or nest a second loop there explodes. This is a well-documented sharp edge: see the python-sdk report of a FastMCP tool blocking asyncio.

The failure was a false negative manufactured by the observer. The component under observation never reported anything wrong; the act of observing it incorrectly is what produced the "down" signal.

The fix

Stop running a nested event loop on the framework's loop thread. Offload the synchronous bridge to a worker thread that does not already own a loop.

The canonical pattern, straight from the asyncio docs that recommend run_in_executor / a ThreadPoolExecutor to run blocking or loop-owning code off the event-loop thread:

• Inside the async handler, `await loop.run_in_executor(pool, blocking_call)` (or hand the sync bridge to an explicit `ThreadPoolExecutor`).
• The worker thread has no running loop, so `call_sync` / `anyio.run()` is free to spin up its own loop and complete.
• The result flows back to the awaiting handler without ever nesting a loop on the original thread.

`nest_asyncio.apply()` is the other commonly cited escape hatch - it monkeypatches asyncio to permit nested event loops - but it patches global asyncio behavior and is best avoided in a shared daemon; the thread-offload approach is the surgical fix.

Equally important: the probe was rewritten so that "my own probe raised" is no longer collapsed into "the dependency is down." A probe error and a dependency error are different facts and must be reported as different facts.

Apply it

The transferable rules, none of them specific to asyncio:

• Distrust a status string until the named component confirms it. "daemon_alive: false" is the wrapper's opinion. Go ask the daemon directly (its own internal `call_sync`, a raw socket probe, `psql`, `dig`, `curl`) before you touch the accused component. In this incident the daemon's own check said alive while the wrapper said dead - that single contradiction was the whole diagnosis.
• A health check that can fail for its own reasons must distinguish "probe failed" from "subject failed." Catch-all `except: return down` is how observers fabricate outages. Surface the probe's own exception separately.
• Suspect the newest / thinnest layer first. The database and daemon were old and stable; the capability wrapper was the recently-added layer carrying the novel failure mode. When a long-stable component is suddenly "down," weight your suspicion toward whatever was changed or added most recently.
• Never start a loop where a loop already runs. If you must call sync-blocking or loop-owning code from inside an async context, offload it to a thread pool (loop.run_in_executor) rather than nesting `asyncio.run()` / `anyio.run()`.
• After the fix, re-verify through the same proxy that lied. Confirm the wrapper now reports `daemon_alive: true` AND that the daemon's direct check still agrees. A fix that only satisfies one of the two has not closed the contradiction. Restart the layer so the corrected code is actually loaded - a patched-but-not-reloaded process is its own false signal.

The meta-principle: verify with the source, not the proxy. Every health dashboard, status badge, and capability flag is a proxy. Proxies have bugs. When a proxy and the source disagree, that disagreement is data - and the proxy is usually the one that's wrong.