Behavior lives in the daemon, not the web deck

intermediate5 min read · updated 2026-06-20

Behavior lives in the daemon, not the web deck

What we tried

The product is split into two processes that look like one thing from the outside:

• a web deck (the UI / dashboard server) that renders pages and accepts requests, and
• a daemon (the long-running brain/memory/core process) that the deck delegates to over IPC.

We changed real behavior — edits to the brain, the memory layer, and core routing — and then restarted the web deck to "deploy" the change. We reloaded the page, re-ran the same query, and saw the exact old behavior. So we edited again, restarted the deck again, tested again. Nothing moved. The natural (wrong) conclusion in the moment is "my edit didn't take" or "the file didn't save," which sends you editing code that was already correct.

We burned roughly 4 full edit→restart-UI→retest loops chasing a phantom before noticing the request was being answered by a process we never restarted.

What broke

The request path is `deck → IPC → daemon`. The deck does the rendering and the talking; the daemon holds the loaded code in memory and does the actual work. Restarting the deck reloads the transport, not the logic.

Two well-documented mechanics make this inevitable, not a quirk of our stack:

• Imported code is cached in memory for the life of the process. In Python, every imported module is stored in `sys.modules`, and on a repeat import "Python doesn't actually reevaluate the code for that module: it just gives us back the same module object as before" (1 cache per module). A long-running daemon imported `brain`, `memory`, and `core` once at startup; your file edits on disk are invisible to it until the process dies and a fresh one re-imports. The official guidance for picking up changes is to start a new process — `importlib.reload()` exists but leaves old class/instance references dangling, so it is not a safe drop-in (see the importlib docs).
• The same trap exists one layer down at the service manager. systemd caches unit config in memory and won't see an edited unit file until you explicitly `daemon-reload` and then restart the service (source). Editing the thing on disk and expecting the running process to notice is the same mistake whether the "thing" is a Python module or a unit file.

So the deck was a red herring twice over: it neither holds the edited logic nor is it the process that needs reloading.

The fix

Restart the daemon, not the deck. In our setup the daemon runs under a supervisor (launchd), so the deploy is simply: send `SIGTERM` to the daemon PID and let the supervisor respawn a fresh process that re-imports the edited code. Respawn lands in about 5 seconds. Do not force-rebuild and do not restart the web deck — neither touches the loaded logic, and a force-rebuild can orphan OS-level grants on adhoc-signed binaries.

The mental model that prevents the relapse:

• The deck renders. The daemon decides. If you changed what the answer is (brain/memory/core), restart the daemon. If you changed how a page looks or routes a request, restart the deck.
• The canonical source tree is the one the daemon actually imports. If the project root is reached through a symlink, edit the real path, not a stale mirror — otherwise you restart the daemon and it re-imports the old code anyway.

Apply it

• Map the processes before you "deploy." Write down, for your own system: which process renders the UI, and which process holds the business logic in memory. They are usually not the same one.
• Trace one request end to end. If the UI delegates over IPC/HTTP/queue to a worker, the worker is what runs your edited code. Restarting the front door changes nothing.
• Restart the process that imported the changed module. For a supervised daemon: `SIGTERM` the daemon PID and let the supervisor respawn it. Confirm a new PID before retesting.
• Verify against behavior, not against "I restarted something." Re-run the exact case that was wrong and confirm the new output. Same output + new PID means you restarted the wrong process — go back to step 1.
• Don't reach for hot-reload as the fix. `importlib.reload()` and friends leave stale references to old classes and instances (Python Morsels); a clean process restart is the reliable production move.
• Pin the canonical path. If the daemon's working tree is a symlink, edit the real directory and confirm that's what the running process imported, so a clean restart can't silently re-load old code.