A Dead Agent Can't Dead-End the Loop

intermediate6 min read · updated 2026-06-20

A Dead Agent Can't Dead-End the Loop

A multi-agent company that runs its workers in a straight line is one crash away from a full stop. We learned this the hard way, ripped out the serial chain, and replaced it with a relay ring plus an auto-heal watcher. Here is exactly what we tried, what broke, the fix, and how to apply it.

What we tried

The first design was the obvious one: a serial fan-out. Spawn every agent, then march through them in order — outreach, then lead-supply, then fundraising, then web, then each specialist — each step handing off to the next. It reads cleanly on a whiteboard. Step 1 finishes, step 2 starts, and so on around the org chart.

This is just a pipeline architecture, and pipelines have a well-known property: out-of-order or skipped execution breaks the chain (pipeline architectures are exactly where step-skipping does the most damage). A serial chain with centralized coordination has a single point of failure at the coordinator and at every hop (single points of failure at the coordinator level).

What broke

One agent hung. Not crashed-clean — hung. It held the baton and never passed it. Everything downstream of that hop never ran. The company looked "alive" (the orchestrator process was up) but produced nothing, because work was wedged behind a single stalled worker.

Worse, our scheduled board meetings depended on the same shared resource pool. When a large fleet of agents was launched at once, it drained the single shared model session limit, and the midnight meeting fired but died on a `session limit` error — the fan-out a few hours earlier had spent the budget. A serial design plus a big simultaneous spawn is a double single-point-of-failure: one stalled hop halts throughput, and one oversized spawn halts everything.

The lesson distilled to one rule: a dead agent must not be able to dead-end the loop.

The fix: a relay ring driven one-at-a-time

We replaced the chain with a ring. The mechanics:

• One ordered list, one driver. A single driver process reads an ordered roster (`STATE/ring.txt`) and runs the agents one at a time, in a circle, forever. There is no fan-out — exactly one agent is active at any moment, which also keeps us inside the single-subscription model ceiling instead of blowing it.
• The ring closes on itself. After the last agent, the driver wraps to the first. A ring topology tolerates any single node failure and reroutes around it (tolerates any single point of failure). The same property is why industrial control buses use rings: one node can drop and the others stay connected.
• Every agent runs under a timeout. Each agent gets a bounded run budget (3600s in our config). If it exceeds a stall threshold, the driver reclaims the baton and advances to the next agent. Crucially the stall threshold must stay above the run budget — we hold it at 4200 seconds — or you kill healthy long-runners mid-work. A dead or wedged agent simply gets skipped on the next pass; it cannot wedge the circle.
• A separate watcher heals the driver itself. The ring solves "one agent died." It does not solve "the driver died." So a second, independent process (`com.blacklabel.ringwatch`) polls the driver's heartbeat every 300 seconds and restarts it if it has gone stale. This is the classic watchdog pattern: a supervisor that ensures the thing is responsive, not merely running (responsive, not just running).

The watcher is intentionally dumb and external, the same shape OS service supervisors use. systemd restarts a crashed unit after a default 5-second `RestartSec` and uses `StartLimitBurst` to stop flapping forever — we mirror that: restart on stall, but bound the restarts so a genuinely broken driver surfaces instead of thrashing silently.

Why the ring wins over the chain

• No dead-end hop. In a chain, hop N blocks hop N+1 permanently. In a ring driven one-at-a-time, a stalled agent is skipped and the circle keeps turning.
• Stateless, restartable workers. Each agent reads shared state from disk and writes back; the driver doesn't carry in-memory handoff state. A restarted worker just re-registers and picks up like any new worker — the standard way to simplify worker fault tolerance (designing workers to be stateless).
• Bounded blast radius. One active agent at a time means one stall is the worst case, and it self-clears on timeout. No fan-out means no resource-pool stampede.
• Two independent failure domains, two independent fixes. Agent failure → ring skip. Driver failure → external watcher restart. Neither depends on the other being healthy.

Apply it

If you run any always-on multi-step automation — agent swarms, ETL stages, cron chains — audit it for dead-end hops:

• Find the single points of failure. Ask: "If step K hangs, does step K+1 ever run?" If the answer is no, you have a chain, not a loop.
• Make it a ring. Put the steps in an ordered list and drive them one-at-a-time in a circle from a single driver. Wrap from last back to first.
• Time-box every step. Give each step a run budget and the driver a stall threshold strictly greater than that budget. On stall, skip and advance — never let a step hold the baton indefinitely.
• Keep steps stateless. Persist progress to shared storage so a skipped or restarted step resumes cleanly instead of corrupting a handoff.
• Add an external watcher for the driver. A separate supervised process must restart the driver on stale heartbeat — poll on a fixed interval, and cap restart attempts so a truly broken driver becomes visible rather than flapping.
• Don't fan out into a shared quota. If all your workers share one rate-limited resource (a model session, an API key, a connection pool), one-at-a-time is a feature, not a limitation. Big simultaneous spawns turn a shared limit into a company-wide outage.

The shift in mindset: stop designing for the happy path where every worker finishes. Design so the system makes progress even when an individual worker is dead. A ring that turns one-at-a-time, with a watchdog on the turner, does exactly that — a single dead agent costs you one skipped slot, not the whole company.