Never prune the data your engines learn from, and never use a banned feed: archive bars append-only so the system warm-starts from history instead of booting cold

intermediate6 min read · updated 2026-06-20

Never prune the data your engines learn from, and never use a banned feed

The signal-generating engines in this system learn from market history: backtests, edge-proving, and warmup all depend on a continuous, trustworthy record of price bars. Two cheap-looking decisions can quietly destroy that record — deleting old data to "save space," and pulling history from a convenient-but-untrusted feed. Both are forbidden here, and the reason is the same: the data your system learns from is not disposable infrastructure. It is the product. This entry covers what we tried, what broke, the fix, and how to apply the rule anywhere you keep a learning corpus.

What we tried

Two tempting shortcuts kept resurfacing:

• "Just prune the old bars." Bar tables grow forever. The instinct is to add a retention window — keep 90 days, drop the rest — the way you would for application logs.
• "Just grab the history from Yahoo." When an engine boots cold and needs backfill, the fastest path is a free public endpoint (Yahoo Finance / `yfinance` / `query1.finance`). It is one HTTP call away and requires no session.

Both feel like harmless plumbing. Neither is.

What broke

• Pruning silently erodes the thing the engines exist to learn from. An append-only log is valuable precisely because nothing can be modified or removed after it is written — that is what makes it trustworthy as a single source of truth you can replay (Azure Event Sourcing). A retention cron that drops "old" bars is indistinguishable, after the fact, from data corruption: a backtest that ran fine last quarter now returns different numbers because its inputs are gone, and you cannot tell whether the strategy changed or the history did. Lost history is not recoverable — there is no second copy of last Tuesday's tape.

• A second writer / overwrite path quietly mutates history. The danger isn't only an explicit `DELETE`. An UPSERT that overwrites on conflict will silently rewrite a bar that already exists, so a buggy or out-of-order ingest can corrupt a known-good row with no error. Append-only means append-only: the only safe conflict policy is to skip, not to overwrite.

• A banned feed poisons the corpus with data you can't trust. A feed that backfills, adjusts, or silently revises bars makes your stored history disagree with what actually printed. Once a bad bar is in the permanent store, every backtest and every edge proof downstream of it inherits the error. Garbage history produces confident, wrong edge claims — the worst possible failure for a trading system, because it looks like a real result.

• The engines boot blind because they archive but never replay. This is the open wound the rule exposes. The engines faithfully write every bar to permanent storage — and then, on restart, ignore all of it. They run a fixed warmup of 600 seconds (~10 minutes) of cold observation before they will act, re-learning context they already have on disk. You paid to store the history and then refused to use it.

The fix

1. Make the store physically append-only. The canonical bars table uses an idempotent insert that refuses to mutate existing rows:

```sql

INSERT INTO bars (symbol, ts, open, high, low, close, volume)

VALUES (...)

ON CONFLICT (symbol, ts) DO NOTHING;

```

`ON CONFLICT ... DO NOTHING` leaves any pre-existing row exactly as it was and raises no error, which makes re-ingesting the same window safe and idempotent (PostgreSQL INSERT docs). Crucially, choose `DO NOTHING`, not `DO UPDATE`: `DO UPDATE` would let a later, possibly-wrong write clobber a good bar. The conflict key `(symbol, ts)` is the immutability guarantee.

2. Ban retention/prune crons on the learning corpus. There is no `DELETE FROM bars` in production — the only deletes that exist are tests cleaning up their own rows. Append-only logs treat the data as a permanent, sequential record you can re-consume after a crash rather than a buffer you trim (QuestDB: append-only storage; Write-Ahead Log). Keep a second permanent copy too — each engine appends its own `bars_log.csv` — so the corpus survives a database mishap.

3. Pin the trusted feed; allow no fallback to the banned one. The real feed (here, WealthCharts via the Chrome CDP bridge) is the only source, enforced in config rather than left to runtime choice: `DATA_FEED=chrome`, `DISABLE_YAHOO=true`, and no Yahoo seed for warmup. A "fallback feed" is not a safety net — it is a path through which untrusted data enters the permanent store. Forbid it explicitly.

4. Warm-start from the history you already kept. This is the payoff. On boot, replay stored bars during the existing warmup window instead of waiting blindly. Replaying a log to reconstruct current state is exactly what append-only storage is for (Azure Event Sourcing), and for large histories a snapshot plus a short tail of recent bars keeps rehydration fast. Done right, an engine restart costs near-zero blind time because it rehydrates from `bars` before the first live tick.

Apply it

• Separate "logs you trim" from "data you learn from." Application logs can rotate. The corpus your models, backtests, or evals depend on is permanent — never put a retention window on it.
• Append-only means no overwrite, not just no delete. Use `ON CONFLICT DO NOTHING`, keep a single writer, and treat any path that mutates an existing record as a bug.
• One trusted source, zero fallbacks. A convenient backup feed that revises data will poison everything downstream of it. Pin the good source in config and disable the bad one explicitly.
• If you stored it, use it on boot. Archiving history and then booting cold is wasted money and needless blind time. Replay from the permanent store during warmup so a restart isn't a regression.
• Test the immutability, not just the happy path. Add a test that re-ingests an existing bar and asserts the original row is unchanged — that is the regression lock that keeps a future "small optimization" from turning your source of truth into a guess.