Phase 2: Can our software factory survive a crash during overnight runs?

A real enterprise build runs for hours, sometimes overnight. Until now, our factory died with its process a power blip, an out-of-memory kill, or a routine restart eight hours in would lose all the in-progress work, and worse, force us to redo and re-pay for the parts that were already finished. Phase 1 proved the factory can safely change existing code. Phase 2 gave it a memory that survives a crash, and a way to pick up exactly where it left off. We chose the simplest possible foundation a built-in database, no new moving parts and we proved it the honest way: we started a real multi-part build, killed it dead in the middle with kill -9, and restarted it. It correctly recognised the work already completed, spent nothing redoing it, cleaned up the half-finished mess the crash left behind, and ran the rest to completion. Verdict: the factory can now be trusted to run long, unattended jobs without losing work or money to a crash.

What we set out to solve

The factory builds software by running many coding jobs in parallel over a long stretch of time. That long horizon is the whole point for an enterprise real work isn't a five-minute toy build but it's also the danger.

The problem was blunt: everything lived in memory, in a single process. If that process stopped for any reason a crash, a server reboot, someone closing a laptop every record of what had been done vanished. Restarting meant starting over from zero. On a job that had already run for hours and spent real money on AI coding sessions, that's not just slow; it's paying twice for the same work, and there's no guarantee the second run even lines up with the first.

So Phase 2 asked: can the factory remember its progress durably, and resume after a crash without re-doing or re-paying for work that was already finished? Every later ambition (jobs that wait days for a human approval, long-running deployment monitoring, scheduled overnight runs) assumes this foundation exists.

The key decision we had to get right

There's an industry of heavyweight "workflow engine" products built for exactly this problem. The tempting move was to adopt one. We deliberately didn't not yet. Reaching for a heavy external system this early would have added a large new dependency, operational complexity, and lock-in, to buy durability we could get far more cheaply.

Instead we chose a built-in database that ships with the language itself nothing new to install, nothing new to operate and designed it single-machine first. The important part: we did this behind a clean boundary, so if the day comes that the factory needs to spread across many machines, the heavier engine can slot in behind the same interface without rewriting everything around it. Cheap to choose now, and not a trap later. This is the classic keystone decision: ruinous to retrofit, so we made it early and made it reversible.

What we learned along the way

The revealing discovery was that the factory was already writing down its progress after every step but nothing ever read it back. The receipts were being filed and then ignored. Half the work of "durability" turned out to be not "start recording" but "actually trust and use what we already recorded."

And that surfaced the subtler, more important insight: you can't blindly trust the record. A saved note might say "this piece was finished and merged," but in the meantime that merge could have been rolled back. So resuming safely isn't just reading the file it's reconciling the record against reality. Before the factory treats any piece as done, it now checks the project's actual version history to confirm the finished work is genuinely still there. If the record and reality disagree, reality wins and the piece is rebuilt. Trust, but verify.

What we resolved

• The factory now has durable memory. Task state and the full provenance trail are written to a real on-disk database as the run progresses the first time these records survive the process that created them.
• It can resume after a crash. A restarted run reads its own history, confirms what's genuinely finished against the project's version history, skips exactly that, cleans up any half-built leftovers, and continues.
• No new dependency, no new operational burden the whole thing runs on machinery already present, behind a boundary that leaves the door open to something heavier later.

How we proved it

This is the part that matters, because "it should resume" is not proof.

We didn't simulate a crash politely. We ran a real, full multi-wave build and, partway through after some pieces had merged but with waves still unstarted and one piece caught half-built we kill -9'd the whole thing (the most brutal, no-cleanup way to kill a program). Then we restarted it in resume mode.

What it did, verifiably:

1. Recognised the finished work it checked the already-merged pieces against the project's actual git history and accepted them as done.
2. Spent nothing redoing them not one AI coding session was launched for the completed pieces. Zero re-spend. This is the whole ballgame.
3. Cleaned up the crash debris the one piece left half-built when the process died was detected and cleared away, not left to corrupt the resumed run.
4. Finished the job the remaining, never-started work ran through to completion, and the final record showed the right pieces, once each, no duplicates.

That is the exit criterion the phase set for itself, met on a real run rather than a unit test. The factory survived the worst kind of interruption and continued as if it had never happened without paying twice.

What we're carrying forward (the honest caveats)

The verdict is "yes," but we're clear about the edges:

1. One machine for now. The durable foundation is single-machine, which is the right scope today. Spreading the work across many machines is a later step and, by design, one that slots in behind the same boundary without a rewrite.
2. Smarter automatic retries are still to come. Cleanly resuming after a crash is done; a richer policy for automatically retrying individual flaky steps with backoff is a scoped follow-up, not required for the crash-safety guarantee.
3. Tamper-proof audit records are a later hardening step. The provenance trail is now durable and append-only, which is the substance of an audit trail; the full cryptographic "cannot be altered even by an insider" guarantee is a dedicated later task.

None of these change the answer. They're the difference between "survives crashes on one machine" (true today) and "survives anything at any scale" (the road ahead).

Bottom line for a decision-maker

• The risky keystone is placed. The factory can run long, unattended jobs and survive a crash without losing work proven by killing a real run dead and watching it recover.
• It doesn't pay twice. Completed work is recognised and skipped on resume; re-spend was measured at zero.
• We kept it cheap and reversible. No heavyweight new system, no lock-in, and a clean boundary that leaves room to scale up later.
• The plan holds. With durability in place, the factory was cleared to take on the work that depends on surviving long horizons and, in the phases since, the requirements-oracle work of Phase 3.