Phase 1: Can our automated software factory safely change real software?

We set out to answer a single make-or-break question: can our automated "software factory" safely modify software that already exists not just build brand-new toy apps? We tested it on a real, widely-used open-source program, gave it three genuine improvement tasks of increasing difficulty, and it completed all three correctly, on the first try, for about 55 cents total, without breaking anything. Crucially, we also proved that its main safety net actually works it catches mistakes rather than just claiming to. Along the way we caught one of our own safety nets silently switched off, fixed it, and only then trusted the results. Verdict: it works. Proceed with three specific improvements folded into the next phase. (For the full technical breakdown of methodology and metrics, see the detailed Phase 1 report.)

What we set out to solve

Think of the factory as a robot software engineer. Until now, we'd only proven it could build new programs from a blank page. That's the easy case.

The hard case and the one that actually matters for a business is changing software that already exists. Real companies rarely start from scratch; they have millions of lines of existing code, and the job is to evolve it without breaking the parts that already work. That's genuinely dangerous: a change can pass all its own tests while quietly breaking something elsewhere that nobody wrote a test for.

So Phase 1 asked the go/no-go question: can the robot safely change existing code, with enough guardrails that we'd trust it to work unsupervised? If the answer were no, the whole product strategy would need to change direction. Every later phase was on hold until we knew.

How we tested it (in real-world terms)

We didn't test on our own code that would be grading our own homework. We picked a real, popular open-source program (a widely-used tool called "click," downloaded millions of times, with tens of thousands of lines of code) and gave the robot three real jobs, chosen to escalate the difficulty:

1. The easy one add a brand-new small feature. (Does it add without breaking?)
2. The tricky one rewrite the internals of an existing feature while keeping its behavior exactly the same. (This is the dangerous case the "silently breaks something" trap.)
3. The hard one a change that ripples across multiple files and affects other parts of the program. (Does it understand the blast radius?)

For each job we put a hard spending cap on the robot (like a prepaid debit card that stops working at the limit), so a runaway process couldn't burn through budget overnight.

What we learned along the way

The uncomfortable discovery: before we could trust any result, we found that one of our most important safety nets the one specifically designed to catch the "silently breaks something" trap was switched off without anyone noticing. The system's own documentation claimed it was running. It wasn't.

This is a classic Goodhart-defenses failure: the metric (the rail's docstring claiming it's active) had decoupled from the reality (the code never calling it). We caught it, switched it on properly, and then ran the real tests. This was the single most valuable lesson of Phase 1: a safety net you haven't verified is running is the same as having no safety net and our reporting was confidently telling us it was fine. (It's now on our list to make any switched-off safety net raise an alarm instead of staying quiet.)

What we resolved

With the safety net genuinely switched on, all three jobs succeeded:

• Nothing broke. The open-source program's own test suite over 1,600 independent checks kept passing throughout, and grew by 19 new checks the robot wrote itself.
• It stayed cheap. Total cost was about 55 cents, roughly 2% of the budget we'd allowed. The spending cap was never close to being hit.
• No human had to step in. Each job went from request to finished, verified, merged change with no manual correction.

How we proved we actually resolved it

This is the part that matters most, because "it seemed to work" is not proof.

The dangerous job was #2: rewrite a feature's internals while keeping its behavior identical. Here's how we proved the safety net is real and not just decorative:

1. Before the robot touched anything, we took a precise "oracle-strength snapshot" of how the existing feature behaved across a range of inputs including the tricky edge cases.
2. The robot then rewrote the internals. Its new version had to match every single snapshot, exactly, or the change would be automatically rejected. It matched.
3. Then we deliberately sabotaged the finished code with a subtle bug the kind of one-character mistake a tired engineer might make. The safety net immediately caught it and flagged the change as broken. We undid the sabotage and it went green again.

That third step is the proof. It shows the safety net doesn't just rubber-stamp whatever the robot produces it genuinely catches real mistakes, including the subtle, silent kind that are the whole reason this is hard. The guardrail has teeth.

What we're improving next (the honest caveats)

The verdict is "proceed," but we're carrying three specific fixes into the next phase rather than pretending it's perfect:

1. Keep the receipts. The factory currently does rigorous safety checks but then throws away the detailed evidence, keeping only a summary. For a system whose whole selling point is auditability, we need to permanently record the proof of every check.
2. Make the strongest safety net automatic, not opt-in. In the hardest job, that behavior-snapshot net had to be aimed manually. We want it to protect everything by default, so a gap can't be left open by omission.
3. Make silence impossible. The switched-off-safety-net problem must never recur quietly a disabled guardrail should set off an alarm, not pass unnoticed.

None of these change the answer. They make the "yes" trustworthy enough to leave running unattended, which is the real bar.

Bottom line for a decision-maker

• The risky question is answered: yes. The factory can safely change real, existing software, and we proved the safety nets have teeth.
• It's cheap and fast. Three real changes to a real program for pocket change, no human babysitting.
• We're honest about the gaps. We found and fixed a real hole in our own safety story, and we're carrying three concrete improvements forward rather than declaring victory.
• The strategy holds. No pivot needed. Phase 2 making the factory survive crashes and long overnight runs so it can operate as a real service is now cleared to begin.