Phase 1 Go/No-Go: The Brownfield Capability Spike

Phase 1 of docs/ROADMAP.md retires the roadmap's single largest risk: can the factory reliably and safely modify an existing codebase inside its gate system at all? If it cannot, the enterprise value proposition fails and the architecture must pivot (toward 8090's own model export work orders over MCP for humans/agents in IDEs to execute) before any of the well-understood engineering money in Phases 2–7 is spent. This document records the exit criterion (T1.6) and its verdict.

Status

The Phase 1 machinery is complete and unit-tested end to end against bundled fixtures, and has now been exercised against a real, external, mid-size repository with a live model. This section records the actual runs.

One correction landed before the runs could count. The driver's docstring claimed the characterization rail (T1.5a) a test oracle that snapshots existing behaviour was active, but _execute never called the (unit-tested) characterization.py module the rail was dead in the driver. It is now wired: each change slice may author characterize: example calls, which are snapshot against the current code and committed as the pre-change baseline before any builder runs. See "Observed failure modes."

What "done" means (the exit criterion)

The factory completes three real change requests on one real external repository (~10–50k LOC, not authored by the factory) end to end with gates green: indexed, decomposed against the index, built, criticized (tests + characterization + scoped mutation), merged with the budget kill-switch and characterization rails active throughout.

Pick a real OSS Python or TypeScript project as the target. Suggested change classes (one of each exercises a different risk):

1. Additive add a new public function/endpoint to an existing module (tests the decomposer's modifies + export-preservation path).
2. Behavioural change an existing function's internals while keeping its contract (tests characterization: the preserved-export snapshot must stay green while the new behaviour lands).
3. Cross-module a change whose blast radius touches a consumer (tests impact-based test selection and the boundary critic).

How to run each change

# 0. baseline: clone the target, commit it, index it
git -C path/to/target log -1 # ensure a clean committed baseline

# 1. author a change-request PRD (factory blocks with `modifies` +
# `preserves_exports`); see tests/fixtures/brownfield_change.md for shape

# 2. dry run validates the change against the real repo, spends nothing
python demo/run_brownfield_change.py --repo path/to/target \
 --change change_1.md --arch

# 3. execute under a hard budget ceiling
python demo/run_brownfield_change.py --repo path/to/target \
 --change change_1.md --arch --budget 5.00 --execute

Each run writes a RunRecord and a JSON-lines event log under .factory/runs/; harvest the metrics below from them.

Results

Every change decomposed against the real index, snapshot its preserved behaviour (where applicable), built on the first builder attempt, passed the critic (tests + scoped mutation), and merged. Click's own full suite grew from 1660 → 1679 passing across the three merges (19 net-new tests authored by the factory) with zero pre-existing tests broken.

Target repository: pallets/click 8.4.2, Python, ~12.4k LOC under src/ (+~14k LOC of tests), commit b2e30a175449cfda909ee4fbf4a29a6a071cad53. Profile: python-src (new; src-layout, PYTHONPATH=src, ambient pytest no install/venv, so critic worktrees stay disposable).

Budget ceiling used: $5.00 per change (total headroom $15). Never hit actual total spend $0.554, ~2% of ceiling. The budget_check event fired allow on every dispatch. (The kill-switch's halt path is proven by tests/test_budget* in CI, not by these runs, which never approached the ceiling.)

Per-change notes

• Change 1 (additive) center_text() in click.formatting. The builder added a clean, correctly-documented helper and 7 unit tests; existing exports untouched. Three wrap_text behaviour pins rode along green. Scoped mutation on formatting.py (verified separately): kill rate 0.67 over 6 mutants meaningful evidence that stays tractable because it mutates only the changed file, not all of click. Nothing surprising; the simplest class, as intended.
• Change 2 (behavioural) refactor make_default_short_help internals. This is the flagship result. The builder genuinely restructured the word-accumulation/truncation loop (explicit result_words accumulator replacing the index-rewind) while keeping the contract bit-identical. Six committed characterization pins spanning the empty, short, over-length, sentence-end, custom-max_length, and -marker paths all stayed green. The rail was proven non-vacuous: injecting a ..-vs-... regression into the merged code trips 3 of the 6 pins; reverting restores green. This is exactly the enterprise failure the greenfield pipeline never sees (a change that passes its own tests while moving an unwritten invariant), converted into gate evidence.
• Change 3 (cross-module) CommaSeparated param type, re-exported from the package root. Touched types.py + __init__.py + tests; the impact selector pulled 14 test files whose transitive imports intersect the changed modules, and the derived-architecture boundary critic enforced the component graph. The builder followed click's existing from .types import X as X convention and ParamType idioms (including __repr__), added 7 tests; from click import CommaSeparated resolves and converts correctly. The driver correctly flagged that its 29 preserved exports carry no characterize: calls and are therefore unpinned (see failure modes) a real observability win, surfaced rather than hidden.

Observed failure modes

No defect merged. No change produced wrong, unsafe, or unmergeable code. The issues below are gaps the exercise surfaced, not gate escapes:

1. Dead characterization rail (caught before the runs, now fixed). The driver never invoked the characterization module despite its docstring the T1.5a rail would have been silently absent from the "runs with rails active." Fixed by wiring pin_baseline into _execute. Category: the harness lied about its own coverage. This is the single most important finding: without it, change 2's headline result would have been vacuous.
2. preserves_exports without characterize: is unpinned (surfaced, not caught). Change 3 declared 29 preserved exports but supplied no example calls, so none of them were behaviourally pinned only the reality/existence and import-boundary checks applied. The driver prints a note naming the unpinned exports, which is the honest behaviour, but a stronger design would auto-generate representative inputs (the LLM-assisted characterization path the T1.5a module docstring defers). Category: rail is opt-in per export.
3. Mutation kill-rate is computed but not persisted (observability gap). The critic runs scoped mutation and puts the kill rate in its in-memory evidence bundle, but the RunRecord stores only summary fields so the number doesn't survive to the run record and had to be reproduced separately for this report. Category: the factory discards evidence of its own rigor the opposite of provenance in factory design exactly the gap roadmap §Phase 2 / T2.2 (durable store) and T2.5 (observability) address.
4. Characterization is Python-only (known Phase 1 scope). The rails don't yet cover the TypeScript profile; the earlier bulletproof-react brownfield run (PR #12) therefore exercised decomposition + build + e2e but not characterization/mutation. A fully-rails-active TS run is future work.

Recommendation & Decision

Drafted recommendation: PROCEED with adjustments.

The go/no-go question can the factory reliably and safely modify an existing codebase inside its gate system? is answered yes on the evidence: three change requests of increasing blast radius, on a real 12k-LOC external repo, each merged first-try with gates green, no defect escaping, at trivial cost ($0.55 total), with the characterization rail demonstrably catching an injected regression. The architecture does not need to pivot to work-order export.

But three adjustments should be folded into Phase 2's scope before the factory is trusted unattended on brownfield work each is an evidence/rigor gap the runs surfaced, not a capability failure:

1. Persist critic evidence (mutation kill-rate, per-verb results, characterization pass/fail) into the durable store fold into T2.2. The factory currently discards the numbers that prove it did its job.
2. Strengthen the characterization rail from opt-in to default auto-derive representative inputs for preserves_exports that carry no characterize: calls (the LLM-assisted path the T1.5a module already anticipates), so a preserved export is never silently unpinned. New sub-task under Phase 3 (oracle strength, alongside T3.3/T3.4).
3. Add a harness self-check so a rail that a driver forgets to invoke fails loudly rather than passing silently (the dead-characterization bug). Cheap; fold into T2.5 observability.

• [ ] PROCEED unchanged Move to Phase 2 as written.
• [x] PROCEED with adjustments (drafted) Fold the three items above into Phase 2/3 scope, then proceed.
• [ ] PIVOT to work-order export not indicated by the evidence.

Decision: PROCEED with adjustments    Decided by: Peter Scheffer (operator)    Date: 2026-07-16

(The operator approved the drafted recommendation. Phase 2 proceeds; the three adjustments are folded into Phase 2/3 scope as listed above. For an executive summary of this phase's results, see Phase 1: Can our automated software factory safely change real software?)

What Phase 1 delivered (harness inventory)

Every item below is committed with tests; this is what the runs above exercise.

Decisions closed (see docs/DECISIONS.md / docs/ROADMAP.md §5): D-009 enforcement half (budget). Decisions advanced: the brownfield capability that Phases 2–7 build on.

Reproducing these runs

# baseline: pallets/click 8.4.2 at ~/Code/click-factory-test, on a `main` branch
git clone https://github.com/pallets/click ~/Code/click-factory-test
git -C ~/Code/click-factory-test switch -C main 8.4.2

# each change (dry-run drops --execute --budget and spends nothing):
python demo/run_brownfield_change.py --repo ~/Code/click-factory-test \
 --change docs/reports/t16/change_1_additive.md \
 --profile python-src --pkg-root src --arch \
 --budget 5.00 --execute --runs-dir /tmp/factory-t16/runs
# …change_2_behavioural.md, change_3_crossmodule.md likewise, in order

The three change-request PRDs are committed under docs/reports/t16/. Run records + JSON-lines event logs land in the --runs-dir.