What Is Hermetic Replay in a Software Factory?

The factory improves itself. It harvests escaped defects as regression cases, trains new critics, and promotes better components through shadow, canary, and live. But there is a problem that breaks all of this if left unsolved: environmental drift. If the repo changed between when you scored the incumbent and when you score the candidate, how do you know the score difference came from the component and not from the environment? Hermetic replay is the answer. It freezes the entire evaluation context at task time and re-runs against that exact snapshot.

The Drift Problem

Without hermetic replay, every eval comparison is suspect. A dependency gets upgraded. A test suite gains coverage. A deployment pipeline changes its behavior. Any of these can make a new component look better or worse than the incumbent, even if the component itself did not change. The score delta is real. The cause is unknown. That is noise, not signal.

The factory solves this by capturing an EvalSnapshot at task time: the repo SHA, the input artifacts, and the policy pack versions that were in force. When a new component needs scoring, the replay harness checks out that exact SHA in a fresh worktree, re-runs the critic, and captures new evidence. The repo, the inputs, and the policies are identical in both runs. The only variable is the component under test.

How Replay Works

The ReplayHarness orchestrates this. It takes an EvalSnapshot and an eval case, checks out the frozen commit via git worktree, re-runs the critic if one is registered, and validates determinism by comparing evidence fingerprints. The same snapshot can be replayed a thousand times and produce identical verdicts. Determinism is not assumed. It is checked, via canonical serialization, every single time.

This is not containerized replay. It uses local git worktrees with bytecode purging and pinned toolchain profiles. Containers are a future enhancement. The key property works today: frozen repo state plus controlled execution produces reproducible evidence.

What Can Be Replayed

Decision components are the primary target: critics and risk-scorers. These produce deterministic output from frozen input. Gates are deterministic by design. All three share the property that the same input always produces the same verdict. Same code, same test suite, same verdict. Code-writing builders are not replayed deterministically, because code generation is inherently non-deterministic. Builders are scored statistically instead, via paired significance tests across multiple seeds and runs.

This boundary matters. Hermetic replay is not a universal tool. It is a precision instrument for the parts of the factory that should be deterministic, making their evaluation trustworthy enough to base promotion decisions on.

Why It Is the Hard Prerequisite

The decisions register calls D-011 the biggest ticket, and for good reason. Without hermetic replay, the entire eval-driven improvement loop is built on unreliable comparisons. The factory cannot tell if a component is actually better or if the environment just shifted. The promotion.py module deliberately raises NotImplementedError at every hook that would depend on replay, rather than building on an unproven substrate. The sequence is mandated: D-011 first, then D-012 statistical gates, then D-013 Goodhart defenses. Never reversed.

Hermetic replay is the foundation that makes everything else in the eval loop trustworthy. Without it, the factory is guessing.