How Eval and Promotion: Shadow, Canary Work in a Software Factory

Every factory component, whether it is a new prompt bundle, a model routing rule, an adapter image, or a policy pack, goes through the same graduated promotion pipeline before it touches production traffic. The pipeline has four mandatory stages: offline eval, shadow, canary, and live. Each stage is gated by statistical significance checks and guard-metric non-inferiority conditions. Nothing skips a stage. This is how the factory safely improves itself over time.

The Promotion Pipeline

Every promotable thing in the factory is modeled as a ComponentVersion. The pipeline definition (core/promotion.py) enforces strict ordering: candidate to offline eval to shadow to canary to live. A component cannot advance to the next stage until the current stage passes its gate conditions. If a stage fails, the component is rejected or falls back to the incumbent.

The pipeline is designed around the flywheel described in the factory's eval loop: production incidents are harvested as eval cases with frozen snapshots, candidate components are scored against those frozen cases, and the gate decides whether they can advance. Each escaped defect becomes a permanent regression test. The factory gets better over time because every failure feeds back into the eval corpus. This is the universal adoption mechanism that the factory uses to continuously improve, and the promotion pipeline is the runtime that executes it.

Offline Eval: Scoring Against History

The first stage is offline eval. The candidate component is scored against a held-out corpus of historical eval cases. This is completely offline. No production traffic touches the candidate. No users are affected. The candidate's performance is compared to the incumbent's performance on the same corpus.

Offline eval uses the factory's hermetic replay infrastructure (DECISION[D-011]) to produce deterministic results. The frozen snapshot of code, artifacts, and policy versions is checked out, the critic and scorer are re-run, and the new evidence is compared against the historical baseline. Without hermetic replay, offline eval would be noise: you could not tell whether a score change came from the component change or from environmental drift.

The statistical harness (StatisticalHarness) implements a paired t-test with alpha set to 0.05. The candidate must beat the incumbent with statistical significance. If the p-value is 0.05 or higher, the candidate stays with the incumbent. The gate is conservative by design: fail-open means stay on what is already working.

Shadow: Watching Without Affecting

Offline eval is not enough. A held-out corpus can only test against what you already know. Shadow mode puts the candidate alongside the live component on real production traffic, but the candidate's output is never acted on. It runs in parallel, purely observational.

Shadow mode is the enter_shadow method in PromotionPipeline. The incumbent handles every request as normal. The candidate processes the same requests on the side. The factory compares the two sets of outputs: latency, quality scores, safety metrics, any observable difference. If the candidate's guard metrics degrade relative to the incumbent, the gate catches it.

The key here is that shadow mode reveals issues that offline eval could not surface: unexpected input distributions, performance regressions under real load, and interactions with other live components. The candidate sees real traffic but does not affect any user. This is where the factory discovers that a change that looked good on offline eval has problems in the wild.

Canary: Small Risk, Real Traffic

If the candidate passes shadow mode, it enters canary. Canary is the first stage where the candidate actually handles real user requests. But only a small fraction of traffic goes through it. The canary_traffic_pct parameter controls the percentage.

The canary runs the candidate on a small, controlled slice of traffic. The factory monitors the same guard metrics as in shadow mode, but now the metrics reflect real outcomes, not just observational comparisons. A component in canary that degrades latency, increases error rates, or produces lower-quality outputs is detected and stopped immediately.

Canary exists because there is no substitute for real outcomes. Shadow mode can measure observational differences, but only canary measures the actual effect of the candidate's decisions on real work. The small traffic percentage limits blast radius. If the candidate fails, only a tiny fraction of users or tasks are affected, and the factory rolls back to the incumbent.

Live: Full Promotion

The final stage is live. The candidate becomes the new incumbent. All traffic goes through it. The old incumbent is retired or kept as a fallback for rollback scenarios.

Live does not mean the component is done being evaluated. The factory continues to monitor guard metrics on the live component. If performance degrades over time (due to data drift, model staleness, or other factors), the component may be demoted and a new candidate cycle begins. The pipeline is not a one-way door. The factory can always promote a better candidate and demote a weakening incumbent.

The Gate at Every Stage

Every stage transition is gated by two conditions: statistical significance and guard-metric non-inferiority.

Statistical significance means the candidate's target metric must improve over the incumbent with enough evidence to be confident the improvement is real, not random. The StatisticalHarness uses a paired t-test for this, but the decision (DECISION[D-012]) notes that production-grade statistical rigor may eventually require additional methods: paired comparisons, multiple random seeds, and significance thresholds tuned to the domain.

Guard-metric non-inferiority means every guard metric (metrics that are not being directly optimized but must not be sacrificed) is allowed to drop by at most a configured tolerance. A component cannot win promotion by trading away safety or quality to inflate the one metric being optimized. The guard tolerances are defined per component version and enforced by the pipeline before any stage advance. This prevents Goodhart's Law from operating at the pipeline level: you cannot game the eval by targeting a single metric while letting everything else slide.

For an in-depth look at how the factory measures the quality of its evaluation decisions, see What Is a Test Oracle in an Autonomous Software Factory? and What Is an Oracle-Strength Floor?.