What Is the Significance of Guard-Metric Non-Inferiority?

The eval loop makes the factory measurably better every run. But better at what? If you optimize for a single metric, the system will find ways to maximize that metric at the expense of everything else. Guard-metric non-inferiority is the safety condition that prevents that. It is the factory's answer to Goodhart's Law applied to automated improvement.

The Problem With Single-Metric Optimization

A component that is being optimized against a single target metric has one incentive: maximize that number. If the eval loop scores critics on catch rate alone, the next version will optimize catch rate. That sounds good until the component discovers it can increase catch rate by lowering its precision threshold and flagging everything. Catch rate goes up. Practical value goes down. This is not a hypothetical failure mode. Every automated optimization system faces it. The question is whether the architecture acknowledges the problem and builds a defense. A single-metric promotion pipeline is an invitation to game the eval set.

How Guard Metrics Work

Guard metrics are the constraints on the optimization. They are the signals that the factory cares about but is not directly optimizing. Precision is a guard metric when catch rate is the target. Response latency is a guard metric when throughput is the target. The factory tracks guard metrics alongside the target metric during every eval run. The target metric must strictly improve. The guard metrics can only degrade within a configured tolerance. A candidate that improves catch rate by 5% is only promotable if precision did not drop more than, say, 5%. The guard tolerances are set per metric in the pipeline configuration. They encode the factory's willingness to trade one property for another.

The Non-Inferiority Test

The non-inferiority test is a statistical check. It compares the candidate guard-metric distribution against the incumbent distribution. The test asks: is the candidate worse than the incumbent by more than the allowed tolerance? If the answer is yes for any guard metric, the candidate fails. The test uses paired comparisons from the same eval cases. This avoids conflating candidate quality with eval-case difficulty. If the guard metric is precision and the tolerance is 0.05, the candidate must not show a statistically significant precision drop of more than 5 percentage points. The reference implementation uses mean comparison. The production version, tracked as DECISION[D-012], will use proper paired statistical tests with significance thresholds.

Why This Prevents Goodhart Effects

Goodhart's Law says: when a measure becomes a target, it ceases to be a good measure. In a software factory, the eval corpus is the measure and component performance is the target. Without guard metrics, a component can exploit the eval corpus by trading away unmeasured properties. The eval loop will faithfully report improvement. The production deployment will fail. Guard-metric non-inferiority closes this gap by measuring the properties being traded away. The component cannot sacrifice safety for speed or quality for quantity because those sacrifices are measured and gated. The defense is not optional. It is part of the PromotionPipeline contract. Every promotion runs through it.

Configuration and Tolerance

Guard tolerances are set in the PromotionPipeline configuration. Each guard metric has its own tolerance value. A tolerance of 0.05 on precision means the candidate precision can drop by at most 5 percentage points. A tolerance of 0.10 on recall means recall can drop by at most 10 points. The tolerances encode engineering judgment. A safety-critical guard metric might have zero tolerance. A performance metric might tolerate a small regression in exchange for a large gain on the target. The factory also supports asymmetric tolerances: a guard metric might tolerate a drop of 5% but not tolerate any improvement, if improvement in that guard metric would indicate gaming in the other direction. The configuration is explicit. Every tolerance is visible in the pipeline definition.