What Is the Purpose of Goodhart Defenses in AI and Agentic Loop Coding?

Goodhart's Law is not a theoretical concern when an automated improver can see its own eval set. It is an engineering constraint that the architecture must handle explicitly. A component that is repeatedly evaluated against the same corpus will eventually overfit. The improvements will be real on the eval set and imaginary in production. Goodhart defenses are the mechanisms that prevent this. They split the corpus, detect overfitting, and refresh stale cases.

Goodhart's Law in the Factory

The factory eval loop scores candidate components against a corpus of eval cases harvested from production incidents. This is the measure. The component is optimized against this measure. Over time, the component will discover patterns in the corpus that correlate with high scores but do not generalize. This is Goodhart's Law in action. The measure ceases to be a good measure because it became the target. The factory addresses this at two levels. Guard-metric non-inferiority prevents the component from trading away unmeasured properties. Goodhart defenses prevent the component from overfitting to the eval corpus itself.

Hold-Out Validation

Hold-out validation splits the eval corpus into two sets: a training set (80%) and a hold-out set (20%). The training set is used for the paired t-test that determines whether the candidate beats the incumbent on the target metric. The hold-out set is sealed. It is never used during the training evaluation. Only after the candidate passes the training test does the factory score it against the hold-out set. If the gap between training performance and hold-out performance exceeds 10%, the candidate is rejected for overfitting. The threshold is configurable. A candidate that scores 0.95 on training but 0.82 on hold-out shows a 13% gap. It is rejected. A candidate that scores 0.95 on training and 0.88 on hold-out shows a 7% gap. It passes.

Corpus Refresh

Hold-out validation detects overfitting to the current corpus. Corpus refresh prevents the corpus itself from becoming stale. Each eval case tracks its creation date and the last time it contributed to a promotion. A refresh interval, defaulting to 30 days, determines when a case is rotated out. When a case reaches its refresh date, it is removed from the next eval run. New cases are harvested from recent production incidents. This ensures the corpus reflects the current threat landscape and codebase state. A component that optimized against a six-month-old set of regression cases will find those cases gone. It must generalize to stay competitive. There is a tension here. A frequently refreshed corpus makes overfitting harder but also makes longitudinal comparisons more difficult. The factory addresses this through the statistical gate, which compares candidate and incumbent on the same corpus at the same time.

Integration With the Promotion Pipeline

The HoldOutHarness wraps the StatisticalHarness inside the PromotionPipeline. The pipeline runs the paired t-test on the training set. If the candidate passes and validate_holdout is enabled, the pipeline scores the candidate on the hold-out set. The overfitting check runs automatically. No manual step is required. The CorpusRefresh tracker is initialized alongside the pipeline. It monitors promotion events and signals when a case should be refreshed. The integration is layered: statistical significance first, then hold-out validation, then corpus refresh. Each layer catches a different failure mode.

Why This Matters for Self-Improving Systems

A self-improving software factory without Goodhart defenses is a system that will eventually fool itself. The eval loop will report improving scores. The production system will stagnate or regress. The disconnect is invisible because the eval set becomes a closed world. The defenses break the closed world. Hold-out validation introduces an unseen test set. Corpus refresh introduces time pressure. The component must improve on cases it has not seen before, against a corpus that evolves. This is the difference between a system that memorizes and a system that learns. The factory is designed for the latter.