What Is an Oracle-Strength Floor in an Autonomous Factory?

A test oracle is only as good as its measured ability to catch bugs. I covered what a test oracle is in a previous post. This one is about what happens when you stop assuming your tests work and start measuring whether they actually do. That measurement is the oracle-strength floor, and it is how an autonomous factory decides if it can trust itself.

The Test Oracle Problem

An autonomous software factory only works if it can trust its test suite. Most test suites have never been tested themselves. A suite with 100% line coverage and zero real assertions looks impressive on a dashboard but catches nothing. This is problem one of seven in building a dark factory. Weak test oracles are the single biggest risk because everything else rests on them. If the foundation is weak, autonomy built on top is a gamble.

Measuring Oracle Strength

Mutation testing solves this by injecting faults into the code and checking whether the test suite catches them. Kill rate, the percentage of injected mutants the suite detects, becomes the measured oracle strength. A 95% kill rate means the suite catches 95 out of 100 real bugs of that type. This is not a hypothetical score. It is a concrete, repeatable measurement that feeds directly into the factory's governance system. The test suite stops being a black box and becomes a known quantity.

The Mutation Floor

Each code path in the factory's governance configuration carries a mutation_floor, a certified minimum kill rate. The risk_tier_gate checks the measured oracle strength against this floor every time a change moves through the pipeline. If the kill rate drops below the floor, lights-out is revoked. The change gets routed to a human. There is no second chance, no manual override. The factory measures, compares, and acts. The test that proves this works is straightforward: test_weak_oracle_loses_lights_out encodes the guarantee as a unit test in the gate engine.

Autonomy Capped at Measured Strength

This is the core principle. A path running at T0, full lights-out, has earned that tier through demonstrated test suite effectiveness. The moment that effectiveness degrades, autonomy degrades with it. A change spanning paths at different tiers inherits the strictest tier. A high-risk change routes to a human regardless of oracle strength. The system fails safe without a human watching the dashboard. Autonomy is not a feature you turn on. It is a property you measure, certify, and revoke when the evidence no longer supports it.

The Feedback Loop

When an escaped defect reaches production, the factory walks the provenance chain backwards. It traces from the deployed change to the gate that approved it, through the evidence bundle that informed the gate, to the test oracle that failed to catch it. If the oracle-strength floor caught a degradation before release, that trace becomes a learning signal. The evals corpus grows. The next candidate component faces a harder test. The floor is not a static number. It is part of a closed loop that makes the factory measurably better over time, one escaped defect at a time.

Beyond the Threshold

The oracle-strength floor is the simplest form of this pattern. It checks one number against one threshold. But the same idea extends to risk-band routing, where high-risk changes go to a human regardless of test coverage. It extends to sampled-for-audit, where statistical process control varies the sampling rate based on risk. And it extends to the eval-driven improvement loop itself, where every promoted component must prove it is statistically better than the incumbent. The pattern is the same in every case. Trust is replaced by measurement. And measurement has a floor.