What Is Risk Scorer by Calibration in Software Factories?

The risk-band routing system needs a risk scorer. The question is how to judge whether a risk scorer is any good. The naive approach is to check whether individual predictions match individual outcomes. Did this change turn out to be risky? That approach fails because risk is probabilistic. A well-calibrated risk scorer might predict a 30% chance of incident for a change, and the change might turn out fine. That does not mean the scorer was wrong. It means the 30% probability landed in the 70% safe zone. Calibration-based evaluation solves this.

Why Accuracy Is Not Enough

Accuracy asks: did this specific prediction match this specific outcome? For a risk scorer producing probability estimates, accuracy is the wrong question. A change predicted at 20% risk that turns out safe was correct in one sense. A change predicted at 80% risk that turns out safe was wrong. But both probabilities were valid statements about the scorer's belief. The difference is confidence. A risk scorer that always predicts 50% for everything will be accurate about half the time but provides no useful signal. A risk scorer that predicts 90% for high-risk changes and 10% for low-risk changes is useful even if individual predictions sometimes miss. Accuracy cannot distinguish these two. Calibration can.

What Calibration Means

Calibration measures whether a scorer's stated probabilities match observed frequencies over many predictions. If a scorer predicts 30% risk for 100 changes, and 30 of those changes actually result in incidents, the scorer is perfectly calibrated at the 30% level. If only 10 result in incidents, the scorer is overconfident. If 50 result in incidents, the scorer is underconfident. Calibration is a property of the scorer's probability statements as a distribution, not as individual points. It answers the question: when this scorer says X%, should we expect X% of those cases to go wrong?

Brier Score as the Measure

The Brier score is the mean squared error between the predicted probability and the actual binary outcome. For each prediction, subtract the actual outcome (0 or 1) from the predicted probability (0.0 to 1.0), square the difference, and average across all predictions. A perfectly calibrated scorer with infinitely many predictions would approach a Brier score of 0. A scorer that always predicts 50% would achieve a Brier score of 0.25 on balanced outcomes. A scorer that is confidently and consistently wrong would score worse. The Brier score is a proper scoring rule. It cannot be gamed by hedging or exaggerating. The only way to improve your Brier score is to be better calibrated.

How Calibration Feeds Risk-Band Routing

The factory's risk-band routing uses discrete bands: low, medium, high. The risk scorer produces a continuous probability estimate that gets mapped to a band. If the scorer is well-calibrated, the band mapping is trustworthy. Changes in the 10-20% probability range map to low. Changes in the 40-60% range map to medium. Changes above 80% map to high. The Brier score gives the factory a way to validate that the mapping is working. A rising Brier score means the scorer is becoming less reliable. That triggers investigation or replacement of the scorer component. The band thresholds can also be tuned against the scorer's calibration curve, answering: at what predicted probability does the factory start seeing real incidents?

The Danger of Confident Wrongness

The most dangerous risk scorer is not one that is inaccurate. It is one that is confidently wrong. A scorer that predicts 90% risk for every change that turns out safe is worse than useless. It creates a false sense of security when it says low risk and desensitises the team when it says high risk. Calibration-based evaluation catches this. The Brier score will be high because the predictions are far from the outcomes. The scorer will be flagged for replacement. Without calibration, the scorer could produce plausible-looking risk bands that are completely detached from reality. The factory would route changes based on noise. This is why the risk scorer by calibration is tracked as DECISION[D-007]: it is the hardest and most important component to get right.