Confidence Calibration: An Introduction With Application to Quality Improvement

Abstract

A probabilistic forecast is one that assigns a probability (or likelihood) to the occurrence of an event. Radiologists commonly make probabilistic judgments in their reports, even if these predictions are not explicitly expressed as numbers. There are calls for radiologists to commit to their probabilistic predictions in a standardized fashion; however, without a mechanism for feedback, there is no opportunity for improvement. Analysis techniques familiar to radiologists (eg, calculation of sensitivity and specificity and construction of receiver operating characteristics curves) have a blind spot with regard to calibration of these probabilities to reality and are the main obstacle to improvement along this axis. We review statistical and graphical methods for calibration analysis in wider use outside the medical literature and present a framework for implementation of these techniques for quality improvement and radiologist self-assessment.

Keywords: Brier score; confidence calibration; physician judgment.