Reliable support: Measuring calibration of likelihood ratios

Abstract

Calculation of likelihood ratios (LR) in evidence evaluation still presents major challenges in many forensic disciplines: for instance, an incorrect selection of databases, a bad choice of statistical models, low quantity and bad quality of the evidence are factors that may lead to likelihood ratios supporting the wrong proposition in a given case. However, measuring performance of LR values is not straightforward, and adequate metrics should be defined and used. With this objective, in this work we describe the concept of calibration, a property of a set of LR values. We highlight that some desirable behavior of LR values happens if they are well calibrated. Moreover, we propose a tool for representing performance, the Empirical Cross-Entropy (ECE) plot, showing that it can explicitly measure calibration of LR values. We finally describe some examples using speech evidence, where the usefulness of ECE plots and the measurement of calibration is shown.