Hypothesis testing procedure for binary and multi-class F1 -scores in the paired design

Abstract

In modern medicine, medical tests are used for various purposes including diagnosis, disease screening, prognosis, and risk prediction. To quantify the performance of the binary medical test, we often use sensitivity, specificity, and negative and positive predictive values as measures. Additionally, the $F_1$ -score, which is defined as the harmonic mean of precision (positive predictive value) and recall (sensitivity), has come to be used in the medical field due to its favorable characteristics. The $F_1$ -score has been extended for multi-class classification, and two types of $F_1$ -scores have been proposed for multi-class classification: a micro-averaged $F_1$ -score and a macro-averaged $F_1$ -score. The micro-averaged $F_1$ -score pools per-sample classifications across classes and then calculates the overall $F_1$ -score, whereas the macro-averaged $F_1$ -score computes an arithmetic mean of the $F_1$ -scores for each class. Additionally, Sokolova and Lapalme ${}^1$ gave an alternative definition of the macro-averaged $F_1$ -score as the harmonic mean of the arithmetic means of the precision and recall over classes. Although some statistical methods of inference for binary and multi-class $F_1$ -scores have been proposed, the methodology development of hypothesis testing procedure for them has not been fully progressing yet. Therefore, we aim to develop hypothesis testing procedure for comparing two $F_1$ -scores in paired study design based on the large sample multivariate central limit theorem.

Keywords: F 1 $$ {F}_1 $$ measures; delta-method; multi-class classification; precision; recall.