Confidence intervals and sample size planning for optimal cutpoints

Abstract

Various methods are available to determine optimal cutpoints for diagnostic measures. Unfortunately, many authors fail to report the precision at which these optimal cutpoints are being estimated and use sample sizes that are not suitable to achieve an adequate precision. The aim of the present study is to evaluate methods to estimate the variance of cutpoint estimations based on published descriptive statistics ('post-hoc') and to discuss sample size planning for estimating cutpoints. We performed a simulation study using widely-used methods to optimize the Youden index (empirical, normal, and transformed normal method) and three methods to determine confidence intervals (the delta method, the parametric bootstrap, and the nonparametric bootstrap). We found that both the delta method and the parametric bootstrap are suitable for post-hoc calculation of confidence intervals, depending on the sample size, the distribution of marker values, and the correctness of model assumptions. On average, the parametric bootstrap in combination with normal-theory-based cutpoint estimation has the best coverage. The delta method performs very well for normally distributed data, except in small samples, and is computationally more efficient. Obviously, not every combination of distributions, cutpoint optimization methods, and optimized metrics can be simulated and a lot of the literature is concerned specifically with cutpoints and confidence intervals for the Youden index. This complicates sample size planning for studies that estimate optimal cutpoints. As a practical tool, we introduce a web-application that allows for running simulations of width and coverage of confidence intervals using the percentile bootstrap with various distributions and cutpoint optimization methods.