Background: The contrasting groups' standard setting method is commonly used for consequences analysis in validity studies for performance in medicine and surgery. The method identifies a pass/fail cut-off score, from which it is possible to determine false positives and false negatives based on observed numbers in each group. Since groups in validity studies are often small, e.g., due to a limited number of experts, these analyses are sensitive to outliers on the normal distribution curve.
Methods: We propose that these shortcomings can be addressed in a simple manner using the cumulative distribution function.
Results: We demonstrate considerable absolute differences between the observed false positives/negatives and the theoretical false positives/negatives. In addition, several important examples are given.
Conclusions: We propose that a better reporting strategy is to report theoretical false positives and false negatives together with the observed false positives and negatives, and we have developed an Excel sheet to facilitate such calculations.
Trial registration: Not relevant.
Keywords: Contrasting groups; False negatives; False positives; Medical education; Messick’s validity framework; Standard setting.