The problem of measurement bias in comparing selected subgroups

Abstract

Estimates of subgroup differences are routinely used as part of a comprehensive validation system, and these estimates serve a critical role, including evaluating adverse impact. Unfortunately, under direct range restriction, a selected mean ( ${\hat{\mu }}_t^{\text{'}}$ ) is a biased estimator of the population mean ${\mu }_x$ as well as the selected true score mean ${\mu }_t^{\text{'}}$ . This is due partly to measurement bias. This bias, as we show, is a factor of the selection ratio, the reliability of the measure, and the variance of the distribution. This measurement bias renders a subgroup comparison questionable when the subgroups have different selection ratios. The selected subgroup comparison is further complicated by the fact that the subgroup variances will be unequal in most situations where the selection ratios are not equal. We address these problems and present a corrected estimate of the mean difference, as well as an estimate of Cohen's d* that estimates the true score difference between two selected populations, $({\mu }_{\mathrm{tA}}^{\text{'}}-{\mu }_{\mathrm{tB}}^{\text{'}})/{\sigma }_t$ . In addition, we show that the measurement bias is not present under indirect range restriction. Thus, the observed selected mean ${\hat{\mu }}_t^{\text{'}}$ is an unbiased estimator of selected true score mean ${\mu }_{\mathrm{ty}}^{\text{'}}$ . However, it is not an unbiased estimator of the population mean ${\mu }_y$ . These results have important implications for selection research, particularly when validating instruments.

Keywords: Cohen’s d; measurement bias; subgroup difference.