Measures of the Degree of Departure from Ignorable Sample Selection

Roderick J A Little; Brady T West; Philip S Boonstra; Jingwei Hu

doi:10.1093/jssam/smz023

Measures of the Degree of Departure from Ignorable Sample Selection

J Surv Stat Methodol. 2020 Nov;8(5):932-964. doi: 10.1093/jssam/smz023. Epub 2019 Aug 29.

Authors

Roderick J A Little¹, Brady T West², Philip S Boonstra³, Jingwei Hu⁴

Affiliations

¹ Professor of Biostatistics at the School of Public Health and Research Professor in the Survey Methodology Program (SMP), Survey Research Center (SRC), Institute for Social Research (ISR), University of Michigan, 1420 Washington Heights, Ann Arbor, MI 48109-2029, USA.
² Research Associate Professor in the Survey Methodology Program (SMP), Survey Research Center (SRC), Institute for Social Research (ISR), University of Michigan, 426 Thompson Street, Ann Arbor, MI 48106-1248, USA.
³ Research Assistant Professor of Biostatistics in the School of Public Health, University of Michigan, 1415 Washington Heights, Ann Arbor, MI 48109, USA.
⁴ Survey Research Director at SurveyPlus Ltd., 1079 Nanhai Street, Shuma Building 201A, Shenzhen, Guangdong 518023, China.

Abstract

With the current focus of survey researchers on "big data" that are not selected by probability sampling, measures of the degree of potential sampling bias arising from this nonrandom selection are sorely needed. Existing indices of this degree of departure from probability sampling, like the R-indicator, are based on functions of the propensity of inclusion in the sample, estimated by modeling the inclusion probability as a function of auxiliary variables. These methods are agnostic about the relationship between the inclusion probability and survey outcomes, which is a crucial feature of the problem. We propose a simple index of degree of departure from ignorable sample selection that corrects this deficiency, which we call the standardized measure of unadjusted bias (SMUB). The index is based on normal pattern-mixture models for nonresponse applied to this sample selection problem and is grounded in the model-based framework of nonignorable selection first proposed in the context of nonresponse by Don Rubin in 1976. The index depends on an inestimable parameter that measures the deviation from selection at random, which ranges between the values zero and one. We propose the use of a central value of this parameter, 0.5, for computing a point index, and computing the values of SMUB at zero and one to provide a range of the index in a sensitivity analysis. We also provide a fully Bayesian approach for computing credible intervals for the SMUB, reflecting uncertainty in the values of all of the input parameters. The proposed methods have been implemented in R and are illustrated using real data from the National Survey of Family Growth.

Keywords: Measures of selection bias; National Survey of Family Growth; Nonignorable sample selection; Nonprobability sampling; Sampling bias.

© The Author(s) 2019. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com.