Analysis of residuals in contingency tables: another nail in the coffin of conditional approaches to significance testing

Behav Res Methods. 2015 Mar;47(1):147-61. doi: 10.3758/s13428-014-0472-0.

Abstract

Omnibus tests of significance in contingency tables use statistics of the chi-square type. When the null is rejected, residual analyses are conducted to identify cells in which observed frequencies differ significantly from expected frequencies. Residual analyses are thus conditioned on a significant omnibus test. Conditional approaches have been shown to substantially alter type I error rates in cases involving t tests conditional on the results of a test of equality of variances, or tests of regression coefficients conditional on the results of tests of heteroscedasticity. We show that residual analyses conditional on a significant omnibus test are also affected by this problem, yielding type I error rates that can be up to 6 times larger than nominal rates, depending on the size of the table and the form of the marginal distributions. We explored several unconditional approaches in search for a method that maintains the nominal type I error rate and found out that a bootstrap correction for multiple testing achieved this goal. The validity of this approach is documented for two-way contingency tables in the contexts of tests of independence, tests of homogeneity, and fitting psychometric functions. Computer code in MATLAB and R to conduct these analyses is provided as Supplementary Material.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Biometry
  • Chi-Square Distribution*
  • Computing Methodologies*
  • Humans
  • Multivariate Analysis*
  • Psychometrics / methods*
  • Reproducibility of Results
  • Systems Analysis