Investigation Into the Effects of Using Normal Distribution Theory Methodology for Likert Scale Patient-Reported Outcome Data From Varying Underlying Distributions Including Floor/Ceiling Effects

Abstract

Objectives: Utilization of parametric or nonparametric methods for testing Likert scale data is often debated. This 2-part simulation study aims to investigate the sampling distribution of various Likert scale distributions (including floor/ceiling effects) and analyze the effectiveness of using parametric versus nonparametric tests with varying sample sizes.

Methods: We simulated populations from parametric distributions binned into Likert scales. In study 1, replicates were sampled from each distribution with sizes ranging from 5 to 150 observations, calculating means with simulated 95% CIs at each sample size. In study 2, floor/ceiling effects were introduced such that the proportion of patients responding with the lowest rating varied from approximately 40% to 90%. Two-sample tests were then conducted for the 90% floor effect distribution against all other floor distributions to determine effectiveness of parametric versus nonparametric methods via 2-sided pooled t tests and Wilcoxon rank-sum tests. Coverage of the difference in means, realized P values, relative efficiency, measures of agreement in direction, and conclusion of tests were plotted by sample size.

Results: The sampling distributions of the 1-sample means and SDs for most distributions converged quickly to Gaussian, with 95% coverage. One- and 2-sample t tests of the mean demonstrated acceptable coverage, type I error, and agreement.

Conclusions: Simulations confirm that the sampling distribution of the mean rapidly approaches normality and appropriate tests provide adequate coverage and type I error. Two-sample t tests demonstrate appropriateness and increased statistical power gained by using parametric over nonparametric approaches, suggesting t tests should be implemented with few restrictions.

Keywords: parametric versus nonparametric tests; patient-reported outcomes.