Purpose: Auditory training has been shown to reduce rater variability in perceptual voice assessment. Because rater variability is also a central issue in the auditory-perceptual assessment of dysarthria, this study sought to determine if training produces a meaningful change in rater reliability, criterion validity, and scaling magnitude of four features: overall speech impairment, articulatory imprecision, monotony, and slow rate.
Method: Forty-four nonexperts randomized to training and nontraining listener groups completed a pretest and posttest. Only the former group underwent auditory training between pre- and posttests. For both testing and training, listeners rated samples from speakers with amyotrophic lateral sclerosis (ALS), speakers with Parkinson's disease (PD), and neurologically healthy control speakers using separate visual analog scales (VASs) for each of the four features. Intraclass correlation coefficients were used to compare inter- and intrarater reliability between pre- and posttest for both listener groups. For criterion validity, severity ratings from the two nonexpert listener groups were compared to those of two experienced listeners for all four features. To determine changes in scaling magnitude, raw VAS scores for each feature were compared from pre- to posttest within the two nonexpert listener groups. Scaling changes were also compared between the two listener groups for the pre- and posttest conditions.
Results and conclusions: In the training group, a meaningful improvement in interrater reliability was observed for some features in all three speaker groups, but not in the nontraining group. In contrast, for intrarater reliability, in the nontraining group, a meaningful improvement was observed for many features in all three speaker groups, but only for PD monotony and slow rate in the training group. All ratings from the nonexpert listeners were valid except for monotony. Raw VAS scores did not meaningfully change from pre- to posttest for any of the features, but there was a trend toward lower scores posttraining, mainly for the ALS samples. Modifications to the auditory training paradigm to further improve reliability and validity, along with future goals for optimizing training, are discussed.