Objective: To explore the factors that influence the stability of evaluation results judged by a jury through a standard research on perceptual evaluation measurements of voice quality.
Methods: Voice samples from 300 patients with dysphonia and 100 control subjects with normal voice were recorded and assessed by a jury composed of 6 experienced listeners from different hospitals. The voice samples were discourse voices and ordered randomly 3 times, and the mean of 3 evaluations using visual analogue scale were the final results. The jury was instructed to classify voice samples according to the G (grade), R (rough) and B (breathy) components of the GRBAS scale on a 4-point scale ranging from 0 for normal to 3 for severe dysphonia. Κ value was used to analyze the concordance of evaluation results and regression analysis was used to research the effects of the extent of voice disorder to the stability of perceptual evaluation.
Results: The discordance of evaluation existed both between the jury and in listeners themselves. The concordance of listeners themselves of each evaluation parameter was not bad, good, or even very good, and the concordance of evaluation of G was the best (κ value: 0.46 - 0.85), then R (κ value: 0.41 - 0.84) and B (κ value: 0.41 - 0.81). The concordance between the jury was worse than that in themselves. And except a listener whose concordance of evaluation was under the requirement, the concordance of evaluation of G was the best (κ value: 0.43 - 0.96), then R (κ value: 0.33 - 0.78) and B (κ value: 0.002 - 0.45). The stability of evaluation of normal voice and severe voice disorder was better than mild and moderate voice disorder.
Conclusions: The discordance between the jury was the main factor that influence the stability of perceptual evaluation. The evaluation parameters and extent of voice disorder will influence the stability of perceptual evaluation of the jury.