Problems With Reliability of Speech Variables for Use in Quality Registries for Cleft Lip and Palate-Experiences From the Swedish Cleft Lip and Palate Registry

Abstract

Objective: To test the reliability of the speech data in the Swedish quality registry for cleft lip and palate.

Design: Retrospective study.

Setting: Primary care university hospital.

Participants: Ninety-four children born with cleft palate with or without cleft lip between 2005 and 2009 who had been assessed and registered in the quality registry at the age of 5 years.

Main outcome measures: Data in the registry on percent oral consonants correct, percent oral errors, percent nonoral errors, perceived velopharyngeal function (PVPF), and intelligibility were compared with results based on reassessments by 3 independent raters from audio recordings. Agreement was calculated by the intraclass correlation coefficient (ICC), quadratic weighted kappa, and percentage agreement.

Results: Absolute agreement calculated by average measures ICC for percent oral consonants correct, percent oral errors, and percent nonoral errors was above >0.90. Single measures ICC for percent oral consonants correct was 0.82, for percent oral errors 0.69, and for percent nonoral errors 0.83. The kappa coefficient for PVPF was 0.5 to 0.59 and for intelligibility 0.65 to 0.77. Exact percentage agreement for PVPF was 33% and for intelligibility 47.8%.

Conclusions: The data on oral consonants correct and nonoral errors in the quality registry seem to be reliable. The data on oral errors, PVPF, and intelligibility should be interpreted with caution. If differences among treatment centers are detected, one should go back and examine the collected raw data before drawing any definitive conclusions about treatment outcome.

Keywords: cleft lip and palate; quality registry; reliability; speech evaluation; validity.