Inter-rater reliability of the diagnosis of otitis media based on otoscopic images and wideband tympanometry measurements

Abstract

Objectives: This study aims to investigate the inter-rater reliability and agreement of the diagnosis of otitis media with effusion, acute otitis media, and no effusion cases based on an otoscopy image and in some cases an additional wideband tympanometry measurement of the patient.

Methods: 1409 cases were examined and diagnosed by an otolaryngologist in the clinic, and otoscopy examination and wideband tympanometry (WBT) measurement were conducted. Afterwards, four otolaryngologists (Ear, Nose, and Throat doctors, ENTs), who did not perform the acute examination of the patients, evaluated the otoscopy images and WBT measurements results for diagnosis (acute otitis media, otitis media with effusion, or no effusion). They also specified their diagnostic certainty for each case, and reported whether they used the image, wideband tympanometry, or both, for diagnosis.

Results: All four ENTs agreed on the diagnosis in 57% of the cases, with a pairwise agreement of 74%, and a Light's Kappa of 0.58. There are, however, large differences in agreement and certainty between the three diagnoses. Acute otitis media yields the highest agreement (77% between all four ENTs) and certainty (0.90), while no effusion shows much lower agreement and certainty (34% and 0.58, respectively). There is a positive correlation between certainty and agreement between the ENTs across all cases, and both certainty and agreement increase for cases where a WBT measurement is shown in addition to the otoscopy image.

Conclusions: The inter-rater reliability between four ENTs was high when diagnosing acute otitis media and lower when diagnosing otitis media with effusion. However, WBT can add valuable information to get closer to the ground-truth diagnosis without myringotomy. Furthermore, the diagnostic certainty increases when the WBT is examined together with the otoscopy image.

Keywords: Agreement; Diagnosis; Inter-rater reliability; Otitis media; Otoscopy; Wideband tympanometry.