A head-to-head comparison of the intra- and interobserver agreement of COVID-RADS and CO-RADS grading systems in a population with high estimated prevalence of COVID-19

Nikita Sushentsev; Vlad Bura; Maruša Kotnik; Grigoriy Shiryaev; Iztok Caglic; Jonathan Weir-McCall; Tristan Barrett

doi:10.1259/bjro.20200053

A head-to-head comparison of the intra- and interobserver agreement of COVID-RADS and CO-RADS grading systems in a population with high estimated prevalence of COVID-19

BJR Open. 2020 Dec 11;2(1):20200053. doi: 10.1259/bjro.20200053. eCollection 2020.

Authors

Nikita Sushentsev¹, Vlad Bura², Maruša Kotnik³, Grigoriy Shiryaev⁴, Iztok Caglic¹, Jonathan Weir-McCall, Tristan Barrett¹

Affiliations

¹ Department of Radiology, Addenbrooke's Hospital and University of Cambridge, Cambridge, United Kingdom.
² Department of Radiology, County Clinical Emergency Hospital, Cluj-Napoca, Romania.
³ Royal Papworth Hospital NHS Foundation Trust, Cambridge, United Kingdom.
⁴ COVID-19 Center, National Medical Research Center of Cardiology, Moscow, Russian Federation, Russia.

Abstract

Objective: To evaluate the inter- and intraobserver agreement of COVID-RADS and CO-RADS reporting systems among differently experienced radiologists in a population with high estimated prevalence of COVID-19.

Methods and materials: Chest CT scans of patients with clinically-epidemiologically diagnosed COVID-19 were retrieved from an open-source MosMedData data set, randomised, and independently assigned COVID-RADS and CO-RADS grades by an abdominal radiology fellow, thoracic imaging fellow and a consultant cardiothoracic radiologist. The inter- and intraobserver agreement of the two systems were assessed using the Fleiss' and Cohen's κ coefficients, respectively.

Results: A total of 200 studies were included in the analysis. Both systems demonstrated moderate interobserver agreement, with κ values of 0.51 [95% confidence interval (CI): 0.46-0.56] and 0.55 (95% CI: 0.50-0.59) for COVID-RADS and CO-RADS, respectively. When COVID-RADS and CO-RADS grades were dichotomised at cut-off values of 2B and 4 to evaluate the agreement between grades representing different levels of clinical suspicion for COVID-19, the interobserver agreement became substantial with κ values of 0.74 (95% CI: 0.66-0.82) for COVID-RADS and 0.73 (95% CI: 0.65-0.81) for CO-RADS. The median intraobserver agreement was considerably higher for CO-RADS reaching 0.81 (95% CI: 0.43-0.76) compared with 0.60 (95% CI: 0.43-0.76) of COVID-RADS.

Conclusions: COVID-RADS and CO-RADS showed comparable interobserver agreement, which was moderate when grades were compared head-to-head and substantial when grades were dichotomised to better reflect the underlying levels of suspicion for COVID-19. The median intraobserver agreement of CO-RADS was, however, considerably higher compared with COVID-RADS.

Advances in knowledge: This paper provides a comprehensive review of the newly introduced COVID-19 chest CT reporting systems, which will help radiologists of all sub-specialties and experience levels make an informed decision on which system to use in their own practice.