Inter-rater reliability of the extended Composite Quality Score (CQS-2)

Steffen Mickenautsch; Stefan Rupf; Ivana Miletić; Ulf Tilman Strähle; Richard Sturm; Faheema Kimmie-Dhansay; Kata Vidosusić; Veerasamy Yengopal

doi:10.3389/fmed.2023.1201517

Inter-rater reliability of the extended Composite Quality Score (CQS-2)

Front Med (Lausanne). 2023 Aug 17:10:1201517. doi: 10.3389/fmed.2023.1201517. eCollection 2023.

Authors

Steffen Mickenautsch^{1

2

3}, Stefan Rupf⁴, Ivana Miletić⁵, Ulf Tilman Strähle⁴, Richard Sturm⁶, Faheema Kimmie-Dhansay⁷, Kata Vidosusić⁵, Veerasamy Yengopal¹

Affiliations

¹ Faculty of Dentistry, University of the Western Cape, Bellville, South Africa.
² Department of Community Dentistry, Faculty of Health Sciences, School of Oral Health Sciences, University of the Witwatersrand, Johannesburg, South Africa.
³ Review Centre for Health Science Research, Johannesburg, South Africa.
⁴ Synoptic Dentistry, Saarland University, Homburg, Germany.
⁵ Department of Endodontics and Restorative Dentistry, School of Dental Medicine, University of Zagreb, Zagreb, Croatia.
⁶ Department of Operative, Preventive and Paediatric Dentistry, Charité - Universitätsmedizin Berlin, Berlin, Germany.
⁷ Department of Community Oral Health, Faculty of Dentistry, University of the Western Cape, Bellville, South Africa.

Abstract

Aim: To establish the inter-rater reliability of the Composite Quality Score (CQS-2) and to test the null hypothesis that it did not differ significantly from that of the first CQS version (CQS-1).

Materials and methods: Four independent raters were selected to rate 45 clinical trial reports using CQS-1 and CQS-2. The raters remained unaware of each other's participation in this study until all rating had been completed. Each rater received only one rating template at a time in a random sequence for CQS-1 and CQS-2 rating. Raters completed each template and sent these back to the principal investigator. Each rater received their next template 2 weeks after submission of the completed previous template. The inter-rater reliabilities for the overall appraisal score of the CQS-1 and the CQS-2 were established by using the Brennan-Prediger coefficient (BPC). The coefficients of both CQS versions were compared by using the two-sample z-test. During secondary analysis, the BPCs for every criterion and each corroboration level for both CQS versions were established.

Results: The BPC for the CQS-1 was 0.85 (95% CI: 0.64-1.00) and for the CQS-2 it was 1.00 (95% CI: 0.94-1.00), suggesting a very high inter-rater reliability for both. The difference between the two CQS versions was statistically not significant (p = 0.17). The null hypothesis was accepted.

Conclusion: The CQS-2 is still under development, This study shows that it is associated with a very high inter-rater reliability, which did not statistically significantly differ from that of the CQS-1. The promising results of this study warrant further investigation in the applicability of the CQS-2 as an appraisal tool for prospective controlled clinical therapy trials.

Keywords: Composite Quality Score; clinical rial; inter-rater reliability; systematic review; trial appraisal.