Assessing the Performance of the Clinician-reported Genetic Testing Utility InDEx (C-GUIDE): Further Evidence of Inter-rater Reliability

Robin Z Hayeems; Stephanie Luca; Lauren Chad; Nada Quercia; Bowen Xiao; Alomgir Hossain; M Stephen Meyn; Eleanor Pullenayegum; Wendy J Ungar

doi:10.1016/j.clinthera.2023.07.006

Assessing the Performance of the Clinician-reported Genetic Testing Utility InDEx (C-GUIDE): Further Evidence of Inter-rater Reliability

Clin Ther. 2023 Aug;45(8):729-735. doi: 10.1016/j.clinthera.2023.07.006. Epub 2023 Jul 27.

Authors

Robin Z Hayeems¹, Stephanie Luca², Lauren Chad³, Nada Quercia⁴, Bowen Xiao², Alomgir Hossain⁵, M Stephen Meyn⁶, Eleanor Pullenayegum⁷, Wendy J Ungar⁸

Affiliations

¹ Child Health Evaluative Sciences, Peter Gilgan Centre for Research and Learning, The Hospital for Sick Children, Toronto, Ontario, Canada; Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada. Electronic address: robin.hayeems@sickkids.ca.
² Child Health Evaluative Sciences, Peter Gilgan Centre for Research and Learning, The Hospital for Sick Children, Toronto, Ontario, Canada.
³ Division of Clinical and Metabolic Genetics, The Hospital for Sick Children, Toronto, Ontario, Canada; Department of Pediatrics, University of Toronto, Toronto, Ontario, Canada.
⁴ Division of Clinical and Metabolic Genetics, The Hospital for Sick Children, Toronto, Ontario, Canada; Department of Genetic Counselling, The Hospital for Sick Children, Toronto, Ontario, Canada; Department of Molecular Genetics, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
⁵ Biostatistics, Design and Analysis Unit, The Hospital for Sick Children, Toronto, Ontario, Canada.
⁶ Center for Human Genomics and Precision Medicine, The University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, USA.
⁷ Child Health Evaluative Sciences, Peter Gilgan Centre for Research and Learning, The Hospital for Sick Children, Toronto, Ontario, Canada; Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada.
⁸ Child Health Evaluative Sciences, Peter Gilgan Centre for Research and Learning, The Hospital for Sick Children, Toronto, Ontario, Canada; Institute of Health Policy Management and Evaluation, University of Toronto, Toronto, Ontario, Canada.

PMID: 37516567
DOI: 10.1016/j.clinthera.2023.07.006

Abstract

Purpose: Advanced genomic and genetic testing technologies are quickly diffusing into clinical practice, but standardized approaches to assessing their clinical utility are limited. Previous work developed and generated preliminary evidence of validity for a novel outcome measure, the Clinician-reported Genetic testing Utility InDEx (C-GUIDE). C-GUIDE is a 17-item measure that captures the utility of genetic testing from the providers' perspective. Preliminary evidence of its inter-rater reliability was obtained through a clinical vignette study. The purpose of this study was to further assess its inter-rater reliability using actual clinical cases.

Methods: One genetic counselor and one medical geneticist independently completed C-GUIDE Version 1.1 after genetic test results were disclosed to a shared set of 42 patients. Raters also completed a case description questionnaire, including information about the patient's age, indication for testing, and type of test performed. Inter-rater reliability was assessed by comparing the raters' C-GUIDE scores using ANOVA to generate intra-class correlation coefficients (ICCs), absolute agreement, and mixed repeated measures ANOVA.

Findings: Of the 42 patients studied, the most common indications for testing were hearing loss (n = 18) and craniosynostosis (n = 11), and the most common tests ordered were gene panels (n = 20) and microarrays (n = 10). Test results were diagnostic or partially diagnostic for 11 patients, potentially diagnostic for 14 patients, or nondiagnostic for 17 patients. The overall ICC was 0.95 (95% CI, 0.89-0.97) and absolute agreement was acceptable (>70%) for 15 individual items. Inter-rater agreement was excellent (ICC > 0.90) for 8 items, good (ICC = 0.75-0.89) for 3 items, moderate (ICC = 0.50-0.74) for 4 items and poor (ICC < 0.50) for 2 items. Absolute agreement was unacceptable (<70%), and rater agreement was fair (ICC = 0.40-0.59) for 2 items. For the global rating, the ICC was 0.62 (95% CI, 0.39-0.77), and the absolute agreement was 61.9%.

Implications: Rater instructions for item completion have been modified to improve consistency of item interpretation. Although further assessments of reliability are warranted after modifications, these findings provide additional tentative evidence of C-GUIDE's inter-rater reliability and suggest that it may be useful as a strategy for measuring the value of genetic testing, as perceived by genetics providers.

Keywords: Clinical utility; Inter-rater reliability; Measurement index; Outcomes research.