Comparison of the MOdified NARanjo Causality Scale (MONARCSi) for Individual Case Safety Reports vs. a Reference Standard

Abstract

Introduction: In 2018, we published the MONARCSi algorithmic decision support tool showing high inter-rater agreement, moderate sensitivity, and high specificity compared with drug-event pairs (DEPs) previously reviewed using current, industry-established approaches. Following publication, MONARCSi was implemented as a prototype system to facilitate medical review of individual case safety reports (ICSRs). This paper presents subsequent evaluation of MONARCSi-supported causality assessments against an independent, best achievable reference standard.

Objective: This paper describes the development of an independent reference standard (i.e., reference comparator) using a sample of DEPs evaluated by Roche subject matter experts (SMEs) and subsequent performance analysis for both the reference standard and MONARCSi.

Methods: Roche collected a random sample of 131 DEPs evaluated by an external vendor using the MONARCSi prototype during 2020, and collectively referred to as the VMON (Vendor using the MONARCSi system for medical review) dataset. An internal group of causality SMEs (aka CAUSMET) were recruited and trained to assess the same DEPs independently using the MONARCSi structure with Global Introspection to determine their individual assessments of causality. The CAUSMET final causality was determined using a majority voting rule.

Results: Binary comparison of the aggregate results showed substantial agreement (Gwet kappa = 0.81) between the VMON and reference standard CAUSMET assessments. Bayesian latent class modeling showed that both the reference standard and VMON assessments exhibited similar high posterior mean sensitivity and specificity (CAUSMET: 89 and 93%, respectively; VMON: 87 and 94%, respectively). Finally, comparison of the sensitivity and specificity suggested no obvious difference across groups.

Conclusion: Analysis of causality results from the assessments by independent internal SMEs using MONARCSi shows there is no obvious difference in performance between the aggregate CAUSMET and VMON assessments based on the comparison of specificity and sensitivity. These results further support use of MONARCSi as a decision support tool for evaluating drug-event causality in a consistent and documentable manner.