Sensitivity and specificity of alternative screening methods for systematic reviews using text mining tools

Jimmy Li; Joudy Kabouji; Sarah Bouhadoun; Sarah Tanveer; Kristian B Filion; Genevieve Gore; Colin Bruce Josephson; Churl-Su Kwon; Nathalie Jette; Prisca Rachel Bauer; Gregory S Day; Ann Subota; Jodie I Roberts; Sara Lukmanji; Khara Sauro; Adnane Alaoui Ismaili; Feriel Rahmani; Khadidja Chelabi; Yasmine Kerdougli; Nour Meryem Seulami; Aminata Soumana; Sarah Khalil; Noémie Maynard; Mark Robert Keezer

doi:10.1016/j.jclinepi.2023.07.010

Sensitivity and specificity of alternative screening methods for systematic reviews using text mining tools

J Clin Epidemiol. 2023 Oct:162:72-80. doi: 10.1016/j.jclinepi.2023.07.010. Epub 2023 Jul 26.

Authors

Jimmy Li¹, Joudy Kabouji², Sarah Bouhadoun³, Sarah Tanveer⁴, Kristian B Filion⁵, Genevieve Gore⁶, Colin Bruce Josephson⁷, Churl-Su Kwon⁸, Nathalie Jette⁹, Prisca Rachel Bauer¹⁰, Gregory S Day¹¹, Ann Subota¹², Jodie I Roberts¹³, Sara Lukmanji¹³, Khara Sauro¹⁴, Adnane Alaoui Ismaili¹⁵, Feriel Rahmani¹⁶, Khadidja Chelabi¹⁷, Yasmine Kerdougli¹⁸, Nour Meryem Seulami¹⁹, Aminata Soumana¹⁸, Sarah Khalil¹⁸, Noémie Maynard¹⁵, Mark Robert Keezer²⁰

Affiliations

¹ Neurology Division, Centre Hospitalier de l'Université de Sherbrooke (CHUS), Sherbrooke, Canada; Centre de Recherche du Centre Hospitalier de l'Université de Montréal (CRCHUM), Montreal, Canada.
² Department of Pharmacy, University of Laval, Quebec City, Canada.
³ Department of Neurology, McGill University, Montreal, Canada.
⁴ Department of Pharmaceutical Health Services Research, University of Maryland, Baltimore, MD, USA.
⁵ Departments of Medicine and of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Canada; Centre for Clinical Epidemiology, Jewish General Hospital - Lady Davis Institute, Montreal, Canada.
⁶ Schulich Library of Physical Sciences, Life Sciences, and Engineering, McGill University, Montreal, Canada.
⁷ Department of Clinical Neurosciences, University of Calgary, Calgary, Canada; Department of Community Health Sciences, University of Calgary, Calgary, Canada; O'Brien Institute for Public Health, University of Calgary, Calgary, Canada; Hotchkiss Brain Institute, University of Calgary, Calgary, Canada; Center for Health Informatics, University of Calgary, Calgary, Canada.
⁸ Department of Neurology, Epidemiology, Neurosurgery and the Gertrude H. Sergievsky Center, Columbia University, New York, NY, USA.
⁹ Department of Clinical Neurosciences, University of Calgary, Calgary, Canada; Department of Community Health Sciences, University of Calgary, Calgary, Canada.
¹⁰ Department of Psychosomatic Medicine and Psychotherapy, Faculty of Medicine, Medical Center - University of Freiburg, Freiburg, Germany.
¹¹ Department of Neurology, Mayo Clinic Florida, Jacksonville, FL, USA.
¹² Department of Community Health Sciences, University of Calgary, Calgary, Canada; Department of Medicine, University of Calgary, Calgary, Canada.
¹³ Department of Community Health Sciences, University of Calgary, Calgary, Canada.
¹⁴ Department of Community Health Sciences, University of Calgary, Calgary, Canada; Department of Surgery, University of Calgary, Calgary, Canada; Department of Oncology & Arnie Charbonneau Cancer Institute, University of Calgary, Calgary, Canada.
¹⁵ Department of Internal Medicine, McGill University, Montreal, Canada.
¹⁶ Faculty of Medicine, McGill University, Montreal, Canada.
¹⁷ Department of Pediatrics, McGill University, Montreal, Canada.
¹⁸ Department of Family Medicine, McGill University, Montreal, Canada.
¹⁹ Department of Emergency Medicine, McGill University, Montreal, Canada.
²⁰ Centre de Recherche du Centre Hospitalier de l'Université de Montréal (CRCHUM), Montreal, Canada; Departments of Medicine and of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, Canada; Department of Neurosciences, Université de Montréal, Montreal, Canada; School of Public Health, Université de Montréal, Montreal, Canada. Electronic address: mark.keezer@umontreal.ca.

PMID: 37506951
DOI: 10.1016/j.jclinepi.2023.07.010

Abstract

Objectives: To evaluate the impact of text mining (TM) on the sensitivity and specificity of title and abstract screening strategies for systematic reviews (SRs).

Study design and setting: Twenty reviewers each evaluated a 500-citation set. We compared five screening methods: conventional double screen (CDS), single screen, double screen with TM, combined double screen and single screen with TM, and single screen with TM. Rayyan, Abstrackr, and SWIFT-Review were used for each TM method. The results of a published SR were used as the reference standard.

Results: The mean sensitivity and specificity achieved by CDS were 97.0% (95% confidence interval [CI]: 94.7, 99.3) and 95.0% (95% CI: 93.0, 97.1). When compared with single screen, CDS provided a greater sensitivity without a decrease in specificity. Rayyan, Abstrackr, and SWIFT-Review identified all relevant studies. Specificity was often higher for TM-assisted methods than that for CDS, although with mean differences of only one-to-two percentage points. For every 500 citations not requiring manual screening, 216 minutes (95% CI: 169, 264) could be saved.

Conclusion: TM-assisted screening methods resulted in similar sensitivity and modestly improved specificity as compared to CDS. The time saved with TM makes this a promising new tool for SR.

Keywords: Abstrackr; Artificial intelligence; Diagnostic study; Knowledge synthesis; Machine learning; Rayyan; SWIFT-Review; Sensitivity; Specificity.

MeSH terms

Data Mining* / methods
Humans
Publications*
Sensitivity and Specificity
Systematic Reviews as Topic