In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving

Abstract

Objectives: The aim of this study is to describe and pilot a novel method for continuously identifying newly published trials relevant to a systematic review, enabled by combining artificial intelligence (AI) with human expertise.

Study design and setting: We used RobotReviewer LIVE to keep a review of COVID-19 vaccination trials updated from February to August 2021. We compared the papers identified by the system with those found by the conventional manual process by the review team.

Results: The manual update searches (last search date July 2021) retrieved 135 abstracts, of which 31 were included after screening (23% precision, 100% recall). By the same date, the automated system retrieved 56 abstracts, of which 31 were included after manual screening (55% precision, 100% recall). Key limitations of the system include that it is limited to searches of PubMed/MEDLINE, and considers only randomized controlled trial reports. We aim to address these limitations in future. The system is available as open-source software for further piloting and evaluation.

Conclusion: Our system identified all relevant studies, reduced manual screening work, and enabled rolling updates on publication of new primary research.

Keywords: Artificial intelligence; Evidence based medicine; Living systematic reviews; Machine learning; Natural language processing; Systematic reviews.