Reproducibility in machine learning for health research: Still a ways to go

Matthew B A McDermott; Shirly Wang; Nikki Marinsek; Rajesh Ranganath; Luca Foschini; Marzyeh Ghassemi

doi:10.1126/scitranslmed.abb1655

Reproducibility in machine learning for health research: Still a ways to go

Sci Transl Med. 2021 Mar 24;13(586):eabb1655. doi: 10.1126/scitranslmed.abb1655.

Authors

Matthew B A McDermott¹, Shirly Wang^{2

3}, Nikki Marinsek⁴, Rajesh Ranganath⁵, Luca Foschini⁴, Marzyeh Ghassemi^{2

6

7}

Affiliations

¹ Massachusetts Institute of Technology, Cambridge, MA 02139, USA. mmd@mit.edu.
² Department of Computer Science, University of Toronto, Toronto, ON M5T 3A1, Canada.
³ Layer 6 AI, TD Bank Group, Toronto, ON M5G 1M1, Canada.
⁴ Evidation Health Inc., San Mateo, CA 94402, USA.
⁵ Center for Data Science and Department of Computer Science, Courant Institute of Mathematical Sciences, New York University, New York, NY 10011, USA ; Department of Population Health, NYU Langone Health, New York, NY, USA 10016.
⁶ Vector Institute, University of Toronto, Toronto, ON M5G 1M1, Canada.
⁷ Department of Medicine, University of Toronto, Toronto, ON M5S 1A8, Canada.

PMID: 33762434
DOI: 10.1126/scitranslmed.abb1655

Abstract

Machine learning for health must be reproducible to ensure reliable clinical use. We evaluated 511 scientific papers across several machine learning subfields and found that machine learning for health compared poorly to other areas regarding reproducibility metrics, such as dataset and code accessibility. We propose recommendations to address this problem.

Publication types

Research Support, N.I.H., Extramural
Research Support, Non-U.S. Gov't
Review

MeSH terms

Machine Learning*
Reproducibility of Results