A descriptive appraisal of quality of reporting in a cohort of machine learning studies in anesthesiology

Rishi Kothari; Catherine Chiu; Mira Moukheiber; Matthew Jehiro; Andrew Bishara; Christine Lee; Romain Pirracchio; Leo Anthony Celi

doi:10.1016/j.accpm.2022.101126

A descriptive appraisal of quality of reporting in a cohort of machine learning studies in anesthesiology

Anaesth Crit Care Pain Med. 2022 Oct;41(5):101126. doi: 10.1016/j.accpm.2022.101126. Epub 2022 Jul 8.

Authors

Rishi Kothari¹, Catherine Chiu², Mira Moukheiber³, Matthew Jehiro⁴, Andrew Bishara², Christine Lee⁵, Romain Pirracchio², Leo Anthony Celi⁶

Affiliations

¹ Department of Anesthesia and Perioperative Care, University of California San Francisco, San Francisco, CA 4143, USA. Electronic address: rishi.p.kothari@gmail.com.
² Department of Anesthesia and Perioperative Care, University of California San Francisco, San Francisco, CA 4143, USA.
³ Picower Institute for Learning & Memory, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
⁴ Department of Biostatistics, State University of New York at Buffalo, Buffalo, NY 14260, USA.
⁵ Edwards Lifesciences, Critical Care, Irvine, CA 92614, USA.
⁶ Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA; Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA 02215, USA; Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA 02115, USA.

PMID: 35811037
DOI: 10.1016/j.accpm.2022.101126

Abstract

Background: The field of machine learning is being employed more and more in medicine. However, studies have shown that the quality of published studies frequently lacks completeness and adherence to published reporting guidelines. This assessment has not been done in the subspecialty of anesthesiology.

Methods: We appraised the quality of reporting of a convenience sample of 67 peer-reviewed publications sourced from the scoping review by Hashimoto et al. Each publication was appraised on the presence of reporting elements (reporting compliance) selected from 4 peer-reviewed guidelines for reporting on machine learning studies. Results are described in several cross sections, including by section of manuscript (e.g. abstract, introduction, etc.), year of publication, impact factor of journal, and impact of publication.

Results: On average, reporting compliance was 64% ± 13%. There was marked heterogeneity of reporting based on section of manuscript. There was a mild trend towards increased quality of reporting with increasing impact factor of journal of publication and increasing average number of citations per year since publication, and no trend regarding recency of publication.

Conclusion: The quality of reporting of machine learning studies in anesthesiology is on par with other fields, but can benefit from improvement, especially in presenting methodology, results, and discussion points, including interpretation of models and pitfalls therein. Clinicians in today's learning health systems will benefit from skills in appraisal of evidence. Several reporting guidelines have been released, and updates to mainstream guidelines are under development, which we hope will usher in improvement in reporting quality.

Keywords: Appraisal of evidence; Artificial intelligence; Learning health system; Machine learning.

MeSH terms

Anesthesiology* / methods
Cohort Studies
Humans
Machine Learning
Research Design