Considering patient clinical history impacts performance of machine learning models in predicting course of multiple sclerosis

PLoS One. 2020 Mar 20;15(3):e0230219. doi: 10.1371/journal.pone.0230219. eCollection 2020.

Abstract

Multiple Sclerosis (MS) progresses at an unpredictable rate, but predictions on the disease course in each patient would be extremely useful to tailor therapy to the individual needs. We explore different machine learning (ML) approaches to predict whether a patient will shift from the initial Relapsing-Remitting (RR) to the Secondary Progressive (SP) form of the disease, using only "real world" data available in clinical routine. The clinical records of 1624 outpatients (207 in the SP phase) attending the MS service of Sant'Andrea hospital, Rome, Italy, were used. Predictions at 180, 360 or 720 days from the last visit were obtained considering either the data of the last available visit (Visit-Oriented setting), comparing four classical ML methods (Random Forest, Support Vector Machine, K-Nearest Neighbours and AdaBoost) or the whole clinical history of each patient (History-Oriented setting), using a Recurrent Neural Network model, specifically designed for historical data. Missing values were handled by removing either all clinical records presenting at least one missing parameter (Feature-saving approach) or the 3 clinical parameters which contained missing values (Record-saving approach). The performances of the classifiers were rated using common indicators, such as Recall (or Sensitivity) and Precision (or Positive predictive value). In the visit-oriented setting, the Record-saving approach yielded Recall values from 70% to 100%, but low Precision (5% to 10%), which however increased to 50% when considering only predictions for which the model returned a probability above a given "confidence threshold". For the History-oriented setting, both indicators increased as prediction time lengthened, reaching values of 67% (Recall) and 42% (Precision) at 720 days. We show how "real world" data can be effectively used to forecast the evolution of MS, leading to high Recall values and propose innovative approaches to improve Precision towards clinically useful values.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Adolescent
  • Adult
  • Algorithms
  • Child
  • Disease Progression
  • Female
  • Humans
  • Machine Learning
  • Male
  • Middle Aged
  • Multiple Sclerosis / pathology*
  • Neural Networks, Computer
  • Probability
  • Rome
  • Support Vector Machine
  • Young Adult

Associated data

  • figshare/10.6084/m9.figshare.11902854

Grants and funding

LP acknowledges partial support from the project ‘MIME-BCI: Mindfulness Meditation training supported by Brain-Computer Interfaces” (2016) (No PI1161550696379A) from Sapienza, University of Rome (www.uniroma1.it). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. LP and RS acknowledge financial support from the Sapienza University of Rome (www.uniroma1.it) within the project "Network medicine-based machine learning and graph theory algorithms for precision oncology" (2019). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript AT and AZ acknowledge financial support from the Italian Ministry of University and Research (www.miur.gov.it) within the project CRISIS LAB PNR 2011-2013. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.