Adoption of electronic medical records in hospitals generates a large amount of data. Health care professionals can easily lose their sight on the important insights of the patients' clinical and medical history. Although machine learning algorithms have already proved their significance in healthcare research, remains a challenge translation and dissemination of fully automated prediction algorithms from research to decision support at the point of care. In this paper, we address the effect of changes in the characteristics of data over time on the performance of deployed models for the use case of predicting delirium in hospitalised patients. We have analysed the stability of models trained with subsets of data from one single year (2012, 2013...2016, respectively), and tested the models with data from 2017. Our results show that in the case of delirium prediction, the models were stable over time, indicating that re-training the models is not necessary e.g. once per year might be more than sufficient.
Keywords: machine learning; model deployment; model stability; prediction.