Optimizing Existing Mental Health Screening Methods in a Dementia Screening and Risk Factor App: Observational Machine Learning Study

Narayan Kuleindiren; Raphael Paul Rifkin-Zybutz; Monika Johal; Hamzah Selim; Itai Palmon; Aaron Lin; Yizhou Yu; Ali Alim-Marvasti; Mohammad Mahmud

doi:10.2196/31209

Optimizing Existing Mental Health Screening Methods in a Dementia Screening and Risk Factor App: Observational Machine Learning Study

JMIR Form Res. 2022 Mar 22;6(3):e31209. doi: 10.2196/31209.

Authors

Narayan Kuleindiren^#¹, Raphael Paul Rifkin-Zybutz^#¹, Monika Johal^{1

2}, Hamzah Selim^{1

3}, Itai Palmon^{1

4}, Aaron Lin^{1

5}, Yizhou Yu¹, Ali Alim-Marvasti^{1

6}, Mohammad Mahmud^{1

7}

Affiliations

¹ Mindset Technologies Ltd, London, United Kingdom.
² Imperial College School of Medicine, Faculty of Medicine, Imperial College London, London, United Kingdom.
³ Medical School, University College London, London, United Kingdom.
⁴ Medical School, University of Michigan, Ann Arbor, MI, United States.
⁵ Medical School, University of Birmingham, Birmingham, United Kingdom.
⁶ Queen Square Institute of Neurology, University College London, London, United Kingdom.
⁷ Department of Brain Sciences, Imperial College London, London, United Kingdom.

^# Contributed equally.

PMID: 35315786
PMCID: PMC8984825
DOI: 10.2196/31209

Abstract

Background: Mindstep is an app that aims to improve dementia screening by assessing cognition and risk factors. It considers important clinical risk factors, including prodromal symptoms, mental health disorders, and differential diagnoses of dementia. The 9-item Patient Health Questionnaire for depression (PHQ-9) and the 7-item Generalized Anxiety Disorder Scale (GAD-7) are widely validated and commonly used scales used in screening for depression and anxiety disorders, respectively. Shortened versions of both (PHQ-2/GAD-2) have been produced.

Objective: We sought to develop a method that maintained the brevity of these shorter questionnaires while maintaining the better precision of the original questionnaires.

Methods: Single questions were designed to encompass symptoms covered in the original questionnaires. Answers to these questions were combined with PHQ-2/GAD-2, and anonymized risk factors were collected by Mindset4Dementia from 2235 users. Machine learning models were trained to use these single questions in combination with data already collected by the app: age, response to a joke, and reporting of functional impairment to predict binary and continuous outcomes as measured using PHQ-9/GAD-7. Our model was developed with a training data set by using 10-fold cross-validation and a holdout testing data set and compared to results from using the shorter questionnaires (PHQ-2/GAD-2) alone to benchmark performance.

Results: We were able to achieve superior performance in predicting PHQ-9/GAD-7 screening cutoffs compared to PHQ-2 (difference in area under the curve 0.04, 95% CI 0.00-0.08, P=.02) but not GAD-2 (difference in area under the curve 0.00, 95% CI -0.02 to 0.03, P=.42). Regression models were able to accurately predict total questionnaire scores in PHQ-9 (R²=0.655, mean absolute error=2.267) and GAD-7 (R²=0.837, mean absolute error=1.780).

Conclusions: We app-adapted PHQ-4 by adding brief summary questions about factors normally covered in the longer questionnaires. We additionally trained machine learning models that used the wide range of additional information already collected in Mindstep to make a short app-based screening tool for affective disorders, which appears to have superior or equivalent performance to well-established methods.

Keywords: anxiety; cognition; dementia; depression; machine learning; precision; prediction; questionnaire; research method; risk factors; screening.

©Narayan Kuleindiren, Raphael Paul Rifkin-Zybutz, Monika Johal, Hamzah Selim, Itai Palmon, Aaron Lin, Yizhou Yu, Ali Alim-Marvasti, Mohammad Mahmud. Originally published in JMIR Formative Research (https://formative.jmir.org), 22.03.2022.