Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry

Zhiyi Chen; Bowen Hu; Xuerong Liu; Benjamin Becker; Simon B Eickhoff; Kuan Miao; Xingmei Gu; Yancheng Tang; Xin Dai; Chao Li; Artemiy Leonov; Zhibing Xiao; Zhengzhi Feng; Ji Chen; Hu Chuan-Peng

doi:10.1186/s12916-023-02941-4

Sampling inequalities affect generalization of neuroimaging-based diagnostic classifiers in psychiatry

BMC Med. 2023 Jul 3;21(1):241. doi: 10.1186/s12916-023-02941-4.

Authors

Zhiyi Chen^{1

2}, Bowen Hu³, Xuerong Liu⁴, Benjamin Becker^{5

6}, Simon B Eickhoff⁷, Kuan Miao⁴, Xingmei Gu⁴, Yancheng Tang⁸, Xin Dai³, Chao Li⁹, Artemiy Leonov¹⁰, Zhibing Xiao¹¹, Zhengzhi Feng⁴, Ji Chen^{12

13}, Hu Chuan-Peng¹⁴

Affiliations

¹ Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China. chenzhiyi@tmmu.edu.cn.
² Faculty of Psychology, Southwest University, Chongqing, China. chenzhiyi@tmmu.edu.cn.
³ Faculty of Psychology, Southwest University, Chongqing, China.
⁴ Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, Chongqing, China.
⁵ The Center of Psychosomatic Medicine, Sichuan Provincial Center for Mental Health, Sichuan Provincial People's Hospital, Chengdu, China.
⁶ The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Laboratory for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, China.
⁷ Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf, Düsseldorf, Germany.
⁸ School of Business and Management, Shanghai International Studies University, Shanghai, China.
⁹ Department of Radiology, The Third Affiliated Hospital, Sun Yat-Sen University, Guangdong, China.
¹⁰ School of Psychology, Clark University, Worcester, MA, USA.
¹¹ State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University, Beijing, China.
¹² Department of Psychology and Behavioral Sciences, Zhejiang University, Hangzhou, China. ji.chen@zju.edu.cn.
¹³ Department of Psychiatry, The Fourth Affiliated Hospital, Zhejiang University School of Medicine, Yiwu, Zhejiang, China. ji.chen@zju.edu.cn.
¹⁴ School of Psychology, Nanjing Normal University, Nanjing, China.

Abstract

Background: The development of machine learning models for aiding in the diagnosis of mental disorder is recognized as a significant breakthrough in the field of psychiatry. However, clinical practice of such models remains a challenge, with poor generalizability being a major limitation.

Methods: Here, we conducted a pre-registered meta-research assessment on neuroimaging-based models in the psychiatric literature, quantitatively examining global and regional sampling issues over recent decades, from a view that has been relatively underexplored. A total of 476 studies (n = 118,137) were included in the current assessment. Based on these findings, we built a comprehensive 5-star rating system to quantitatively evaluate the quality of existing machine learning models for psychiatric diagnoses.

Results: A global sampling inequality in these models was revealed quantitatively (sampling Gini coefficient (G) = 0.81, p < .01), varying across different countries (regions) (e.g., China, G = 0.47; the USA, G = 0.58; Germany, G = 0.78; the UK, G = 0.87). Furthermore, the severity of this sampling inequality was significantly predicted by national economic levels (β = - 2.75, p < .001, R²_adj = 0.40; r = - .84, 95% CI: - .41 to - .97), and was plausibly predictable for model performance, with higher sampling inequality for reporting higher classification accuracy. Further analyses showed that lack of independent testing (84.24% of models, 95% CI: 81.0-87.5%), improper cross-validation (51.68% of models, 95% CI: 47.2-56.2%), and poor technical transparency (87.8% of models, 95% CI: 84.9-90.8%)/availability (80.88% of models, 95% CI: 77.3-84.4%) are prevailing in current diagnostic classifiers despite improvements over time. Relating to these observations, model performances were found decreased in studies with independent cross-country sampling validations (all p < .001, BF₁₀ > 15). In light of this, we proposed a purpose-built quantitative assessment checklist, which demonstrated that the overall ratings of these models increased by publication year but were negatively associated with model performance.

Conclusions: Together, improving sampling economic equality and hence the quality of machine learning models may be a crucial facet to plausibly translating neuroimaging-based diagnostic classifiers into clinical practice.

Keywords: Diagnostic classification; Meta-analysis; Neuroimaging; Psychiatric machine learning; Sampling inequalities.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Humans
Machine Learning
Neuroimaging
Psychiatry*
Psychotic Disorders*
Research Design