Consistency and Sensitivity Evaluation of the Saudi Arabia Mental Health Surveillance System (MHSS): Hypothesis Generation and Testing

Abstract

Background: Public health surveillance systems should be evaluated periodically, and the evaluation should include recommendations for improving the system's quality and efficiency. Each surveillance system may have a unique situation in which evaluating its quality depends on its methodology, aims, and other factors, such as the frequency of repeating the survey in the case of survey-based surveillance.

Objective: As the consistency of the surveillance system to capture demographic data and its sensitivity to monitor the intended health-related event are important indicators of the quality of the surveillance system, the aim of this article is to evaluate the Saudi Arabia Mental Health Surveillance System (MHSS) in terms of consistency and sensitivity via the scientific hypothesis testing process.

Methods: The quality of the MHSS was assessed by examining (1) the consistency of the main demographic variables and (2) the sensitivity to changes in score between the 2 mental health screening tools used in the MHSS and between the 3 waves collected in 3 consecutive months. The assessment uses all data collected via the MHSS between May 2020 and July 2020. The first null hypothesis predicted there were differences between the distributions of the demographic variables between the 3 waves. The second predicted there were no differences between the scores of the Patient Health Questionnaire 9 (PHQ-9) and the Generalized Anxiety Disorder 7-item scale (GAD-7) between the 3 waves.

Results: In terms of sampling variables (age, gender, and region), there were no significant differences between the 3 waves in age, using one-way ANOVA, nor in gender and region, using the chi-square test. In addition, there were no significant differences between the 3 waves in all other demographic variables, except in the income variable. However, in terms of the PHQ-9 score, the one-way ANOVA (F_2,12334=8.05; P<.001) showed significant differences between waves. Similarly, significant differences between waves were found in the GAD-7 score (F_2,12334=7.09; P=.001).

Conclusions: The MHSS showed a consistent distribution of the sample demographic variables, while being sensitive to the changes in mental health scores across waves. The MHSS can generate an acceptable level of consistency and sensitivity to monitor mental health trends.

International registered report identifier (irrid): RR2-10.2196/23748.

Keywords: data quality; evaluation; mental health; screening; surveillance; surveillance system evaluation; surveillance system quality.