We aimed to evaluate the efficacy of physical exercise in ameliorating depressive symptoms in patients with cognitive impairment. The databases of PubMed, EMBASE, Web of Science, the Cochrane Library, PsycINFO, China National Knowledge Infrastructure, WanFang, and WeiPu (VIP) were searched to identify randomized controlled trials (RCTs) that involved physical exercise for patients with cognitive impairment. A random effects model and a fixed effects model were used to calculate the pooled effect size. Twenty-one studies were identified. The meta-analysis showed that physical exercise significantly ameliorated depressive symptoms (standardized mean difference [SMD] = -0.23; 95% confidence interval [CI], -0.39 to -0.07; p = 0.004). In addition, beneficial improvements in neuropsychiatric symptoms (mean difference, -4.62; 95% CI, -9.07 to -0.16, p = 0.04), quality of life (SMD = 0.23; 95% CI, 0.01-0.46; p = 0.04), and activities of daily living (SMD = 0.27; 95% CI, 0.12-0.43; p = 0.0005) were observed in our study. No significant improvements were found in anxiety or apathy. Nevertheless, further high-quality, multicenter RCTs are needed to identify the clinical value of our results.