Evidence accumulated from the last decade has proven that silent cerebrovascular lesions (SCLs) and their underlying pathogenic processes contribute to cognitive decline in the elderly. However, the distinct effects of each type of the lesions on cognitive performance remain unclear. Moreover, research data from Chinese elderly with SCLs is scarce. In this study, 398 otherwise healthy hypertensive elderly subjects (median age 72 years) were included and assessed. All participates were required to complete a battery of structured neuropsychological assessment, including forward and backward digit span tests, symbol digit modalities test, Stroop test, verbal fluency test and Montreal Cognitive Assessment. These tests were used to assess attention, executive function, information processing speed, language, memory and visuospatial function. A multi-sequence 3T MRI scanning was arranged within one month of the neuropsychological assessment to evaluate the burden of SCLs. SCLs were rated visually. Cerebral microbleeds (CMBs) and silent lacunes (SLs) were identified as strictly lobar CMBs and SLs or deep CMBs and SLs according to their locations, respectively. Similarly, white matter hyperintensities (WMHs) were separated into periventricular WMHs (PVHs) and deep WMHs (DWMHs). A series of linear regression models were used to assess the correlation between each type of SCLs and individual cognitive function domain. The results showed that CMBs tend to impair language-related cognition. Deep SLs affect executive function, but this association disappeared after controlling for other types of SCLs. PVHs, rather than DWMHs, are associated with cognitive decline, especially in executive function and processing speed. It is concluded that different aspects of SCLs have differential impact on cognitive performance in hypertensive elderly Chinese.