The aim of this study is to describe the frequency of isolated, subtle, neurological abnormalities (ISNAs) in a large population of neurologically and cognitively healthy subjects and to compare ISNAs to various types of MRI-detected cerebrovascular lesions and subcortical brain atrophy in different age classes. 907 subjects were selected from a large, prospective hospital-based study. At baseline neurological examination, 17 ISNAs were selected. Primitive reflexes were the most common ISNAs (35.8%), while dysphagia was the most rarely encountered (0.3%). Measures of small vessel disease, i.e., deep and subcortical white matter hyperintensity and lacunar infarcts as well as subcortical atrophy, were variously associated with ISNAs. In the adult group, the ISNAs were associated with hypertriglyceridemia, TIA, and subcortical lacunar infarcts, while in the elderly-old group they were associated with arterial hypertension, subcortical white matter hyperintensity, and subcortical atrophy. An increased risk of ISNAs was associated with lacunae and white matter hyperintensity in the parietal region. This study shows that white matter hyperintensity, lacunae, and subcortical atrophy are associated with an increased risk of ISNAs in cognitively and neurologically healthy aging subjects. ISNAs are not benign signs. Therefore, adults and elderly people presenting with ISNAs should have access to accurate history and diagnosis to prevent progression of small vessel disease and future neurological and cognitive disabilities.