The APOE-ε4 allele is a known genetic risk for Alzheimer's disease (AD). Thus, it can be reasoned that the APOE-ε4 allele would also impact neurodegeneration-associated structural brain changes. Here we probe if the APOE-ε4 genotype directly modulates the human brain's gray matter using a neural network trained on the whole-brain gray matter images from the cognitively normally aging (CN) and AD individuals. To investigate the linkage between the APOE-ε4 allele and whole-brain (voxel-wise) gray matter, we systematically profile our investigation in multiple classification tasks, including diagnostic classification and APOE-ε4 classification conjointly as well as independently. Results suggest that although the MRI data can reliably track and reflect neurodegenerative changes in the brain cross-sectionally, the APOE-ε4 status may not be distinguishable correspondingly. The nonexistence of a direct and convincing modulative effect of APOE-ε4 on the whole-brain gray matter indicates that the gray matter changes may be independent of the APOE-ε4 status, and instead characterize a non-APOE, comorbid mechanism in AD.