Residential environments are important in the daily lives of individuals, especially in terms of cognitive psychophysiology. This study compared the neural responses to two extreme residential environments: comfortable versus uncomfortable. Thirteen healthy individuals underwent a 3.0-T functional MRI while viewing images representing comfortable and uncomfortable residential environments. During exposure to the comfortable scenery, significant activation was observed in the calcarine gyrus selectively, whereas the uncomfortable scenery induced significant activation in the ventrolateral prefrontal cortex and anterior cingulate cortex, which are related to cognitive control, as well as in the medial prefrontal cortex, hippocampus, parahippocampal gyrus, amygdala, and insula, which are involved in the regulation and expression of emotion. Such differential brain activation patterns may reflect the neural networks mediating cognitive and emotional responses to residential environments. These findings would be helpful for understanding the neural mechanism associated with human comfortability for their residential environments.