Countless studies in animals have shown how housing environments and behaviors can significantly affect anxiety and brain health, giving valuable insight as to whether this is applicable in the human context. The relationship between housing, behavior, brain health, and mental wellbeing in humans remains poorly understood. We therefore explored the interaction of housing quality, weekend/holiday sedentary behavior, brain structure, and anxiety in healthy Japanese adults. Whole-brain structural magnetic resonance imaging (MRI) methods based on gray matter volume and fractional anisotropy were used as markers for brain health. Correlation tests were conducted, and then adjusted for multiple comparisons using the False Discovery Rate method. Housing quality and weekend/holiday sedentary behavior were associated with fractional anisotropy, but not with gray matter volume. Fractional anisotropy showed significant associations with anxiety. Lastly, both weekend/holiday sedentary behavior and housing quality were indirectly associated with anxiety through fractional anisotropy. These results add to the limited evidence surrounding the relationship among housing, behavior, and the brain. Furthermore, these results show that behavior and housing qualities can have an indirect impact on anxiety through neurobiological markers such as fractional anisotropy.