Previous research has shown that the built environment plays a crucial role for health-related quality of life (HRQoL) and health care utilization. But, there is limited evidence on the independence of this association from lifestyle and social environment. The objective of this cross-sectional study was to investigate these associations, independent of the social environment, physical activity and body mass index (BMI). We used data from the third follow-up of the Swiss study on Air Pollution and Lung and Heart diseases In Adults (SAPALDIA), a population based cohort with associated biobank. Covariate adjusted multiple quantile and polytomous logistic regressions were performed to test associations of variables describing the perceived built environment with HRQoL and health care utilization. Higher HRQoL and less health care utilization were associated with less reported transportation noise annoyance. Higher HRQoL was also associated with greater satisfaction with the living environment and more perceived access to greenspaces. These results were independent of the social environment (living alone and social engagement) and lifestyle (physical activity level and BMI). This study provides further evidence that the built environment should be designed to integrate living and green spaces but separate living and traffic spaces in order to improve health and wellbeing and potentially save health care costs.