This study evaluates the impact of breast cancer (BC) in health related quality of life (HRQL) and in psychological distress (PD) during the initial phases of the disease and looks for contributing factors. A multicentric case-control study, EpiGEICAM, was carried out. Incident BC cases and age- and residence- matched controls were included. Clinical, epidemiological, HRQL (SF-36) and PD information (GHQ-28) was collected. We used multivariable logistic regression models to estimate OR of low HRQL and of PD in cases compared to controls, and to identify factors associated with low HRQL and with PD. Among 896 BC cases and 890 control women, cases had poorer scores than both, the reference population and the control group, in all SF-36 scales. BC women with lower education, younger, active workers, never smokers, those with comorbidities, in stage IV and with surgical treatment had lower physical HRQL; factors associated with low mental HRQL were dissatisfaction with social support, being current smoker and having children. Cases had a fivefold increased odds of PD compared to controls. Managing comorbidities and trying to promote social support, especially in younger and less educated women, could improve well-being of BC patients.