Physical health-related quality of life scores have been, inconsistently, associated with breast cancer prognosis. This analysis examined whether change in physical health scores were related to outcomes in women with a history of breast cancer. 2343 breast cancer survivors in a randomized diet trial provided self-reported assessment of physical health-related quality of life at baseline and year 1. Based on change in physical health score, participants were grouped into subpopulations of decreased physical health, no/minimal changes, and increased physical health. Cox regression analysis assessed whether change in physical health (from baseline to year 1) predicted disease-free and overall survival; hazard ratio (HR) was the measure of association. There were 294 additional breast cancer events and 162 deaths among women followed for 7.3 years. Improvements in physical health were associated with younger age, lower BMI, being employed, not receiving tamoxifen, lower physical activity, and lower baseline physical and mental health. There was no association of change in physical health with additional breast cancer events or mortality among women diagnosed ≤ 2 years before study enrollment. However, among women who entered the study >2 years post-diagnosis, the HR for increased compared to decreased physical health was 0.38 (95% CI, 0.16-0.85) for all-cause mortality. These results appear to support testing an intervention to improve physical health in breast cancer patients among patients after the acute stage of treatment.