Breast cancer is a heterogeneous disease with variable survival outcomes. Pathologists grade the microscopic appearance of breast tissue using the Nottingham criteria, which is qualitative and does not account for non-cancerous elements within the tumor microenvironment (TME). We present the Histomic Prognostic Signature (HiPS), a comprehensive, interpretable scoring of the survival risk incurred by breast TME morphology. HiPS uses deep learning to accurately map cellular and tissue structures in order to measure epithelial, stromal, immune, and spatial interaction features. It was developed using a population-level cohort from the Cancer Prevention Study (CPS)-II and validated using data from three independent cohorts, including the PLCO trial, CPS-3, and The Cancer Genome Atlas. HiPS consistently outperformed pathologists' performance in predicting survival outcomes, independent of TNM stage and pertinent variables. This was largely driven by stromal and immune features. In conclusion, HiPS is a robustly validated biomarker to support pathologists and improve prognosis.