Among breast cancer patients treated with neoadjuvant chemotherapy (NAC) and mastectomy, locoregional recurrence (LRR) rates are unclear in women with ER+ tumors treated with adjuvant endocrine therapy without postmastectomy radiation (PMRT). To determine if PMRT is needed in these patients, we compared LRR rates of patients with ER+ tumors (treated with adjuvant endocrine therapy) with women who have non-ER+ tumors. 85 consecutive breast cancer patients (87 breast tumors) treated with NAC and mastectomy without PMRT were reviewed. Patients were divided by residual nodal disease (ypN) status (ypN+ versus ypN0) and then stratified by receptor subtype. Among ypN+ patients (n = 35), five-year LRR risk in patients with ER+, Her2+, and triple negative tumors was 5%, 33%, and 37%, respectively (p = 0.02). Among ypN+/ER+ patients, lymphovascular invasion and grade three disease increased the five-year LRR risk to 13% and 11%, respectively. Among ypN0 patients (n = 52), five-year LRR risk in patients with ER+, Her2+, and triple negative tumors was 7%, 22%, and 6%, respectively (p = 0.71). In women with ER+ tumors and residual nodal disease, endocrine therapy may be sufficient adjuvant treatment, except in patients with lymphovascular invasion or grade three tumors where PMRT may still be indicated.