Tumor-infiltrating lymphocyte (TIL) levels have prognostic and predictive values in treatment-naïve breast cancers. However, there have been controversies regarding TIL subset changes and their clinical implications in post-treatment breast cancers. This study aimed to explore change and prognostic significance of TIL subset infiltration after primary systemic therapy (PST) in breast cancer. One-hundred-fifty-five patients who had residual disease after anthracycline- or anthracycline plus taxane-based PST were included. The quantities of intratumoral and stromal TIL subsets (CD8+, CD4+, and FOXP3+ TILs) in pre- and post-PST breast cancer samples, as well as changes between them, were analyzed along with their correlations with clinicopathologic features and outcome of patients. As a whole, intratumoral CD8+ and CD4+ TILs increased after PST while stromal TILs decreased. Both intratumoral and stromal FOXP3+ TILs decreased after PST. The chemo-sensitive group [residual cancer burden (RCB) class I and II] showed the same pattern of change in intratumoral CD8+ TILs as the whole group, whereas the chemo-resistant group (RCB class III) showed no significant change in intratumoral CD8+ TIL infiltration after PST. Survival analyses for each TIL subset as well as their ratios revealed that high levels of intratumoral, stromal, and total CD8+ TIL infiltration after PST were independent predictors of longer patient survival. In subgroup analyses, CD8+ TIL infiltration after PST revealed prognostic significance in the chemo-resistant group but not in the chemo-sensitive group. In conclusion, infiltration of CD8+, CD4+, and FOXP3+ TIL changed after PST in the intratumoral and stromal compartments. Especially, increase of intratumoral CD8+ TILs was associated with chemo-responsiveness. Moreover, CD8+ TIL status in residual tumors after PST may be used as a potential prognostic marker in breast cancer patients who receive PST and provide additional prognostic information to chemo-resistant group.