The involvement of the immune system for the course of breast cancer, as evidenced by varying degrees of lymphocyte infiltration (LI) into the tumor is still poorly understood. The aim of this study was to evaluate the prognostic value of LI in breast cancer samples using microarray-based screening for LI-associated genes. Starting from the observation that most published ER gene signatures are heavily influenced by the LI effect, we developed and applied a novel approach to dissect molecular signatures. Further, a meta-analysis encompassing 1,044 hybridizations showed that LI alone is not sufficient to highlight breast cancer patients with different prognosis. However, for ER positive patients, high LI was associated with shorter survival times, whereas for ER negative patients, high LI is significantly associated with longer survival. Annotation of LI, in addition to ER status, is important for breast cancer patient prognosis and may have implications for the future treatment of breast cancer.