We report the development and implementation of an intraoperative polarization-sensitive optical coherence tomography (PS-OCT) system for enhancing breast cancer detection. A total of 3440 PS-OCT images were intraoperatively acquired from 9 human breast specimens diagnosed by H&E histology as healthy fibro-adipose tissue (n = 2), healthy stroma (n = 2), or invasive ductal carcinoma (IDC, n = 5). A standard OCT-based metric (coefficient of variation (CV)) and PS-OCT-based metrics sensitive to biological tissue from birefringence (i.e., retardation and degree of polarization uniformity (DOPU)) were derived from 398 statistically different and independent images selected by correlation coefficient analysis. We found the standard OCT-based metric and PS-OCT-based metrics were complementary for the differentiation of healthy fibro-adipose tissue, healthy stroma, and IDC. While the CV of fibro-adipose tissue was significantly higher (p<0.001) than those of either stroma or IDC, the CV difference between stroma and IDC was minimal. On the other hand, stroma was associated with significantly higher (p<0.001) retardation and significantly lower (p<0.001) DOPU as compared to IDC. By leveraging the complementary information acquired by the intraoperative PS-OCT system, healthy fibro-adipose tissue, healthy stroma, and IDC can be differentiated with an accuracy of 89.4%, demonstrating the potential of PS-OCT as an adjunct modality for enhanced intraoperative differentiation of human breast cancer.