Intraoperative delineation of breast cancer is a major challenge. An effective breast tissue screening technique may reduce the risk of re-excision during surgery by specifically identifying positive margins. In this study, a high-resolution automated full-field polarization-sensitive optical coherence tomography (FF-PS-OCT) system was developed to classify healthy and malignant human breast tissue from quantitative phase retardation information of the tissues in ex vivo. Twelve breast tissue samples [four healthy, eight malignant (cancerous)] were imaged with the FF-PS-OCT system and the different phase features were extracted from the acquired OCT images (106), based on the differences in the optical signatures of the healthy and malignant tissues. A linear support vector model classifier was trained using 75 images, with a sensitivity of 92.10% and specificity of 89.18% was achieved. Thirty-one images were used to test the model, with a sensitivity of 90.90% and specificity of 85.0% achieved.