In breast cancer, 20%-30% of cases require a second surgery because of incomplete excision of malignant tissues. Therefore, to avoid the risk of recurrence, accurate detection of the cancer margin by the clinician or surgeons needs some assistance. In this paper, an automated volumetric analysis of normal and breast cancer tissue is done by a machine learning algorithm to separate them into two classes. The proposed method is based on a support-vector-machine-based classifier by dissociating 10 features extracted from the A-line, texture, and phase map by the swept-source optical coherence tomographic intensity and phase images. A set of 88 freshly excised breast tissue [44 normal and 44 cancers (invasive ductal carcinoma tissues)] samples from 22 patients was used in our study. The algorithm successfully classifies the cancerous tissue with sensitivity, specificity, and accuracy of 91.56%, 93.86%, and 92.71% respectively. The present computational technique is fast, simple, and sensitive, and extracts features from the whole volume of the tissue, which does not require any special tissue preparation nor an expert to analyze the breast cancer as required in histopathology. Diagnosis of breast cancer by extracting quantitative features from optical coherence tomographic images could be a potentially powerful method for cancer detection and would be a valuable tool for a fine-needle-guided biopsy.