The Dynamic Optical Breast Imaging technology is a promising breast cancer diagnosis approach based on tumor angiogenesis or vascular change detection which generally causes an increased blood volume in tumor. By applying sustained pressure to breast tissue under red light, the tissue with abnormal vascularization exhibits different dynamic behaviors of optical properties compared with normal breast tissue. In this paper, we explore a comprehensive classification method to discriminate malignant from benign lesions based on the Dynamic Optical breast Imaging technology. Firstly, we propose an automatic Point of Reference (POR) and Point of Interest (POI) selection algorithm from input images for following comparison procedures. Secondly, an automatic Margin Shape Patterns (MSP) recognition algorithm for Region of Interest (ROI) is explored. Furthermore, we define a new significant temporal and contextual feature named Temporal Curves Similarity Index (TCSI), with the aim of better characterizing and quantifying the difference inside the same breast. Finally, Support Vector Machine (SVM) is utilized for comprehensive classification. Experimental results, sensitivity of 91% and specificity of 71%, on our clinical database verify the efficiency of the proposed method.