Fine-grained classification of breast tumors is crucial for early diagnosis and timely treatment. Most fine-grained visual classification approaches focus on learning 'informative' visual patterns, which depend on the attention of the network, instead of 'discriminative' patterns, which interpretably contribute to classification. In this paper, we propose to extract discriminative patterns from informative patterns by utilizing the prior information of the dataset. The proposed method can detect the rough contour of the tumor area without boundary ground-truth guidance. At the same time, different masks are generated from the rough contour to reflect prior information on breast cancer. Moreover, a soft-labeling approach is utilized to replace the original BI-RADS label. Our model is trained using image-level object labels and interprets its results via a rough segmentation of tumor parts. Extensive experiments show that our approach achieves a significant performance increase on our BI-RADS classification dataset.