Breast conserving surgery aims at the complete removal of malignant lesions while minimizing healthy tissue loss. To ensure the balance between complete resection of the cancer and conservation of healthy tissue, intra-operative margin assessment is necessary. Deep ultraviolet (DUV) fluorescence scanning microscope provides fast whole-surface-imaging (WSI) of excised tissue with contrast between malignant and normal tissues. Then, an automated breast cancer classification method on DUV images is required for intra-operative margin assessment. Deep learning shows the promising results in breast cancer classification, but limited DUV image dataset poses overfitting challenge to train the robust network. To tackle this challenge, we partition the DUV WSI image into small patches and extract pathological features for each patch from a pre-trained network using a transfer learning approach. We feed pathological features into a decision-tree-based classifier and fuse patch-level classification results based on regional importance to determine malignant or benign WSI. Experimental results on 60 DUV images show that our proposed method outperforms the standard deep learning classification in terms of improving the classification performance and identifying cancerous regions.