Detection of metastatic breast cancer lesions is a challenging task in breast cancer treatment. The recent advancements in deep learning gained attention owing to its robustness, particularly in addressing automated segmentation and classification issues in medical images. In this paper, we proposed a modified Swin Transformer model (mST) integrated with a novel Multi-Level Adaptive Feature Fusion (MLAFF) Module. We constructed a modified Swin Transformer network comprising of a Local Transferable MSA (LT-MSA) and a Global Transferable MSA (GT-MSA) in addition to a Feed Forward Network (FFN). Our novel Multi-Level Adaptive Feature Fusion (MLAFF) module iteratively combines the features throughout multiple transformers. We utilized a pre-trained deep learning model U-Net and trained it on mammography utilizing Transfer Learning for automated segmentation. The proposed method, mST-MLAFF, is used for breast cancer classification into normal, benign, and malignant classes. Our model outperformed comparison methods based on U-Net and Swin Transformer in breast metastatic lesion segmentation on the seven benchmark datasets, namely INBreast, DDSM, MIAS, CBIS-DDSM, MIMBCD-UI, KAU-BCMD, and Mammographic Masses. Our model achieved 98% Dice-Similarity coefficient (DSC) for segmentation and an average of 94.5% accuracy for classification, whereas U-Net based model achieved 92% DSC and Swin Transformer achieved 93% DSC. Extensive performance evaluation of our model on benchmark datasets shows the potential of our model for breast cancer classification.Clinical relevance- This research work is focused on assisting the radiologist in the early detection and classification of breast cancer. A single mammography image is analyzed in less than a minute for automated segmentation and classification into malignant and benign classes.