This paper proposes a novel local feature descriptor coined as a local instant-and-center-symmetric neighbor-based pattern of the extrema-images (LINPE) to detect breast abnormalities in thermal breast images. It is a hybrid descriptor that combines two different feature descriptors: one is the inverse-probability difference extrema (IpDE), and another is the local instant and center-symmetric neighbor-based pattern (LICsNP). IpDE is developed to compute the intensity-inhomogeneity-invariant feature-based image of the breast thermogram. Besides, the LICsNP is intended to capture the local microstructure pattern information in the IpDE image. A new paradigm, named Broad Learning (BL) network, is introduced here as a classifier to differentiate the healthy and sick breast thermograms efficiently. The efficacy of the proposed system is quantitatively validated on the images of DMR-IR and DBT-TU-JU databases. Extensive experimentation on these databases with an average accuracy of 96.90% and 94%, respectively, justifies proposed system's superiority in the differentiation of healthy and sick breast thermograms over the other related existing state-of-the-art methods. The proposed system also performs consistently in the presence of noise and rotational changes.