Generative artificial intelligence algorithms have shown to be successful in exploring large chemical spaces and designing novel and diverse molecules. There has been considerable interest in developing predictive models using artificial intelligence for drug-like properties, which can potentially reduce the late-stage attrition of drug candidates or predict the properties of novel AI-designed molecules. Concurrently, it is important to understand the contribution of functional groups toward these properties and modify them to obtain property-optimized lead compounds. As a result, there is an increasing interest in the development of explainable property prediction models. However, current explainable approaches are mostly atom-based, where, often, only a fraction of a fragment is shown to be significant. To address the above challenges, we have developed a novel domain-aware molecular fragmentation approach termed post-processing of BRICS (pBRICS), which can fragment small molecules into their functional groups. Multitask models were developed to predict various properties, including the absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties. The fragment importance was explained using the gradient-weighted class activation mapping (Grad-CAM) approach. The method was validated on data sets of experimentally available matched molecular pairs (MMPs). The explanations from the model can be useful for medicinal chemists to identify the fragments responsible for poor drug-like properties and optimize the molecule. The explainability approach was also used to identify the reason behind false positive and false negative MMP predictions. Based on evidence from the existing literature and our analysis, some of these mispredictions were justified. We propose that the quantity, quality, and diversity of the training data will improve the accuracy of property prediction algorithms for novel molecules.