Speech assessment is an important part of the rehabilitation process for patients with aphasia (PWA). Mandarin speech lucidity features such as articulation, fluency, and tone influence the meaning of the spoken utterance and overall speech clarity. Automatic assessment of these features is important for an efficient assessment of the aphasic speech. Hence, in this paper, a standardized automatic speech lucidity assessment method for Mandarin-speaking aphasic patients using a machine learning based technique is presented. The proposed assessment method adopts the Chinese Rehabilitation Research Center Aphasia Examination (CRRCAE) standard as a guideline. Quadrature based high-resolution time-frequency images with a convolutional neural network (CNN) are utilized to develop a method that can map the relationship between the severity level of aphasic patients' speech and the three speech lucidity features. The results show a linear relationship with statistically significant correlations between the normalized true-class output activations (TCOA) of the CNN model and patients' articulation, fluency, and tone scores, i.e., 0.71 (p < 0.001), 0.60 (p < 0.001) and 0.58 (p < 0.001), respectively. The linearity of the proposed Mandarin aphasic speech assessment method and its significant correlation with the speech severity levels show the efficacy of the method in predicting the severity of impaired Mandarin speech. The outcome of this research envisages assisting speech-language pathologists in Mandarin-speech impairment assessment and promoting early support discharge; hence could alleviate the stress that the healthcare system is currently experiencing in China nationwide. The framework of the proposed Mandarin aphasic speech assessment method can be readily extended to other languages.