Physical therapists play a crucial role in guiding patients through effective and safe rehabilitation processes according to medical guidelines. However, due to the therapist-patient imbalance, it is neither economical nor feasible for therapists to provide guidance to every patient during recovery sessions. Automated assessment of physical rehabilitation can help with this problem, but accurately quantifying patients' training movements and providing meaningful feedback poses a challenge. In this paper, an Expert-knowledge-based Graph Convolutional approach is proposed to automate the assessment of the quality of physical rehabilitation exercises. This approach utilizes experts' knowledge to improve the spatial feature extraction ability of the Graph Convolutional module and a Gated pooling module for feature aggregation. Additionally, a Transformer module is employed to capture long-range temporal dependencies in the movements. The attention scores and weight matrix obtained through this approach can serve as interpretability tools to help therapists understand the assessment model and assist patients in improving their exercises. The effectiveness of the proposed method is verified on the KIMORE dataset, achieving state-of-the-art performance compared to existing models. Experimental results also illustrate the interpretability of the method in both spatial and temporal dimensions.