Quantitative assessment of hand function can assist therapists in providing appropriate rehabilitation strategies, which plays an essential role in post-stroke rehabilitation. Conventionally, the assessment process relies heavily on clinical experience and lacks quantitative analysis. To quantitatively assess the hand motor function of patients with post-stroke hemiplegia, this study proposes a novel multi-modality fusion assessment framework. This framework includes three components: the kinematic feature extraction based on a graph convolutional network (HAGCN), the surface electromyography (sEMG) signal processing based on a multi-layer long short term memory (LSTM) network, and the quantitative assessment based on the multi-modality fusion. To the best of the authors' knowledge, this is the first study of applying a graph convolution network to assess the hand motor function. We also collect the kinematic data and sEMG data from 70 subjects who completed 28 types of hand movements. Therapists first graded patients using traditional motor assessment scales (Brunnstrom Scale and Fugl-Meyer Assessment Scale) and further refined the patient's motor assessment result by their experience. Then, we trained the HAGCN and LSTM networks and quantitatively assessed each patient based on the proposed assessment framework. Finally, the Spearman correlation coefficient (SC) between the assessment result of this study and the traditional scale are 0.908 and 0.967, demonstrating a significant correlation between the proposed assessment and the traditional scale scores. In addition, the SC value between the score of this study and the refined hand motor function is 0.997, indicating the "ceiling effect" of some traditional scales can be avoided.