Multimodal data play an important role in the diagnosis of brain diseases. This study constructs a whole-brain functional connectivity network based on functional MRI data, uses non-imaging data with demographic information to complement the classification task for diagnosing subjects, and proposes a multimodal and across-site WL-DeepGCN-based method for classification to diagnose autism spectrum disorder (ASD). This method is used to resolve the existing problem that deep learning ASD identification cannot efficiently utilize multimodal data. In the WL-DeepGCN, a weight-learning network is used to represent the similarity of non-imaging data in the latent space, introducing a new approach for constructing population graph edge weights, and we find that it is beneficial and robust to define pairwise associations in the latent space rather than the input space. We propose a graph convolutional neural network residual connectivity approach to reduce the information loss due to convolution operations by introducing residual units to avoid gradient disappearance and gradient explosion. Furthermore, an EdgeDrop strategy makes the node connections sparser by randomly dropping edges in the raw graph, and its introduction can alleviate the overfitting and oversmoothing problems in the DeepGCN training process. We compare the WL-DeepGCN model with competitive models based on the same topics and nested 10-fold cross-validation show that our method achieves 77.27% accuracy and 0.83 AUC for ASD identification, bringing substantial performance gains.