Predicting the binding affinity of drug target is essential to reduce drug development costs and cycles. Recently, several deep learning-based methods have been proposed to utilize the structural or sequential information of drugs and targets to predict the drug-target binding affinity (DTA). However, methods that rely solely on sequence features do not consider hydrogen atom data, which may result in information loss. Graph-based methods may contain information that is not directly related to the prediction process. Additionally, the lack of structured division can limit the representation of characteristics. To address these issues, we propose a multimodal DTA prediction model using graph local substructures, called MLSDTA. This model comprehensively integrates the graph and sequence modal information from drugs and targets, achieving multimodal fusion through a cross-attention approach for multimodal features. Additionally, adaptive structure aware pooling is applied to generate graphs containing local substructural information. The model also utilizes the DropNode strategy to enhance the distinctions between different molecules. Experiments on two benchmark datasets have shown that MLSDTA outperforms current state-of-the-art models, demonstrating the feasibility of MLSDTA.