Accurate prediction of protein-ligand binding affinities can significantly advance the development of drug discovery. Several graph neural network (GNN)-based methods learn representations of protein-ligand complexes via modeling intermolecule interactions and spatial structures (e.g., distances and angles) of complexes. However, these methods fail to emphasize the importance of bonds and learn hierarchical structures of complexes, which are significant for binding affinity prediction. In this article, we propose the structure-aware graph attention diffusion network (SGADN) to incorporate both distance and angle information for efficient spatial structure learning. We model complexes as line graphs with distance and angle information, focusing on bonds as nodes. Then we perform line graph attention diffusion layers (LGADLs) on line graphs to explore long-range bond node interactions and enhance spatial structure learning. Furthermore, we propose an attentive pooling layer (APL) to refine the hierarchical structures in complexes. Extensive experimental studies on two benchmarks demonstrate the superiority of SGADN for binding affinity prediction.