Light scattering is a pervasive problem in many areas. Recently, deep learning was implemented in speckle reconstruction. To better investigate the key feature extraction and generalization abilities of the networks for sparse pattern reconstruction, we develop the "one-to-all" self-attention armed convolutional neural network (SACNN). It can extract the local and global speckle properties of different types of sparse patterns, unseen glass diffusers, and untrained detection positions. We quantitatively analyzed the performance and generalization ability of the SACNN using scientific indicators and found that, compared with convolutional neural networks, the Pearson correlation coefficient, structural similarity measure, and Jaccard index for the validation datasets increased by more than 10% when SACNN was used. Moreover, SACNN is capable of reconstructing features 75 times beyond the memory effect range for a 120 grits diffuser. Our work paves the way to boost the field of view and depth of field for various sparse patterns with complex scatters, especially in deep tissue imaging.