Atmospheric turbulence, a pervasive and complex physical phenomenon, challenges optical imaging across various applications. This paper presents the Alternating Spatial-Frequency (ASF)-Transformer, a learning-based method for neutralizing the impact of atmospheric turbulence on optical imaging. Drawing inspiration from split-step propagation and correlated imaging principles, we propose the Alternating Learning in Spatial and Frequency domains (LASF) mechanism. This mechanism utilizes two specially designed transformer blocks that alternate between the spatial and Fourier domains. Assisted by the proposed patch FFT loss, our model can enhance the recovery of intricate textures without the need for generative adversarial networks (GANs). Evaluated across diverse test mediums, our model demonstrated state-of-the-art performance in comparison to recent methods. The ASF-Transformer diverges from mainstream GAN-based solutions, offering a new strategy to combat image degradation introduced by atmospheric turbulence. Additionally, this work provides insights into neural network architecture by integrating principles from optical theory, paving the way for innovative neural network designs in the future.