Heart sound analysis plays an important role in early detecting heart disease. However, manual detection requires doctors with extensive clinical experience, which increases uncertainty for the task, especially in medically underdeveloped areas. This paper proposes a robust neural network structure with an improved attention module for automatic classification of heart sound wave. In the preprocessing stage, noise removal with Butterworth bandpass filter is first adopted, and then heart sound recordings are converted into time-frequency spectrum by short-time Fourier transform (STFT). The model is driven by STFT spectrum. It automatically extracts features through four down sample blocks with different filters. Subsequently, an improved attention module based on Squeeze-and-Excitation module and coordinate attention module is developed for feature fusion. Finally, the neural network will give a category for heart sound waves based on the learned features. The global average pooling layer is adopted for reducing the model's weight and avoiding overfitting, while focal loss is further introduced as the loss function to minimize the data imbalance problem. Validation experiments have been conducted on two publicly available datasets, and the results well demonstrate the effectiveness and advantages of our method.