Various physical information can be leaked while the encryption algorithm is running in the device. Side-channel analysis exploits these leakages to recover keys. Due to the sensitivity of deep learning to the data features, the efficiency and accuracy of side channel analysis are effectively improved with the application of deep learning algorithms. However, a considerable part of existing reserches are based on traditional neural networks. The effectiveness of key recovery is improved by increasing the size of the network. However, the computational complexity of the algorithm increases accordingly. Problems such as overfitting, low training efficiency, and low feature extraction ability also occur. In this paper, we construct an improved lightweight convolutional neural network based on the feature fusion network. The new network and the traditional neural networks are respectively applied to the side-channel analysis for comparative experiments. The results show that the new network has faster convergence, better robustness and higher accuracy. No overfitting has occurred. A heatmap visualization method was introduced for analysis. The new network has higher heat value and more concentration in the key interval. Side-channel analysis based on feature fusion network has better performance, compared with the ones based on traditional neural networks.