In this Letter, we propose an attention-based neural network specially designed for the challenging task of polarimetric image denoising. In particular, the channel attention mechanism is used to effectively extract the features underlying the polarimetric images by rescaling the contributions of channels in the network. In addition, we also design the adaptive polarization loss to make the network focus on the polarization information. Experiments show that our method can well restore the details flooded by serious noise and outperforms previous methods. Moreover, the underlying mechanism of channel attention is revealed visually.