Reflection removal is of great significance for high-level computer vision tasks. Most existing methods separate reflections relying heavily on the quality of intermediate prediction or under certain special constraints. However, these methods ignore the inherent correlation between the background and reflection, which may lead to unsatisfactory results with undesired artifacts. Polarized images contain unique optical characteristics that can facilitate reflection removal. In this paper, we present, to the best of our knowledge, a novel two-stage polarized image reflection removal network with difference feature attention guidance. Specifically, our model takes multi-channel polarized images and Stokes parameters as input and utilizes the optical characteristics of reflected and transmitted light to alleviate the ill-posed nature. It adopts a simple yet effective two-stage structure that first predicts the reflection layer and then refines the transmission layer capitalizing on the special relationship between reflection and transmission light. The difference feature attention guidance module (DFAG) is elaborated to diminish the dependence on intermediate consequences and better suppress reflection. It mitigates the reflection components from the observation and generates the supplement and enhancement to the transmission features. Extensive experiments on the real-world polarized dataset demonstrate the superiority of our method in comparison to the state-of-the-art methods.