With the vigorous development of vision techniques, simultaneous localization and mapping (SLAM) has shown the capability of navigating autonomous robots in global-navigation-satellite-system-denied environments. However, the long-term robust navigation of lightweight autonomous robots in outdoor environments with complex interferences, such as illumination change, dynamic objects, and electromagnetic interference, is still a great challenge. In this paper, a polarization sensor-aided SLAM (POL-SLAM) that can provide absolute heading constraints for pure SLAM is proposed. POL-SLAM is a lightweight, tightly coupled system consisting of a polarization sensor and binocular camera. By means of an initialization that uses a polarization sensor, an absolute heading angle for the entire map is designed. Additionally, an algorithm to eliminate mismatching points using the matching point vector is proposed. The objective function of bundle adjustment is then deduced according to the re-projection error and polarization sensor. The vehicle test shows that the yaw and trajectory accuracies of POL-SLAM are significantly improved compared to pure SLAM. The yaw and trajectory accuracies are increased by 43.1% and 36.6%, respectively. These results indicate that the proposed POL-SLAM can improve the reliability and robustness of pure SLAM and can be used in lightweight autonomous robots in outdoor environments.