For structure parameter calibration of a binocular stereo vision system equipped with polarizers, the optimal calibration polarization angle needs to be determined. There are no corresponding solutions for the determination of the optimal polarization angle of structure parameter calibration. Furthermore, existing research considers the polarization angle that causes the image to possess the lowest brightness (gray value) as the optimal polarization angle. This reduces image contrast, and eventually the texture information of the image is lost, which also affects the accuracy of feature extraction. In this paper, we propose a new calibration method for the structure parameters of a binocular stereo vision system equipped with polarizers. We calculated the pose of the target relative to each camera for different polarization angles. The sum of the object-space errors corresponding to each polarization angle was considered as the evaluation criterion to determine the optimal calibration polarization angle. The calibration of structure parameters was finished using images captured on the premise of the optimal calibration polarization angle. This angle can also be considered as the reference of the polarization angle for measurement. Experiment results show that using the calibration results of our method, the reconstructed length error of a 275×200mm target was less than ±0.052mm, the reconstructed linear displacement error was less than ±0.048mm for the range of 0-30 mm, and the reconstructed rotary angle error was less than ±0.048∘ for the range of-30∘-30∘.