When calibrating a line-structured light vision system using a planar target, noise easily affects the solution of the coordinates of light stripe points at the camera coordinate frame. Therefore, the planar target must be placed in the measurement space many times to capture more target images for improving calibration stability and achieving relatively high calibration accuracy. This complicates the calibration process. This paper proposes a calibration method considering the measurement baselines of a planar target. The planar target is placed only two times, and two target images are captured correspondingly. A three-point subset is made up of the two calibration points that form the measurement baseline with the longest 2D projection and any other calibration point. In this way, it is less affected by noise when using the three-point subsets to establish the equations. Then, we use the lengths of the measurement baselines provided by all three-point subsets and their 2D projections to solve the coordinates of light stripe points at the camera coordinate frame more accurately to calibrate the line-structured light vision system. Both the simulation and actual experiment results demonstrate the feasibility of our method. Based on our calibration method, the RMS error is 0.035 mm for length measurement and 0.054 mm for height measurement. Compared with other existing methods, our method needs only two target images. It can also achieve more accurate calibration results than the other methods. In addition, our calibration method increases the applicability of the line-structured light measurement method by reducing the number of target swings.