A spacecraft-borne optical navigation camera is one of the key instruments for optical autonomous navigation, and the line of sight (LOS) of the camera directly affects the accuracy of navigation. We developed an on-orbit calibration approach for a navigation camera to ensure the accuracy, in which a stepwise calibration is performed, external parameters estimated, and internal parameters estimated in a generalized camera frame determined by external parameters. In addition, we proposed a batch and sequential on-orbit estimation method to save on-orbit computing power, and established a strategy to reject misidentified reference stars while keeping consistency in on-orbit parameters estimation. After the calibration, the accuracy of LOS in inertial frame satisfied the need in optical autonomous navigation. The results have been proven precise and robust in three experiments.