Vision ray calibration provides imaging properties of cameras for application in optical metrology by identifying an independent vision ray for each sensor pixel. Due to this generic description of imaging properties, setups of multiple cameras can be considered as one imaging device. This enables holistic calibration of such setups with the same algorithm that is used for the calibration of a single camera. Obtaining reference points for the calculation of independent vision rays requires knowledge of the parameters of the calibration setup. This is achieved by numerical optimization which comes with high computational effort due to the large amount of calibration data. Using the collinearity of reference points corresponding to individual sensor pixels as the measure of accuracy of system parameters, we derived a cost function that does not require explicit calculation of vision rays. We analytically derived formulae for gradient and Hessian matrix of this cost function to improve computational efficiency of vision ray calibration. Fringe projection measurements using a holistically vision ray calibrated system of two cameras demonstrate the effectiveness of our approach. To the best of our knowledge, neither any explicit description of vision ray calibration calculations nor the application of vision ray calibration in holistic camera system calibration can be found in literature.