In fringe projection profilometry, errors related to projectors are not easy to compensate for, as a projector is much more difficult to calibrate than a camera. Immune to projector errors, the depth recovering method based on pixel cross-ratio invariance enables circumventing this issue by calculating the depth of a point from the shift of its camera pixel instead of from its fringe phase. With this existing technique, however, one has to search three reference phase maps along epipolar lines for pixels having the same phases. Doing so increases the measurement time significantly. To improve measurement efficiency, this paper derives, from the pixel cross-ratio invariance, a generic function representing the relationship between the depths and the corresponding pixel shifts and suggests a calibration method for determining its coefficient matrices. Using this function allows us to recover object depths just by searching a single reference phase map, thus reducing the time duration for data processing to about one-third. Besides, different from the previous method, which depends on exactly three reference phase maps, the proposed method calculates the function coefficients from more reference phase maps in the least-squares sense and denoises the benchmark reference phases by use of averaging technique, thus improving measurement accuracy. Experimental results demonstrate this method to be effective in improving measurement accuracy and efficiency.