X-ray mirrors with single-digit nanometer height errors are required to preserve the quality of ultra-intense photon beams produced at synchrotron or free electron laser sources. To fabricate suitable X-ray mirrors, accurate metrology data is needed for deterministic polishing machines. Fizeau phase-shifting interferometers are optimized to achieve accurate results under nulled conditions. However, for curved or aspheric mirrors, a limited choice of reference optic often necessitates measurement under non-nulled conditions, which can introduce retrace error. Using experimental measurements of a multi-tilted calibration mirror, we have developed an empirical model of Fizeau retrace error, based on Zernike polynomial fitting. We demonstrate that the model is in good agreement with measurements of ultra-high quality, weakly-curved X-ray mirrors with sags of only a few tens of microns. Removing the predicted retrace error improves the measurement accuracy for full aperture, single shot, Fizeau interferometry to < 2 nm RMS.