Phase measuring deflectometry is a powerful measurement tool of optical surfaces, but the measuring accuracy relies on the quality of system calibration. Calibration errors arise from the oversimplified imaging models, error accumulation and amplification, and the bias in numerical optimization. A holistic calibration method is proposed to shorten the error propagation chain. The descriptive prowess of the imaging system is enhanced by calculating each incident ray independently and compensating the systematic errors resulting from the form error of the calibration mirror. Finally, a holonomic framework prior is defined to guarantee the calibration reliability by utilizing the physical constraints of the measurement system. Experimental results demonstrate that the proposed method improves measurement accuracy by at least 38% compared to traditional approaches.