Phase-shifting digital holography is a digital measurement technology of a complex optical field profile that uses focal plane array detectors without the loss of bandwidth. It has been known that the accuracy of phase-shifting digital holography is limited mainly by the phase tolerance of reference. In this Letter, it is revealed that the uncertainty in an expected signal is highly dependent on the phase of the signal itself, as well as the phase tolerance of the reference. Based on the uncertainty analysis, we propose a novel scheme of phase-shifting digital holography that exploits an uncertainty property to enhance the measurement accuracy even under significant reference phase tolerance.