In digital holography, the phase is most important, and the quality of the reconstructed phase determines the final reconstructed image effect. However, noise is inevitably introduced in the process of recording the hologram. For regions without object light, the phase has a random distribution, which affects the final phase quality. This kind of noise is called abnormal phase fluctuations in this paper. The correlation between amplitude and phase in digital holography is used to judge whether there is useful phase information. Through structural similarity and the light-dark relationship, a credible probability mask is introduced to extract the phase that needs to be preserved. The simulation and experimental results show that abnormal phase fluctuations are successfully removed, and the useful phase information is retained.