This paper presents analytical modelling for describing the speckle noise decorrelation in phase data from two- or multiple-wavelength digital holography. A novel expression for the modulus of the coherence factor is proposed for the case of two-wavelength speckle decorrelation from imaging roughness and surface shape through an optical system. The expression permits us to estimate the speckle decorrelation phase noise in surface shape measurements. The theoretical analysis is supported by realistic simulations including both the surface roughness and shape. The results demonstrate the very good agreement between the modulus of the coherence factor estimated with the simulation and the one calculated with theory.