The wrapped phase patterns obtained from an object composed of different materials have uneven gray values. In this paper, we improve the dilated-blocks-based deep convolution neural network (DBDNet) and build a new dataset for restoring the uneven gray values of uneven wrapped phase patterns as well as eliminating the speckle noise. In our method, we improve the structure of dilated blocks in DBDNet to enhance the ability of obtaining full scales of gray values and speckle noise information in the uneven phase patterns. We use the combined MS_SSIM+L1 loss function to improve the denoising and restoration performance of our method. We compare three representative networks ResNet-based, ADNet, and BRDNet in denoising with our proposed method. We test the three compared methods and our method on one group of computer-simulated and one group of experimentally obtained uneven noisy wrapped phase patterns from a dynamic measurement. We also conduct the ablation experiments on the improved model structure and the combined loss function used in our method. The denoising performance has been evaluated quantitatively and qualitatively. The denoising results demonstrate that our proposed method can reduce high speckle noise, restore the uneven gray values of wrapped phase patterns, and get better results than the compared methods.