The wrapped phase patterns of objects with varying materials exhibit uneven gray values. Phase unwrapping is a tricky problem from a single wrapped phase pattern in electronic speckle pattern interferometry (ESPI) due to the gray unevenness and noise. In this paper, we propose a convolutional neural network (CNN) model named UN-PUNet for phase unwrapping from a single wrapped phase pattern with uneven grayscale and noise. UN-PUNet leverages the benefits of a dual-branch encoder structure, a multi-scale feature fusion structure, a convolutional block attention module, and skip connections. Additionally, we have created an abundant dataset for phase unwrapping with varying degrees of unevenness, fringe density, and noise levels. We also propose a mixed loss function MS_SSIM + L2. Employing the proposed dataset and loss function, we can successfully train the UN-PUNet, ultimately realizing effective and robust phase unwrapping from a single uneven and noisy wrapped phase pattern. We evaluate the performance of our method on both simulated and experimental ESPI wrapped phase patterns, comparing it with DLPU, VUR-Net, and PU-M-Net. The unwrapping performance is assessed quantitatively and qualitatively. Furthermore, we conduct ablation experiments to evaluate the impact of different loss functions and the attention module utilized in our method. The results demonstrate that our proposed method outperforms the compared methods, eliminating the need for pre-processing, post-processing procedures, and parameter fine-tuning. Moreover, our method effectively solves the phase unwrapping problem while preserving the structure and shape, eliminating speckle noise, and addressing uneven grayscale.