Phase Measuring Deflectometry (PMD) and Structured-Light Modulation Analysis Technique (SMAT) perform effectively in shape and defect measurements of specular objects, but the difficulty of giving consideration to accuracy and speed has also restricted the further development and application of them. Inspired by recent successes of deep learning techniques for computational imaging, we demonstrate for the first time that deep learning techniques can be used to recover high-precision modulation distributions of specular surfaces from a single-frame fringe pattern under SMAT, enabling fast and high-quality defect detection of specular surfaces. This method can also be applied to recover higher-precision phase distributions of specular surfaces from a single-frame fringe pattern under PMD, so as to realize the 3D shape measurement. In this paper, we combine depthwise separable convolution, residual structure and U-Net to build an improved U-Net network. The experimental results prove that the method has excellent performance in the phase and modulation retrieval of specular surfaces, which almost reach the accuracy of the results obtained by ten-step phase-shifting method.