High-quality denoising of optical interference images usually requires preliminary prediction of the noise level. Although blind denoising can filter the image at the pixel level without noise prediction, it inevitably loses a significant amount of phase information. This paper proposes a fast and high-quality denoising algorithm for optical interference images that combines the merits of a principal component analysis (PCA) and residual neural networks. The PCA is used to analyze the image noise and, in turn, establishes an accurate mapping between the estimated and true noise levels. The mapping helps to select a suitable residual neural network model for image processing, which maximizes the retention of image information and reduces the effect of noise. In addition, a comprehensive evaluation factor to account for the time complexity and denoising effect of the algorithm is proposed, since time complexity can be a dominant concern in some cases of actual measurement. The performance of the denoising algorithm and the effectiveness of the evaluation criterion are demonstrated to be high by processing a set of optical interference images and benchmarking with other denoising algorithms. The proposed algorithm outperforms the previously reported counterparts in a specific area of optical interference image preprocessing and provides an alternative paradigm for other denoising problems of optics, such as holograms and structured light measurements.