Speckle reduction remains a critical issue for ultrasound image processing and analysis. The nonlocal means (NLM) filter has recently attached much attention due to its competitive despeckling performance. However, the existing NLM methods usually determine the similarity between two patches by directly utilizing the gray-level information of the noisy image, which renders it difficult to represent the structural similarity of ultrasound images effectively. To address this problem, the NLM method based on the simple deep learning baseline named PCANet is proposed by introducing the intrinsic features of image patches extracted by this network rather than the pixel intensities into the pixel similarity computation. In this approach, the improved two-stage PCANet is proposed by using Parametric Rectified Linear Unit (PReLU) activation function instead of the binary hashing and block histograms in the original PCANet. This model is firstly trained on the ultrasound database to learn the convolution kernels. Then, the trained PCANet is utilized to extract the intrinsic features from the image patches in the pre-denoised version of the noisy image to be despeckled. These obtained features are concatenated together to determine the structural similarity between image patches in the NLM method, based on which the weighted mean of all pixels in a search window is computed to produce the final despeckled image. Extensive experiments have been conducted on a variety of images to demonstrate the superiority of the proposed method over several well-known despeckling algorithm and the PCANet based NLM method using ReLU function and sigmoid function. Visual inspection indicates that the proposed method outperforms the compared methods in reducing speckle noise and preserving image details. The quantitative comparisons show that among all the evaluated methods, our method produces the best structural similarity index metrics (SSIM) values for the synthetic image, as well as the highest equivalent number of looks (ENL) value for the simulated image and the clinical ultrasound images.