The high speckle contrast in reflectance confocal microscopy is perhaps the most limiting factor on this imaging modality, particularly in high scattering samples such as biological tissues. In this Letter, we propose and numerically analyze a method for speckle reduction that uses simple lateral shifting of the confocal pinhole in several directions, which results in reduced speckle contrast and only a moderate penalty in both lateral and axial resolutions. By simulating free-space electromagnetic wave propagation through a high-numerical-aperture (NA) confocal imaging system, and assuming only single-scattering events, we characterize the 3D point-spread function (PSF) that results from full-aperture pinhole shifting. Simple summation of four different pinhole-shifted images resulted in a 36% reduction in speckle contrast, with reductions of only 17% and 60% in the lateral and axial resolutions, respectively. This method could be particularly useful in noninvasive microscopy for clinical diagnosis, where fluorescence labeling is impractical and high image quality is imperative for achieving accurate diagnosis.