For lensless diffraction imaging, it is a challenging dilemma to achieve a large field of view (FOV) and high resolution with a small amount of data at the same time. Ptychography can reconstruct the high-resolution image and illumination light simultaneously. But the illumination is limited to a small size by a probe in typical ptychography. For large samples, it takes much time to collect abundant patterns and has strict requirements for the computing power of computers. Another widely applied method, multi-height measurement, can realize a wide FOV with several holograms. But, the recovered image is easily destroyed by the background noise. In this Letter, a lensless diffraction imaging method by three-dimensional scanning is proposed. All positions of the object are different in three directions instead of scanning schemes only on a plane or along the optic axis, so more diversity of diffraction information is obtained. We apply the illumination without the limit of a confined aperture, which means that the imaging FOV of a pattern is equal to the size of the utilized image sensor. In comparison with the multi-height method, our method can separate the illumination background noise from the retrieved object. Consequently, the proposed method realized high resolution and contrast, large FOV, and the removal of background noise simultaneously. Experimental validations and comparisons with other methods are presented.