In optical scanning holography (OSH), the coherence properties of the acquired holograms depend on the single-pixel size, i.e., the active area of the photodetector. For the first time, to the best of our knowledge, we have demonstrated coherent, partial coherent, and incoherent three-dimensional (3D) imaging by experiment in such a single-pixel digital holographic recording system. We have found, for the incoherent mode of OSH, in which the detector of the largest active area is applied, the 3D location of a diffusely reflecting object can be successfully retrieved without speckle noise. For the partial coherent mode employing a smaller pixel size of the detector, significant speckles and randomly distributed bright spots appear among the reconstructed images. For the coherent mode of OSH when the size of the pixel is vanishingly small, the bright spots disappear. However, the speckle remains and the signal-to-noise ratio is low.