Incoherent digital holography (IDH) enables passive 3D imaging under spatially incoherent light; however, the reconstructed images are seriously affected by detector noise. Herein, we derive theoretical sampling requirements for IDH to reduce this noise via simple postprocessing based on spatial averaging. The derived theory provides a significant insight that the sampling requirements vary depending on the recording geometry. By judiciously choosing the number of pixels used for spatial averaging based on the proposed theory, noise can be reduced without losing spatial resolution. We then experimentally verify the derived theory and show that the associated adaptive spatial averaging technique is a practical and powerful way of improving 3D image quality.