Holographic optical coherence tomography (OCT) is a powerful imaging technique, but its ability to reveal low-reflectivity features is limited. In this study, we performed holographic OCT by incoherently averaging volumes with changing diffuse illumination of numerical aperture (NA) equal to the detection NA. While the reduction of speckle from singly scattered light is only modest, we discovered that speckle from multiply scattered light can be arbitrarily reduced, resulting in substantial improvements in image quality. This technique also offers the advantage of suppressing noises arising from spatial coherence, and can be implemented with a partially spatially incoherent light source for further mitigation of multiple scattering. Finally, we show that although holographic reconstruction capabilities are increasingly lost with decreasing spatial coherence, they can be retained over an axial range sufficient to standard OCT applications.