We report a single-shot three-dimensional (3D) topographical imaging method, optical coherence factor (OCF) imaging, which uses optical coherence as the contrast mechanism to acquire the surface height (${z}$z-direction) information of an object. A 4-f imaging system records the light field reflected from the surface of the object. The illumination of the imaging system comes from a laser source with the optical coherence length comparable to the depth of field (DoF) of the optical system. Off-axis holographic recording is used to retrieve the coherence factor from the interference fringes, which is then converted to ${z}$z-direction information. In this experiment, we validate our 3D imaging results comparing them to axial scanning full-field optical coherence tomography images. We also analyze the contrast mechanism of OCF and show that it is able to provide additional information over conventional coherent and incoherent imaging using the same imaging setup. This single-shot computationally efficient method may have potential applications in industrial quality control inspection.