In terahertz digital holography, the off-axis configuration is the appropriate choice when the investigated object is non-sparse and complex. The limitation of recording distance in the off-axis configuration restricts the imaging quality. Either low-resolution or spectra overlap can potentially occur. We propose an iterative phase-retrieval approach to improve the quality of reconstruction results obtained from an off-axis hologram. One additional capture of object wave intensity is recorded to perform iterative phase retrieval with off-axis reconstruction as the initial guess. Apodization operation can be applied to the object wave intensity capture to suppress undesired border diffraction effects. The image quality using the proposed method has been improved both from simulation and experimental verification.