We propose a swept source-based digital holographic phase microscopy technique. By scanning source wavelength, a series of on-axis interferograms can be obtained for accurate determination of the sample phase using spectral domain interferometry. With these sample spectra, sources of undesirable interference artifacts, often significant in holographic systems, can be identified and avoided by placing the sample signal at a spectral frequency with a clean background. Pathlength sensitivity better than 0.3 nm can, thus, be achieved. The quantitative pathlength image of live sperm cells is obtained with clear identification of morphological features. In addition, the availability of sample spectrum also permits the retrieval of its spectroscopic information. The wavelength-dependent refractive indices of indocyanine green solution are obtained to demonstrate this capability.