Quantitative phase imaging (QPI) exploits sample-induced changes in the optical field to analyze biological specimens in a label-free manner. However, the quantitative nature of QPI makes it susceptible to optical aberrations. We propose a method for calibrating pupil aberrations by imaging a sample of interest. The proposed method recovers pupil information by utilizing the cross-spectral density between optical fields at different incident angles and allows both thin and weakly scattering three-dimensional samples for calibration. We experimentally validate the proposed method by imaging various samples, including a resolution target, breast tissue, and a polystyrene bead, and demonstrate aberration-free two- and three-dimensional QPI.