A special data processing method for the terahertz frequency domain reflection spectroscopy system has been proposed to analyze the characteristics of a polar mixed liquid. This novel and practical measurement system is characterized by a simpler optical structure and a tunable output frequency range of 0.1-1 THz. By Hilbert transform, stationary wavelet transform, and time domain zero setting method, the reflection coefficient that has been eliminated by the noise and the Fabry-Pérot effect can be obtained by the self-reference calibration method. Then, the dielectric function of the ethanol/n-hexane and propanol/n-hexane mixtures with different mixture ratios can be extracted by this method. In addition, a significant deviation can be observed between the imaginary part of the measured dielectric function and the ideal calculated value. These results indicate that during the mixing process of polar and nonpolar liquids, the hydroxyl functional groups of alcohols significantly change the molecular arrangement pattern of the mixture. The arrangement pattern will result in the formation of the new permanent dipole moment. This study lays a solid foundation for future research on the microscopic mechanism of intermolecular interaction using terahertz frequency domain reflection spectroscopy.