As an alternative solution to the lack of cost-effective multipixel terahertz cameras, terahertz single-pixel imaging that is free from pixel-by-pixel mechanical scanning has been attracting increasing attention. Such a technique relies on illuminating the object with a series of spatial light patterns and recording with a single-pixel detector for each one of them. This leads to a trade-off between the acquisition time and the image quality, hindering practical applications. Here, we tackle this challenge and demonstrate high-efficiency terahertz single-pixel imaging based on physically enhanced deep learning networks for both pattern generation and image reconstruction. Simulation and experimental results show that this strategy is much more efficient than the classical terahertz single-pixel imaging methods based on Hadamard or Fourier patterns, and can reconstruct high-quality terahertz images with a significantly reduced number of measurements, corresponding to an ultra-low sampling ratio down to 1.56%. The efficiency, robustness and generalization of the developed approach are also experimentally validated using different types of objects and different image resolutions, and clear image reconstruction with a low sampling ratio of 3.12% is demonstrated. The developed method speeds up the terahertz single-pixel imaging while reserving high image quality, and advances its real-time applications in security, industry, and scientific research.