For spectrometers, baseline drift seriously affects the measurement and quantitative analysis of spectral data. Deep learning has recently emerged as a powerful method for baseline correction. However, the dependence on vast amounts of paired data and the difficulty in obtaining spectral data limit the performance and development of deep learning-based methods. Therefore, we solve these problems from the network architecture and training framework. For the network architecture, a Learned Feature Fusion (LFF) module is designed to improve the performance of U-net, and a three-stage training frame is proposed to train this network. Specifically, the LFF module is designed to adaptively integrate features from different scales, greatly improving the performance of U-net. For the training frame, stage 1 uses airPLS to ameliorate the problem of vast amounts of paired data, stage 2 uses synthetic spectra to further ease reliance on real spectra, and stage 3 uses contrastive learning to reduce the gap between synthesized and real spectra. The experiments show that the proposed method is a powerful tool for baseline correction and possesses potential for application in spectral quantitative analysis.