Low phosphorus use efficiency (PUE) is one of the main problems of acidic soil that limit the crop growth. Therefore, in the present study, we investigated the response of crop yield and PUE to the long-term application of fertilizers and quicklime (CaO) in the acidic soil under wheat-maize rotation system. Treatments included, CK (no fertilization), NP (inorganic nitrogen and P fertilization), NPK (inorganic N, P and potassium fertilization), NPKS (NPK + straw return), NPCa (NP + lime), NPKCa (NPK + lime) and NPKSCa (NPKS + lime). Results showed that, fertilizer without lime treatments, significantly (p ≤ 0.05) decreased soil pH and crop yield, compared to the fertilizer with lime treatments during the period of 2012-2018. Average among years, compared to the CK treatment, wheat grain yield increased by 138%, 213%, 198%, 547%, 688% and 626%, respectively and maize yield increased by 687%, 1887%, 1651%, 2605%, 5047% and 5077%, respectively, under the NP, NPK, NPKS, NPCa, NPKCa and NPKSCa treatments. Lime application significantly increased soil exchangeable base cations (Ca2+ and Mg2+) and decreased Al3+ cation. Compared to the NP treatment, phosphorus use efficiency (PUE) increased by 220%, 212%, 409%, 807% and 795%, respectively, under the NPK, NPKS, NPCa, NPKCa and NPKSCa treatments. Soil pH showed significant negative relationship with exchangeable Al3+ and soil total N. While, soil pH showed significant (p ≤ 0.05) positive relationship with exchangeable Ca2+, PUE and annual crop yield. PUE was highly negatively correlated with soil exchangeable Al3+. In addition, soil exchangeable Ca2+, pH, exchangeable Al3+ and available N were the most influencing factors of crop yield. Therefore, we concluded that lime application is an effective strategy to mitigate soil acidification and to increase PUE through increasing exchangeable base cations and reducing the acidic cations for high crop yield in acidic soil.