The traditional aeromagnetic compensation model does not consider the negative effects of the gradient of the geomagnetic background field. Especially in emergency aeromagnetic surveys, this gradient effect is obvious and can reduce the performance of magnetic compensation. To solve this problem, we propose a method of reducing the gradient effect of the magnetic sensor for aeromagnetic compensation. A magnetic gradient model of the Earth's background field is established for the aeromagnetic compensation, and the gradient effect is then suppressed using altitude data. An experimental platform was built, and flight tests were conducted. Comparison results show that the adoption of the proposed method increased the improvement ratio (IR) of our compensator on two flights from 20.4 and 21.3 to 25.9 and 26.0, with the growth rates of the IR, respectively, reaching 27% and 22%.