Magnetic interferential compensation plays a vital role in geomagnetic vector measurement applications. Traditional compensation accounts for only the permanent interferences, induced field interferences, and eddy-current interferences. However, nonlinear magnetic interferences are found, which also have a great impact on measurement, and it cannot be fully characterized by a linear compensation model. This paper proposes a new compensation method based on a back propagation neural network, which can reduce the influence of the linear model on compensation accuracy due to its good nonlinear mapping capabilities. The high-quality network training requires representative datasets, yet it is a common problem in the engineering field. To provide adequate data, this paper adopts a 3D Helmholtz coil to restore the magnetic signal of a geomagnetic vector measurement system. A 3D Helmholtz coil is more flexible and practical than the geomagnetic vector measurement system itself when generating abundant data under different postures and applications. Simulations and experiments are both conducted to prove the superiority of the proposed method. According to the experiment, the proposed method can reduce the root mean square errors of north, east, and vertical components and the total intensity from 73.25, 68.54, 70.45, and 101.77 nT to 23.35, 23.58, 27.42, and 29.72 nT, respectively, compared with the traditional method.