Aiming at the engineering requirements of reducing the volume and improving the vibration characteristics of gears in the three-cylinder engine balanced system, a design and optimization method of gear structure is proposed based on the Design of Experiments (DOE) and proxy models. The paper analyzes the structure improvement process based on the gear design model and technical index requirements. By designing the plane characteristics of the weight-increasing module, the weight-reducing module and the elastic module, the calculation model of balance performance indices such as mass, moment of inertia and unbalance of new balanced vibration reduction gear are constructed. Then, a more efficient design method is proposed based on dynamic simulation and multidisciplinary optimization design platform (Isight). The results show that the new design method of gear structure can effectively reduce the structure improvement cycle. At the same time, the improved structure can reduce the thickness of the weight-increasing module by 6.3 mm and the vibration attenuation by more than 90%.