Soliton microcombs generated by the third-order nonlinearity of microresonators exhibit high coherence, low noise, and stable spectra envelopes, which can be designed for many applications. However, conventional dispersion engineering based design methods require iteratively solving Maxwell's equations through time-consuming electromagnetic field simulations until a local optimum is obtained. Moreover, the overall inverse design from soliton microcomb to the microcavity geometry has not been systematically investigated. In this paper, we propose a high accuracy microcomb-to-geometry inverse design method based on the genetic algorithm (GA) and deep neural network (DNN), which effectively optimizes dispersive wave position and power. The method uses the Lugiato-Lefever equation and GA (LLE-GA) to obtain second- and higher-order dispersions from a target microcomb, and it utilizes a pre-trained forward DNN combined with GA (FDNN-GA) to obtain microcavity geometry. The results show that the dispersive wave position deviations of the inverse designed MgF2 and Si3N4 microresonators are less than 0.5%, and the power deviations are less than 5 dB, which demonstrates good versatility and effectiveness of our method for various materials and structures.