Conventional design methods for computer-generated holograms often rely on the scalar diffraction theory because the calculation effort of rigorous simulations is too high. But for sub-wavelength lateral feature sizes or large deflection angles, the performance of realized elements will show distinct deviations from the expected scalar behavior. We propose a new design method that overcomes this issue by incorporating high-speed semi-rigorous simulation techniques that allow the modeling of light propagation at an accuracy close to the rigorous methods. This includes an approach to solve the inverse problem of calculating a geometric structure that is able to form a certain physical field distribution.