With the rapid evolution of synchrotron X-ray sources, the demand for high-precision X-ray mirrors has greatly increased. Single nanometer profile error is required to keep imaging capability at the diffraction limit. Ion Beam Figuring (IBF), as a highly deterministic surfacing technique, has been used for ultra-precision finishing of mirrors. One crucial step that guides the IBF process is dwell time calculation. A valid dwell time solution should be non-negative and duplicate the shape of the desired removal map. Another important aspect is to minimize the total dwell time. In this study, we propose a Robust Iterative Fourier Transform-based dwell time Algorithm (RIFTA) that automatically fulfills these requirements. First, the thresholded inverse filtering in Fourier transform-based deconvolution is stabilized and automated by optimizing the threshold value using the Nelder-Mead simplex algorithm. Second, a novel two-level iterative scheme is proposed to guarantee the minimized total dwell time with its non-negativity at each dwell point. Third, a bicubic resampling is employed to flexibly adapt the calculated dwell time map to any IBF process intervals. The performance of RIFTA is first studied with simulation, followed by a comparison with the other state-of-the-art dwell time algorithms. We then demonstrate with an experiment that, using the dwell time calculated by the RIFTA, the total dwell time is shortened by a factor of two and the RMS in a 5 × 50 mm clear aperture was reduced from 3.4 nm to 1.1 nm after one IBF run, which proves the effectiveness and the efficiency of the proposed algorithm.