Computer-Controlled Optical Surfacing (CCOS) has been greatly developed and widely used for precision optical fabrication in the past three decades. It relies on robust dwell time solutions to determine how long the polishing tools must dwell at certain points over the surfaces to achieve the expected forms. However, as dwell time calculations are modeled as ill-posed deconvolution, it is always non-trivial to reach a reliable solution that 1) is non-negative, since CCOS systems are not capable of adding materials, 2) minimizes the residual in the clear aperture 3) minimizes the total dwell time to guarantee the stability and efficiency of CCOS processes, 4) can be flexibly adapted to different tool paths, 5) the parameter tuning of the algorithm is simple, and 6) the computational cost is reasonable. In this study, we propose a novel Universal Dwell time Optimization (UDO) model that universally satisfies these criteria. First, the matrix-based discretization of the convolutional polishing model is employed so that dwell time can be flexibly calculated for arbitrary dwell points. Second, UDO simplifies the inverse deconvolution as a forward scalar optimization for the first time, which drastically increases the solution stability and the computational efficiency. Finally, the dwell time solution is improved by a robust iterative refinement and a total dwell time reduction scheme. The superiority and general applicability of the proposed algorithm are verified on the simulations of different CCOS processes. A real application of UDO in improving a synchrotron X-ray mirror using Ion Beam Figuring (IBF) is then demonstrated. The simulation indicates that the estimated residual in the 92.3 mm × 15.7 mm CA can be reduced from 6.32 nm Root Mean Square (RMS) to 0.20 nm RMS in 3.37 min. After one IBF process, the measured residual in the CA converges to 0.19 nm RMS, which coincides with the simulation.