Annealing has proven highly successful in finding minima in a cost landscape. Yet, depending on the landscape, systems often converge towards local minima rather than global ones. In this Letter, we analyze the conditions for which annealing is approximately successful in finite time. We connect annealing to stochastic thermodynamics to derive a general bound on the distance between the system state at the end of the annealing and the ground state of the landscape. This distance depends on the amount of state updates of the system and the accumulation of nonequilibrium energy, two protocol and energy landscape-dependent quantities which we show are in a trade-off relation. We describe how to bound the two quantities both analytically and physically. This offers a general approach to assess the performance of annealing from accessible parameters, both for simulated and physical implementations.