A remarkable feature of quantum many-body systems is the orthogonality catastrophe that describes their extensively growing sensitivity to local perturbations and plays an important role in condensed matter physics. Here we show that the dynamics of the orthogonality catastrophe can be fully characterized by the quantum speed limit and, more specifically, that any quenched quantum many-body system, whose variance in ground state energy scales with the system size, exhibits the orthogonality catastrophe. Our rigorous findings are demonstrated by two paradigmatic classes of many-body systems-the trapped Fermi gas and the long-range interacting Lipkin-Meshkov-Glick spin model.