Achieving effectively adiabatic dynamics is a ubiquitous goal in almost all areas of quantum physics. Here, we study the speed with which a quantum system can be driven when employing transitionless quantum driving. As a main result, we establish a rigorous link between this speed, the quantum speed limit, and the (energetic) cost of implementing such a shortcut to adiabaticity. Interestingly, this link elucidates a trade-off between speed and cost, namely, that instantaneous manipulation is impossible as it requires an infinite cost. These findings are illustrated for two experimentally relevant systems-the parametric oscillator and the Landau-Zener model-which reveal that the spectral gap governs the quantum speed limit as well as the cost for realizing the shortcut.