Energy relaxation between two electronic states of a molecule is mediated by a set of relevant vibrational states. We describe this fundamental process in a fully quantum mechanical framework based on first principles. This approach explains population transfer rates as well as describes the entire transient absorption signal as vibronic transitions between electronic states. By applying this vibronic energy relaxation approach to carotenoids, we show that β-carotene's transient absorption signal can be understood without invoking the intensely debated S* electronic state. For a carotenoid with longer chain length, we find that vibronic energy relaxation does not suffice to explain all features in the transient absorption spectra, which we relate to an increased ground state structural inhomogeneity. Our modeling approach is generally applicable to photophysical deactivation processes in molecules. As such, it represents a well-founded alternative to data fitting techniques such as global target analysis.