We investigate the nature of the S* excited state in carotenoids by performing a series of pump-probe experiments with sub-20 fs time resolution on spirilloxanthin in a polymethyl-methacrylate matrix varying the sample temperature. Following photoexcitation, we observe sub-200 fs internal conversion of the bright S2 state into the lower-lying S1 and S* states, which in turn relax to the ground state on a picosecond time scale. Upon cooling down the sample to 77 K, we observe a systematic decrease of the S*/S1 ratio. This result can be explained by assuming two thermally populated ground state isomers. The higher lying one generates the S* state, which can then be effectively frozen out by cooling. These findings are supported by quantum chemical modeling and provide strong evidence for the existence and importance of ground state isomers in the photophysics of carotenoids.