The reversibly switchable fluorescent proteins Dronpa, rsFastLime, rsKame, Padron, and bsDronpa feature the same chromophore but display a 40 nm variation in absorption maxima and an only 18 nm variation in emission maxima. In the present contribution, we employ QM/MM models to investigate the mechanism of such remarkably different spectral variations, which are caused by just a few amino acid replacements. We show that the models, which are based on CASPT2//CASSCF level of QM theory, reproduce the observed trends in absorption maxima, with only a 3.5 kcal/mol blue-shift, and in emission maxima, with an even smaller 1.5 kcal/mol blue-shift with respect to the observed quantities. In order to explain the variations across the series, we look at the chromophore's electronic structure change during absorption and emission. Such analysis indicates that a change in charge-transfer character, which is more pronounced during absorption, triggers a cascade of hydrogen-bond-network rearrangements, suggesting preparation to an isomerization event. We also show how the contribution of Arg 89 and Arg 64 residues to the chromophore conformational changes correlate with the spectral variations in absorption and emission. Furthermore, we describe how the conical intersection stability is related to the protein's photophysical properties. While for the Dronpa, rsFastLime, and rsKame triad, the stability correlates with the photoswitching speed, this does not happen for bsDronpa and Padron, suggesting a less obvious photoisomerization mechanism.