The absorption and emission spectra of most luminescent, pi-conjugated, organic molecules are the mirror image of each other. In some cases, however, this symmetry is severely broken. In the present work, the asymmetry between the absorption and fluorescence spectra in molecular systems consisting of para-linked phenyl rings is studied. The vibronic structure of the emission and absorption bands is calculated from ab initio quantum chemical methods and a subsequent, rigorous Franck-Condon treatment. Good agreement with experiment is achieved. A clear relation can be established between the strongly anharmonic double-well potential for the phenylene ring librations around the long molecular axis and the observed deviation from the mirror image symmetry. Consequences for related compounds and temperature dependent optical measurements are also discussed.