We used a systematic approach to shed light on the inherent differences in perylenes, namely monoimides versus diimides, including coplanarity and dipole moment, and their impact on singlet fission (SF) by designing, synthesizing, and probing a full fledged series of phenylene- and naphthalene-linked dimers. Next to changing the functionality of the perylene core, we probed the effect of the spacers and their varying degrees of rotational freedom, molecular electrostatic potentials, and intramolecular interactions on the SF-mechanism and -efficiencies. An arsenal of spectroscopic techniques revealed that for perylene-monoimides, a strong charge-transfer mixing with the singlet and triplet excited states restricts SF and yields low triplet quantum yields. This is accompanied by an up-conversion channel that includes geminate triplet-triplet recombination. Using perylene-diimides alters the SF-mechanism by populating a charge-separated-state intermediate, which either favors or shuts-down SF. Napthylene-spacers bring about higher triplet quantum yields and overall better SF-performance for all perylene-monoimides and perylene-diimides. The key to better SF-performance is rotational freedom because it facilitates the overall excited-state polarization and amplifies intramolecular interactions between chromophores.