Singlet excitons in π-stacked molecular crystals can split into two triplet excitons in a process called singlet fission that opens a route to carrier multiplication in photovoltaics. To resolve controversies about the mechanism of singlet fission, we have developed a first principles nonadiabatic quantum dynamical model that reveals the critical role of molecular stacking symmetry and provides a unified picture of coherent versus thermally activated singlet fission mechanisms in different acenes. The slip-stacked equilibrium packing structure of pentacene derivatives is found to enhance ultrafast singlet fission mediated by a coherent superexchange mechanism via higher-lying charge transfer states. By contrast, the electronic couplings for singlet fission strictly vanish at the C(2h) symmetric equilibrium π stacking of rubrene. In this case, singlet fission is driven by excitations of symmetry-breaking intermolecular vibrations, rationalizing the experimentally observed temperature dependence. Design rules for optimal singlet fission materials therefore need to account for the interplay of molecular π-stacking symmetry and phonon-induced coherent or thermally activated mechanisms.