We investigate a class of non-emissive conjugated polymers with very short excited state lifetimes believed to undergo singlet fission and relaxation to mid-gap forbidden excited states. Poly(3-decylthieneylenvinylene) (P3DTV) and its heavy atom analog, poly(3-decylseleneylenvinylene) (P3DSV), are strongly aggregating conjugated polymers that experience large excited state displacements along multiple vibrational modes. We demonstrate this Franck-Condon vibrational activity effectively disperses excitation energy into multiple non-radiative channels that can be explained using a simple, two-state potential energy surface model. Resonance Raman spectroscopy is sensitive to early Franck-Condon vibrational activity and we observe rich harmonic progressions involving multiple high frequency CC backbone symmetric stretching motions (∼1000-1600 cm-1) in both systems reflecting mode-specific excited state geometrical displacements. Transient absorption spectra confirm that efficient non-radiative processes dominate excited state relaxation dynamics which are confined to π-stacked aggregated chains. Surprisingly, we found little influence of the heteroatom consistent with efficient vibrational energy dissipation. Our results highlight the importance of aggregation and multi-dimensional Franck-Condon vibrational dynamics on the ability to harvest excitons, which are not usually considered in materials design and optimization schemes.