Intramolecular singlet fission (iSF) materials provide remarkable advantages in terms of tunable electronic structures, and quantum chemistry studies have indicated strong electronic coupling modulation by high frequency phonon modes. In this work, we formulate a microscopic model of iSF with simultaneous diagonal and off-diagonal coupling to high-frequency modes. A nonperturbative treatment, the Dirac-Frenkel time-dependent variational approach is adopted using the multiple Davydov trial states. It is shown that both diagonal and off-diagonal coupling can aid efficient singlet fission if excitonic coupling is weak, and fission is only facilitated by diagonal coupling if excitonic coupling is strong. In the presence of off-diagonal coupling, it is found that high frequency modes create additional fission channels for rapid iSF. Results presented here may help provide guiding principles for design of efficient singlet fission materials by directly tuning singlet-triplet interstate coupling.