Orbital degrees of freedom mediating an interaction between spin and lattice were predicted to raise strong magnetoelectric effect, i.e., to realize an efficient coupling between magnetic and ferroelectric orders. However, the effect of orbital fluctuations has been considered only in a few magnetoelectric materials, as orbital-degeneracy driven Jahn-Teller effect rarely couples to polarization. Here, we explore the spin-lattice coupling in multiferroic Swedenborgites with mixed valence and Jahn-Teller active transition metal ions on a stacked triangular and Kagome lattice using infrared and dielectric spectroscopy. On one hand, in CaBaM_{4}O_{7} (M=Co, Fe), we observe a strong magnetic-order-induced shift in the phonon frequencies and a corresponding large change in the dielectric response. Remarkably, as an unusual manifestation of the spin-phonon coupling, the spin fluctuations reduce the phonon lifetime by one order of magnitude at the magnetic phase transitions. On the other hand, lattice vibrations, dielectric response, and electric polarization show no variation at the Néel temperature of CaBaFe_{2}Co_{2}O_{7}, which is built up by orbital singlet ions. Our results provide a showcase for orbital degrees of freedom enhanced magnetoelectric coupling via the example of Swedenborgites.