Phonon spectroscopy is used to investigate the mechanism of current-induced spin torques in nonmagnetic/ferromagnetic (N/F) point contacts. Magnetization excitations observed in the magneto-conductance of the point contacts are pronounced for diffusive and thermal contacts, where the electrons experience significant scattering in the contact region. We find no magnetic excitations in highly ballistic contacts. Our results show that impurity scattering at the N/F interface is the origin of the new single-interface spin torque effect.