We use single-cycle THz fields and the femtosecond magneto-optical Kerr effect to, respectively, excite and probe the magnetization dynamics in two thin-film ferromagnets with different lattice structures: crystalline Fe and amorphous CoFeB. We observe Landau-Lifshitz-torque magnetization dynamics of comparable magnitude in both systems, but only the amorphous sample shows ultrafast demagnetization caused by the spin-lattice depolarization of the THz-induced ultrafast spin current. Quantitative modeling shows that such spin-lattice scattering events occur on similar time scales than the conventional spin conserving electronic scattering (∼30 fs). This is significantly faster than optical laser-induced demagnetization. THz conductivity measurements point towards the influence of lattice disorder in amorphous CoFeB as the driving force for enhanced spin-lattice scattering.