Phonons are displacements of atoms around their rest positions in a crystalline solid. They carry sound and heat, but are not classically associated with magnetism. Here, we show that phonons are, in fact, sensitive to magnetic fields, even in diamagnetic materials. We do so by demonstrating experimentally that acoustic phonons in a diamagnetic semiconductor (InSb) scatter more strongly from one another when a magnetic field is applied. We attribute this observation to the magnetic-field sensitivity of the anharmonicity of the interatomic bonds that govern the probability of phonon-phonon interactions. The displacements of atoms locally affect the orbital motion of valence band electrons, which, in the presence of an external magnetic field, spatially modulates the orbital diamagnetism around the displaced atoms. The spatial gradient in magnetic moment results in an anharmonic magnetic force exerted on the displaced atom. The process is modelled by ab initio calculations that, without the use of a single adjustable parameter, reproduce the observed 12% decrease in the lattice thermal conductivity under a 7 T magnetic field at a temperature of 5.2 K.