In this paper, we present a detailed Raman study of the non-multiferroic compounds PrMnO 3 and NdMnO 3 and the multiferroic compounds TbMnO 3 and DyMnO 3 as a function of temperature and magnetic field. All studied systems show anomalous phonon shifts close to the Néel transition T N . In PrMnO 3 and NdMnO 3, the frequency softenings are partly attributed to an orbital-spin-phonon coupling whereas in TbMnO 3 and DyMnO 3, the relatively weak frequency shifts are rather attributed to an expansion of the Mn-O bond lengths. On the other hand, the frequencies of TbMnO 3 phonons are shifted as a function of magnetic field, while those of PrMnO 3 remain unaffected. These frequency shifts are interpreted in terms of local oxygen rearrangements under magnetic field that could play an important role in the multiferroicity of TbMnO 3 and DyMnO 3.