We show that a soliton in a high-order spatial mode of a multi-mode fiber can completely lose its shot-to-shot coherence due to a noise seed with energy orders of magnitude below that of the soliton. The total degradation of shot-to-shot coherence is caused by a very strong recently demonstrated intermodal nonlinear effect, soliton self-mode conversion. The results indicate that the robustness of solitons against perturbations is not entirely applicable in the presence of intermodal nonlinearities, and, more generally, that certain single-mode results cannot be trivially extrapolated to multi-mode fibers.