In the absence of confinement, localization of waves takes place due to randomness or nonlinearity and relies on their phase coherence. We quantitatively probe the sensitivity of localized wave packets to random phase fluctuations and confirm the necessity of phase coherence for localization. Decoherence resulting from a dynamical random environment leads to diffusive spreading and destroys linear and nonlinear localization. We find that maximal spreading is achieved for optimal phase fluctuation characteristics, which is a consequence of the competition between diffusion due to decoherence and ballistic transport within the mean free path distance.