The presence of geometric phases is known to affect the dynamics of the systems involved. Here, we consider a quantum degree of freedom, moving in a dissipative environment, whose dynamics is described by a Langevin equation with quantum noise. We show that geometric phases enter the stochastic noise terms. Specifically, we consider small ferromagnetic particles (nanomagnets) or quantum dots close to Stoner instability, and investigate the dynamics of the total magnetization in the presence of tunneling coupling to the metallic leads. We generalize the Ambegaokar-Eckern-Schön effective action and the corresponding semiclassical equations of motion from the U(1) case of the charge degree of freedom to the SU(2) case of the magnetization. The Langevin forces (torques) in these equations are strongly influenced by the geometric phase. As a first but nontrivial application, we predict low temperature quantum diffusion of the magnetization on the Bloch sphere, which is governed by the geometric phase. We propose a protocol for experimental observation of this phenomenon.