Making a "which-way" measurement (WWM) to identify which slit a particle goes through in a double-slit apparatus will reduce the visibility of interference fringes. There has been a long-standing controversy over whether this can be attributed to an uncontrollable momentum transfer. Here, by reconstructing the Bohmian trajectories of single photons, we experimentally obtain the distribution of momentum change. For our WWM, the change we see is not a momentum kick that occurs at the point of the WWM, but rather one that nonclassically accumulates during the propagation of the photons. We further confirm a quantitative relation between the loss of visibility consequent on a WWM and the total (late-time) momentum disturbance. Our results emphasize the role of the Bohmian momentum in giving an intuitive picture of wave-particle duality and complementarity.