The observed cloud-level atmospheric circulation on the outer planets of the Solar System is dominated by strong east-west jet streams. The depth of these winds is a crucial unknown in constraining their overall dynamics, energetics and internal structures. There are two approaches to explaining the existence of these strong winds. The first suggests that the jets are driven by shallow atmospheric processes near the surface, whereas the second suggests that the atmospheric dynamics extend deeply into the planetary interiors. Here we report that on Uranus and Neptune the depth of the atmospheric dynamics can be revealed by the planets' respective gravity fields. We show that the measured fourth-order gravity harmonic, J4, constrains the dynamics to the outermost 0.15 per cent of the total mass of Uranus and the outermost 0.2 per cent of the total mass of Neptune. This provides a stronger limit to the depth of the dynamical atmosphere than previously suggested, and shows that the dynamics are confined to a thin weather layer no more than about 1,000 kilometres deep on both planets.