Sympathetic cooling is the process of energy exchange between a system and a colder bath. We investigate this fundamental process in an atom-ion experiment where the system is composed of a single ion trapped in a radio-frequency Paul trap and prepared in a classical oscillatory motion with total energy of ∼200 K, and the bath is an ultracold cloud of atoms at μK temperature. We directly observe the sympathetic cooling dynamics with single-shot energy measurements during one to several collisions in two distinct regimes. In one, collisions predominantly cool the system with very efficient momentum transfer leading to cooling in only a few collisions. In the other, collisions can both cool and heat the system due to nonequilibrium dynamics in the presence of the ion trap's oscillating electric fields. While the bulk of our observations agree well with a molecular-dynamics simulation of hard-sphere (Langevin) collisions, a measurement of the scattering angle distribution reveals forward-scattering (glancing) collisions which are beyond the Langevin model. This work paves the way for further nonequilibrium and collision dynamics studies using the well-controlled atom-ion system.