We use confocal microscopy and time-resolved light scattering to investigate plasticity in a colloidal polycrystal, following the evolution of the network of grain boundaries as the sample is submitted to thousands of shear deformation cycles. The grain boundary motion is found to be ballistic, with a velocity distribution function exhibiting nontrivial power law tails. The shear-induced dynamics initially slow down, similarly to the aging of the spontaneous dynamics in glassy materials, but eventually reach a steady state. Surprisingly, the crossover time between the initial aging regime and the steady state decreases with increasing probed length scale, hinting at a hierarchical organization of the grain boundary dynamics.