Granular materials such as sand, powders and foams are ubiquitous in daily life and in industrial and geotechnical applications. These disordered systems form stable structures when unperturbed, but in the presence of external influences such as tapping or shear they 'relax', becoming fluid in nature. It is often assumed that the relaxation dynamics of granular systems is similar to that of thermal glass-forming systems. However, so far it has not been possible to determine experimentally the dynamic properties of three-dimensional granular systems at the particle level. This lack of experimental data, combined with the fact that the motion of granular particles involves friction (whereas the motion of particles in thermal glass-forming systems does not), means that an accurate description of the relaxation dynamics of granular materials is lacking. Here we use X-ray tomography to determine the microscale relaxation dynamics of hard granular ellipsoids subject to an oscillatory shear. We find that the distribution of the displacements of the ellipsoids is well described by a Gumbel law (which is similar to a Gaussian distribution for small displacements but has a heavier tail for larger displacements), with a shape parameter that is independent of the amplitude of the shear strain and of the time. Despite this universality, the mean squared displacement of an individual ellipsoid follows a power law as a function of time, with an exponent that does depend on the strain amplitude and time. We argue that these results are related to microscale relaxation mechanisms that involve friction and memory effects (whereby the motion of an ellipsoid at a given point in time depends on its previous motion). Our observations demonstrate that, at the particle level, the dynamic behaviour of granular systems is qualitatively different from that of thermal glass-forming systems, and is instead more similar to that of complex fluids. We conclude that granular materials can relax even when the driving strain is weak.