Understanding the ultrafast excitation and transport dynamics of plasmon-driven hot carriers is critical to the development of optoelectronics, photochemistry, and solar-energy harvesting. However, the ultrashort time and length scales associated with the behavior of these highly out-of-equilibrium carriers have impaired experimental verification of ab initio quantum theories. Here, we present an approach to studying plasmonic hot-carrier dynamics that analyzes the temporal waveform of coherent terahertz bursts radiated by photo-ejected hot carriers from designer nano-antennas with a broken symmetry. For ballistic carriers ejected from gold antennas, we find an ~11-femtosecond timescale composed of the plasmon lifetime and ballistic transport time. Polarization- and phase-sensitive detection of terahertz fields further grant direct access to their ballistic transport trajectory. Our approach opens explorations of ultrafast carrier dynamics in optically excited nanostructures.