Retardation between sensation and action is an inherent biological trait. Here we study its effect in the Vicsek model, which is a paradigmatic swarm model. We find that (1) a discrete time delay in the orientational interactions diminishes the ability of strongly aligned swarms to follow a leader and, in return, increases their stability against random orientation fluctuations; (2) both longer delays and higher speeds favor ballistic over diffusive spreading of information (orientation) through the swarm; (3) for short delays, the mean change in the total orientation (the order parameter) scales linearly in a small orientational bias of the leaders and inversely in the delay time, while its variance first increases and then saturates with increasing delays; and (4) the linear response breaks down when orientation conservation is broken.