The relative entropy between the joint probability distribution of backward and forward sequences is used to quantify time asymmetry (or irreversibility) for stationary time series. The parallel with the thermodynamic theory of nonequilibrium steady states allows us to link the degree of asymmetry in the time signal with the distance from equilibrium and the lack of detailed balance among its states. We study the statistics of time asymmetry in terms of the fluctuation theorem, showing that this type of relationship derives from simple general symmetries valid for any stationary time series.