It is usually expected that time delays cause oscillations in dynamical systems-there might appear cycles around stationary points. Here we study random walks with asymmetric time delays. In our models, the probability of a walker to move to the right or to the left depends on the difference between two state-dependent fitness functions evaluated at two different times. We observe a different behavior-a dependence of the mean position of the walker on time delays. Moreover, the effect of time delays is reversed when one shifts (in a symmetric way) fitness functions. This is a joint effect of both stochasticity and time delays present in the system. The position of a random walker may be interpreted as a frequency of a given strategy in discrete replicator dynamics of evolutionary games. Then our results show that the effect of asymmetric time delays on the equilibrium structure of a population depends not only on time delays, but also on details of the fitness functions.