We examine birth-death processes with state dependent transition probabilities and at least one absorbing boundary. In evolution, this describes selection acting on two different types in a finite population where reproductive events occur successively. If the two types have equal fitness the system performs a random walk. If one type has a fitness advantage it is favored by selection, which introduces a bias (asymmetry) in the transition probabilities. How long does it take until advantageous mutants have invaded and taken over? Surprisingly, we find that the average time of such a process can increase, even if the mutant type always has a fitness advantage. We discuss this finding for the Moran process and develop a simplified model which allows a more intuitive understanding. We show that this effect can occur for weak but nonvanishing bias (selection) in the state dependent transition rates and infer the scaling with system size. We also address the Wright-Fisher model commonly used in population genetics, which shows that this stochastic slowdown is not restricted to birth-death processes.