The ability to efficiently direct visual attention to salient features in the environment is a critical function of the visual system. The finding that people are slower to detect a target that appears at a previously cued location is thought to reflect a mechanism known as inhibition of return (IOR). Past research has shown that difficult target discriminations result in a greater amount of time needed to inhibit previously attended locations (i.e., a delayed onset of inhibition), suggesting that task difficulty plays a critical role in the allocation of attention. In this study, IOR was measured at a wide range of SOAs while participants detected either a perceptually degraded target or a standard, high luminance target. When responses were made to a perceptually degraded target, the time course of IOR was delayed by approximately 250 ms (relative to the control group), suggesting that the difficulty in detecting targets also influences the allocation of attention. The results are consistent with the notion that IOR is not simply a reflexive subcortical mechanism but rather involves top-down attentional control settings.