During binocular rivalry, incompatible images presented dichoptically compete for perceptual dominance. It has long been debated whether binocular rivalry can be controlled by attention. Most studies have shown that voluntary control over binocular rivalry is limited. We sought to remove attention from binocular rivalry by presenting a concurrent task. Diverting attention slowed the rivalry alternation rate, and did so in proportion to the difficulty of the concurrent task. Even a very demanding distractor task, however, did not arrest rivalry alternations completely. Given that diverting attention was equivalent to lowering the contrast of the rival stimuli, the ability of attention to speed binocular rivalry is most likely due to an increase in the effective contrast of the stimuli through boosting the gain of the cortical response. This increase in effective contrast will ultimately lead to a perceptual switch, thereby limiting voluntary control. Thus, attention speeds rivalry alternations, but has no inherent control over the rivalry process.