Studies of perceptual learning consistently found that improvement is stimulus specific. These findings were interpreted as indicating an early cortical learning site. In line with this interpretation, we consider two alternative hypotheses: the 'earliest modification' and the 'output-level modification' assumptions, which respectively assume that learning occurs within the earliest representation which is selective for the trained stimuli, or at cortical levels receiving its output. We studied performance in a pop-out task using light bar distractor elements of one orientation, and a target element rotated by 30 degrees (or 90 degrees). We tested the alternative hypotheses by examining pop-out learning through an initial training phase, a subsequent learning stage with swapped target and distracted orientations, and a final re-test with the originally trained stimuli. We found learning does not transfer across orientation swapping. However, following training with swapped orientations, a similar performance level is reached as with original orientations. That is, learning neither facilitates nor interferes to a substantial degree with subsequent performance with altered stimuli. Furthermore, this re-training does not hamper performance with the original trained stimuli. If training changed the earliest orientation selective representation (specializing it for performance of the particular performed task) it would necessarily affect performance with swapped orientations, as well. The co-existence of similar asymptotes for apparently conflicting stimulus sets refutes the 'earliest modification' hypothesis, supporting the alternative 'output level modification' hypothesis. We conclude that secondary cortical processing levels use outputs from the earliest orientation representation to compute higher order structures, promoting and improving successful task performance.