Sometimes we fail to notice distinctive or unusual items (inattentional blindness), while other times we remember distinctive items more than expected items (the von Restorff effect). A three-factor framework is presented and tested in two experiments in an attempt to reconcile these seemingly contradictory phenomena. Memory for different types of unexpected stimuli was tested after an easy or difficult Stroop color-naming task. Highly arousing taboo words were well remembered even when the difficult Stroop task limited attentional resources. However, a conceptual isolation effect was only observed when the nature of the category change was highlighted by the Stroop task, the Stroop task was easy, and/or the isolated targets enjoyed a retrieval advantage relative to comparison targets. As proposed in the three-factor framework, the arousing qualities of the stimuli, the attentional demands of the primary task, and the relevance of isolated features at encoding and retrieval combine to produce inattentional blindness and the von Restorff effect.