Working memory theories make opposing predictions as to whether the disruptive effect of task-irrelevant sound on serial recall should be attenuated after repeated exposure to the auditory distractors. Although evidence of habituation has emerged after a passive listening phase, previous attempts to observe habituation to to-be ignored distractors on a trial-by-trial basis have proven to be fruitless. With the present study, we suggest that habituation to auditory distractors occurs, but has often been overlooked because past attempts to measure habituation in the irrelevant-sound paradigm were not sensitive enough. In a series of four experiments, the disruptive effects of to-be-ignored speech and music relative to a quiet control condition were markedly reduced after eight repetitions, regardless of whether trials were presented in blocks (Exp. 1) or in a random order (Exp. 2). The auditory distractor's playback direction (forward, backward) had no effect (Exp. 3). The same results were obtained when the auditory distractors were only presented in a retention interval after the presentation of the to-be-remembered items (Exp. 4). This pattern is only consistent with theoretical accounts that allow for attentional processes to interfere with the maintenance of information in working memory.