The present study investigated whether faces capture attention regardless of attentional set. The presentation of a face as a distractor during a visual search has been shown to impair performance relative to when the face was absent, implying that faces automatically attract attention. If attentional control is contingent on the observer's current goal, faces should not capture attention when they are irrelevant to the observer's attentional set. Previous studies demonstrating face-induced attentional capture used faces that were relevant to the task. Thus, a task in which faces were completely irrelevant to the observer's set was created. Participants identified a target letter among heterogeneously colored non-targets while ignoring a peripheral facial image that appeared as a brief distractor. No face-specific capture was observed when the target-distractor stimulus onset asynchrony (SOA) was long (Experiment 1). When the SOA was shortened, attentional capture by irrelevant faces was observed (Experiment 2). Experiment 3 extended this finding to all conditions, regardless of the attractiveness of faces. No such capture effect was found in Experiment 4 with inverted-face distractors. These results indicate that completely task-irrelevant faces break through top-down attentional set given a brief distractor-target SOA.