It is commonly accepted that attention is spontaneously biased towards faces and eyes. However, the role of stimulus features and task settings in this finding has not yet been systematically investigated. Here, we tested if faces and facial features bias attention spontaneously when stimulus factors, task properties, response conditions, and eye movements are controlled. In three experiments, participants viewed face, house, and control scrambled face-house images in an upright and inverted orientation. The task was to discriminate a target that appeared with equal probability at the previous location of the face, house, or the control image. In all experiments, our data indicated no spontaneous biasing of attention for targets occurring at the previous location of the face. Experiment 3, which measured oculomotor biasing, suggested a reliable but infrequent saccadic bias towards the eye region of upright faces. Importantly, these results did not reflect our specific laboratory settings, as in Experiment 4, we present a full replication of a classic finding in the literature demonstrating reliable social attention bias. Together, these data suggest that attentional biasing for social information is task and context mediated, and less robust than originally thought.