Under suitable conditions, pupillary dilation is a reliable index of processing activity. Pupil size was tracked in male and female observers following the presentation of face stimuli for an age-judgment task. The eyes on the faces were either directed towards the observer or deviated to the side. Pupil dilation accompanied processing of the faces, but female observers showed significantly more sustained pupil dilation when viewing direct- than deviated-gaze faces over the period 3 to 7 s after stimulus onset, regardless of stimulus sex. In contrast, male observers did not show a consistent pattern in response to either the gaze or sex of the face stimuli. These findings indicate a sex difference in the processing of gaze direction and suggest that females, but not males, apply increased effort to processing socially relevant (direct-gaze) than irrelevant (deviated-gaze) faces. They also demonstrate that pupillary measurement can potentially provide new insights into the processing of even visual input, provided reflexes are sufficiently controlled.