Gaze direction plays a central role in face recognition. Previous research suggests that faces with direct gaze are better remembered than faces with averted gaze. We compared recognition of faces with direct versus averted gaze in male versus female participants. A total of 52 adults (23 females, 29 males) and 46 children (25 females, 21 males) completed a computerised task that assessed their recognition of faces with direct gaze and faces with averted gaze. Adult male participants showed superior recognition of faces with direct gaze compared to faces with averted gaze. There was no difference between recognition of direct and averted gaze faces for the adult female participants. Children did not demonstrate this sex difference; rather, both male and female youth participants showed better recognition of faces with direct gaze compared to averted gaze. A large body of previous research has revealed superior recognition of faces with direct, compared to averted gaze. However, relatively few studies have examined sex differences. Our findings suggest that gaze direction has differential effects on face recognition for adult males and females, but not for children. These findings have implications for previous explanations of better recognition for direct versus averted gaze.