Females are generally better than males at recognizing facial emotions. However, it is not entirely clear whether and in what way females may also excel at non-affective face recognition. Here, we tested males and females on two perceptual face recognition tasks that involved only neutral expressions: detection and identity discrimination. On face detection (Experiment 1), females were significantly more accurate than males in detecting upright faces. This gender difference was reduced during inverted face detection, and not present during tree detection, suggesting that the magnitude of the gender difference for performance co-varies with the extent to which face processing mechanisms are involved. On facial identity discrimination (Experiment 2), females again outperformed males, particularly when face images were masked by visual noise, or the delay between comparison face images was extended from 0.5 to 3s. These results reveal a female advantage in processing face-specific information and underscore the role of perceptual factors in socially relevant gender differences.