Previous research has demonstrated sex differences in face processing at both neural and behavioural levels. The present study examined the role of handedness and sexual orientation as mediators of this effect. We compared the performance of LH (left-handed) and RH (right-handed) heterosexual and homosexual male and female participants on a face recognition memory task. Our main findings were that homosexual males have better face recognition memory than both heterosexual males and homosexual women. We also demonstrate better face processing in women than in men. Finally, LH heterosexual participants had better face recognition than LH homosexual participants and also tended to be better than RH heterosexual participants. These findings are consistent with differences in the organisation and laterality of face-processing mechanisms as a function of sex, handedness, and sexual orientation.