We examined the effect of facial expression on the assignment of gender to facial images. A computational analysis of the facial images was applied to examine whether physical aspects of the face itself induced this effect. Thirty-six observers rated the degree of masculinity of the faces of 48 men, and the degree of femininity of the faces of 48 women. Half of the faces had a neutral facial expression, and the other half was smiling. Smiling significantly reduced the perceived masculinity of men's faces, especially for male observers, whereas no effect of smiling on femininity ratings was obtained for women's faces. A principal component analysis was conducted on the matrix of pixel luminance values for each facial image x all the images. The third principle component explained a relatively high proportion of the variance of both facial expressions and gender of face. These results suggest that the effect of smiling on the assignment of gender is caused, at least in part, by the physical relationship between facial expression and face gender.