The majority of work on the perception of gloss has been performed using smooth surfaces (e.g., spheres). Previous studies that have employed more complex surfaces reported that increasing mesoscale roughness increases perceived gloss [Psychol. Sci.19, 196 (2008), J. Vis.10(9), 13 (2010), Curr. Biol.22, 1909 (2012)]. We show that the use of realistic rendering conditions is important and that, in contrast to [Psychol. Sci.19, 196 (2008), J. Vis.10(9), 13 (2010)], after a certain point increasing roughness further actually reduces glossiness. We investigate five image statistics of estimated highlights and show that for our stimuli, one in particular, which we term "percentage of highlight area," is highly correlated with perceived gloss. We investigate a simple model that explains the unimodal, nonmonotonic relationship between mesoscale roughness and percentage highlight area.