The current study assesses the intercoder reliability of one of the most important skin tone measurement instruments-the Massey-Martin scale. This scale is used in several high-profile social surveys, but has not yet been psychometrically evaluated. The current evaluation is only possible because, for the first time, the General Social Survey's 2010-2014 panel used the instrument to guide interviewers' skin tone observation of the same respondents in two different years (2012 and 2014). Despite the widespread use of the Massey-Martin scale to investigate potential effects of skin tone on social attitudes and outcomes, the data suggest that the measure has low intercoder reliability. Implications for researchers and survey practitioners are discussed.