In 1999, Wilson and Batterham proposed a new approach to assessing the test-retest stability of psychometric questionnaires. They recommended assessing the proportion of agreement - that is, the proportion of participants that record the same response to an item - using a test-retest design. They went on to use a bootstrapping technique to estimate the uncertainty of the proportion of agreement. The aims of this short communication are (1) to demonstrate that the sampling distribution of the proportion of agreement is well known (the binomial distribution), making the technique of 'bootstrapping' redundant, and (2) to suggest a much simpler, more sensitive method of assessing the stability of a psychometric questionnaire, based on the test-retest differences (within-individuals) for each item. Adopting methods similar to Wilson and Batterham, 97 sport students completed the Social Physique Anxiety Scale on two occasions. Test-retest differences were calculated for each item. Our results show that the proportion of agreement ignores the nature of disagreement. Items 4 and 11 showed similar agreement (44.3% and 43.3% respectively), but 89 of the participants (91.8%) differed by just +/-1 point when responding to item 4, indicating a relatively stable item. In contrast, only 78 of the participants (80.4%) recorded a difference within +/- 1 point when responding to item 11, suggesting quite contrasting stability for the two items. We recommend that, when assessing the stability of self-report questionnaires using a 5-point scale, most participants (90%) should record test-retest differences within a reference value of +/- 1.