Neuropsychological instruments continue to proliferate the field, while studies systematically addressing their accuracy and standardization have lagged behind. This paper examines the standard scoring criteria on a widely used neuropsychological test, the Wisconsin Card Sorting Test (WCST). Specifically, the scoring criteria for the perseveration variables are complex and interrater reliability had not yet been established. Study 1 revealed excellent interrater reliability between three individuals experienced with the WCST who had independently learned the scoring procedures. Study 2 demonstrated excellent interrater agreement among six novice raters. Some of the novice scorers used supplemental scoring instructions that were developed in an attempt to improve understanding of the manual. The supplemental scoring instructions reduced training time by 41% without affecting the high reliability between the raters. This paper also illustrates an appropriate use of generalizability theory, a powerful statistical method to examine reliability.