Problem statement and background: Examinees can make three types of errors on the short-menu questions in the Clinical Reasoning Skills component of the Medical Council of Canada's Qualifying Examination Part I: (1) failing to select any correct responses, (2) selecting too many responses, or (3) selecting a response that is inappropriate or harmful to the patient. This study compared the information provided by equal and differential weighting of these errors.
Method: The item response theory nominal model was applied to fit examinees' response patterns on the 1998 test.
Results: Differential error weighting resulted in improved model fit and increased test information for examinees in the lower half of the achievement continuum.
Conclusion: Differential error weighting appears promising. The pass score is near the lower end of the achievement continuum; therefore, this approach may improve the accuracy of pass-fail decisions.