Background and objectives: Multiple choice examinations assess learners' attainment of medical knowledge. Developing multiple choice examinations that discriminate among learners is difficult and time-consuming. Many institutions avoid this effort by using the National Board of Medical Examiners (NBME) subject examinations, which can also provide comparisons to a national norm. The family medicine subject examination has been criticized, however, because the test's content does not reflect the learning expected during the clerkship. Additionally, the test results cannot guide clerkship directors sufficiently to help students study or to improve the curriculum.
Methods: Family medicine clerkships at three different institutions used a common 75-item examination based on the textbook Essentials of Family Medicine, Fifth Edition, for one academic year. Data were pooled and analyzed. The Raush Item Response Theory assessed student and item performance.
Results: A total of 451 students took the examination. Across the three schools: (1) item separations (Rasch) were high (8.64), indicating good spread in item difficulty, (2) person separations were lower (1.65), indicating that medical students are likely a relatively homogeneous group, (3) Rasch item reliabilities were strong (ranging from .96-.99), and (4) Rasch person reliabilities (.54-.73) were lower. True internal consistencies across items as measured by the Kuder-Richardson 20 (KR-20) reliabilities were just adequate at .71-.77.
Conclusions: By pooling resources, clerkship directors can share the creation and implementation of a written examination that has acceptable reliability and greater face validity than the NBME subject examination. They also have more control over examination content and can guide students' learning and curriculum improvements more accurately.