Effect of changing multiple choice questions from "all of the above" to "select all that apply"

Abstract

Introduction: The purpose of this study was to describe the effect of converting multiple choice questions (MCQs) that include an "all of the above" (AOTA) answer option to a "select all that apply" (SATA) question type on question performance.

Methods: A summative assessment at the end of the first professional pharmacy year was comprised of approximately 50 multiple choice questions covering material from all courses taught. Eight questions contained AOTA answer options and were converted to SATA items in the subsequent year by eliminating the AOTA option and including the words "select all that apply" in the stem. Majority of the other questions included on the exam remained the same between the two years. Item difficulty, item discrimination, point biserial, and distractor efficiency were used to compare the MCQs on exams in the two years.

Results: The AOTA questions were significantly easier and less discriminating than the SATA items. The performance of the remaining questions on the exam did not differ between the years. The distractor efficiency increased significantly when the questions were converted to SATA items.

Conclusions: MCQs with AOTA answer options are discouraged due to poor item construction resulting in poor discrimination between high and low performing students. The AOTA questions are easily converted to the SATA format. The result of this conversion is a more difficult and more discriminating question with all answer options chosen, which prevents students from easily guessing the correct answer.

Keywords: All of the above; Multiple choice questions; Question performance; Select all that apply.