A Comprehensive, Multi-modal Evaluation of the Assessment System of an Undergraduate Research Methodology Course: Translating Theory into Practice

Hamza Mohammad Abdulghani; Gominda G Ponnamperuma; Farah Ahmad; Zubair Amin

A Comprehensive, Multi-modal Evaluation of the Assessment System of an Undergraduate Research Methodology Course: Translating Theory into Practice

Pak J Med Sci. 2014 Mar;30(2):227-32.

Authors

Hamza Mohammad Abdulghani¹, Gominda G Ponnamperuma², Farah Ahmad³, Zubair Amin⁴

Affiliations

¹ Hamza Mohammad Abdulghani, FRCGP, Associate Professor, College of Medicine, King Saud University, P.O. Box: 230155, Riyadh 11321, Kingdom of Saudi Arabia.
² Gominda G. Ponnamperuma, PhD, College of Medicine, University of Colombo, Colombo, Sri Lanka.
³ Farah Ahmad, PhD, Department of Medical Education, College of Medicine, King Saud University.
⁴ Zubair Amin, MD, Associate Professor, Department of Pediatrics, College of Medicine, National University of Singapore, Singapore.

PMID: 24772117
PMCID: PMC3998984

Abstract

Objective: To evaluate assessment system of the 'Research Methodology Course' using utility criteria (i.e. validity, reliability, acceptability, educational impact, and cost-effectiveness). This study demonstrates comprehensive evaluation of assessment system and suggests a framework for similar courses.

Methods: Qualitative and quantitative methods used for evaluation of the course assessment components (50 MCQ, 3 Short Answer Questions (SAQ) and research project) using the utility criteria. RESULTS of multiple evaluation methods for all the assessment components were collected and interpreted together to arrive at holistic judgments, rather than judgments based on individual methods or individual assessment.

Results: Face validity, evaluated using a self-administered questionnaire (response rate-88.7%) disclosed that the students perceived that there was an imbalance in the contents covered by the assessment. This was confirmed by the assessment blueprint. Construct validity was affected by the low correlation between MCQ and SAQ scores (r=0.326). There was a higher correlation between the project and MCQ (r=0.466)/SAQ (r=0.463) scores. Construct validity was also affected by the presence of recall type of MCQs (70%; 35/50), item construction flaws and non-functioning distractors. High discriminating indices (>0.35) were found in MCQs with moderate difficulty indices (0.3-0.7). Reliability of the MCQs was 0.75 which could be improved up to 0.8 by increasing the number of MCQs to at least 70. A positive educational impact was found in the form of the research project assessment driving students to present/publish their work in conferences/peer reviewed journals. Cost per student to complete the course was US$164.50.

Conclusions: The multi-modal evaluation of an assessment system is feasible and provides thorough and diagnostic information. Utility of the assessment system could be further improved by modifying the psychometrically inappropriate assessment items.

Keywords: Assessment; Evaluation; Research Course; Utility Criteria.