Validation of the behavior and concept based assessment of professionalism competence in postgraduate first-year residents

Abstract

Background: The evaluation of professional behaviors and concepts of postgraduate first-year (PGY1) residents has been identified as an area for development. This study examined the efficiency of the professionalism-assessing objective structured clinical examination (OSCE), 360° evaluation, and mini-Clinical Examination Exercise scores (mini-CEX; p-OSCE, p-360° evaluation, and p-mini-CEX scores).

Methods: Between January 2009 and January 2012, 189 PGY1 residents were evaluated for behavior- and concept-based professionalism competence based on the above three methods using two checklists unique to each case. Data were analyzed for reliability, inter-rater agreement, interval changes, and gender-related difference for each method.

Results: The test reliabilities of p-OSCE, p-360° evaluation, and p-mini-CEX were acceptable. Further, the reliability of concept and combined p-OSCE was higher than that of behavior p-OSCE. In addition, the concept OSCE p-scores and behavior 360° evaluation p-scores were significantly improved after 6 months of training. The inter-rater agreements were relatively good in p-OSCE and p-360° evaluation. Interestingly, male PGY1 residents had higher behavior 360° evaluation p-scores from nurses than those of females, whereas female PGY1 residents had higher behavior 360° evaluation p-scores from patients than those of males. Behavior and concept OSCE p-scores were positively correlated with behavior 360° evaluation p-scores. In comparison with p-360° evaluation, the combination of p-360° evaluation + OSCE + mini-CEX significantly increases their reliabilities.

Conclusion: The current study suggests that the p-OSCE, p-360° evaluation, and p-mini-CEX are feasible methods for evaluating professionalism in clinical training of PGY1 residents. Combination of the above three evaluations, participation, and support from multiple constituencies and multiple representatives provides good reliability and adds credibility in the assessment of professionalism competence.