Developing a Competency-Based Learning and Assessment System for Residency Training: Analysis Study of User Requirements and Acceptance

Abstract

Background: The increasingly complex medical environment highlights the importance of milestones and entrustable professional activities (EPAs) to realize the ideals of competency-based medical education (CBME). However, if enormous amounts of assessment results need to be compiled, the development of a digital system to manage, integrate, and synthesize learning and assessment data will be necessary. Furthermore, this system should be able to facilitate real-time assessment with feedback and therefore enhance users' learning through coaching in the moment in the clinical workplace.

Objective: The main purpose of this study was to develop a competency-based electronic platform system to provide resident physicians with clinical assessments and learning in order to enhance the learning of trainees and reduce the burden of assessments.

Methods: A competency-based learning and assessment system (CBLAS) for residency training was designed, developed, and evaluated in this study. Opinion interviews and a focus group consensus meeting of key users, including trainees, clinical teachers, and administrative staff, were conducted as needs assessments. The structure of the CBLAS was designed according to the thematic analysis of needs assessments. Clinical teachers' acceptance of using CBME assessments, according to the constructs of attitude, perceived usefulness, and perceived ease of use, was surveyed in the beginning and half a year after implementation of the CBLAS. Additionally, the satisfaction of using the CBLAS, according to information, system, and service qualities, was surveyed after implementation.

Results: The main functions of the CBLAS, including milestones, EPAs, learning portfolios, teacher/student feedback, e-books, learning materials, assessment progress tracking, and statistical analysis of assessment results, were designed and developed for responding to nine themes, which emerged from the needs assessments of the three user groups. Twenty clinical teachers responded to the CBME assessment acceptance surveys before and after CBLAS implementation, which revealed a significant improvement in the factor of "attitude" (P=.02) but no significant differences in the two factors of "usefulness" (P=.09) and "ease of use" (P=.58) for CBME assessments. Furthermore, satisfaction surveys were performed in 117 users, and 87.2% (102/117) were satisfied with the CBLAS in terms of information, system, and service qualities. There was no significant difference in satisfaction among different user groups.

Conclusions: The CBLAS is a user-centered platform that supports clinical teachers' assessment exercises and residents' learning, as well as administrative work for staff according to users' needs assessments and operationalized features of CBME assessments. With the system, clinical teachers had a more positive attitude to conduct the assessment activities of milestones and EPAs and learners could arrange their study schedules to enhance their learning effectiveness. The CBLAS sheds light on how to effectively design and develop a digital system to execute milestone- and EPA-based assessments for enhancing competency-based education among residents, according to our experiences in Taiwan.

Keywords: assessment; competency-based learning; entrustable professional activities; learning platform; milestones.