Shared Mental Models Among Clinical Competency Committees in the Context of Time-Variable, Competency-Based Advancement to Residency

Alan Schwartz; Dorene F Balmer; Emily Borman-Shoap; Alan Chin; Duncan Henry; Bruce E Herman; Patricia Hobday; James H Lee; Sara Multerer; Ross E Myers; Keith Ponitz; Adam Rosenberg; Jennifer B Soep; Daniel C West; Robert Englander

doi:10.1097/ACM.0000000000003638

Shared Mental Models Among Clinical Competency Committees in the Context of Time-Variable, Competency-Based Advancement to Residency

Acad Med. 2020 Nov;95(11S Association of American Medical Colleges Learn Serve Lead: Proceedings of the 59th Annual Research in Medical Education Presentations):S95-S102. doi: 10.1097/ACM.0000000000003638.

Authors

Alan Schwartz¹, Dorene F Balmer², Emily Borman-Shoap³, Alan Chin⁴, Duncan Henry⁵, Bruce E Herman⁶, Patricia Hobday⁷, James H Lee⁸, Sara Multerer⁹, Ross E Myers¹⁰, Keith Ponitz¹¹, Adam Rosenberg¹², Jennifer B Soep¹³, Daniel C West¹⁴, Robert Englander¹⁵

Affiliations

¹ A. Schwartz is the Michael Reese Endowed Professor of Medical Education and research professor, pediatrics, University of Illinois College of Medicine, and network director, Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN), Chicago, Illinois.
² D.F. Balmer is associate professor, pediatrics, The Children's Hospital of Pennsylvania and University of Pennsylvania, Philadelphia, Pennsylvania.
³ E. Borman-Shoap is vice chair of education, pediatric residency program director, and assistant professor, pediatrics, University of Minnesota, Minneapolis, Minnesota.
⁴ A. Chin is assistant clinical professor, pediatrics, University of California, Los Angeles, Mattel Children's Hospital, Los Angeles, California.
⁵ D. Henry is associate program director for assessment, and clinical competency committee chair, pediatric residency, University of California, San Francisco, San Francisco, California.
⁶ B.E. Herman is professor and vice chair of education and residency programs, pediatrics, University of Utah School of Medicine, Salt Lake City, Utah.
⁷ P. Hobday is assistant professor, pediatrics, University of Minnesota Medical School, Minneapolis, Minnesota.
⁸ J.H. Lee is associate director, pediatrics residency program, and assistant professor, pediatrics, the David Geffen School of Medicine at University of California, Los Angeles, Los Angeles, California.
⁹ S. Multerer is director, pediatric residency program, and associate professor, pediatric hospital medicine, University of Louisville Department of Pediatrics and Norton Children's Hospital, Louisville, Kentucky.
¹⁰ R.E. Myers is associate director, pediatric residency program, and associate professor, pediatrics, Rainbow Babies and Children's Hospital and Case Western Reserve University School of Medicine, Cleveland, Ohio.
¹¹ K. Ponitz is director, pediatric residency training program, pediatrics, Rainbow Babies and Children's Hospital, Cleveland, Ohio.
¹² A. Rosenberg is professor of pediatrics and program director, pediatric residency, University of Colorado School of Medicine, Children's Hospital Colorado, Aurora, Colorado.
¹³ J.B. Soep is associate professor, pediatrics, University of Colorado, Aurora, Colorado.
¹⁴ D.C. West is professor and associate chair for education, pediatrics, Children's Hospital of Philadelphia and Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania.
¹⁵ R. Englander is associate dean, undergraduate medical education, and professor, pediatrics, University of Minnesota Medical School, Minneapolis, Minnesota.

PMID: 32769469
DOI: 10.1097/ACM.0000000000003638

Abstract

Purpose: To evaluate response process validity evidence for clinical competency committee (CCC) assessments of first-year residents on a subset of General Pediatrics Entrustable Professional Activities (EPAs) and milestones in the context of a national pilot of competency-based, time-variable (CBTV) advancement from undergraduate to graduate medical education.

Method: Assessments of 2 EPAs and 8 milestones made by the trainees' actual CCCs and 2 different blinded "virtual" CCCs for 48 first-year pediatrics residents at 4 residency programs between 2016 and 2018 were compared. Residents had 3 different training paths from medical school to residency: time-variable graduation at the same institution as their residency, time-fixed graduation at the same institution, or time-fixed graduation from a different institution. Assessments were compared using ordinal mixed-effects models.

Results: Actual CCCs assigned residents higher scores than virtual CCCs on milestones and one EPA's supervision levels. Residents who graduated from a different institution than their residency received lower milestone ratings than either group from the same institution; CBTV residents received higher ratings on one milestone (ICS4) and similar ratings on all others compared with non-CBTV residents who completed medical school at the same institution.

Conclusions: First-year residents who graduated from CBTV medical school programs were assessed as having the same level of competence as residents who graduated from traditional medical school programs, but response process evidence suggests that members of CCCs may also draw on undocumented personal knowledge of the learner to draw conclusions about resident competence.

Publication types

Comparative Study
Research Support, Non-U.S. Gov't

MeSH terms

Clinical Competence / standards*
Education, Medical, Undergraduate / standards
Internship and Residency / standards*
Models, Psychological*
Time Factors