An Application of Kane's Validity Framework to Evaluate Formative and Summative Assessment Instruments for Telesimulations in Clinical Lactation

Abstract

Introduction: Health professional learners have limited exposure to breastfeeding patients from diverse backgrounds in clinical rotations. Instead, simulation-based training is used for lactation skills training. There are no validated or standardized simulations and assessment rubrics for lactation. In this pilot, breastfeeding telesimulations with standardized patients (SPs) wearing a high-fidelity breast model matching their skin tone were developed. The validity of Formative and Summative Assessment Rubrics (FAR, SAR) were assessed following Kane's validity framework. The objective was to provide initial evidence for the validity of the FAR and SAR as constructs of competence in lactation support at the entry to practice or practice level.

Methods: Three breastfeeding case scenarios, FAR, and SAR were developed and evaluated with clinical lactation specialists (evaluators, n = 17) and SPs. The FAR was used in practice telesimulations where SPs' (n = 14) performance and telesimulation feasibility were assessed. The FAR was updated in preparation for a pilot study where medical students (n = 13) completed the 3 telesimulations. In the pilot, the updated FAR was used by SPs (n = 6) to assess medical students' performance of clinical skills. After the pilot, rubrics were updated after focus groups with SPs and discussions with evaluators. Evaluators (n = 3) graded students' posttelesimulation documentations using the SAR. Cronbach ɑ level and the intraclass correlation coefficient were assessed iteratively to collect evidence for the scoring, generalizability, and extrapolation of the FAR and SAR according to Kane's framework.

Results: The FAR and SAR were found to have acceptable internal consistency and moderate to high interrater reliability (intraclass correlation coefficient, 0.55-0.94), which provided evidence for scoring and generalizability of the instruments. Evaluators agreed that SPs' performances were realistic (5.6/6), and SPs' feedback was organized (5.5/6) and helpful (5.6/6), which provided evidence for extrapolation.

Conclusions: Initial evidence for validity of scoring, generalization, and extrapolation FAR and SAR (according to Kane's framework) in assessing health professional learner's performance of clinical lactation skills has been presented. These results from a pilot study suggest that the FAR and SAR are reliable instruments for assessing learners' clinical performance in a breastfeeding-focused telesimulation where the SP wears a high-fidelity breast model matching their skin tone. Additional studies will be required to collect evidence according to all 4 categories of Kane's framework for the validity of the FAR and SAR.