Filling a void: developing a standard subjective assessment tool for surgical simulation through focused review of current practices

Abstract

Background: Simulation-based education (SBE) has advanced greatly in surgery the past decade, partly through development of validated performance measurement. Standard measures are commonly used to evaluate performance (eg, Objective Structured Assessment of Technical Skills, Fundamentals of Laparoscopic Surgery, Global Operative Assessment of Laparoscopic Skills [GOALS]). However, subjective measures are necessary for ensuring content validity during evaluation of novel simulations or simulators. Although commonly assessed, there are no standardized instruments for such surveys of participants. This lack of standardization limits the ability to compare simulations and/or simulators. We performed a focused literature review to assess current uses of subjective measures, and develop a template for a standardized assessment tool.

Methods: A representative sample of recently collected subjective measures was generated through a focused literature review of the journals Surgery, Journal of Surgical Education, and Journal of the American College of Surgeons (January 2008-November 2012) using the key words "surgical" and "simulation."

Results: Of the 137 articles relevant to development and/or evaluation of surgical skills curricula or simulators, 19 (12%) reported subjective measures from participants. Ten domains were identified, including Self-efficacy/Confidence/Comfort (11, 57%), Model quality/characteristics (7, 37%), Educational/Program value (6, 32%), Previous experience with procedure/simulation (5, 26%) Relevance to practice (3, 16%), Quality- trainer feedback (3, 16%), Quality-experience (2, 11%), Attitude toward specific aspects (2, 11%), Satisfaction/Enjoyability (2, 11%), Ability to perform relevant task(s) (2, 11%). Response options varied and included 4-, 5-, and 10-point rating scales, Visual Analog Scales, and open written responses.

Conclusion: These results suggest that simulation experiences are not currently assessed in a systematic manner. However, analysis of the literature suggests that the full range of subjective measures commonly used for subjective assessment could be addressed by a unified assessment instrument. To this end, the Michigan Standard Simulation Experience Scale (MiSSES) template has been developed and is available on-line. Such a tool would provide practitioners a freely available resource used to measure performance and preferences in SBE.