Instruments to evaluate non-technical skills during high fidelity simulation: A systematic review

Abstract

Introduction: High Fidelity Simulations (HFS) are increasingly used to develop Non-Technical Skills (NTS) in healthcare providers, medical and nursing students. Instruments to measure NTS are needed to evaluate the healthcare providers' (HCPs) performance during HFS. The aim of this systematic review is to describe the domains, items, characteristics and psychometric properties of instruments devised to evaluate the NTS of HCPs during HFS.

Methods: A systematic review of the literature was performed according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). Studies were retrieved from PubMed, Cinahl, Web of Science, Cochrane Library, ProQuest and PubPsych. Studies evaluating the measurement properties of instruments used to assess NTS during HFS training were included. Pairs of independent reviewers determined the eligibility, extracted and evaluated the data. Risk of bias and appraisal of the methodological quality of the studies was assessed using the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist, and the quality of the evidence with the Grading of Recommendations, Assessment, Development and Evaluation (GRADE).

Results: A total of 3,953 articles were screened. A total of 110 reports were assessed for eligibility and 26 studies were included. Studies were conducted in Europe/United Kingdom (n = 13; 50%), North America/Australia (n = 12; 46%) and Thailand (n = 1; 4%). The NTS instruments reported in this review included from 1 to 14 domains (median of 4, Q₁ = 3.75, Q₃ = 5) and from 3 to 63 items (median of 15, Q₁ = 10, Q₃ = 19.75). Out of 19 NTS assessment instruments for HFS, the Team Emergency Assessment Measure (TEAM) can be recommended for use to assess NTS. All the other instruments require further research to assess their quality in order to be recommended for use during HFS training. Eight NTS instruments had a positive overall rating of their content validity with at least a moderate quality of evidence.

Conclusion: Among a large variety of published instruments, TEAM can be recommended for use to assess NTS during HFS. Evidence is still limited on essential aspects of validity and reliability of all the other NTS instruments included in this review. Further research is warranted to establish their performance in order to be reliably used for HFS.

Keywords: assessment and evaluation; crew resource management; high fidelity simulation; human error; non-technical skills; psychometrics; reproducibility of results; teamwork.