Development and validation of the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI)

Liane Ginsburg; Matthias Hoben; Whitney Berta; Malcolm Doupe; Carole A Estabrooks; Peter G Norton; Colin Reid; Ariane Geerts; Adrian Wagg

doi:10.1136/bmjqs-2023-016001

Development and validation of the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI)

BMJ Qual Saf. 2024 Jan 19;33(2):98-108. doi: 10.1136/bmjqs-2023-016001.

Authors

Liane Ginsburg¹, Matthias Hoben², Whitney Berta³, Malcolm Doupe^{4

5}, Carole A Estabrooks², Peter G Norton⁶, Colin Reid⁷, Ariane Geerts⁸, Adrian Wagg⁹

Affiliations

¹ School of Health Policy and Management, Faculty of Health, York University, Toronto, Ontario, Canada lgins@yorku.ca.
² Faculty of Nursing, University of Alberta, Edmonton, Alberta, Canada.
³ Institute of Health Policy Management and Evaluation, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
⁴ Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba, Canada.
⁵ Centre for Care Research, Western Norway University of Applied Sciences, Bergen, Norway.
⁶ Department of Family Medicine, Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada.
⁷ School of Health and Exercise Science, The University of British Columbia, Kelowna, British Columbia, Canada.
⁸ School of Kinesiology and Health Science, Faculty of Health, York University, Toronto, Ontario, Canada.
⁹ Department of Medicine, University of Alberta, Edmonton, Alberta, Canada.

Abstract

Background: In many quality improvement (QI) and other complex interventions, assessing the fidelity with which participants 'enact' intervention activities (ie, implement them as intended) is underexplored. Adapting the evaluative approach used in objective structured clinical examinations, we aimed to develop and validate a practical approach to assessing fidelity enactment-the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI).

Methods: We developed the OFES-CI to evaluate enactment of the SCOPE QI intervention, which teaches nursing home teams to use plan-do-study-act (PDSA) cycles. The OFES-CI was piloted and revised early in SCOPE with good inter-rater reliability, so we proceeded with a single rater. An intraclass correlation coefficient (ICC) was used to assess inter-rater reliability. For 27 SCOPE teams, we used ICC to compare two methods for assessing fidelity enactment: (1) OFES-CI ratings provided by one of five trained experts who observed structured 6 min PDSA progress presentations made at the end of SCOPE, (2) average rating of two coders' deductive content analysis of qualitative process evaluation data collected during the final 3 months of SCOPE (our gold standard).

Results: Using Cicchetti's classification, inter-rater reliability between two coders who derived the gold standard enactment score was 'excellent' (ICC=0.93, 95% CI=0.85 to 0.97). Inter-rater reliability between the OFES-CI and the gold standard was good (ICC=0.71, 95% CI=0.46 to 0.86), after removing one team where open-text comments were discrepant with the rating. Rater feedback suggests the OFES-CI has strong face validity and positive implementation qualities (acceptability, easy to use, low training requirements).

Conclusions: The OFES-CI provides a promising novel approach for assessing fidelity enactment in QI and other complex interventions. It demonstrates good reliability against our gold standard assessment approach and addresses the practicality problem in fidelity assessment by virtue of its suitable implementation qualities. Steps for adapting the OFES-CI to other complex interventions are offered.

Keywords: Evaluation methodology; Implementation science; Medical education; Quality improvement.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Data Accuracy*
Feedback
Humans
Quality Improvement*
Reproducibility of Results