Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties

Andres Jung; Julia Balzer; Tobias Braun; Kerstin Luedtke

doi:10.1186/s12874-022-01561-5

Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties

BMC Med Res Methodol. 2022 Apr 6;22(1):100. doi: 10.1186/s12874-022-01561-5.

Authors

Andres Jung¹, Julia Balzer², Tobias Braun^{3

4}, Kerstin Luedtke⁵

Affiliations

¹ Institute of Health Sciences, Department of Physiotherapy, Pain and Exercise Research Luebeck (P.E.R.L), Universität zu Lübeck, Ratzeburger Allee 160, 23562, Lübeck, Germany. a.jung@uni-luebeck.de.
² Faculty of Applied Public Health, European University of Applied Sciences, Werftstr. 5, 18057, Rostock, Germany.
³ Division of Physiotherapy, Department of Applied Health Sciences, Hochschule für Gesundheit (University of Applied Sciences), Gesundheitscampus 6‑8, 44801, Bochum, Germany.
⁴ Department of Health, HSD Hochschule Döpfer (University of Applied Sciences), Waidmarkt 9, 50676, Cologne, Germany.
⁵ Institute of Health Sciences, Department of Physiotherapy, Pain and Exercise Research Luebeck (P.E.R.L), Universität zu Lübeck, Ratzeburger Allee 160, 23562, Lübeck, Germany.

Abstract

Background: Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs. It was further, to evaluate the quality of identified tools and to recommend the use of individual tools to assess the external validity of RCTs in future systematic reviews.

Methods: A two-phase systematic literature search was performed in four databases: PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies investigating the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines.

Results: 38 publications reporting on the development or validation of 28 included tools were included. For 61% (17/28) of the included tools, there was no evidence for measurement properties. For the remaining tools, reliability was the most frequently assessed property. Reliability was judged as "sufficient" for three tools (very low certainty of evidence). Content validity was rated as "sufficient" for one tool (moderate certainty of evidence).

Conclusions: Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a better suitable tool.

Trial registration: Prospective registration at Open Science Framework (OSF): https://doi.org/10.17605/OSF.IO/PTG4D .

Keywords: Applicability; External validity; Generalizability; Measurement properties; Randomized controlled trial; Tools.

Publication types

Research Support, Non-U.S. Gov't
Review
Systematic Review

MeSH terms

Consensus
Humans
Randomized Controlled Trials as Topic*
Reproducibility of Results