Measuring test-retest reliability (TRR) of AMSTAR provides moderate to perfect agreement - a contribution to the discussion of the importance of TRR in relation to the psychometric properties of assessment tools

Stefanie Bühn; Peggy Ober; Tim Mathes; Uta Wegewitz; Anja Jacobs; Dawid Pieper

doi:10.1186/s12874-021-01231-y

Measuring test-retest reliability (TRR) of AMSTAR provides moderate to perfect agreement - a contribution to the discussion of the importance of TRR in relation to the psychometric properties of assessment tools

BMC Med Res Methodol. 2021 Mar 11;21(1):51. doi: 10.1186/s12874-021-01231-y.

Authors

Stefanie Bühn¹, Peggy Ober², Tim Mathes³, Uta Wegewitz⁴, Anja Jacobs⁵, Dawid Pieper³

Affiliations

¹ Institute for Research in Operative Medicine, Faculty of Health - School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Building 38, 51109, Cologne, Germany. Stefanie.Buehn@uni-wh.de.
² LIFE Child, LIFE Leipzig Research Center for Civilization Diseases, Leipzig University, Ph.-Rosenthal-Str. 27, 04103, Leipzig, Germany.
³ Institute for Research in Operative Medicine, Faculty of Health - School of Medicine, Witten/Herdecke University, Ostmerheimer Str. 200, Building 38, 51109, Cologne, Germany.
⁴ Federal Institute for Occupational Safety and Health (BAuA), Nöldnerstr. 40-42, 10317, Berlin, Germany.
⁵ Federal Joint Committee (Healthcare), Gutenbergstraße 13, 10587, Berlin, Germany.

Abstract

Background: Systematic Reviews (SRs) can build the groundwork for evidence-based health care decision-making. A sound methodological quality of SRs is crucial. AMSTAR (A Measurement Tool to Assess Systematic Reviews) is a widely used tool developed to assess the methodological quality of SRs of randomized controlled trials (RCTs). Research shows that AMSTAR seems to be valid and reliable in terms of interrater reliability (IRR), but the test retest reliability (TRR) of AMSTAR has never been investigated. In our study we investigated the TRR of AMSTAR to evaluate the importance of its measurement and contribute to the discussion of the measurement properties of AMSTAR and other quality assessment tools.

Methods: Seven raters at three institutions independently assessed the methodological quality of SRs in the field of occupational health with AMSTAR. Between the first and second ratings was a timespan of approximately two years. Answers were dichotomized, and we calculated the TRR of all raters and AMSTAR items using Gwet's AC1 coefficient. To investigate the impact of variation in the ratings over time, we obtained summary scores for each review.

Results: AMSTAR item 4 (Was the status of publication used as an inclusion criterion?) provided the lowest median TRR of 0.53 (moderate agreement). Perfect agreement of all reviewers was detected for AMSTAR-item 1 with a Gwet's AC1 of 1, which represented perfect agreement. The median TRR of the single raters varied between 0.69 (substantial agreement) and 0.89 (almost perfect agreement). Variation of two or more points in yes-scored AMSTAR items was observed in 65% (73/112) of all assessments.

Conclusions: The high variation between the first and second AMSTAR ratings suggests that consideration of the TRR is important when evaluating the psychometric properties of AMSTAR.. However, more evidence is needed to investigate this neglected issue of measurement properties. Our results may initiate discussion of the importance of considering the TRR of assessment tools. A further examination of the TRR of AMSTAR, as well as other recently established rating tools such as AMSTAR 2 and ROBIS (Risk Of Bias In Systematic reviews), would be useful.

Keywords: AMSTAR; Psychometric properties; Quality assessment tool; Reliability; Systematic reviews; Test-retest-reliability.

Publication types

Research Support, Non-U.S. Gov't

MeSH terms

Bias
Humans
Psychometrics
Reproducibility of Results
Research Report*
Systematic Reviews as Topic