Comparison of four heterogeneity measures for meta-analysis

Abstract

Rationale, aims, and objectives: Heterogeneity is a critical issue in meta-analysis, because it implies the appropriateness of combining the collected studies and impacts the reliability of the synthesized results. The Q test is a traditional method to assess heterogeneity; however, because it does not have an intuitive interpretation for clinicians and often has low statistical power, many meta-analysts alter to use some measures, such as the I² statistic, to quantify the extent of heterogeneity. This article aims at providing a summary of available tools to assess heterogeneity and comparing their performance.

Methods: We reviewed four heterogeneity measures (I² , ${\hat{R}}_I$ , ${\hat{R}}_M$ , and ${\hat{R}}_b$ ) and illustrated how they could be treated as test statistics like the Q statistic. These measures were compared with respect to statistical power based on simulations driven by three real-data examples. The pairwise agreement among the four measures was also evaluated using Cohen's κ coefficient.

Results: Generally, ${\hat{R}}_I$ was slightly more powerful than the Q test, while its type I error rate might be slightly inflated. The power of I² was fairly close to that of Q. The ${\hat{R}}_M$ and ${\hat{R}}_b$ statistics might have low powers in some cases. Because the differences between the powers of I² , ${\hat{R}}_I$ , and Q were often tiny, meta-analysts might not expect I² and ${\hat{R}}_I$ to yield significant heterogeneity if the Q test failed to do so. In addition, I² and ${\hat{R}}_I$ had fairly good agreement based on the simulated meta-analyses, but all other pairs of heterogeneity measures generally had poor agreement.

Conclusion: The I² and ${\hat{R}}_I$ statistics are recommended for measuring heterogeneity. Meta-analysts should use the heterogeneity measures as descriptive statistics which have intuitive interpretations from the clinical perspective, instead of determining the significance of heterogeneity simply based on their magnitudes.

Keywords: I2 statistic; heterogeneity; meta-analysis; statistical power.