Background: Cumulative meta-analysis typically involves performing an updated meta-analysis every time when new trials are added to a series of similar trials, which by definition involves multiple inspections. Neither the commonly used random effects model nor the conventional group sequential method can control the type I error for many practical situations. In our previous research, Lan et al. (Lan KKG, Hu M-X, Cappelleri JC. Applying the law of iterated logarithm to cumulative meta-analysis of a continuous endpoint. Statistica Sinica 2003; 13: 1135-45) proposed an approach based on the law of iterated logarithm (LIL) to this problem for the continuous case.
Purpose: The study is an extension and generalization of our previous research to binary outcomes. Although it is based on the same LIL principle, we found the discrete case much more complex and the results from the continuous case do not apply to the binary case. The simulation study presented here is also more extensive.
Methods: The LIL based method ;penalizes' the Z-value of the test statistic to account for multiple tests and for the estimation of heterogeneity in treatment effects across studies. It involves an adjustment factor, which is directly related to the control of type I error and determined through extensive simulations under various conditions.
Results: With an adjustment factor of 2, the LIL-based test statistics controls the overall type I error when odds ratio or relative risk is the parameter of interest. For risk difference, the adjustment factor can be reduced to 1.5. More inspections may require a larger adjustment factor, but the required adjustment factor stabilizes after 25 inspections.
Limitations: It will be ideal if the adjustment factor can be obtained theoretically through a statistical model. Unfortunately, real life data are too complex and we have to solve the problem through simulation. However, for large number of inspections, the adjustment factor will have a limited effect and the type I error is controlled mainly by the LIL.
Conclusions: The LIL method controls the overall type I error for a very broad range of practical situations with a binary outcome, and the LIL works properly in controlling the type I error rates as the number of inspections becomes large.