Objectives: The objectives of this exploratory study were to: (1) describe the frequency of selective outcome reporting (SOR) and selective analysis reporting (SAR) in randomized controlled trials (RCTs) included in reviews of comparative effectiveness for outcomes of benefit; (2) explore potential predictors for SOR and SAR; and (3) assess the reliability and validity of the Outcome Reporting Bias in Trials (ORBIT) classification system for missing or incomplete outcome reporting.
Data Sources and Methods: We selected three comparative effectiveness reviews (CERs) funded by the Agency for Healthcare Research and Quality that included drug–drug comparisons. Within each CER, we then specified one outcome that fulfilled explicit criteria (the “index outcome”) and examined the RCTs in the CER that reported that outcome. We then searched trial registries for study registration information and results for each RCT. Using available registry information to complement information in the methods section of the publication, we determined the frequency of SOR and SAR, and we examined prespecified predictors of SOR and SAR. Lastly, using the ORBIT classification of SOR, we attempted to examine the inter-rater reliability of ORBIT and its validity, comparing information contained within the publication to assessments of SOR, using the additional information obtained from trial registries.
Results:
RCTs published in 2005 or later and reporting the index outcome were not consistently listed in trial registries, with 29 percent, 67 percent, and 75 percent of trials registered for each of the three CERs. In addition, publications did not consistently report trial registration. Results were infrequently listed in
Conclusions:
We identified numerous challenges in identifying and characterizing SOR and SAR in this pilot study of three CERs. Existing tools were suboptimal: ORBIT does not encompass the type of SOR and SAR where results in the publication were not prespecified in the methods section or in the registry. The design of our study (focusing on RCTs with results in the CER) precluded identifying certain types of SOR where the outcomes were not reported at all in the study. The presentation and content of