Rationale, aims and objectives: Systematic reviews, meta-analyses and clinical guidelines (reviews) are intended to inform clinical practice, and in this sense can be thought of as scientific truthmakers. High-quality controlled trials should align to this truth, and method quality markers should predict truth status. We sought to determine in what way controlled trial quality relates to scientific truth, and to determine predictive utility of trial quality and bibliographic markers.
Method: A sample of reviews in rehabilitation medicine was examined. Two scientific truth dimensions were established based on review outcomes. Quality and bibliographic markers were extracted from associated trials for use in a regression analysis of their predictive utility for trial truth status. Probability analysis was undertaken to examine judgments of future trial truth status.
Results: Of the 93 trials included in contemporaneous reviews, overall, n = 45 (48%) were true. Randomization was found more in true trials than false trials in one truth dimension (P = 0.03). Intention-to-treat analysis was close to significant in one truth dimension (P = 0.058), being more commonly used in false trials. There were no other significant differences in quality or bibliographic variables between true and false trials. Regression analysis revealed no significant predictors of trial truth status. Probability analysis reported that the reasonable chance of future trials being true was between 2 and 5%, based on a uniform prior.
Conclusions: The findings are at odds with what is considered gold-standard research methods, but in line with previous reports. Further work should focus on scientific dynamics within healthcare research and evidence-based practice constructs.
Keywords: evidence-based practice; philosophy; randomized controlled trials; rehabilitation; systematic reviews; truth.
© 2012 John Wiley & Sons Ltd.