Pitfalls in retrospective analyses of biomarkers: a case study with metastatic melanoma patients

Abstract

Background: Reliable prognostic biomarkers of survival and response to treatment are clearly important in oncology, and many studies have been carried out with the objective of identifying new prognostic biomarkers. Retrospective analysis of blood banked from patients is a frequently used paradigm for these studies. We describe a new study of the association of serum biomarker level with overall survival in melanoma patients, and the problems encountered in carrying it out.

Methods: Blood samples from 56 patients with stage IV metastatic melanoma were drawn prior to initiation of any treatment for their disease. Sera from the samples were stored for up to 94 months at -80°C, and were subsequently thawed at the same time and tested by multiplex Luminex assay for 30 analytes (cytokines, chemokines and growth factors). Cox regression analysis was used to assess the association between these analytes and time-to-death.

Results: Of the 30 analytes, 17 were associated with survival, most strongly so, and in all cases, a higher analyte level was associated with increased survival. In addition, the correlations of the levels of all possible pairs of analytes were all positive and in almost all cases highly significant. However, these results are artifacts that arise from the combination of two peculiarities of the data: the apparent decrease in analyte level with storage time, and the uniformly shorter storage times of the samples from censored patients than the storage times of the samples from patients who died.

Conclusions: All retrospective studies can have hidden biases, and thus investigators should not claim new findings before examining the data in detail with the goal of determining whether the findings could be spurious. There were several suspicious findings in our initial analyses: too many analytes found significant, too many very small p-values, a uniformly positive association of analyte level with survival, and a uniformly positive correlation between analyte levels. We were convinced that these findings must be artifacts, and further analyses showed that the findings could be explained by an apparent decrease of analyte level storage time.