Systematic Review of Methods in Low-Consensus Fields: Supporting Commensuration through `Construct-Centered Methods Aggregation' in the Case of Climate Change Vulnerability Research

PLoS One. 2016 Feb 22;11(2):e0149071. doi: 10.1371/journal.pone.0149071. eCollection 2016.

Abstract

There is increasing interest in using systematic review to synthesize evidence on the social and environmental effects of and adaptations to climate change. Use of systematic review for evidence in this field is complicated by the heterogeneity of methods used and by uneven reporting. In order to facilitate synthesis of results and design of subsequent research a method, construct-centered methods aggregation, was designed to 1) provide a transparent, valid and reliable description of research methods, 2) support comparability of primary studies and 3) contribute to a shared empirical basis for improving research practice. Rather than taking research reports at face value, research designs are reviewed through inductive analysis. This involves bottom-up identification of constructs, definitions and operationalizations; assessment of concepts' commensurability through comparison of definitions; identification of theoretical frameworks through patterns of construct use; and integration of transparently reported and valid operationalizations into ideal-type research frameworks. Through the integration of reliable bottom-up inductive coding from operationalizations and top-down coding driven from stated theory with expert interpretation, construct-centered methods aggregation enabled both resolution of heterogeneity within identically named constructs and merging of differently labeled but identical constructs. These two processes allowed transparent, rigorous and contextually sensitive synthesis of the research presented in an uneven set of reports undertaken in a heterogenous field. If adopted more broadly, construct-centered methods aggregation may contribute to the emergence of a valid, empirically-grounded description of methods used in primary research. These descriptions may function as a set of expectations that improves the transparency of reporting and as an evolving comprehensive framework that supports both interpretation of existing and design of future research.

Publication types

  • Research Support, Non-U.S. Gov't
  • Review
  • Systematic Review

MeSH terms

  • Climate Change*
  • Models, Theoretical*
  • Research Design*

Grants and funding

This work was support by International Livestock Research Institute (Grant Number: none, https://www.ilri.org/, Recipient: Peter Tamás, Wageningen University). The role of the funders in this research is as follows. Study design: The funders had no role in designing the study other than approval. In fact, the study design did not promise nor did it meet the Funder’s initial expectations (a review of evidence). Data collection: The funder provided the initial set of articles examined. This initial set was supplemented by a sampling procedure that the funder did not influence. Data analysis: The funder served as the “expert” at two points during data analysis. Their role in this process is fully documented. Decision to publish: This article was not initiated by the funder. The funder was asked to support and agreed to participate in publication of this article. Their desired outputs were the technical report and the GEC article. The article submitted to PLONE was driven entirely by academic interest. Preparation of the manuscript: The funder prepared the first draft of the conclusion and commented on other sections. The authors asked the funder to prepare the first draft of the conclusion as neither of the WUR authors are experts in the subject matter and the authors were concerned that their findings be framed in a manner that was engaging to scholars in climate studies.