Assessment of variables that influence agreement between reviewers for Foot & Ankle International

Abstract

Introduction: A common criticism of the peer-review process is the often disparate nature of reviewer recommendations when a decision is rendered which belies the supposed uniformity of the process. The purpose of this investigation was to examine level of agreement between reviewers for Foot & Ankle International (FAI) and analyze variables which may have influenced agreement in order to better understand the peer-review process.

Methods: Approval to conduct this investigation was obtained from the Executive Board and Editor in Chief of FAI. All manuscripts submitted to FAI during the calendar year 2016 which underwent formal peer-review were included in the analysis. For each reviewed manuscript, demographic data was collected regarding specific reviewer and manuscript characteristics in a de-identified manner.

Results: 442 manuscripts underwent formal blinded peer-review by two independent reviewers during the study period. Only 199 manuscripts (45%) had a decision rendered in which both reviewers agreed on the same initial recommendation. There were no differences in demographic characteristics between the group of reviewers who agreed as compared to those who disagreed on the initial round of peer review. A similar number of indexed peer-reviewed publications between reviewers correlated with increased levels of agreement.

Conclusions: During the study period, there was 45% initial agreement between reviewers for FAI when assessing the same manuscript. Aside from research productivity, no other reviewer-specific variables examined in this investigation were found to correlate with agreement. Specific recommendations and changes may be considered to increase the efficiency and effectiveness of the peer-review process.

Keywords: Foot and Ankle International journal; Peer review; Publishing standards; Reviewers´agreement.