Aim: To compare reviewers recommendations and editorial decisions in The Lancet, a high-impact journal, and Croatian Medical Journal (CMJ), a small general medical journal.
Method: Case study of research manuscripts submitted to the CMJ (n=140 manuscripts; 308 review forms) and a sample of similar manuscripts submitted to The Lancet (n=141; 348 review forms) in 1999 and the first half of 2000. Reviewers recommendations and their influence on editorial decisions on manuscripts were analyzed by logistic regression. Agreement between reviewers was assessed by the kappa statistic.
Results: Although reviewers scores were identical in the two journals (median=3 for both journals, range 0 to 5), Lancet reviewers more often recommended rejection than CMJ reviewers (44% vs 17%; chi-square=52.1, p=0.029), and agreed best on rejection (kappa=0.29 [95% CI=0.04 to 0.53] vs kappa=-0.04 [-0.45 to 0.36] for CMJ). Lancet editors were even stricter than their reviewers and accepted for publication only 53% [95% CI=37 to 68] of manuscripts graded acceptable by the reviewers, compared with 85% [73 to 91] for CMJ editors (chi-square=10.0, p=0.001). For nine questions about manuscript quality in the review form, multiple regression analysis showed significant association between editorial decision and reviewers scores for the suitability of research design (B=0.70, OR=2.01, 95% CI=1.40 to 2.89, p<0.001) and discussion of systematic/random error (B=0.32, OR=1.38 [1.03 to 1.85], p=0.031) for The Lancet, and scores for novelty of information (B=0.56, OR=1.75 [1.35 to 2.27], p<0.001) for the CMJ.
Conclusion: Reviewers of big journals, such as The Lancet, are stricter in their recommendations reviewers of a smaller journal, such as the CMJ. The Lancet editors rely on reviewers to identify methodologically superior studies, whereas CMJ editors look for the novelty of information in a manuscript.