Bibliometric quantification is frequently used as metrics for the evaluation of the scientific performance of researchers and institutions. The researchers' merit is usually assessed by the analysis of quantitative parameters such as the number of publications, the impact factor of journals, the total number of citations, or the h-index, although the limitations in translating these indicators into the impact of the outcome of scientific production are a matter of harsh criticism. To assess, based on factual evidences, the validity of traditional bibliometric analyses to conclude on the impact of papers to advance the state of the art, we carried out an innovative methodology on selected publications (test set). This methodology is based on identifying those citations of the test set papers that truly embed the methods, concepts, or hypotheses to build new knowledge and formulate conclusions. The results show that the percentage of citations that reflect the real impact of the papers of the test set has an average value of 12.4% of total citations and is not related to the impact factor of the journal where the test set papers were published. In conclusion, our analysis demonstrates factually, using experimental data, the total failure of using quantitative bulk citation analyses to conclude on the scientific impact of publications. Only a careful analysis of how the work described in papers was embedded on the subsequent work and/or conclusions of others can tell about the real contribution of a published work to the development of new knowledge and advancement of science.
Keywords: impact; metrics; number of citations; research; scientific evaluation.
© 2020 Federation of European Biochemical Societies.