Essential guidelines for computational method benchmarking

Lukas M Weber; Wouter Saelens; Robrecht Cannoodt; Charlotte Soneson; Alexander Hapfelmeier; Paul P Gardner; Anne-Laure Boulesteix; Yvan Saeys; Mark D Robinson

doi:10.1186/s13059-019-1738-8

Essential guidelines for computational method benchmarking

Genome Biol. 2019 Jun 20;20(1):125. doi: 10.1186/s13059-019-1738-8.

Authors

Lukas M Weber^{1

2}, Wouter Saelens^{3

4}, Robrecht Cannoodt^{3

4}, Charlotte Soneson^{1

2

5}, Alexander Hapfelmeier⁶, Paul P Gardner⁷, Anne-Laure Boulesteix⁸, Yvan Saeys^{9

10}, Mark D Robinson^{11

12}

Affiliations

¹ Institute of Molecular Life Sciences, University of Zurich, 8057, Zurich, Switzerland.
² SIB Swiss Institute of Bioinformatics, University of Zurich, 8057, Zurich, Switzerland.
³ Data Mining and Modelling for Biomedicine, VIB Center for Inflammation Research, 9052, Ghent, Belgium.
⁴ Department of Applied Mathematics, Computer Science and Statistics, Ghent University, 9000, Ghent, Belgium.
⁵ Present address: Friedrich Miescher Institute for Biomedical Research and SIB Swiss Institute of Bioinformatics, 4058, Basel, Switzerland.
⁶ Institute of Medical Informatics, Statistics and Epidemiology, Technical University of Munich, 81675, Munich, Germany.
⁷ Department of Biochemistry, University of Otago, Dunedin, 9016, New Zealand.
⁸ Institute for Medical Information Processing, Biometry and Epidemiology, Ludwig-Maximilians-University, 81377, Munich, Germany.
⁹ Data Mining and Modelling for Biomedicine, VIB Center for Inflammation Research, 9052, Ghent, Belgium. yvan.saeys@ugent.be.
¹⁰ Department of Applied Mathematics, Computer Science and Statistics, Ghent University, 9000, Ghent, Belgium. yvan.saeys@ugent.be.
¹¹ Institute of Molecular Life Sciences, University of Zurich, 8057, Zurich, Switzerland. mark.robinson@imls.uzh.ch.
¹² SIB Swiss Institute of Bioinformatics, University of Zurich, 8057, Zurich, Switzerland. mark.robinson@imls.uzh.ch.

Abstract

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology.

Publication types

Research Support, Non-U.S. Gov't
Review

MeSH terms

Benchmarking
Computational Biology / standards*
Datasets as Topic
Guidelines as Topic*
Publishing
Research Design
Software