Using single-cell cytometry to illustrate integrated multi-perspective evaluation of clustering algorithms using Pareto fronts

Givanna H Putri; Irena Koprinska; Thomas M Ashhurst; Nicholas J C King; Mark N Read

doi:10.1093/bioinformatics/btab038

Using single-cell cytometry to illustrate integrated multi-perspective evaluation of clustering algorithms using Pareto fronts

Bioinformatics. 2021 Jan 28:btab038. doi: 10.1093/bioinformatics/btab038. Online ahead of print.

Authors

Givanna H Putri^{1

2}, Irena Koprinska¹, Thomas M Ashhurst^{3

2}, Nicholas J C King^{3

4

2}, Mark N Read^{1

5

2}

Affiliations

¹ School of Computer Science, The University of Sydney, Sydney, 2006, Australia.
² Charles Perkins Centre, The University of Sydney, Sydney, 2006, Australia.
³ Sydney Cytometry Facility, The University of Sydney and Centenary Institute, Sydney, 2006, Australia.
⁴ Discipline of Pathology, The University of Sydney, Sydney, 2006, Australia.
⁵ Westmead Initiative, The University of Sydney, Sydney, 2006, Australia.

PMID: 33508103
DOI: 10.1093/bioinformatics/btab038

Abstract

Motivation: Many 'automated gating' algorithms now exist to cluster cytometry and single cell sequencing data into discrete populations. Comparative algorithm evaluations on benchmark datasets rely either on a single performance metric, or a few metrics considered independently of one another. However, single metrics emphasise different aspects of clustering performance and do not rank clustering solutions in the same order. This underlies the lack of consensus between comparative studies regarding optimal clustering algorithms and undermines the translatability of results onto other non-benchmark datasets.

Results: We propose the Pareto fronts framework as an integrative evaluation protocol, wherein individual metrics are instead leveraged as complementary perspectives. Judged superior are algorithms that provide the best trade-off between the multiple metrics considered simultaneously. This yields a more comprehensive and complete view of clustering performance. Moreover, by broadly and systematically sampling algorithm parameter values using the Latin Hypercube sampling method, our evaluation protocol minimises (un)fortunate parameter value selections as confounding factors. Furthermore, it reveals how meticulously each algorithm must be tuned in order to obtain good results, vital knowledge for users with novel data. We exemplify the protocol by conducting a comparative study between three clustering algorithms (ChronoClust, FlowSOM and Phenograph) using four common performance metrics applied across four cytometry benchmark datasets. To our knowledge, this is the first time Pareto fronts have been used to evaluate the performance of clustering algorithms in any application domain.

Availability: Implementation of our Pareto front methodology and all scripts to reproduce this article are available at https://github.com/ghar1821/ParetoBench.