A framework for nucleus and overlapping cytoplasm segmentation in cervical cytology extended depth of field and volume images

Abstract

We propose a framework to detect and segment nuclei and segment overlapping cytoplasm in cervical cytology images. This is a challenging task due to folded cervical cells with spurious edges, poor contrast of cytoplasm and presence of neutrophils and artifacts. The algorithm segments nuclei and cell clumps in extended depth of field (EDF) images and uses volume images to segment overlapping cytoplasm. The boundaries are first approximated by a defined similarity metric and are refined in two steps by reducing concavity, iterative smoothing and outliers removal. We evaluated our framework on two public datasets provided in the first and second overlapping cervical cell segmentation challenges (ISBI 2014 and 2015). The results show that our method outperforms other state-of-the-art algorithms on both datasets. The results on the ISBI 2014 dataset show that our method missed less than 5% of cells when the pairwise cell overlapping degree was not higher than 0.3 and it missed only 7% of cells on average in a dataset of 810 synthetic images with 4860 (overlapping) cells. On the same dataset, it outperforms other state-of-the-art methods in nucleus detection with precision 0.961 and recall 0.933. The results on the ISBI 2015 dataset containing real cervical EDF images show that our method misses around 20% of cells in EDF images where a segmentation is considered a miss if it has dice similarity coefficient not greater than 0.7. The 20% miss rate is around half of the miss rate of two other recent methods.

Keywords: Cervical cytology; Cytoplasm segmentation; Multi-layer cytology; Nucleus detection; Overlapping cell segmentation.