Accurate classification of malignant cells from benign ones can significantly enhance cancer diagnosis and prognosis by detection of circulating tumor cells (CTCs). We have investigated two approaches of quantitative morphology and polarization diffraction imaging on two prostate cell types to evaluate their feasibility as single-cell assay methods toward CTC detection after cell enrichment. The two cell types have been measured by a confocal imaging method to obtain their three-dimensional morphology parameters and by a polarization diffraction imaging flow cytometry (p-DIFC) method to obtain image texture parameters. The support vector machine algorithm was applied to examine the accuracy of cell classification with the morphology and diffraction image parameters. Despite larger mean values of cell and nuclear sizes of the cancerous prostate cells than the normal ones, it has been shown that the morphologic parameters cannot serve as effective classifiers. In contrast, accurate classification of the two prostate cell types can be achieved with high classification accuracies on measured data acquired separately in three measurements. These results provide strong evidence that the p-DIFC method has the potential to yield morphology-related “fingerprints” for accurate and label-free classification of the two prostate cell types.