Metastatic tumors have poor prognoses for progression-free and overall survival for all cancer patients. Rare circulating tumor cells (CTCs) and rarer circulating tumor cell clusters (CTCCs) are potential biomarkers of metastatic growth, with CTCCs representing an increased risk factor for metastasis. Current detection platforms are optimized for ex vivo detection of CTCs only. Microfluidic chips and size exclusion methods have been proposed for CTCC detection; however, they lack in vivo utility and real-time monitoring capability. Confocal backscatter and fluorescence flow cytometry (BSFC) has been used for label-free detection of CTCCs in whole blood based on machine learning (ML) enabled peak classification. Here, we expand to a deep-learning (DL) -based, peak detection and classification model to detect CTCCs in whole blood data. We demonstrate that DL-based BSFC has a low false alarm rate of 0.78 events/min with a high Pearson correlation coefficient of 0.943 between detected events and expected events. DL-based BSFC of whole blood maintains a detection purity of 72% and a sensitivity of 35.3% for both homotypic and heterotypic CTCCs starting at a minimum size of two cells. We also demonstrate through artificial spiking studies that DL-based BSFC is sensitive to changes in the number of CTCCs present in the samples and does not add variability in detection beyond the expected variability from Poisson statistics. The performance established by DL-based BSFC motivates its use for in vivo detection of CTCCs. Further developments of label-free BSFC to enhance throughput could lead to critical applications in the clinical detection of CTCCs and ex vivo isolation of CTCC from whole blood with minimal disruption and processing steps.