Label-free image identification of circulating rare cells, such as circulating tumor cells within peripheral blood nucleated cells (PBNCs), the vast majority of which are white blood cells (WBCs), remains challenging. We previously described developing label-free image cytometry for classifying live cells using computer vision technology for pattern recognition, based on the subcellular structure of the quantitative phase microscopy images. We applied our image recognition methods to cells flowing in a flow cytometer microfluidic channel, and differentiated WBCs from cancer cell lines (area under receiver operating characteristic curve = 0.957). We then applied this method to healthy volunteers' and advanced cancer patients' blood samples and found that the non-WBC fraction rates (NWBC-FRs), defined as the percentage of cells classified as non-WBCs of the total PBNCs, were significantly higher in cancer patients than in healthy volunteers. Furthermore, we monitored NWBC-FRs over the therapeutic courses in cancer patients, which revealed the potential ability in monitoring the clinical status during therapy. Our image recognition system has the potential to provide a morphological diagnostic tool for circulating rare cells as non-WBC fractions.