Immunoprofiling is useful for predicting prognosis in various malignancies and provides targets for immunotherapy. Quantitative multispectral imaging system, which allows simultaneous detection of multiple immune markers, is a novel method for examining the tumor immune environment. We compared the expression levels of various surface markers in immune cells between colitis-associated cancer (CAC) and sporadic colorectal cancer (CRC) and evaluated the clinical usefulness of immunoprofiling in CRC. Tumor specimens from 24 CAC patients and 48 sporadic CRC patients, matched by age, sex, and tumor location to CAC, were included in the analysis. The expression levels of CD3, CD8, Foxp3, and programmed death-ligand 1 (PD-L1) in immune cells at the invasive margins of tumor tissues were evaluated by quantitative multispectral imaging. The CAC group had significantly less levels of cells expressing CD3, CD8, Foxp3, or PD-L1 (all, p < 0.01). In the CAC group, patients whose immune cells had high expression of CD3+ and CD8+ had better overall survival. The immune profiling patterns of CAC patients were significantly distinct from those of sporadic CRC patients, suggesting that CAC and sporadic CRC have distinct disease phenotypes. Immunoprofiling can be helpful for evaluation of clinical prognosis in CAC.