Systematic interrogation of tumor-infiltrating immune cells (TIICs) is key to the prediction of clinical outcome and development of immunotherapies. However, little is known about the TIICs of hepatocellular carcinoma (HCC) and its impact on the prognosis of patients and potential for immunotherapy. We applied CIBERSORT of 1090 tumors to infer the infiltration of 22 subsets of TIICs using gene expression data. Unsupervised clustering analysis by 22 TIICs revealed 4 clusters of tumors, mainly defined by macrophages and T cells, with distinct prognosis and associations with immune checkpoint molecules, including PD-1, CD274, CTLA-4, LAG-3 and IFNG. We found tumors with decreased number of M1 macrophages or increased regulatory T cells were associated with poor prognosis. Based on the multivariate Cox analysis, a nomogram was also established for clinical application. In conclusion, composition of the TIICs in HCC was quite different, which is an important determinant of prognosis with great potential to identify candidates for immunotherapy.