PD-1/PD-L1 checkpoint blockade has demonstrated great success in cancer immunotherapy. Small-molecule PD-L1 inhibitors also attract significant research interests but remain challenging in the efficacy and safety. Carbohydrate moiety and carbohydrate-binding proteins (lectins) play important roles in immune modulation including antigen recognition and presenting. Herein, we reported a novel strategy to strengthen the immunotherapeutic effect of small-molecule PD-L1 inhibitors by introducing sugar motifs, which may utilize the carbohydrate-mediated immune enhancement for cancer treatment. The data revealed that glycoside compounds containing mannose or N-acetylglucosamine exhibited the best results in IFN-γ secretion. Moreover, compared to the nonglycosylated compounds, glycosides C3 and C15 demonstrated significant lower cytotoxicity and effective in vivo antitumor potency in the CT26 and melanoma B16-F10 tumor models with good tolerance. Notably, tumor-infiltrating lymphocyte (TIL) analysis validated increased CD3+, CD4+, CD8+, and granzyme B+ T cells after glycoside treatments. This work presents a new concept to improve the immunotherapy.