T cells play a central role in cancer immunotherapy, but we lack systematic comparison of the heterogeneity and dynamics of tumor-infiltrating T cells across cancer types. We built a single-cell RNA-sequencing pan-cancer atlas of T cells for 316 donors across 21 cancer types and revealed distinct T cell composition patterns. We found multiple state-transition paths in the exhaustion of CD8+ T cells and the preference of those paths among different tumor types. Certain T cell populations showed specific correlation with patient properties such as mutation burden, shedding light on the possible determinants of the tumor microenvironment. T cell compositions within tumors alone could classify cancer patients into groups with clinical trait specificity, providing new insights into T cell immunity and precision immunotherapy targeting T cells.