Alterations in three-dimensional (3D) genome structures are associated with cancer1-5. However, how genome folding evolves and diversifies during subclonal cancer progression in the native tissue environment remains unknown. Here, we leveraged a genome-wide chromatin tracing technology to directly visualize 3D genome folding in situ in a faithful Kras-driven mouse model of lung adenocarcinoma (LUAD)6, generating the first single-cell 3D genome atlas of any cancer. We discovered stereotypical 3D genome alterations during cancer development, including a striking structural bottleneck in preinvasive adenomas prior to progression to LUAD, indicating a stringent selection on the 3D genome early in cancer progression. We further showed that the 3D genome precisely encodes cancer states in single cells, despite considerable cell-to-cell heterogeneity. Finally, evolutionary changes in 3D genome compartmentalization - partially regulated by polycomb group protein Rnf2 through its ubiquitin ligase-independent activity - reveal novel genetic drivers and suppressors of LUAD progression. Our results demonstrate the importance of mapping the single-cell cancer 3D genome and the potential to identify new diagnostic and therapeutic biomarkers from 3D genomic architectures.