Large-scale genomic surveys of crop germplasm are important for understanding the genetic architecture of favorable traits. The genomic basis of geographic differentiation and fiber improvement in cultivated cotton is poorly understood. Here, we analyzed 3,248 tetraploid cotton genomes and confirmed that the extensive chromosome inversions on chromosomes A06 and A08 underlies the geographic differentiation in cultivated Gossypium hirsutum. We further revealed that the haplotypic diversity originated from landraces, which might be essential for understanding adaptative evolution in cultivated cotton. Introgression and association analyses identified new fiber quality-related loci and demonstrated that the introgressed alleles from two diploid cottons had a large effect on fiber quality improvement. These loci provided the potential power to overcome the bottleneck in fiber quality improvement. Our study uncovered several critical genomic signatures generated by historical breeding effects in cotton and a wealth of data that enrich genomic resources for the research community.