Maximizing crop yield while at the same time minimizing crop failure for sustainable agriculture requires a better understanding of the impacts of plant breeding on crop genetic diversity. This review identifies knowledge gaps and shows the need for more research into genetic diversity changes under plant breeding. Modern plant breeding has made a profound impact on food production and will continue to play a vital role in world food security. For sustainable agriculture, a compromise should be sought between maximizing crop yield under changing climate and minimizing crop failure under unfavorable conditions. Such a compromise requires better understanding of the impacts of plant breeding on crop genetic diversity. Efforts have been made over the last three decades to assess crop genetic diversity using molecular marker technologies. However, these assessments have revealed some temporal diversity patterns that are largely inconsistent with our perception that modern plant breeding reduces crop genetic diversity. An attempt was made in this review to explain such discrepancies by examining empirical assessments of crop genetic diversity and theoretical investigations of genetic diversity changes over time under artificial selection. It was found that many crop genetic diversity assessments were not designed to assess diversity impacts from specific plant breeding programs, while others were experimentally inadequate and contained technical biases from the sampling of cultivars and genomes. Little attention has been paid to theoretical investigations on crop genetic diversity changes from plant breeding. A computer simulation of five simplified breeding schemes showed the substantial effects of plant breeding on the retention of heterozygosity over generations. It is clear that more efforts are needed to investigate crop genetic diversity in space and time under plant breeding to achieve sustainable crop production.