The apple (Malus domestica) is one of the world's most commercially important perennial crops and its improvement has been the focus of human effort for thousands of years. Here, we genetically characterise over 1000 apple accessions from the United States Department of Agriculture (USDA) germplasm collection using over 30,000 single-nucleotide polymorphisms (SNPs). We confirm the close genetic relationship between modern apple cultivars and their primary progenitor species, Malus sieversii from Central Asia, and find that cider apples derive more of their ancestry from the European crabapple, Malus sylvestris, than do dessert apples. We determine that most of the USDA collection is a large complex pedigree: over half of the collection is interconnected by a series of first-degree relationships. In addition, 15% of the accessions have a first-degree relationship with one of the top 8 cultivars produced in the USA. With the exception of 'Honeycrisp', the top 8 cultivars are interconnected to each other via pedigree relationships. The cultivars 'Golden Delicious' and 'Red Delicious' were found to have over 60 first-degree relatives, consistent with their repeated use by apple breeders. We detected a signature of intense selection for red skin and provide evidence that breeders also selected for increased firmness. Our results suggest that Americans are eating apples largely from a single family tree and that the apple's future improvement will benefit from increased exploitation of its tremendous natural genetic diversity.