Despite some inconsistent reporting of symptoms, studies have demonstrated that at least 20% of individuals infected with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) will remain asymptomatic. Although most global efforts have focused on understanding factors underlying severe illness in COVID-19 (coronavirus disease of 2019), the examination of asymptomatic infection provides a unique opportunity to consider early disease and immunologic features promoting rapid viral clearance. Owing to its critical role in the immune response, we postulated that variation in the human leukocyte antigen (HLA) loci may underly processes mediating asymptomatic infection. We enrolled 29,947 individuals registered in the National Marrow Donor Program for whom high-resolution HLA genotyping data were available in the UCSF Citizen Science smartphone-based study designed to track COVID-19 symptoms and outcomes. Our discovery cohort (n=1428) was comprised of unvaccinated, self-identified subjects who reported a positive test result for SARS-CoV-2. We tested for association of five HLA loci (HLA-A, -B, -C, -DRB1, -DQB1) with disease course and identified a strong association of HLA-B*15:01 with asymptomatic infection, and reproduced this association in two independent cohorts. Suggesting that this genetic association is due to pre-existing T-cell immunity, we show that T cells from pre-pandemic individuals carrying HLA-B*15:01 were reactive to the immunodominant SARS-CoV-2 S-derived peptide NQKLIANQF, and 100% of the reactive cells displayed memory phenotype. Finally, we characterize the protein structure of HLA-B*15:01-peptide complexes, demonstrating that the NQKLIANQF peptide from SARS-CoV-2, and the highly homologous NQKLIANAF from seasonal coronaviruses OC43-CoV and HKU1-CoV, share similar ability to be stabilized and presented by HLA-B*15:01, providing the molecular basis for T-cell cross-reactivity and HLA-B*15:01-mediated pre-existing immunity.