Vaccine-induced immunity may impact subsequent de novo responses to drifted epitopes in SARS-CoV-2 variants, but this has been difficult to quantify due to the challenges in recruiting unvaccinated control groups whose first exposure to SARS-CoV-2 is a primary infection. Through local, statewide, and national SARS-CoV-2 testing programs, we were able to recruit cohorts of individuals who had recovered from either primary or post-vaccination infections by either the Delta or Omicron BA.1 variants. Regardless of variant, we observed greater Spike-specific and neutralizing antibody responses in post-vaccination infections than in those who were infected without prior vaccination. Through analysis of variant-specific memory B cells as markers of de novo responses, we observed that Delta and Omicron BA.1 infections led to a marked shift in immunodominance in which some drifted epitopes elicited minimal responses, even in primary infections. Prior immunity through vaccination had a small negative impact on these de novo responses, but this did not correlate with cross-reactive memory B cells, arguing against competitive inhibition of naïve B cells. We conclude that dampened de novo B cell responses against drifted epitopes are mostly a function of altered immunodominance hierarchies that are apparent even in primary infections, with a more modest contribution from pre-existing immunity, perhaps due to accelerated antigen clearance.