Adaptive immune responses are induced by vaccination and infection, yet little is known about how CD4+ T cell memory differs when primed in these two contexts. Notably, viral infection is generally associated with higher levels of systemic inflammation than is vaccination. To assess whether the inflammatory milieu at the time of CD4+ T cell priming has long-term effects on memory, we compared Spike-specific memory CD4+ T cells in 22 individuals around the time of the participants' third SARS-CoV-2 mRNA vaccination, with stratification by whether the participants' first exposure to Spike was via virus or mRNA vaccine. Multimodal single-cell profiling of Spike-specific CD4+ T cells revealed 755 differentially expressed genes that distinguished infection- and vaccine-primed memory CD4+ T cells. Spike-specific CD4+ T cells from infection-primed individuals had strong enrichment for cytotoxicity and interferon signaling genes, whereas Spike-specific CD4+ T cells from vaccine-primed individuals were enriched for proliferative pathways by gene set enrichment analysis. Moreover, Spike-specific memory CD4+ T cells established by infection had distinct epigenetic landscapes driven by enrichment of IRF-family transcription factors, relative to T cells established by mRNA vaccination. This transcriptional imprint was minimally altered following subsequent mRNA vaccination or breakthrough infection, reflecting the strong bias induced by the inflammatory environment during initial memory differentiation. Together, these data suggest that the inflammatory context during CD4+ T cell priming is durably imprinted in the memory state at transcriptional and epigenetic levels, which has implications for personalization of vaccination based on prior infection history.