Binge drinking (BD), a harmful pattern of alcohol consumption, is common during adolescence. Young adults with alcohol use disorders exhibit hippocampal alterations and episodic memory deficits. However, it is not known how these difficulties progress in community BD adolescents. Our objective was to analyze the relationship between BD trajectory and verbal episodic memory during the developmental period spanning from adolescence and to early adulthood. An initial sample of 155 male and female first-year university students with no other risk factors were followed over six years. Participants were classified as stable non-BDs, stable BDs and ex-BDs according to the third AUDIT item. At baseline, participants comprised 36 ♂/ 40 ♀ non-BDs (18.58 years), 40 ♂/ 39 ♀ BDs (18.87 years), and at the third follow-up, they comprised 8 ♂/ 8 ♀ stable non-BDs (25.49 years), 2 ♂/ 2 ♀ stable BDs (25.40) and 8 ♂/ 12 ♀ ex-BDs (24.97 years). Episodic memory was assessed four times with the Logical Memory subtest (WMS-III) and the Rey Auditory Verbal Learning Test (RAVLT). Generalized linear mixed models were applied. The results showed that, relative to non-BDs, stable BDs presented difficulties in immediate and delayed recall in the Logical Memory subtest. These difficulties remained stable over time. The short-term ex-BDs continued to display difficulties in immediate and delayed recall in the Logical Memory subtest, but long-term ex-BDs did not. The effects were not influenced by age of alcohol onset, frequency of cannabis use, tobacco use or psychopathological distress. In conclusion, BD during adolescence and young adulthood is associated with episodic memory deficits. Abandoning the BD pattern may lead to partial recovery. These findings are consistent with the vulnerability of the adolescent hippocampus to the neurotoxic effects of alcohol.