Severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), the causative agent of the pandemic human respiratory illness COVID-19, is a global health emergency. While severe acute disease has been linked to an expansion of antibody-secreting plasmablasts, we sought to identify B cell responses that correlated with positive clinical outcomes in convalescent patients. We characterized the peripheral blood B cell immunophenotype and plasma antibody responses in 40 recovered non-hospitalized COVID-19 subjects that were enrolled as donors in a convalescent plasma treatment study. We observed a significant negative correlation between the frequency of peripheral blood memory B cells and the duration of symptoms for convalescent subjects. Memory B cell subsets in convalescent subjects were composed of classical CD24+ class-switched memory B cells, but also activated CD24-negative and natural unswitched CD27+ IgD+ IgM+ subsets. Memory B cell frequency was significantly correlated with both IgG1 and IgM responses to the SARS-CoV-2 spike protein receptor binding domain (RBD). IgM+ memory, but not switched memory, directly correlated with virus-specific antibody responses, and remained stable over time. Our findings suggest that the frequency of memory B cells is a critical indicator of disease resolution, and that IgM+ memory B cells play an important role in SARS-CoV-2 immunity.