Understanding how antibody responses to SARS-CoV-2 evolve during infection may provide important insight into therapeutic approaches and vaccination for COVID-19. Here we profile the antibody responses of 162 COVID-19 symptomatic patients in the COVID-BioB cohort followed longitudinally for up to eight months from symptom onset to find SARS-CoV-2 neutralization, as well as antibodies either recognizing SARS-CoV-2 spike antigens and nucleoprotein, or specific for S2 antigen of seasonal beta-coronaviruses and hemagglutinin of the H1N1 flu virus. The presence of neutralizing antibodies within the first weeks from symptoms onset correlates with time to a negative swab result (p = 0.002), while the lack of neutralizing capacity correlates with an increased risk of a fatal outcome (p = 0.008). Neutralizing antibody titers progressively drop after 5-8 weeks but are still detectable up to 8 months in the majority of recovered patients regardless of age or co-morbidities, with IgG to spike antigens providing the best correlate of neutralization. Antibody responses to seasonal coronaviruses are temporarily boosted, and parallel those to SARS-CoV-2 without dampening the specific response or worsening disease progression. Our results thus suggest compromised immune responses to the SARS-CoV-2 spike to be a major trait of COVID-19 patients with critical conditions, and thereby inform on the planning of COVID-19 patient care and therapy prioritization.