Our understanding of the protective immunity, particularly the long-term dynamics of neutralizing antibody (NAbs) response to SARS-CoV-2, is currently limited. We enrolled a cohort of 545 COVID-19 patients from Hubei, China, who were followed up up to 7 months, and determined the dynamics of NAbs to SARS-CoV-2 by using a surrogate virus neutralization test (sVNT). In our validation study, sVNT IC50 titers and the neutralization rate measured at a single dilution (1:20) were well correlated with FRNT titers (r = 0.85 and 0.84, respectively). The median time to seroconversion of NAbs was 5.5 days post onset of symptoms. The rate of positive sVNT was 52% in the first week, reached 100% in the third week, and remained above 97% till 6 months post onset. Quantitatively, NAbs peaked in the fourth week and only a quarter of patients had an estimated peak titer of >1000. NAbs declined with a half-time of 61 days (95% CI: 49-80 days) within the first two months, and the decay deaccelerated to a half-time of 104 days (95% CI: 86-130 days) afterward. The peak levels of NAbs were positively associated with severity of COVID-19 and age, while negatively associated with serum albumin levels. The observation that the low-moderate peak neutralizing activity and fast decay of NAbs in most naturally infected individuals called for caution in evaluating the feasibility of antibody-based therapy and vaccine durability. NAbs response positively correlated with disease severity, warning for the possibility of repeat infection in patients with mild COVID-19.