HIV infection can be effectively controlled by anti-retroviral therapy (ART) in most patients. However therapy must be continued for life, because interruption of ART leads to rapid recrudescence of infection from long-lived latently infected cells. A number of approaches are currently being developed to 'purge' the reservoir of latently infected cells in order to either eliminate infection completely, or significantly delay the time to viral recrudescence after therapy interruption. A fundamental question in HIV research is how frequently the virus reactivates from latency, and thus how much the reservoir might need to be reduced to produce a prolonged antiretroviral-free HIV remission. Here we provide the first direct estimates of the frequency of viral recrudescence after ART interruption, combining data from four independent cohorts of patients undergoing treatment interruption, comprising 100 patients in total. We estimate that viral replication is initiated on average once every ≈6 days (range 5.1- 7.6 days). This rate is around 24 times lower than previous thought, and is very similar across the cohorts. In addition, we analyse data on the ratios of different 'reactivation founder' viruses in a separate cohort of patients undergoing ART-interruption, and estimate the frequency of successful reactivation to be once every 3.6 days. This suggests that a reduction in the reservoir size of around 50-70-fold would be required to increase the average time-to-recrudescence to about one year, and thus achieve at least a short period of anti-retroviral free HIV remission. Our analyses suggests that time-to-recrudescence studies will need to be large in order to detect modest changes in the reservoir, and that macaque models of SIV latency may have much higher frequencies of viral recrudescence after ART interruption than seen in human HIV infection. Understanding the mean frequency of recrudescence from latency is an important first step in approaches to prolong antiretroviral-free viral remission in HIV.