The current outbreak of the coronavirus disease 2019 (COVID-19) is an unprecedented example of how fast an infectious disease can spread around the globe (especially in urban areas) and the enormous impact it causes on public health and socio-economic activities. Despite the recent surge of investigations about different aspects of the COVID-19 pandemic, we still know little about the effects of city size on the propagation of this disease in urban areas. Here we investigate how the number of cases and deaths by COVID-19 scale with the population of Brazilian cities. Our results indicate small towns are proportionally more affected by COVID-19 during the initial spread of the disease, such that the cumulative numbers of cases and deaths per capita initially decrease with population size. However, during the long-term course of the pandemic, this urban advantage vanishes and large cities start to exhibit higher incidence of cases and deaths, such that every 1% rise in population is associated with a 0.14% increase in the number of fatalities per capita after about four months since the first two daily deaths. We argue that these patterns may be related to the existence of proportionally more health infrastructure in the largest cities and a lower proportion of older adults in large urban areas. We also find the initial growth rate of cases and deaths to be higher in large cities; however, these growth rates tend to decrease in large cities and to increase in small ones over time.