Rapid assessments have been emerging on the effects of COVID-19, yet rigorous analyses remain scant. Here, rigorous evidence of the impacts of COVID-19 on several livelihood outcomes are presented, with a particular focus on heterogenous effects of COVID-19. We use a household-level panel dataset consisting of 880 data points collected in rural Bangladesh in 2018 and 2020, and employ difference-in-differences with fixed effects regression techniques. Results suggest that COVID-19 had significant and heterogenous effects on livelihood outcomes. Agricultural production and share of production sold were reduced, especially for rice crops. Further, diet diversity and education expenditure were reduced for the total sample. Households primarily affected by (fear of) sickness had a significantly lower agricultural production, share of crop market sales, and lower health and education expenditure, compared to households affected by other COVID-19 effects, such as travel restrictions. In turn, (fear of) sickness and the correlated reduced incidence of leaving the house, resulted in higher off-farm incomes suggesting that households engage in less physically demanding and localized work. Policy-makers need to be cognizant of these heterogenous COVID-19 effects and formulate policies that are targeted at those households that are most vulnerable (e.g., unable/willing to leave the house due to (fear of) sickness).