In the absence of pharmaceutical interventions, social distancing is being used worldwide to curb the spread of COVID-19. The impact of these measures has been inconsistent, with some regions rapidly nearing disease elimination and others seeing delayed peaks or nearly flat epidemic curves. Here we build a stochastic epidemic model to examine the effects of COVID-19 clinical progression and transmission network structure on the outcomes of social distancing interventions. Our simulations show that long delays between the adoption of control measures and observed declines in cases, hospitalizations, and deaths occur in many scenarios. We find that the strength of within-household transmission is a critical determinant of success, governing the timing and size of the epidemic peak, the rate of decline, individual risks of infection, and the success of partial relaxation measures. The structure of residual external connections, driven by workforce participation and essential businesses, interacts to determine outcomes. We suggest limited conditions under which the formation of household "bubbles" can be safe. These findings can improve future predictions of the timescale and efficacy of interventions needed to control second waves of COVID-19 as well as other similar outbreaks, and highlight the need for better quantification and control of household transmission.