In the aftermath of shock events, policy responses tend to be crafted under significant time constraints and high levels of uncertainty. The extent to which individuals comply with different policy designs can further influence how effective the policy responses are and how equitably their impacts are distributed in the population. Tools which allow policymakers to model different crisis trajectories, policy responses, and behavioral scenarios ex ante can provide crucial timely support in the decision-making process. Set in the context of COVID-19 shelter in place policies, in this paper we present the COVID-19 Policy Evaluation (CoPE) tool, which is an agent-based modeling framework that enables researchers and policymakers to anticipate the relative impacts of policy decisions. Specifically, this framework illuminates the extent to which policy design features and behavioral responsiveness influence the efficacy and equity of policy responses to shock events. We show that while an early policy response can be highly effective, the impact of the timing is moderated by other aspects of policy design such as duration and targeting of the policy, as well as societal aspects such as trust and compliance among the population. More importantly, we show that even policies that are more effective overall can have disproportionate impacts on vulnerable populations. By disaggregating the impact of different policy design elements on different population groups, we provide an additional tool for policymakers to use in the design of targeted strategies for disproportionately affected populations.