The rapid early spread of COVID-19 in the US was experienced very differently by different socioeconomic groups and business industries. In this study, we study aggregate mobility patterns of New York City and Chicago to identify the relationship between the amount of interpersonal contact between people in urban neighborhoods and the disparity in the growth of positive cases among these groups. We introduce an aggregate spatiotemporal contact density index (CDI) to measure the strength of this interpersonal contact using mobility data collected from mobile phones, and combine it with social distancing metrics to show its effect on positive case growth. With the help of structural equations modeling, we find that the effect of CDI on case growth was consistently positive and that it remained consistently higher in lower-income neighborhoods, suggesting a causal path of income on case growth via CDI. Using the CDI, schools and restaurants are identified as high contact density industries, and the estimation suggests that implementing specific mobility restrictions on these point-of-interest categories is most effective. This analysis can be useful in providing insights for government officials targeting specific population groups and businesses to reduce infection spread as reopening efforts continue to expand across the nation.