Understanding factors that promote or prevent adherence to recommended health behaviors is essential for developing effective health programs, particularly among lower income populations who carry a disproportionate burden of disease. We conducted in-depth qualitative interviews (n = 64) with low-income Black and Latina women who shared the experience of requiring diagnostic follow-up after having a screening mammography with abnormal findings. We found that in addition to holding negative and fatalistic cancer-related beliefs, the social context of these women was largely defined by multiple challenges and major life stressors, factors that may interfere with their ability to attain health. Factors commonly mentioned included competing health issues, economic hardship, demanding caretaking responsibilities and relationships, insurance-related challenges, distrust of health care providers, and inflexible work policies. Black women also reported discrimination and medical mistrust, whereas Latinas experienced difficulties associated with immigration and social isolation. These results suggest that effective health interventions not only address change among individuals but must also change health care systems and social policies in order to reduce health disparities.