Adults caring for children in HIV-endemic communities are at risk for poor psychological outcomes. However, we still have a limited understanding of how various HIV impacts--including caregiver's own HIV illness, responsibilities of caring for a child orphaned by AIDS, or both--affect psychological outcomes among caregivers. Furthermore, few studies have explored the relationship between stigma, HIV, and psychological outcomes among caregivers of children in HIV-endemic communities. A cross-sectional survey conducted from 2009 to 2010 assessed anxiety among 2477 caregivers of children in HIV-endemic South Africa. Chi-square tested differences in anxiety among caregivers living with HIV, caregivers of a child orphaned by AIDS, and caregivers affected with both conditions. Multivariate logistic regressions identified whether the relationship between HIV impacts and anxiety remained after controlling for socio-demographic co-factors. Mediation analysis tested the relationship between stigma, HIV, and anxiety. The odds of meeting threshold criteria for clinically relevant anxiety symptoms were two and a half times greater among caregivers living with HIV compared to nonaffected caregivers. The odds of meeting threshold criteria for clinically relevant anxiety symptoms were greatest among caregivers living with HIV and caring for a child orphaned by AIDS. Exposure to AIDS-related stigma partially mediated the relationship between HIV and anxiety. Interventions are needed to address caregiver psychological health, particularly among caregivers affected with both conditions of living with HIV and caring for a child orphaned by AIDS.