Previous studies evaluated the effects of psychosocial stress on influenza vaccine responses. However, there were methodological limitations. This study aims to determine whether chronic stress is associated with poorer influenza-specific immune responses to influenza vaccines in Hong Kong Chinese elderly people. This is a prospective study with a 12-week follow-up. Subjects were recruited from government general out-patient clinics, non-government organizations, and public housing estates in Hong Kong. Participants include 55 caregivers of spouses with chronic conditions that impaired their activities of daily living and 61 age- and sex-matched non-caregivers. A single-dose trivalent influenza vaccine was given to all subjects by intramuscular ingestion. Blood samples were collected before vaccination, at 6 weeks, and at 12 weeks after vaccination. Influenza vaccine strain-specific antibody titers were measured by the hemagglutination inhibition method. Lymphocyte subsets were analyzed for ratios and absolute counts, and cytokine concentration were measured by flow cytometry. Validated scales were used to assess psychological (depressive symptoms, perceived stress, and caregiver strain), social (multidimensional social support scale), and lifestyle factors (physical exercise, cigarette smoking, and alcohol consumption) at baseline prior to vaccination. Demographic and socioeconomic variables were also collected. Albumin levels were measured as an indicator for nutritional status in subjects. Caregivers had statistically significant (p < 0.05) lower cell-mediated immune responses to influenza vaccination at 12 weeks when compared with those of the controls. No differences in humoral immune response to vaccination were observed between caregivers and controls. Hong Kong Chinese elderly who experience chronic stress have a significantly lower cell-mediated immune response to influenza vaccination when compared with non-caregivers.