There is an urgent need to understand the nature of immune responses generated against SARS-CoV-2, to better inform risk-mitigation strategies for people living with HIV (PLWH). Although not all PLWH are considered immunosuppressed, residual cellular immune deficiency and ongoing inflammation could influence COVID-19 disease severity, the evolution and durability of protective memory responses. Here, we performed an integrated analysis, characterizing the nature, breadth and magnitude of SARS-CoV-2-specific immune responses in PLWH, controlled on ART, and HIV negative subjects. Both groups were in the convalescent phase of predominately mild COVID-19 disease. The majority of PLWH mounted SARS-CoV-2 Spike- and Nucleoprotein-specific antibodies with neutralizing activity and SARS-CoV-2-specific T cell responses, as measured by ELISpot, at levels comparable to HIV negative subjects. T cell responses against Spike, Membrane and Nucleocapsid were the most prominent, with SARS-CoV-2-specific CD4 T cells outnumbering CD8 T cells. Notably, the overall magnitude of SARS-CoV-2-specific T cell responses related to the size of the naive CD4 T cell pool and the CD4:CD8 ratio in PLWH, in whom disparate antibody and T cell responses were observed. Both humoral and cellular responses to SARS-CoV-2 were detected at 5-7 months post-infection, providing evidence of medium-term durability of responses irrespective of HIV serostatus. Incomplete immune reconstitution on ART and a low CD4:CD8 ratio could, however, hamper the development of immunity to SARS-CoV-2 and serve as a useful tool for risk stratification of PLWH. These findings have implications for the individual management and potential effectiveness of vaccination against SARS-CoV-2 in PLWH.