The hallmarks of COVID-19 are higher pathogenicity and mortality in the elderly compared to children. Examining baseline SARS-CoV-2 cross-reactive immunological responses, induced by circulating human coronaviruses (hCoVs), is needed to understand such divergent clinical outcomes. Here we show analysis of coronavirus antibody responses of pre-pandemic healthy children (n = 89), adults (n = 98), elderly (n = 57), and COVID-19 patients (n = 50) by systems serology. Moderate levels of cross-reactive, but non-neutralizing, SARS-CoV-2 antibodies are detected in pre-pandemic healthy individuals. SARS-CoV-2 antigen-specific Fcγ receptor binding accurately distinguishes COVID-19 patients from healthy individuals, suggesting that SARS-CoV-2 infection induces qualitative changes to antibody Fc, enhancing Fcγ receptor engagement. Higher cross-reactive SARS-CoV-2 IgA and IgG are observed in healthy elderly, while healthy children display elevated SARS-CoV-2 IgM, suggesting that children have fewer hCoV exposures, resulting in less-experienced but more polyreactive humoral immunity. Age-dependent analysis of COVID-19 patients, confirms elevated class-switched antibodies in elderly, while children have stronger Fc responses which we demonstrate are functionally different. These insights will inform COVID-19 vaccination strategies, improved serological diagnostics and therapeutics.