Immunoglobulin detection is essential for diagnosing progression of SARS-CoV-2 infection, for which SARS-CoV-2 IgG is one of the most important indexes. In this paper, Ag nanoparticles with ultrathin Au shells (∼2 nm) embedded with 4-mercaptobenzoic acid (MBA) (AgMBA@Au) were manufactured via a ligand-assisted epitaxial growth method and integrated into lateral flow immunoassay (LFIA) for colorimetric and SERS dual-mode detection of SARS-CoV-2 IgG. AgMBA@Au possessed not only the surface chemistry advantages of Au but also the superior optical characteristics of Ag. Moreover, the nanogap between the Ag core and the Au shell also greatly enhanced the Raman signal. After being modified with anti-human antibodies, AgMBA@Au recognized and combined with SARS-CoV-2 IgG, which was captured by the SARS-CoV-2 spike protein on the T line. Qualitative analysis was achieved by visually observing the color of the T line, and quantitative analysis was conducted by measuring the SERS signal with a sensitivity four orders of magnitude higher (detection limit: 0.22 pg/mL). The intra-assay and inter-assay variation coefficients were 7.7 and 10.3%, respectively, and other proteins at concentrations of 10 to 20 times higher than those of SARS-CoV-2 IgG could hardly produce distinguishable signals, confirming good reproducibility and specificity. Finally, this method was used to detect 107 clinical serum samples. The results agreed well with those obtained from enzyme-linked immunosorbent assay kits and were significantly better than those of the colloidal gold test strips. Therefore, this dual-mode LFIA has great potential in clinical practical applications and can be used to screen and trace the early immune response of SARS-CoV-2.