As an immune response to COVID-19 infection, patients develop SARS-CoV-2-specific IgM/IgG antibodies. Here, we compare the performance of a conventional lateral flow assay (LFA) with a surface-enhanced Raman scattering (SERS)-based LFA test for the detection of SARS-CoV-2-specific IgM/IgG in sera of COVID-19 patients. Sensitive detection of IgM might enable early serological diagnosis of acute infections. Rapid detection in serum using a custom-built SERS reader is at least an order of magnitude more sensitive than the conventional LFAs with naked-eye detection. For absolute quantification and the determination of the limit of detection (LOD), a set of reference measurements using purified (total) IgM in buffer was performed. In this purified system, the sensitivity of SERS detection is even 7 orders of magnitude higher: the LOD for SERS was ca. 100 fg/mL compared to ca. 1 μg/mL for the naked-eye detection. This outlines the high potential of SERS-based LFAs in point-of-care testing once the interference of serum components with the gold conjugates and the nitrocellulose membrane is minimized.