Viruses interact with numerous host factors to facilitate viral replication and to dampen antiviral defense mechanisms. We currently have a limited mechanistic understanding of how SARS-CoV-2 binds host factors and the functional role of these interactions. Here, we uncover a novel interaction between the viral NSP3 protein and the fragile X mental retardation proteins (FMRPs: FMR1 and FXR1-2). SARS-CoV-2 NSP3 mutant viruses preventing FMRP binding have attenuated replication in vitro and have delayed disease onset in vivo. We show that a unique peptide motif in NSP3 binds directly to the two central KH domains of FMRPs and that this interaction is disrupted by the I304N mutation found in a patient with fragile X syndrome. NSP3 binding to FMRPs disrupts their interaction with the stress granule component UBAP2L through direct competition with a peptide motif in UBAP2L to prevent FMRP incorporation into stress granules. Collectively, our results provide novel insight into how SARS-CoV-2 hijacks host cell proteins for efficient infection and provides molecular insight to the possible underlying molecular defects in fragile X syndrome.