The infrapatellar fat pad (IPFP) and synovium play essential roles in maintaining knee joint homeostasis and in the progression of osteoarthritis (OA). The cellular and transcriptional mechanisms regulating the function of these specialized tissues under healthy and diseased conditions are largely unknown. Here, single-cell and single-nuclei RNA sequencing of human IPFP and synovial tissues were performed to elucidate the cellular composition and transcriptional profile. Computational trajectory analysis revealed that dipeptidyl peptidase 4+ mesenchymal cells function as a common progenitor for IPFP adipocytes and synovial lining layer fibroblasts, suggesting that IPFP and synovium represent an integrated tissue unit. OA induced a profibrotic and inflammatory phenotype in mesenchymal lineage cells with biglycan+ intermediate fibroblasts as a major contributor to OA fibrosis. Apolipoprotein E (APOE) signaling from intermediate fibroblasts and macrophages was identified as a critical regulatory factor. Ex vivo incubation of human cartilage with soluble APOE accelerated proteoglycan degeneration. Inhibition of APOE signaling by intra-articular injection of an anti-APOE neutralizing antibody attenuated the progression of collagenase-induced OA in mice, demonstrating a detrimental effect of APOE on cartilage. Our studies provide a framework for designing further therapeutic strategies for OA by describing the cellular and transcriptional landscape of human IPFP and synovium in healthy versus OA joints.