Preclinical models of Alzheimer's disease (AD) suggest APOE modulates brain function in structures vulnerable to AD pathophysiology. However, genome-wide association studies now demonstrate that AD risk is shaped by a broader polygenic architecture, estimated via polygenic risk scoring (AD-PRS). Despite this breakthrough, the effect of AD-PRS on brain function in young individuals remains unknown. In a large sample (N = 608) of young, asymptomatic individuals, we measure the impact of both (i) APOE and (ii) AD-PRS on a vulnerable cortico-limbic scene-processing network heavily implicated in AD pathophysiology. Integrity of this network, which includes the hippocampus (HC), is fundamental for maintaining cognitive function during ageing. We show that AD-PRS, not APOE, selectively influences activity within the HC in response to scenes, while other perceptual nodes remained intact. This work highlights the impact of polygenic contributions to brain function beyond APOE, which could aid potential therapeutic/interventional strategies in the detection and prevention of AD.