What determines the functional organization of cortex? One hypothesis is that innate connectivity patterns, either structural or functional connectivity, set up a scaffold upon which functional specialization can later take place. We tested this hypothesis by asking whether the visual word form area (VWFA), an experience-driven region, was already functionally connected to proto language networks in neonates scanned within one week of birth. Using the data from the Human Connectone Project (HCP) and the Developing Human Connectome Project (dHCP), we calculated intrinsic functional connectivity during resting-state functional magnetic resonance imaging (fMRI), and found that neonates showed similar functional connectivity patterns to adults. We observed that (1) language regions connected more strongly with the putative VWFA than other adjacent ventral visual regions that also show foveal bias, and (2) the VWFA connected more strongly with frontotemporal language regions than with regions adjacent to these language regions. These data suggest that the location of the VWFA is earmarked at birth due to its connectivity with the language network, providing evidence that innate connectivity instructs the later refinement of cortex.