Older adults often have difficulty understanding speech in a noisy environment or with multiple speakers. In such situations, binaural hearing improves the signal-to-noise ratio. How does this binaural advantage change with increasing age? Using magnetoencephalography, we recorded cortical activity evoked by changes in interaural phase differences of amplitude-modulated tones. These responses occurred for frequencies up to 1225 Hz in young subjects but only up to 940 Hz in middle-aged and 760 Hz in older adults. Behavioral thresholds also decreased with increasing age but were more variable, likely because some older adults make effective use of compensatory mechanisms. The reduced frequency range for binaural hearing became significant in middle age, before decline in hearing sensation and the morphology of cortical responses, which became apparent only in the older subjects. This study provides evidence from human physiological data for the early onset of biological aging in binaural hearing.