Hearing facial identities: brain correlates of face--voice integration in person identification

Abstract

Audiovisual integration (AVI) is a well-known aspect of speech perception, but integration of facial and vocal information is also important for speaker recognition. We recently demonstrated AVI in the recognition of familiar (but not unfamiliar) speakers. Specifically, systematic behavioural benefits and costs in recognizing a familiar voice occur when the voice is combined with a time-synchronised articulating face of corresponding or noncorresponding speaker identity, respectively (Schweinberger et al., 2007; Robertson and Schweinberger, 2010). Here we report an experiment assessing event-related brain potentials (ERPs) in this novel paradigm, while participants recognized familiar speakers presented in (1) Voice only, (2) voice with identity-corresponding and (3) noncorresponding time-synchronised speaking faces, as well as (4) Face only conditions. Audiovisual speaker identity correspondence influenced only later ERPs around 250-600 msec, with increased negativity for noncorresponding identities at central electrodes. Strikingly, when compared with the ERPs from both unimodal conditions, both audiovisual conditions led to a much earlier onset of frontocentral negativity, with maximal differences around 50-80 msec. Moreover, audiovisual stimuli elicited larger N170 responses than Face only stimuli. These findings suggest that the perception of a voice and a time-synchronised articulating face triggers remarkably early and mandatory mechanisms of audiovisual processing, although the correspondence or discrepancy in audiovisual speaker identity may only be computed ∼200msec later.