Temporal dynamics of audiovisual affective processing

Abstract

This study used event-related potentials (ERPs) to investigate the time course of auditory, visual, and audiovisual affective processing. Stimuli consisted of naturalistic silent videos, instrumental music clips, or combination of the two, with valence varied at three levels for each modality and arousal matched across valence conditions. Affective ratings of the unimodal and multimodal stimuli showed evidence of visual dominance, congruency, and negativity dominance effects. ERP results for unimodal presentations revealed valence effects in early components for both modalities, but only for the visual condition in a late positive potential. The ERP results for multimodal presentations showed effects for both visual valence and auditory valence in three components, early N200, P300 and LPP. A modeling analysis of the N200 component suggested its role in the visual dominance effect, which was further supported by a correlation between behavioral visual dominance scores and the early ERP components. Significant congruency comparisons were also found for N200 amplitudes, suggesting that congruency effects may occur early. Consistent differences between negative and positive valence were found for both visual and auditory modalities in the P300 at anterior electrode clusters, suggesting a potential source for the negativity dominance effect observed behaviorally. The separation between negative and positive valence also occurred at LPP for the visual modality. Significant auditory valence modulation was found for the LPP, implying an integration effect in which valence sensitivity of the LPP emerged for the audiovisual condition. These results provide a basis for mapping out the temporal dynamics of audiovisual affective processing.

Keywords: Affect; Audiovisual; Congruency; ERP; Negativity dominance; Visual dominance.