Elucidating the influences of embodiment and conceptual metaphor on lexical and non-speech tone learning

Abstract

In the contexts of language learning and music processing, hand gestures conveying acoustic information visually influence perception of speech and non-speech sounds (Connell et al., 2013; Morett & Chang, 2015). Currently, it is unclear whether this effect is due to these gestures' use of the human body to highlight relevant features of language (embodiment) or the cross-modal mapping between the visual motion trajectories of these gestures and corresponding auditory features (conceptual metaphor). To address this question, we examined identification of the pitch contours of lexical tones and non-speech analogs learned with pitch gesture, comparable dot motion, or no motion. Critically, pitch gesture and dot motion were either congruent or incongruent with the vertical conceptual metaphor of pitch. Consistent with our hypotheses, we found that identification accuracy increased for tones learned with congruent pitch gesture and dot motion, whereas it remained stable or decreased for tones learned with incongruent pitch gesture and dot motion. These findings provide the first evidence that both embodied and non-embodied visual stimuli congruent with the vertical conceptual metaphor of pitch enhance lexical and non-speech tone learning. Thus, they illuminate the influences of conceptual metaphor and embodiment on lexical and non-speech auditory perception, providing insight into how they can be leveraged to enhance language learning and music processing.

Keywords: Conceptual metaphor; Embodied cognition; Gesture; Lexical tone; Multisensory integration.