A computational lens into how music characterizes genre in film

PLoS One. 2021 Apr 8;16(4):e0249957. doi: 10.1371/journal.pone.0249957. eCollection 2021.

Abstract

Film music varies tremendously across genre in order to bring about different responses in an audience. For instance, composers may evoke passion in a romantic scene with lush string passages or inspire fear throughout horror films with inharmonious drones. This study investigates such phenomena through a quantitative evaluation of music that is associated with different film genres. We construct supervised neural network models with various pooling mechanisms to predict a film's genre from its soundtrack. We use these models to compare handcrafted music information retrieval (MIR) features against VGGish audio embedding features, finding similar performance with the top-performing architectures. We examine the best-performing MIR feature model through permutation feature importance (PFI), determining that mel-frequency cepstral coefficient (MFCC) and tonal features are most indicative of musical differences between genres. We investigate the interaction between musical and visual features with a cross-modal analysis, and do not find compelling evidence that music characteristic of a certain genre implies low-level visual features associated with that genre. Furthermore, we provide software code to replicate this study at https://github.com/usc-sail/mica-music-in-media. This work adds to our understanding of music's use in multi-modal contexts and offers the potential for future inquiry into human affective experiences.

Publication types

  • Comparative Study
  • Research Support, Non-U.S. Gov't

MeSH terms

  • Humans
  • Motion Pictures / classification*
  • Music / psychology*
  • Smart Glasses
  • Software
  • Supervised Machine Learning
  • Visual Perception

Grants and funding

The study was done at the Center for Computational Media Intelligence at USC, which is supported by a research award from Google. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.