Production studies have shown that speakers of languages with larger phoneme inventories expand their acoustic space relative to languages with smaller inventories [Bradlow, A. (1995). A comparative acoustic study of English and Spanish vowels. Journal of the Acoustical Society of America, 97(3), 1916-1924; Jongman, A., Fourakis, M., & Sereno, J. (1989). The acoustic vowel space of Modern Greek and German. Language Speech, 32, 221-248]. In this study, we investigated whether this acoustic expansion in production has a perceptual correlate, that is, whether the perceived distance between pairs of sounds separated by equal acoustic distances varies as a function of inventory size or organization. We used magnetoencephalography, specifically the mismatch field response (MMF), and compared two language groups, French and Spanish, whose vowel inventories differ in size and organization. Our results show that the MMF is sensitive to inventory size but not organization, suggesting that speakers of languages with larger inventories perceive the same sounds as less similar than speakers with smaller inventories.