Adequate mathematical competencies are currently indispensable in professional and social life. However, mathematics is often associated with stress and frustration and the confrontation with tasks that require mathematical knowledge triggers anxiety in many children. We examined if there is a relationship between math anxiety and changes in brain structure in children with and without developmental dyscalculia. Our findings showed that math anxiety is related to altered brain structure. In particular, the right amygdala volume was reduced in individuals with higher math anxiety. In conclusion, math anxiety not only hinders children in arithmetic development, but it is associated with altered brain structure in areas related to fear processing. This emphasizes the far-reaching outcome emotional factors in mathematical cognition can have and encourages educators and researchers alike to consider math anxiety to prevent detrimental long-term consequences on school achievement and quality of life, especially in children with developmental dyscalculia.