Immigrants might be perceived as a threat to a country's jobs, security, and cultural identity. In this study, we aimed to test whether individuals with higher numerical, scientific, and economic literacy were more polarized in their perception of immigration, depending on their cultural worldview orientation. We measured these variables in a representative sample of citizens in a medium-sized city in northern Italy. We found evidence that numerical, scientific, and economic literacy polarize concerns about immigration aligning them to people's worldview orientations. Individuals with higher numerical, economic, and scientific literacy were less concerned about immigration if they held an egalitarian-communitarian worldview, while they were more concerned about immigration if they held a hierarchical-individualistic worldview. On the contrary, individuals with less numerical, economic, and scientific literacy did not show a polarized perception of immigration. Results reveal that citizens with higher knowledge and ability presented a more polarized perception of immigration. Conclusions highlight the central role of cultural worldviews over information theories in shaping concerns about immigration.