Inorganic nanoparticles (NPs), particularly iron oxide (IO) and gold (Au) NPs, are widely used in a variety of biomedical applications, such as diagnosis and cancer therapy. As an important component of host defense in organisms, macrophages play a crucial role in responding to foreign substances, such as nanoparticles. Thus, it is of utmost importance to understand the nanotoxicity effects on the immune system by investigating the influences of such nanoparticles. In this study, we found that macrophages can take up large amounts of amphiphilic polymer (PMA)-modified Au and IO NPs, which will induce macrophage cell vacuolization and enhance macrophage polarization. This mechanism is an essential part of the immune response in vivo. In addition, we report that smaller-sized nanoparticles (ca. 4 nm) show more significant effects on the macrophage polarization and caused lysosomal damage compared to larger nanoparticles (ca. 14 nm). Moreover, the amount of NP uptake in macrophages decreases upon trapping the PMA with PEG, resulting in reduced vacuolization and a reduced immune response. We hypothesize that vacuoles are formed in large amounts during NP uptake by macrophages, which enhances the immune response and induces macrophages toward M1 polarization. These findings are potentially useful for disease treatment and understanding the immune response when NPs are used in vitro and in vivo.