Phthalate esters, which are known endocrine disruptors, are ubiquitously present throughout indoor environments. Leaching from building materials may be a major source of phthalate esters. In this study, we evaluated phthalate ester concentrations in dust samples from 64 classrooms located in 50 nursery schools and explored the critical factors affecting phthalate concentrations, especially with regard to building materials. Dust was sampled by a modified vacuuming method, and building materials were assessed using a portable X-ray fluorescence (XRF) analyzer to determine whether they contained polyvinyl chloride. Di-n-butyl phthalate (DBP), di(2-ethylhexyl) phthalate (DEHP), and di-isononyl phthalate (DINP) were the most frequently detected phthalates. Of these, DEHP was the most abundant phthalate, with a geometric mean of 3170 μg/g dust, and concentrations were significantly correlated with the area of polyvinyl chloride (PVC)-verified flooring. DINP, which has not been well-reported in other studies, was the second-most abundant phthalate, with a geometric mean of 688 μg/g dust, and showed a critical relationship with the number of children in the institution and the agency operating the nursery school. This is the first study to verify the sources of phthalates with an XRF analyzer and to evaluate the relationship between phthalate concentrations and PVC-verified materials.