In this study, we assessed the factor structure and construct validity of the parent-reported Inventory of Callous-Unemotional Traits (ICU) among school-aged children and adolescents, aged 6 to 15 years, in a community setting in Japan (n = 10,936). We investigated 15 models that have been reported in previous studies and used confirmatory factor analyses to determine a model that might actually be the best-fit among these. We then examined the correlations between the score of ICU and the Strengths and Difficulties Questionnaire (SDQ) in the best fit model and the three-factor bifactor (3FBF) model with the original ICU through cross-sectional and longitudinal analysis to determine the concurrent and predictive validity of the ICU. The results showed that the best-fit model was the two-factor bifactor (2FBF) model with a revised version of the ICU with 12 items, excluding all but one item of unemotional factors. The cross-sectional and longitudinal analysis showed that higher general callous-unemotional factor scores, callousness and uncaring specific factor scores were significantly associated with a higher level of conduct problems and a lower level of prosocial behaviors in the SDQ. These tendencies were shown both in the 2FBF model with the revised version of the ICU and the 3FBF model with the original ICU. We conclude that the 2FBF model was useful for school-aged community samples, as it predicts increases in conduct problems and decreases in prosocial behavior with fewer items than the 3FBF model.