Students' academic performance has been routinely assessed in various subjects, including arts education. The current study uses Rasch methodology to investigate item quality for an annual state-wide arts assessment program administered to 4th grade students. All multiple-choice items were previously analyzed through the true score theory (TST) framework to examine item difficulty, differential item functioning (DIF), and distractor quality. However, these traditional methods are sample-specific, and score interpretations are limited to the particular group being tested. Rasch methodology provides a sample-free framework for item analysis. This approach has the advantage of producing sample-invariant item parameters and using goodness-of-fit criteria to detect problematic items, leading to more accurate item analysis results. Study results suggest that majority of the items performed well and the test was appropriate to its intended audience and evaluation purpose. It also validates the test score interpretation and the use of this assessment program.