The impact of the chemical structure on ion transport, nanoscale morphology, and dynamics in polymerized imidazolium-based ionic liquids is investigated by broadband dielectric spectroscopy and X-ray scattering, complemented with atomistic molecular dynamics simulations. Anion volume is found to correlate strongly with Tg-independent ionic conductivities spanning more than 3 orders of magnitude. In addition, a systematic increase in alkyl side chain length results in about one decade decrease in Tg-independent ionic conductivity correlating with an increase in the characteristic backbone-to-backbone distances found from scattering and simulations. The quantitative comparison between ion sizes, morphology, and ionic conductivity underscores the need for polymerized ionic liquids with small counterions and short alkyl side chain length in order to obtain polymer electrolytes with higher ionic conductivity.