Mixed-valent transition metal compounds display complex structural, electronic and magnetic properties which can often be exquisitely tuned. Here the charge-ordered state of stoichiometric CaFe3O5 is probed using neutron powder diffraction, Monte Carlo simulation and symmetry analysis. Magnetic ordering is dominated by the formation of ferromagnetic Fe3+-Fe2+-Fe3+ trimers which are evident above the magnetic ordering transition. Between TN = 289 K and 281 K an incommensurate magnetically ordered phase develops due to magnetic frustration, but a spin Jahn-Teller distortion lifts the frustration and enables the magnetic ordering to lock in to a charge-ordered commensurate state at lower temperatures. Stoichiometric CaFe3O5 exhibits single phase behaviour throughout and avoids the phase separation into two distinct crystallographic phases with different magnetic structures and Fe valence distributions reported recently, which likely occurs due to partial Fe2+ for Ca2+ substitution. This underlines the sensitivity of the magnetism and chemistry of these mixed-valent systems to composition.