Core/shell nanowire (NW) heterostructures based on III-V semiconductors and related alloys are attractive for optoelectronic and photonic applications owing to the ability to modify their electronic structure via bandgap and strain engineering. Post-growth thermal annealing of such NWs is often involved during device fabrication and can also be used to improve their optical and transport properties. However, effects of such annealing on alloy disorder and strain in core/shell NWs are not fully understood. In this work we investigate these effects in novel core/shell/shell GaAs/GaNAs/GaAs NWs grown by molecular beam epitaxy on (111) Si substrates. By employing polarization-resolved photoluminescence measurements, we show that annealing (i) improves overall alloy uniformity due to suppressed long-range fluctuations in the N composition; (ii) reduces local strain within N clusters acting as quantum dot emitters; and (iii) leads to partial relaxation of the global strain caused by the lattice mismatch between GaNAs and GaAs. Our results, therefore, underline applicability of such treatment for improving optical quality of NWs from highly-mismatched alloys. They also call for caution when using ex-situ annealing in strain-engineered NW heterostructures.