The properties of crystalline solids can be significantly modified by deliberately introducing point defects. Understanding these effects, however, requires understanding the changes in geometry and electronic structure of the host material. Here we report the effect of forming anion vacancies, via dehydroxylation, in a hexagonal tungsten-bronze-structured iron oxyfluoride, which has potential use as a lithium-ion battery cathode. Our combined pair distribution function and density functional theory analysis indicates that oxygen vacancy formation is accompanied by spontaneous rearrangement of fluorine anions and vacancies, producing dual pyramidal (FeF4)-O-(FeF4) structural units containing 5-fold-coordinated Fe atoms. The addition of lattice oxygen introduces new electronic states above the top of the valence band, with a corresponding reduction in the optical band gap from 4.05 to 2.05 eV. This band gap reduction relative to the FeF3 parent material is correlated with a significant improvement in lithium insertion capability relative to a defect-free compound.