Solid-state nuclear magnetic resonance spectroscopy, X-ray diffraction, and Raman spectroscopy were used to show that Cu2ZnSnS4 (CZTS) bulk solids grown in the presence of oxygen had improved cation ordering compared to bulk solids grown without oxygen. Oxygen was shown to have negligible solubility in the CZTS phase. The addition of oxygen resulted in the formation of SnO2, leading to Sn-deficient CZTS. At the highest oxygen levels, other phases such as Cu9S5 and ZnS were observed. Beneficial ordering was only observed in samples produced with more than 2 at. % oxygen in the precursor materials but did not occur in samples designed with excess Sn and O. Thus, it is the removal of Sn and formation of Sn-deficient CZTS that improves ordering rather than the presence of SnO2 or O alone. These results indicate that using oxygen or air annealing to tailor the Sn content of CZTS followed by an etching step to remove SnO2 may significantly improve the properties of CZTS.