We demonstrate colloidal, layer-by-layer growth of metal oxide shells on InP quantum dots (QDs) at room temperature. We show with computational modeling that native InP QD surface oxides give rise to nonradiative pathways due to the presence of surface-localized dark states near the band edges. Replacing surface indium with zinc to form a ZnO shell results in reduced nonradiative decay and a density of states at the valence band edge that resembles defect-free, stoichiometric InP. We then developed a synthetic strategy using stoichiometric amounts of common atomic layer deposition precursors in alternating cycles to achieve layer-by-layer growth. Metal-oxide-shelled InP QDs show bulk and local structural perturbations as determined by X-ray diffraction and extended X-ray absorption fine structure spectroscopy. Upon growing ZnSe shells of varying thickness on the oxide-shelled QDs, we observe increased photoluminescence (PL) quantum yields and narrowing of the emission linewidths that we attribute to decreased ion diffusion to the shell, as supported by phosphorus X-ray emission spectroscopy. These results present a versatile strategy to control QD interfaces for novel heterostructure design by leveraging surface oxides. This work also contributes to our understanding of the connections between structural complexity and PL properties in technologically relevant colloidal optoelectronic materials.