Atomic layer deposition (ALD) is one of the backbones for today's electronic device fabrication. A critical property of ALD is the layer-by-layer growth, which gives rise to the atomic-scale accuracy. However, the growth rate - or growth per cycle - can differ significantly depending on the type of system, molecules used, and several other experimental parameters. Typically, ALD growth rates are constant in subsequent ALD cycles, making ALD an outstanding deposition technique. However, contrary to this steady-state - when the ALD process can be entirely decoupled from the substrate on which the material is grown - the deposition's early stage does not appear to follow the same kinetics, chemistry, and growth rate. Instead, it is to a large extent determined by the surface composition of the substrate. Here, we present evidence of oxygen relocation from the substrate-based oxide, either the thermal or native oxide of InAs, to the overlayer of HfO2 in the initial ALD phase. This phenomenon enables control of the thickness of the initial ALD layer by controlling the surface conditions of the substrate prior to ALD. On the other hand, we observe a complete removal of the native oxide from InAs already during the first ALD half-cycle, even if the thickness of the oxide layer exceeds one monolayer, together with a self-limiting thickness of the ALD layer of a maximum of one monolayer of HfO2. These observations not only highlight several limitations of the widely used ligand exchange model, but they also give promise for a better control of the industrially important self-cleaning effect of III-V semiconductors, which is crucial for future generation high-speed MOS.