This study aims to elucidate the aggregation and agglomeration behavior of TiO2 and ZrO2 nanoparticles during the nonaqueous synthesis. We found that zirconia nanoparticles immediately form spherical-like aggregates after nucleation with a homogeneous size of 200 nm, which can be related to the metastable state of the nuclei and the reduction of surface free energy. These aggregates further agglomerate, following a diffusion-limited colloid agglomeration mechanism that is additionally supported by the high fractal dimension of the resulting agglomerates. In contrast, TiO2 nanoparticles randomly orient and follow a reaction-limited colloid agglomeration mechanism that leads to a dense network of particles throughout the entire reaction volume. We performed in situ laser light transmission measurements and showed that particle formation starts earlier than previously reported. A complex population balance equation model was developed that is able to simulate particle aggregation as well as agglomeration, which eventually allowed us to distinguish between both phenomena. Hence, we were able to investigate the respective agglomeration kinetics with great agreement to our experimental data.