Non-polar magnetic nanoparticles agglomerate upon cooling. This process is followed by in situ small angle X-ray scattering to assess structural properties of the emerging agglomerates. On the length scale of a few particle diameters, no differences are found between the agglomerates of small (d = 12 nm) and large (d = 22 nm) nanoparticles. Hard-sphere like random packing with a local packing fraction of η = 0.4 is seen. On larger length scales, small particles form compact superstructures, while large particles arrange into agglomerates that resemble chain-like structures in SAXS. This can be explained by directed magnetic dipole interactions that dominate larger particles, while isotropic van der Waals interaction governs the agglomeration of smaller particles.