The anharmonicity of atomic motion limits the thermal conductivity in crystalline solids. However, a microscopic understanding of the mechanisms active in strong thermal insulators is lacking. In this Letter, we classify 465 experimentally known materials with respect to their anharmonicity and perform fully anharmonic ab initio Green-Kubo calculations for 58 of them, finding 28 thermal insulators with κ<10 W/mK including 6 with ultralow κ≲1 W/mK. Our analysis reveals that the underlying strong anharmonic dynamics is driven by the exploration of metastable intrinsic defect geometries. This is at variance with the frequently applied perturbative approach, in which the dynamics is assumed to evolve around a single stable geometry.