The effect of the structural inhomogeneity and oxygen defects on the thermal conductivity of polycrystalline aluminum nitride (AlN) thin films deposited on single-crystal silicon substrates is experimentally and theoretically investigated. The influence of the evolution of crystal structure, grain size, and out-of plane disorientation along the cross plane of the films on their thermal conductivity is analyzed. The impact of oxygen-related defects on thermal conduction is studied in AlN/AlN multilayered samples. Microstructure, texture, and grain size of the films were characterized by X-ray diffraction and scanning and transmission electron microscopy. The measured thermal conductivity obtained with the 3-omega technique for a single and multiple layers of AlN is in fairly good agreement with the theoretical predictions of our model, which is developed by considering a serial assembly of grain distributions. An effective thermal conductivity of 5.92 W m(-1) K(-1) is measured for a 1107.5 nm-thick multilayer structure, which represents a reduction of 20% of the thermal conductivity of an AlN monolayer with approximately the same thickness, due to oxygen impurities at the interface of AlN layers. Our results show that the reduction of the thermal conductivity as the film thickness is scaled down, is strongly determined by the structural inhomogeneities inside the sputtered films. The origin of this non-homogeneity and the effect on phonon scattering are also discussed.