Owing to their intrinsically low thermal conductivity and chemical diversity, materials within the I-V-VI2 family, and especially AgBiSe2, have recently attracted interest as promising thermoelectric materials. However, further investigations are needed in order to develop a more fundamental understanding of the origin of the low thermal conductivity in AgBiSe2, to evaluate possible stereochemical activity of the 6s2 lone pair of Bi3+, and to further elaborate on chemical design approaches for influencing the occurring phase transitions. In this work, a combination of temperature-dependent X-ray diffraction, Rietveld refinements of laboratory X-ray diffraction data, and pair distribution function analyses of synchrotron X-ray diffraction data is used to tackle the influence of Sb substitution within AgBi1-xSbxSe2 (0 ⩽ x ⩽ 0.15) on the phase transitions, local distortions, and off-centering of the structure. This work shows that, similar to other lone-pair-containing materials, local off-centering and distortions can be found in AgBiSe2. Furthermore, electronic and thermal transport measurements, in combination with the modeling of point-defect scattering, highlight the importance of structural characterizations toward understanding changes induced by elemental substitutions. This work provides new insights into the structure-transport correlations of the thermoelectric AgBiSe2.