AgBiSe2 has recently been shown to exhibit promising thermoelectric properties due to the low intrinsic thermal conductivity, stemming from a large degree of lattice anharmonicity. While samples synthesized via solid-state routes usually exhibit n-type behavior, p-type transport is seen in samples based on solution synthetic routes possibly due to Ag vacancies. Using a combined approach of synchrotron diffraction, thermoelectric transport measurements and thermal transport modeling, we show the influence of synthetically induced Ag vacancies on the structure of AgBiSe2 and the thermoelectric transport. We identify the degree of anti-site disorder of Ag and Bi due to the occurring phase transformation and the influence of the vacancy content on metal ordering. Additionally, we show that anti-site disorder and vacancies act as scattering centers for phonons, leading to enhanced point defect scattering in this interesting thermoelectric material.