BAs was predicted to have an unusually high thermal conductivity with a room temperature value of 2000 W m^{-1} K^{-1}, comparable to that of diamond. However, the experimentally measured thermal conductivity of BAs single crystals is still lower than this value. To identify the origin of this large inconsistency, we investigate the lattice structure and potential defects in BAs single crystals at the atomic scale using aberration-corrected scanning transmission electron microscopy (STEM). Rather than finding a large concentration of As vacancies (V_{As}), as widely thought to dominate the thermal resistance in BAs, our STEM results show an enhanced intensity of some B columns and a reduced intensity of some As columns, suggesting the presence of antisite defects with As_{B} (As atom on a B site) and B_{As} (B atom on an As site). Additional calculations show that the antisite pair with As_{B} next to B_{As} is preferred energetically among the different types of point defects investigated and confirm that such defects lower the thermal conductivity for BAs. Using a concentration of 1.8(8)% (6.6±3.0×10^{20} cm^{-3} in density) for the antisite pairs estimated from STEM images, the thermal conductivity is estimated to be 65-100 W m^{-1} K^{-1}, in reasonable agreement with our measured value. Our study suggests that As_{B}-B_{As} antisite pairs are the primary lattice defects suppressing thermal conductivity of BAs. Possible approaches are proposed for the growth of high-quality crystals or films with high thermal conductivity. Employing a combination of state-of-the-art synthesis, STEM characterization, theory, and physical insight, this work models a path toward identifying and understanding defect-limited material functionality.