Systematic experimental and theoretical research on the role of microstructure and interface thermal resistance on the thermal conductivity of the PbTe-CoSb3 bulk polycrystalline composite is presented. In particular, the correlation between the particle size of the dispersed phase and interface thermal resistance (Rint) on the phonon thermal conductivity (κph) is discussed. With this aim, a series of PbTe-CoSb3 polycrystalline composite materials with different particle sizes of CoSb3 was prepared. The structural (XRD) and microstructural analysis (SEM/EDXS) confirmed the intended chemical and phase compositions. Acoustic impedance difference (ΔZ) was determined from measured sound velocities in PbTe and CoSb3 phases. It is shown that κph of the composite may be reduced when particle size of the dispersed phase (CoSb3) is smaller than the critical value of ∼230 nm. This relationship was concluded to be crucial for controlling the heat transport phenomena in composite thermoelectric materials. The selection of the components with different elastic properties (acoustic impedance) and particle size smaller than Kapitza radius leads to a new direction in the engineering of composite TE materials with designed thermal properties.