We studied the temperature dependence of the structural relaxation in poly(vinyl acetate) near the glass transition temperature with single molecule spectroscopy from Tg-1 K to Tg+12 K. The temperature dependence of the observed relaxation times matches results from bulk experiments; the observed relaxation times are, however, 80-fold slower than those from bulk experiments at the same temperature. We attribute this factor to the size of the probe molecule. The individual relaxation times of the single molecule environments are distributed normally on a logarithmic time scale, confirming that the dynamics in poly(vinyl acetate) is heterogeneous. The width of the distribution of individual relaxation times is essentially independent of temperature. The observed full width at half maximum (FWHM) on a logarithmic time axis is approximately 0.7, corresponding to a factor of about 5-fold, significantly narrower than the dielectric spectrum of the same material with a FWHM of about 2.0 on a logarithmic time axis, corresponding to a factor of about 100-fold. We explain this narrow width as the effect of temporal averaging of single molecule fluorescence signals over numerous environments due to a limited lifetime of the probed heterogeneities, indicating that heterogeneities are dynamic. We determine a loose upper limit for the ratio of the structural relaxation time to the lifetime of the heterogeneities (the rate memory parameter) of Q<80 for the range of investigated temperatures.