Single photons are pivotal building blocks for photonic quantum technologies. Semiconductor quantum dots are promising candidates for optimal single photon sources in terms of purity, brightness and indistinguishability. Here we embed quantum dots into bullseye cavities with a backside dielectric mirror to enhance the collection efficiency up to near 90%. Experimentally, we achieve a collection efficiency of 30%. The auto-correlation measurements reveal a multiphoton probability below 0.05±0.005. A moderate Purcell factor of 3.1 is observed. Furthermore, we propose a scheme for laser integration as well as fiber coupling. Our results represent a step forward to the practical plug-and-play single photon sources.