We have fabricated all-dielectric high-Q optical pillar resonators with embedded colloidal CdSe/ZnS quantum dots or rods as light emitters by focused ion beam milling. Three-dimensional light confinement and distinct pillar microcavity modes are observed. Results from a waveguide model for the mode patterns and their spectral positions are in excellent agreement with the experimental data. Cavities with elliptical cross sections show higher quality factors in the short axis direction than do circular resonators of the same cross-sectional area.