To improve color conversion performance for color display application, we study the near-field-induced nanoscale-cavity effects on the emission efficiency and Förster resonance energy transfer (FRET) under the condition of surface plasmon (SP) coupling by inserting colloidal quantum dots (QDs) and synthesized Ag nanoparticles (NPs) into surface nano-holes fabricated on a GaN template and an InGaN/GaN quantum-well (QW) template. In the QW template, the inserted Ag NPs are close to either QWs or QDs for producing three-body SP coupling to enhance color conversion. Time-resolved and continuous-wave photoluminescence (PL) behaviors of the QW- and QD-emitting lights are investigated. The comparison between the nano-hole samples and the reference samples of surface QD/Ag NP shows that the nanoscale-cavity effect of the nano-hole leads to the enhancements of QD emission, FRET between QDs, and FRET from QW into QD. The SP coupling induced by the inserted Ag NPs can enhance the QD emission and FRET from QW into QD. Its result is further enhanced through the nanoscale-cavity effect. The relative continuous-wave PL intensities among different color components also show the similar behaviors. By introducing SP coupling to a color conversion device with the FRET process in a nanoscale cavity structure, we can significantly improve the color conversion efficiency. Simulation results confirm the basic observations in experiment.