A novel spectral method with variable transformation, the adaptive Hermite-Gauss decomposition method (A-HGDM), has been developed and applied to the analysis of three-dimensional (3D) dielectric structures. The proposed method includes an optimization strategy to automatically find the quasi-optimum numerical parameters of the variable transformation with low computational effort. The technique has been tested by analyzing two typical 3D dielectric structures: the rectangular step-index waveguide and the rib-waveguide directional coupler. In both cases, the A-HGDM increases the accuracy of the Hermite-Gauss decomposition method (HGDM), especially when the mode is near cutoff, and improves the computational efficiency of previously published optimization strategies (optimized HGDM).