We investigate the third-order nonlinear optical properties of epsilon-near-zero (ENZ) Au/dye-doped fused silica multilayered metamaterials in the visible spectral range for TM incident by using nonlocal effective medium theory at different incidence angles. The nonlocal response affects the permittivity of anisotropic metamaterials when the thickness of the layer cannot be much smaller than the incident wavelength. By doping pump dye gain material within the dielectric layer to compensate for the metal loss, the imaginary part of the effective permittivity is reduced to 10-4, and the optical nonlinear refractive index and nonlinear absorption coefficient are enhanced. The real and imaginary parts of the permittivity are simultaneously minimized when the central emission wavelength of the gain material is close to the ENZ wavelength, and the nonlinear refraction coefficient reaches the order of 10-5 cm2/W, which is five orders of magnitude larger than that of the nonlinear response of the metamaterial without the gain medium. Our results demonstrate that a smaller imaginary part of the permittivity can be obtained by doping gain materials within the dielectric layer; it offers the promise of designing metamaterials with large nonlinearity at arbitrary wavelengths.
Keywords: epsilon-near-zero (ENZ); gain materials; multilayered metamaterials; nonlocal effective medium theory; third-order nonlinear.