We present an asymmetric encryption scheme for hyperspectral images using hybrid chaotic maps (HCMs) and an equal modulus decomposition tree (EMDT) structure in a discrete multiple-parameter fractional Fourier transform (dmpFrFT) domain. The original hyperspectral image was scrambled by an HCM and then encrypted into asymmetric ciphertext using the EMDT. In the EMDT, each pair of the band images of the scrambled hyperspectral image were regarded as leaf nodes, while the encryption modules using chaotic random phase mask, dmpFrFT, and improved equal modulus decomposition were regarded as branch nodes, and the encryption process was implemented along the paths from the leaf nodes to the topmost branch node. The EMDT structure could provide multiparameter encryption, real-valued output, and different pairs of band images with different secret keys and encryption/decryption paths. Compared with the previous optical encryption approaches for hyperspectral images, our asymmetric cryptosystem had larger key space, less data amount of storage and transmission, and stronger resistance to statistical attacks. Various numerical simulations verified the performance of our proposed asymmetric cryptosystem.